ta-ching chen

2 minute read

 Table of Contents

Story 

I ran log parser within docker containers to parse log file that mount from the host, and everything goes well until daily log rotation. Container seems failed to mount the correct file foo.log but foo.log.1.

Container mounts the old log file

Try It Yourself 

You can try it with following steps

  • docker-compose.yml
dummy-log-parser:
  image: alpine:latest
  volumes:
    - /tmp/foo.log:/tmp/foo.log:ro
  command:
    - "sleep"
    - "3600"
  • execute following commands
$ touch /tmp/foo.log
$ docker-compose up -d
$ echo 'foo' >> /tmp/foo.log
  • check file content in container, and you should see foo in foo.log
$ docker exec -it <container> sh 
$ cat /tmp/foo.log
foo
  • the tricky part
# on host
$ mv /tmp/foo.log /tmp/foo.log.1
$ touch /tmp/foo.log
$ echo 'foo bar foo bar' >> /tmp/foo.log
$ cat /tmp/foo.log
foo bar foo bar
$ echo 'old log file' >> /tmp/foo.log.1
$ cat /tmp/foo.log.1
foo
old log file
# in container
$ cat /tmp/foo.log
foo
old log file

WHAT?

What happened 

The filesystem has an inode table that stores the mapping of file and inode number. When container ups, the log file is mounted into the container with specific inode number. Even the file name is changed on host, the inode number remains the same. That’s why the file in container still points to the old log file.

Solution 

Mount directory rather than mounting file!

dummy-log-parser:
  image: alpine:latest
  volumes:
    - /tmp/:/tmp/:ro
  command:
    - "sleep"
    - "3600"

Reference 

comments powered by Disqus