793dd1b257
This very simple program reads from `/dev/kmsg` and logs the output. Signed-off-by: David Scott <dave.scott@docker.com>
4.0 KiB
Logging
By default LinuxKit will write onboot and service logs directly to files in
/var/log and /var/log/onboot.
It is tricky to write the logs to a disk or a network service as no disks
or networks are available until the onboot containers run. We work around
this by splitting the logging into 2 pieces:
memlogd: an in-memory circular buffer which receives logs (including all the earlyonbootlogs)- a log writing
servicethat starts later and can download and process the logs frommemlogd
To use this new logging system, you should add the memlogd container to
the init block in the LinuxKit yml. On boot memlogd will be started
from init.d and it will listen on a Unix domain socket:
/var/run/linuxkit-external-logging.sock
The init/service process will look for this socket and redirect the
stdout and stderr of both onboot and services to memlogd.
memlogd: an in-memory circular buffer
The memlogd daemon reads the logs from the onboot and services containers
and stores them together with a timestamp and the name of the originating
container in a circular buffer in memory.
The contents of the circular buffer can be read over the Unix domain socket
/var/run/memlogq.sock
The circular buffer has a fixed size (overridden by the command-line argument
-max-lines) and when it fills up, the oldest messages will be overwritten.
To store the logs somewhere more permanent, for example a disk or a remote
network service, a service should be added to the yaml which connects to
memlogd and streams the logs. The logwrite service described below shows
how to do this.
Message format
The format used to read logs is similar to kmsg:
<timestamp>,<log>;<body>
where <timestamp> is an RFC3339-formatted timestamp, <log> is the name of
the log (e.g. docker-ce.out) and <body> is the output. The <log> must
not contain the character ;.
logwrite: writing logs to disk
The service pkg/logwrite connects to memlogd and streams the logs to files
in /var/log. The logs are automatically rotated; by default each file has
a maximum size of 1 MiB and up to 10 files are kept per log. The arguments
-max-log-files and -max-log-size can be used to override these defaults.
Here is an example log file:
# cat /var/log/onboot.001-dhcpcd.out
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: waiting for carrier
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: carrier acquired
2018-07-08T09:16:53Z onboot.001-dhcpcd.out DUID 00:01:00:01:22:d4:93:05:02:50:00
:00:00:06
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: IAID 00:00:00:06
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: adding address fe80::f346:56a6:590d:5ea4
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: soliciting an IPv6 router
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: soliciting a DHCP lease
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: offered 192.168.65.8 from 192.168.65.1 `vpnkit'
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: leased 192.168.65.8 for 7200 se
conds
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: adding route to 192.168.65.0/24
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: adding default route via 192.16
8.65.1
2018-07-08T09:16:53Z onboot.001-dhcpcd.out exiting due to oneshot
2018-07-08T09:16:53Z onboot.001-dhcpcd.out dhcpcd exited
Current issues and limitations:
- No docker logger plugin support yet - it could be nice to add support to memlogd, so the docker container logs would also be gathered in one place
- No syslog compatibility at the moment and
/dev/logdoesn’t exist. This socket could be created to keep syslog compatibility, e.g. by using https://github.com/mcuadros/go-syslog. Processes that require syslog should then be able to log directly to memlogd. - Currently no direct external hooks exposed - but options available that
could be added. Should also be possible to pipe output to e.g.
oklogfromlogread(https://github.com/oklog/oklog)