Merge pull request #1640 from MagnusS/memlogd-exp

RFC: Prototype for system logs

projects/README.md

@@ -19,6 +19,7 @@ If you want to create a project, please submit a pull request to create a new di
 - [Swarmd](swarmd) Standalone swarmkit based orchestrator
 - [Landlock LSM](landlock/) programmatic access control
 - [Clear Containers](clear-containers/) Clear Containers image
+- [Logging](logging/) Experimental logging tools
 
 ## Current projects not yet documented
 - VMWare support (VMWare)

projects/logging/README.md

@@ -0,0 +1,54 @@
+### Logging tools
+
+Experimental logging tools for linuxkit. 
+
+This project currently provides three tools for system logs; `logwrite`, `logread` and `memlogd` (+ `startmemlogd` to run `memlogd` with `runc`).
+
+`memlogd` is the daemon that keeps logs in a circular buffer in memory. It is started automatically by `init`/`startmemlogd` in a runc container. It is passed two sockets - one that allows clients to dump/follow the logs and one that can be used to send open file descriptors to `memlogd`. When `memlogd` receives a file descriptor it will read from the file descriptor and timestamp and append the content to the in-memory log until the file is closed.
+
+`logwrite` executes a command and will send stderr and stdout to `memlogd`. It does this by opening a socketpair for stdout and stderr and then sends the file descriptors to memlogd, before executing a specified command. Output is also sent to normal stderr/stdin. For example, `logwrite ls` will show the output both in the console and record it in the logs.
+
+`logread` connects to memlogd and dumps the ring buffer. Parameters `-f` and `-F` can be used to follow the logs and disable the initial log dump (it behaves similar to busybox’ `logread`)
+
+Init is modified to run all `onboot` and `service` containers wrapped in`logwrite` and to run `/usr/bin/startmemlogd`.
+
+New sockets:
+`/tmp/memlogd.sock` — sock_dgram which accepts an fd and a null-terminated source description
+`/tmp/memlogdq.sock` — sock_stream to ask to dump/follow logs
+
+Usage examples:
+```
+/ # logread -f
+2017-04-15T15:37:37Z memlogd memlogd started
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: waiting for carrier
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: carrier acquired
+2017-04-15T15:37:37Z 002-dhcpcd.stdout DUID 00:01:00:01:20:84:fa:c1:02:50:00:00:00:24
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: IAID 00:00:00:24
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: adding address fe80::84e3:ca52:2590:fe80
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: soliciting an IPv6 router
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: soliciting a DHCP lease
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: offered 192.168.65.37 from 192.168.65.1 `vpnkit'
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: leased 192.168.65.37 for 7199 seconds
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: adding route to 192.168.65.0/24
+2017-04-15T15:37:37Z 002-dhcpcd.stdout eth0: adding default route via 192.168.65.1
+2017-04-15T15:37:37Z 002-dhcpcd.stdout exiting due to oneshot
+2017-04-15T15:37:37Z 002-dhcpcd.stdout dhcpcd exited
+2017-04-15T15:37:37Z rngd.stderr Unable to open file: /dev/tpm0
+^C
+/ # logwrite echo testing123
+testing123
+/ # logread | tail -n1
+2017-04-15T15:37:45Z echo.stdout testing123
+/ # echo -en "GET / HTTP/1.0\n\n" | nc localhost 80 > /dev/null
+/ # logread | grep nginx
+2017-04-15T15:42:40Z nginx.stdout 127.0.0.1 - - [15/Apr/2017:15:42:40 +0000] "GET / HTTP/1.0" 200 612 "-" "-" "-"
+```
+
+Current issues and limitations:
+
+- The moby tool only supports onboot and service containers. `memlogd` runs as a special container that is managed by init, as it needs fd’s created in advance. To work around this a memlogd container is exported during build. The init-section in the yml is used to extract it to `/containers/init/memlogd` with a pre-created `config.json`.
+- No docker logger plugin support yet - it could be nice to add support to memlogd, so the docker container logs would also be gathered in one place
+- No syslog compatibility at the moment and `/dev/log` doesn’t exist. This socket could be created to keep syslog compatibility, e.g. by using https://github.com/mcuadros/go-syslog. Processes that require syslog should then be able to log directly to memlogd.
+- Kernel messages not read on startup yet (but can be captured with `logwrite dmesg`)
+- Currently no direct external hooks exposed - but options available that could be added. Should also be possible to pipe output to e.g. `oklog` from `logread` (https://github.com/oklog/oklog)
+

projects/logging/examples/logging.yml

@@ -0,0 +1,60 @@
+kernel:
+  image: "mobylinux/kernel:4.9.x"
+  cmdline: "console=ttyS0 console=tty0 page_poison=1"
+init:
+  - linuxkit/init:b5c88b78cd9cc73ed83b45f66bc9de618223768a # with runc, logwrite, startmemlogd
+  - mobylinux/runc:b0fb122e10dbb7e4e45115177a61a3f8d68c19a9
+  - mobylinux/containerd:18eaf72f3f4f9a9f29ca1951f66df701f873060b # unmodified containerd, from pre pr #1636
+  - mobylinux/ca-certificates:eabc5a6e59f05aa91529d80e9a595b85b046f935
+  - linuxkit/memlogd:9b5834189f598f43c507f6938077113906f51012
+onboot:
+  - name: sysctl
+    image: "mobylinux/sysctl:2cf2f9d5b4d314ba1bfc22b2fe931924af666d8c"
+    net: host
+    pid: host
+    ipc: host
+    capabilities:
+     - CAP_SYS_ADMIN
+    readonly: true
+  - name: binfmt
+    image: "linuxkit/binfmt:8881283ac627be1542811bd25c85e7782aebc692"
+    binds:
+     - /proc/sys/fs/binfmt_misc:/binfmt_misc
+    readonly: true
+  - name: dhcpcd
+    image: "linuxkit/dhcpcd:48e249ebef6a521eed886b3bce032db69fbb4afa"
+    binds:
+     - /var:/var
+     - /tmp/etc:/etc
+    capabilities:
+     - CAP_NET_ADMIN
+     - CAP_NET_BIND_SERVICE
+     - CAP_NET_RAW
+    net: host
+    command: ["/sbin/dhcpcd", "--nobackground", "-f", "/dhcpcd.conf", "-1"]
+services:
+  - name: rngd
+    image: "mobylinux/rngd:3dad6dd43270fa632ac031e99d1947f20b22eec9"
+    capabilities:
+     - CAP_SYS_ADMIN
+    oomScoreAdj: -800
+    readonly: true
+  - name: nginx
+    image: "nginx:alpine"
+    capabilities:
+     - CAP_NET_BIND_SERVICE
+     - CAP_CHOWN
+     - CAP_SETUID
+     - CAP_SETGID
+     - CAP_DAC_OVERRIDE
+    net: host
+files:
+  - path: etc/docker/daemon.json
+    contents: '{"debug": true}'
+trust:
+  image:
+    - mobylinux/kernel
+outputs:
+  - format: kernel+initrd
+  - format: iso-bios
+  - format: iso-efi

projects/logging/pkg/init/.gitignore

@@ -0,0 +1,2 @@
+sbin/
+usr/

projects/logging/pkg/init/Dockerfile

@@ -0,0 +1,9 @@
+FROM alpine:3.5
+
+RUN \
+  apk --no-cache update && \
+  apk --no-cache upgrade -a && \
+  apk --no-cache add \
+  && rm -rf /var/cache/apk/*
+
+COPY . ./

projects/logging/pkg/init/Makefile

@@ -0,0 +1,38 @@
+C_COMPILE=linuxkit/c-compile:63b085bbaec1aa7c42a7bd22a4b1c350d900617d@sha256:286e3a729c7a0b1a605ae150235416190f9f430c29b00e65fa50ff73158998e5
+START_STOP_DAEMON=sbin/start-stop-daemon
+
+default: push
+
+$(START_STOP_DAEMON): start-stop-daemon.c
+	mkdir -p $(dir $@)
+	tar cf - $^ | docker run --rm --net=none --log-driver=none -i $(C_COMPILE) -o $@ | tar xf -
+
+.PHONY: tag push
+
+BASE=alpine:3.5
+IMAGE=init
+
+ETC=$(shell find etc -type f)
+
+hash: Dockerfile $(ETC) init $(START_STOP_DAEMON)
+	DOCKER_CONTENT_TRUST=1 docker pull $(BASE)
+	tar cf - $^ | docker build --no-cache -t $(IMAGE):build -
+	docker run --rm $(IMAGE):build sh -c 'cat $^ /lib/apk/db/installed | sha1sum' | sed 's/ .*//' > $@
+
+push: hash
+	docker pull linuxkit/$(IMAGE):$(shell cat hash) || \
+		(docker tag $(IMAGE):build linuxkit/$(IMAGE):$(shell cat hash) && \
+		 docker push linuxkit/$(IMAGE):$(shell cat hash))
+	docker rmi $(IMAGE):build
+	rm -f hash
+
+tag: hash
+	docker pull linuxkit/$(IMAGE):$(shell cat hash) || \
+		docker tag $(IMAGE):build linuxkit/$(IMAGE):$(shell cat hash)
+	docker rmi $(IMAGE):build
+	rm -f hash
+
+clean:
+	rm -rf hash sbin usr
+
+.DELETE_ON_ERROR:

projects/logging/pkg/init/etc/init.d/containerd

@@ -0,0 +1,9 @@
+#!/bin/sh
+
+# bring up containerd
+ulimit -n 1048576
+ulimit -p unlimited
+
+printf "\nStarting containerd\n"
+mkdir -p /var/log
+exec /usr/bin/containerd

projects/logging/pkg/init/etc/init.d/containers

@@ -0,0 +1,36 @@
+#!/bin/sh
+
+# start memlogd container
+
+/usr/bin/startmemlogd
+
+# start onboot containers, run to completion
+
+if [ -d /containers/onboot ]
+then
+	for f in $(find /containers/onboot -mindepth 1 -maxdepth 1 | sort)
+	do
+		base="$(basename $f)"
+		/bin/mount --bind "$f/rootfs" "$f/rootfs"
+		mount -o remount,rw "$f/rootfs"
+		/usr/bin/logwrite -n "$(basename $f)" /usr/bin/runc run --bundle "$f" "$(basename $f)"
+		printf " - $base\n"
+	done
+fi
+
+# start service containers
+
+if [ -d /containers/services ]
+then
+	for f in $(find /containers/services -mindepth 1 -maxdepth 1 | sort)
+	do
+		base="$(basename $f)"
+		/bin/mount --bind "$f/rootfs" "$f/rootfs"
+		mount -o remount,rw "$f/rootfs"
+		log="/var/log/$base.log"
+		/usr/bin/logwrite -n "$(basename $f)" /sbin/start-stop-daemon --start --pidfile /run/$base.pid --exec /usr/bin/runc -- run --bundle "$f" --pid-file /run/$base.pid "$(basename $f)" </dev/null 2>$log >$log &
+		printf " - $base\n"
+	done
+fi
+
+wait

projects/logging/pkg/init/etc/init.d/rcS

@@ -0,0 +1,114 @@
+#!/bin/sh
+
+# mount filesystems
+mkdir -p -m 0755 /proc /run /tmp /sys /dev
+
+mount -n -t proc proc /proc -o ndodev,nosuid,noexec,relatime
+
+mount -n -t tmpfs tmpfs /run -o nodev,nosuid,noexec,relatime,size=10%,mode=755
+mount -n -t tmpfs tmpfs /tmp -o nodev,nosuid,noexec,relatime,size=10%,mode=1777
+
+# mount devfs
+mount -n -t devtmpfs dev /dev -o nosuid,noexec,relatime,size=10m,nr_inodes=248418,mode=755
+# devices
+[ -c /dev/console ] || mknod -m 600 /dev/console c 5 1
+[ -c /dev/tty1 ] || mknod -m 620 /dev/tty1 c 4 1
+[ -c /dev/tty ] || mknod -m 666 /dev/tty c 5 0
+
+[ -c /dev/null ] || mknod -m 666 /dev/null c 1 3
+[ -c /dev/kmsg ] || mknod -m 660 /dev/kmsg c 1 11
+
+# extra symbolic links not provided by default
+[ -e /dev/fd ] || ln -snf /proc/self/fd /dev/fd
+[ -e /dev/stdin ] || ln -snf /proc/self/fd/0 /dev/stdin
+[ -e /dev/stdout ] || ln -snf /proc/self/fd/1 /dev/stdout
+[ -e /dev/stderr ] || ln -snf /proc/self/fd/2 /dev/stderr
+[ -e /proc/kcore ] && ln -snf /proc/kcore /dev/core
+
+# devfs filesystems
+mkdir -p -m 1777 /dev/mqueue
+mkdir -p -m 1777 /dev/shm
+mkdir -p -m 0755 /dev/pts
+mount -n -t mqueue -o noexec,nosuid,nodev mqueue /dev/mqueue
+mount -n -t tmpfs -o noexec,nosuid,nodev,mode=1777 shm /dev/shm
+mount -n -t devpts -o noexec,nosuid,gid=5,mode=0620 devpts /dev/pts
+
+# mount sysfs
+sysfs_opts=nodev,noexec,nosuid
+mount -n -t sysfs -o ${sysfs_opts} sysfs /sys
+[ -d /sys/kernel/security ] && mount -n -t securityfs -o ${sysfs_opts} securityfs /sys/kernel/security
+[ -d /sys/kernel/debug ] && mount -n -t debugfs -o ${sysfs_opts} debugfs /sys/kernel/debug
+[ -d /sys/kernel/config ] && mount -n -t configfs -o ${sysfs_opts} configfs /sys/kernel/config
+[ -d /sys/fs/fuse/connections ] && mount -n -t fusectl -o ${sysfs_opts} fusectl /sys/fs/fuse/connections
+[ -d /sys/fs/selinux ] && mount -n -t selinuxfs -o nosuid,noexec selinuxfs /sys/fs/selinux
+[ -d /sys/fs/pstore ] && mount -n -t pstore pstore -o ${sysfs_opts} /sys/fs/pstore
+[ -d /sys/firmware/efi/efivars ] && mount -n -t efivarfs -o ro,${sysfs_opts} efivarfs /sys/firmware/efi/efivars
+
+# misc /proc mounted fs
+[ -d /proc/sys/fs/binfmt_misc ] && mount -t binfmt_misc -o nodev,noexec,nosuid binfmt_misc /proc/sys/fs/binfmt_misc
+
+# mount cgroups
+mount -n -t tmpfs -o nodev,noexec,nosuid,mode=755,size=10m cgroup_root /sys/fs/cgroup
+
+while read name hier groups enabled rest
+do
+	case "${enabled}" in
+	1)	mkdir -p /sys/fs/cgroup/${name}
+		mount -n -t cgroup -o ${sysfs_opts},${name} ${name} /sys/fs/cgroup/${name}
+	;;
+	esac
+done < /proc/cgroups
+
+# use hierarchy for memory
+echo 1 > /sys/fs/cgroup/memory/memory.use_hierarchy
+
+# for compatibility
+mkdir -p /sys/fs/cgroup/systemd
+mount -t cgroup -o none,name=systemd cgroup /sys/fs/cgroup/systemd
+
+# start mdev for hotplug
+echo "/sbin/mdev" > /proc/sys/kernel/hotplug
+
+# mdev -s will not create /dev/usb[1-9] devices with recent kernels
+# so we trigger hotplug events for usb for now
+for i in $(find /sys/devices -name 'usb[0-9]*'); do
+	[ -e $i/uevent ] && echo add > $i/uevent
+done
+
+mdev -s
+
+# set hostname
+if [ -s /etc/hostname ]
+then
+	hostname -F /etc/hostname
+fi
+
+if [ $(hostname) = "moby" -a -f /sys/class/net/eth0/address ]
+then
+	mac=$(cat /sys/class/net/eth0/address)
+	hostname moby-$(echo $mac | sed 's/://g')
+fi
+
+# set system clock from hwclock
+hwclock --hctosys --utc
+
+# bring up loopback interface
+ip addr add 127.0.0.1/8 dev lo brd + scope host
+ip route add 127.0.0.0/8 dev lo scope host
+ip link set lo up
+
+# for containerising dhcpcd and other containers that need writable etc
+mkdir /tmp/etc
+mv /etc/resolv.conf /tmp/etc/resolv.conf
+ln -snf /tmp/etc/resolv.conf /etc/resolv.conf
+
+# remount rootfs as readonly
+mount -o remount,ro /
+
+# make /var writeable and shared
+mount -o bind /var /var
+mount -o remount,rw,nodev,nosuid,noexec,relatime /var /var
+mount --make-rshared /var
+
+# make / rshared
+mount --make-rshared /

projects/logging/pkg/init/etc/inittab

@@ -0,0 +1,15 @@
+# /etc/inittab
+
+::sysinit:/etc/init.d/rcS
+::once:/etc/init.d/containerd
+::once:/etc/init.d/containers
+
+# Stuff to do for the 3-finger salute
+::ctrlaltdel:/sbin/reboot
+
+# Stuff to do before rebooting
+::shutdown:/usr/sbin/killall5 -15
+::shutdown:/bin/sleep 5
+::shutdown:/usr/sbin/killall5 -9
+::shutdown:/bin/echo "Unmounting filesystems"
+::shutdown:/bin/umount -a -r

projects/logging/pkg/init/etc/issue

@@ -0,0 +1,12 @@
+
+Welcome to LinuxKit
+
+                        ##         .
+                  ## ## ##        ==
+               ## ## ## ## ##    ===
+           /"""""""""""""""""\___/ ===
+      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
+           \______ o           __/
+             \    \         __/
+              \____\_______/
+

projects/logging/pkg/init/init

@@ -0,0 +1,45 @@
+#!/bin/sh
+
+setup_console() {
+	tty=${1%,*}
+	speed=${1#*,}
+	inittab="$2"
+	securetty="$3"
+	line=
+	term="linux"
+	[ "$speed" = "$1" ] && speed=115200
+
+	case "$tty" in
+	ttyS*|ttyAMA*|ttyUSB*|ttyMFD*)
+		line="-L"
+		term="vt100"
+		;;
+	tty?)
+		line=""
+		speed="38400"
+		term=""
+		;;
+	esac
+	# skip consoles already in inittab
+	grep -q "^$tty:" "$inittab" && return
+
+	echo "$tty::once:cat /etc/issue" >> "$inittab"
+	echo "$tty::respawn:/sbin/getty -n -l /bin/sh $line $speed $tty $term" >> "$inittab"
+	if ! grep -q -w "$tty" "$securetty"; then
+		echo "$tty" >> "$securetty"
+	fi
+}
+
+/bin/mount -t tmpfs tmpfs /mnt
+
+/bin/cp -a / /mnt 2>/dev/null
+
+/bin/mount -t proc -o noexec,nosuid,nodev proc /proc
+for opt in $(cat /proc/cmdline); do
+	case "$opt" in
+	console=*)
+		setup_console ${opt#console=} /mnt/etc/inittab /mnt/etc/securetty;;
+	esac
+done
+
+exec /bin/busybox switch_root /mnt /sbin/init

projects/logging/pkg/memlogd/.gitignore

@@ -0,0 +1,5 @@
+usr
+hash
+containers
+.*
+sbin

projects/logging/pkg/memlogd/Dockerfile

@@ -0,0 +1,3 @@
+FROM scratch
+COPY . ./
+WORKDIR /

projects/logging/pkg/memlogd/Dockerfile.memlogd

@@ -0,0 +1,3 @@
+FROM scratch
+COPY . ./
+CMD ["/usr/bin/memlogd","-fd","3"]

projects/logging/pkg/memlogd/Makefile

@@ -0,0 +1,66 @@
+GO_COMPILE=mobylinux/go-compile:3afebc59c5cde31024493c3f91e6102d584a30b9@sha256:e0786141ea7df8ba5735b63f2a24b4ade9eae5a02b0e04c4fca33b425ec69b0a
+
+SHA_IMAGE=alpine:3.5@sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
+
+MEMLOGD_BINARY=usr/bin/memlogd
+LOGWRITE_BINARY=usr/bin/logwrite
+STARTMEMLOGD_BINARY=usr/bin/startmemlogd
+LOGREAD_BINARY=sbin/logread
+
+IMAGE=memlogd
+
+.PHONY: tag push clean container
+default: tag
+
+DEPS=$(MEMLOGD_BINARY) $(LOGWRITE_BINARY) $(STARTMEMLOGD_BINARY) $(LOGREAD_BINARY)
+
+$(MEMLOGD_BINARY): cmd/memlogd/main.go
+	mkdir -p $(dir $@)
+	tar -Ccmd/memlogd -cf - main.go | docker run --rm --net=none --log-driver=none -i $(GO_COMPILE) -o $@ | tar xf -
+
+$(LOGWRITE_BINARY): cmd/logwrite/main.go
+	mkdir -p $(dir $@)
+	tar -Ccmd/logwrite -cf - main.go | docker run --rm --net=none --log-driver=none -i $(GO_COMPILE) -o $@ | tar xf -
+
+$(STARTMEMLOGD_BINARY): cmd/startmemlogd/main.go
+	mkdir -p $(dir $@)
+	tar -Ccmd/startmemlogd -cf - main.go | docker run --rm --net=none --log-driver=none -i $(GO_COMPILE) -o $@ | tar xf -
+
+$(LOGREAD_BINARY): cmd/logread/main.go
+	mkdir -p $(dir $@)
+	tar -Ccmd/logread -cf - main.go | docker run --rm --net=none --log-driver=none -i $(GO_COMPILE) -o $@ | tar xf -
+
+containers: $(MEMLOGD_BINARY) Dockerfile.memlogd config.json
+	mkdir -p containers/init/memlogd/rootfs
+	tar -cf - $^ | docker build -f Dockerfile.memlogd -t $(IMAGE):build1 --no-cache -
+	docker create --name $(IMAGE)-build1 $(IMAGE):build1
+	docker export $(IMAGE)-build1 | tar -Ccontainers/init/memlogd/rootfs -xv -
+	docker rm $(IMAGE)-build1
+	docker rmi $(IMAGE):build1
+	mv containers/init/memlogd/rootfs/Dockerfile.memlogd containers/init/memlogd/rootfs/Dockerfile
+	mv containers/init/memlogd/rootfs/config.json containers/init/memlogd
+
+container: Dockerfile $(LOGWRITE_BINARY) $(STARTMEMLOGD_BINARY) $(LOGREAD_BINARY) containers
+	tar cf - $^ | docker build --no-cache -t $(IMAGE):build -
+
+hash: Dockerfile Dockerfile.memlogd $(DEPS)
+	find $^ -type f | xargs cat | docker run --rm -i $(SHA_IMAGE) sha1sum - | sed 's/ .*//' > hash
+
+push: hash container
+	docker pull linuxkit/$(IMAGE):$(shell cat hash) || \
+		(docker tag $(IMAGE):build linuxkit/$(IMAGE):$(shell cat hash) && \
+		 docker push linuxkit/$(IMAGE):$(shell cat hash))
+	docker rmi $(IMAGE):build
+	rm -f hash
+
+tag: hash container
+	docker pull linuxkit/$(IMAGE):$(shell cat hash) || \
+		docker tag $(IMAGE):build linuxkit/$(IMAGE):$(shell cat hash)
+	docker rmi $(IMAGE):build
+	rm -f hash
+
+clean:
+	rm -rf hash usr containers sbin
+
+.DELETE_ON_ERROR:
+

projects/logging/pkg/memlogd/cmd/logread/main.go

@@ -0,0 +1,52 @@
+package main
+
+import (
+	"bufio"
+	"flag"
+	"net"
+	"os"
+)
+
+const (
+	logDump byte = iota
+	logFollow
+	logDumpFollow
+)
+
+func main() {
+	var err error
+
+	var socketPath string
+	var follow bool
+	var dumpFollow bool
+
+	flag.StringVar(&socketPath, "socket", "/tmp/memlogdq.sock", "memlogd log query socket")
+	flag.BoolVar(&dumpFollow, "F", false, "dump log, then follow")
+	flag.BoolVar(&follow, "f", false, "follow log buffer")
+	flag.Parse()
+
+	addr := net.UnixAddr{socketPath, "unix"}
+	conn, err := net.DialUnix("unix", nil, &addr)
+	if err != nil {
+		panic(err)
+	}
+	defer conn.Close()
+
+	var n int
+	switch {
+	case dumpFollow:
+		n, err = conn.Write([]byte{logDumpFollow})
+	case follow && !dumpFollow:
+		n, err = conn.Write([]byte{logFollow})
+	default:
+		n, err = conn.Write([]byte{logDump})
+	}
+
+	if err != nil || n < 1 {
+		panic(err)
+	}
+
+	r := bufio.NewReader(conn)
+	r.WriteTo(os.Stdout)
+
+}

projects/logging/pkg/memlogd/cmd/logwrite/main.go

@@ -0,0 +1,97 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"syscall"
+)
+
+func getLogFileSocketPair() (*os.File, int) {
+	fds, err := syscall.Socketpair(syscall.AF_UNIX, syscall.SOCK_STREAM, 0)
+	if err != nil {
+		panic(err)
+	}
+
+	localFd := fds[0]
+	remoteFd := fds[1]
+
+	localLogFile := os.NewFile(uintptr(localFd), "")
+	return localLogFile, remoteFd
+}
+
+func sendFD(conn *net.UnixConn, remoteAddr *net.UnixAddr, source string, fd int) error {
+	oobs := syscall.UnixRights(fd)
+	_, _, err := conn.WriteMsgUnix([]byte(source), oobs, remoteAddr)
+	return err
+}
+
+func main() {
+	var err error
+	var ok bool
+
+	var serverSocket string
+	var name string
+
+	flag.StringVar(&serverSocket, "socket", "/tmp/memlogd.sock", "socket to pass fd's to memlogd")
+	flag.StringVar(&name, "n", "", "name of sender, defaults to first argument if left blank")
+	flag.Parse()
+	args := flag.Args()
+
+	if len(args) < 1 {
+		log.Fatal("no command specified")
+	}
+
+	if name == "" {
+		name = args[0]
+	}
+
+	localStdoutLog, remoteStdoutFd := getLogFileSocketPair()
+	localStderrLog, remoteStderrFd := getLogFileSocketPair()
+
+	var outSocket int
+	if outSocket, err = syscall.Socket(syscall.AF_UNIX, syscall.SOCK_DGRAM, 0); err != nil {
+		log.Fatal("Unable to create socket: ", err)
+	}
+
+	var outFile net.Conn
+	if outFile, err = net.FileConn(os.NewFile(uintptr(outSocket), "")); err != nil {
+		log.Fatal(err)
+	}
+
+	var conn *net.UnixConn
+	if conn, ok = outFile.(*net.UnixConn); !ok {
+		log.Fatal("Internal error, invalid cast.")
+	}
+
+	raddr := net.UnixAddr{Name: serverSocket, Net: "unixgram"}
+
+	if err = sendFD(conn, &raddr, name+".stdout", remoteStdoutFd); err != nil {
+		log.Fatal("fd stdout send failed: ", err)
+	}
+
+	if err = sendFD(conn, &raddr, name+".stderr", remoteStderrFd); err != nil {
+		log.Fatal("fd stderr send failed: ", err)
+	}
+
+	cmd := exec.Command(args[0], args[1:]...)
+	outStderr := io.MultiWriter(localStderrLog, os.Stderr)
+	outStdout := io.MultiWriter(localStdoutLog, os.Stdout)
+	cmd.Stderr = outStderr
+	cmd.Stdout = outStdout
+	if err = cmd.Run(); err != nil {
+		if exitError, ok := err.(*exec.ExitError); ok {
+			// exit with exit code from process
+			status := exitError.Sys().(syscall.WaitStatus)
+			os.Exit(status.ExitStatus())
+		} else {
+			// no exit code, report error and exit 1
+			fmt.Println(err)
+			os.Exit(1)
+		}
+	}
+}

projects/logging/pkg/memlogd/cmd/memlogd/main.go

@@ -0,0 +1,315 @@
+package main
+
+import (
+	"bufio"
+	"bytes"
+	"container/list"
+	"container/ring"
+	"flag"
+	"fmt"
+	"io"
+	"log"
+	"net"
+	"os"
+	"sync"
+	"syscall"
+	"time"
+)
+
+type logEntry struct {
+	time   time.Time
+	source string
+	msg    string
+}
+
+type fdMessage struct {
+	name string
+	fd   int
+}
+
+type logMode byte
+
+const (
+	logDump logMode = iota
+	logFollow
+	logDumpFollow
+)
+
+type queryMessage struct {
+	conn *net.UnixConn
+	mode logMode
+}
+
+type connListener struct {
+	conn      *net.UnixConn
+	cond      *sync.Cond // condition and mutex used to notify listeners of more data
+	buffer    bytes.Buffer
+	err       error
+	exitOnEOF bool // exit instead of blocking if no more data in read buffer
+}
+
+func doLog(logCh chan logEntry, msg string) {
+	logCh <- logEntry{time: time.Now(), source: "memlogd", msg: msg}
+	return
+}
+
+func logQueryHandler(l *connListener) {
+	defer l.conn.Close()
+
+	data := make([]byte, 0xffff)
+
+	l.cond.L.Lock()
+	for {
+		var n, remaining int
+		var rerr, werr error
+
+		for rerr == nil && werr == nil {
+			if n, rerr = l.buffer.Read(data); n == 0 { // process data before checking error
+				break // exit read loop to wait for more data
+			}
+			l.cond.L.Unlock()
+
+			remaining = n
+			w := data
+			for remaining > 0 && werr == nil {
+				w = data[:remaining]
+				n, werr = l.conn.Write(w)
+				w = w[n:]
+				remaining = remaining - n
+			}
+
+			l.cond.L.Lock()
+		}
+
+		// check errors
+		if werr != nil {
+			l.err = werr
+			l.cond.L.Unlock()
+			break
+		}
+
+		if rerr != nil && rerr != io.EOF { // EOF is ok, just wait for more data
+			l.err = rerr
+			l.cond.L.Unlock()
+			break
+		}
+		if l.exitOnEOF && rerr == io.EOF { // ... unless we should exit on EOF
+			l.err = nil
+			l.cond.L.Unlock()
+			break
+		}
+		l.cond.Wait() // unlock and wait for more data
+	}
+}
+
+func (msg *logEntry) String() string {
+	return fmt.Sprintf("%s %s %s", msg.time.Format(time.RFC3339), msg.source, msg.msg)
+}
+
+func ringBufferHandler(ringSize int, logCh chan logEntry, queryMsgChan chan queryMessage) {
+	// Anything that interacts with the ring buffer goes through this handler
+	ring := ring.New(ringSize)
+	listeners := list.New()
+
+	for {
+		select {
+		case msg := <-logCh:
+			fmt.Printf("%s\n", msg.String())
+			// add log entry
+			ring.Value = msg
+			ring = ring.Next()
+
+			// send to listeners
+			var l *connListener
+			var remove []*list.Element
+			for e := listeners.Front(); e != nil; e = e.Next() {
+				l = e.Value.(*connListener)
+				if l.err != nil {
+					remove = append(remove, e)
+					continue
+				}
+				l.cond.L.Lock()
+				l.buffer.WriteString(fmt.Sprintf("%s\n", msg.String()))
+				l.cond.L.Unlock()
+				l.cond.Signal()
+			}
+			if len(remove) > 0 { // remove listeners that returned errors
+				for _, e := range remove {
+					l = e.Value.(*connListener)
+					fmt.Println("Removing connection, error: ", l.err)
+					listeners.Remove(e)
+				}
+			}
+
+		case msg := <-queryMsgChan:
+			l := connListener{conn: msg.conn, cond: sync.NewCond(&sync.Mutex{}), err: nil, exitOnEOF: (msg.mode == logDump)}
+			listeners.PushBack(&l)
+			go logQueryHandler(&l)
+			if msg.mode == logDumpFollow || msg.mode == logDump {
+				l.cond.L.Lock()
+				// fill with current data in buffer
+				ring.Do(func(f interface{}) {
+					if msg, ok := f.(logEntry); ok {
+						s := fmt.Sprintf("%s\n", msg.String())
+						l.buffer.WriteString(s)
+					}
+				})
+				l.cond.L.Unlock()
+				l.cond.Signal() // signal handler that more data is available
+			}
+		}
+	}
+}
+
+func receiveQueryHandler(l *net.UnixListener, logCh chan logEntry, queryMsgChan chan queryMessage) {
+	for {
+		var conn *net.UnixConn
+		var err error
+		if conn, err = l.AcceptUnix(); err != nil {
+			doLog(logCh, fmt.Sprintf("Connection error %s", err))
+			continue
+		}
+		mode := make([]byte, 1)
+		n, err := conn.Read(mode)
+		if err != nil || n != 1 {
+			doLog(logCh, fmt.Sprintf("No mode received: %s", err))
+		}
+		queryMsgChan <- queryMessage{conn, logMode(mode[0])}
+	}
+}
+
+func receiveFdHandler(conn *net.UnixConn, logCh chan logEntry, fdMsgChan chan fdMessage) {
+	oob := make([]byte, 512)
+	b := make([]byte, 512)
+
+	for {
+		n, oobn, _, _, err := conn.ReadMsgUnix(b, oob)
+		if err != nil {
+			doLog(logCh, fmt.Sprintf("ERROR: Unable to read oob data: %s", err.Error()))
+			continue
+		}
+
+		if oobn == 0 {
+			continue
+		}
+
+		oobmsgs, err := syscall.ParseSocketControlMessage(oob[:oobn])
+		if err != nil {
+			doLog(logCh, fmt.Sprintf("ERROR: Failed to parse socket control message: %s", err.Error()))
+			continue
+		}
+
+		for _, oobmsg := range oobmsgs {
+			r, err := syscall.ParseUnixRights(&oobmsg)
+			if err != nil {
+				doLog(logCh, fmt.Sprintf("ERROR: Failed to parse UNIX rights in oob data: %s", err.Error()))
+				continue
+			}
+			for _, fd := range r {
+				name := ""
+				if n > 0 {
+					name = string(b[:n])
+				}
+				fdMsgChan <- fdMessage{name: name, fd: fd}
+			}
+		}
+	}
+}
+
+func readLogFromFd(maxLineLen int, fd int, source string, logCh chan logEntry) {
+	f := os.NewFile(uintptr(fd), "")
+	defer f.Close()
+
+	r := bufio.NewReader(f)
+	l, isPrefix, err := r.ReadLine()
+	var buffer bytes.Buffer
+
+	for err == nil {
+		buffer.Write(l)
+		for isPrefix {
+			l, isPrefix, err = r.ReadLine()
+			if err != nil {
+				break
+			}
+			if buffer.Len() < maxLineLen {
+				buffer.Write(l)
+			}
+		}
+		if buffer.Len() > maxLineLen {
+			buffer.Truncate(maxLineLen)
+		}
+		logCh <- logEntry{time: time.Now(), source: source, msg: buffer.String()}
+		buffer.Reset()
+
+		l, isPrefix, err = r.ReadLine()
+	}
+}
+
+func main() {
+	var err error
+
+	var socketQueryPath string
+	var passedQueryFD int
+	var socketLogPath string
+	var passedLogFD int
+	var linesInBuffer int
+	var lineMaxLength int
+
+	flag.StringVar(&socketQueryPath, "socket-query", "/tmp/memlogdq.sock", "unix domain socket for responding to log queries. Overridden by -fd-query")
+	flag.StringVar(&socketLogPath, "socket-log", "/tmp/memlogd.sock", "unix domain socket to listen for new fds to add to log. Overridden by -fd-log")
+	flag.IntVar(&passedLogFD, "fd-log", -1, "an existing SOCK_DGRAM socket for receiving fd's. Overrides -socket-log.")
+	flag.IntVar(&passedQueryFD, "fd-query", -1, "an existing SOCK_STREAM for receiving log read requets. Overrides -socket-query.")
+	flag.IntVar(&linesInBuffer, "max-lines", 5000, "Number of log lines to keep in memory")
+	flag.IntVar(&lineMaxLength, "max-line-len", 1024, "Maximum line length recorded. Additional bytes are dropped.")
+
+	flag.Parse()
+
+	var connLogFd *net.UnixConn
+	if passedLogFD == -1 { // no fd on command line, use socket path
+		addr := net.UnixAddr{socketLogPath, "unixgram"}
+		if connLogFd, err = net.ListenUnixgram("unixgram", &addr); err != nil {
+			log.Fatal("Unable to open socket: ", err)
+		}
+		defer os.Remove(addr.Name)
+	} else { // use given fd
+		var f net.Conn
+		if f, err = net.FileConn(os.NewFile(uintptr(passedLogFD), "")); err != nil {
+			log.Fatal("Unable to open fd: ", err)
+		}
+		connLogFd = f.(*net.UnixConn)
+	}
+	defer connLogFd.Close()
+
+	var connQuery *net.UnixListener
+	if passedQueryFD == -1 { // no fd on command line, use socket path
+		addr := net.UnixAddr{socketQueryPath, "unix"}
+		if connQuery, err = net.ListenUnix("unix", &addr); err != nil {
+			log.Fatal("Unable to open socket: ", err)
+		}
+		defer os.Remove(addr.Name)
+	} else { // use given fd
+		var f net.Listener
+		if f, err = net.FileListener(os.NewFile(uintptr(passedQueryFD), "")); err != nil {
+			log.Fatal("Unable to open fd: ", err)
+		}
+		connQuery = f.(*net.UnixListener)
+	}
+	defer connQuery.Close()
+
+	logCh := make(chan logEntry)
+	fdMsgChan := make(chan fdMessage)
+	queryMsgChan := make(chan queryMessage)
+
+	go receiveFdHandler(connLogFd, logCh, fdMsgChan)
+	go receiveQueryHandler(connQuery, logCh, queryMsgChan)
+	go ringBufferHandler(linesInBuffer, logCh, queryMsgChan)
+
+	doLog(logCh, "memlogd started")
+
+	for true {
+		select {
+		case msg := <-fdMsgChan: // incoming fd
+			go readLogFromFd(lineMaxLength, msg.fd, msg.name, logCh)
+		}
+	}
+}

projects/logging/pkg/memlogd/cmd/startmemlogd/main.go

@@ -0,0 +1,76 @@
+package main
+
+import (
+	"flag"
+	"fmt"
+	"log"
+	"net"
+	"os"
+	"os/exec"
+	"syscall"
+)
+
+func main() {
+	var socketLogPath string
+	var socketQueryPath string
+	var memlogdBundle string
+	var pidFile string
+	var detach bool
+	flag.StringVar(&socketLogPath, "socket-log", "/tmp/memlogd.sock", "path to fd logging socket. Created and passed to logging container. Existing socket will be removed.")
+	flag.StringVar(&socketQueryPath, "socket-query", "/tmp/memlogdq.sock", "path to query socket. Created and passed to logging container. Existing socket will be removed.")
+	flag.StringVar(&memlogdBundle, "bundle", "/containers/init/memlogd", "runc bundle with memlogd")
+	flag.StringVar(&pidFile, "pid-file", "/run/memlogd.pid", "path to pid file")
+	flag.BoolVar(&detach, "detach", true, "detach from subprocess")
+	flag.Parse()
+
+	laddr := net.UnixAddr{socketLogPath, "unixgram"}
+	os.Remove(laddr.Name) // remove existing socket
+	lconn, err := net.ListenUnixgram("unixgram", &laddr)
+	if err != nil {
+		panic(err)
+	}
+	lfd, err := lconn.File()
+	if err != nil {
+		panic(err)
+	}
+
+	qaddr := net.UnixAddr{socketQueryPath, "unix"}
+	os.Remove(qaddr.Name) // remove existing socket
+	qconn, err := net.ListenUnix("unix", &qaddr)
+	if err != nil {
+		panic(err)
+	}
+	qfd, err := qconn.File()
+	if err != nil {
+		panic(err)
+	}
+
+	cmd := exec.Command("/sbin/start-stop-daemon", "--start", "--pidfile", pidFile,
+		"--exec", "/usr/bin/runc", "--", "run", "--preserve-fds=2",
+		"--bundle", memlogdBundle,
+		"--pid-file", pidFile, "memlogd")
+	log.Println(cmd.Args)
+	cmd.ExtraFiles = append(cmd.ExtraFiles, lfd, qfd)
+	cmd.Stdout = os.Stdout
+	cmd.Stderr = os.Stderr
+	if err := cmd.Start(); err != nil {
+		panic(err)
+	}
+	if detach {
+		if err := cmd.Process.Release(); err != nil {
+			panic(err)
+		}
+	} else {
+		if err := cmd.Wait(); err != nil {
+			if exitError, ok := err.(*exec.ExitError); ok {
+				// exit with exit code from process
+				status := exitError.Sys().(syscall.WaitStatus)
+				os.Exit(status.ExitStatus())
+			} else {
+				// no exit code, report error and exit 1
+				fmt.Println(err)
+				os.Exit(1)
+			}
+		}
+	}
+}

projects/logging/pkg/memlogd/config.json

@@ -0,0 +1,115 @@
+{
+    "ociVersion": "1.0.0-rc5-dev",
+    "platform": {
+        "os": "linux",
+        "arch": "amd64"
+    },
+    "process": {
+        "consoleSize": {
+            "height": 0,
+            "width": 0
+        },
+        "user": {
+            "uid": 0,
+            "gid": 0
+        },
+        "args": [
+            "/usr/bin/memlogd",
+            "-fd-log",
+            "3",
+            "-fd-query",
+            "4"
+        ],
+        "env": [
+            "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
+        ],
+        "cwd": "/",
+        "capabilities": {}
+    },
+    "root": {
+        "path": "rootfs",
+        "readonly": true
+    },
+    "mounts": [
+        {
+            "destination": "/proc",
+            "type": "proc",
+            "source": "proc",
+            "options": [
+                "nosuid",
+                "nodev",
+                "noexec",
+                "relatime"
+            ]
+        },
+        {
+            "destination": "/dev",
+            "type": "tmpfs",
+            "source": "tmpfs",
+            "options": [
+                "nosuid",
+                "strictatime",
+                "mode=755",
+                "size=65536k",
+                "ro"
+            ]
+        },
+        {
+            "destination": "/sys",
+            "type": "sysfs",
+            "source": "sysfs",
+            "options": [
+                "nosuid",
+                "noexec",
+                "nodev",
+                "ro"
+            ]
+        },
+        {
+            "destination": "/dev/pts",
+            "type": "devpts",
+            "source": "devpts",
+            "options": [
+                "nosuid",
+                "noexec",
+                "newinstance",
+                "ptmxmode=0666",
+                "mode=0620"
+            ]
+        },
+        {
+            "destination": "/sys/fs/cgroup",
+            "type": "cgroup",
+            "source": "cgroup",
+            "options": [
+                "nosuid",
+                "noexec",
+                "nodev",
+                "relatime",
+                "ro"
+            ]
+        }
+    ],
+    "linux": {
+        "resources": {
+            "disableOOMKiller": false
+        },
+        "namespaces": [
+            {
+                "type": "network"
+            },
+            {
+                "type": "pid"
+            },
+            {
+                "type": "ipc"
+            },
+            {
+                "type": "uts"
+            },
+            {
+                "type": "mount"
+            }
+        ]
+    }
+}