From 2540bb95692f049affce6b0f670a2f46e319b245 Mon Sep 17 00:00:00 2001
From: Graham Whaley <graham.whaley@intel.com>
Date: Mon, 2 Jul 2018 11:22:57 +0100
Subject: [PATCH] Dev: Document journald limitations and workarounds

Journald, by default on some systems, will rate limit log messages,
and in the case of our 'enable full debug', will likely drop some
of our debug.
Document how to identify if this is happening, and how to configure
`systemd-journald` appropriately.

Fixes: #181

Signed-off-by: Graham Whaley <graham.whaley@intel.com>
---
 Developer-Guide.md | 50 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)

diff --git a/Developer-Guide.md b/Developer-Guide.md
index f122caac9f..88f66fba46 100644
--- a/Developer-Guide.md
+++ b/Developer-Guide.md
@@ -6,6 +6,9 @@
     * [Check hardware requirements](#check-hardware-requirements)
     * [Configure to use initrd or rootfs image](#configure-to-use-initrd-or-rootfs-image)
     * [Enable full debug](#enable-full-debug)
+        * [journald rate limiting](#journald-rate-limiting)
+            * [systemd-journald suppressing messages](#systemd-journald-suppressing-messages)
+            * [Disabling systemd-journald rate limiting](#disabling-systemd-journald-rate-limiting)
 * [Build and install Kata proxy](#build-and-install-kata-proxy)
 * [Build and install Kata shim](#build-and-install-kata-shim)
 * [Create and install rootfs and initrd image](#create-and-install-rootfs-and-initrd-image)
@@ -131,6 +134,53 @@ $ sudo sed -i -e 's/^# *\(enable_debug\).*=.*$/\1 = true/g' /usr/share/defaults/
 $ sudo sed -i -e 's/^kernel_params = "\(.*\)"/kernel_params = "\1 agent.log=debug"/g' /usr/share/defaults/kata-containers/configuration.toml
 ```
 
+### journald rate limiting
+
+Enabling [full debug](#enable-full-debug) results in the Kata components generating
+large amounts of logging, which by default is stored in the system log. Depending on
+your system configuration, it is possible that some events might be discarded by the
+system logging daemon. The following shows how to determine this for `systemd-journald`,
+and offers possible workarounds and fixes.
+
+> **Note** The method of implementation can vary between Operating System installations.
+> Amend these instructions as necessary to your system implementation,
+> and consult with your system administrator for the appropriate configuration.
+
+#### `systemd-journald` suppressing messages
+
+`systemd-journald` can be configured to rate limit the number of journal entries
+it stores. When messages are suppressed, it is noted in the logs. This can be checked
+for by looking for those notifications, such as:
+
+```sh
+$ sudo journalctl --since today | fgrep Suppressed
+Jun 29 14:51:17 mymachine systemd-journald[346]: Suppressed 4150 messages from /system.slice/docker.service
+```
+
+This message indicates that a number of log messages from the `docker.service` slice were
+suppressed. In such a case, you can expect to have incomplete logging information
+stored from the Kata Containers components.
+
+#### Disabling `systemd-journald` rate limiting
+
+In order to capture complete logs from the Kata Containers components, you
+need to reduce or disable the `systemd-journald` rate limit. Configure
+this at the global `systemd-journald` level, and it will apply to all system slices.
+
+To disable `systemd-journald` rate limiting at the global level, edit the file
+`/etc/systemd/journald.conf`, and add/uncomment the following lines:
+
+```
+RateLimitInterval=0s
+RateLimitBurst=0
+```
+
+Restart `systemd-journald` for the changes to take effect:
+
+```sh
+$ sudo systemctl restart systemd-journald
+```
+
 # Build and install Kata proxy
 
 ```