github/linuxkit
1
0
Fork 0
mirror of https://github.com/linuxkit/linuxkit.git synced 2025-07-20 09:39:08 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #3095 from djs55/more-logging

Browse Source 
Support log rotation and logging of /dev/kmsg
This commit is contained in:
Rolf Neugebauer 2018-07-09 22:34:43 +01:00 committed by GitHub
commit 20bd54c6b0
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
19 changed files with 917 additions and 26 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

52
docs/logging.md
View File

@ -39,8 +39,8 @@ The circular buffer has a fixed size (overridden by the command-line argument
To store the logs somewhere more permanent, for example a disk or a remote
network service, a service should be added to the yaml which connects to
`memlogd` and streams the logs. The example program `logread` in the `memlogd`
package demonstrates how to do this.
`memlogd` and streams the logs. The `logwrite` service described below shows
how to do this.
### Message format
@ -52,33 +52,35 @@ where `<timestamp>` is an RFC3339-formatted timestamp, `<log>` is the name of
the log (e.g. `docker-ce.out`) and `<body>` is the output. The `<log>` must
not contain the character `;`.
### Usage examples
## logwrite: writing logs to disk
The service `pkg/logwrite` connects to `memlogd` and streams the logs to files
in `/var/log`. The logs are automatically rotated; by default each file has
a maximum size of 1 MiB and up to 10 files are kept per log. The arguments
`-max-log-files` and `-max-log-size` can be used to override these defaults.
Here is an example log file:
```
/ # logread -f
2018-07-05T13:22:32Z,memlogd;memlogd started
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: waiting for carrier
2018-07-05T13:22:32Z,onboot.001-dhcpcd.err;eth0: could not detect a useable init
system
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: carrier acquired
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;DUID 00:01:00:01:22:d0:d8:18:02:50:00:00:00:02
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: IAID 00:00:00:02
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: adding address fe80::d33a:3936:
2ee4:5c8c
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: soliciting an IPv6 router
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: soliciting a DHCP lease
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: offered 192.168.65.4 from 192.1
68.65.1 `vpnkit'
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: leased 192.168.65.4 for 7200 se
# cat /var/log/onboot.001-dhcpcd.out
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: waiting for carrier
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: carrier acquired
2018-07-08T09:16:53Z onboot.001-dhcpcd.out DUID 00:01:00:01:22:d4:93:05:02:50:00
:00:00:06
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: IAID 00:00:00:06
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: adding address fe80::f346:56a6:590d:5ea4
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: soliciting an IPv6 router
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: soliciting a DHCP lease
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: offered 192.168.65.8 from 192.168.65.1 `vpnkit'
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: leased 192.168.65.8 for 7200 se
conds
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: adding route to 192.168.65.0/24
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;eth0: adding default route via 192.16
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: adding route to 192.168.65.0/24
2018-07-08T09:16:53Z onboot.001-dhcpcd.out eth0: adding default route via 192.16
8.65.1
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;exiting due to oneshot
2018-07-05T13:22:32Z,onboot.001-dhcpcd.out;dhcpcd exited
^C
2018-07-08T09:16:53Z onboot.001-dhcpcd.out exiting due to oneshot
2018-07-08T09:16:53Z onboot.001-dhcpcd.out dhcpcd exited
```
Current issues and limitations:
## Current issues and limitations:
- No docker logger plugin support yet - it could be nice to add support to
memlogd, so the docker container logs would also be gathered in one place
@ -86,8 +88,6 @@ Current issues and limitations:
socket could be created to keep syslog compatibility, e.g. by using
https://github.com/mcuadros/go-syslog. Processes that require syslog should
then be able to log directly to memlogd.
- Kernel messages not read on startup yet (but can be captured with
`logwrite dmesg`)
- Currently no direct external hooks exposed - but options available that
could be added. Should also be possible to pipe output to e.g. `oklog`
from `logread` (https://github.com/oklog/oklog)

4
examples/logging.yml
View File

@ -24,6 +24,10 @@ services:
- name: write-to-the-logs
image: alpine
command: ["/bin/sh", "-c", "while /bin/true; do echo hello $(date); sleep 1; done" ]
- name: write-and-rotate-logs
image: linuxkit/logwrite:7859c102a963828fd9c5aa3837db9600483220c7
- name: kmsg
image: linuxkit/kmsg:3dfa0d5b4027ecc16089f27fbcffa15d6aa5438e
trust:
org:
- linuxkit

14
pkg/kmsg/Dockerfile Normal file
View File

@ -0,0 +1,14 @@
FROM linuxkit/alpine:6264e5b39af8eb1da7ffa4c05a7ccc597da01197 AS mirror
RUN apk add --no-cache go musl-dev linux-headers
ENV GOPATH=/go PATH=$PATH:/go/bin
COPY . /go/src/kmsg/
RUN go-compile.sh /go/src/kmsg
FROM scratch
ENTRYPOINT []
CMD []
WORKDIR /
COPY --from=mirror /go/bin/kmsg /usr/bin/kmsg
CMD ["/usr/bin/kmsg"]

6
pkg/kmsg/build.yml Normal file
View File

@ -0,0 +1,6 @@
image: kmsg
config:
binds:
- /dev/kmsg:/dev/kmsg
capabilities:
- CAP_SYSLOG

25
pkg/kmsg/main.go Normal file
View File

@ -0,0 +1,25 @@
package main
// Log the kernel log buffer (from /dev/kmsg)
import (
"fmt"
"log"
"time"
"github.com/euank/go-kmsg-parser/kmsgparser"
)
func main() {
parser, err := kmsgparser.NewParser()
if err != nil {
log.Fatalf("unable to create parser: %v", err)
}
defer parser.Close()
kmsg := parser.Parse()
for msg := range kmsg {
fmt.Printf("(%d) - %s: %s", msg.SequenceNumber, msg.Timestamp.Format(time.RFC3339Nano), msg.Message)
}
}

1
pkg/kmsg/vendor.conf Normal file
View File

@ -0,0 +1 @@
github.com/euank/go-kmsg-parser 5ba4d492e455a77d25dcf0d2c4acc9f2afebef4e

201
pkg/kmsg/vendor/github.com/euank/go-kmsg-parser/LICENSE generated vendored Normal file
View File

@ -0,0 +1,201 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "{}"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright {yyyy} {name of copyright owner}
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

19
pkg/kmsg/vendor/github.com/euank/go-kmsg-parser/README.md generated vendored Normal file
View File

@ -0,0 +1,19 @@
# go-kmsg-parser
[![Build Status](https://travis-ci.org/euank/go-kmsg-parser.svg?branch=master)](https://travis-ci.org/euank/go-kmsg-parser)
This repository contains a library to allow parsing the `/dev/kmsg` device in
Linux. This device provides a read-write interface to the Linux Kernel's ring
buffer.
In addition to the library, a simple cli-tool that functions similarly to
`dmesg --ctime --follow` is included. This code serves both as a usage example
and as a simple way to verify it works how you'd expect on a given system.
# Contributions
Welcome
# License
Apache 2.0

200
pkg/kmsg/vendor/github.com/euank/go-kmsg-parser/kmsgparser/kmsgparser.go generated vendored Normal file
View File

@ -0,0 +1,200 @@
/*
Copyright 2016 Euan Kemp
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
// Package kmsgparser implements a parser for the Linux `/dev/kmsg` format.
// More information about this format may be found here:
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
// Some parts of it are slightly inspired by rsyslog's contrib module:
// https://github.com/rsyslog/rsyslog/blob/v8.22.0/contrib/imkmsg/kmsg.c
package kmsgparser
import (
"fmt"
"io"
"os"
"strconv"
"strings"
"syscall"
"time"
)
// Parser is a parser for the kernel ring buffer found at /dev/kmsg
type Parser interface {
// SeekEnd moves the parser to the end of the kmsg queue.
SeekEnd() error
// Parse provides a channel of messages read from the kernel ring buffer.
// When first called, it will read the existing ringbuffer, after which it will emit new messages as they occur.
Parse() <-chan Message
// SetLogger sets the logger that will be used to report malformed kernel
// ringbuffer lines or unexpected kmsg read errors.
SetLogger(Logger)
// Close closes the underlying kmsg reader for this parser
Close() error
}
// Message represents a given kmsg logline, including its timestamp (as
// calculated based on offset from boot time), its possibly multi-line body,
// and so on. More information about these mssages may be found here:
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
type Message struct {
Priority int
SequenceNumber int
Timestamp time.Time
Message string
}
func NewParser() (Parser, error) {
f, err := os.Open("/dev/kmsg")
if err != nil {
return nil, err
}
bootTime, err := getBootTime()
if err != nil {
return nil, err
}
return &parser{
log: &StandardLogger{nil},
kmsgReader: f,
bootTime: bootTime,
}, nil
}
type ReadSeekCloser interface {
io.ReadCloser
io.Seeker
}
type parser struct {
log Logger
kmsgReader ReadSeekCloser
bootTime time.Time
}
func getBootTime() (time.Time, error) {
var sysinfo syscall.Sysinfo_t
err := syscall.Sysinfo(&sysinfo)
if err != nil {
return time.Time{}, fmt.Errorf("could not get boot time: %v", err)
}
// sysinfo only has seconds
return time.Now().Add(-1 * (time.Duration(sysinfo.Uptime) * time.Second)), nil
}
func (p *parser) SetLogger(log Logger) {
p.log = log
}
func (p *parser) Close() error {
return p.kmsgReader.Close()
}
func (p *parser) SeekEnd() error {
_, err := p.kmsgReader.Seek(0, os.SEEK_END)
return err
}
// Parse will read from the provided reader and provide a channel of messages
// parsed.
// If the provided reader *is not* a proper Linux kmsg device, Parse might not
// behave correctly since it relies on specific behavior of `/dev/kmsg`
//
// A goroutine is created to process the provided reader. The goroutine will
// exit when the given reader is closed.
// Closing the passed in reader will cause the goroutine to exit.
func (p *parser) Parse() <-chan Message {
output := make(chan Message, 1)
go func() {
defer close(output)
msg := make([]byte, 8192)
for {
// Each read call gives us one full message.
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
n, err := p.kmsgReader.Read(msg)
if err != nil {
if err == syscall.EPIPE {
p.log.Warningf("short read from kmsg; skipping")
continue
}
if err == io.EOF {
p.log.Infof("kmsg reader closed, shutting down")
return
}
p.log.Errorf("error reading /dev/kmsg: %v", err)
return
}
msgStr := string(msg[:n])
message, err := p.parseMessage(msgStr)
if err != nil {
p.log.Warningf("unable to parse kmsg message %q: %v", msgStr, err)
continue
}
output <- message
}
}()
return output
}
func (p *parser) parseMessage(input string) (Message, error) {
// Format:
// PRIORITY,SEQUENCE_NUM,TIMESTAMP,-;MESSAGE
parts := strings.SplitN(input, ";", 2)
if len(parts) != 2 {
return Message{}, fmt.Errorf("invalid kmsg; must contain a ';'")
}
metadata, message := parts[0], parts[1]
metadataParts := strings.Split(metadata, ",")
if len(metadataParts) < 3 {
return Message{}, fmt.Errorf("invalid kmsg: must contain at least 3 ',' separated pieces at the start")
}
priority, sequence, timestamp := metadataParts[0], metadataParts[1], metadataParts[2]
prioNum, err := strconv.Atoi(priority)
if err != nil {
return Message{}, fmt.Errorf("could not parse %q as priority: %v", priority, err)
}
sequenceNum, err := strconv.Atoi(sequence)
if err != nil {
return Message{}, fmt.Errorf("could not parse %q as sequence number: %v", priority, err)
}
timestampUsFromBoot, err := strconv.ParseInt(timestamp, 10, 64)
if err != nil {
return Message{}, fmt.Errorf("could not parse %q as timestamp: %v", priority, err)
}
// timestamp is offset in microsecond from boottime.
msgTime := p.bootTime.Add(time.Duration(timestampUsFromBoot) * time.Microsecond)
return Message{
Priority: prioNum,
SequenceNumber: sequenceNum,
Timestamp: msgTime,
Message: message,
}, nil
}

55
pkg/kmsg/vendor/github.com/euank/go-kmsg-parser/kmsgparser/log.go generated vendored Normal file
View File

@ -0,0 +1,55 @@
/*
Copyright 2016 Euan Kemp
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package kmsgparser
import stdlog "log"
// Logger is a glog compatible logging interface
// The StandardLogger struct can be used to wrap a log.Logger from the golang
// "log" package to create a standard a logger fulfilling this interface as
// well.
type Logger interface {
Warningf(string, ...interface{})
Infof(string, ...interface{})
Errorf(string, ...interface{})
}
// StandardLogger adapts the "log" package's Logger interface to be a Logger
type StandardLogger struct {
*stdlog.Logger
}
func (s *StandardLogger) Warningf(fmt string, args ...interface{}) {
if s.Logger == nil {
return
}
s.Logger.Printf("[WARNING] "+fmt, args)
}
func (s *StandardLogger) Infof(fmt string, args ...interface{}) {
if s.Logger == nil {
return
}
s.Logger.Printf("[INFO] "+fmt, args)
}
func (s *StandardLogger) Errorf(fmt string, args ...interface{}) {
if s.Logger == nil {
return
}
s.Logger.Printf("[INFO] "+fmt, args)
}

14
pkg/logwrite/Dockerfile Normal file
View File

@ -0,0 +1,14 @@
FROM linuxkit/alpine:6264e5b39af8eb1da7ffa4c05a7ccc597da01197 AS build
RUN apk add --no-cache go musl-dev
ENV GOPATH=/go PATH=$PATH:/go/bin
COPY logwrite.go /go/src/logwrite/
RUN go-compile.sh /go/src/logwrite
FROM scratch
ENTRYPOINT []
CMD []
WORKDIR /
COPY --from=build /go/bin/logwrite usr/bin/logwrite
CMD ["/usr/bin/logwrite"]

5
pkg/logwrite/build.yml Normal file
View File

@ -0,0 +1,5 @@
image: logwrite
config:
binds:
- /var/run:/var/run
- /var/log:/var/log

203
pkg/logwrite/logwrite.go Normal file
View File

@ -0,0 +1,203 @@
package main
// Write logs to files and perform rotation.
import (
"bufio"
"errors"
"flag"
"fmt"
"io"
"log"
"net"
"os"
"path/filepath"
"strings"
"time"
)
// These must be kept in sync with memlogd:
const (
logDump byte = iota
logFollow
logDumpFollow
)
const mb = 1024 * 1024
// LogMessage is a message received from memlogd.
type LogMessage struct {
Time time.Time // time message was received by memlogd
Name string // name of the service that wrote the message
Message string // body of the message
}
func (m *LogMessage) String() string {
return m.Time.Format(time.RFC3339) + " " + m.Name + " " + m.Message
}
// ParseLogMessage reconstructs a LogMessage from a line of text which looks like:
// <timestamp>,<origin>;<body>
func ParseLogMessage(line string) (*LogMessage, error) {
bits := strings.SplitN(line, ";", 2)
if len(bits) != 2 {
return nil, errors.New("Failed to parse log message: " + line)
}
bits2 := strings.Split(bits[0], ",")
if len(bits2) < 2 {
// There could be more parameters in future
return nil, errors.New("Failed to parse log message: " + line)
}
Time, err := time.Parse(time.RFC3339, bits2[0])
if err != nil {
return nil, err
}
return &LogMessage{
Time: Time,
Name: bits2[1],
Message: bits[1],
}, nil
}
// LogFile is where we write LogMessages to
type LogFile struct {
File *os.File // active file handle
Name string // filename of log file
Dir string // log file directory
BytesWritten int // total number of bytes written so far
}
// NewLogFile creates a new LogFile.
func NewLogFile(dir, name string) (*LogFile, error) {
// If the log exists already we want to append to it.
f, err := os.OpenFile(filepath.Join(dir, name), os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return nil, err
}
fi, err := f.Stat()
if err != nil {
return nil, err
}
return &LogFile{
File: f,
Name: name,
Dir: dir,
BytesWritten: int(fi.Size()),
}, nil
}
// Write appends a message to the log file
func (l *LogFile) Write(m *LogMessage) error {
s := m.String()
_, err := io.WriteString(l.File, s)
if err == nil {
l.BytesWritten += len(s)
}
return err
}
// Close a log file
func (l *LogFile) Close() error {
return l.File.Close()
}
// Rotate closes the current log file, rotates the files and creates an empty log file.
func (l *LogFile) Rotate(maxLogFiles int) error {
if err := l.File.Close(); err != nil {
return err
}
path := filepath.Join(l.Dir, l.Name)
for i := maxLogFiles - 1; i >= 0; i-- {
newerFile := fmt.Sprintf("%s.%d", path, i-1)
// special case: if index is 0 we omit the suffix i.e. we expect
// foo foo.1 foo.2 up to foo.<maxLogFiles-1>
if i == 0 {
newerFile = path
}
olderFile := fmt.Sprintf("%s.%d", path, i)
// overwrite the olderFile with the newerFile
err := os.Rename(newerFile, olderFile)
if os.IsNotExist(err) {
// the newerFile does not exist
continue
}
if err != nil {
return err
}
}
f, err := os.Create(path)
if err != nil {
return err
}
l.File = f
l.BytesWritten = 0
return nil
}
func main() {
socketPath := flag.String("socket", "/var/run/memlogdq.sock", "memlogd log query socket")
logDir := flag.String("log-dir", "/var/log", "Directory containing log files")
maxLogFiles := flag.Int("max-log-files", 10, "Maximum number of rotated log files before deletion")
maxLogSize := flag.Int("max-log-size", mb, "Maximum size of a log file before rotation")
flag.Parse()
addr := net.UnixAddr{
Name: *socketPath,
Net: "unix",
}
conn, err := net.DialUnix("unix", nil, &addr)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
n, err := conn.Write([]byte{logDumpFollow})
if err != nil || n < 1 {
log.Fatalf("Failed to write request to memlogd socket: %v", err)
}
// map of service name to active log file
logs := make(map[string]*LogFile)
r := bufio.NewReader(conn)
for {
line, err := r.ReadString('\n')
if err == io.EOF {
return
}
if err != nil {
log.Fatalf("Failed to read from memlogd: %v", err)
}
msg, err := ParseLogMessage(line)
if err != nil {
log.Println(err)
continue
}
if strings.HasPrefix(msg.Name, "logwrite") {
// don't log our own output in a loop
continue
}
var logF *LogFile
var ok bool
if logF, ok = logs[msg.Name]; !ok {
logF, err = NewLogFile(*logDir, msg.Name)
if err != nil {
log.Printf("Failed to create log file %s: %v", msg.Name, err)
continue
}
logs[msg.Name] = logF
}
if err = logF.Write(msg); err != nil {
log.Printf("Failed to write to log file %s: %v", msg.Name, err)
if err := logF.Close(); err != nil {
log.Printf("Failed to close log file %s: %v", msg.Name, err)
}
delete(logs, msg.Name)
continue
}
if logF.BytesWritten > *maxLogSize {
logF.Rotate(*maxLogFiles)
}
}
}

14
test/cases/040_packages/030_logwrite/check.sh Executable file
View File

@ -0,0 +1,14 @@
#!/bin/sh
for i in $(seq 1 20); do
if [ -e /var/log/fill-the-logs.out.0 ]; then
printf "logwrite test suite PASSED\n" > /dev/console
/sbin/poweroff -f
fi
sleep 1
done
printf "logwrite test suite FAILED\n" > /dev/console
echo "contents of /var/log:" > /dev/console
ls -l /var/log > /dev/console
/sbin/poweroff -f

24
test/cases/040_packages/030_logwrite/test.sh Normal file
View File

@ -0,0 +1,24 @@
#!/bin/sh
# SUMMARY: Check that the logwrite package works
# LABELS:
# REPEAT:
set -e
# Source libraries. Uncomment if needed/defined
#. "${RT_LIB}"
. "${RT_PROJECT_ROOT}/_lib/lib.sh"
NAME=logwrite
clean_up() {
rm -rf ${NAME}-*
}
trap clean_up EXIT
# Test code goes here
linuxkit build -disable-content-trust -format kernel+initrd -name "${NAME}" test.yml
RESULT="$(linuxkit run ${NAME})"
echo "${RESULT}"
echo "${RESULT}" | grep -q "suite PASSED"
exit 0

35
test/cases/040_packages/030_logwrite/test.yml Normal file
View File

@ -0,0 +1,35 @@
kernel:
image: linuxkit/kernel:4.14.53
cmdline: "console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:6cc1442112980c889230b6449df09d5b48de6854
- linuxkit/runc:v0.4
- linuxkit/containerd:f2bc1bda1ab18146967fa1a149800aaf14bee81b
- linuxkit/ca-certificates:v0.4
- linuxkit/memlogd:883f0d46e7d3ae2d787e8acb496da115a4707cbc
services:
# A service which generates logs of log messages
- name: fill-the-logs
image: alpine
command: ["/bin/sh", "-c", "while /bin/true; do echo hello $(date); done" ]
- name: write-and-rotate-logs
image: linuxkit/logwrite:7859c102a963828fd9c5aa3837db9600483220c7
command: ["/usr/bin/logwrite", "-max-log-size", "1024"]
- name: check-the-logs
image: alpine:3.8
binds:
- /check.sh:/check.sh
- /dev/console:/dev/console
- /var/log:/var/log
command: ["sh", "./check.sh"]
pid: host
capabilities:
- CAP_SYS_BOOT
files:
- path: check.sh
source: ./check.sh
trust:
org:
- linuxkit
image:
- alpine:3.8

15
test/cases/040_packages/031_kmsg/check.sh Executable file
View File

@ -0,0 +1,15 @@
#!/bin/sh
for i in $(seq 1 20); do
# Look for a common kernel log message
if grep "SCSI subsystem initialized" /var/log/kmsg.out 2>/dev/null; then
printf "kmsg test suite PASSED\n" > /dev/console
/sbin/poweroff -f
fi
sleep 1
done
printf "kmsg test suite FAILED\n" > /dev/console
echo "contents of /var/log/kmsg.out:" > /dev/console
cat /var/log/kmsg.out > /dev/console
/sbin/poweroff -f

24
test/cases/040_packages/031_kmsg/test.sh Normal file
View File

@ -0,0 +1,24 @@
#!/bin/sh
# SUMMARY: Check that the kmsg package works
# LABELS:
# REPEAT:
set -e
# Source libraries. Uncomment if needed/defined
#. "${RT_LIB}"
. "${RT_PROJECT_ROOT}/_lib/lib.sh"
NAME=kmsg
clean_up() {
rm -rf ${NAME}-*
}
trap clean_up EXIT
# Test code goes here
linuxkit build -disable-content-trust -format kernel+initrd -name "${NAME}" test.yml
RESULT="$(linuxkit run ${NAME})"
echo "${RESULT}"
echo "${RESULT}" | grep -q "suite PASSED"
exit 0

32
test/cases/040_packages/031_kmsg/test.yml Normal file
View File

@ -0,0 +1,32 @@
kernel:
image: linuxkit/kernel:4.14.53
cmdline: "console=ttyS0 console=ttyAMA0"
init:
- linuxkit/init:6cc1442112980c889230b6449df09d5b48de6854
- linuxkit/runc:v0.4
- linuxkit/containerd:f2bc1bda1ab18146967fa1a149800aaf14bee81b
- linuxkit/ca-certificates:v0.4
- linuxkit/memlogd:883f0d46e7d3ae2d787e8acb496da115a4707cbc
services:
- name: kmsg
image: linuxkit/kmsg:3dfa0d5b4027ecc16089f27fbcffa15d6aa5438e
- name: write-and-rotate-logs
image: linuxkit/logwrite:7859c102a963828fd9c5aa3837db9600483220c7
- name: check-the-logs
image: alpine:3.8
binds:
- /check.sh:/check.sh
- /dev/console:/dev/console
- /var/log:/var/log
command: ["sh", "./check.sh"]
pid: host
capabilities:
- CAP_SYS_BOOT
files:
- path: check.sh
source: ./check.sh
trust:
org:
- linuxkit
image:
- alpine:3.8