github/kata-containers
1
0
Fork 1
mirror of https://github.com/kata-containers/kata-containers.git synced 2025-04-30 04:34:27 +00:00
Code Issues Projects Releases Wiki Activity

runtime: add enable_debug_console configuration item for agent

Browse Source 
Set enable_debug_console=true in Kata's congiguration file,
runtime will pass `agent.debug_console`
and `agent.debug_console_vport=1026` to agent.

Fixes: #245

Signed-off-by: bin liu <bin@hyper.sh>
This commit is contained in:
bin liu 2020-07-27 13:09:34 +08:00
parent febdf8f68c
commit 484a595f1a
13 changed files with 202 additions and 125 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

71
docs/Developer-Guide.md
View File

@ -34,19 +34,20 @@
* [Troubleshoot Kata Containers](#troubleshoot-kata-containers)
* [Appendices](#appendices)
* [Checking Docker default runtime](#checking-docker-default-runtime)
* [Set up a debug console(the easy way)](#set-up-a-debug-consolethe-easy-way)
* [Enable agent debug console](#enable-agent-debug-console)
* [Start `kata-monitor`](#start-kata-monitor)
* [Connect to debug console](#connect-to-debug-console)
* [Set up a debug console(the traditional way)](#set-up-a-debug-consolethe-traditional-way)
* [Create a custom image containing a shell](#create-a-custom-image-containing-a-shell)
* [Create a debug systemd service](#create-a-debug-systemd-service)
* [Build the debug image](#build-the-debug-image)
* [Configure runtime for custom debug image](#configure-runtime-for-custom-debug-image)
* [Ensure debug options are valid](#ensure-debug-options-are-valid)
* [Create a container](#create-a-container)
* [Connect to the virtual machine using the debug console](#connect-to-the-virtual-machine-using-the-debug-console)
* [Obtain details of the image](#obtain-details-of-the-image)
* [Set up a debug console](#set-up-a-debug-console)
* [Simple debug console setup](#simple-debug-console-setup)
* [Enable agent debug console](#enable-agent-debug-console)
* [Start `kata-monitor`](#start-kata-monitor)
* [Connect to debug console](#connect-to-debug-console)
* [Traditional debug console setup](#traditional-simple-debug-console-setup)
* [Create a custom image containing a shell](#create-a-custom-image-containing-a-shell)
* [Create a debug systemd service](#create-a-debug-systemd-service)
* [Build the debug image](#build-the-debug-image)
* [Configure runtime for custom debug image](#configure-runtime-for-custom-debug-image)
* [Ensure debug options are valid](#ensure-debug-options-are-valid)
* [Create a container](#create-a-container)
* [Connect to the virtual machine using the debug console](#connect-to-the-virtual-machine-using-the-debug-console)
* [Obtain details of the image](#obtain-details-of-the-image)
* [Capturing kernel boot logs](#capturing-kernel-boot-logs)
# Warning
@ -437,24 +438,30 @@ See [Set up a debug console](#set-up-a-debug-console).
```
$ sudo docker info 2>/dev/null | grep -i "default runtime" | cut -d: -f2- | grep -q runc && echo "SUCCESS" || echo "ERROR: Incorrect default Docker runtime"
```
## Set up a debug console
## Set up a debug console(The easy way)
Kata containers provides two ways to connect to the guest. One is using traditional login service, which needs additional works. In contrast the simple debug console is easy to setup.
Kata containers 2.0 support a shell simulated *console* for quickly debug purpose. This approach use `vsock` to connect shell running inside guest started by agent. The good aspect is that we need not modify guest image or despite using what device that hypervisors support. Only `/bin/sh` or `/bin/bash` are necessary.
### Simple debug console setup
### Enable agent debug console
Kata Containers 2.0 supports a shell simulated *console* for quick debug purpose. This approach uses VSOCK to
connect to the shell running inside the guest which the agent starts. This method only requires the guest image to
contain either `/bin/sh` or `/bin/bash`.
Change your `configuration.toml`, add agent debug parameters.
#### Enable agent debug console
Enable debug_console_enabled in the configuration.toml configuration file:
```
kernel_params = "agent.debug_console agent.debug_console_vport=1026"
[agent.kata]
debug_console_enabled = true
```
Sandboxes created using this parameters will start a shell in guest if new connection is accept from `vsock`.
This will pass `agent.debug_console agent.debug_console_vport=1026` to agent as kernel parameters, and sandboxes created using this parameters will start a shell in guest if new connection is accept from VSOCK.
### Start `kata-monitor`
#### Start `kata-monitor`
`kata-runitime exec` need `kata-monitor` to get the sandbox's `vsock` address to connect to, firt start `kata-monitor`.
The `kata-runtime exec` command needs `kata-monitor` to get the sandbox's `vsock` address to connect to, firt start `kata-monitor`.
```
$ sudo kata-monitor
@ -463,9 +470,9 @@ $ sudo kata-monitor
`kata-monitor` will serve at `localhost:8090` by default.
### Connect to debug console
#### Connect to debug console
Command `kata-runitime exec` is used to connect to the debug console.
Command `kata-runtime exec` is used to connect to the debug console.
```
$ kata-runtime exec 1a9ab65be63b8b03dfd0c75036d27f0ed09eab38abb45337fea83acd3cd7bacd
@ -477,9 +484,9 @@ bash-4.2# exit
exit
```
If you want to access guest OS through a traditional way, see [Set up a debug console(the traditional way)](#set-up-a-debug-console-the-traditional-way).
If you want to access guest OS through a traditional way, see [Traditional debug console setup)](#traditional-debug-console-setup).
## Set up a debug console(the traditional way)
### Traditional debug console setup
By default you cannot login to a virtual machine, since this can be sensitive
from a security perspective. Also, allowing logins would require additional
@ -506,7 +513,7 @@ the following steps (using rootfs or initrd image).
>
> Once these steps are taken you can connect to the virtual machine using the [debug console](Developer-Guide.md#connect-to-the-virtual-machine-using-the-debug-console).
### Create a custom image containing a shell
#### Create a custom image containing a shell
To login to a virtual machine, you must
[create a custom rootfs](#create-a-rootfs-image) or [custom initrd](#create-an-initrd-image---optional)
@ -521,7 +528,7 @@ $ export ROOTFS_DIR=${GOPATH}/src/github.com/kata-containers/kata-containers/too
$ script -fec 'sudo -E GOPATH=$GOPATH USE_DOCKER=true EXTRA_PKGS="bash coreutils" ./rootfs.sh centos'
```
### Create a debug systemd service
#### Create a debug systemd service
Create the service file that starts the shell in the rootfs directory:
@ -550,12 +557,12 @@ Add a dependency to start the debug console:
$ sudo sed -i '$a Requires=kata-debug.service' ${ROOTFS_DIR}/lib/systemd/system/kata-containers.target
```
### Build the debug image
#### Build the debug image
Follow the instructions in the [Build a rootfs image](#build-a-rootfs-image)
section when using rootfs, or when using initrd, complete the steps in the [Build an initrd image](#build-an-initrd-image) section.
### Configure runtime for custom debug image
#### Configure runtime for custom debug image
Install the image:
@ -580,7 +587,7 @@ $ (cd /usr/share/kata-containers && sudo ln -sf "$name" kata-containers.img)
**Note**: You should take care to undo this change after you finish debugging
to avoid all subsequently created containers from using the debug image.
### Create a container
#### Create a container
Create a container as normal. For example using crictl:
@ -588,7 +595,7 @@ Create a container as normal. For example using crictl:
$ sudo crictl run -r kata container.yaml pod.yaml
```
### Connect to the virtual machine using the debug console
#### Connect to the virtual machine using the debug console
```
$ id=$(sudo crictl pods --no-trunc -q)
@ -601,7 +608,7 @@ $ sudo socat "stdin,raw,echo=0,escape=0x11" "unix-connect:${console}"
To disconnect from the virtual machine, type `CONTROL+q` (hold down the
`CONTROL` key and press `q`).
### Obtain details of the image
## Obtain details of the image
If the image is created using
[osbuilder](../tools/osbuilder), the following YAML

92
src/agent/src/main.rs
View File

@ -531,18 +531,21 @@ fn run_debug_console_shell(logger: &Logger, shell: &str, socket_fd: RawFd) -> Re
let master_fd = pseduo.master;
let debug_shell_logger = logger.clone();
// channel that used to sync between thread and main process
let (tx, rx) = mpsc::channel::<i32>();
// start a thread to do IO copy between socket and pseduo.master
thread::spawn(move || {
let mut master_reader = unsafe { File::from_raw_fd(master_fd) };
let mut socket_writer = unsafe { File::from_raw_fd(socket_fd) };
let mut socket_reader = unsafe { File::from_raw_fd(socket_fd) };
let mut master_writer = unsafe { File::from_raw_fd(master_fd) };
let mut socket_reader = unsafe { File::from_raw_fd(socket_fd) };
let mut socket_writer = unsafe { File::from_raw_fd(socket_fd) };
loop {
let mut fd_set = FdSet::new();
fd_set.insert(rfd);
fd_set.insert(master_fd);
fd_set.insert(socket_fd);
fd_set.insert(rfd);
match select(
Some(fd_set.highest().unwrap() + 1),
@ -557,36 +560,7 @@ fn run_debug_console_shell(logger: &Logger, shell: &str, socket_fd: RawFd) -> Re
continue;
} else {
error!(debug_shell_logger, "select error {:?}", e);
break;
}
}
}
if fd_set.contains(master_fd) {
match io_copy(&mut master_reader, &mut socket_writer) {
Ok(0) => {
debug!(debug_shell_logger, "master fd closed");
break;
}
Ok(_) => {}
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue,
Err(e) => {
error!(debug_shell_logger, "read master fd error {:?}", e);
break;
}
}
}
if fd_set.contains(socket_fd) {
match io_copy(&mut socket_reader, &mut master_writer) {
Ok(0) => {
debug!(debug_shell_logger, "master fd closed");
break;
}
Ok(_) => {}
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue,
Err(e) => {
error!(debug_shell_logger, "read master fd error {:?}", e);
tx.send(1).unwrap();
break;
}
}
@ -595,22 +569,66 @@ fn run_debug_console_shell(logger: &Logger, shell: &str, socket_fd: RawFd) -> Re
if fd_set.contains(rfd) {
info!(
debug_shell_logger,
"debug shelll process {} exited", child_pid
"debug shell process {} exited", child_pid
);
tx.send(1).unwrap();
break;
}
if fd_set.contains(master_fd) {
match io_copy(&mut master_reader, &mut socket_writer) {
Ok(0) => {
debug!(debug_shell_logger, "master fd closed");
tx.send(1).unwrap();
break;
}
Ok(_) => {}
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue,
Err(e) => {
error!(debug_shell_logger, "read master fd error {:?}", e);
tx.send(1).unwrap();
break;
}
}
}
if fd_set.contains(socket_fd) {
match io_copy(&mut socket_reader, &mut master_writer) {
Ok(0) => {
debug!(debug_shell_logger, "socket fd closed");
tx.send(1).unwrap();
break;
}
Ok(_) => {}
Err(ref e) if e.kind() == std::io::ErrorKind::Interrupted => continue,
Err(e) => {
error!(debug_shell_logger, "read socket fd error {:?}", e);
tx.send(1).unwrap();
break;
}
}
}
}
});
let wait_status = wait::waitpid(child_pid, None);
info!(logger, "debug console exit code: {:?}", wait_status);
info!(logger, "debug console process exit code: {:?}", wait_status);
info!(logger, "notify debug monitor thread to exit");
// close pipe to exit select loop
let _ = close(wfd);
// wait for thread exit.
let _ = rx.recv().unwrap();
info!(logger, "debug monitor thread has exited");
// close files
let _ = close(rfd);
let _ = close(master_fd);
let _ = close(slave_fd);
}
Err(err) => {
let msg = format!("fork error: {:?}", err);
return Err(ErrorKind::ErrorCode(msg).into());
return Err(anyhow!("fork error: {:?}", err));
}
}

1
src/runtime/cli/config-generated.go.in
View File

@ -39,7 +39,6 @@ var version = "@VERSION@"
// project-specific command names
var envCmd = fmt.Sprintf("%s-env", projectPrefix)
var checkCmd = fmt.Sprintf("%s-check", projectPrefix)
var execCmd = "exec"
// project-specific option names
var configFilePathOption = fmt.Sprintf("%s-config", projectPrefix)

7
src/runtime/cli/config/configuration-acrn.toml.in
View File

@ -127,6 +127,13 @@ block_device_driver = "@DEFBLOCKSTORAGEDRIVER_ACRN@"
#trace_mode = "dynamic"
#trace_type = "isolated"
# Enable debug console.
# If enabled, user can connect guest OS running inside hypervisor
# through "kata-runtime exec <sandbox-id>" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional

6
src/runtime/cli/config/configuration-clh.toml.in
View File

@ -125,6 +125,12 @@ block_device_driver = "virtio-blk"
#trace_mode = "dynamic"
#trace_type = "isolated"
# Enable debug console.
# If enabled, user can connect guest OS running inside hypervisor
# through "kata-runtime exec <sandbox-id>" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the

7
src/runtime/cli/config/configuration-fc.toml.in
View File

@ -256,6 +256,13 @@ block_device_driver = "@DEFBLOCKSTORAGEDRIVER_FC@"
#
kernel_modules=[]
# Enable debug console.
# If enabled, user can connect guest OS running inside hypervisor
# through "kata-runtime exec <sandbox-id>" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional

6
src/runtime/cli/config/configuration-qemu-virtiofs.toml.in
View File

@ -352,6 +352,12 @@ vhost_user_store_path = "@DEFVHOSTUSERSTOREPATH@"
#
kernel_modules=[]
# Enable debug console.
# If enabled, user can connect guest OS running inside hypervisor
# through "kata-runtime exec <sandbox-id>" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the

6
src/runtime/cli/config/configuration-qemu.toml.in
View File

@ -375,6 +375,12 @@ vhost_user_store_path = "@DEFVHOSTUSERSTOREPATH@"
#
kernel_modules=[]
# Enable debug console.
# If enabled, user can connect guest OS running inside hypervisor
# through "kata-runtime exec <sandbox-id>" command
#debug_console_enabled = true
[netmon]
# If enabled, the network monitoring process gets started when the

32
src/runtime/cli/kata-exec.go
View File

@ -28,9 +28,16 @@ import (
const (
// The buffer size used to specify the buffer for IO streams copy
bufSize = 32 << 10
bufSize = 1024 * 2
defaultTimeout = 3 * time.Second
subCommandName = "exec"
// command-line parameters name
paramKataMonitorAddr = "kata-monitor-addr"
paramDebugConsolePort = "kata-debug-port"
defaultKernelParamDebugConsoleVPortValue = 1026
defaultParamKataMonitorAddr = "http://localhost:8090"
)
var (
@ -43,15 +50,15 @@ var (
)
var kataExecCLICommand = cli.Command{
Name: execCmd,
Name: subCommandName,
Usage: "Enter into guest by debug console",
Flags: []cli.Flag{
cli.StringFlag{
Name: "monitor-addr",
Name: paramKataMonitorAddr,
Usage: "Kata monitor listen address.",
},
cli.Uint64Flag{
Name: "debug-port",
Name: paramDebugConsolePort,
Usage: "Port that debug console is listening on.",
},
},
@ -60,17 +67,17 @@ var kataExecCLICommand = cli.Command{
if err != nil {
return err
}
span, _ := katautils.Trace(ctx, "exec")
span, _ := katautils.Trace(ctx, subCommandName)
defer span.Finish()
endPoint := context.String("monitor-addr")
endPoint := context.String(paramKataMonitorAddr)
if endPoint == "" {
endPoint = "http://localhost:8090"
endPoint = defaultParamKataMonitorAddr
}
port := context.Uint64("debug-port")
port := context.Uint64(paramDebugConsolePort)
if port == 0 {
port = 1026
port = defaultKernelParamDebugConsoleVPortValue
}
sandboxID := context.Args().Get(0)
@ -188,8 +195,11 @@ func getConn(endPoint, sandboxID string, port uint64) (net.Conn, error) {
switch addr.Scheme {
case clientUtils.VSockSocketScheme:
// vsock://31513974:1024
shimAddr := clientUtils.VSockSocketScheme + ":" + addr.Host
shimAddr = strings.Replace(shimAddr, ":1024", fmt.Sprintf(":%d", port), -1)
cidAndPort := strings.Split(addr.Host, ":")
if len(cidAndPort) != 2 {
return nil, fmt.Errorf("Invalid vsock scheme: %s", sock)
}
shimAddr := fmt.Sprintf("%s:%s:%d", clientUtils.VSockSocketScheme, cidAndPort[0], port)
return clientUtils.VsockDialer(shimAddr, defaultTimeout)
case clientUtils.HybridVSockScheme:

41
src/runtime/pkg/katautils/config.go
View File

@ -132,11 +132,12 @@ type runtime struct {
}
type agent struct {
Debug bool `toml:"enable_debug"`
Tracing bool `toml:"enable_tracing"`
TraceMode string `toml:"trace_mode"`
TraceType string `toml:"trace_type"`
KernelModules []string `toml:"kernel_modules"`
Debug bool `toml:"enable_debug"`
Tracing bool `toml:"enable_tracing"`
TraceMode string `toml:"trace_mode"`
TraceType string `toml:"trace_type"`
KernelModules []string `toml:"kernel_modules"`
DebugConsoleEnabled bool `toml:"debug_console_enabled"`
}
type netmon struct {
@ -441,6 +442,10 @@ func (h hypervisor) getIOMMUPlatform() bool {
return h.IOMMUPlatform
}
func (a agent) debugConsoleEnabled() bool {
return a.DebugConsoleEnabled
}
func (a agent) debug() bool {
return a.Debug
}
@ -866,23 +871,15 @@ func updateRuntimeConfigHypervisor(configPath string, tomlConf tomlConfig, confi
}
func updateRuntimeConfigAgent(configPath string, tomlConf tomlConfig, config *oci.RuntimeConfig, builtIn bool) error {
if builtIn {
config.AgentConfig = vc.KataAgentConfig{
LongLiveConn: true,
Debug: config.AgentConfig.Debug,
KernelModules: config.AgentConfig.KernelModules,
}
return nil
}
for _, agent := range tomlConf.Agent {
config.AgentConfig = vc.KataAgentConfig{
Debug: agent.debug(),
Trace: agent.trace(),
TraceMode: agent.traceMode(),
TraceType: agent.traceType(),
KernelModules: agent.kernelModules(),
LongLiveConn: true,
Debug: agent.debug(),
Trace: agent.trace(),
TraceMode: agent.traceMode(),
TraceType: agent.traceType(),
KernelModules: agent.kernelModules(),
EnableDebugConsole: agent.debugConsoleEnabled(),
}
}
@ -1026,12 +1023,10 @@ func initConfig() (config oci.RuntimeConfig, err error) {
return oci.RuntimeConfig{}, err
}
defaultAgentConfig := vc.KataAgentConfig{}
config = oci.RuntimeConfig{
HypervisorType: defaultHypervisor,
HypervisorConfig: GetDefaultHypervisorConfig(),
AgentConfig: defaultAgentConfig,
AgentConfig: vc.KataAgentConfig{},
}
return config, nil

13
src/runtime/pkg/katautils/config_test.go
View File

@ -167,7 +167,9 @@ func createAllRuntimeConfigFiles(dir, hypervisor string) (config testRuntimeConf
VirtioFSCache: defaultVirtioFSCacheMode,
}
agentConfig := vc.KataAgentConfig{}
agentConfig := vc.KataAgentConfig{
LongLiveConn: true,
}
netmonConfig := vc.NetmonConfig{
Path: netmonPath,
@ -519,7 +521,8 @@ func TestMinimalRuntimeConfig(t *testing.T) {
# Runtime configuration file
[agent.kata]
debug_console_enabled=true
kernel_modules=["a", "b", "c"]
[netmon]
path = "` + netmonPath + `"
`
@ -576,7 +579,11 @@ func TestMinimalRuntimeConfig(t *testing.T) {
VirtioFSCache: defaultVirtioFSCacheMode,
}
expectedAgentConfig := vc.KataAgentConfig{}
expectedAgentConfig := vc.KataAgentConfig{
LongLiveConn: true,
EnableDebugConsole: true,
KernelModules: []string{"a", "b", "c"},
}
expectedNetmonConfig := vc.NetmonConfig{
Path: netmonPath,

35
src/runtime/virtcontainers/kata_agent.go
View File

@ -52,6 +52,11 @@ const (
// path to vfio devices
vfioPath = "/dev/vfio/"
// enable debug console
kernelParamDebugConsole = "agent.debug_console"
kernelParamDebugConsoleVPort = "agent.debug_console_vport"
kernelParamDebugConsoleVPortValue = "1026"
)
var (
@ -195,13 +200,14 @@ func ephemeralPath() string {
// KataAgentConfig is a structure storing information needed
// to reach the Kata Containers agent.
type KataAgentConfig struct {
LongLiveConn bool
Debug bool
Trace bool
ContainerPipeSize uint32
TraceMode string
TraceType string
KernelModules []string
LongLiveConn bool
Debug bool
Trace bool
EnableDebugConsole bool
ContainerPipeSize uint32
TraceMode string
TraceType string
KernelModules []string
}
// KataAgentState is the structure describing the data stored from this
@ -294,6 +300,11 @@ func KataAgentKernelParams(config KataAgentConfig) []Param {
params = append(params, Param{Key: vcAnnotations.ContainerPipeSizeKernelParam, Value: containerPipeSize})
}
if config.EnableDebugConsole {
params = append(params, Param{Key: kernelParamDebugConsole, Value: ""})
params = append(params, Param{Key: kernelParamDebugConsoleVPort, Value: kernelParamDebugConsoleVPortValue})
}
return params
}
@ -1208,16 +1219,6 @@ func (k *kataAgent) buildContainerRootfs(sandbox *Sandbox, c *Container, rootPat
return nil, nil
}
func (k *kataAgent) hasAgentDebugConsole(sandbox *Sandbox) bool {
for _, p := range sandbox.config.HypervisorConfig.KernelParams {
if p.Key == "agent.debug_console" {
k.Logger().Info("agent has debug console")
return true
}
}
return false
}
func (k *kataAgent) createContainer(sandbox *Sandbox, c *Container) (p *Process, err error) {
span, _ := k.trace("createContainer")
defer span.Finish()

10
src/runtime/virtcontainers/vm_test.go
View File

@ -101,7 +101,15 @@ func TestVMConfigGrpc(t *testing.T) {
config := VMConfig{
HypervisorType: QemuHypervisor,
HypervisorConfig: newQemuConfig(),
AgentConfig: KataAgentConfig{true, false, false, 0, "", "", []string{}},
AgentConfig: KataAgentConfig{
LongLiveConn: true,
Debug: false,
Trace: false,
EnableDebugConsole: false,
ContainerPipeSize: 0,
TraceMode: "",
TraceType: "",
KernelModules: []string{}},
}
p, err := config.ToGrpc()