diff --git a/docs/how-to/how-to-use-virtio-fs-nydus-with-kata.md b/docs/how-to/how-to-use-virtio-fs-nydus-with-kata.md index 0ea8bf73f7..dd5bb8d5c0 100644 --- a/docs/how-to/how-to-use-virtio-fs-nydus-with-kata.md +++ b/docs/how-to/how-to-use-virtio-fs-nydus-with-kata.md @@ -2,57 +2,432 @@ ## Introduction -Refer to [kata-`nydus`-design](../design/kata-nydus-design.md) for introduction and `nydus` has supported Kata Containers with hypervisor `QEMU` and `CLH` currently. +Nydus is a container image acceleration service that provides fast container startup and on-demand data loading. Kata Containers integrates with Nydus through virtio-fs, supporting two operational modes: -## How to +### Operational Modes -You can use Kata Containers with `nydus` as follows, +1. **Standalone Mode (virtio-fs-nydus)** + - Used with QEMU and Cloud-Hypervisor + - Nydusd runs as an independent process + - Mounts RAFS through nydusd and lets the guest kernel assemble overlayfs -1. Use [`nydus` latest branch](https://github.com/dragonflyoss/image-service); +2. **Inline Mode (inline-virtio-fs) / Builtin Nydus** + - Used with Dragonball VMM + - Nydusd is built into the VMM (builtin nydus) + - Lower resource overhead and better performance + - Suitable for lightweight VM scenarios -2. Deploy `nydus` environment as [`Nydus` Setup for Containerd Environment](https://github.com/dragonflyoss/image-service/blob/master/docs/containerd-env-setup.md); +Refer to [kata-nydus-design](../design/kata-nydus-design.md) for detailed design documentation. -3. Start `nydus-snapshotter` with `enable_nydus_overlayfs` enabled; +## Architecture Overview -4. Use [kata-containers](https://github.com/kata-containers/kata-containers) `latest` branch to compile and build `kata-containers.img`; +### Standalone Mode Architecture -5. Update `configuration-qemu.toml` or `configuration-clh.toml`to include: +``` +┌─────────────────────────────────────────────────────────┐ +│ Host System │ +│ ┌──────────────┐ ┌──────────────────────┐ │ +│ │ nydusd │◄────────┤ nydus-snapshotter │ │ +│ │ (standalone)│ │ │ │ +│ └──────┬───────┘ └──────────────────────┘ │ +│ │ virtiofs │ +│ ▼ │ +│ ┌──────────────┐ │ +│ │ QEMU │ │ +│ │ / Cloud-Hypervisor │ +│ └──────────────┘ │ +└─────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Guest VM │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ /run/kata-containers/shared/ │ │ +│ │ ├── containers//rootfs (overlay) │ │ +│ │ │ ├── upperdir │ │ +│ │ │ ├── workdir │ │ +│ │ │ └── lowerdir (from rafs) │ │ +│ │ └── rafs//lowerdir (nydus image) │ │ +│ └──────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### Inline Mode (Builtin Nydus) Architecture + +``` +┌─────────────────────────────────────────────────────────┐ +│ Host System │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ nydus-snapshotter │ │ +│ └──────────────────────────────────────────────┘ │ +│ │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ Dragonball VMM │ │ +│ │ ┌────────────────────────────────────────┐ │ │ +│ │ │ Builtin Nydusd (virtiofs server) │ │ │ +│ │ │ ┌──────────────────────────────────┐ │ │ │ +│ │ │ │ Vfs (Virtual File System) │ │ │ │ +│ │ │ │ ├── Rafs (nydus image backend) │ │ │ │ +│ │ │ │ └── PassthroughFs │ │ │ │ +│ │ │ └──────────────────────────────────┘ │ │ │ +│ │ └────────────────────────────────────────┘ │ │ +│ └──────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────┐ +│ Guest VM │ +│ ┌──────────────────────────────────────────────┐ │ +│ │ /run/kata-containers/shared/containers/ │ │ +│ │ ├── /rootfs (overlay mount) │ │ +│ │ ├── /rootfs_lower/ (Rafs mount) │ │ +│ │ └── /snapshotdir/ │ │ +│ │ ├── fs/ (upperdir) │ │ +│ │ └── work/ (workdir) │ │ +│ │ │ │ +│ │ Guest Kernel Overlay Assembly: │ │ +│ │ overlay lowerdir=rootfs_lower/ │ │ +│ │ upperdir=snapshotdir/fs/ │ │ +│ │ workdir=snapshotdir/work/ │ │ +│ └──────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────┘ +``` + +### Key Components + +- **nydusd**: The nydus daemon that provides: + - Virtiofs server for file sharing + - Rafs (Registry Accelerated File System) image mounting + - Passthrough filesystem for host-guest file sharing + +- **nydus-snapshotter**: Containerd snapshotter that: + - Manages nydus image layers + - Provides RAFS bootstrap/config/snapshot directory information to Kata + - Prepares container rootfs + - Communicates with Kata runtime + +- **Builtin Nydus (Inline Mode)**: Integrated nydus in Dragonball VMM: + - Runs inside the VMM process, no separate daemon needed + - Uses fuse_backend_rs Vfs to manage multiple filesystem backends + - Supports Rafs for nydus image mounting + - Supports PassthroughFs for host directory sharing + - Lower memory footprint and faster startup + - Overlay filesystem assembled by guest kernel instead of nydusd + +## Prerequisites + +### 1. Install Nydus + +Use the [nydus latest branch](https://github.com/dragonflyoss/image-service) and build nydusd: + +```bash +git clone https://github.com/dragonflyoss/image-service.git +cd image-service +make nydusd +``` + +### 2. Deploy Nydus Environment + +Deploy nydus environment as described in [Nydus Setup for Containerd Environment](https://github.com/dragonflyoss/image-service/blob/master/docs/containerd-env-setup.md). + +### 3. Start Nydus Snapshotter + +Start `nydus-snapshotter` with `enable_nydus_overlayfs` enabled: + +```bash +./nydus-snapshotter --enable-nydus-overlayfs +``` + +### 4. Build Kata Containers + +Use [kata-containers](https://github.com/kata-containers/kata-containers) `latest` branch to compile and build `kata-containers.img`. + +## Configuration + +### Shared Filesystem Types + +Kata Containers supports the following shared filesystem types: + +| Type | Description | Hypervisor | +|------|-------------|------------| +| `virtio-fs` | Standard virtio-fs with virtiofsd | QEMU, Cloud-Hypervisor | +| `virtio-fs-nydus` | Virtio-fs with standalone nydusd | QEMU, Cloud-Hypervisor | +| `inline-virtio-fs` | Inline virtio-fs with builtin nydus | Dragonball | +| `none` | Disable shared filesystem | All | + +### Configuration for QEMU + +Update `configuration-qemu.toml` (for the go runtime) or `configuration-qemu-runtime-rs.toml` (for runtime-rs): + +```toml +# Enable virtio-fs-nydus for standalone mode +shared_fs = "virtio-fs-nydus" + +# Path to nydusd binary (required for virtio-fs-nydus) +virtio_fs_daemon = "/usr/local/bin/nydusd" + +# Optional: Extra arguments for nydusd +# Example: virtio_fs_extra_args = ["--log-level", "debug", "--threads", "4"] +virtio_fs_extra_args = [] + +# Cache mode for virtio-fs (never, auto, always) +virtio_fs_cache = "never" + +# Optional: Enable DAX for better performance +virtio_fs_is_dax = false +``` + +### Configuration for Cloud-Hypervisor + +Update `configuration-clh.toml` (for the go runtime) or `configuration-clh-runtime-rs.toml` (for runtime-rs): ```toml shared_fs = "virtio-fs-nydus" -virtio_fs_daemon = "" +virtio_fs_daemon = "/usr/local/bin/nydusd" virtio_fs_extra_args = [] +virtio_fs_cache = "never" ``` -6. run `crictl run -r kata nydus-container.yaml nydus-sandbox.yaml`; +### Configuration for Dragonball (Inline Mode / Builtin Nydus) -The `nydus-sandbox.yaml` looks like below: +For Dragonball VMM, use inline mode (builtin nydus), update `configuration-dragonball.toml`: + +```toml +shared_fs = "inline-virtio-fs" +# Note: virtio_fs_daemon is not needed for inline mode +# nydusd is built into Dragonball VMM +virtio_fs_cache = "never" +``` + +## How Nydusd Works in Kata + +### Nydusd Startup Process + +#### Standalone Mode + +When using `virtio-fs-nydus`, Kata runtime starts nydusd with the following parameters: + +```bash +nydusd virtiofs \ + --hybrid-mode \ + --log-level info \ + --apisock /path/to/nydusd-api.sock \ + --sock /path/to/virtiofs.sock +``` + +Key features: + +- **Hybrid Mode**: Enables both Rafs and passthrough filesystem support +- **API Socket**: Provides HTTP API for runtime to mount Rafs images +- **Virtiofs Socket**: Used by hypervisor for virtio-fs communication + +After nydusd starts, Kata runtime automatically: + +1. Waits for the nydusd API server to be ready +2. Mounts passthrough_fs at `/containers` within the nydusd virtiofs namespace + - This maps to `/run/kata-containers/shared/containers/` in the guest + - The passthrough_fs provides the writable layer for container overlay + +#### Inline Mode (Builtin Nydus) + +When using `inline-virtio-fs` with Dragonball: + +- Nydusd is built into the Dragonball VMM binary +- No separate process or daemon startup required +- Virtiofs server initializes automatically during VMM boot +- Filesystem backends (Rafs, PassthroughFs) are mounted via VMM API +- Lower resource usage and faster initialization + +### Filesystem Layout in Guest + +#### Standalone Mode (virtio-fs-nydus) + +``` +/run/kata-containers/shared/ # virtiofs mount point +├── containers// # passthrough_fs from host +│ ├── rootfs/ # container rootfs mount point +│ └── snapshotdir/ # snapshot directory +│ ├── fs/ # upperdir (writable layer) +│ └── work/ # workdir (overlay work directory) +└── rafs//lowerdir/ # Rafs mount (nydus image) +``` + +#### Inline Mode (inline-virtio-fs / Builtin Nydus) + +``` +/run/kata-containers/shared/containers/ +├── / +│ ├── rootfs/ # container rootfs mount point +│ ├── rootfs_lower/ # Rafs mount (lowerdir) +│ └── snapshotdir/ +│ ├── fs/ # upperdir +│ └── work/ # workdir +└── passthrough/ # passthrough filesystem +``` + +### Overlay Filesystem Assembly + +#### Standalone Mode (Guest Kernel Overlay) + +The nydus snapshotter provides the RAFS bootstrap path, nydus config, and +snapshot directory through the container rootfs mount options. Runtime-rs passes +the RAFS source/config to nydusd, then asks kata-agent to assemble the writable +overlay in the guest kernel with: + +- **Lowerdir**: Rafs mount point (nydus image) +- **Upperdir**: Writable layer from snapshot directory +- **Workdir**: Overlay work directory + +The nydusd mount request only mounts RAFS: + +```json +{ + "fs_type": "rafs", + "source": "/path/to/bootstrap", + "config": "{...nydus config...}" +} +``` + +Runtime-rs then sends an overlay `Storage` to kata-agent whose lowerdir points +at `/run/kata-containers/shared/rafs//lowerdir` and whose upper/work dirs +come from `/run/kata-containers/shared/containers//snapshotdir/`. + +#### Inline Mode (Guest Kernel Overlay) + +The guest kernel assembles overlay filesystem: + +- Kata agent receives overlay mount information +- Lowerdir points to Rafs mount (rootfs_lower/) +- Upperdir and workdir from snapshot directory +- No native overlay support in builtin nydusd + +## Usage Examples + +### Running Containers with nerdctl + +```bash +$sudo nerdctl run --snapshotter nydus --runtime io.containerd.kata.v2 --net=none --rm -it ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 lsblk +NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS +pmem0 259:0 0 254M 1 disk +`-pmem0p1 259:1 0 253M 1 part +``` + +### Using with Kubernetes + +Create a Pod specification: ```yaml +apiVersion: v1 +kind: Pod metadata: - attempt: 1 - name: nydus-sandbox - uid: nydus-uid - namespace: default -log_directory: /tmp -linux: - security_context: - namespace_options: - network: 2 + name: nydus-test + annotations: + io.containerd.osfeature: "nydus.remoteimage.v1" +spec: + runtimeClassName: kata + containers: + - name: test + image: ghcr.io/dragonflyoss/image-service/ubuntu:nydus-nightly-v5 + command: ["/bin/sleep", "600"] +``` + +## Advanced Configuration + +### Nydusd Extra Arguments + +You can pass additional arguments to nydusd through configuration: + +```toml +virtio_fs_extra_args = ["--log-level", "debug", "--threads", "4"] +``` + +### Prefetch Configuration + +Enable prefetch for faster container startup: + +```toml +# Path to prefetch file list +prefetch_list_path = "/path/to/prefetch_file.list" +``` + +### Performance Tuning + +#### Cache Mode + +Choose appropriate cache mode based on your use case: + +- **never**: No caching, always fetch from host (default, safest) +- **auto**: Cache with timeout, good for read-heavy workloads +- **always**: Aggressive caching, best performance but may see stale data + +```toml +virtio_fs_cache = "auto" +``` + +#### DAX (Direct Access) + +Enable DAX for memory-mapped I/O: + +```toml +virtio_fs_is_dax = true +virtio_fs_cache_size = 1024 # Size in MiB +``` + +## Debugging and Path Mappings + +### Debug Logging + +Enable debug logging for nydusd: + +```toml +virtio_fs_extra_args = ["--log-level", "debug"] +``` + +Or via annotation: + +```yaml annotations: - "io.containerd.osfeature": "nydus.remoteimage.v1" + io.katacontainers.config.hypervisor.virtio_fs_extra_args: "--log-level=debug" ``` -The `nydus-container.yaml` looks like below: +### Path Mappings -```yaml -metadata: - name: nydus-container -image: - image: localhost:5000/ubuntu-nydus:latest -command: - - /bin/sleep -args: - - 600 -log_path: container.1.log -``` +#### Standalone Mode (virtio-fs-nydus) + +| Component | Host Path | Guest Path | Notes | +|-----------|-----------|------------|-------| +| Virtiofs mount | N/A | `/run/kata-containers/shared/` | Root virtiofs mount point | +| Passthrough FS | `/run/kata-containers/shared/sandboxes//rw/` | `/run/kata-containers/shared/containers/` | Mounted at `/containers` in nydusd namespace | +| Rafs mount | Bootstrap path from snapshotter | `/run/kata-containers/shared/rafs//lowerdir` | Mounted via nydusd API | +| Container rootfs | `/run/kata-containers/shared/sandboxes//rw//rootfs` | `/run/kata-containers/shared/containers//rootfs` | Overlay mount point | +| Snapshot dir | From snapshotter | `/run/kata-containers/shared/containers//snapshotdir/` | Contains upperdir and workdir | + +#### Inline Mode (inline-virtio-fs) + +| Component | Host Path | Guest Path | Notes | +|-----------|-----------|------------|-------| +| Virtiofs mount | N/A | `/run/kata-containers/shared/containers/` | Root virtiofs mount point | +| Passthrough FS | `/run/kata-containers/shared/sandboxes//rw/passthrough/` | `/run/kata-containers/shared/containers/passthrough/` | Uses PASSTHROUGH_FS_DIR | +| Rafs mount | Bootstrap path from snapshotter | `/run/kata-containers/shared/containers//rootfs_lower/` | Mounted via DeviceManager | +| Container rootfs | `/run/kata-containers/shared/sandboxes//rw/passthrough//rootfs` | `/run/kata-containers/shared/containers//rootfs` | Overlay mount point | + +## Comparison: Standalone vs Inline Mode + +| Feature | Standalone (virtio-fs-nydus) | Inline (inline-virtio-fs) | +|---------|------------------------------|---------------------------| +| Hypervisor | QEMU, Cloud-Hypervisor | Dragonball | +| Nydusd Process | Independent process | Built into VMM | +| Overlay Support | Guest kernel | Guest kernel | +| Performance | Good | Good | +| Resource Usage | Higher (separate process) | Lower (integrated) | +| Flexibility | More configurable | Less configurable | +| Use Case | General purpose | Lightweight VMs | +| Startup Time | Slower (daemon startup) | Faster (no daemon) | +| Memory Overhead | Higher | Lower | + +## References + +- [Nydus Image Service](https://github.com/dragonflyoss/image-service) +- [Nydus Setup for Containerd](https://github.com/dragonflyoss/image-service/blob/master/docs/containerd-env-setup.md) +- [Kata Containers with Nydus Design](../design/kata-nydus-design.md) +- [Virtio-fs Documentation](https://virtio-fs.gitlab.io/) diff --git a/tests/spellcheck/kata-dictionary.txt b/tests/spellcheck/kata-dictionary.txt index 2e962083f2..68d230242c 100644 --- a/tests/spellcheck/kata-dictionary.txt +++ b/tests/spellcheck/kata-dictionary.txt @@ -76,6 +76,16 @@ Sharedfs Initdata fsmerge fsmerged +Rafs +virtiofs +virtiofsd +Virtiofs +nydusd +Lowerdir +lowerdir +Upperdir +upperdir + # Networking & Communication netns @@ -93,6 +103,7 @@ coredump CPUSET crio nerdctl +crictl dockershim dentries hypercalls