MVGAL Design Document

Implementation status: Source metadata is 0.7.31. The source changelog documents through 0.7.31. Treat design/API descriptions as available only where the relevant code path and runtime capability are verified; unsupported kernel submission and VRAM allocation return -EOPNOTSUPP.

Source version: 0.7.31 | Last Updated: October 2026

Design Goals

1. Multi-vendor GPU coordination — a design goal, not an achieved behaviour: MVGAL does not make arbitrary applications see one physical GPU; explicit, workload-specific coordination is the supported shape.
2. Interception without source changes — a Vulkan layer, OpenCL ICD, CUDA shim and LD_PRELOAD hooks exist, but per-call forwarding and multi-device correctness require validation and interception is opt-in per launch.
3. Vendor-agnostic scheduling — AMD, NVIDIA, Intel, and Moore Threads GPUs work together.
4. Safety-first — kernel module with DRM registration, Rust safety crates for critical paths.
5. Low overhead — target <2% frame time regression for single-GPU pass-through mode.

Architecture Decisions

Decision 1: Kernel Module as DRM Meta-Driver

Chosen approach: A Linux kernel module (mvgal.ko) that registers a DRM device and exposes a character device, /dev/mvgal0.

There are two distinct ioctl namespaces, and they are easy to confuse:

The device registers with driver_features = DRIVER_RENDER | DRIVER_HAVE_IRQ | DRIVER_GEM and major/minor 0.2. Registration is deliberately minimal and logical-only: GPU discovery happens in userspace.

Alternatives considered: - Userspace-only daemon: Simpler to develop, no kernel patching required. Rejected because DMA-BUF zero-copy and cross-device synchronization require kernel-level buffer management. - Upstream DRM proposal: Would integrate natively with the DRM subsystem. Deferred because upstream review cycle is 6-12 months and vendor driver compatibility is uncertain. - eBPF-based approach: Modern, safe, no module loading. Rejected because eBPF programs cannot manage GPU memory or synchronize across vendors at the required latency.

Tradeoffs: - (+) Full control over buffer management and scheduling - (+) DMA-BUF import/export across vendors - (-) Requires module signing and MOK enrollment for Secure Boot - (-) Must track kernel API changes across releases

Decision 2: C++20 Daemon with IPC over Unix Socket

Chosen approach: mvgald is a C++20 daemon communicating with clients via Unix socket (/run/mvgal/mvgal.sock) using a binary protocol and peer-credential authentication.

The wire header is ipc_message_header_t (packed): magic, version, message_type, payload_size, request_id. The magic is MVGAL_IPC_MAGIC 0x4D564741 ("MVGA") and the protocol version is 1. Message types are the eleven mvgal_ipc_message_type_t values from MVGAL_IPC_MSG_PING (0) through MVGAL_IPC_MSG_ERROR (10).

Note: MVGAL_NET_MAGIC 0x4D56474E ("MVGN") is a different constant, used by the network-pooling protocol in mvgal_network.h. The local IPC socket does not use it.

Authentication reads the peer's credentials with getsockopt(SOL_SOCKET, SO_PEERCRED) and admits root, the daemon's own uid, and members of the mvgal group (falling back to the daemon's primary group). The standard video group is also accepted, and the client's supplementary groups are read from /proc/<pid>/status — getgroups() would return the daemon's groups, not the client's, so it is deliberately not used.

Alternatives considered: - D-Bus as primary IPC: Simpler, well-supported. Rejected because D-Bus adds ~50μs per message round-trip; GPU workload submission needs sub-10μs latency. - Shared memory ring buffer: Lowest latency. Rejected because it requires mmap setup per client and complicates authentication. - gRPC/Protobuf: Cross-language, schema-defined. Rejected because it adds ~100KB binary dependency and serialization overhead.

Tradeoffs: - (+) Low latency (~2-5μs per IPC call) - (+) Native Unix credential verification - (-) Custom binary protocol requires versioning discipline - (-) No automatic cross-language binding generation

Decision 3: LD_PRELOAD Interception for All APIs

Chosen approach: Interception scaffolding via Vulkan implicit layer, OpenCL ICD wrapper, CUDA function hooking (40+ functions), and LD_PRELOAD for OpenGL/D3D/Metal/WebGPU. Per-call forwarding and multi-device correctness require validation; interception is opt-in per launch.

Alternatives considered: - Library wrapping (link-time): Compile against MVGAL lib instead of vendor lib. Rejected because it requires recompilation and breaks binary compatibility. - ptrace-based interception: No library modification needed. Rejected because ptrace has severe performance penalties and breaks anti-cheat. - Binary patching: Modify vendor libraries in-place. Rejected because it violates vendor licenses and breaks with updates.

Tradeoffs: - (+) No application source changes required when the layer/ICD is enabled - (+) Applies to unmodified binaries in principle (Steam games, AI frameworks) — validated paths vary by API, driver and workload - (-) Environment variable setup required per-launch - (-) Anti-cheat software may flag LD_PRELOAD

Decision 4: DMA-BUF Zero-Copy with Host-RAM Fallback

Chosen approach: Three-tier memory transfer: (1) DMA-BUF zero-copy between GPUs sharing the same IOMMU group, (2) PCIe P2P for GPUs on the same root complex, (3) host-RAM staging as universal fallback.

Alternatives considered: - CUDA peer-to-peer only: NVIDIA-to-NVIDIA in principle. Rejected because MVGAL targets AMD+NVIDIA+Intel combinations — cross-vendor paths remain capability-probed and require validation. - ROCm hipMemcpyPeer: AMD-specific. Rejected for the same reason. - Always host-staged: Simplest implementation. Rejected because it adds a full host round-trip per cross-GPU transfer for large buffers.

Tradeoffs: - (+) Best-case performance with zero-copy - (+) Universal fallback ensures correctness - (-) Complex fallback logic with three code paths - (-) DMA-BUF support varies by driver version

Decision 5: Rust for Safety-Critical Subsystems

Chosen approach: Three Rust crates with C FFI exports for use by the C++ daemon.

The directory names and the Cargo package names differ, which is an easy thing to get wrong:

runtime/safe/lib.rs re-exports them as modules (fence_manager, memory_safety, capability_model), and safe/ffi_tests/tests/ffi_integration.rs imports them under the same aliases — which is why the module names and the package names can appear to conflict. pub struct GpuCapability is defined at safe/capability_model/src/lib.rs:37. A fourth crate, safe/ffi_tests, exercises the FFI boundary.

Alternatives considered: - All C++ with RAII: Would keep the codebase single-language. Rejected because Rust's ownership model eliminates entire classes of bugs (use-after-free, data races) that are critical in GPU synchronization code. - All Rust: Would maximize safety. Rejected because the kernel module must be C (Linux requirement) and the existing IPC/scheduler code is mature C++. - Valgrind/ASan in CI: Runtime detection. Rejected as insufficient alone — static analysis via Rust borrow checker is strictly better for prevention.

Tradeoffs: - (+) Memory safety without garbage collection - (+) Compile-time enforcement of concurrency safety - (-) FFI boundary adds complexity - (-) Two build systems (Cargo + CMake)

Decision 6: 13 Scheduling Strategies

Chosen approach: The C enum mvgal_distribution_strategy_t (include/mvgal/mvgal_types.h:69-84) has 13 members — twelve auto-numbered 0..11 plus CUSTOM = 100:

ROUND_ROBIN 0 · AFR 1 · SFR 2 · AUTO 3 · COMPUTE_OFFLOAD 4 · HYBRID 5 · SINGLE_GPU 6 · TASK 7 · AI_DRIVEN 8 · RLD 9 · REP 10 · PPL 11 · CUSTOM 100

Auto-detect (AUTO) selects a strategy based on workload type.

Three different strategy vocabularies exist in the tree, and they are not the same list:

Set the strategy with mvgal_set_strategy() (mvgal.h:197) or the scheduler-specific mvgal_scheduler_set_strategy() (mvgal_scheduler.h:324) — these are two distinct APIs, not duplicates.

Alternatives considered: - Fixed single strategy: Simplest. Rejected because different workloads benefit from different strategies (AFR for gaming, compute offload for AI). - User-configured only: No auto-detect. Rejected because most users won't know which strategy is best. - ML-based adaptive: Uses the AI scheduling mode. Deferred because it requires training data and adds model loading overhead.

Tradeoffs: - (+) Covers all common use cases - (+) Auto-detect reduces user configuration burden - (-) 13 strategies increase code surface area - (-) Auto-detect heuristics may be wrong for edge cases

Decision 7: Synthetic Topology at the Discovery Seam (v0.7.18)

Multi-vendor code paths — cross-vendor DMA-BUF, P2P capability negotiation, heterogeneous scheduling, per-pair bandwidth costing — only misbehave on machines nobody has. MVGAL therefore lets a test describe a machine that does not exist, and substitutes that description for the result of hardware rescan.

The alternative designs were rejected for specific reasons:

• Mock each subsystem independently. Rejected: it would prove that a mocked scheduler picks the right strategy, which was never in doubt. The bugs worth catching live between discovery, capability negotiation and the cost model.
• Require real multi-vendor hardware. Rejected: a test that needs four GPUs from four vendors in one box does not run.
• A separate fake-only code path. Rejected: the fake path would not be the path production takes, so passing tests would prove nothing.

Why the discovery seam specifically: it is the single point every downstream consumer already goes through. Substituting there means the scheduler, memory manager and P2P cost model are exercised through their real entry points, with no test-only branches inside them.

Tradeoffs: - (+) Heterogeneous paths become testable on any machine, deterministically - (+) Thermal, throttling and bandwidth behaviour are modelled, so time and load are inputs rather than fixtures - (-) The backend shapes what discovery reports; it creates no devices and proves nothing about hardware - (-) It is a public header, so the test surface is part of the shipped ABI

Data Flow

The kernel path carries discovery and capability reporting only: vendor VRAM allocation and kernel-side submission return -EOPNOTSUPP because the native drivers own the hardware. The diagram shows the intended control flow, not a claim that MVGAL submits work itself.

Security Model

• Socket permissions: /run/mvgal/mvgal.sock is mode 0660, owned root:mvgal — falling back to the daemon's primary group if the mvgal group does not exist. The run directory /run/mvgal is chowned to the same gid and made 0775. (Changed in v0.7.14; earlier releases chmod'ed the socket 0660 without a matching chown, leaving it root:root and unreachable by any non-root user. Both the C++ and the legacy C daemon now apply this, and mvgal-compat reports the socket's real uid:gid and mode instead of advising you to join a group that did not own it.)
• IPC authentication: SO_PEERCRED on the Unix socket. Admitted: root, the daemon's own uid, and members of the mvgal group (falling back to the daemon's primary group) or of the standard video group, including via supplementary group membership.
• Kernel module: Signed at install time; MOK enrollment required for Secure Boot. The signing keypair lives in /var/lib/mvgal/keys/ (directory 0700, private key 0600) and sign-file is installed to /usr/lib/mvgal/sign-file.
• Device nodes: /dev/mvgal* and /dev/mvgal_ntsync are root:mvgal, mode 0660 (since v0.7.31). Two mechanisms set this: config/99-mvgal.rules (SUBSYSTEM=="mvgal", MODE="0660", GROUP="mvgal") establishes the group and mode at device creation, and the privileged helper then chown root:mvgal + chmod 660 each node after loading the modules. (Prior to v0.7.31 the udev rules used GROUP="video" and MODE="0666", which locked out users who were in the mvgal group but not video and left the nodes world-writable. The helper used chown root:video to match.) The helper deliberately does not leave the nodes world-writable.
• pkexec: All privileged operations go through /usr/lib/mvgal/mvgal-pkexec-helper.sh, and the daemon start is a separate /usr/sbin/mvgald action. data/com.mvgal.policy defines exactly these two actions, both auth_admin, and both with exec.allow_gui=false. (v0.7.14 replaced ten orphaned action IDs — none of which was ever annotated on an executable, so polkit fell back to its anonymous org.freedesktop.policykit.exec prompt — with the two that match reality.)
• No firmware flashing: Vendor-overriding firmware operations are explicitly excluded

Performance Characteristics

These figures are design targets, not measurements. The project publishes no benchmark data backing them, and the same numbers were carried forward across earlier revisions without being re-measured. Treat the "Notes" column as the intent of the design; the absolute values are unverified.

The only figures in this document that are backed by code inspection are structural — the ioctl counts, the strategy count, and the protocol constants. Everything measured is currently aspirational.

Limitations and Known Issues

1. No direct upstream kernel integration — the module must be built and signed per-kernel.
2. Anti-cheat compatibility — LD_PRELOAD interception may be flagged by kernel-level anti-cheat (EAC, BattlEye).
3. Network GPU pooling — exploratory design only; not a verified production execution path.
4. AI scheduling — the ML-based scheduler requires training data and is not yet deployed. The AI_DRIVEN strategy exists in the enum, but no trained model ships, and the ai_scheduler.model_path setting in the config file is validated with stat() at load time: a path that does not exist yields a diagnostic and a NULL model, not a crash.
5. Collective communication — the AllReduce/AllGather/Broadcast library is a stub; no UCX/UCC integration.
6. Four char-device ioctls are declared but unimplemented — EXPORT_DMABUF, IMPORT_DMABUF, ALLOC_CROSS_VENDOR, and FREE_CROSS_VENDOR are defined in the UAPI header and return -EINVAL from the dispatcher. They are reserved for future DMA-BUF and cross-vendor work.
7. VRAM allocation and compute submission are not functional — as of v0.7.12 these paths return -EOPNOTSUPP. See MEMORY for what that means for the memory architecture, and STATUS for the capability matrix.
8. Steam runtime behaviour is unverified on hardware — the frame pacer's vsync wait and the Steam compatdata paths have not been observed on a multi-GPU machine. Everything in STEAM_INTEGRATION is either read from source or explicitly marked unverified.