pivid

Experimental video code for Linux / Raspberry Pi

Building

1. Use a Raspberry Pi with a fully updated bullseye install and 2G+ RAM.
2. Edit /boot/config.txt as follows, and reboot:

#dtoverlay=vc4-fkms-v3d  # Comment/remove old dtoverlay=vc4* lines
# Use full KMS and H.265 (HEVC) decoding, 512M CMA for frames
dtoverlay=vc4-kms-v3d,cma-512
dtoverlay=rpivid-v4l2

3. Run ./dev_setup.py. (Works On My Machine™, YMMV)
4. Run ninja -C build to run the actual build (this is the only part to repeat after edits).
5. Run binaries from build (like build/pivid_test_playback).
6. If things get weird, rm -rf build and start over with dev_setup.py.

Notable programs:

• pivid_test_playback - play a video file via KMS/DRM (stop X first)
• pivid_inspect_avformat - print a summary of streams in a video file
• pivid_inspect_kms - print capabilities & properties of KMS/DRM devices
• pivid_inspect_v4l2 - print capabilities & properties of V4L2 devices

Use --help to see usage (and/or read the source).

Notes and links

Raspberry Pi specifics

• All about accelerated video on the Raspberry Pi - my notes
• kernel.org: V3D Graphics Driver - RPi 4 GPU kernel driver docs
• rpi kernel source: drivers/gpu/drm/v3d - RPi 4 GPU kernel driver source
• rpi kernel source: include/uapi/v3d_drm.h - ioctl defs for RPi 4 GPU kernel driver

Graphics output: DRM and KMS

• Wikipedia: Direct Rendering Manager - a good overview
• Blog post: "From pre-history to beyond the global thermonuclear war" - Linux graphics history
• LWN article: Atomic mode setting design overview - the current KMS API
• Man page: Direct Rendering Manager - Kernel Mode-Setting - incomplete but helpful
• Man page: Direct Rendering Manager - Memory Management - incomplete but helpful
• kernel.org: Linux GPU Driver Userland Interfaces - basic notes on the kernel/user interface
• kernel.org: KMS Properties - exhaustive object property list
• kernel source: include/uapi/drm/drm.h and drm_mode.h - kernel/user headers
• kernel source: include/drm/drm_print.h - debugging definitions
• ST Micro: DRM/KMS Overview - decent general docs from a chip vendor
• ST Micro: How to trace and debug the framework - an especially useful section
• NVIDIA Jetson Linux API: Direct Rendering Manager - API reference, a bit NVIDIA-specific
• libdrm - library wrapper; see xf86drm.h and xf86drmMode.h (not X-specific despite "xf86")
• libgbm - GPU allocation helper library; see gbm.h
• modetest - command line tool (in the libdrm-tests Debian package)
• kmscube - oft-referenced KMS/GL example program
• kms++ - C++ KMS wrapper & utilities

Notes on DRM and KMS:

• These interfaces are almost completely undocumented. Learn by examples.
• But, many examples use "legacy" interfaces, prefer "atomic" update interfaces.

Video decoding: V4L2

• kernel.org: Video for Linux API - kernel/user interface
• kernel.org: Video for Linux - Memory-to-Memory Stateful Video Decoder Interface
• kernel source: include/uapi/linux/videodev2.h - kernel/user header
• libv4l - thin library wrapper with format conversion; see libv4l2.h
• v4l-utils - useful tools, especially v4l2-ctl
• libavformat - for unpacking containers (.mp4, .mkv); see avformat.h and avio.h

Zero-copy buffer sharing

• kernel.org: Buffer Sharing and Synchronization - kernel buffer management (and user interface)
• kernel.org: Linux GPU Memory Management - PRIME buffer sharing - exporting GPU buffers as "dma-bufs"
• kernel.org: Video for Linux - Streaming I/O (DMA buffer importing) - using "dma-buf" objects in V4L2
• hello_drmprime - nice example of hardware H.264/H.265 sending to DRM/KMS with zero copy

Notes on memory management:

• GPUs have all kinds of memory architectures. (The RPi is simple, everything is in CPU RAM without dedicated GPU RAM.)
• DRM/KMS requires you to define a "framebuffer", giving it one or more "buffer objects" (aka "GEM handles") (opaque int32).
• "DRM-PRIME" is a fancy name for ioctl's (DRM_IOCTL_PRIME_FD_TO_HANDLE and vice versa) to convert GEM handles to/from "dma-buf" descriptors.
• On the Pi, memory is simple, so buffers allocated anywhere (V4L2, DRM, etc) can be used anywhere else (shared via "dma-buf").
• Of course you need to get the data format right.