eBPF¶
Overview¶
Instead of relying on static counters and gauges exposed by the operating system, eBPF enables the collection & in-kernel aggregation of custom metrics and generation of visibility events based on a wide range of possible sources.
- ebpf.io
- Infrastructure including the Linux Kernel, Compilers (LLVM, gcc), Libraries (Go, C/C++, Rust)
- Applications
Projects¶
Observability¶
- Pixie, Kubernetes observability for developers, auto-instrumented, scriptable.
- Coroot, Kubernetes Observability, implements service maps using eBPF.
- Parca, Continuous Profiling
- ebpf_exporter, Prometheus exporter for custom eBPF metrics
- OpenTelemetry eBPF Collectors, low level kernel telemetry data on a host Kernel, from the cloud or within a Kubernetes cluster.
- Kepler (Kubernetes Efficient Power Level Exporter) uses eBPF to probe energy-related system stats and exports them as Prometheus metrics.
Security¶
- Cilium, network connectivity security and observability. The Cilium Story So Far, April 2023.
- Tracee, Runtime Security and Forensics
- Falco, Kubernetes threat detection engine. Use case example: Package dependency scanning with GitLab Package Hunter
- Deepfence ThreatMapper, Runtime Threat Management and Attack Path Enumeration for Cloud Native
Security PoCs:
- Boopkit, Linux eBPF backdoor over TCP. Spawn reverse shells, RCE, on prior privileged access.
SRE/DevOps¶
- Inspektor Gadget, A collection of eBPF-based gadgets to debug and inspect Kubernetes apps and resources
- Caretta, instant Kubernetes service dependency map in Grafana, using VictoriaMetrics as backend.
- BumbleBee, build, run and distribute eBPF programs using OCI images.
- q, surface linux networking metrics with eBPF by Kris Nova.
Zero code instrumentation:
- Odigos provides distributed tracing without code changes. Instantly monitor any application using OpenTelemetry and eBPF.
- Deepflow implemented Zero Code data collection with eBPF for metrics, distributed tracing, request logs and function profiling.
Hot Topics¶
- Past, Present, & Future of eBPF in Cloud Native Observability - Frederic Branczyk & Natalie Serrino, KubeCon EU 2023.
- The Power of eBPF for Cloud Native Systems is a comprehensive deep-dive into cloud-native, IoT and Edge computing, and ideas how to monetize eBPF. Suggest diving into eBPF and its capabilities.
- eBPF: Why now, introduction and deep dive
- eBPF report by Liz Rice
- Bypassing eBPF-based Security Enforcement Tools
- eCHO Episode 93: BPF Signing
Use cases¶
- Learn how eBPF can help minimize "observability tax"
- Distributed patterns compared: Frameworks vs. K8s vs. Service Mesh vs. eBPF by Matthias Haeussler and Tiffany Jernigan at Devoxx UK
- Hello eBPF! Goodbye Sidecars by Liz Rice
- Life Without Sidecars - Is eBPF's Promise Too Good to Be True?
Learning Resources¶
- Learning eBPF book by Liz Rice, published March 2023.
- Learning eBPF tutorial by Isovalent
- Learning eBPF for better Observability workshop at Cloudland 2023
- Learning eBPF for better Observability article on InfoQ.com - learning experience step-by-step by Michael Friedrich, published May 2023.
- Debugging Production: eBPF Chaos article on InfoQ.com - eBPF use cases for production, and how to verify reliability with chaos engineering by Michael Friedrich, published June 2023.
- eBPF learning story shared by Michael Friedrich in their talk "From Monitoring to Observability: eBPF Chaos" at Config Management Camp 2023.
- Capture The Flag Challenges for eBPF Summit 2022
- awesome-ebpf
Newsletters¶
Books and blog posts¶
- Learning eBPF by Liz Rice.
- BPF Performance Tools (Book)
- How we diagnosed and resolved Redis latency spikes with BPF and other tools is a thorough learning walkthrough from a problem, analysis, attempts, to final solutions.
- BlackHat Arsenal 2022: Detecting Linux kernel rootkits with Aqua Tracee
- Measuring CPU usage of eBPF programs with Inspektor Gadget
Development¶
- Learning eBPF Tracing: Tutorials and Examples (2019) recommended
- bpftrace
- bcc (BPF Compiler Collection)´
- libbpf-bootstrap: Examples that provide different use cases, for example traffic monitoring using XDP, written in Rust.
- An eBPF tutorial to try out the bpftrace framework
- The art of writing eBPF programs: a primer.
- Getting Started on Kubernetes observability with eBPF
- buzzer, a fuzzer toolchain to write eBPF fuzzing strategies. These generate random eBPF programs and then validate that they do not have unexpected behavior in running on a Linux Kernel.
Development Tools¶
- eBPF explorer is Web UI that lets you see all the maps and programs in eBPF subsystem.
Development: XDP¶
Everything focussed on network communication and XDP (eXpress Data Path).
- Writing an eBPF/XDP load-balancer in Rust
- Building an XDP eBPF Program with C and Golang: A Step-by-Step Guide
- BPFAgent: eBPF for Monitoring at DoorDash
Testing and CI/CD¶
CO-RE (Compile Once, Run Everywhere)¶
- The Challenge with Deploying eBPF Into the Wild
- Andrii Nakryiko: BPF CO-RE reference guide
- Andrii Nakryiko: BPF CO-RE (Compile Once – Run Everywhere)
Debugging Tips¶
- Elastic blog: Code coverage for eBPF programs
- Andrii Nakryiko: Guide to bpf_trace_printk() and bpf_printk()
eBPF Libraries¶
- cilium/ebpf-go (Go) - Use case examples
- aquasecurity/libbpfgo (Go)
- libbpf (C/C++)
- Wrapped by aquasecurity/libbpfgo
- libbpf-rs (Rust)
- aya-rs (Rust)
Platforms¶
Events¶
- eBPF Summit 2022 summary in the opsindev.news newsletter
- eBPF day at KubeCon EU 2022, summary in the opsindev.news newsletter
Meetups¶
- 54. #EveryoneCanContribute Cafe: Pixie for Kubernetes Observability
- 52. #EveryoneCanContribute Cafe: Learned at KubeCon EU, feat. Cilium Tetragon first try
- 49. #EveryoneCanContribute Cafe: Aqua Security and Open Source
- 42. #EveryoneCanContribute cafe: Falco and GitLab Package Hunter
- 32. #EveryoneCanContribute cafe: Continuous Profiling with Polar Signals