eBPF¶
Overview¶
Instead of relying on static counters and gauges exposed by the operating system, eBPF enables the collection & in-kernel aggregation of custom metrics and generation of visibility events based on a wide range of possible sources.
- ebpf.io
- Infrastructure including the Linux Kernel, Compilers (LLVM, gcc), Libraries (Go, C/C++, Rust)
- Applications
Projects¶
Observability¶
- Pixie, Kubernetes observability for developers, auto-instrumented, scriptable.
- Coroot, Kubernetes Observability, implements service maps using eBPF.
- Parca, Continuous Profiling
- ebpf_exporter, Prometheus exporter for custom eBPF metrics
- OpenTelemetry eBPF Collectors, low level kernel telemetry data on a host Kernel, from the cloud or within a Kubernetes cluster.
Security¶
- Cilium, network connectivity security and observability. The Cilium Story So Far, April 2023.
- Tracee, Runtime Security and Forensics
- Falco, Kubernetes threat detection engine. Use case example: Package dependency scanning with GitLab Package Hunter
Security PoCs:
- Boopkit, Linux eBPF backdoor over TCP. Spawn reverse shells, RCE, on prior privileged access.
SRE/DevOps¶
- Inspektor Gadget, A collection of eBPF-based gadgets to debug and inspect Kubernetes apps and resources
- Caretta, instant Kubernetes service dependency map in Grafana, using VictoriaMetrics as backend.
- BumbleBee, build, run and distribute eBPF programs using OCI images.
- q, surface linux networking metrics with eBPF by Kris Nova.
Hot Topics¶
- The Power of eBPF for Cloud Native Systems is a comprehensive deep-dive into cloud-native, IoT and Edge computing, and ideas how to monetize eBPF. Suggest watching Hello eBPF! Goodbye Sidecars by Liz Rice as additional learning insight, and dive into eBPF and its capabilities.
- Learn how eBPF can help minimize "observability tax"
- eBPF: Why now, introduction and deep dive
- eBPF report by Liz Rice
- Bypassing eBPF-based Security Enforcement Tools
Learning Resources¶
- Learning eBPF book by Liz Rice, published March 2023.
- Learning eBPF for better Observability article on InfoQ.com - learning experience step-by-step by Michael Friedrich, published May 2023.
- eBPF learning story shared by Michael Friedrich in their talk "From Monitoring to Observability: eBPF Chaos" at Config Management Camp 2023.
- Capture The Flag Challenges for eBPF Summit 2022
- awesome-ebpf
Newsletters¶
Books and blog posts¶
- Learning eBPF by Liz Rice.
- BPF Performance Tools (Book)
- How we diagnosed and resolved Redis latency spikes with BPF and other tools is a thorough learning walkthrough from a problem, analysis, attempts, to final solutions.
- BlackHat Arsenal 2022: Detecting Linux kernel rootkits with Aqua Tracee
- Measuring CPU usage of eBPF programs with Inspektor Gadget
Development¶
- Learning eBPF Tracing: Tutorials and Examples (2019) recommended
- bpftrace
- bcc (BPF Compiler Collection)´
- libbpf-bootstrap: Examples that provide different use cases, for example traffic monitoring using XDP, written in Rust.
- An eBPF tutorial to try out the bpftrace framework
- The art of writing eBPF programs: a primer.
- Writing an eBPF/XDP load-balancer in Rust
- Getting Started on Kubernetes observability with eBPF
CO-RE (Compile Once, Run Everywhere)¶
- The Challenge with Deploying eBPF Into the Wild
- Andrii Nakryiko: BPF CO-RE reference guide
- Andrii Nakryiko: BPF CO-RE (Compile Once – Run Everywhere)
Debugging Tips¶
- Elastic blog: Code coverage for eBPF programs
- Andrii Nakryiko: Guide to bpf_trace_printk() and bpf_printk()
eBPF Libraries¶
- cilium/ebpf-go (Go) - Use case examples
- aquasecurity/libbpfgo (Go)
- libbpf (C/C++)
- Wrapped by aquasecurity/libbpfgo
- libbpf-rs (Rust)
- redbpf (Rust)
- aya-rs (Rust)
- Program lifecycle
- Used by ebpfguard to implement Linux security policies.
- Parca agent used Aya but migrated to libbpf
Platforms¶
Events¶
- eBPF Summit 2022 summary in the opsindev.news newsletter
- eBPF day at KubeCon EU 2022, summary in the opsindev.news newsletter
Meetups¶
- 54. #EveryoneCanContribute Cafe: Pixie for Kubernetes Observability
- 52. #EveryoneCanContribute Cafe: Learned at KubeCon EU, feat. Cilium Tetragon first try
- 49. #EveryoneCanContribute Cafe: Aqua Security and Open Source
- 42. #EveryoneCanContribute cafe: Falco and GitLab Package Hunter
- 32. #EveryoneCanContribute cafe: Continuous Profiling with Polar Signals