eBPF

Overview

Instead of relying on static counters and gauges exposed by the operating system, eBPF enables the collection & in-kernel aggregation of custom metrics and generation of visibility events based on a wide range of possible sources.

• ebpf.io
  • Infrastructure including the Linux Kernel, Compilers (LLVM, gcc), Libraries (Go, C/C++, Rust)
  • Applications

Projects

Observability

• Pixie, Kubernetes observability for developers, auto-instrumented, scriptable.
• Coroot, Kubernetes Observability, implements service maps using eBPF.
• Parca, Continuous Profiling
• ebpf_exporter, Prometheus exporter for custom eBPF metrics
• OpenTelemetry eBPF Collectors, low level kernel telemetry data on a host Kernel, from the cloud or within a Kubernetes cluster.

Security

• Cilium, network connectivity security and observability. The Cilium Story So Far, April 2023.
  • Tetragon
• Tracee, Runtime Security and Forensics
• Falco, Kubernetes threat detection engine. Use case example: Package dependency scanning with GitLab Package Hunter

Security PoCs:

• Boopkit, Linux eBPF backdoor over TCP. Spawn reverse shells, RCE, on prior privileged access.

SRE/DevOps

• Inspektor Gadget, A collection of eBPF-based gadgets to debug and inspect Kubernetes apps and resources
• Caretta, instant Kubernetes service dependency map in Grafana, using VictoriaMetrics as backend.
• BumbleBee, build, run and distribute eBPF programs using OCI images.
• q, surface linux networking metrics with eBPF by Kris Nova.

Hot Topics

• The Power of eBPF for Cloud Native Systems is a comprehensive deep-dive into cloud-native, IoT and Edge computing, and ideas how to monetize eBPF. Suggest watching Hello eBPF! Goodbye Sidecars by Liz Rice as additional learning insight, and dive into eBPF and its capabilities.
• Learn how eBPF can help minimize "observability tax"
• eBPF: Why now, introduction and deep dive
• eBPF report by Liz Rice
• Bypassing eBPF-based Security Enforcement Tools

Learning Resources

• Learning eBPF book by Liz Rice, published March 2023.
• Learning eBPF for better Observability article on InfoQ.com - learning experience step-by-step by Michael Friedrich, published May 2023.
• eBPF learning story shared by Michael Friedrich in their talk "From Monitoring to Observability: eBPF Chaos" at Config Management Camp 2023.
• Capture The Flag Challenges for eBPF Summit 2022
• awesome-ebpf

Newsletters

• eCHO newsletter
• opsindev.news newsletter
  • The inner dev learning eBPF since Feb 2023

Books and blog posts

• Learning eBPF by Liz Rice.
• BPF Performance Tools (Book)
• How we diagnosed and resolved Redis latency spikes with BPF and other tools is a thorough learning walkthrough from a problem, analysis, attempts, to final solutions.
• BlackHat Arsenal 2022: Detecting Linux kernel rootkits with Aqua Tracee
• Measuring CPU usage of eBPF programs with Inspektor Gadget

Development

• Learning eBPF Tracing: Tutorials and Examples (2019) recommended
• bpftrace
• bcc (BPF Compiler Collection)´
• libbpf-bootstrap: Examples that provide different use cases, for example traffic monitoring using XDP, written in Rust.
• An eBPF tutorial to try out the bpftrace framework
• The art of writing eBPF programs: a primer.
• Writing an eBPF/XDP load-balancer in Rust
• Getting Started on Kubernetes observability with eBPF

CO-RE (Compile Once, Run Everywhere)

• The Challenge with Deploying eBPF Into the Wild
• Andrii Nakryiko: BPF CO-RE reference guide
• Andrii Nakryiko: BPF CO-RE (Compile Once – Run Everywhere)

Debugging Tips

• Elastic blog: Code coverage for eBPF programs
• Andrii Nakryiko: Guide to bpf_trace_printk() and bpf_printk()

eBPF Libraries

• cilium/ebpf-go (Go) - Use case examples
• aquasecurity/libbpfgo (Go)
  • Used by the Parca agent
• libbpf (C/C++)
  • Wrapped by aquasecurity/libbpfgo
  • libbpf-rs (Rust)
• redbpf (Rust)
• aya-rs (Rust)
  • Program lifecycle
  • Used by ebpfguard to implement Linux security policies.
  • Parca agent used Aya but migrated to libbpf

Platforms

• Get started with eBPF using BumbleBee

Events

• eBPF Summit 2022 summary in the opsindev.news newsletter
• eBPF day at KubeCon EU 2022, summary in the opsindev.news newsletter

Meetups

• 54. #EveryoneCanContribute Cafe: Pixie for Kubernetes Observability
• 52. #EveryoneCanContribute Cafe: Learned at KubeCon EU, feat. Cilium Tetragon first try
• 49. #EveryoneCanContribute Cafe: Aqua Security and Open Source
• 42. #EveryoneCanContribute cafe: Falco and GitLab Package Hunter
• 32. #EveryoneCanContribute cafe: Continuous Profiling with Polar Signals