How Do I Profile eBPF Code? Expert Techniques and Tools for Optimal Performance

Introduction

eBPF (extended Berkeley Packet Filter) has revolutionized how we observe, secure, and optimize Linux systems. By allowing safe execution of sandboxed programs inside the kernel, eBPF powers tools for tracing, networking, security, and performance monitoring. However, as with any kernel-level code, eBPF programs themselves can become performance bottlenecks. Profiling eBPF code is essential to detect excessive overhead, missed events, or unintended side effects.

This guide answers the practical question: How do I profile eBPF code? We’ll walk through the most reliable tools, step‑by‑step workflows, and real‑world examples. Whether you’re debugging a custom kprobe or tuning an XDP firewall, these techniques help you identify where your eBPF program spends its time and how to make it faster.

Why Profile eBPF Code?

Before diving into tools, it’s important to understand what can go wrong in an eBPF program:

• High overhead per event – especially when hooking frequent functions (e.g., tcp_rcv_establish).
• Map contention – multiple CPUs updating the same hash map can degrade performance.
• Verifier limits – programs that approach the complexity limit may fail or run inefficiently.
• Tail call chains – improper tail call planning can increase latency.

Profiling helps you answer: “Is my eBPF program itself slowing down the workload it’s observing?”

Key Tools for eBPF Profiling

The eBPF ecosystem includes several powerful profiling instruments. The table below summarizes the most common ones.

bpftrace is perfect for a quick “is this eBPF program a problem?” scan. BCC provides deeper analysis with tools like funclatency, trace, and profile. perf can sample eBPF program executions via the bpf_prog_run tracepoint. The kernel also exposes per‑program statistics under /sys/kernel/debug/tracing/events/bpf/.

Step‑by‑Step Profiling Workflow

1. Identify the eBPF Program and Its Hook Point

Use bpftool prog list to see all loaded eBPF programs and their IDs, types (kprobe, tracepoint, xdp, etc.), and attached targets.

bpftool prog list

Note the program ID and the kernel function it hooks (e.g., tcp_connect).

2. Measure Execution Latency with bpftrace

A simple one‑liner can show the distribution of execution times for a specific eBPF program. For example, to profile all kprobe programs that hook tcp_connect:

bpftrace -e 'kprobe:tcp_connect { @start[tid] = nsecs; } kretprobe:tcp_connect /@start[tid]/ { @us = hist((nsecs - @start[tid]) / 1000); delete(@start[tid]); }'

This captures the duration of each eBPF program run and prints a histogram. If you see many events taking >10μs, you likely need to optimize.

3. Use BCC’s funclatency for Function‑Level Timing

If your eBPF program calls a specific kernel function (e.g., bpf_map_lookup_elem), you can measure how long that helper takes:

funclatency bpf_map_lookup_elem

BCC’s trace tool can also count how many times a specific helper is invoked per second.

4. Sample eBPF Program Runs with perf

The bpf_prog_run tracepoint fires every time an eBPF program executes. Use perf record to collect stack samples and then perf report to see which programs consume most CPU:

perf record -e bpf:bpf_prog_run -ag -- sleep 10
perf report

This shows you the distribution of eBPF program runs across different CPUs and the user‑space call stacks that trigger them.

5. Examine eBPF‑Specific Statistics from the Kernel

The kernel keeps counters for each eBPF program:

cat /proc/sys/kernel/bpf_stats_enabled  # check if enabled (must be 1)
cat /sys/kernel/debug/tracing/events/bpf/bpf_prog_get_info/format

However, the easiest way is to use bpftool prog show id <ID>:

bpftool prog show id 23 --pretty

Look for run_cnt (total number of runs) and run_time_ns (total time spent executing). Average run time = run_time_ns / run_cnt. Multiply by events per second to estimate overhead.

Advanced Profiling Techniques

Sampling vs. Tracing

• Sampling (e.g., perf with -F 99) touches the system less and can expose hot eBPF programs without skewing the results.
• Tracing (e.g., bpftrace on every event) gives exact per‑event data but may itself add overhead. For frequent hooks, start with sampling.

Verifier Complexity Analysis

Set the BPF verifier log level to 2 (verbose) to see the number of instructions processed. A program close to the 1‑million‑instruction limit can cause high CPU usage during verification only, but the runtime is unaffected. Still, it’s a red flag if your program uses many helper calls.

bpftool prog load program.o  /sys/fs/bpf/prog  -p 2

Map Profiling

Map operations (lookup, update, delete) are often the biggest cost. Use BCC’s map_profile (or write a small bpftrace script to count map helper calls).

Example bpftrace script to count map lookups:

kprobe:bpf_map_lookup_elem
{
    @map_lookups = count();
}

If lookups exceed millions per second, consider using PERCPU maps to avoid lock contention.

CPU‑Level Distribution

Use bpftrace to count executions per CPU:

bpftrace -e 'tracepoint:bpf:bpf_prog_run { @[cpu] = count(); }'

Unexpectedly high counts on a single CPU may indicate an imbalance in RSS (receive side scaling) for XDP programs.

Common Pitfalls and Best Practices

• Excessive helper calls – each bpf_map_lookup_elem or bpf_get_current_pid_tgid takes time. Minimize them inside loops.
• Large global maps – per‑CPU maps reduce contention but increase memory. Profile both lookup speed and memory footprint.
• Ignoring tail call overhead – tail calls are fast (single jump), but chaining many tails can degrade performance. Keep chains short.
• Forgetting bpf_stats_enabled – without this sysctl set to 1, you won’t have per‑program run time counters.
• Profiling during low load – always profile under realistic load to see the effect of contention.

Real‑World Example: Profiling an XDP Drop Program

Imagine an XDP program that drops all IPv4 packets from a specific source IP. It uses a hash map to store the blacklist. The program runs on a 40 Gbps interface, but we suspect it adds too much latency.

1. List programs: bpftool prog list | grep xdp → ID 42.
2. Check run time: bpftool prog show id 42 → run_time_ns = 1,234,567,890 and run_cnt = 50,000,000. Average = 24.7 ns per packet. Quite good, but maybe maps are slow.
3. Profile map lookups: Use bpftrace to count bpf_map_lookup_elem calls. Discover that each packet triggers two lookups (one for IPv4 check, one for source IP). That’s fine.
4. Check CPU distribution: bpftrace -e 'tracepoint:bpf:bpf_prog_run { @[cpu] = count(); }' shows 70% of runs on CPU 0. The interface RSS is unbalanced. Fixing the RSS hash prevents flow pinning.
5. Result: After balancing, average latency stays low, but throughput improves because CPU 0 no longer saturates.

Integrating Profiling Data into Monitoring Platforms

Once you’ve identified the key metrics (run count, average latency, map utilization), it’s beneficial to feed them into a centralized observability system. Many organizations export eBPF statistics to Prometheus or stream them to Grafana for dashboards and alerting.

ASI Biont supports connecting to Grafana through its API, enabling you to visualize eBPF profiling data alongside your application metrics. For details, see asibiont.com/courses.

Conclusion

Profiling eBPF code doesn’t require kernel wizardry. With tools like bpftrace, BCC, and perf, you can quickly identify overhead, contention, and inefficiencies in your eBPF programs. Start with bpftool prog show to get baseline statistics, move to sampling with perf, and dive into specific helpers with bpftrace. Always profile under realistic load and watch for map contention and CPU imbalance.

By regularly profiling eBPF code, you ensure your observability and security tools remain practically invisible – exactly as they were designed to be.