High CPU usage due to frequent kallsyms reloading

[bobrik]

As noted in #1019, there's a problem with elevated CPU usage when parsing kallsyms for heavy bpf users.

There's some more context in that PR from #1019 (comment) and down.

The problem seems two-fold:

1. If there are many bpf symbols, the parsing is slowed down. There's a patch to fix this upstream from me here and an alternative from @florianl here. This takes care of slowness when reading bpf symbols, but it doesn't address the fact that the baseline is bad too:

2. If there's a significant churn in bpf symbols due to frequent loading in unloading (from bpftool prog show or anything else), just happens all the time with a 100ms pause:

opentelemetry-ebpf-profiler/kallsyms/kallsyms.go

Lines 578 to 583 in db2fc46

	case <-s.reloadSymbols:
	// Just trigger reloading of symbols with small delay to batch
	// potentially multiple module loads.
	if nextKallsymsReload == noTimeout {
	nextKallsymsReload = time.After(100 * time.Millisecond)
	}

The pause helps with update coalescing, but it's still 100% of on-CPU time for ~100ms with a 100ms pause, which is a non-trivial amount.

---

Given that we already subscribe to bpf_ksym_add and it receives the following struct as an argument, we should have all the information to add symbols without full /proc/kallsyms re-parsing:

struct bpf_ksym {
	unsigned long		 start;
	unsigned long		 end;
	char			 name[KSYM_NAME_LEN];
	// ...
};

[bobrik]

I tried working on this a little bit. My general idea:

1. Attach kprobes to bpf_ksym_add and bpf_ksym_del. Send updates to userspace with (op, start, end, name) where op is either add or del.
2. Parse /proc/kallsyms like one normally would on start-up.
3. Start processing updates from kprobes and adjust the mapping in userspace.
4. If there happen to be updates from module loading, re-parse /proc/kallsyms like normal. Keep processing the updates.
5. If you happen to fail to send an update, trigger full /proc/kallsyms parsing as it is done today.

This would effectively remove any parsing or /proc/kallsyms for users who do not load any modules, which is approximately everybody in a steady state.

In terms of races:

1. If we process an add already seen through /proc/kallsyms: no-op.
2. If we process a del already seen through /proc/kallsyms: no-op.
3. Additions and removals are ordered: you can't del a program before you add it.
4. Update processing should be blocked while /proc/kallsyms is being read.

---

One problem here is that KSYM_NAME_LEN is 512 bytes, which means it doesn't fit on the stack.

With bpf_perf_event_output you don't have the luxury of reserving some space in a ringbuf with bpf_ringbuf_reserve to work around this. Until the minimum kernel is bumped in #1178, having a completely different path for bpf ksyms for newer kernels doesn't feel right.

In the meantime, perhaps the 100ms reload delay should be configurable. We'd set it to something like 60s as a little stale bpf programs are more tolerable than expensive parsing in a loop. It is more likely that interesting and expensive ebpf programs are the long lived ones anyway.

[fabled]

Have you looked at using perf_event_open with PERF_RECORD_KSYMBOL which would dynamically report the symbol names on a dedicated channel? This works since Linux 5.0.

[bobrik]

I was not aware of it, but it seems like it could work here.

[bobrik]

Support in go-perf: elastic/go-perf#6.

[bobrik]

Quick & ugly proof of concept: bobrik@5927903.

PR: #1198.