Can't build BPF program on Fedora 40 and Debian 13 (Linux 6.10)

[lnicola]

Just a heads-up, since someone else might run into this:

=== [0x.tools] xcapture-bpf 2.0.3 BETA by Tanel Poder. Fedora Linux 40 6.10.6 x86_64
===  Loading BPF...
In file included from /virtual/main.c:31:
In file included from include/uapi/linux/ptrace.h:183:
In file included from arch/x86/include/asm/ptrace.h:175:
In file included from arch/x86/include/asm/paravirt_types.h:12:
In file included from arch/x86/include/asm/nospec-branch.h:15:
arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
   47 |                 return this_cpu_read_const(const_pcpu_hot.current_task);
      |                        ^
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:30: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
arch/x86/include/asm/percpu.h:93:28: note: expanded from macro '__my_cpu_ptr'
   93 | #define __my_cpu_ptr(ptr)       (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr)
      |                                  ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:31:
In file included from include/uapi/linux/ptrace.h:183:
In file included from arch/x86/include/asm/ptrace.h:175:
In file included from arch/x86/include/asm/paravirt_types.h:12:
In file included from arch/x86/include/asm/nospec-branch.h:15:
arch/x86/include/asm/current.h:47:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:9: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:32:
In file included from include/linux/sched.h:13:
arch/x86/include/asm/processor.h:543:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
  543 |                 return this_cpu_read_const(const_pcpu_hot.top_of_stack);
      |                        ^
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:30: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                                     ^
arch/x86/include/asm/percpu.h:93:28: note: expanded from macro '__my_cpu_ptr'
   93 | #define __my_cpu_ptr(ptr)       (__my_cpu_type(*(ptr))*)(__force uintptr_t)(ptr)
      |                                  ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:32:
In file included from include/linux/sched.h:13:
arch/x86/include/asm/processor.h:543:10: warning: multiple identical address spaces specified for type [-Wduplicate-decl-specifier]
arch/x86/include/asm/percpu.h:456:34: note: expanded from macro 'this_cpu_read_const'
  456 | #define this_cpu_read_const(pcp)        __raw_cpu_read(, pcp)
      |                                         ^
arch/x86/include/asm/percpu.h:426:9: note: expanded from macro '__raw_cpu_read'
  426 |         *(qual __my_cpu_type(pcp) *)__my_cpu_ptr(&(pcp));               \
      |                ^
arch/x86/include/asm/percpu.h:92:40: note: expanded from macro '__my_cpu_type'
   92 | #define __my_cpu_type(var)      typeof(var) __percpu_seg_override
      |                                             ^
arch/x86/include/asm/percpu.h:45:31: note: expanded from macro '__percpu_seg_override'
   45 | #define __percpu_seg_override   __seg_gs
      |                                 ^
<built-in>:349:33: note: expanded from macro '__seg_gs'
  349 | #define __seg_gs __attribute__((address_space(256)))
      |                                 ^
In file included from /virtual/main.c:34:
In file included from include/linux/syscalls.h:93:
In file included from include/trace/syscall.h:7:
In file included from include/linux/trace_events.h:10:
In file included from include/linux/perf_event.h:62:
In file included from include/linux/security.h:35:
include/linux/bpf.h:348:10: error: invalid application of 'sizeof' to an incomplete type 'struct bpf_wq'
  348 |                 return sizeof(struct bpf_wq);
      |                        ^     ~~~~~~~~~~~~~~~
include/linux/bpf.h:348:24: note: forward declaration of 'struct bpf_wq'
  348 |                 return sizeof(struct bpf_wq);
      |                                      ^
include/linux/bpf.h:377:10: error: invalid application of '__alignof' to an incomplete type 'struct bpf_wq'
  377 |                 return __alignof__(struct bpf_wq);
      |                        ^          ~~~~~~~~~~~~~~~
include/linux/bpf.h:377:29: note: forward declaration of 'struct bpf_wq'
  377 |                 return __alignof__(struct bpf_wq);
      |                                           ^
4 warnings and 2 errors generated.
Traceback (most recent call last):
  File "/home/grayshade/Projects/0xtools/bin/xcapture-bpf", line 474, in <module>
    b = BPF(text= ifdef + bpf_text)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/bcc/__init__.py", line 479, in __init__
    raise Exception("Failed to compile BPF module %s" % (src_file or "<text>"))
Exception: Failed to compile BPF module <text>

You can ignore the warnings (biolatency shows them too), but I don't understand the problem with the missing struct. It's defined in /usr/src/kernels/6.10.6-200.fc40.x86_64/include/uapi/linux/bpf.h:

struct bpf_wq {                                                                                                                                                                                                                       
  __u64 __opaque[2];                                                                                                                                                                                                                  
} __attribute__((aligned(8)));                                                                                                                                                                                                        

Which /usr/src/kernels/6.10.6-200.fc40.x86_64/include/linux/bpf.h includes:

#include <uapi/linux/bpf.h>

// ...

static inline u32 btf_field_type_size(enum btf_field_type type)
{
        switch (type) {
        case BPF_SPIN_LOCK:
                return sizeof(struct bpf_spin_lock);
        case BPF_TIMER:
                return sizeof(struct bpf_timer);
        case BPF_WORKQUEUE:
                return sizeof(struct bpf_wq);
        case BPF_KPTR_UNREF:
        case BPF_KPTR_REF:
        case BPF_KPTR_PERCPU:
                return sizeof(u64);
        case BPF_LIST_HEAD:
                return sizeof(struct bpf_list_head);
        case BPF_LIST_NODE:
                return sizeof(struct bpf_list_node);
        case BPF_RB_ROOT:
                return sizeof(struct bpf_rb_root);
        case BPF_RB_NODE:
                return sizeof(struct bpf_rb_node);
        case BPF_REFCOUNT:
                return sizeof(struct bpf_refcount);
        default:
                WARN_ON_ONCE(1);
                return 0;
        }
}

---

TL;DR: this fails to build on 6.10:

from bcc import BPF

bpf_text = """
#include <uapi/linux/bpf.h>
#include <linux/syscalls.h>
"""
BPF(text=bpf_text)

[tanelpoder]

Interesting, I'll test it on Debian 13 and see what's up. Does Debian 13 have 6.10 kernel as a default or did you upgrade the kernel separately?

There's a chance that since BCC is somewhat deprecated (in favor to libbpf approach), maybe there's some header/kernel incompatibility, like what I saw on Ubuntu 22.04.

My next step is to build v3 based on libbpf (using BTF/CO-RE), this should avoid future kernel/header incompatibility issues, in addition to all the other benefits.

[lnicola]

Does Debian 13 have 6.10 kernel as a default or did you upgrade the kernel separately?

apt update && apt upgrade will give you 6.10.6.

[tanelpoder]

It took a while, but I finally attempted to test on Debian 13. I didn't realize that it's not a GA release yet... there's a good chance that you'd need to separately download & build the latest bcc toolset (instead of the bundled package) to make it work.

I saw various issues on Ubuntu 22.04 too (24.04 works ok) - given that bcc is pretty much deprecated in favor or proper pre-compiled binaries that rely on BTF and CO-RE (using libbpf), I stopped working on v2 and started working on v3 that does everything right according to the modern state of eBPF. The highly in-flux prototype of v3 (using libbpf + BTF + CO-RE) is in the "next" directory:

• https://github.com/tanelpoder/0xtools/tree/master/next

Once that's out, it should not have any issues with missing or incompatible headers (even if people are using non-standard kernels that usually come with the distro, etc).

I did demo some absolute basics of the new approach (BPF task_iterator, etc) at my talk with Liz Rice a few months ago (the last 10 mins of the video or so):

• https://www.youtube.com/watch?v=DFYLkYwgaGE

I'm leaving this issue open, in case anyone else with a similar issue ends up coming here. Thanks!

[kdenis4]

Hi Tanel,
Still keep having installation issues on Ubuntu 22.04, there is no possibility to use HWE in closed production, also seems the 'next' branch is not available also any more. Could you please advice whether there is any way to get your tools working here, on Ubuntu 22.04?

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.5 LTS
Release:	22.04
Codename:	jammy
uname -r
5.15.0-160-generic

[tanelpoder]

Hello @kdenis4.

Try this (after pulling the latest changes). I had replaced the regular make with cmake (for building RPM and DEB packages, but I apparently forgot to update the readme for building for the "older" kernels (previously it was just "make old"):

cd 0xtools/xcapture
make cleanx
cmake -DOLD_KERNEL_SUPPORT=ON -B build -S .
cmake --build build

Then run xcapture. That worked on Ubuntu 22.04 with 5.15 kernel in my tests:

cd build
sudo TZ=:/etc/localtime chrt -r 30 xcapture

Any issues, let me know, we'll figure it out :-)