To kick things off in this second era of my blog (ha!), I thought I’d write up an interesting investigation I recently conducted based on an, on its surface, simple question from a colleague (paraphrased):
Can we use BPF to attribute a drop in a netfilter ruleset to the rule that decided to drop the packet?
This turned out to be not quite straight-forward, because of the way the netfilter code is structured (as we will see below), but it was interesting because it allowed me to experiment with attaching kprobes into the middle of the function. Something I knew was possible using BPF, but not something I had really experimented with in practice before.
The anatomy of a netfilter ruleset
To expand a bit on what the question really means, let’s have a look at the
nft
ruleset currently installed on my laptop (which is what I used to test
this):
$ sudo nft -a list ruleset
table inet filter { # handle 6
chain input { # handle 1
type filter hook input priority filter; policy drop;
ct state { established, related } accept # handle 5
ct state invalid drop # handle 6
tcp dport 10000 drop # handle 7
iifname "lo" accept # handle 8
iifname "veth0" accept # handle 9
iifname "eth0" accept # handle 10
ip protocol icmp accept # handle 11
ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept # handle 13
}
chain forward { # handle 2
type filter hook forward priority filter; policy drop;
}
chain output { # handle 3
type filter hook output priority filter; policy accept;
}
}
Looking at the input
chain, each line is a rule that will be executed in
sequence, and if a rule matches the packet and contains a verdict (accept
or
drop
), processing of further rules will stop, rending that verdict as the
final one for the packet. What we want to do is be able to hook into the
processing and whenever a drop
verdict is reached, look at both the packet
data (to know which packet it was, e.g., to identify the flow it belongs to),
and also know which rule was the cause of the drop verdict so we can
troubleshoot our ruleset.
As can be seen from the output above, each rule has a ‘handle’ that identifies it with its chain. So whenever a particular packet is dropped, what we want is to find the handle of the rule that dropped it. I added the (redundant) rule with handle 7 as a test rule just for experimenting.
Before we jump down the BPF rabbit hole for this, I should mention that
netfilter itself does have some facilities for getting this information. For one
thing, you can add a counter
statement to the rule if you just want statistics
on how often that rule is hit. Or you can use the netfilter logging facilities
to get per-packet output in system logs or passed to a userspace socket. But
those require modifications of the ruleset, and are either too low granularity
(the counter), or fairly heavy weight (the logging), so let’s see if we can get
the result we want with BPF.
Diving into the netfilter kernel code
To find a good place to place our hook, let’s first have a look at how netfilter
executes its rules. The main kernel function that executes a netfilter chain is
nft_do_chain()
which lives in net/netfilter/nf_tables_core.c
and looks like
this:
unsigned int
nft_do_chain(struct nft_pktinfo *pkt, void *priv)
{
const struct nft_chain *chain = priv, *basechain = chain;
const struct nft_rule_dp *rule, *last_rule;
const struct net *net = nft_net(pkt);
const struct nft_expr *expr, *last;
struct nft_regs regs = {};
unsigned int stackptr = 0;
struct nft_jumpstack jumpstack[NFT_JUMP_STACK_SIZE];
bool genbit = READ_ONCE(net->nft.gencursor);
struct nft_rule_blob *blob;
struct nft_traceinfo info;
info.trace = false;
if (static_branch_unlikely(&nft_trace_enabled))
nft_trace_init(&info, pkt, ®s.verdict, basechain);
do_chain:
if (genbit)
blob = rcu_dereference(chain->blob_gen_1);
else
blob = rcu_dereference(chain->blob_gen_0);
rule = (struct nft_rule_dp *)blob->data;
last_rule = (void *)blob->data + blob->size;
next_rule:
regs.verdict.code = NFT_CONTINUE;
for (; rule < last_rule; rule = nft_rule_next(rule)) {
nft_rule_dp_for_each_expr(expr, last, rule) {
if (expr->ops == &nft_cmp_fast_ops)
nft_cmp_fast_eval(expr, ®s);
else if (expr->ops == &nft_cmp16_fast_ops)
nft_cmp16_fast_eval(expr, ®s);
else if (expr->ops == &nft_bitwise_fast_ops)
nft_bitwise_fast_eval(expr, ®s);
else if (expr->ops != &nft_payload_fast_ops ||
!nft_payload_fast_eval(expr, ®s, pkt))
expr_call_ops_eval(expr, ®s, pkt);
if (regs.verdict.code != NFT_CONTINUE)
break;
}
switch (regs.verdict.code) {
case NFT_BREAK:
regs.verdict.code = NFT_CONTINUE;
nft_trace_copy_nftrace(pkt, &info);
continue;
case NFT_CONTINUE:
nft_trace_packet(pkt, &info, chain, rule,
NFT_TRACETYPE_RULE);
continue;
}
break;
}
nft_trace_verdict(&info, chain, rule, ®s);
switch (regs.verdict.code & NF_VERDICT_MASK) {
case NF_ACCEPT:
case NF_DROP:
case NF_QUEUE:
case NF_STOLEN:
return regs.verdict.code;
}
switch (regs.verdict.code) {
case NFT_JUMP:
if (WARN_ON_ONCE(stackptr >= NFT_JUMP_STACK_SIZE))
return NF_DROP;
jumpstack[stackptr].chain = chain;
jumpstack[stackptr].rule = nft_rule_next(rule);
jumpstack[stackptr].last_rule = last_rule;
stackptr++;
fallthrough;
case NFT_GOTO:
chain = regs.verdict.chain;
goto do_chain;
case NFT_CONTINUE:
case NFT_RETURN:
break;
default:
WARN_ON_ONCE(1);
}
if (stackptr > 0) {
stackptr--;
chain = jumpstack[stackptr].chain;
rule = jumpstack[stackptr].rule;
last_rule = jumpstack[stackptr].last_rule;
goto next_rule;
}
nft_trace_packet(pkt, &info, basechain, NULL, NFT_TRACETYPE_POLICY);
if (static_branch_unlikely(&nft_counters_enabled))
nft_update_chain_stats(basechain, pkt);
return nft_base_chain(basechain)->policy;
}
The main rule execution bit here is in the two nested for-loops right after the
next_rule
label. Netfilter is internally based on a virtual machine for rule
execution, and each rule is translated into expressions, which correspond
roughly to the words in a rule definition above. The details of this are not
really important for the problem at hand, so I’ll skip them. It’s enough to
simply note that the outer loop goes through all the rules in the chain, and the
inner one executes each expression in the chain, breaking out of the loop if the
expression reaches a verdict.
Looking a bit further down in the code, the call to nft_trace_verdict()
springs out as really the obvious place to hook in. Indeed, this trace function
is part of netfilter’s own internal trace infrastructure; the function is a
small inline function looking like this:
static inline void nft_trace_verdict(struct nft_traceinfo *info,
const struct nft_chain *chain,
const struct nft_rule_dp *rule,
const struct nft_regs *regs)
{
if (static_branch_unlikely(&nft_trace_enabled)) {
info->rule = rule;
__nft_trace_verdict(info, chain, regs);
}
}
The static_branch_unlikely()
means that the real function implementing the
facility will be skipped in the common case: it’s compiled as a noop instruction
that will always be skipped over, and only become active when the netfilter
tracing infrastructure is enabled. The use of static branches is a performance
optimisation technique used in various places in the kernel to avoid the runtime
overhead of a feature that is only enabled in rare cases (and enabling it
involves the kernel dynamically rewriting its own code, which is cool, but we’re
also going to skip over the details of this here).
The netfilter tracing functionality actually allows us to see exactly what we want, but it’s not turned on by default, has some overhead associated with it (there’s also a setup function further up in the function), and we can’t really execute our own code there to look at the packet data.
Besides, we’re exploring BPF functionality here, so let’s see if we can replicate the functionality with BPF instead!
Attaching kprobes mid-function
The BPF tracing infrastructure makes it possible to attach probes to any
function in the running kernel (with some exceptions). To see all the kprobes
in the running kernel, use bpftrace -l
- but be warned, it’s a long list!
By default a kprobe attaches at the start of a function and can access the
function arguments, and there’s a kretprobe
that can be attached at the point
where a function returns, accessing its return value. However, it is also
possible to supply an offset to a kprobe, which is an instruction offset
inside the function itself, meaning we can attach to an arbitrary instruction in
the machine code of the compiled function.1 Which is exactly what we need - we
just need to figure out which instruction to attach to!
To do this, we’ll need to look at the disassembled code of the compiled
function. There are various ways of doing this, one is to use gdb
, but I found
that I preferred objdump
because it can also visualise the jumps around the
function using nice coloured ASCII art arrows. There are quite a few jumps in
this function because the control flow is so complex (just look at all those
nested loops and goto statements!), so this was quite useful.
Here is the output of the disassembled version of the function as it exists in
my 6.2.10-arch1-1
distribution kernel, in all its glory (scroll horizontally
to see all of it):
$ objdump --disassemble=nft_do_chain -rw -Mintel --visualize-jumps=extended-color --disassembler-color=extended nf_tables.ko nf_tables.ko: file format elf64-x86-64 Disassembly of section .text: 0000000000000110 <nft_do_chain>: 110: f3 0f 1e fa endbr64 11: R_X86_64_PLT32 __fentry__-0x4 21: R_X86_64_PLT32 __x86_return_thunk-0x4 2d: R_X86_64_PLT32 nft_trace_notify-0x4 51: R_X86_64_PLT32 __fentry__-0x4 65: R_X86_64_PC32 pcpu_hot 71: R_X86_64_PC32 this_cpu_off-0x4 80: R_X86_64_32S .text+0x62 90: R_X86_64_PLT32 __local_bh_enable_ip-0x4 95: R_X86_64_PLT32 __x86_return_thunk-0x4 b1: R_X86_64_PLT32 __fentry__-0x4 114: /-- e8 00 00 00 00 call 119 <nft_do_chain+0x9> 115: R_X86_64_PLT32 __fentry__-0x4 119: \-> 41 57 push r15 11b: b9 0a 00 00 00 mov ecx,0xa 120: 41 56 push r14 122: 49 89 fe mov r14,rdi 125: 41 55 push r13 127: 41 54 push r12 129: 55 push rbp 12a: 48 89 f5 mov rbp,rsi 12d: 53 push rbx 12e: 48 81 ec 20 02 00 00 sub rsp,0x220 135: 65 48 8b 04 25 28 00 00 00 mov rax,QWORD PTR gs:0x28 13e: 48 89 84 24 18 02 00 00 mov QWORD PTR [rsp+0x218],rax 146: 48 8b 47 08 mov rax,QWORD PTR [rdi+0x8] 14a: 48 8d 5c 24 48 lea rbx,[rsp+0x48] 14f: 48 8d 94 24 98 00 00 00 lea rdx,[rsp+0x98] 157: 48 89 df mov rdi,rbx 15a: 48 8b 70 20 mov rsi,QWORD PTR [rax+0x20] 15e: 31 c0 xor eax,eax 160: f3 48 ab rep stos QWORD PTR es:[rdi],rax 163: b9 30 00 00 00 mov ecx,0x30 168: 48 89 d7 mov rdi,rdx 16b: f3 48 ab rep stos QWORD PTR es:[rdi],rax 16e: b9 06 00 00 00 mov ecx,0x6 173: 48 8d 7c 24 18 lea rdi,[rsp+0x18] 178: 44 0f b6 be 28 0c 00 00 movzx r15d,BYTE PTR [rsi+0xc28] 180: f3 48 ab rep stos QWORD PTR es:[rdi],rax 183: 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 188: /----------------------------------------------------------------------------------------------------------------> 44 88 7c 24 0f mov BYTE PTR [rsp+0xf],r15b 18d: | 45 31 e4 xor r12d,r12d 190: | 80 7c 24 0f 00 cmp BYTE PTR [rsp+0xf],0x0 195: | 48 89 2c 24 mov QWORD PTR [rsp],rbp 199: | 48 8b 04 24 mov rax,QWORD PTR [rsp] 19d: | 48 89 6c 24 10 mov QWORD PTR [rsp+0x10],rbp 1a2: | /----------------------------------- 0f 84 c5 01 00 00 je 36d <nft_do_chain+0x25d> 1a8: | | /----------------------------> 48 8b 40 08 mov rax,QWORD PTR [rax+0x8] 1ac: | | /--|----------------------------> 48 8b 28 mov rbp,QWORD PTR [rax] 1af: | | | | 4c 8d 68 08 lea r13,[rax+0x8] 1b3: | | | | c7 44 24 48 ff ff ff ff mov DWORD PTR [rsp+0x48],0xffffffff 1bb: | | | | 4c 01 ed add rbp,r13 1be: | | | | 49 39 ed cmp r13,rbp 1c1: | /-----------------------------------|--|--|----------------------------- 0f 83 d3 02 00 00 jae 49a <nft_do_chain+0x38a> 1c7: | | /--------------------------|--|--|----------------------------> 44 89 64 24 08 mov DWORD PTR [rsp+0x8],r12d 1cc: | | | | | | 49 89 ec mov r12,rbp 1cf: | | | | | | /-------> 41 0f b7 45 00 movzx eax,WORD PTR [r13+0x0] 1d4: | | | | | | | 49 8d 6d 08 lea rbp,[r13+0x8] 1d8: | | | | | | | 66 d1 e8 shr ax,1 1db: | | | | | | | 25 ff 0f 00 00 and eax,0xfff 1e0: | | | | | | | 4d 8d 7c 05 08 lea r15,[r13+rax*1+0x8] 1e5: | | | | | | | 4c 39 fd cmp rbp,r15 1e8: | | | | | | | /----- 0f 85 de 00 00 00 jne 2cc <nft_do_chain+0x1bc> 1ee: | /--------------------------------------|--------|--------------------------|--|--|--------------------|--|----- e9 3f 04 00 00 jmp 632 <nft_do_chain+0x522> 1f3: | | | | | | | | | /-> 48 3d 00 00 00 00 cmp rax,0x0 1f5: R_X86_64_32S nft_cmp16_fast_ops 1f9: | | | | | | | /--------------|--|--|-- 0f 84 91 01 00 00 je 390 <nft_do_chain+0x280> 1ff: | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 201: R_X86_64_32S nft_bitwise_fast_ops 205: | | | | | | | | /-----------|--|--|-- 0f 84 6a 01 00 00 je 375 <nft_do_chain+0x265> 20b: | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 20d: R_X86_64_32S nft_payload_fast_ops 211: | | | | | | | /--|--|-----------|--|--|-- 0f 84 b2 01 00 00 je 3c9 <nft_do_chain+0x2b9> 217: | | | | /--|--|--|--|--|--|-----------|--|--|-> 48 8b 00 mov rax,QWORD PTR [rax] 21a: | | | | | | | | | | | | | | 4c 89 f2 mov rdx,r14 21d: | | | | | | | | | | | | | | 48 89 de mov rsi,rbx 220: | | | | | | | | | | | | | | 48 89 ef mov rdi,rbp 223: | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 225: R_X86_64_32S nft_payload_eval 229: | | | | /-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 8b 02 00 00 je 4ba <nft_do_chain+0x3aa> 22f: | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 231: R_X86_64_32S nft_cmp_eval 235: | | | | /-----------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 c1 02 00 00 je 4fc <nft_do_chain+0x3ec> 23b: | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 23d: R_X86_64_32S nft_counter_eval 241: | | | | | /--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 bf 02 00 00 je 506 <nft_do_chain+0x3f6> 247: | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 249: R_X86_64_32S nft_meta_get_eval 24d: | | | /--|-----|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 c8 02 00 00 je 51b <nft_do_chain+0x40b> 253: | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 255: R_X86_64_32S nft_lookup_eval 259: | | | | | /--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 c6 02 00 00 je 525 <nft_do_chain+0x415> 25f: | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 261: R_X86_64_32S nft_range_eval 265: | | /-----|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 13 03 00 00 je 57e <nft_do_chain+0x46e> 26b: | | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 26d: R_X86_64_32S nft_immediate_eval 271: | | | /--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 11 03 00 00 je 588 <nft_do_chain+0x478> 277: | | | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 279: R_X86_64_32S nft_byteorder_eval 27d: | | /-----------------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 3e 03 00 00 je 5c1 <nft_do_chain+0x4b1> 283: | | | | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 285: R_X86_64_32S nft_dynset_eval 289: | | | /--------------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 3c 03 00 00 je 5cb <nft_do_chain+0x4bb> 28f: | | | | | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 291: R_X86_64_32S nft_rt_get_eval 295: | | | | /-----------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 3a 03 00 00 je 5d5 <nft_do_chain+0x4c5> 29b: | | | | | | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 29d: R_X86_64_32S nft_bitwise_eval 2a1: | | | | | /--------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 38 03 00 00 je 5df <nft_do_chain+0x4cf> 2a7: | | | | | | /-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- e8 00 00 00 00 call 2ac <nft_do_chain+0x19c> 2a8: R_X86_64_PLT32 __x86_indirect_thunk_rax-0x4 2ac: | | | | | | >-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-> 8b 54 24 48 mov edx,DWORD PTR [rsp+0x48] 2b0: | | | | | | | | | | | | | | | | | | | | | | | | | | 83 fa ff cmp edx,0xffffffff 2b3: | | /-----------|--|--|--|--|-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 85 19 04 00 00 jne 6d2 <nft_do_chain+0x5c2> 2b9: | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b 45 00 mov rax,QWORD PTR [rbp+0x0] 2bd: | | | | | | | | | | | | | | | | | | | | | | | | | | | 8b 40 10 mov eax,DWORD PTR [rax+0x10] 2c0: | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 c5 add rbp,rax 2c3: | | | | | | | | | | | | | | | | | | | | | | | | | | | 49 39 ef cmp r15,rbp 2c6: | | | /-----|--|--|--|--|-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 f8 01 00 00 je 4c4 <nft_do_chain+0x3b4> 2cc: | | | | | | | | | | | | | | | | | | | | | | | | | | \--|-> 48 8b 45 00 mov rax,QWORD PTR [rbp+0x0] 2d0: | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 3d 00 00 00 00 cmp rax,0x0 2d2: R_X86_64_32S nft_cmp_fast_ops 2d6: | | | | | | | | | | | | | | | | | | | | | | | | | | \-- 0f 85 17 ff ff ff jne 1f3 <nft_do_chain+0xe3> 2dc: | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 45 10 movzx eax,BYTE PTR [rbp+0x10] 2e0: | | | | | | | | | | | | | | | | | | | | | | | | | | 8b 44 84 48 mov eax,DWORD PTR [rsp+rax*4+0x48] 2e4: | | | | | | | | | | | | | | | | | | | | | | | | | | 23 45 0c and eax,DWORD PTR [rbp+0xc] 2e7: | | | | | | | | | | | | | | | | | | | | | | | | | | 3b 45 08 cmp eax,DWORD PTR [rbp+0x8] 2ea: | | | | | | | | | | | | | | | | | | | | | | | | | | 0f 94 c0 sete al 2ed: | | | | | | | | | | | | | | | | | | | | | | | | | | 3a 45 12 cmp al,BYTE PTR [rbp+0x12] 2f0: | | | | | | | | +-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|-------- 75 ba jne 2ac <nft_do_chain+0x19c> 2f2: | | | /--|-----|--|--|--|--|-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|-------> c7 44 24 48 ff ff ff ff mov DWORD PTR [rsp+0x48],0xffffffff 2fa: | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 2ff: | | | | | | | | | | | | | | | | | | /-----|-----|--|--|--|--|--|--|-----------|-------> 41 0f b7 45 00 movzx eax,WORD PTR [r13+0x0] 304: | | | | | | | | | | | | | | | | | | | | | | | | | | | | 66 d1 e8 shr ax,1 307: | | | | | | | | | | | | | | | | | | | | | | | | | | | | 25 ff 0f 00 00 and eax,0xfff 30c: | | | | | | | | | | | | | | | | | | | | | | | | | | | | 4d 8d 6c 05 08 lea r13,[r13+rax*1+0x8] 311: | | | | | | | | | | | | | | | | | | | | | | | | | | | | 4d 39 e5 cmp r13,r12 314: | | | | | | | | | | | | | | | | | | | | | | | | | | | \-------- 0f 82 b5 fe ff ff jb 1cf <nft_do_chain+0xbf> 31a: | | | | | | | | | | | | | | | | | | | | | | | | | | | 4c 89 e5 mov rbp,r12 31d: | | | | | | | | | | | | | | | | | | | | | | | | | | | 8b 54 24 48 mov edx,DWORD PTR [rsp+0x48] 321: | | | | | | | | | | | | | | | | | | | | | | | | | | | 44 8b 64 24 08 mov r12d,DWORD PTR [rsp+0x8] 326: | | | | | | | | | | /--|--|--|-----|--|--|--|--|--|-----|-----|--|--|--|--|--|--|-------------------> 0f 1f 44 00 00 nop DWORD PTR [rax+rax*1+0x0] 32b: | | | | | | | | | | | | | | | | | | | | | | | | | | | | f6 c2 fc test dl,0xfc 32e: | | | | | /--|--|--|--|--|--|--|--|--|-----|--|--|--|--|--|-----|-----|--|--|--|--|--|--|-------------------- 0f 84 1b 02 00 00 je 54f <nft_do_chain+0x43f> 334: | | | | | | | | | | | | | | | | | | | | | | /--|--|--|--|--|--|--|-------------------> 83 fa fd cmp edx,0xfffffffd 337: | | | | | | | | | | | | | | | | | | | | | /--|--|--|--|--|--|--|--|--|-------------------- 0f 84 ac 02 00 00 je 5e9 <nft_do_chain+0x4d9> 33d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | /----------------- 0f 87 0a 01 00 00 ja 44d <nft_do_chain+0x33d> 343: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 83 fa fb cmp edx,0xfffffffb 346: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | /-------------- 0f 84 08 01 00 00 je 454 <nft_do_chain+0x344> 34c: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 83 fa fc cmp edx,0xfffffffc 34f: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | /----------- 0f 85 fd 00 00 00 jne 452 <nft_do_chain+0x342> 355: | | | | | | | | | | | | | | | /--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|----------> 48 8b 44 24 50 mov rax,QWORD PTR [rsp+0x50] 35a: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 80 7c 24 0f 00 cmp BYTE PTR [rsp+0xf],0x0 35f: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 89 04 24 mov QWORD PTR [rsp],rax 363: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b 04 24 mov rax,QWORD PTR [rsp] 367: | | | | | | | | | | | | | | | | | | | | | | | | | | | | \--|--|--|--|--|--|----------- 0f 85 3b fe ff ff jne 1a8 <nft_do_chain+0x98> 36d: | | | | | | | | | | | | | | | | | | | | | | | | | | \--|-----|--|--|--|--|--|----------> 48 8b 00 mov rax,QWORD PTR [rax] 370: | | | | | | | | | | | | | | | | | | | | | | | | | | \-----|--|--|--|--|--|----------- e9 37 fe ff ff jmp 1ac <nft_do_chain+0x9c> 375: | | | | | | | | | | | | | | | | | | | | | | | | | | | | \--|--|--|----------> 0f b6 45 10 movzx eax,BYTE PTR [rbp+0x10] 379: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 55 11 movzx edx,BYTE PTR [rbp+0x11] 37d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 8b 44 84 48 mov eax,DWORD PTR [rsp+rax*4+0x48] 381: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 23 45 08 and eax,DWORD PTR [rbp+0x8] 384: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 33 45 0c xor eax,DWORD PTR [rbp+0xc] 387: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 89 44 94 48 mov DWORD PTR [rsp+rdx*4+0x48],eax 38b: | | | | | | | | | | +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|-----------|--|-----|--|--|----------- e9 1c ff ff ff jmp 2ac <nft_do_chain+0x19c> 390: | | | | | | | | | | | | | | | | | | | | | | | | | | | \-----|--|--|----------> 0f b6 45 28 movzx eax,BYTE PTR [rbp+0x28] 394: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 31 d2 xor edx,edx 396: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8d 0c 83 lea rcx,[rbx+rax*4] 39a: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b 01 mov rax,QWORD PTR [rcx] 39d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 23 45 18 and rax,QWORD PTR [rbp+0x18] 3a1: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 3b 45 08 cmp rax,QWORD PTR [rbp+0x8] 3a5: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | /-- 75 11 jne 3b8 <nft_do_chain+0x2a8> 3a7: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 31 d2 xor edx,edx 3a9: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b 41 08 mov rax,QWORD PTR [rcx+0x8] 3ad: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 23 45 20 and rax,QWORD PTR [rbp+0x20] 3b1: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 3b 45 10 cmp rax,QWORD PTR [rbp+0x10] 3b5: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f 94 c2 sete dl 3b8: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | \-> 0f b6 45 2a movzx eax,BYTE PTR [rbp+0x2a] 3bc: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 39 d0 cmp eax,edx 3be: | | | | | | | | | | +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|-----------|--------|--|--|----------- 0f 85 e8 fe ff ff jne 2ac <nft_do_chain+0x19c> 3c4: | | | +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|-----------|--------|--|--|----------- e9 29 ff ff ff jmp 2f2 <nft_do_chain+0x1e2> 3c9: | | | | | | | | | | | | | | | | | | | | | | | | | | \--------|--|--|----------> 80 7d 08 01 cmp BYTE PTR [rbp+0x8],0x1 3cd: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 49 8b 36 mov rsi,QWORD PTR [r14] 3d0: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | /----- 0f 84 ce 00 00 00 je 4a4 <nft_do_chain+0x394> 3d6: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 41 f6 46 10 01 test BYTE PTR [r14+0x10],0x1 3db: | | | | | | | | | | | | | | | | | | | | | | | | | +--------------------|--|--|-----|----- 0f 84 36 fe ff ff je 217 <nft_do_chain+0x107> 3e1: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b7 96 b8 00 00 00 movzx edx,WORD PTR [rsi+0xb8] 3e8: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 41 0f b7 7e 14 movzx edi,WORD PTR [r14+0x14] 3ed: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b 8e c8 00 00 00 mov rcx,QWORD PTR [rsi+0xc8] 3f4: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 fa add rdx,rdi 3f7: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 ca add rdx,rcx 3fa: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | /-> 0f b6 7d 09 movzx edi,BYTE PTR [rbp+0x9] 3fe: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 8b b6 bc 00 00 00 mov esi,DWORD PTR [rsi+0xbc] 404: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 fa add rdx,rdi 407: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 7d 0a movzx edi,BYTE PTR [rbp+0xa] 40b: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 f1 add rcx,rsi 40e: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 d7 add rdi,rdx 411: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 39 f9 cmp rcx,rdi 414: | | | | | | | | | | | | | | | | | | | | | | | | | \--------------------|--|--|-----|--|-- 0f 82 fd fd ff ff jb 217 <nft_do_chain+0x107> 41a: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 4d 0b movzx ecx,BYTE PTR [rbp+0xb] 41e: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | c7 44 8c 48 00 00 00 00 mov DWORD PTR [rsp+rcx*4+0x48],0x0 426: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8d 34 8b lea rsi,[rbx+rcx*4] 42a: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 89 c8 mov rax,rcx 42d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 4d 0a movzx ecx,BYTE PTR [rbp+0xa] 431: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 80 f9 02 cmp cl,0x2 434: | | | | | | | | | | | | | | | | | | | | | | | | | | | | /--|--|-- 0f 84 d6 00 00 00 je 510 <nft_do_chain+0x400> 43a: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 80 f9 04 cmp cl,0x4 43d: | | | | | | | | | | | | | | | | | | | | | | | | | /--|--|--|--|--|--|-- 0f 84 4f 01 00 00 je 592 <nft_do_chain+0x482> 443: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 02 movzx eax,BYTE PTR [rdx] 446: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 88 06 mov BYTE PTR [rsi],al 448: | | | | | | | | | | +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--------------------|--|--|--|--|--|--|-- e9 5f fe ff ff jmp 2ac <nft_do_chain+0x19c> 44d: | | | | | | | | | | | | | | | | | | | | | | | | | | \--|--|--|--|--|-> 83 fa ff cmp edx,0xffffffff 450: | | | | | | | | | | | | | | | | | | | | | | | | | | +--|--|--|--|-- 74 02 je 454 <nft_do_chain+0x344> 452: | | | | | | | | | | | | | | | | | | | | | | | | | | | \--|--|--|-> 0f 0b ud2 454: | | | | | | | | | | | | | | | | | | | | | | | | | | \-----|--|--|-> 45 85 e4 test r12d,r12d 457: | | | | | | | | | | | | | | | | | | | | | | | | | | /-----|--|--|-- 0f 84 19 02 00 00 je 676 <nft_do_chain+0x566> 45d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 41 8d 44 24 ff lea eax,[r12-0x1] 462: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | c7 44 24 48 ff ff ff ff mov DWORD PTR [rsp+0x48],0xffffffff 46a: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 49 89 c4 mov r12,rax 46d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8d 04 40 lea rax,[rax+rax*2] 471: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 c1 e0 03 shl rax,0x3 475: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b bc 04 98 00 00 00 mov rdi,QWORD PTR [rsp+rax*1+0x98] 47d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 4c 8b ac 04 a0 00 00 00 mov r13,QWORD PTR [rsp+rax*1+0xa0] 485: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b ac 04 a8 00 00 00 mov rbp,QWORD PTR [rsp+rax*1+0xa8] 48d: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 89 3c 24 mov QWORD PTR [rsp],rdi 491: | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | 49 39 ed cmp r13,rbp 494: | | | | | | | | | | | | | | | | | \--|--|--|--|--|--|--|--------------------|-----|-----|--|--|-- 0f 82 2d fd ff ff jb 1c7 <nft_do_chain+0xb7> 49a: | | | | | | | | | | | | | | \--|--|-----|--|--|--|--|--|--|--------------------|-----|-----|--|--|-> ba ff ff ff ff mov edx,0xffffffff 49f: | | | | | | | | | | | +--|--|-----|--|-----|--|--|--|--|--|--|--------------------|-----|-----|--|--|-- e9 82 fe ff ff jmp 326 <nft_do_chain+0x216> 4a4: | | | | | | | | | | | | | | | | | | | | | | | | | | \--|-> 48 8b 8e c8 00 00 00 mov rcx,QWORD PTR [rsi+0xc8] 4ab: | | | | | | | | | | | | | | | | | | | | | | | | | | | 0f b7 96 b8 00 00 00 movzx edx,WORD PTR [rsi+0xb8] 4b2: | | | | | | | | | | | | | | | | | | | | | | | | | | | 48 01 ca add rdx,rcx 4b5: | | | | | | | | | | | | | | | | | | | | | | | | | | \-- e9 40 ff ff ff jmp 3fa <nft_do_chain+0x2ea> 4ba: | | | | | | | | | | | | | | | | | | | | | \--|--------------------|-----|-----|-----/-X e8 00 00 00 00 call 4bf <nft_do_chain+0x3af> 4bb: R_X86_64_PLT32 nft_payload_eval-0x4 4bf: | | | | | | | | | | +--|--|--|-----|--|-----|--|--|--|--|-----|--------------------|-----|-----|-----\-X e9 e8 fd ff ff jmp 2ac <nft_do_chain+0x19c> 4c4: | | | | >--|--|--|--|--|--|--|--|--|-----|--|-----|--|--|--|--|-----|--------------------|-----|-----|-------> 66 90 xchg ax,ax 4c6: | | | | | | | | | | | | | | | | | | | +--|-----|--------------------|-----|-----|-------- e9 34 fe ff ff jmp 2ff <nft_do_chain+0x1ef> 4cb: | | | | | | | | | | | | | | | | | | | | | | | | | 49 8b 06 mov rax,QWORD PTR [r14] 4ce: | | | | | | | | | | | | | | | | | | | | | | | | | 48 8b 34 24 mov rsi,QWORD PTR [rsp] 4d2: | | | | | | | | | | | | | | | | | | | | | | | | | ba 03 00 00 00 mov edx,0x3 4d7: | | | | | | | | | | | | | | | | | | | | | | | | | 48 8d 7c 24 18 lea rdi,[rsp+0x18] 4dc: | | | | | | | | | | | | | | | | | | | | | | | | | 4c 89 6c 24 38 mov QWORD PTR [rsp+0x38],r13 4e1: | | | | | | | | | | | | | | | | | | | | | | | | | 0f b6 80 80 00 00 00 movzx eax,BYTE PTR [rax+0x80] 4e8: | | | | | | | | | | | | | | | | | | | | | | | | | c0 e8 04 shr al,0x4 4eb: | | | | | | | | | | | | | | | | | | | | | | | | | 83 e0 01 and eax,0x1 4ee: | | | | | | | | | | | | | | | | | | | | | | | | | 88 44 24 19 mov BYTE PTR [rsp+0x19],al 4f2: | | | | | | | | | | | | | | | | | | | | | | | | | e8 19 fb ff ff call 10 <__nft_trace_packet> 4f7: | | | | | | | | | | | | | | | | | | | +--|-----|--------------------|-----|-----|-------- e9 03 fe ff ff jmp 2ff <nft_do_chain+0x1ef> 4fc: | | | | | | | | | | | | | | | | | \--|--|--|-----|--------------------|-----|-----|-----/-X e8 00 00 00 00 call 501 <nft_do_chain+0x3f1> 4fd: R_X86_64_PLT32 nft_cmp_eval-0x4 501: | | | | | | | | | | +--|--|--|-----|--|-----|-----|--|--|-----|--------------------|-----|-----|-----\-X e9 a6 fd ff ff jmp 2ac <nft_do_chain+0x19c> 506: | | | | | | | | | | | | | | | | | \--|--|-----|--------------------|-----|-----|-----/-X e8 00 00 00 00 call 50b <nft_do_chain+0x3fb> 507: R_X86_64_PLT32 nft_counter_eval-0x4 50b: | | | | | | | | | | +--|--|--|-----|--|-----|--------|--|-----|--------------------|-----|-----|-----\-X e9 9c fd ff ff jmp 2ac <nft_do_chain+0x19c> 510: | | | | | | | | | | | | | | | | | | | | | | \-------> 0f b7 02 movzx eax,WORD PTR [rdx] 513: | | | | | | | | | | | | | | | | | | | | | | 66 89 06 mov WORD PTR [rsi],ax 516: | | | | | | | | | | +--|--|--|-----|--|-----|--------|--|-----|--------------------|-----|-------------- e9 91 fd ff ff jmp 2ac <nft_do_chain+0x19c> 51b: | | | | | | | | | | | | | | | \-----|--------|--|-----|--------------------|-----|-----------/-X e8 00 00 00 00 call 520 <nft_do_chain+0x410> 51c: R_X86_64_PLT32 nft_meta_get_eval-0x4 520: | | | | | | | | | | +--|--|--|-----|--------|--------|--|-----|--------------------|-----|-----------\-X e9 87 fd ff ff jmp 2ac <nft_do_chain+0x19c> 525: | | | | | | | | | | | | | | | \--------|--|-----|--------------------|-----|-----------/-X e8 00 00 00 00 call 52a <nft_do_chain+0x41a> 526: R_X86_64_PLT32 nft_lookup_eval-0x4 52a: | | | | | | | | | | +--|--|--|-----|-----------------|--|-----|--------------------|-----|-----------\-X e9 7d fd ff ff jmp 2ac <nft_do_chain+0x19c> 52f: | | | | | | | | | | | | | | | | | | | | 48 8b 34 24 mov rsi,QWORD PTR [rsp] 533: | | | | | | | | | | | | | | | | | | | | 48 8d 7c 24 18 lea rdi,[rsp+0x18] 538: | | | | | | | | | | | | | | | | | | | | 4c 89 6c 24 38 mov QWORD PTR [rsp+0x38],r13 53d: | | | | | | | | | | | | | | | | | | | | e8 6e fb ff ff call b0 <__nft_trace_verdict.isra.0> 542: | | | | | | | | | | | | | | | | | | | | 8b 54 24 48 mov edx,DWORD PTR [rsp+0x48] 546: | | | | | | | | | | | | | | | | | | | | f6 c2 fc test dl,0xfc 549: | | | | | | | | | | | | | | | | | \--------------------|-----|-------------- 0f 85 e5 fd ff ff jne 334 <nft_do_chain+0x224> 54f: | | | | | >--|--|--|--|--|--|--|--|-----|-----------------|--|--------------------------|-----|-------------> 48 8b 84 24 18 02 00 00 mov rax,QWORD PTR [rsp+0x218] 557: | | | | | | | | | | | | | | | | | | | 65 48 2b 04 25 28 00 00 00 sub rax,QWORD PTR gs:0x28 560: | | | | | | | | | | | | | | | | | | | /----------- 0f 85 67 01 00 00 jne 6cd <nft_do_chain+0x5bd> 566: | | | | | | | | | | | | | | | | | | | | 48 81 c4 20 02 00 00 add rsp,0x220 56d: | | | | | | | | | | | | | | | | | | | | 89 d0 mov eax,edx 56f: | | | | | | | | | | | | | | | | | | | | 5b pop rbx 570: | | | | | | | | | | | | | | | | | | | | 5d pop rbp 571: | | | | | | | | | | | | | | | | | | | | 41 5c pop r12 573: | | | | | | | | | | | | | | | | | | | | 41 5d pop r13 575: | | | | | | | | | | | | | | | | | | | | 41 5e pop r14 577: | | | | | | | | | | | | | | | | | | | | 41 5f pop r15 579: | | | | | | | | | | | | +--|-----|-----------------|--|--------------------------|-----|--|----------- e9 00 00 00 00 jmp 57e <nft_do_chain+0x46e> 57a: R_X86_64_PLT32 __x86_return_thunk-0x4 57e: | | | | | | | | | | | | \--|-----|-----------------|--|--------------------------|-----|--|--------/-X e8 00 00 00 00 call 583 <nft_do_chain+0x473> 57f: R_X86_64_PLT32 nft_range_eval-0x4 583: | | | | | | | | | | +--|-----|-----|-----------------|--|--------------------------|-----|--|--------\-X e9 24 fd ff ff jmp 2ac <nft_do_chain+0x19c> 588: | | | | | | | | | | | | \-----|-----------------|--|--------------------------|-----|--|--------/-X e8 00 00 00 00 call 58d <nft_do_chain+0x47d> 589: R_X86_64_PLT32 nft_immediate_eval-0x4 58d: | | | | | | | | | | +--|-----------|-----------------|--|--------------------------|-----|--|--------\-X e9 1a fd ff ff jmp 2ac <nft_do_chain+0x19c> 592: | | | | | | | | | | | | | | | \-----|--|----------> 8b 12 mov edx,DWORD PTR [rdx] 594: | | | | | | | | | | | | | | | | | 89 54 84 48 mov DWORD PTR [rsp+rax*4+0x48],edx 598: | | | | | | | | | | +--|-----------|-----------------|--|--------------------------------|--|----------- e9 0f fd ff ff jmp 2ac <nft_do_chain+0x19c> 59d: | | | | | | | | | | | | | | | | | 80 7c 24 18 00 cmp BYTE PTR [rsp+0x18],0x0 5a2: | | | | | | | | | | | | | +--|--------------------------------|--|----------- 0f 84 57 fd ff ff je 2ff <nft_do_chain+0x1ef> 5a8: | | | | | | | | | | | | | | | | | 49 8b 06 mov rax,QWORD PTR [r14] 5ab: | | | | | | | | | | | | | | | | | 0f b6 80 80 00 00 00 movzx eax,BYTE PTR [rax+0x80] 5b2: | | | | | | | | | | | | | | | | | c0 e8 04 shr al,0x4 5b5: | | | | | | | | | | | | | | | | | 83 e0 01 and eax,0x1 5b8: | | | | | | | | | | | | | | | | | 88 44 24 19 mov BYTE PTR [rsp+0x19],al 5bc: | | | | | | | | | | | | | \--|--------------------------------|--|----------- e9 3e fd ff ff jmp 2ff <nft_do_chain+0x1ef> 5c1: | | | | | | \--|--|--|--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00 call 5c6 <nft_do_chain+0x4b6> 5c2: R_X86_64_PLT32 nft_byteorder_eval-0x4 5c6: | | | | | | | | | +--|-----------|--------------------|--------------------------------|--|--------\-X e9 e1 fc ff ff jmp 2ac <nft_do_chain+0x19c> 5cb: | | | | | | \--|--|--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00 call 5d0 <nft_do_chain+0x4c0> 5cc: R_X86_64_PLT32 nft_dynset_eval-0x4 5d0: | | | | | | | | +--|-----------|--------------------|--------------------------------|--|--------\-X e9 d7 fc ff ff jmp 2ac <nft_do_chain+0x19c> 5d5: | | | | | | \--|--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00 call 5da <nft_do_chain+0x4ca> 5d6: R_X86_64_PLT32 nft_rt_get_eval-0x4 5da: | | | | | | | +--|-----------|--------------------|--------------------------------|--|--------\-X e9 cd fc ff ff jmp 2ac <nft_do_chain+0x19c> 5df: | | | | | | \--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00 call 5e4 <nft_do_chain+0x4d4> 5e0: R_X86_64_PLT32 nft_bitwise_eval-0x4 5e4: | | | | | | \--|-----------|--------------------|--------------------------------|--|--------\-X e9 c3 fc ff ff jmp 2ac <nft_do_chain+0x19c> 5e9: | | | | | | | | \--------------------------------|--|----------> 41 83 fc 0f cmp r12d,0xf 5ed: | | | | | | | | | | /----- 77 7e ja 66d <nft_do_chain+0x55d> 5ef: | | | | | | | | | | | 41 0f b7 55 00 movzx edx,WORD PTR [r13+0x0] 5f4: | | | | | | | | | | | 44 89 e0 mov eax,r12d 5f7: | | | | | | | | | | | 48 8b 3c 24 mov rdi,QWORD PTR [rsp] 5fb: | | | | | | | | | | | 41 83 c4 01 add r12d,0x1 5ff: | | | | | | | | | | | 48 8d 04 40 lea rax,[rax+rax*2] 603: | | | | | | | | | | | 66 d1 ea shr dx,1 606: | | | | | | | | | | | 48 c1 e0 03 shl rax,0x3 60a: | | | | | | | | | | | 81 e2 ff 0f 00 00 and edx,0xfff 610: | | | | | | | | | | | 48 89 bc 04 98 00 00 00 mov QWORD PTR [rsp+rax*1+0x98],rdi 618: | | | | | | | | | | | 49 8d 54 15 08 lea rdx,[r13+rdx*1+0x8] 61d: | | | | | | | | | | | 48 89 ac 04 a8 00 00 00 mov QWORD PTR [rsp+rax*1+0xa8],rbp 625: | | | | | | | | | | | 48 89 94 04 a0 00 00 00 mov QWORD PTR [rsp+rax*1+0xa0],rdx 62d: | | | | | | | \-----------------------------------------------------|--|-----|----- e9 23 fd ff ff jmp 355 <nft_do_chain+0x245> 632: | \--|--|--|--|-----------------|-----------------------------------------------------------------|--|-----|----> 8b 54 24 48 mov edx,DWORD PTR [rsp+0x48] 636: | | | | | | | | | 83 fa fe cmp edx,0xfffffffe 639: | | +--|--|-----------------|-----------------------------------------------------------------|--|-----|----- 0f 84 b3 fc ff ff je 2f2 <nft_do_chain+0x1e2> 63f: | | | | | | | | | 83 fa ff cmp edx,0xffffffff 642: | | | \--|-----------------|-----------------------------------------------------------------|--|-----|----- 0f 84 7c fe ff ff je 4c4 <nft_do_chain+0x3b4> 648: | | | | | | | /--|----> 4c 89 e5 mov rbp,r12 64b: | | | | | | | | | 44 8b 64 24 08 mov r12d,DWORD PTR [rsp+0x8] 650: | | | | \-----------------------------------------------------------------|--|--|--|----- e9 d1 fc ff ff jmp 326 <nft_do_chain+0x216> 655: | | | | | | | | 48 89 e9 mov rcx,rbp 658: | | | | | | | | 48 89 da mov rdx,rbx 65b: | | | | | | | | 48 8d 7c 24 18 lea rdi,[rsp+0x18] 660: | | | | | | | | 4c 89 f6 mov rsi,r14 663: | | | | | | | | /-- e8 00 00 00 00 call 668 <nft_do_chain+0x558> 664: R_X86_64_PLT32 nft_trace_init-0x4 668: \-----|--|-----|-----------------------------------------------------------------------------------|--|--|--|--\-X e9 1b fb ff ff jmp 188 <nft_do_chain+0x78> 66d: | | | | | | \----> 0f 0b ud2 66f: | | | | | | 31 d2 xor edx,edx 671: | | +-----------------------------------------------------------------------------------|--|--|-------- e9 d9 fe ff ff jmp 54f <nft_do_chain+0x43f> 676: | | | \--|--|-------> 48 8b 6c 24 10 mov rbp,QWORD PTR [rsp+0x10] 67b: | | | | | 66 90 xchg ax,ax 67d: | | | | | /-> 66 90 xchg ax,ax 67f: | | | | | | 0f b6 55 f0 movzx edx,BYTE PTR [rbp-0x10] 683: | | +--------------------------------------------------------------------------------------|--|-----|-- e9 c7 fe ff ff jmp 54f <nft_do_chain+0x43f> 688: | | | | | | 4c 89 f6 mov rsi,r14 68b: | | | | | | 48 89 ef mov rdi,rbp 68e: | | | | | | e8 bd f9 ff ff call 50 <nft_update_chain_stats> 693: | | | | | | 0f b6 55 f0 movzx edx,BYTE PTR [rbp-0x10] 697: | | \--------------------------------------------------------------------------------------|--|-----|-- e9 b3 fe ff ff jmp 54f <nft_do_chain+0x43f> 69c: | | | | | 49 8b 06 mov rax,QWORD PTR [r14] 69f: | | | | | ba 01 00 00 00 mov edx,0x1 6a4: | | | | | 48 89 ee mov rsi,rbp 6a7: | | | | | 48 8d 7c 24 18 lea rdi,[rsp+0x18] 6ac: | | | | | 48 c7 44 24 38 00 00 00 00 mov QWORD PTR [rsp+0x38],0x0 6b5: | | | | | 0f b6 80 80 00 00 00 movzx eax,BYTE PTR [rax+0x80] 6bc: | | | | | c0 e8 04 shr al,0x4 6bf: | | | | | 83 e0 01 and eax,0x1 6c2: | | | | | 88 44 24 19 mov BYTE PTR [rsp+0x19],al 6c6: | | | | | e8 45 f9 ff ff call 10 <__nft_trace_packet> 6cb: | | | | \-- eb b0 jmp 67d <nft_do_chain+0x56d> 6cd: +--|--------------------------------------------------------------------------------------------\--|-------X e8 00 00 00 00 call 6d2 <nft_do_chain+0x5c2> 6ce: R_X86_64_PLT32 __stack_chk_fail-0x4 6d2: \--|-----------------------------------------------------------------------------------------------|-------> 83 fa fe cmp edx,0xfffffffe 6d5: | \-------- 0f 85 6d ff ff ff jne 648 <nft_do_chain+0x538> 6db: \-------------------------------------------------------------------------------------------------------- e9 12 fc ff ff jmp 2f2 <nft_do_chain+0x1e2>
Now, I would love to be able to say that I could immediately follow the flow of that and zero in on the instruction of interest, but truth be told, I am not actually very good at reading assembly output. Especially not x86_64 assembly (so many instructions! and what’s with those weird names for the registers?). However, armed with a handy reference for the register names that some googling turned up, and another one for the instruction names, I set out to try and make sense of it all.
Fancy ASCII graphs notwithstanding, I quickly gave up on tracing the execution
flow from the beginning of the function. Instead, I looked for branches in the C
source above that were near the place I wanted to hook in. We already identified
that the call site for nft_trace_verdict()
would be a good place, but that is
obscured some by the static_call
infrastructure. However, right after it there
is this branch, which turned out to be a better candidate:
switch (regs.verdict.code & NF_VERDICT_MASK) {
case NF_ACCEPT:
case NF_DROP:
case NF_QUEUE:
case NF_STOLEN:
return regs.verdict.code;
}
To find this in the disassembly, first notice that the NF_VERDICT_MATCH
is
0xff
, and the values of those four verdict codes are 0, 1, 2 and 3. This means
that this statement compiles down to a check for regs.verdict.code & 0xfc
, and
if that is non-zero, the function returns. There’s a test
instruction for
such an AND test, so let’s look for this in the disassembled output above (just
search for 0xfc
).
There’s one such instruction at offset 0x32b, and another at 0x546, both of them
testing the same dl
register. By using bpftrace
to just print out a simple
message at each site, we can quickly determine if they seem to be executed when
the event we’re interested in, by a oneliner like this one2:
$ sudo bpftrace -e 'kprobe:nft_do_chain+0x21b { printf("Here!\n"); }'
This quickly printed out a bunch of messages when I generated traffic on the
system, so that seemed to be the right place. Now we just need to figure out how
we are going to access the values we need: the struct nft_rule_dp
pointer that
leads to the rule, and the pkt
pointer that allows up to look at the packet
data itself.
The latter one is actually fairly straight-forward: the pkt
pointer is passed
in as the first argument to the function when it starts, meaning it starts out
in the rdi
register. And one of the first instructions in the function is a
mov r14,rdi
, copying that value into r14
, where it stays for the rest of
the function execution (there are no other instructions putting a new value into
the r14
register until the function returns).
To obtain the other pointer, I noticed that the call to the netfilter tracing
function is in the output above; it appears as call b0
<__nft_trace_verdict.isra.0>
. And right before it, there’s an instruction
storing the value of the r13
register into a value on the stack which is what
the inline wrapper function (nft_trace_verdict()
) does with the rule
pointer. So the rule pointer is probably in r13
!
Putting it all together
With the above information, we can construct a small bpftrace
probe that can
read out the values we need. We have two pointers stored in registers r13
and
r14
, and bpftrace simply lets us cast those pointers to their struct values
and dereference them to get at the values we really want3. Due to the BPF tracing
magic, this is safe even if the pointers turn out to be invalid: We don’t crash
the kernel by a bad pointer deref, we’ll just get an invalid value back. So if
we get reasonable values, we know that we are poking at the right bit of memory.
To try it out, I constructed the following bpftrace script:
#!/usr/bin/bpftrace
kprobe:nft_do_chain+0x21b {
if ((reg("dx") & 0xff) == 0) {
printf("Packet with len %d dropped by rule with handle %d\n",
((struct nft_pktinfo *)reg("r14"))->skb->len,
*((uint64 *)reg("r13")) >> 13);
}
}
This first looks at the return code, which we determined above is in the dx
register4. NFT_DROP
has a value of 0, and we’re
only interested in drops, so we only print something if this was a drop
verdict - that’s the if statement in the bpftrace script above.
When we do get a drop verdict, we simply print out a message on the console with
the packet length and the handle of the rule that gave the verdict. I chose the
length because this is stored directly in struct sk_buff
as a len field, so
it’s easy to get at by just doing a bit of pointer walking. If we wanted to look
at the packet data we’d have to do a bit more work, but for this short
verification the length will suffice.
To get the rule handle, we have to be slightly more creative: the nft_rule_dp
struct
contains a u64 as its first member that is defined as a bitfield with three
members, the handle being one of those. However, BTF doesn’t support bitfields,
so we can’t just dereference the struct member. Instead we just cast the pointer
directly to a u64
and manually shift it by the required 13 bits to get the
handle.
Saving the script above as nfdrop.bt
and running it, we get output like this
(while producing test traffic from another machine that we know will get dropped):
$ sudo ./nfdrop.bt
Attaching 1 probe...
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
^C
Success! Handle 7 is the extra rule I added to the ruleset to test with, and
this output happens whenever I produce traffic with a destination port of 10000
like the rule specifies. Dumping the traffic with tcpdump
confirms that the
packet size is 60 bytes, so we’re getting sane values. This means that we’ve
successfully probed into the middle of the kernel function and can get at both
the rule and packet data at the place where the verdict is set! 🥳
Caveats
Now, the obvious problem with all this is that while the resulting script above is quite small, it is also entirely specific to the kernel binary that I am currently running on my own machine. Any change in the code, or even a change in how the compiler generates the code of that function, will invalidate it as that will change the function offsets and/or the register usage.
This means that it is not really a general solution to the problem, and if this were to be part of a monitoring solution that someone wants to deploy on multiple systems, the script would potentially have to be adjusted for each one. I don’t know of any automated way to generate this either, so this would probably have to be done manually. However, for this demonstration that doesn’t really matter, and I thought it was quite interesting to poke into all this and figure out exactly how kprobes in the middle of a function works!
Summary
The above shows how to use kprobes attached to the middle of a kernel function to get at data that is not otherwise available because there is no useful attach point to get at it. This is an incredibly powerful technique to obtain any kind of information from a running kernel - I did all this on my regular laptop without changing anything in the kernel code, or even rebooting it!
Getting the information out involved quite a lot of manual work to decipher the function code and find the attach point, but this can be worth it for certain use cases. And many functions are less complex than the one we were looking at here, in which case finding the right attach point can be easier.
I certainly found the exploration above interesting, and if you’re still reading this far down, I’m hoping you did as well! 😅
This works by rewriting the target instruction so that it becomes an instruction that raises a processor exception, and then executing the kprobe itself in the exception handler, before executing the original instruction and jumping back to the function itself so execution can continue. See the kprobes documentation for more details.
[return]Note that the offsets in the objdump output starts at 0x110, so we have to subtract that value, yielding the
[return]0x21b
value in thebpftrace
script.It used to be the case that you had to also include the header files with the struct definition you were interested in into your bpftrace script. However, newer versions of bpftrace will automatically look at the BTF information of the running kernel (if the kernel is built with BTF support) and determine the struct layout for us, so we can just use the struct name directly. Pretty neat.
[return]The
[return]reg()
function is a bpftrace internal function which returns the value of a named register; bpftrace uses slightly different register names than those in the objdump output, so this corresponds to the earlierdl
register we saw as an argument to thetest
instruction.