To kick things off in this second era of my blog (ha!), I thought I’d write up an interesting investigation I recently conducted based on an, on its surface, simple question from a colleague (paraphrased):

Can we use BPF to attribute a drop in a netfilter ruleset to the rule that decided to drop the packet?

This turned out to be not quite straight-forward, because of the way the netfilter code is structured (as we will see below), but it was interesting because it allowed me to experiment with attaching kprobes into the middle of the function. Something I knew was possible using BPF, but not something I had really experimented with in practice before.

The anatomy of a netfilter ruleset

To expand a bit on what the question really means, let’s have a look at the nft ruleset currently installed on my laptop (which is what I used to test this):

$ sudo nft -a list ruleset
table inet filter { # handle 6
        chain input { # handle 1
                type filter hook input priority filter; policy drop;
                ct state { established, related } accept # handle 5
                ct state invalid drop # handle 6
                tcp dport 10000 drop # handle 7
                iifname "lo" accept # handle 8
                iifname "veth0" accept # handle 9
                iifname "eth0" accept # handle 10
                ip protocol icmp accept # handle 11
                ip6 nexthdr ipv6-icmp icmpv6 type { destination-unreachable, packet-too-big, time-exceeded, parameter-problem, echo-request, echo-reply, nd-router-advert, nd-neighbor-solicit, nd-neighbor-advert } accept # handle 13
        }

        chain forward { # handle 2
                type filter hook forward priority filter; policy drop;
        }

        chain output { # handle 3
                type filter hook output priority filter; policy accept;
        }
}

Looking at the input chain, each line is a rule that will be executed in sequence, and if a rule matches the packet and contains a verdict (accept or drop), processing of further rules will stop, rending that verdict as the final one for the packet. What we want to do is be able to hook into the processing and whenever a drop verdict is reached, look at both the packet data (to know which packet it was, e.g., to identify the flow it belongs to), and also know which rule was the cause of the drop verdict so we can troubleshoot our ruleset.

As can be seen from the output above, each rule has a ‘handle’ that identifies it with its chain. So whenever a particular packet is dropped, what we want is to find the handle of the rule that dropped it. I added the (redundant) rule with handle 7 as a test rule just for experimenting.

Before we jump down the BPF rabbit hole for this, I should mention that netfilter itself does have some facilities for getting this information. For one thing, you can add a counter statement to the rule if you just want statistics on how often that rule is hit. Or you can use the netfilter logging facilities to get per-packet output in system logs or passed to a userspace socket. But those require modifications of the ruleset, and are either too low granularity (the counter), or fairly heavy weight (the logging), so let’s see if we can get the result we want with BPF.

Diving into the netfilter kernel code

To find a good place to place our hook, let’s first have a look at how netfilter executes its rules. The main kernel function that executes a netfilter chain is nft_do_chain() which lives in net/netfilter/nf_tables_core.c and looks like this:

unsigned int
nft_do_chain(struct nft_pktinfo *pkt, void *priv)
{
	const struct nft_chain *chain = priv, *basechain = chain;
	const struct nft_rule_dp *rule, *last_rule;
	const struct net *net = nft_net(pkt);
	const struct nft_expr *expr, *last;
	struct nft_regs regs = {};
	unsigned int stackptr = 0;
	struct nft_jumpstack jumpstack[NFT_JUMP_STACK_SIZE];
	bool genbit = READ_ONCE(net->nft.gencursor);
	struct nft_rule_blob *blob;
	struct nft_traceinfo info;

	info.trace = false;
	if (static_branch_unlikely(&nft_trace_enabled))
		nft_trace_init(&info, pkt, &regs.verdict, basechain);
do_chain:
	if (genbit)
		blob = rcu_dereference(chain->blob_gen_1);
	else
		blob = rcu_dereference(chain->blob_gen_0);

	rule = (struct nft_rule_dp *)blob->data;
	last_rule = (void *)blob->data + blob->size;
next_rule:
	regs.verdict.code = NFT_CONTINUE;
	for (; rule < last_rule; rule = nft_rule_next(rule)) {
		nft_rule_dp_for_each_expr(expr, last, rule) {
			if (expr->ops == &nft_cmp_fast_ops)
				nft_cmp_fast_eval(expr, &regs);
			else if (expr->ops == &nft_cmp16_fast_ops)
				nft_cmp16_fast_eval(expr, &regs);
			else if (expr->ops == &nft_bitwise_fast_ops)
				nft_bitwise_fast_eval(expr, &regs);
			else if (expr->ops != &nft_payload_fast_ops ||
				 !nft_payload_fast_eval(expr, &regs, pkt))
				expr_call_ops_eval(expr, &regs, pkt);

			if (regs.verdict.code != NFT_CONTINUE)
				break;
		}

		switch (regs.verdict.code) {
		case NFT_BREAK:
			regs.verdict.code = NFT_CONTINUE;
			nft_trace_copy_nftrace(pkt, &info);
			continue;
		case NFT_CONTINUE:
			nft_trace_packet(pkt, &info, chain, rule,
					 NFT_TRACETYPE_RULE);
			continue;
		}
		break;
	}

	nft_trace_verdict(&info, chain, rule, &regs);

	switch (regs.verdict.code & NF_VERDICT_MASK) {
	case NF_ACCEPT:
	case NF_DROP:
	case NF_QUEUE:
	case NF_STOLEN:
		return regs.verdict.code;
	}

	switch (regs.verdict.code) {
	case NFT_JUMP:
		if (WARN_ON_ONCE(stackptr >= NFT_JUMP_STACK_SIZE))
			return NF_DROP;
		jumpstack[stackptr].chain = chain;
		jumpstack[stackptr].rule = nft_rule_next(rule);
		jumpstack[stackptr].last_rule = last_rule;
		stackptr++;
		fallthrough;
	case NFT_GOTO:
		chain = regs.verdict.chain;
		goto do_chain;
	case NFT_CONTINUE:
	case NFT_RETURN:
		break;
	default:
		WARN_ON_ONCE(1);
	}

	if (stackptr > 0) {
		stackptr--;
		chain = jumpstack[stackptr].chain;
		rule = jumpstack[stackptr].rule;
		last_rule = jumpstack[stackptr].last_rule;
		goto next_rule;
	}

	nft_trace_packet(pkt, &info, basechain, NULL, NFT_TRACETYPE_POLICY);

	if (static_branch_unlikely(&nft_counters_enabled))
		nft_update_chain_stats(basechain, pkt);

	return nft_base_chain(basechain)->policy;
}

The main rule execution bit here is in the two nested for-loops right after the next_rule label. Netfilter is internally based on a virtual machine for rule execution, and each rule is translated into expressions, which correspond roughly to the words in a rule definition above. The details of this are not really important for the problem at hand, so I’ll skip them. It’s enough to simply note that the outer loop goes through all the rules in the chain, and the inner one executes each expression in the chain, breaking out of the loop if the expression reaches a verdict.

Looking a bit further down in the code, the call to nft_trace_verdict() springs out as really the obvious place to hook in. Indeed, this trace function is part of netfilter’s own internal trace infrastructure; the function is a small inline function looking like this:

static inline void nft_trace_verdict(struct nft_traceinfo *info,
				     const struct nft_chain *chain,
				     const struct nft_rule_dp *rule,
				     const struct nft_regs *regs)
{
	if (static_branch_unlikely(&nft_trace_enabled)) {
		info->rule = rule;
		__nft_trace_verdict(info, chain, regs);
	}
}

The static_branch_unlikely() means that the real function implementing the facility will be skipped in the common case: it’s compiled as a noop instruction that will always be skipped over, and only become active when the netfilter tracing infrastructure is enabled. The use of static branches is a performance optimisation technique used in various places in the kernel to avoid the runtime overhead of a feature that is only enabled in rare cases (and enabling it involves the kernel dynamically rewriting its own code, which is cool, but we’re also going to skip over the details of this here).

The netfilter tracing functionality actually allows us to see exactly what we want, but it’s not turned on by default, has some overhead associated with it (there’s also a setup function further up in the function), and we can’t really execute our own code there to look at the packet data.

Besides, we’re exploring BPF functionality here, so let’s see if we can replicate the functionality with BPF instead!

Attaching kprobes mid-function

The BPF tracing infrastructure makes it possible to attach probes to any function in the running kernel (with some exceptions). To see all the kprobes in the running kernel, use bpftrace -l - but be warned, it’s a long list!

By default a kprobe attaches at the start of a function and can access the function arguments, and there’s a kretprobe that can be attached at the point where a function returns, accessing its return value. However, it is also possible to supply an offset to a kprobe, which is an instruction offset inside the function itself, meaning we can attach to an arbitrary instruction in the machine code of the compiled function.1 Which is exactly what we need - we just need to figure out which instruction to attach to!

To do this, we’ll need to look at the disassembled code of the compiled function. There are various ways of doing this, one is to use gdb, but I found that I preferred objdump because it can also visualise the jumps around the function using nice coloured ASCII art arrows. There are quite a few jumps in this function because the control flow is so complex (just look at all those nested loops and goto statements!), so this was quite useful.

Here is the output of the disassembled version of the function as it exists in my 6.2.10-arch1-1 distribution kernel, in all its glory (scroll horizontally to see all of it):

$ objdump --disassemble=nft_do_chain -rw -Mintel --visualize-jumps=extended-color --disassembler-color=extended nf_tables.ko

nf_tables.ko:     file format elf64-x86-64


Disassembly of section .text:

0000000000000110 <nft_do_chain>:
     110:	                                                                                                                   f3 0f 1e fa          	endbr64	11: R_X86_64_PLT32	__fentry__-0x4
	21: R_X86_64_PLT32	__x86_return_thunk-0x4
	2d: R_X86_64_PLT32	nft_trace_notify-0x4
	51: R_X86_64_PLT32	__fentry__-0x4
	65: R_X86_64_PC32	pcpu_hot
	71: R_X86_64_PC32	this_cpu_off-0x4
	80: R_X86_64_32S	.text+0x62
	90: R_X86_64_PLT32	__local_bh_enable_ip-0x4
	95: R_X86_64_PLT32	__x86_return_thunk-0x4
	b1: R_X86_64_PLT32	__fentry__-0x4
     114:	                                                                                                               /-- e8 00 00 00 00       	call   119 <nft_do_chain+0x9>	115: R_X86_64_PLT32	__fentry__-0x4
     119:	                                                                                                               \-> 41 57                	push   r15
     11b:	                                                                                                                   b9 0a 00 00 00       	mov    ecx,0xa
     120:	                                                                                                                   41 56                	push   r14
     122:	                                                                                                                   49 89 fe             	mov    r14,rdi
     125:	                                                                                                                   41 55                	push   r13
     127:	                                                                                                                   41 54                	push   r12
     129:	                                                                                                                   55                   	push   rbp
     12a:	                                                                                                                   48 89 f5             	mov    rbp,rsi
     12d:	                                                                                                                   53                   	push   rbx
     12e:	                                                                                                                   48 81 ec 20 02 00 00 	sub    rsp,0x220
     135:	                                                                                                                   65 48 8b 04 25 28 00 00 00 	mov    rax,QWORD PTR gs:0x28
     13e:	                                                                                                                   48 89 84 24 18 02 00 00 	mov    QWORD PTR [rsp+0x218],rax
     146:	                                                                                                                   48 8b 47 08          	mov    rax,QWORD PTR [rdi+0x8]
     14a:	                                                                                                                   48 8d 5c 24 48       	lea    rbx,[rsp+0x48]
     14f:	                                                                                                                   48 8d 94 24 98 00 00 00 	lea    rdx,[rsp+0x98]
     157:	                                                                                                                   48 89 df             	mov    rdi,rbx
     15a:	                                                                                                                   48 8b 70 20          	mov    rsi,QWORD PTR [rax+0x20]
     15e:	                                                                                                                   31 c0                	xor    eax,eax
     160:	                                                                                                                   f3 48 ab             	rep stos QWORD PTR es:[rdi],rax
     163:	                                                                                                                   b9 30 00 00 00       	mov    ecx,0x30
     168:	                                                                                                                   48 89 d7             	mov    rdi,rdx
     16b:	                                                                                                                   f3 48 ab             	rep stos QWORD PTR es:[rdi],rax
     16e:	                                                                                                                   b9 06 00 00 00       	mov    ecx,0x6
     173:	                                                                                                                   48 8d 7c 24 18       	lea    rdi,[rsp+0x18]
     178:	                                                                                                                   44 0f b6 be 28 0c 00 00 	movzx  r15d,BYTE PTR [rsi+0xc28]
     180:	                                                                                                                   f3 48 ab             	rep stos QWORD PTR es:[rdi],rax
     183:	                                                                                                                   0f 1f 44 00 00       	nop    DWORD PTR [rax+rax*1+0x0]
     188:	/----------------------------------------------------------------------------------------------------------------> 44 88 7c 24 0f       	mov    BYTE PTR [rsp+0xf],r15b
     18d:	|                                                                                                                  45 31 e4             	xor    r12d,r12d
     190:	|                                                                                                                  80 7c 24 0f 00       	cmp    BYTE PTR [rsp+0xf],0x0
     195:	|                                                                                                                  48 89 2c 24          	mov    QWORD PTR [rsp],rbp
     199:	|                                                                                                                  48 8b 04 24          	mov    rax,QWORD PTR [rsp]
     19d:	|                                                                                                                  48 89 6c 24 10       	mov    QWORD PTR [rsp+0x10],rbp
     1a2:	|                                                                             /----------------------------------- 0f 84 c5 01 00 00    	je     36d <nft_do_chain+0x25d>
     1a8:	|                                                                             |     /----------------------------> 48 8b 40 08          	mov    rax,QWORD PTR [rax+0x8]
     1ac:	|                                                                             |  /--|----------------------------> 48 8b 28             	mov    rbp,QWORD PTR [rax]
     1af:	|                                                                             |  |  |                              4c 8d 68 08          	lea    r13,[rax+0x8]
     1b3:	|                                                                             |  |  |                              c7 44 24 48 ff ff ff ff 	mov    DWORD PTR [rsp+0x48],0xffffffff
     1bb:	|                                                                             |  |  |                              4c 01 ed             	add    rbp,r13
     1be:	|                                                                             |  |  |                              49 39 ed             	cmp    r13,rbp
     1c1:	|                                         /-----------------------------------|--|--|----------------------------- 0f 83 d3 02 00 00    	jae    49a <nft_do_chain+0x38a>
     1c7:	|                                         |        /--------------------------|--|--|----------------------------> 44 89 64 24 08       	mov    DWORD PTR [rsp+0x8],r12d
     1cc:	|                                         |        |                          |  |  |                              49 89 ec             	mov    r12,rbp
     1cf:	|                                         |        |                          |  |  |                    /-------> 41 0f b7 45 00       	movzx  eax,WORD PTR [r13+0x0]
     1d4:	|                                         |        |                          |  |  |                    |         49 8d 6d 08          	lea    rbp,[r13+0x8]
     1d8:	|                                         |        |                          |  |  |                    |         66 d1 e8             	shr    ax,1
     1db:	|                                         |        |                          |  |  |                    |         25 ff 0f 00 00       	and    eax,0xfff
     1e0:	|                                         |        |                          |  |  |                    |         4d 8d 7c 05 08       	lea    r15,[r13+rax*1+0x8]
     1e5:	|                                         |        |                          |  |  |                    |         4c 39 fd             	cmp    rbp,r15
     1e8:	|                                         |        |                          |  |  |                    |  /----- 0f 85 de 00 00 00    	jne    2cc <nft_do_chain+0x1bc>
     1ee:	|  /--------------------------------------|--------|--------------------------|--|--|--------------------|--|----- e9 3f 04 00 00       	jmp    632 <nft_do_chain+0x522>
     1f3:	|  |                                      |        |                          |  |  |                    |  |  /-> 48 3d 00 00 00 00    	cmp    rax,0x0	1f5: R_X86_64_32S	nft_cmp16_fast_ops
     1f9:	|  |                                      |        |                          |  |  |     /--------------|--|--|-- 0f 84 91 01 00 00    	je     390 <nft_do_chain+0x280>
     1ff:	|  |                                      |        |                          |  |  |     |              |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	201: R_X86_64_32S	nft_bitwise_fast_ops
     205:	|  |                                      |        |                          |  |  |     |  /-----------|--|--|-- 0f 84 6a 01 00 00    	je     375 <nft_do_chain+0x265>
     20b:	|  |                                      |        |                          |  |  |     |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	20d: R_X86_64_32S	nft_payload_fast_ops
     211:	|  |                                      |        |                          |  |  |  /--|--|-----------|--|--|-- 0f 84 b2 01 00 00    	je     3c9 <nft_do_chain+0x2b9>
     217:	|  |                                      |        |                       /--|--|--|--|--|--|-----------|--|--|-> 48 8b 00             	mov    rax,QWORD PTR [rax]
     21a:	|  |                                      |        |                       |  |  |  |  |  |  |           |  |  |   4c 89 f2             	mov    rdx,r14
     21d:	|  |                                      |        |                       |  |  |  |  |  |  |           |  |  |   48 89 de             	mov    rsi,rbx
     220:	|  |                                      |        |                       |  |  |  |  |  |  |           |  |  |   48 89 ef             	mov    rdi,rbp
     223:	|  |                                      |        |                       |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	225: R_X86_64_32S	nft_payload_eval
     229:	|  |                                      |        |                 /-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 8b 02 00 00    	je     4ba <nft_do_chain+0x3aa>
     22f:	|  |                                      |        |                 |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	231: R_X86_64_32S	nft_cmp_eval
     235:	|  |                                      |        |     /-----------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 c1 02 00 00    	je     4fc <nft_do_chain+0x3ec>
     23b:	|  |                                      |        |     |           |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	23d: R_X86_64_32S	nft_counter_eval
     241:	|  |                                      |        |     |  /--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 bf 02 00 00    	je     506 <nft_do_chain+0x3f6>
     247:	|  |                                      |        |     |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	249: R_X86_64_32S	nft_meta_get_eval
     24d:	|  |                                      |     /--|-----|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 c8 02 00 00    	je     51b <nft_do_chain+0x40b>
     253:	|  |                                      |     |  |     |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	255: R_X86_64_32S	nft_lookup_eval
     259:	|  |                                      |     |  |  /--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 c6 02 00 00    	je     525 <nft_do_chain+0x415>
     25f:	|  |                                      |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	261: R_X86_64_32S	nft_range_eval
     265:	|  |                                /-----|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 13 03 00 00    	je     57e <nft_do_chain+0x46e>
     26b:	|  |                                |     |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	26d: R_X86_64_32S	nft_immediate_eval
     271:	|  |                                |  /--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 11 03 00 00    	je     588 <nft_do_chain+0x478>
     277:	|  |                                |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	279: R_X86_64_32S	nft_byteorder_eval
     27d:	|  |              /-----------------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 3e 03 00 00    	je     5c1 <nft_do_chain+0x4b1>
     283:	|  |              |                 |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	285: R_X86_64_32S	nft_dynset_eval
     289:	|  |              |  /--------------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 3c 03 00 00    	je     5cb <nft_do_chain+0x4bb>
     28f:	|  |              |  |              |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	291: R_X86_64_32S	nft_rt_get_eval
     295:	|  |              |  |  /-----------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 3a 03 00 00    	je     5d5 <nft_do_chain+0x4c5>
     29b:	|  |              |  |  |           |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 3d 00 00 00 00    	cmp    rax,0x0	29d: R_X86_64_32S	nft_bitwise_eval
     2a1:	|  |              |  |  |  /--------|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 38 03 00 00    	je     5df <nft_do_chain+0x4cf>
     2a7:	|  |              |  |  |  |  /-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- e8 00 00 00 00       	call   2ac <nft_do_chain+0x19c>	2a8: R_X86_64_PLT32	__x86_indirect_thunk_rax-0x4
     2ac:	|  |              |  |  |  |  >-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-> 8b 54 24 48          	mov    edx,DWORD PTR [rsp+0x48]
     2b0:	|  |              |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   83 fa ff             	cmp    edx,0xffffffff
     2b3:	|  |  /-----------|--|--|--|--|-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 85 19 04 00 00    	jne    6d2 <nft_do_chain+0x5c2>
     2b9:	|  |  |           |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 8b 45 00          	mov    rax,QWORD PTR [rbp+0x0]
     2bd:	|  |  |           |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   8b 40 10             	mov    eax,DWORD PTR [rax+0x10]
     2c0:	|  |  |           |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   48 01 c5             	add    rbp,rax
     2c3:	|  |  |           |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  |  |   49 39 ef             	cmp    r15,rbp
     2c6:	|  |  |     /-----|--|--|--|--|-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|--|--|-- 0f 84 f8 01 00 00    	je     4c4 <nft_do_chain+0x3b4>
     2cc:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |  \--|-> 48 8b 45 00          	mov    rax,QWORD PTR [rbp+0x0]
     2d0:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |     |   48 3d 00 00 00 00    	cmp    rax,0x0	2d2: R_X86_64_32S	nft_cmp_fast_ops
     2d6:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |     \-- 0f 85 17 ff ff ff    	jne    1f3 <nft_do_chain+0xe3>
     2dc:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         0f b6 45 10          	movzx  eax,BYTE PTR [rbp+0x10]
     2e0:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         8b 44 84 48          	mov    eax,DWORD PTR [rsp+rax*4+0x48]
     2e4:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         23 45 0c             	and    eax,DWORD PTR [rbp+0xc]
     2e7:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         3b 45 08             	cmp    eax,DWORD PTR [rbp+0x8]
     2ea:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         0f 94 c0             	sete   al
     2ed:	|  |  |     |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         3a 45 12             	cmp    al,BYTE PTR [rbp+0x12]
     2f0:	|  |  |     |     |  |  |  |  +-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|-------- 75 ba                	jne    2ac <nft_do_chain+0x19c>
     2f2:	|  |  |  /--|-----|--|--|--|--|-----|--|--|-----|--|--|--|--|--------|-----|--|--|--|--|--|--|-----------|-------> c7 44 24 48 ff ff ff ff 	mov    DWORD PTR [rsp+0x48],0xffffffff
     2fa:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |        |     |  |  |  |  |  |  |           |         0f 1f 44 00 00       	nop    DWORD PTR [rax+rax*1+0x0]
     2ff:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  /-----|-----|--|--|--|--|--|--|-----------|-------> 41 0f b7 45 00       	movzx  eax,WORD PTR [r13+0x0]
     304:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |           |         66 d1 e8             	shr    ax,1
     307:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |           |         25 ff 0f 00 00       	and    eax,0xfff
     30c:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |           |         4d 8d 6c 05 08       	lea    r13,[r13+rax*1+0x8]
     311:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |           |         4d 39 e5             	cmp    r13,r12
     314:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |           \-------- 0f 82 b5 fe ff ff    	jb     1cf <nft_do_chain+0xbf>
     31a:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |                     4c 89 e5             	mov    rbp,r12
     31d:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |                     8b 54 24 48          	mov    edx,DWORD PTR [rsp+0x48]
     321:	|  |  |  |  |     |  |  |  |  |     |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |                     44 8b 64 24 08       	mov    r12d,DWORD PTR [rsp+0x8]
     326:	|  |  |  |  |     |  |  |  |  |  /--|--|--|-----|--|--|--|--|--|-----|-----|--|--|--|--|--|--|-------------------> 0f 1f 44 00 00       	nop    DWORD PTR [rax+rax*1+0x0]
     32b:	|  |  |  |  |     |  |  |  |  |  |  |  |  |     |  |  |  |  |  |     |     |  |  |  |  |  |  |                     f6 c2 fc             	test   dl,0xfc
     32e:	|  |  |  |  |  /--|--|--|--|--|--|--|--|--|-----|--|--|--|--|--|-----|-----|--|--|--|--|--|--|-------------------- 0f 84 1b 02 00 00    	je     54f <nft_do_chain+0x43f>
     334:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |     |  /--|--|--|--|--|--|--|-------------------> 83 fa fd             	cmp    edx,0xfffffffd
     337:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |  /--|--|--|--|--|--|--|--|--|-------------------- 0f 84 ac 02 00 00    	je     5e9 <nft_do_chain+0x4d9>
     33d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  /----------------- 0f 87 0a 01 00 00    	ja     44d <nft_do_chain+0x33d>
     343:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                  83 fa fb             	cmp    edx,0xfffffffb
     346:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  /-------------- 0f 84 08 01 00 00    	je     454 <nft_do_chain+0x344>
     34c:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |               83 fa fc             	cmp    edx,0xfffffffc
     34f:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  /----------- 0f 85 fd 00 00 00    	jne    452 <nft_do_chain+0x342>
     355:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  /--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|----------> 48 8b 44 24 50       	mov    rax,QWORD PTR [rsp+0x50]
     35a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |            80 7c 24 0f 00       	cmp    BYTE PTR [rsp+0xf],0x0
     35f:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |            48 89 04 24          	mov    QWORD PTR [rsp],rax
     363:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |            48 8b 04 24          	mov    rax,QWORD PTR [rsp]
     367:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  \--|--|--|--|--|--|----------- 0f 85 3b fe ff ff    	jne    1a8 <nft_do_chain+0x98>
     36d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  \--|-----|--|--|--|--|--|----------> 48 8b 00             	mov    rax,QWORD PTR [rax]
     370:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |     \-----|--|--|--|--|--|----------- e9 37 fe ff ff       	jmp    1ac <nft_do_chain+0x9c>
     375:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  |  \--|--|--|----------> 0f b6 45 10          	movzx  eax,BYTE PTR [rbp+0x10]
     379:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  |     |  |  |            0f b6 55 11          	movzx  edx,BYTE PTR [rbp+0x11]
     37d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  |     |  |  |            8b 44 84 48          	mov    eax,DWORD PTR [rsp+rax*4+0x48]
     381:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  |     |  |  |            23 45 08             	and    eax,DWORD PTR [rbp+0x8]
     384:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  |     |  |  |            33 45 0c             	xor    eax,DWORD PTR [rbp+0xc]
     387:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  |     |  |  |            89 44 94 48          	mov    DWORD PTR [rsp+rdx*4+0x48],eax
     38b:	|  |  |  |  |  |  |  |  |  |  +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|-----------|--|-----|--|--|----------- e9 1c ff ff ff       	jmp    2ac <nft_do_chain+0x19c>
     390:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |  \-----|--|--|----------> 0f b6 45 28          	movzx  eax,BYTE PTR [rbp+0x28]
     394:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |            31 d2                	xor    edx,edx
     396:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |            48 8d 0c 83          	lea    rcx,[rbx+rax*4]
     39a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |            48 8b 01             	mov    rax,QWORD PTR [rcx]
     39d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |            48 23 45 18          	and    rax,QWORD PTR [rbp+0x18]
     3a1:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |            48 3b 45 08          	cmp    rax,QWORD PTR [rbp+0x8]
     3a5:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        /-- 75 11                	jne    3b8 <nft_do_chain+0x2a8>
     3a7:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        |   31 d2                	xor    edx,edx
     3a9:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        |   48 8b 41 08          	mov    rax,QWORD PTR [rcx+0x8]
     3ad:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        |   48 23 45 20          	and    rax,QWORD PTR [rbp+0x20]
     3b1:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        |   48 3b 45 10          	cmp    rax,QWORD PTR [rbp+0x10]
     3b5:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        |   0f 94 c2             	sete   dl
     3b8:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |        \-> 0f b6 45 2a          	movzx  eax,BYTE PTR [rbp+0x2a]
     3bc:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           |        |  |  |            39 d0                	cmp    eax,edx
     3be:	|  |  |  |  |  |  |  |  |  |  +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|-----------|--------|--|--|----------- 0f 85 e8 fe ff ff    	jne    2ac <nft_do_chain+0x19c>
     3c4:	|  |  |  +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|--|-----------|--------|--|--|----------- e9 29 ff ff ff       	jmp    2f2 <nft_do_chain+0x1e2>
     3c9:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |           \--------|--|--|----------> 80 7d 08 01          	cmp    BYTE PTR [rbp+0x8],0x1
     3cd:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |            49 8b 36             	mov    rsi,QWORD PTR [r14]
     3d0:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     /----- 0f 84 ce 00 00 00    	je     4a4 <nft_do_chain+0x394>
     3d6:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |      41 f6 46 10 01       	test   BYTE PTR [r14+0x10],0x1
     3db:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  +--------------------|--|--|-----|----- 0f 84 36 fe ff ff    	je     217 <nft_do_chain+0x107>
     3e1:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |      0f b7 96 b8 00 00 00 	movzx  edx,WORD PTR [rsi+0xb8]
     3e8:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |      41 0f b7 7e 14       	movzx  edi,WORD PTR [r14+0x14]
     3ed:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |      48 8b 8e c8 00 00 00 	mov    rcx,QWORD PTR [rsi+0xc8]
     3f4:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |      48 01 fa             	add    rdx,rdi
     3f7:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |      48 01 ca             	add    rdx,rcx
     3fa:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  /-> 0f b6 7d 09          	movzx  edi,BYTE PTR [rbp+0x9]
     3fe:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  |   8b b6 bc 00 00 00    	mov    esi,DWORD PTR [rsi+0xbc]
     404:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  |   48 01 fa             	add    rdx,rdi
     407:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  |   0f b6 7d 0a          	movzx  edi,BYTE PTR [rbp+0xa]
     40b:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  |   48 01 f1             	add    rcx,rsi
     40e:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  |   48 01 d7             	add    rdi,rdx
     411:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |     |  |   48 39 f9             	cmp    rcx,rdi
     414:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  \--------------------|--|--|-----|--|-- 0f 82 fd fd ff ff    	jb     217 <nft_do_chain+0x107>
     41a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |     |  |   0f b6 4d 0b          	movzx  ecx,BYTE PTR [rbp+0xb]
     41e:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |     |  |   c7 44 8c 48 00 00 00 00 	mov    DWORD PTR [rsp+rcx*4+0x48],0x0
     426:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |     |  |   48 8d 34 8b          	lea    rsi,[rbx+rcx*4]
     42a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |     |  |   48 89 c8             	mov    rax,rcx
     42d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |     |  |   0f b6 4d 0a          	movzx  ecx,BYTE PTR [rbp+0xa]
     431:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |     |  |   80 f9 02             	cmp    cl,0x2
     434:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |  /--|--|-- 0f 84 d6 00 00 00    	je     510 <nft_do_chain+0x400>
     43a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                       |  |  |  |  |  |   80 f9 04             	cmp    cl,0x4
     43d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    /--|--|--|--|--|--|-- 0f 84 4f 01 00 00    	je     592 <nft_do_chain+0x482>
     443:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |  |  |  |  |   0f b6 02             	movzx  eax,BYTE PTR [rdx]
     446:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  |  |  |  |  |  |   88 06                	mov    BYTE PTR [rsi],al
     448:	|  |  |  |  |  |  |  |  |  |  +--|--|--|--|--|--|--|--|--|--|--|--|--|--|--------------------|--|--|--|--|--|--|-- e9 5f fe ff ff       	jmp    2ac <nft_do_chain+0x19c>
     44d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |  \--|--|--|--|--|-> 83 fa ff             	cmp    edx,0xffffffff
     450:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     +--|--|--|--|-- 74 02                	je     454 <nft_do_chain+0x344>
     452:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |  \--|--|--|-> 0f 0b                	ud2
     454:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     \-----|--|--|-> 45 85 e4             	test   r12d,r12d
     457:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     /-----|--|--|-- 0f 84 19 02 00 00    	je     676 <nft_do_chain+0x566>
     45d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   41 8d 44 24 ff       	lea    eax,[r12-0x1]
     462:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   c7 44 24 48 ff ff ff ff 	mov    DWORD PTR [rsp+0x48],0xffffffff
     46a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   49 89 c4             	mov    r12,rax
     46d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   48 8d 04 40          	lea    rax,[rax+rax*2]
     471:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   48 c1 e0 03          	shl    rax,0x3
     475:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   48 8b bc 04 98 00 00 00 	mov    rdi,QWORD PTR [rsp+rax*1+0x98]
     47d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   4c 8b ac 04 a0 00 00 00 	mov    r13,QWORD PTR [rsp+rax*1+0xa0]
     485:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   48 8b ac 04 a8 00 00 00 	mov    rbp,QWORD PTR [rsp+rax*1+0xa8]
     48d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   48 89 3c 24          	mov    QWORD PTR [rsp],rdi
     491:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |                    |     |     |  |  |   49 39 ed             	cmp    r13,rbp
     494:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  \--|--|--|--|--|--|--|--------------------|-----|-----|--|--|-- 0f 82 2d fd ff ff    	jb     1c7 <nft_do_chain+0xb7>
     49a:	|  |  |  |  |  |  |  |  |  |  |  |  |  |  \--|--|-----|--|--|--|--|--|--|--------------------|-----|-----|--|--|-> ba ff ff ff ff       	mov    edx,0xffffffff
     49f:	|  |  |  |  |  |  |  |  |  |  |  +--|--|-----|--|-----|--|--|--|--|--|--|--------------------|-----|-----|--|--|-- e9 82 fe ff ff       	jmp    326 <nft_do_chain+0x216>
     4a4:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |  |  |                    |     |     |  \--|-> 48 8b 8e c8 00 00 00 	mov    rcx,QWORD PTR [rsi+0xc8]
     4ab:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |  |  |                    |     |     |     |   0f b7 96 b8 00 00 00 	movzx  edx,WORD PTR [rsi+0xb8]
     4b2:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |  |  |                    |     |     |     |   48 01 ca             	add    rdx,rcx
     4b5:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |  |  |                    |     |     |     \-- e9 40 ff ff ff       	jmp    3fa <nft_do_chain+0x2ea>
     4ba:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |  \--|--------------------|-----|-----|-----/-X e8 00 00 00 00       	call   4bf <nft_do_chain+0x3af>	4bb: R_X86_64_PLT32	nft_payload_eval-0x4
     4bf:	|  |  |  |  |  |  |  |  |  |  +--|--|--|-----|--|-----|--|--|--|--|-----|--------------------|-----|-----|-----\-X e9 e8 fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     4c4:	|  |  |  |  >--|--|--|--|--|--|--|--|--|-----|--|-----|--|--|--|--|-----|--------------------|-----|-----|-------> 66 90                	xchg   ax,ax
     4c6:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  +--|-----|--------------------|-----|-----|-------- e9 34 fe ff ff       	jmp    2ff <nft_do_chain+0x1ef>
     4cb:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         49 8b 06             	mov    rax,QWORD PTR [r14]
     4ce:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         48 8b 34 24          	mov    rsi,QWORD PTR [rsp]
     4d2:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         ba 03 00 00 00       	mov    edx,0x3
     4d7:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         48 8d 7c 24 18       	lea    rdi,[rsp+0x18]
     4dc:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         4c 89 6c 24 38       	mov    QWORD PTR [rsp+0x38],r13
     4e1:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         0f b6 80 80 00 00 00 	movzx  eax,BYTE PTR [rax+0x80]
     4e8:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         c0 e8 04             	shr    al,0x4
     4eb:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         83 e0 01             	and    eax,0x1
     4ee:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         88 44 24 19          	mov    BYTE PTR [rsp+0x19],al
     4f2:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  |  |     |                    |     |     |         e8 19 fb ff ff       	call   10 <__nft_trace_packet>
     4f7:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  |  |  +--|-----|--------------------|-----|-----|-------- e9 03 fe ff ff       	jmp    2ff <nft_do_chain+0x1ef>
     4fc:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |  \--|--|--|-----|--------------------|-----|-----|-----/-X e8 00 00 00 00       	call   501 <nft_do_chain+0x3f1>	4fd: R_X86_64_PLT32	nft_cmp_eval-0x4
     501:	|  |  |  |  |  |  |  |  |  |  +--|--|--|-----|--|-----|-----|--|--|-----|--------------------|-----|-----|-----\-X e9 a6 fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     506:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |     \--|--|-----|--------------------|-----|-----|-----/-X e8 00 00 00 00       	call   50b <nft_do_chain+0x3fb>	507: R_X86_64_PLT32	nft_counter_eval-0x4
     50b:	|  |  |  |  |  |  |  |  |  |  +--|--|--|-----|--|-----|--------|--|-----|--------------------|-----|-----|-----\-X e9 9c fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     510:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |        |  |     |                    |     |     \-------> 0f b7 02             	movzx  eax,WORD PTR [rdx]
     513:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  |     |        |  |     |                    |     |               66 89 06             	mov    WORD PTR [rsi],ax
     516:	|  |  |  |  |  |  |  |  |  |  +--|--|--|-----|--|-----|--------|--|-----|--------------------|-----|-------------- e9 91 fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     51b:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |  \-----|--------|--|-----|--------------------|-----|-----------/-X e8 00 00 00 00       	call   520 <nft_do_chain+0x410>	51c: R_X86_64_PLT32	nft_meta_get_eval-0x4
     520:	|  |  |  |  |  |  |  |  |  |  +--|--|--|-----|--------|--------|--|-----|--------------------|-----|-----------\-X e9 87 fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     525:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |        \--------|--|-----|--------------------|-----|-----------/-X e8 00 00 00 00       	call   52a <nft_do_chain+0x41a>	526: R_X86_64_PLT32	nft_lookup_eval-0x4
     52a:	|  |  |  |  |  |  |  |  |  |  +--|--|--|-----|-----------------|--|-----|--------------------|-----|-----------\-X e9 7d fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     52f:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     |                    |     |               48 8b 34 24          	mov    rsi,QWORD PTR [rsp]
     533:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     |                    |     |               48 8d 7c 24 18       	lea    rdi,[rsp+0x18]
     538:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     |                    |     |               4c 89 6c 24 38       	mov    QWORD PTR [rsp+0x38],r13
     53d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     |                    |     |               e8 6e fb ff ff       	call   b0 <__nft_trace_verdict.isra.0>
     542:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     |                    |     |               8b 54 24 48          	mov    edx,DWORD PTR [rsp+0x48]
     546:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     |                    |     |               f6 c2 fc             	test   dl,0xfc
     549:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |     \--------------------|-----|-------------- 0f 85 e5 fd ff ff    	jne    334 <nft_do_chain+0x224>
     54f:	|  |  |  |  |  >--|--|--|--|--|--|--|--|-----|-----------------|--|--------------------------|-----|-------------> 48 8b 84 24 18 02 00 00 	mov    rax,QWORD PTR [rsp+0x218]
     557:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |               65 48 2b 04 25 28 00 00 00 	sub    rax,QWORD PTR gs:0x28
     560:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  /----------- 0f 85 67 01 00 00    	jne    6cd <nft_do_chain+0x5bd>
     566:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            48 81 c4 20 02 00 00 	add    rsp,0x220
     56d:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            89 d0                	mov    eax,edx
     56f:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            5b                   	pop    rbx
     570:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            5d                   	pop    rbp
     571:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            41 5c                	pop    r12
     573:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            41 5d                	pop    r13
     575:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            41 5e                	pop    r14
     577:	|  |  |  |  |  |  |  |  |  |  |  |  |  |     |                 |  |                          |     |  |            41 5f                	pop    r15
     579:	|  |  |  |  |  |  |  |  |  |  |  |  +--|-----|-----------------|--|--------------------------|-----|--|----------- e9 00 00 00 00       	jmp    57e <nft_do_chain+0x46e>	57a: R_X86_64_PLT32	__x86_return_thunk-0x4
     57e:	|  |  |  |  |  |  |  |  |  |  |  |  \--|-----|-----------------|--|--------------------------|-----|--|--------/-X e8 00 00 00 00       	call   583 <nft_do_chain+0x473>	57f: R_X86_64_PLT32	nft_range_eval-0x4
     583:	|  |  |  |  |  |  |  |  |  |  +--|-----|-----|-----------------|--|--------------------------|-----|--|--------\-X e9 24 fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     588:	|  |  |  |  |  |  |  |  |  |  |  |     \-----|-----------------|--|--------------------------|-----|--|--------/-X e8 00 00 00 00       	call   58d <nft_do_chain+0x47d>	589: R_X86_64_PLT32	nft_immediate_eval-0x4
     58d:	|  |  |  |  |  |  |  |  |  |  +--|-----------|-----------------|--|--------------------------|-----|--|--------\-X e9 1a fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     592:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                          \-----|--|----------> 8b 12                	mov    edx,DWORD PTR [rdx]
     594:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            89 54 84 48          	mov    DWORD PTR [rsp+rax*4+0x48],edx
     598:	|  |  |  |  |  |  |  |  |  |  +--|-----------|-----------------|--|--------------------------------|--|----------- e9 0f fd ff ff       	jmp    2ac <nft_do_chain+0x19c>
     59d:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            80 7c 24 18 00       	cmp    BYTE PTR [rsp+0x18],0x0
     5a2:	|  |  |  |  |  |  |  |  |  |  |  |           |                 +--|--------------------------------|--|----------- 0f 84 57 fd ff ff    	je     2ff <nft_do_chain+0x1ef>
     5a8:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            49 8b 06             	mov    rax,QWORD PTR [r14]
     5ab:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            0f b6 80 80 00 00 00 	movzx  eax,BYTE PTR [rax+0x80]
     5b2:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            c0 e8 04             	shr    al,0x4
     5b5:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            83 e0 01             	and    eax,0x1
     5b8:	|  |  |  |  |  |  |  |  |  |  |  |           |                 |  |                                |  |            88 44 24 19          	mov    BYTE PTR [rsp+0x19],al
     5bc:	|  |  |  |  |  |  |  |  |  |  |  |           |                 \--|--------------------------------|--|----------- e9 3e fd ff ff       	jmp    2ff <nft_do_chain+0x1ef>
     5c1:	|  |  |  |  |  |  \--|--|--|--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00       	call   5c6 <nft_do_chain+0x4b6>	5c2: R_X86_64_PLT32	nft_byteorder_eval-0x4
     5c6:	|  |  |  |  |  |     |  |  |  +--|-----------|--------------------|--------------------------------|--|--------\-X e9 e1 fc ff ff       	jmp    2ac <nft_do_chain+0x19c>
     5cb:	|  |  |  |  |  |     \--|--|--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00       	call   5d0 <nft_do_chain+0x4c0>	5cc: R_X86_64_PLT32	nft_dynset_eval-0x4
     5d0:	|  |  |  |  |  |        |  |  +--|-----------|--------------------|--------------------------------|--|--------\-X e9 d7 fc ff ff       	jmp    2ac <nft_do_chain+0x19c>
     5d5:	|  |  |  |  |  |        \--|--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00       	call   5da <nft_do_chain+0x4ca>	5d6: R_X86_64_PLT32	nft_rt_get_eval-0x4
     5da:	|  |  |  |  |  |           |  +--|-----------|--------------------|--------------------------------|--|--------\-X e9 cd fc ff ff       	jmp    2ac <nft_do_chain+0x19c>
     5df:	|  |  |  |  |  |           \--|--|-----------|--------------------|--------------------------------|--|--------/-X e8 00 00 00 00       	call   5e4 <nft_do_chain+0x4d4>	5e0: R_X86_64_PLT32	nft_bitwise_eval-0x4
     5e4:	|  |  |  |  |  |              \--|-----------|--------------------|--------------------------------|--|--------\-X e9 c3 fc ff ff       	jmp    2ac <nft_do_chain+0x19c>
     5e9:	|  |  |  |  |  |                 |           |                    \--------------------------------|--|----------> 41 83 fc 0f          	cmp    r12d,0xf
     5ed:	|  |  |  |  |  |                 |           |                                                     |  |     /----- 77 7e                	ja     66d <nft_do_chain+0x55d>
     5ef:	|  |  |  |  |  |                 |           |                                                     |  |     |      41 0f b7 55 00       	movzx  edx,WORD PTR [r13+0x0]
     5f4:	|  |  |  |  |  |                 |           |                                                     |  |     |      44 89 e0             	mov    eax,r12d
     5f7:	|  |  |  |  |  |                 |           |                                                     |  |     |      48 8b 3c 24          	mov    rdi,QWORD PTR [rsp]
     5fb:	|  |  |  |  |  |                 |           |                                                     |  |     |      41 83 c4 01          	add    r12d,0x1
     5ff:	|  |  |  |  |  |                 |           |                                                     |  |     |      48 8d 04 40          	lea    rax,[rax+rax*2]
     603:	|  |  |  |  |  |                 |           |                                                     |  |     |      66 d1 ea             	shr    dx,1
     606:	|  |  |  |  |  |                 |           |                                                     |  |     |      48 c1 e0 03          	shl    rax,0x3
     60a:	|  |  |  |  |  |                 |           |                                                     |  |     |      81 e2 ff 0f 00 00    	and    edx,0xfff
     610:	|  |  |  |  |  |                 |           |                                                     |  |     |      48 89 bc 04 98 00 00 00 	mov    QWORD PTR [rsp+rax*1+0x98],rdi
     618:	|  |  |  |  |  |                 |           |                                                     |  |     |      49 8d 54 15 08       	lea    rdx,[r13+rdx*1+0x8]
     61d:	|  |  |  |  |  |                 |           |                                                     |  |     |      48 89 ac 04 a8 00 00 00 	mov    QWORD PTR [rsp+rax*1+0xa8],rbp
     625:	|  |  |  |  |  |                 |           |                                                     |  |     |      48 89 94 04 a0 00 00 00 	mov    QWORD PTR [rsp+rax*1+0xa0],rdx
     62d:	|  |  |  |  |  |                 |           \-----------------------------------------------------|--|-----|----- e9 23 fd ff ff       	jmp    355 <nft_do_chain+0x245>
     632:	|  \--|--|--|--|-----------------|-----------------------------------------------------------------|--|-----|----> 8b 54 24 48          	mov    edx,DWORD PTR [rsp+0x48]
     636:	|     |  |  |  |                 |                                                                 |  |     |      83 fa fe             	cmp    edx,0xfffffffe
     639:	|     |  +--|--|-----------------|-----------------------------------------------------------------|--|-----|----- 0f 84 b3 fc ff ff    	je     2f2 <nft_do_chain+0x1e2>
     63f:	|     |  |  |  |                 |                                                                 |  |     |      83 fa ff             	cmp    edx,0xffffffff
     642:	|     |  |  \--|-----------------|-----------------------------------------------------------------|--|-----|----- 0f 84 7c fe ff ff    	je     4c4 <nft_do_chain+0x3b4>
     648:	|     |  |     |                 |                                                                 |  |  /--|----> 4c 89 e5             	mov    rbp,r12
     64b:	|     |  |     |                 |                                                                 |  |  |  |      44 8b 64 24 08       	mov    r12d,DWORD PTR [rsp+0x8]
     650:	|     |  |     |                 \-----------------------------------------------------------------|--|--|--|----- e9 d1 fc ff ff       	jmp    326 <nft_do_chain+0x216>
     655:	|     |  |     |                                                                                   |  |  |  |      48 89 e9             	mov    rcx,rbp
     658:	|     |  |     |                                                                                   |  |  |  |      48 89 da             	mov    rdx,rbx
     65b:	|     |  |     |                                                                                   |  |  |  |      48 8d 7c 24 18       	lea    rdi,[rsp+0x18]
     660:	|     |  |     |                                                                                   |  |  |  |      4c 89 f6             	mov    rsi,r14
     663:	|     |  |     |                                                                                   |  |  |  |  /-- e8 00 00 00 00       	call   668 <nft_do_chain+0x558>	664: R_X86_64_PLT32	nft_trace_init-0x4
     668:	\-----|--|-----|-----------------------------------------------------------------------------------|--|--|--|--\-X e9 1b fb ff ff       	jmp    188 <nft_do_chain+0x78>
     66d:	      |  |     |                                                                                   |  |  |  \----> 0f 0b                	ud2
     66f:	      |  |     |                                                                                   |  |  |         31 d2                	xor    edx,edx
     671:	      |  |     +-----------------------------------------------------------------------------------|--|--|-------- e9 d9 fe ff ff       	jmp    54f <nft_do_chain+0x43f>
     676:	      |  |     |                                                                                   \--|--|-------> 48 8b 6c 24 10       	mov    rbp,QWORD PTR [rsp+0x10]
     67b:	      |  |     |                                                                                      |  |         66 90                	xchg   ax,ax
     67d:	      |  |     |                                                                                      |  |     /-> 66 90                	xchg   ax,ax
     67f:	      |  |     |                                                                                      |  |     |   0f b6 55 f0          	movzx  edx,BYTE PTR [rbp-0x10]
     683:	      |  |     +--------------------------------------------------------------------------------------|--|-----|-- e9 c7 fe ff ff       	jmp    54f <nft_do_chain+0x43f>
     688:	      |  |     |                                                                                      |  |     |   4c 89 f6             	mov    rsi,r14
     68b:	      |  |     |                                                                                      |  |     |   48 89 ef             	mov    rdi,rbp
     68e:	      |  |     |                                                                                      |  |     |   e8 bd f9 ff ff       	call   50 <nft_update_chain_stats>
     693:	      |  |     |                                                                                      |  |     |   0f b6 55 f0          	movzx  edx,BYTE PTR [rbp-0x10]
     697:	      |  |     \--------------------------------------------------------------------------------------|--|-----|-- e9 b3 fe ff ff       	jmp    54f <nft_do_chain+0x43f>
     69c:	      |  |                                                                                            |  |     |   49 8b 06             	mov    rax,QWORD PTR [r14]
     69f:	      |  |                                                                                            |  |     |   ba 01 00 00 00       	mov    edx,0x1
     6a4:	      |  |                                                                                            |  |     |   48 89 ee             	mov    rsi,rbp
     6a7:	      |  |                                                                                            |  |     |   48 8d 7c 24 18       	lea    rdi,[rsp+0x18]
     6ac:	      |  |                                                                                            |  |     |   48 c7 44 24 38 00 00 00 00 	mov    QWORD PTR [rsp+0x38],0x0
     6b5:	      |  |                                                                                            |  |     |   0f b6 80 80 00 00 00 	movzx  eax,BYTE PTR [rax+0x80]
     6bc:	      |  |                                                                                            |  |     |   c0 e8 04             	shr    al,0x4
     6bf:	      |  |                                                                                            |  |     |   83 e0 01             	and    eax,0x1
     6c2:	      |  |                                                                                            |  |     |   88 44 24 19          	mov    BYTE PTR [rsp+0x19],al
     6c6:	      |  |                                                                                            |  |     |   e8 45 f9 ff ff       	call   10 <__nft_trace_packet>
     6cb:	      |  |                                                                                            |  |     \-- eb b0                	jmp    67d <nft_do_chain+0x56d>
     6cd:	      +--|--------------------------------------------------------------------------------------------\--|-------X e8 00 00 00 00       	call   6d2 <nft_do_chain+0x5c2>	6ce: R_X86_64_PLT32	__stack_chk_fail-0x4
     6d2:	      \--|-----------------------------------------------------------------------------------------------|-------> 83 fa fe             	cmp    edx,0xfffffffe
     6d5:	         |                                                                                               \-------- 0f 85 6d ff ff ff    	jne    648 <nft_do_chain+0x538>
     6db:	         \-------------------------------------------------------------------------------------------------------- e9 12 fc ff ff       	jmp    2f2 <nft_do_chain+0x1e2>

Now, I would love to be able to say that I could immediately follow the flow of that and zero in on the instruction of interest, but truth be told, I am not actually very good at reading assembly output. Especially not x86_64 assembly (so many instructions! and what’s with those weird names for the registers?). However, armed with a handy reference for the register names that some googling turned up, and another one for the instruction names, I set out to try and make sense of it all.

Fancy ASCII graphs notwithstanding, I quickly gave up on tracing the execution flow from the beginning of the function. Instead, I looked for branches in the C source above that were near the place I wanted to hook in. We already identified that the call site for nft_trace_verdict() would be a good place, but that is obscured some by the static_call infrastructure. However, right after it there is this branch, which turned out to be a better candidate:

	switch (regs.verdict.code & NF_VERDICT_MASK) {
	case NF_ACCEPT:
	case NF_DROP:
	case NF_QUEUE:
	case NF_STOLEN:
		return regs.verdict.code;
	}

To find this in the disassembly, first notice that the NF_VERDICT_MATCH is 0xff, and the values of those four verdict codes are 0, 1, 2 and 3. This means that this statement compiles down to a check for regs.verdict.code & 0xfc, and if that is non-zero, the function returns. There’s a test instruction for such an AND test, so let’s look for this in the disassembled output above (just search for 0xfc).

There’s one such instruction at offset 0x32b, and another at 0x546, both of them testing the same dl register. By using bpftrace to just print out a simple message at each site, we can quickly determine if they seem to be executed when the event we’re interested in, by a oneliner like this one2:

$ sudo bpftrace -e 'kprobe:nft_do_chain+0x21b { printf("Here!\n"); }'

This quickly printed out a bunch of messages when I generated traffic on the system, so that seemed to be the right place. Now we just need to figure out how we are going to access the values we need: the struct nft_rule_dp pointer that leads to the rule, and the pkt pointer that allows up to look at the packet data itself.

The latter one is actually fairly straight-forward: the pkt pointer is passed in as the first argument to the function when it starts, meaning it starts out in the rdi register. And one of the first instructions in the function is a mov r14,rdi, copying that value into r14, where it stays for the rest of the function execution (there are no other instructions putting a new value into the r14 register until the function returns).

To obtain the other pointer, I noticed that the call to the netfilter tracing function is in the output above; it appears as call b0 <__nft_trace_verdict.isra.0>. And right before it, there’s an instruction storing the value of the r13 register into a value on the stack which is what the inline wrapper function (nft_trace_verdict()) does with the rule pointer. So the rule pointer is probably in r13!

Putting it all together

With the above information, we can construct a small bpftrace probe that can read out the values we need. We have two pointers stored in registers r13 and r14, and bpftrace simply lets us cast those pointers to their struct values and dereference them to get at the values we really want3. Due to the BPF tracing magic, this is safe even if the pointers turn out to be invalid: We don’t crash the kernel by a bad pointer deref, we’ll just get an invalid value back. So if we get reasonable values, we know that we are poking at the right bit of memory.

To try it out, I constructed the following bpftrace script:

#!/usr/bin/bpftrace
kprobe:nft_do_chain+0x21b {
  if ((reg("dx") & 0xff) == 0) {
    printf("Packet with len %d dropped by rule with handle %d\n",
           ((struct nft_pktinfo *)reg("r14"))->skb->len,
           *((uint64 *)reg("r13")) >> 13);
  }
}

This first looks at the return code, which we determined above is in the dx register4. NFT_DROP has a value of 0, and we’re only interested in drops, so we only print something if this was a drop verdict - that’s the if statement in the bpftrace script above.

When we do get a drop verdict, we simply print out a message on the console with the packet length and the handle of the rule that gave the verdict. I chose the length because this is stored directly in struct sk_buff as a len field, so it’s easy to get at by just doing a bit of pointer walking. If we wanted to look at the packet data we’d have to do a bit more work, but for this short verification the length will suffice.

To get the rule handle, we have to be slightly more creative: the nft_rule_dp struct contains a u64 as its first member that is defined as a bitfield with three members, the handle being one of those. However, BTF doesn’t support bitfields, so we can’t just dereference the struct member. Instead we just cast the pointer directly to a u64 and manually shift it by the required 13 bits to get the handle.

Saving the script above as nfdrop.bt and running it, we get output like this (while producing test traffic from another machine that we know will get dropped):

$ sudo ./nfdrop.bt
Attaching 1 probe...
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
Packet with len 60 dropped by rule with handle 7
^C

Success! Handle 7 is the extra rule I added to the ruleset to test with, and this output happens whenever I produce traffic with a destination port of 10000 like the rule specifies. Dumping the traffic with tcpdump confirms that the packet size is 60 bytes, so we’re getting sane values. This means that we’ve successfully probed into the middle of the kernel function and can get at both the rule and packet data at the place where the verdict is set! 🥳

Caveats

Now, the obvious problem with all this is that while the resulting script above is quite small, it is also entirely specific to the kernel binary that I am currently running on my own machine. Any change in the code, or even a change in how the compiler generates the code of that function, will invalidate it as that will change the function offsets and/or the register usage.

This means that it is not really a general solution to the problem, and if this were to be part of a monitoring solution that someone wants to deploy on multiple systems, the script would potentially have to be adjusted for each one. I don’t know of any automated way to generate this either, so this would probably have to be done manually. However, for this demonstration that doesn’t really matter, and I thought it was quite interesting to poke into all this and figure out exactly how kprobes in the middle of a function works!

Summary

The above shows how to use kprobes attached to the middle of a kernel function to get at data that is not otherwise available because there is no useful attach point to get at it. This is an incredibly powerful technique to obtain any kind of information from a running kernel - I did all this on my regular laptop without changing anything in the kernel code, or even rebooting it!

Getting the information out involved quite a lot of manual work to decipher the function code and find the attach point, but this can be worth it for certain use cases. And many functions are less complex than the one we were looking at here, in which case finding the right attach point can be easier.

I certainly found the exploration above interesting, and if you’re still reading this far down, I’m hoping you did as well! 😅


  1. This works by rewriting the target instruction so that it becomes an instruction that raises a processor exception, and then executing the kprobe itself in the exception handler, before executing the original instruction and jumping back to the function itself so execution can continue. See the kprobes documentation for more details.

    [return]
  2. Note that the offsets in the objdump output starts at 0x110, so we have to subtract that value, yielding the 0x21b value in the bpftrace script.

    [return]
  3. It used to be the case that you had to also include the header files with the struct definition you were interested in into your bpftrace script. However, newer versions of bpftrace will automatically look at the BTF information of the running kernel (if the kernel is built with BTF support) and determine the struct layout for us, so we can just use the struct name directly. Pretty neat.

    [return]
  4. The reg() function is a bpftrace internal function which returns the value of a named register; bpftrace uses slightly different register names than those in the objdump output, so this corresponds to the earlier dl register we saw as an argument to the test instruction.

    [return]