Tutorial

Tuxscope Lab 4: Network Monitoring

Monitor TCP connections in real time by probing tcp_v4_connect and inet_csk_accept, capture source and destination IPs, ports, and connection direction.

8 min read intermediate
eBPF Infrastructure Linux

Prerequisites

  • Completed Labs 1-3 (Hello eBPF, Syscall Tracing, File I/O Observation)
  • Basic TCP/IP knowledge helpful

Part 4 of 7 in Tuxscope: Linux Kernel Observability with eBPF

Table of Contents

In the previous labs you observed syscalls and file I/O. You could see that a process called write or read, but you could not see where the data was going over the network. In this lab you probe two functions deep in the TCP stack, tcp_v4_connect for outbound connections and inet_csk_accept for inbound connections, to capture IP addresses, ports, and connection direction in real time.

This lab introduces two new techniques: kretprobes (probing function return instead of entry) and bpf_probe_read_kernel (safely reading kernel memory from BPF programs).

The complete source code is at gitlab.com/sfoerster/tuxscope.

Note

Prerequisites You need a Linux system running kernel 5.8 or later with root access and the tuxscope binary built from source. You should have completed Labs 1-3. Familiarity with TCP/IP basics (what ports and IP addresses are, the difference between client and server) is helpful but not strictly required.

TCP connections in the Linux kernel

When a process makes a TCP connection (e.g., curl https://example.com), the path through the kernel looks like this:

 User Space                    Kernel Space
┌──────────┐                  ┌─────────────────────────────────┐
│           │                  │                                 │
│ connect() ┼─────────────────→│  sys_connect()                  │
│           │                  │      │                          │
│           │                  │      v                          │
│           │                  │  tcp_v4_connect()  ← outbound  │
│           │                  │      │                          │
│           │                  │      v                          │
│           │                  │  Send SYN packet                │
│           │                  │                                 │
│ accept()  ┼─────────────────→│  sys_accept()                   │
│           │                  │      │                          │
│           │                  │      v                          │
│           │                  │  inet_csk_accept() ← inbound   │
│           │                  │      │                          │
│           │                  │      v                          │
│           │                  │  Return new socket              │
│           │                  │                                 │
└──────────┘                  └─────────────────────────────────┘

tcp_v4_connect is called when a process initiates an outbound IPv4 TCP connection. Its signature is:

int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len);

The second argument, uaddr, points to a sockaddr_in structure containing the destination IP and port. By reading this structure, you know where the process is connecting to.

inet_csk_accept is called when a listening socket accepts an inbound connection. Its signature is:

struct sock *inet_csk_accept(struct sock *sk, int flags, int *err, bool kern);

The return value is a pointer to the new socket structure representing the accepted connection. The socket contains both the remote (source) and local (destination) addresses. Since the interesting data is in the return value, you need a kretprobe here, not a regular kprobe.

Kretprobes

In Lab 3 you used kprobes, which fire when a function is entered. A kretprobe fires when the function returns. This gives you access to the return value.

 kprobe                              kretprobe
    │                                    │
    v                                    v
┌───────────────────────────────────────────┐
│  function_entry:                          │
│      ... function body ...                │
│      return value;  ──────────────────────│──→ captured here
└───────────────────────────────────────────┘

In Aya, a kretprobe is declared with #[kretprobe] and receives a RetProbeContext. The return value is accessed with ctx.ret().

This is why inet_csk_accept needs a kretprobe: the information you want (the new socket with remote address information) is the return value. At function entry, the new socket does not exist yet.

Reading kernel memory safely: bpf_probe_read_kernel

In a BPF program, you cannot simply dereference a kernel pointer. The BPF verifier will reject any program that does this. Instead, you must use bpf_probe_read_kernel to copy data from a kernel address into your BPF stack:

use aya_ebpf::helpers::bpf_probe_read_kernel;

// WRONG: the verifier will reject this
// let value = *some_kernel_ptr;

// CORRECT: safely copy the value without dereferencing it directly
let value_ptr = some_kernel_ptr as *const u32;
let value: u32 = unsafe { bpf_probe_read_kernel(value_ptr)? };

Why the restriction? The pointer might be invalid: the memory could have been freed, the address might be in an unmapped region, or the data might be in a different address space. bpf_probe_read_kernel handles these cases safely: if the read fails, it returns an error instead of crashing the kernel.

In this lab, you need to follow a chain of pointers to reach the IP address and port. For tcp_v4_connect, the chain is:

uaddr (struct sockaddr *) → sockaddr_in.sin_port
                           → sockaddr_in.sin_addr.s_addr

For inet_csk_accept, the return value is a struct sock *, and you need:

sock → __sk_common.skc_dport     (remote port)
     → __sk_common.skc_rcv_saddr (local IP)
     → __sk_common.skc_daddr     (remote IP)
     → __sk_common.skc_num       (local port)

Each of these requires a bpf_probe_read_kernel call.

The event struct

// tuxscope-common/src/lib.rs

#[repr(C)]
#[derive(Clone, Copy)]
pub struct NetEvent {
    pub pid: u32,
    pub sport: u16,       // source port (remote peer for accept events)
    pub dport: u16,       // destination port (local listener for accept events)
    pub saddr: u32,       // source IPv4 address (network byte order)
    pub daddr: u32,       // destination IPv4 address (network byte order)
    pub timestamp_ns: u64,
    pub comm: [u8; 16],
    pub event_type: u8,   // 0 = connect (outbound), 1 = accept (inbound)
}

IPv4 addresses are stored as 32-bit integers in network byte order (big-endian). Userspace converts them to dotted-quad notation for display.

The event_type field distinguishes outbound connections (connect) from inbound connections (accept), similar to how the op field in Lab 3 distinguished reads from writes.

The eBPF programs

Outbound connections: tcp_v4_connect

// tuxscope-ebpf/src/net.rs

use aya_ebpf::{
    macros::{kprobe, kretprobe, map},
    maps::RingBuf,
    programs::{ProbeContext, RetProbeContext},
    helpers::{
        bpf_get_current_pid_tgid, bpf_ktime_get_ns,
        bpf_get_current_comm, bpf_probe_read_kernel,
    },
};
use tuxscope_common::NetEvent;

#[map]
static EVENTS: RingBuf = RingBuf::with_byte_size(256 * 1024, 0);

#[kprobe]
pub fn trace_tcp_connect(ctx: ProbeContext) -> u32 {
    match try_tcp_connect(&ctx) {
        Ok(()) => 0,
        Err(_) => 1,
    }
}

fn try_tcp_connect(ctx: &ProbeContext) -> Result<(), i64> {
    let pid = (bpf_get_current_pid_tgid() >> 32) as u32;
    let timestamp_ns = unsafe { bpf_ktime_get_ns() };
    let comm = bpf_get_current_comm().map_err(|e| e as i64)?;

    // Argument 1 (index 1) is struct sockaddr *uaddr
    let sockaddr_ptr: *const u8 = ctx.arg(1).ok_or(1i64)?;

    // sockaddr_in layout:
    //   offset 0: sa_family (u16)
    //   offset 2: sin_port  (u16, network byte order)
    //   offset 4: sin_addr  (u32, network byte order)

    let dport: u16 = unsafe {
        bpf_probe_read_kernel((sockaddr_ptr.add(2)) as *const u16)?
    };
    let daddr: u32 = unsafe {
        bpf_probe_read_kernel((sockaddr_ptr.add(4)) as *const u32)?
    };

    let event = NetEvent {
        pid,
        sport: 0,   // source port not yet assigned at connect time
        dport: u16::from_be(dport),
        saddr: 0,   // source address not yet assigned
        daddr,
        timestamp_ns,
        comm,
        event_type: 0, // connect
    };

    if let Some(mut entry) = EVENTS.reserve::<NetEvent>(0) {
        entry.write(event);
        entry.submit(0);
    }

    Ok(())
}

The sockaddr_in structure has a fixed layout defined by POSIX. The port is at byte offset 2 and the address at byte offset 4. Both are in network byte order (big-endian), so the port is converted with u16::from_be() for display. The IP address stays in network byte order and is converted in userspace.

Note

Source port at connect time When tcp_v4_connect is called, the kernel has not yet assigned a source port; that happens later in the connection setup. The source port and address fields are zero in the connect event. If you need the source port, you would probe tcp_v4_connect with a kretprobe and read the socket after the connection is established.

Inbound connections: inet_csk_accept

// tuxscope-ebpf/src/net.rs (continued)

#[kretprobe]
pub fn trace_tcp_accept(ctx: RetProbeContext) -> u32 {
    match try_tcp_accept(&ctx) {
        Ok(()) => 0,
        Err(_) => 1,
    }
}

fn try_tcp_accept(ctx: &RetProbeContext) -> Result<(), i64> {
    // The return value is a struct sock * for the new connection
    let sock_ptr: *const u8 = ctx.ret().ok_or(1i64)?;

    // A null return means accept failed
    if sock_ptr.is_null() {
        return Ok(());
    }

    let pid = (bpf_get_current_pid_tgid() >> 32) as u32;
    let timestamp_ns = unsafe { bpf_ktime_get_ns() };
    let comm = bpf_get_current_comm().map_err(|e| e as i64)?;

    // Read fields from struct sock → __sk_common
    // These offsets come from the kernel's sock_common structure.
    // They can vary between kernel versions — use BTF for production tools.

    // skc_daddr: remote IPv4 address (the client for an accept event)
    let remote_addr: u32 = unsafe {
        bpf_probe_read_kernel(sock_ptr.add(4) as *const u32)?
    };
    // skc_rcv_saddr: local IPv4 address (the listening server)
    let local_addr: u32 = unsafe {
        bpf_probe_read_kernel(sock_ptr.add(8) as *const u32)?
    };
    // skc_dport: remote port (network byte order)
    let remote_port_be: u16 = unsafe {
        bpf_probe_read_kernel(sock_ptr.add(12) as *const u16)?
    };
    // skc_num: local port (host byte order)
    let local_port: u16 = unsafe {
        bpf_probe_read_kernel(sock_ptr.add(14) as *const u16)?
    };

    let event = NetEvent {
        pid,
        sport: u16::from_be(remote_port_be),
        dport: local_port,
        saddr: remote_addr,
        daddr: local_addr,
        timestamp_ns,
        comm,
        event_type: 1, // accept
    };

    if let Some(mut entry) = EVENTS.reserve::<NetEvent>(0) {
        entry.write(event);
        entry.submit(0);
    }

    Ok(())
}

Warning

Struct offsets and kernel versions The offsets into struct sock (4, 8, 12, 14) are specific to one kernel build and CONFIG_* combination. Tuxscope hardcodes them so the program stays small and the moving parts stay visible — this lab does not actually use BTF or CO-RE, despite the recommendation. On any other kernel version, distribution kernel, or CONFIG_INET_DIAG/CONFIG_IPV6 variation, the offsets will be wrong and the addresses you see will be garbage.

Confirm against your kernel before running for real:

pahole -C sock_common /sys/kernel/btf/vmlinux | head -25
# Look for the offset numbers next to skc_daddr, skc_rcv_saddr, skc_dport, skc_num.

Production eBPF tools (libbpf-tools, bpftrace, Cilium) use BTF + CO-RE relocations so the verifier patches the correct offsets in at load time. We cover that pattern in the BTF lab later in the series; until then, treat the addresses you read here as “trust but verify against ss -tnp output”.

Key differences from the connect probe:

  1. #[kretprobe] fires on function return instead of entry. The RetProbeContext provides access to the return value.
  2. ctx.ret() returns the function’s return value, here, a pointer to the new struct sock.
  3. Null check, if accept failed, the return value is null. We skip these.
  4. Full socket information, unlike the connect probe, the accept probe has the complete connection tuple because the TCP handshake is already finished.

The userspace formatter

Userspace converts the raw 32-bit addresses to human-readable dotted-quad notation:

fn format_ipv4(addr: u32) -> String {
    let bytes = addr.to_be_bytes();
    format!("{}.{}.{}.{}", bytes[0], bytes[1], bytes[2], bytes[3])
}

The address is stored in network byte order (big-endian), so to_be_bytes() extracts the octets in the correct order. If you used to_ne_bytes() on a little-endian machine, the octets would be reversed.

Running it

sudo tuxscope net

In another terminal, make some connections:

curl -s https://example.com > /dev/null
curl -s https://httpbin.org/get > /dev/null

The tuxscope output shows:

PID      COMM             TYPE     SRC                 DST
6201     curl             connect  0.0.0.0:0           93.184.216.34:443
6205     curl             connect  0.0.0.0:0           34.227.213.82:443

The source is 0.0.0.0:0 because the kernel has not assigned the source address and port at the time tcp_v4_connect runs. The destination address and port are populated from the sockaddr_in passed by the process.

If you are running a web server, you will also see inbound connections:

PID      COMM             TYPE     SRC                 DST
8401     nginx            accept   203.0.113.42:51234  192.168.1.100:8080
8401     nginx            accept   203.0.113.42:51236  192.168.1.100:8080
8401     nginx            accept   198.51.100.7:49821  192.168.1.100:8080

For accept events, SRC is the remote client and DST is the local server. The source port is the client’s ephemeral port, and the destination port is the listening port (8080). Keeping that convention consistent makes SRC and DST mean the same thing for both connect and accept events.

Filter by PID:

sudo tuxscope net --pid 6201

JSON output:

sudo tuxscope net --format json
{"pid":6201,"event_type":"connect","sport":0,"dport":443,"saddr":"0.0.0.0","daddr":"93.184.216.34","comm":"curl","timestamp_ns":9825200293847}
{"pid":8401,"event_type":"accept","sport":51234,"dport":8080,"saddr":"203.0.113.42","daddr":"192.168.1.100","comm":"nginx","timestamp_ns":9825200312054}

Interpreting network events

Common patterns to look for:

DNS-over-TLS and TCP fallback appear as connect events to port 853 or, less commonly, TCP port 53. Ordinary DNS is usually UDP, so it will not appear in this lab’s output. If you want to trace conventional DNS lookups, add a UDP probe as in Exercise 3.

HTTPS traffic shows up as connect events to port 443. You can see which processes are making outbound HTTPS connections and to which IP addresses. You cannot see the hostname from the IP alone (many sites share IPs), but combining this with DNS traces gives a complete picture.

Service mesh / sidecar patterns appear as many connections to 127.0.0.1, the application talks to a local proxy (Envoy, Linkerd, etc.) which makes the actual outbound connection.

Port scanning looks like rapid connect events to many different ports on the same destination address. If you see this from an unexpected process, investigate.

Note

IPv6 connections This lab only traces IPv4 connections. IPv6 uses tcp_v6_connect and a different socket structure. Adding IPv6 support is a natural extension: the event struct would need 128-bit address fields, and you would add kprobes on the v6 variants of the same functions.

Exercises

  1. Add IPv6 support. Probe tcp_v6_connect and extend the NetEvent struct with 128-bit address fields. Use a union or a separate event type. Display IPv6 addresses in standard notation.

  2. Track connection duration. Use a BPF HashMap to store the timestamp when a connection is established (from the connect or accept event). Probe tcp_close to capture when the connection ends. Compute the duration in userspace. This is the foundation of connection-level latency monitoring.

  3. Correlate with DNS. Probe the udp_sendmsg function to capture outbound DNS queries (port 53). Parse the DNS query to extract the hostname. When a connect event arrives, look up the destination IP in the DNS cache to show the hostname alongside the IP address.

  4. Build a connection summary. After running for a period, print a summary showing the top destination IPs and ports by connection count. Group connections by process name. This gives a high-level view of which processes are talking to which endpoints, useful for auditing and security monitoring.

Progress so far

Over these four labs you have built the core of a kernel observability toolkit:

  • Lab 1, established the eBPF pipeline from kernel to userspace with PerfEventArray
  • Lab 2, traced all syscalls with RingBuf and resolved syscall IDs to names
  • Lab 3, probed the VFS layer with kprobes to measure file I/O volume
  • Lab 4, monitored TCP connections with kprobes and kretprobes, reading kernel data structures safely

Each lab introduced a new eBPF technique: tracepoints, RingBuf, kprobes, kretprobes, and bpf_probe_read_kernel. Together they cover the core patterns used by production observability tools like bpftrace, tcpconnect, biotop, and execsnoop.

What’s next

In Lab 5: Tracing Process Lifecycle, you will attach to three tracepoints simultaneously to trace fork, exec, and exit events in real time and understand how Linux creates, transforms, and destroys processes.