Buffer overflows work because of how the stack is organized. The return address — the value that tells the CPU where to go when a function finishes — sits right next to local variables in memory. If a function copies too much data into a local buffer without checking the size, the overflow spills past the buffer and overwrites the return address.
Understanding why that layout exists, and exactly how functions use the stack, turns buffer overflow exploitation from a magic trick into a predictable mechanical process.
Process Memory Layout
When a Linux program runs, the kernel maps its memory into distinct regions:
High addresses
┌───────────────────────┐
│ Stack │ ← grows downward
│ ↓ │
│ │
│ ↑ │
│ Heap │ ← grows upward
├───────────────────────┤
│ BSS │ ← uninitialized global variables
├───────────────────────┤
│ Data │ ← initialized global variables
├───────────────────────┤
│ Text │ ← executable code (read-only)
├───────────────────────┤
Low addresses- Text: The compiled machine code. Marked read-only and executable. This is where your gadgets live.
- Data / BSS: Global and static variables. Data holds initialized values; BSS holds variables declared but not yet assigned.
- Heap: Dynamically allocated memory (
malloc,new). Grows upward toward higher addresses. - Stack: Function-local data — arguments, local variables, saved registers, return addresses. Grows downward toward lower addresses.
The stack growing downward is key to understanding overflows. When you write past the end of a local buffer, you write toward higher addresses — toward the saved base pointer and return address that sit above your buffer in the stack frame.
Viewing the Layout in GDB
gdb-peda$ vmmap
Start End Perm Name
0x00400000 0x00401000 r-xp ./binary (Text)
0x00600000 0x00601000 rw-p ./binary (Data/BSS)
0x00007ffff7a0d000 0x00007ffff7bcd000 r-xp libc-2.31.so
0x00007ffffffde000 0x00007ffffffff000 rwxp [stack]The stack’s rwx permissions (when NX is disabled) are what allow shellcode execution on the stack. With NX enabled, the stack is rw- — writable but not executable.
Stack Fundamentals
Growth Direction
The stack grows from high addresses to low addresses. When you push a value, ESP/RSP decreases. When you pop, it increases.
This is counterintuitive: “growing” the stack means moving the pointer to a smaller number.
PUSH and POP
push stores a value on the stack and moves the stack pointer:
; x86
push eax ; ESP = ESP - 4, then store EAX at [ESP]
; x64
push rax ; RSP = RSP - 8, then store RAX at [RSP]pop does the reverse:
; x86
pop eax ; Load [ESP] into EAX, then ESP = ESP + 4
; x64
pop rax ; Load [RSP] into RAX, then RSP = RSP + 8The size difference matters: x86 pushes and pops 4 bytes at a time, x64 pushes and pops 8 bytes.
The Stack Pointer
ESP (x86) or RSP (x64) always points to the top of the stack — the most recently pushed value. Every push, pop, call, and ret modifies this register.
Function Calls and the Stack
What CALL Does
The call instruction does two things:
- Pushes the address of the next instruction (the return address) onto the stack
- Jumps to the target function
call 0x401234 ; push RIP+5 onto stack, then jump to 0x401234After call, the stack looks like:
RSP → [ return address ] ← address of instruction after the CALLWhat RET Does
The ret instruction is the inverse:
- Pops the top of the stack into RIP/EIP
- Execution continues at that address
ret ; pop [RSP] into RIP, RSP = RSP + 8This is the fundamental mechanism that buffer overflows exploit. If you overwrite the value that ret will pop into RIP, you control where execution goes.
Function Prologues and Epilogues
Most compiled functions follow a standard pattern for setting up and tearing down their stack frame.
The Prologue
push rbp ; Save the caller's base pointer
mov rbp, rsp ; Set up this function's base pointer
sub rsp, 0x40 ; Allocate space for local variables (64 bytes here)After the prologue:
┌─────────────────────┐
│ return address │ ← pushed by CALL
├─────────────────────┤
RBP → │ saved RBP │ ← pushed by prologue
├─────────────────────┤
│ │
│ local variables │ ← allocated by SUB RSP
│ │
├─────────────────────┤
RSP → │ (top of stack) │
└─────────────────────┘RBP serves as a stable reference point. Local variables are accessed as negative offsets from RBP ([rbp-0x10], [rbp-0x20]), while function arguments (on x86) and the return address are at positive offsets.
The Epilogue
leave ; Equivalent to: mov rsp, rbp; pop rbp
ret ; Pop return address into RIPleave reverses the prologue: it restores RSP to where RBP points (discarding local variables), then pops the saved RBP. After leave, RSP points to the return address, and ret pops it into RIP.
Frame Pointer Omission
Compilers sometimes omit the frame pointer (-fomit-frame-pointer) to free up RBP as a general-purpose register. When this happens, functions access local variables as offsets from RSP directly, and the prologue/epilogue look different:
; Prologue without frame pointer
sub rsp, 0x48 ; Allocate locals (no push rbp / mov rbp, rsp)
; Epilogue without frame pointer
add rsp, 0x48 ; Deallocate locals (no leave)
retThis matters for exploitation because there’s no saved RBP to overwrite — the return address sits directly above the local variables.
Stack Frame Anatomy
Here’s a complete stack frame for a function called with two arguments on x86:
High addresses (toward caller)
┌─────────────────────────┐
│ arg 2 │ [EBP+0x0C]
├─────────────────────────┤
│ arg 1 │ [EBP+0x08]
├─────────────────────────┤
│ return address │ [EBP+0x04] ← pushed by CALL
├─────────────────────────┤
│ saved EBP │ [EBP+0x00] ← pushed by prologue
├─────────────────────────┤
│ local var 1 │ [EBP-0x04]
├─────────────────────────┤
│ local var 2 │ [EBP-0x08]
├─────────────────────────┤
│ local buffer[64] │ [EBP-0x48]
├─────────────────────────┤
ESP → (top of stack)
Low addressesOn x64, the layout is similar but arguments 1–6 are in registers (RDI, RSI, RDX, RCX, R8, R9) rather than on the stack:
High addresses
┌─────────────────────────┐
│ return address │ [RBP+0x08] ← pushed by CALL
├─────────────────────────┤
│ saved RBP │ [RBP+0x00] ← pushed by prologue
├─────────────────────────┤
│ local var 1 │ [RBP-0x08]
├─────────────────────────┤
│ local buffer[64] │ [RBP-0x48]
├─────────────────────────┤
RSP → (top of stack)
Low addressesHow a Buffer Overflow Works
Consider this vulnerable C function:
void vulnerable(char *input) {
char buffer[64];
strcpy(buffer, input); // No bounds checking
}The compiler allocates buffer on the stack. The stack frame looks like:
┌─────────────────────────┐
│ return address │ ← where execution goes after RET
├─────────────────────────┤
│ saved RBP │
├─────────────────────────┤
│ │
│ buffer[64] │ ← strcpy writes here
│ │
├─────────────────────────┤
RSP → (top of stack)strcpy copies bytes starting at the bottom of buffer and moves upward (toward higher addresses). If the input is longer than 64 bytes, the copy overflows past the buffer and overwrites:
- Saved RBP (next 4 or 8 bytes)
- Return address (next 4 or 8 bytes after that)
Normal After overflow
┌────────────────┐ ┌────────────────┐
│ return address │ │ 0x41414141 │ ← overwritten!
├────────────────┤ ├────────────────┤
│ saved RBP │ │ 0x41414141 │ ← overwritten
├────────────────┤ ├────────────────┤
│ buffer (64B) │ │ AAAAAAAAAA... │ ← input data
└────────────────┘ └────────────────┘When vulnerable executes ret, it pops 0x41414141 into EIP/RIP. The CPU tries to execute code at that address — and since we chose the value, we control where execution goes.
Finding the Offset in Practice
The gap between the start of the buffer and the return address depends on the compiler’s layout decisions. You find it empirically:
Step 1: Generate a Unique Pattern
gdb-peda$ pattern create 200This produces a string where every 4-byte (or 8-byte) substring is unique.
Step 2: Crash the Program
gdb-peda$ run <<< 'AAA%AAsAABAA$AAnAACAA...'Step 3: Find the Offset
After the crash, EIP/RIP contains a fragment of the pattern:
gdb-peda$ pattern offset 0x41416d41The reported offset is the exact number of bytes from the start of your input to the return address.
Step 4: Verify
payload = b"A" * offset + b"BBBB"If EIP shows 0x42424242, you have precise control.
Examining the Stack in GDB
View the Stack as Words
gdb-peda$ x/20wx $esp # x86: 20 words (4-byte) from ESP
gdb-peda$ x/20gx $rsp # x64: 20 giant words (8-byte) from RSPView the Current Frame
gdb-peda$ info frame
Stack level 0, frame at 0x7fffffffe4b0:
rip = 0x401156 in vulnerable; saved rip = 0x401189
Arglist at 0x7fffffffe4a0, args:
Locals at 0x7fffffffe4a0The saved rip value is the return address — the one you’re trying to overwrite.
Walk the Call Stack
gdb-peda$ bt
#0 vulnerable () at vuln.c:4
#1 0x0000000000401189 in main () at vuln.c:10View Stack Contents Around RBP
gdb-peda$ x/4gx $rbp
0x7fffffffe4a0: 0x00007fffffffe4c0 0x0000000000401189
saved RBP return addressKey Takeaways
- The stack grows downward (toward lower addresses). Buffers fill upward. This directional mismatch is why overflows reach the return address.
callpushes the return address;retpops it into EIP/RIP. Overwriting that stored return address redirects execution.- Function prologues save the caller’s base pointer and allocate space for locals. Epilogues reverse this with
leave; ret. - The exact distance from your buffer to the return address depends on the compiler and optimization flags — always find it empirically with pattern generation.
- On x86, function arguments sit above the return address on the stack. On x64, the first six arguments are in registers, so the stack frame is simpler.
- Use
vmmapto check memory permissions andx/gx $rspto inspect stack contents during debugging.