# The Stack: Memory Layout and Function Frames > How the stack works in x86 and x64 Linux: memory layout, push/pop mechanics, function prologues and epilogues, and why overflows overwrite the return address. - Published: 2025-08-08 - Difficulty: beginner - Series: Linux Exploitation Fundamentals - Tags: Linux, Assembly, x86, x64, Internals - Source: https://stevenfoerster.com/tutorials/stack-memory-layout-and-function-frames/ ## Prerequisites - Basic understanding of x86/x64 registers - Basic Linux command line knowledge Buffer overflows work because of how the stack is organized. The return address, the value that tells the CPU where to go when a function finishes, sits right next to local variables in memory. If a function copies too much data into a local buffer without checking the size, the overflow spills past the buffer and overwrites the return address. Understanding _why_ that layout exists, and exactly how functions use the stack, turns buffer overflow exploitation from a magic trick into a predictable mechanical process. ## Process Memory Layout When a Linux program runs, the kernel maps its memory into distinct regions: ```text High addresses ┌───────────────────────┐ │ Stack │ ← grows downward │ ↓ │ │ │ │ ↑ │ │ Heap │ ← grows upward ├───────────────────────┤ │ BSS │ ← uninitialized global variables ├───────────────────────┤ │ Data │ ← initialized global variables ├───────────────────────┤ │ Text │ ← executable code (read-only) ├───────────────────────┤ Low addresses ``` - **Text**: The compiled machine code. Marked read-only and executable. This is where your gadgets live. - **Data / BSS**: Global and static variables. Data holds initialized values; BSS holds variables declared but not yet assigned. - **Heap**: Dynamically allocated memory (`malloc`, `new`). Grows upward toward higher addresses. - **Stack**: Function-local data, arguments, local variables, saved registers, return addresses. Grows **downward** toward lower addresses. The stack growing downward is key to understanding overflows. When you write past the end of a local buffer, you write _toward higher addresses_, toward the saved base pointer and return address that sit above your buffer in the stack frame. ### Viewing the Layout in GDB ```text gdb-peda$ vmmap Start End Perm Name 0x00400000 0x00401000 r-xp ./binary (Text) 0x00600000 0x00601000 rw-p ./binary (Data/BSS) 0x00007ffff7a0d000 0x00007ffff7bcd000 r-xp libc-2.31.so 0x00007ffffffde000 0x00007ffffffff000 rwxp [stack] ``` The stack's `rwx` permissions (when NX is disabled) are what allow shellcode execution on the stack. With NX enabled, the stack is `rw-`, writable but not executable. > [!note] > NX (No-Execute), also called DEP (Data Execution Prevention) on Windows, is a memory protection that marks regions as non-executable. When NX is enabled, the stack has `rw-p` permissions instead of `rwxp`, preventing shellcode execution. Use `checksec` to check whether a binary has NX enabled. The `-z execstack` GCC flag disables NX at compile time, making the stack executable for testing. ## Stack Fundamentals ### Growth Direction The stack grows from high addresses to low addresses. When you push a value, ESP/RSP **decreases**. When you pop, it **increases**. This is counterintuitive: "growing" the stack means moving the pointer to a _smaller_ number. ### PUSH and POP `push` stores a value on the stack and moves the stack pointer: ```asm ; x86 push eax ; ESP = ESP - 4, then store EAX at [ESP] ; x64 push rax ; RSP = RSP - 8, then store RAX at [RSP] ``` `pop` does the reverse: ```asm ; x86 pop eax ; Load [ESP] into EAX, then ESP = ESP + 4 ; x64 pop rax ; Load [RSP] into RAX, then RSP = RSP + 8 ``` The size difference matters: x86 pushes and pops 4 bytes at a time, x64 pushes and pops 8 bytes. ### The Stack Pointer ESP (x86) or RSP (x64) always points to the **top** of the stack: the most recently pushed value. Every `push`, `pop`, `call`, and `ret` modifies this register. ## Function Calls and the Stack ### What CALL Does The `call` instruction does two things: 1. Pushes the address of the next instruction (the return address) onto the stack 2. Jumps to the target function ```asm call 0x401234 ; push the address of the next instruction onto ; the stack, then jump to 0x401234 ``` The return address pushed by `call` is the address of the instruction immediately following the `call` itself. You may see this written as "RIP+5" in some references because a near `call` with a 32-bit relative offset is encoded as 5 bytes (1-byte opcode + 4-byte operand), but other `call` encodings have different lengths, so "address of the next instruction" is the more precise description. After `call`, the stack looks like: ```text RSP → [ return address ] ← address of instruction after the CALL ``` ### What RET Does The `ret` instruction is the inverse: 1. Pops the top of the stack into RIP/EIP 2. Execution continues at that address ```asm ret ; pop [RSP] into RIP, RSP = RSP + 8 ``` This is the fundamental mechanism that buffer overflows exploit. If you overwrite the value that `ret` will pop into RIP, you control where execution goes. ## Function Prologues and Epilogues Most compiled functions follow a standard pattern for setting up and tearing down their stack frame. ### The Prologue ```asm push rbp ; Save the caller's base pointer mov rbp, rsp ; Set up this function's base pointer sub rsp, 0x40 ; Allocate space for local variables (64 bytes here) ``` After the prologue: ```text ┌─────────────────────┐ │ return address │ ← pushed by CALL ├─────────────────────┤ RBP → │ saved RBP │ ← pushed by prologue ├─────────────────────┤ │ │ │ local variables │ ← allocated by SUB RSP │ │ ├─────────────────────┤ RSP → │ (top of stack) │ └─────────────────────┘ ``` RBP serves as a stable reference point. Local variables are accessed as negative offsets from RBP (`[rbp-0x10]`, `[rbp-0x20]`), while function arguments (on x86) and the return address are at positive offsets. ### The Epilogue ```asm leave ; Equivalent to: mov rsp, rbp; pop rbp ret ; Pop return address into RIP ``` `leave` reverses the prologue: it restores RSP to where RBP points (discarding local variables), then pops the saved RBP. After `leave`, RSP points to the return address, and `ret` pops it into RIP. ### Frame Pointer Omission Compilers sometimes omit the frame pointer (`-fomit-frame-pointer`) to free up RBP as a general-purpose register. When this happens, functions access local variables as offsets from RSP directly, and the prologue/epilogue look different: ```asm ; Prologue without frame pointer sub rsp, 0x48 ; Allocate locals (no push rbp / mov rbp, rsp) ; Epilogue without frame pointer add rsp, 0x48 ; Deallocate locals (no leave) ret ``` This matters for exploitation because there's no saved RBP to overwrite: the return address sits directly above the local variables. ## Stack Frame Anatomy Here's a complete stack frame for a function called with two arguments on x86: ```text High addresses (toward caller) ┌─────────────────────────┐ │ arg 2 │ [EBP+0x0C] ├─────────────────────────┤ │ arg 1 │ [EBP+0x08] ├─────────────────────────┤ │ return address │ [EBP+0x04] ← pushed by CALL ├─────────────────────────┤ │ saved EBP │ [EBP+0x00] ← pushed by prologue ├─────────────────────────┤ │ local var 1 │ [EBP-0x04] ├─────────────────────────┤ │ local var 2 │ [EBP-0x08] ├─────────────────────────┤ │ local buffer[64] │ [EBP-0x48] ├─────────────────────────┤ ESP → (top of stack) Low addresses ``` On x64, the layout is similar but arguments 1–6 are in registers (RDI, RSI, RDX, RCX, R8, R9) rather than on the stack: ```text High addresses ┌─────────────────────────┐ │ return address │ [RBP+0x08] ← pushed by CALL ├─────────────────────────┤ │ saved RBP │ [RBP+0x00] ← pushed by prologue ├─────────────────────────┤ │ local var 1 │ [RBP-0x08] ├─────────────────────────┤ │ local buffer[64] │ [RBP-0x48] ├─────────────────────────┤ RSP → (top of stack) Low addresses ``` > [!note] > On x64, the System V ABI reserves a 128-byte region below RSP called the red zone. Leaf functions (functions that don't call other functions) can use this space without adjusting RSP. This matters for exploitation because shellcode placed just below RSP may land in the red zone and could be overwritten by signal handlers or other interrupts. ## Nested Function Calls on the Stack When functions call other functions, each gets its own stack frame. The frames stack on top of each other, and `ret` unwinds them in reverse order: ```text High addresses ┌─────────────────────────────┐ │ main()'s stack frame │ │ ┌─────────────────────────┐│ │ │ local variables ││ │ │ saved EBP (prev frame) ││ │ │ return address (to _start)│ │ │ arguments ││ │ └─────────────────────────┘│ ├─────────────────────────────┤ │ foo()'s stack frame │ │ ┌─────────────────────────┐│ │ │ local variables ││ │ │ saved EBP (main's EBP) ││ │ │ return address (to main)││ │ │ arguments from main() ││ │ └─────────────────────────┘│ ├─────────────────────────────┤ │ bar()'s stack frame │ │ ┌─────────────────────────┐│ │ │ local variables ││ │ │ saved EBP (foo's EBP) ││ │ │ return address (to foo) ││ │ │ arguments from foo() ││ │ └─────────────────────────┘│ ├─────────────────────────────┤ ESP → (top of stack) Low addresses ``` Each saved EBP forms a linked list; you can trace back through every caller by following the chain of saved base pointers. This is exactly what GDB's `bt` (backtrace) command does. ## How a Buffer Overflow Works Consider this vulnerable C function: ```c void vulnerable(char *input) { char buffer[64]; strcpy(buffer, input); // No bounds checking } ``` The compiler allocates `buffer` on the stack. The stack frame looks like: ```text ┌─────────────────────────┐ │ return address │ ← where execution goes after RET ├─────────────────────────┤ │ saved RBP │ ├─────────────────────────┤ │ │ │ buffer[64] │ ← strcpy writes here │ │ ├─────────────────────────┤ RSP → (top of stack) ``` `strcpy` copies bytes starting at the bottom of `buffer` and moves **upward** (toward higher addresses). If the input is longer than 64 bytes, the copy overflows past the buffer and overwrites: 1. **Saved RBP** (next 4 or 8 bytes) 2. **Return address** (next 4 or 8 bytes after that) ```text Normal After overflow ┌────────────────┐ ┌────────────────┐ │ return address │ │ 0x41414141 │ ← overwritten! ├────────────────┤ ├────────────────┤ │ saved RBP │ │ 0x41414141 │ ← overwritten ├────────────────┤ ├────────────────┤ │ buffer (64B) │ │ AAAAAAAAAA... │ ← input data └────────────────┘ └────────────────┘ ``` When `vulnerable` executes `ret`, it pops `0x41414141` into EIP/RIP. The CPU tries to execute code at that address, and since we chose the value, we control where execution goes. ## Finding the Offset in Practice The gap between the start of the buffer and the return address depends on the compiler's layout decisions. You find it empirically: > [!note] > Compilers may add padding between local variables for alignment. The actual offset from a buffer to the saved return address may differ from what you calculate by looking at declared variable sizes alone. This is why exploit developers use pattern-based offset discovery (e.g., GDB-PEDA's `pattern create` and `pattern offset`) rather than relying on source-level calculations. ### Step 1: Generate a Unique Pattern ```text gdb-peda$ pattern create 200 ``` This produces a string where every 4-byte (or 8-byte) substring is unique. ### Step 2: Crash the Program ```text gdb-peda$ run <<< 'AAA%AAsAABAA$AAnAACAA...' ``` ### Step 3: Find the Offset After the crash, EIP/RIP contains a fragment of the pattern: ```text gdb-peda$ pattern offset 0x41416d41 ``` The reported offset is the exact number of bytes from the start of your input to the return address. ### Step 4: Verify ```python payload = b"A" * offset + b"BBBB" ``` If EIP shows `0x42424242`, you have precise control. ## Examining the Stack in GDB ### View the Stack as Words ```text gdb-peda$ x/20wx $esp # x86: 20 words (4-byte) from ESP gdb-peda$ x/20gx $rsp # x64: 20 giant words (8-byte) from RSP ``` The GDB examine command `x/20wx` means: examine 20 units, each a **w**ord (4 bytes), displayed in he**x**. Other common size specifiers: `g` for giant word (8 bytes, useful for x64 addresses), `b` for single byte, and `h` for halfword (2 bytes). ### View the Current Frame ```text gdb-peda$ info frame Stack level 0, frame at 0x7fffffffe4b0: rip = 0x401156 in vulnerable; saved rip = 0x401189 Arglist at 0x7fffffffe4a0, args: Locals at 0x7fffffffe4a0 ``` The `saved rip` value is the return address: the one you're trying to overwrite. ### Walk the Call Stack ```text gdb-peda$ bt #0 vulnerable () at vuln.c:4 #1 0x0000000000401189 in main () at vuln.c:10 ``` ### View Stack Contents Around RBP ```text gdb-peda$ x/4gx $rbp 0x7fffffffe4a0: 0x00007fffffffe4c0 0x0000000000401189 saved RBP return address ``` ## Next steps With the stack layout and frame mechanics down, [Redirecting Execution to Hidden Functions](https://stevenfoerster.com/tutorials/redirecting-execution-to-hidden-functions/) puts them to work: overwrite that stored return address to redirect execution to a function that was never meant to run.