ret2libc: reusing the library against NX
NX prevents executing bytes on the stack. ret2libc sidesteps it: why inject code when system is already mapped in libc?
Building the call
Overwrite the return address with the address of system and set up the argument. On x86-64 the argument goes in rdi, so you need a pop rdi; ret gadget pointing at "/bin/sh" (a string present in libc):
[ pop rdi; ret ]
[ addr "/bin/sh" ]
[ addr system ]On x86 (32-bit) it is even simpler: arguments are on the stack, so [system][ret][arg].
Defence
ASLR randomizes the libc base: you need a leak to compute it. Full RELRO, PIE, and stack canaries raise the wall. ret2libc remains the base case from which you move to full ROP when more than one call is needed.