# self-overwritten REP STOS/MOVS, IDA-Pro 5.4 and Ko

once upon a time was MS-DOS and ancient debuggers like Turbo-Debugger, Soft-Ice and many others. and there were anti-debug tricks. one of them was based on self-overwritten REP STOS/MOVS instruction. it worked great against all existing debuggers, including CUP 386 (exe unpacker with build-in CPU emulator).

I used this tricks for years. I would almost forget about if Silviocesare not posted “Anti-debugging prefetch tricks and single stepping through a rep stos/movs” article on his blog (very nice blog, btw).

I was interested: what’s about modern debuggers? what’s about emulators like BOCHS? what’s about IDA-Pro 5.4 with BOCHS-based debugger? imagine, how surprised I was when I found out that IDA-Pro 5.3, Olly 1.10 and Soft-Ice not only can be detected this way, but also lost the control during step over tracing! debugged code just escapes out of the debugger!!! IDA-Pro 5.4 with BOCHS module fails to emulate the self-overwritten REP STOS/MOVS instruction (so it can be detected as well) and lost control on Step Over tracing. only Olly 2.00i recognizes attempts to espace and blocks them, however it can be detected the same way.

for testing reasons I wrote a simple program with self-overwritten REP STOSB command (see source bellow).

xor ebx, ebx
mov al, 43h ; // INC EBX
mov edi, offset end_of
mov ecx, 6
end_of: NOP

download it, load into IDA-Pro 5.3 and start tracing REP STOSB instruction (F7 hot key). what do we see? REP STOSB changes NOP to INC EBX, overwrites four commands and overwrites itself (STOSB). since, during tracing CPU generates a single step exception every iteration, REP STOSB becomes REP INC EBX and as far as we all know, REP works only with string commands, so REP INC EBX is not executed and REP loop finishes with ECX = 1.

now, run the program without tracing. CPU pipelines REP STOSB and executes it until ECX > 0. REP STOSB modifies only data cache, while the instruction is executed on the pipeline and CPU does not recognizes modification of the code, so REP loop finishes with ECX = 0.

Olly 1.10/200i and Soft-Ice also fails to trace self-overwritten REP STOSB instruction. of course, if we trace the program with our hands, it’s easy to set a breakpoint _after_ it and run the code without tracing, but!!! many plug-ins use trace engine for their needs, so the trace engine should work fine and it’s possible to fix debuggers – just before executing REP STOS/MOVS we have to perform some checks and if the command overwrites itself we either set a breakpoint either emulate CPU behavior.

ok, run the program under BOCHS (IDA-Pro 5.4 support a special plug-in, allowing us to debug code on the fly). regardless of whether we trace program or not, the REP loop finishes with ECX = 1. well-known x86-emu plug-in gives us the same result, and this result is definitely wrong.

by the way, did I hear a question: how long CPU is executing overwritten REP STOS/MOVS command? there is no universal answer. it depends on CPU internal behavior and external evens like interrupts. when an interrupt is generated, CPU stops executing overwritten REP STOS/MOVS. a good way to create a pseudo-random generator! I’m going to write about it in the next post.

meanwhile, some CPU have a bug. they executes CLD commands _before_ overwritten REP STOS/MOVS will be stopped or finishes. as result, REP STOS/MOVS changes the direction and hits the memory not supposed to be written. I’m investigating this case now, will publish the result soon.

well, let’s return to our muttons. load the program into IDA-Pro 5.3/5.4 and perform Step Over tracing. move cursor to REP STOSB, press F8 and… the debugger lost the control!!! why? the answer is: to gain control back after REP STOSB command IDA-Pro sets a software breakpoint on the next command. we all know that a software breakpoint it’s just INT 03 (CCh) instruction and in our case this instruction is overwritten by REP STOSB. thus the breakpoint is wiped out and the process is executed until another breakpoint will be triggered. if there is no other breakpoints – the debugged code escapes out of the debugger!

what’s about Olly 1.10 and Soft-Ice?! fast check shows us like they do not lost the control and stop after REP STOSB. but… how they do it? well, they just use hardware breakpoints!!! and if all four hardware breakpoints are in use, the debuggers set a software breakpoint like IDA-Pro does, so they lost control as well. not good!

Olly 2.00i (didn’t check other versions) is the only debugger who is able to detect that the breakpoint was wiped out. if it happens we have a warning message. impressive! Olly 2.00 is a great debugger doubtless!!!



  1. The Prefetch Input Queue bug with REP STOS no longer exists.

    This code runs fine on CPUs before Intel i7, but on i7, it decodes rep stosb when it is overwritten, resulting in an exception (Because rep stosb becomes 00AA ADD BYTE PTR DS:[EDX+909090B8],CH).

    ; Clear initializing code

    mov edi,offset Initialize

    mov ecx,offset EndInitialize – offset Initialize

    xor eax,eax

    rep stosb
    ; Dies here if CPU=i7, or single stepping, or emulating

    mov eax,90909090h




  2. thanks for the comment, it’s interesting, but I have a pretty different situation. the following code sometimes (1/10 times) causes access violation, tries to write memory at 141FFFFh.

    0141FFE2 FD STD
    0141FFE3 B0 90 MOV AL,90
    0141FFE5 B9 00000001 MOV ECX,1000000
    0141FFEA E8 00000000 CALL 0141FFEF
    0141FFEF 5F POP EDI
    0141FFF0 83C7 06 ADD EDI,6
    0141FFF5 90 NOP
    0141FFF6 FC CLD
    0141FFF7 8BC1 MOV EAX,ECX
    0141FFF9 5F POP EDI
    0141FFFA 5E POP ESI
    0141FFFB 5B POP EBX
    0141FFFC 5D POP EBP
    0141FFFD C2 0400 RETN 4

  3. The forward direction attack is also in my Anti-Unpacker paper. :-)
    The Obsidium packer uses it to detect debuggers. It is a neat trick.
    However, Windows does not like the direction flag to be set. It assumes in many places that it is clear, so the behaviour can be quite unexpected.

Leave a comment

Comments are closed.