months ago Bow Sineath (a very clever reverser!) asked me: “does JL [jump is less] instruction check ZF flag?” I said: “well, give me a second to think, well, it’s supposed to check it, otherwise it would act like JLE [jump if less or equal] and besides, JL is synonym of JNGE (jump if not great or equal), so JL should check ZF!“.
but, according to Intel’ manuals JL and JNGE check only if SF != OF. CMOVL/CMOVNGE work the same way. at that time I thought that it’s just a documentation bug and even pointed this out in my presentation on HITB 2008 conference.
but I was wrong!!! I have checked it and found out that JL/JNGE does not check ZF flag!!! to do this I wrote extremely simple POC (if you’re too lazy to type, download source and binary):
mov eax, 002C2h ; S = 1, O = 0, Z = 1
jl jump_is_taken ; ==>
mov p, offset noo
mov eax, 2C2h/push eax/popfd set SF with ZF and clear OF. so, SF != OF, but ZF is set. what CPU is going to do? easy to check with Olly! just load the program and start tracing. ops!!! JL is taken!!! JL ignores ZF!!! x86emu (plug-in for IDA-Pro) acts the same. didn’t check other emulators yet.
well, it’s interesting. why JL (and similar commands) ignores ZF?! guess, normal CPU command (like TEST/CMP/XOR/etc) never set ZF if result is less, so JL just ignores it. but… if we set flags manually or use other tricks… it becomes a real trap!!! consider the listing above and ask your co-worker: is the jump taken or not? I’m sure, some of them will answer: of course, the jump is not going to be taken! a good anti-reversing trick!!! I just wonder – how software is still working on buggy hardware.