I recently researched anti-debugging techniques for VMP 3.8 and 3.9 versions. Although there are minor differences, they are largely the same, and the methods used to bypass them are consistent. Here is a general overview of the process for version 3.9.

Higher versions of VMP use the rdtsc instruction to make system calls along random paths. I used Unicorn to simulate execution. After encountering an rdtsc instruction, I set a breakpoint on the next instruction, then ran the debugger until it hit the breakpoint, and continued simulation until encountering a system call or API call. Then I checked the logs.

As you can see, the simulation stops when it encounters a system call. At this point, the debugger runs until the breakpoint, and then uses the breakpoint callback function to remove the hardware breakpoint. This allows interception of the system call. This is the general process.

Next, we will analyze the anti-debugging methods.

First, let’s talk about the simple PEB check. The debugger has an advanced option: hide debugger. This allows tracing by clearing the debug flag and heap flag.

It is recommended to set a breakpoint on ntclose (avoid setting breakpoints in headers or using hardware breakpoints). Hiding the debugger and placing a breakpoint on ntclose can prevent interruptions from certain API calls that do not affect the debugger. Of course, if a popup appears immediately after setting the breakpoint on ntclose, you should trace it.

As shown above, the system call’s rax indicates the function call number. Here it corresponds to NtSetInformationProcess, and rdx is 0x28. I asked AI, and it said this setting is used to prevent system call tracing. If a callback function is set, the return point from ring 3 after the system call will not be the next instruction, but the callback function you defined. I recommend reproducing this yourself; I already did, so I won’t go into details.

Next, NtOpenFile is called to detect the \??\TitanHide driver. Since this system does not have the driver, there’s no need to worry. If it does exist, you can try setting it to C0000034 to bypass it.

At this point, rax is 0x33, and the string \??\TitanHide can be found via r8.

Then, in the third step, NtQueryInformationProcess queries DebugPort. You can bypass this by modifying the output parameter from -1 (0xffffffffffffffff) to 0.

You can see the value 0xffffffffffffffff—set it to 0.

Next is the fourth step: NtQueryInformationProcess queries DebugObjectHandle. Set rax to C0000353 to bypass it.

After that, some APIs are called and breakpoints are set on RtlFreeHeap. Simulation was not necessary here (this applies only to this program; I simulated a few times just to save time. Other programs may still require simulation). You can try placing breakpoints on RtlFreeHeap; if a breakpoint is detected, simulate again. Try multiple times.

Now look at the fifth anti-debugging point: ZwSetInformationThread hidefromdebugger.

First, set rdx to 0, then let the shellcode perform self-validation. Since this step takes a long time, it’s best not to simulate it. Instead, place a persistent breakpoint on the first byte of the section containing the shellcode.

The program will stop here. At this point, locate the next instruction, which is a jne. Some programs may use obfuscation like call jne or jmp, but that doesn’t matter. You must find the correct jne. This jne will definitely jump to movzx esi, byte ptr ds:[rcx] or one or two instructions above it.

Set a hardware breakpoint on the instruction following the jne, which in the diagram is mov eax, 0x88BAC81D. The program will stop here multiple times (the exact number varies by program). To determine how many times, keep pressing F9 until the debugger window appears. Remember this count, then restart. For this program, stopping 5 times is enough before continuing simulation. (You can continue simulation without interruption if you want, but it takes longer—this is just a time-saving trick.) It’s more like a trial-and-error method I discovered rather than a formal optimization.

Next, the process involves checking four points:

• Querying DebugPort via NtQueryInformationProcess

• Setting ZwSetInformationThread hidefromdebugger to 0

• Setting rdx to 0 to bypass NtClose

• Closing an invalid handle by setting rax to C0000008

(In version 3.8, CloseHandle requires rax to be set to 0.) This triggers a single-step exception detection interrupt.

The first two steps are the same as explained earlier and won’t be repeated.

After ZwSetInformationThread hidefromdebugger finishes execution:

• Version 3.9 stops directly at ntclose

• Version 3.8 stops directly at closehandle

  

Set rax to C0000008 and change rip to ret. Then remove all hardware breakpoints, press F9, and the program will run.

Finally, the simulated execution code, plugin, and demo program are provided. If you can successfully run the program, congratulations! Overcoming VMP anti-debugging mechanisms is now very easy.

The X64DBG plugin code is below. You can compile it with C++ or further develop it.

This is the hidden content, please

• Sign In
• or
• Sign Up

Some errors in the code have been fixed.

Thank you

thx

thhxx