Allocator: Add INT3 codecave scanner

[praydog]

This adds a fallback "allocation" method to the allocator which scans forward for the nearest sequence of INT3 instructions that is >= 10 bytes and >= size of allocation requested. I chose 10 because it's unlikely that this is going to belong to some other kind of code or data in an executable region.

This allows a few things:

1. It reduces the memory footprint of the hook to an E9 patch (5 bytes) if an FF 25 JMP was originally going to be used
2. It uses existing nearby executable memory (the INT3 sequence) as the entire trampoline code
3. This bypasses the FF 25 RIP relative instruction failures entirely if it initially succeeds in the E9 hook function because of point 2

There are some improvements that could be made:

1. The INT3 scan could scan backwards as well as forwards
2. The INT3 scan could also look at other nearby regions of memory rather than the ones in the desired_addresses list This is probably not good because it could be destroyed randomly
3. This could possibly be an optional flag passed to the Allocator to turn this feature on

The reason why this is needed is mainly for large binaries (like Denuvo protected binaries) consuming a large amount of space, as well as the heap allocations that follow that take up 2GB of space around the target binary. This causes random failures if there is a RIP relative instruction in the path of the hook patch.

An alternative method of fixing this is to use an assembler to construct larger variants of the RIP relative instructions. I could see this alternative being necessary for 1 byte short JMPs or other similar instructions in the line of fire.

[ethanporcaro]

It'd be really cool if this could get merged. I tried playing around with it in my own repo, but I couldn't really figure it out. The application I'm hooking stuffs function calls in a ton of int3 padding, so this would be great to use. Let me know how I can help, and I'll try to.

[praydog]

It'd be really cool if this could get merged. I tried playing around with it in my own repo, but I couldn't really figure it out. The application I'm hooking stuffs function calls in a ton of int3 padding, so this would be great to use. Let me know how I can help, and I'll try to.

My fork is used in UEVR if you need to figure out how to pull it. Cursey has made a lot of changes at this point so I may need to redesign it to get it merged.

The usage is not any different than how safetyhook is used normally. It's not optional.

[ethanporcaro]

Cool, thanks. I think the new changes were helping with #52 but I wasn't really able to test it without running into out-of-range issues

…

On Fri, Feb 16, 2024, 3:08 PM praydog ***@***.***> wrote: It'd be really cool if this could get merged. I tried playing around with it in my own repo, but I couldn't really figure it out. The application I'm hooking stuffs function calls in a ton of int3 padding, so this would be great to use. Let me know how I can help, and I'll try to. My fork is used in UEVR if you need to figure out how to pull it. Cursey has made a lot of changes at this point so I may need to redesign it to get it merged. The usage is not any different than how safetyhook is used normally. It's not optional. — Reply to this email directly, view it on GitHub <#45 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKHIPCDLPVADHL4X5ZIV7ILYT7KHBAVCNFSM6AAAAAA7L3PJDWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNBZGQYDINBSGE> . You are receiving this because you commented.Message ID: ***@***.***>

[cursey]

I recently added InlineHook::Flags and MidHook::Flags as a way to opt-into features that may not always be desirable. If this PR is updated to make use of those or a similar system I'd be happy to merge it in.