Automatically find/calculate offsets in binaries that haven't been manually reversed #9
Comments
|
Added a script that can assist in diffing the hex bytes to create patterns for Using that, I also have patterns that seem to uniquely match the arm offsets for 13.3.1, 13.5, and 14.2. It seems the arm patterns need to be a lot longer to make them match uniquely, as they were hitting a lot more duplicates in the binaries. I haven't pushed the patterns I have yet, as I'm wondering if there is an easy way for me to find the minimal length for uniqueness. |
|
Refactored This can be used to optimise the patterns used for finding the offsets to the shortest pattern that still gives a unique match. This should be used against all the test binaries you have, as sometimes a shorter pattern in one breaks the uniqueness for another. In that case, just use the slightly longer one. Repeat until all your test binaries are happy with unique offsets found. Added unoptimised arm patterns in 0xdevalias/poc-re-binsearch@f5471ed Optimised the patterns for x86/arm (for the 13.3.1, 13.5, and 14.2 binaries I had to test against) in 0xdevalias/poc-re-binsearch@5bf9e9e These optimised patterns might need to be adjusted slightly if other binary versions don't work properly with them; but at least for the binaries I have to test against, these seem to work to automagically get the offsets for both x86/arm! 🎉 You can see the existing 'known' offsets here: And can contrast them against the automagically calculated offsets below to ensure correctness:
|
|
I have a mac on I believe 12.7.1 That I am willing to offer as a test case (I use this method and see if it works) if/when this gets implemented. |
|
@rom4ster Are you able to zip up the binary from the following path and upload it somewhere + share a link here as per:
|
Sure https://github.com/rom4ster/IdentityServices/blob/master/identityservicesd%20(1).txt |
@rom4ster Thanks for that :) It looks like my current 'automatically find the offsets' code doesn't fully work on that version of ⇒ sha256sum samples/macos-12.7.1-monterey-identityservicesd
5833338da6350266eda33f5501c5dfc793e0632b52883aa2389c438c02d03718 samples/macos-12.7.1-monterey-identityservicesd⇒ ./find_fat_binary_offsets.py samples/macos-12.7.1-monterey-identityservicesd
-= Universal Binary Sections =-
Architecture 0 (x86_64):
CPU Type: 16777223 (0x1000007)
CPU Subtype: 3 (0x3)
CPU Subtype Capability: 0 (0x0)
Offset: 0x4000 (Valid Mach-O Header: Yes)
Size: 7442432
Align: 14
Architecture 1 (arm64e):
CPU Type: 16777228 (0x100000c)
CPU Subtype: 2 (0x2)
CPU Subtype Capability: 128 (0x80)
Offset: 0x720000 (Valid Mach-O Header: Yes)
Size: 8707856
Align: 14
-= Found Symbol Offsets =-
Offset of _IDSProtoKeyTransparencyTrustedServiceReadFrom in architecture x86_64: 0x0b2278
Offset of _IDSProtoKeyTransparencyTrustedServiceReadFrom in architecture arm64e: 0x0b562c
-= Found Hex Offsets (with pure python fixed sequence search + regex) =-
Architecture 0 (x86_64):
IDSProtoKeyTransparencyTrustedServiceReadFrom: 0xb2278
NACInitAddress: 0x4132e0
NACKeyEstablishmentAddress:
NACSignAddress:
Architecture 1 (arm64e):
IDSProtoKeyTransparencyTrustedServiceReadFrom:
NACInitAddress: 0x5897bc
NACKeyEstablishmentAddress:
NACSignAddress: |
|
Do you know if the specified offsets exist? Like is it just that the symbols are in some other spot, or are the symbols themselves different on 12.7.1 ? I have not had the time to do a manual look with a decompiler (I tried with a hexeditor and it was... not so great). If you do end up finding them manually or something, could you please update the provider or post them here (because might as well if you already did the work and also I kind of want to be able to use that mac.....) |
@rom4ster Manually reversed it (see below), and it looks like all the relevant functions still exist; so it's likely just an issue with the patterns my auto-tool is matching against.
@rom4ster Haha, I can only imagine! That sounds... painful :p
@rom4ster Yeah, absolutely. See below for the offsets I reversed this morning, and I will double check them a bit later + open a PR to have them added to this tool. They should be correct, but I want to be triple sure before I submit a PR with them.
Manually reversing the offsets (assuming I didn't make a mistake):
|
|
@jetfir3 I just saw your PR to add a bunch of other offsets (#12), and was wondering if you'd be able to upload the |
|
Here's a zip: https://oshi.at/vPMG (expires in ~72hrs) Since these are not U2B, no 0x4000 padding calculation will be needed. On 10.14.x and earlier, The patterns below will find all 4 offsets for all of 10.15.x: |
|
The following issue is about |
@jetfir3 Awesome, thanks for that! :) |
|
Added a few more helper scripts + tool/pattern refinements + hex dumps + etc:
For the pattern refinements in those commits, I didn't use the hex dumps/offsets from the # TODO: Need to refactor this script to handle non-fat binaries, and then we can try the following patterns:
# ⇒ ./find_fat_binary_offsets.py --shortest-patterns samples/macos-10.14.6-mojave-x86_64-identityservicesd
# -= Universal Binary Sections =-
# Traceback (most recent call last):
# File "/Users/devalias/dev/0xdevalias/poc-re-binsearch/./find_fat_binary_offsets.py", line 703, in <module>
# main()
# File "/Users/devalias/dev/0xdevalias/poc-re-binsearch/./find_fat_binary_offsets.py", line 120, in main
# print_arch_info(architectures)
# File "/Users/devalias/dev/0xdevalias/poc-re-binsearch/./find_fat_binary_offsets.py", line 289, in print_arch_info
# for i, arch in architectures.items():
# AttributeError: 'str' object has no attribute 'items'
## Patterns from 10.x binaries
# hex_strings = {
# "x86_64": {
# "IDSProtoKeyTransparencyTrustedServiceReadFrom": "554889e54157415641554154534883ec284989f648897dd04c8b2d....4d004d8b45004a8b0c064c8b3d....4d00498b17483b0c160f83..010000488b05....5700488945c0488b05....5700488945c84c8b25....4d004c8b0d....4d00498b0c2441",
# "NACInitAddress": "554889e54157415641554154534881ec481800004989d6488b05......00488b00488945d00f314889d348c1e3204809c348ba8fe3388ee3388ee34889d848f7e248c1ea03488d04d24829c34889d848c1e00448ba7febeb767bf7f7fe4831c24883e307",
# "NACKeyEstablishmentAddress": "554889e54157415641554154534881ec28010000488b05......00488b00488945d04885ff0f94c04885f60f94c108c185d20f94c008c84189d70fb6c08d48064c8d25......0049630c8c488d15e5ffffff4801ca41bd05514f51ffe24656904889f248",
# "NACSignAddress": "554889e54157415641554154534881ec580300004189d14989ff488b05......00488b00488945d0488d95d0fcffff488995d8fcffff89d02548488a9489d381e3109524214189d24181e2a022514a8d3c1289fa81e20000042809c281f20200029489f8",
# },
# "arm64e": {
# "IDSProtoKeyTransparencyTrustedServiceReadFrom": "N/A",
# "NACInitAddress": "N/A",
# "NACKeyEstablishmentAddress": "N/A",
# "NACSignAddress": "N/A",
# },
# }The updated patterns still don't uniquely match for everything, namely, for the
|
|
Opened a PR to add the |
|
|
|
I have extracted the offsets for macos 14.4 beta2. Would it be possible to add them so I can create a new registration code? -= Universal Binary Sections =- -= Found Symbol Offsets =- -= Found Hex Offsets (with pure python fixed sequence search + regex) =- |
|
@TheDave94 You could open a pull request to add them following the similar examples in: See also: |
|
I tried it on my 2017 MacBook Pro running Ventura 13.6.4 and received "No offsets found for 13.6.4/22G513/amd64". From what I've gleaned from this site, it should have worked, no? I ran the python magic script and this was its output, which appears to be in the offsets file: -= Universal Binary Sections =- -= Found Symbol Offsets =- -= Found Hex Offsets (with pure python fixed sequence search + regex) =- |
|
@Marckus That only looks like partial output from my python script (unless it just crashed/exited at that point? Also, what's the sha256sum of the identity service binary? (Which I believe also is included in the full error message) You can then check if that hash is in the offsets file in the code in this repository; that'll confirm whether your version is supported or not. |
|
OK, I'll try that to check. Also, I updated my post as it did crash, I was using the wrong binary. |
It IS in the offsets file, so this is even more puzzling... 2024/03/05 20:53:03 Starting mac-registration-provider 31dc297 |
|
@Marckus Next thing to check is whether the version of this binary you're running is actually the latest one; or at least, one that includes those offsets. Often in situations like this it's because the user is running an old version of this tool. 31dc297 is the commit of the version of the tool you're running; if the offsets file at that code revision doesn't have it; that would be your problem. |
|
Thanks, that was it! A lot of good it'll do me now though, Beeper has apparently pulled iMessage altogether now. I can't even select/add it as a Chat Network any more. :( |
|
Ventura 13.6.5 is out now and I forgot about Beeper and upgraded. If someone wants to put the new lines into offsets.go and rebuild, that would be awesome: // macOS 13.6.5 |
|
You can see more of my methodology for finding the offsets using this tool in the following comments:
|
|
For reference, these are all of the patterns I have in my local copy of the auto-finder tool now: # Symbol to search for
# symbol = "_newLocalDeliveryServiceStatString" # For 10.14.x and earlier
symbol = "_IDSProtoKeyTransparencyTrustedServiceReadFrom" # For 10.15.x onwards
# 10.14.x (and earlier?)
# See also:
# https://github.com/beeper/mac-registration-provider/issues/47#issuecomment-2054451356
# https://github.com/beeper/mac-registration-provider/issues/47#issuecomment-2054839669
# Looking at a bunch of the existing 10.14.x/10.15.x offsets, we can observe that:
#
# - NACKeyEstablishmentAddress - NACSignAddress = 0x14660
# - NACKeyEstablishmentAddress - NACInitAddress = 0xebc0
#
# If we assume that same pattern may hold on this binary, we can attempt to narrow down the likely matching offset:
#
# - NACSignAddress: NACKeyEstablishmentAddress (0x3478a0) - 0x14660 = 0x333240
# - This appears to be one of the offsets listed above for NACSignAddress, so may be what we want.
# - NACInitAddress: NACKeyEstablishmentAddress (0x3478a0) - 0xebc0 = 0x338ce0
# - This appears to be one of the offsets listed above for NACInitAddress, so may be what we want.
# 10.14.x unique match (mostly minified patterns (that still work with 10.15.x))
hex_strings_10_14_x = {
"x86_64": {
f"ReferenceAddress ({symbol})": "554889e54157415641554154534881ec98010000488b05....28",
"NACInitAddress": "554889e54157415641554154534881ec4818",
"NACKeyEstablishmentAddress": "554889e54157415641554154534881ec28010000488b05......00488b00488945d04885",
"NACSignAddress": "554889..4157415641554154534881..580300004189..4989..48..05....",
}
}
# 10.15.x unique match (minified patterns)
hex_strings_10_15_x = {
"x86_64": {
f"ReferenceAddress ({symbol})": "554889e54157415641554154534883ec284989f648897dd04c8b2d",
"NACInitAddress": hex_strings_10_14_x['x86_64']['NACInitAddress'],
"NACKeyEstablishmentAddress": hex_strings_10_14_x['x86_64']['NACKeyEstablishmentAddress'],
"NACSignAddress": hex_strings_10_14_x['x86_64']['NACSignAddress'],
}
}
# Current main one..
hex_strings_modern = {
"x86_64": {
f"ReferenceAddress ({symbol})": "554889e54157415641554154534883ec28..89..48897dd04c8b3d",
"NACInitAddress": "554889e541574156415541545350b87818",
"NACKeyEstablishmentAddress": "554889e54157415641554154534881ec..010000488b05......00488b00488945d04885",
"NACSignAddress": "554889e54157415641554154534881ec..030000........................................................................................................................................................................................48....48..........................................................................................................89............................................................", # TODO: Find a pattern that gets unique matches for this
},
"arm64e": {
f"ReferenceAddress ({symbol})": "7f2303d5ffc301d1fc6f01a9fa6702a9f85f03a9f65704a9f44f05a9fd7b06a9fd830191f30301aa....00........f9..0280b9..68..f8....00........f9....80b9..68..f8....00........f9..01..eb....0054f40300aa............................................................................................................................80b96d6a6df89f010deb....0054..0380b96d6a6df8................................................", # TODO: Find a pattern that gets unique matches for this
"NACInitAddress": "7f2303d5fc6fbaa9fa6701a9f85f02a9f65703a9f44f04a9fd7b05a9fd43019109..8352....00..10....f91f0a3fd6ff0740d1ff....d1....00..08....f9080140f9a8....f8......d2......f2......f2......f2e9",
"NACKeyEstablishmentAddress": "7f2303d5ff....d1fc6f..a9fa67..a9f85f..a9f657..a9f44f..a9fd7b..a9fd..0591....00..08....f9080140f9a8....f8......52",
"NACSignAddress": "7f2303d5fc6fbaa9fa6701a9f85f02a9f65703a9f44f04a9fd7b05a9fd430191ff....d1................08....f9......................................................................................................................................f2......f2......................d2",
},
}
# hex_strings = hex_strings_10_14_x
# hex_strings = hex_strings_10_15_x
hex_strings = hex_strings_modernEdit: Have also pushed these to my repo in 0xdevalias/poc-re-binsearch@573c125 |
|
There is some discussion about finding the offsets for macOS 15 binaries here: Including some of my notes on how to manually reverse the offsets in this comment: |
As I mentioned on my last PR (#7 (comment)) I've been working on a PoC script that is able to find the offsets automagically (at least across the
13.3.1,13.5,14.2x86 binaries I have tested it against so far):The concepts used here could be ported into
mac-registration-providerand used to provide support for platform versions where the proper offsets aren't currently known (on a "it should work, but no guarantees" best case effort)It might also be useful if
mac-registration-providerwas able to prompt the user to submit a PR/support request/similar with the relevant offsets/etc for their platform if they weren't already known (and were identified with this), and/or suggest how they can share the binary for their platform (along with appropriate version details) so the offsets can be calculated/verified, then added to the main tool.The text was updated successfully, but these errors were encountered: