glibc >= 2.34 support #282

dp1 · 2022-09-15T11:39:16Z

Starting with version 2.34, glibc removed symlinks from .deb packages:

* Previously, glibc installed its various shared objects under versioned
  file names such as libc-2.33.so.  The ABI sonames (e.g., libc.so.6)
  were provided as symbolic links.  Starting with glibc 2.34, the shared
  objects are installed under their ABI sonames directly, without
  symbolic links.  This increases compatibility with distribution
  package managers that delete removed files late during the package
  upgrade or downgrade process.

This seems to affect both normal packages and debug symbols, which are now only stored as build-id indexed files. Contents of libc6-dbg_2.34-0ubuntu1_amd64.deb:

dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ tree -a lib
lib
└── debug
    └── .build-id
        ├── 00
        │   ├── 353e7ffbfa963d5e2e219ec8e207543fe1a3a8.debug
        │   └── f20d89ff15a56fa67617724be2c13f3a40a0a3.debug
        ├── 01
        │   ├── 8f46b01a6f2f5e9590cfd293d70b33807bb41f.debug
        │   ├── 9604cce2625c8507da1181a800116a7b37b7c0.debug
        │   └── f1eedae89122b983696c744b7476bb2b4c13be.debug
        ├── 02
        │   └── b3ca6e48e03b04b79e404c1d6472b3126f5991.debug
        ├── 03
[...]

pwninit tries to look for the versioned filenames, which fails in these more recent libcs. This PR addresses both problems. To avoid any regression, I tested against all ubuntu libcs (using libc-database) with the following scripts and verified that no new warnings were added and no files were modified other than adding linkers and unstripping libcs >= 2.34

Warnings:
warning: failed detecting libc version (is the libc an Ubuntu glibc?): failed finding version string: 153 
warning: failed unstripping libc: eu-unstrip exited with failure: exit status: 1: 163 
warning: failed fetching ld: libc deb error: failed to find file in data.tar: 0 -30
warning: failed unstripping libc: libc deb error: failed to find file in data.tar: 26 -10
Total: 342 -40

test_pwninit.py

#!/usr/bin/env python3

"""
Test pwninit against all the ubuntu libcs from libc-database
Run libc-dataase's "./get ubuntu" first to get the libc binaries
"""

import os
import argparse
import sys
import tempfile
import shutil
import subprocess
import multiprocessing
import json
from Crypto.Hash import SHA256
from glob import glob
from tqdm import tqdm

DUMMY_BINARY = "/home/dario/Desktop/ctf/tools/pwninit-test/chal1"
PWNINIT_BINARY = "/home/dario/Desktop/ctf/tools/pwninit/target/release/pwninit"
LIBCDB_FOLDER = "/home/dario/Desktop/ctf/tools/libc-database/db"

libc_binaries = glob(f'{LIBCDB_FOLDER}/*.so')

def get_pwninit_warnings(dir, libc_name):
    output = subprocess.check_output([
        PWNINIT_BINARY, '--bin', 'chal1', '--libc', libc_name
    ], cwd=dir, stderr=subprocess.STDOUT).decode()

    return list(filter(lambda x : x.startswith('warning'), output.splitlines()))

def get_filestats(dir):
    filestats = {}
    for f in os.listdir(dir):
        if f not in ['chal1', 'chal1_patched', 'solve.py']:
            with open(os.path.join(dir, f), 'rb') as fin:
                data = fin.read()
            filestats[f] = {"len": len(data), "sha": SHA256.new(data).hexdigest()}

    return filestats

def process_libc(libc_path):
    libc_name = os.path.basename(libc_path)

    with tempfile.TemporaryDirectory() as dir:
        shutil.copyfile(libc_path, os.path.join(dir, libc_name))
        shutil.copyfile(DUMMY_BINARY, os.path.join(dir, "chal1"))

        warnings = get_pwninit_warnings(dir, libc_name)
        filestats = get_filestats(dir)

        return libc_name, warnings, filestats

def main(output_file):
    results = {}

    with multiprocessing.Pool(64) as pool:
        with tqdm(total = len(libc_binaries)) as pbar:
            for libc_name, warnings, filestats in pool.imap_unordered(process_libc, libc_binaries):

                assert libc_name not in results
                results[libc_name] = {"warnings": warnings, "files": filestats}

                pbar.update(1)

    with open(output_file, 'w') as fout:
        json.dump(results, fout)

if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('output_file', help='Output file to write the results into. JSON')
    parser.add_argument('--force', action='store_true', help='Force output file overwrite')
    args = parser.parse_args()

    if os.path.exists(args.output_file) and not args.force:
        print('Output file already exists. Stopping')
        sys.exit(1)

    main(args.output_file)

diff_results.py

#!/usr/bin/env python3

import json
import argparse
from colorama import Fore, Style

def colored_delta(new_in_b, solved_in_b):
    res = ''
    if new_in_b > 0:
        res += f'{Fore.RED}+{new_in_b} '
    if solved_in_b > 0:
        res += f'{Fore.GREEN}-{solved_in_b} '
    return res.strip() + Style.RESET_ALL

def main(file_a, file_b):
    with open(file_a) as fin:
        data_a = json.load(fin)
    with open(file_b) as fin:
        data_b = json.load(fin)

    print('Files:')
    num_new = 0
    num_deleted = 0
    num_modified = 0

    for libc,metadata in data_b.items():
        files_b = metadata["files"]
        files_a = data_a[libc]["files"]

        files = {}
        for filename,filestats in files_a.items():
            files[filename] = {'a':filestats, 'b':{}}
        for filename,filestats in files_b.items():
            if filename not in files:
                files[filename] = {'a':{}, 'b':{}}
            files[filename]['b'] = filestats

        diff = False
        for filename,data in files.items():
            if data['a'] != data['b']:
                diff = True

        if diff:
            print(libc)
            for filename,data in files.items():
                if data["a"] == data["b"]: continue

                if data["a"] == {}:
                    print(f'  {filename}: new -> {data["b"]["len"]} bytes')
                    num_new += 1
                elif data["b"] == {}:
                    print(f'  {filename}: {data["a"]["len"]} bytes -> deleted')
                    num_deleted += 1
                else:
                    print(f'  {filename}: {data["a"]["len"]} bytes -> {data["b"]["len"]} bytes')
                    num_modified += 1

    print(f'Total: {num_new} new files, {num_deleted} deleted files, {num_modified} modified files')

    warnings: dict[str,dict[str,set[str]]] = {}

    for libc,metadata in data_a.items():
        for warning in metadata["warnings"]:
            if warning not in warnings:
                warnings[warning] = {'a':set(), 'b':set()}
            warnings[warning]['a'].add(libc)

    for libc,metadata in data_b.items():
        for warning in metadata["warnings"]:
            if warning not in warnings:
                warnings[warning] = {'a':set(), 'b':set()}
            warnings[warning]['b'].add(libc)

    print('Warnings:')

    total_in_b = 0
    total_new_in_b = 0
    total_solved_in_b = 0

    for warning,libcs in warnings.items():
        new_in_b = len(libcs['b'].difference(libcs['a']))
        solved_in_b = len(libcs['a'].difference(libcs['b']))

        print(f'{warning}: {len(libcs["b"])} {colored_delta(new_in_b, solved_in_b)}')

        total_in_b += len(libcs['b'])
        total_new_in_b += new_in_b
        total_solved_in_b += solved_in_b

        if len(libcs['b'].difference(libcs['a'])) > 0:
            print(libcs['b'].difference(libcs['a']))

    print(f'Total: {total_in_b} {colored_delta(total_new_in_b, total_solved_in_b)}')


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('file_a', help='Diff from this results file')
    parser.add_argument('file_b', help='Diff to this results file')
    args = parser.parse_args()

    main(args.file_a, args.file_b)

Full output

dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test$ python diff_results.py results/baseline-v3.2.0.json results/ld_version_and_build_id.json 
Files:
libc6-amd64_2.36-0ubuntu2_i386.so
  ld-2.36.so: new -> 228720 bytes
libc6-x32_2.35-0ubuntu3.1_i386.so
  ld-2.35.so: new -> 240936 bytes
libc6-i386_2.34-0ubuntu3_amd64.so
  ld-2.34.so: new -> 213552 bytes
libc6_2.36-0ubuntu2_i386.so
  libc6_2.36-0ubuntu2_i386.so: 2272520 bytes -> 6443980 bytes
  libc.so.6: 2272520 bytes -> 6443980 bytes
  ld-2.36.so: new -> 217732 bytes
libc6_2.35-0ubuntu3_amd64.so
  libc6_2.35-0ubuntu3_amd64.so: 2216304 bytes -> 6619296 bytes
  libc.so.6: 2216304 bytes -> 6619296 bytes
  ld-2.35.so: new -> 240936 bytes
libc6_2.34-0ubuntu3.2_amd64.so
  libc6_2.34-0ubuntu3.2_amd64.so: 2216272 bytes -> 6566152 bytes
  libc.so.6: 2216272 bytes -> 6566152 bytes
  ld-2.34.so: new -> 220376 bytes
libc6_2.34-0ubuntu3.2_i386.so
  libc.so.6: 2272532 bytes -> 6373808 bytes
  libc6_2.34-0ubuntu3.2_i386.so: 2272532 bytes -> 6373808 bytes
  ld-2.34.so: new -> 213552 bytes
libc6-i386_2.36-0ubuntu2_amd64.so
  ld-2.36.so: new -> 217732 bytes
libc6_2.35-0ubuntu3.1_amd64.so
  libc.so.6: 2216304 bytes -> 6616616 bytes
  libc6_2.35-0ubuntu3.1_amd64.so: 2216304 bytes -> 6616616 bytes
  ld-2.35.so: new -> 240936 bytes
libc6_2.35-0ubuntu3_i386.so
  libc.so.6: 2280756 bytes -> 6460268 bytes
  libc6_2.35-0ubuntu3_i386.so: 2280756 bytes -> 6460268 bytes
  ld-2.35.so: new -> 225864 bytes
libc6-x32_2.36-0ubuntu2_amd64.so
  ld-2.36.so: new -> 228720 bytes
libc6_2.34-0ubuntu3_amd64.so
  libc.so.6: 2215936 bytes -> 6565240 bytes
  libc6_2.34-0ubuntu3_amd64.so: 2215936 bytes -> 6565240 bytes
  ld-2.34.so: new -> 220376 bytes
libc6-amd64_2.34-0ubuntu3_i386.so
  ld-2.34.so: new -> 220376 bytes
libc6-x32_2.34-0ubuntu3.2_amd64.so
  ld-2.34.so: new -> 220376 bytes
libc6-amd64_2.34-0ubuntu3.2_i386.so
  ld-2.34.so: new -> 220376 bytes
libc6-x32_2.35-0ubuntu3_amd64.so
  ld-2.35.so: new -> 240936 bytes
libc6_2.35-0ubuntu3.1_i386.so
  libc.so.6: 2280756 bytes -> 6454420 bytes
  libc6_2.35-0ubuntu3.1_i386.so: 2280756 bytes -> 6454420 bytes
  ld-2.35.so: new -> 225864 bytes
libc6-i386_2.34-0ubuntu3.2_amd64.so
  ld-2.34.so: new -> 213552 bytes
libc6-i386_2.35-0ubuntu3.1_amd64.so
  ld-2.35.so: new -> 225864 bytes
libc6-x32_2.34-0ubuntu3_amd64.so
  ld-2.34.so: new -> 220376 bytes
libc6-amd64_2.35-0ubuntu3.1_i386.so
  ld-2.35.so: new -> 240936 bytes
libc6-x32_2.36-0ubuntu2_i386.so
  ld-2.36.so: new -> 228720 bytes
libc6-i386_2.35-0ubuntu3_amd64.so
  ld-2.35.so: new -> 225864 bytes
libc6-x32_2.34-0ubuntu3_i386.so
  ld-2.34.so: new -> 220376 bytes
libc6-x32_2.34-0ubuntu3.2_i386.so
  ld-2.34.so: new -> 220376 bytes
libc6_2.34-0ubuntu3_i386.so
  libc.so.6: 2270640 bytes -> 6371112 bytes
  libc6_2.34-0ubuntu3_i386.so: 2270640 bytes -> 6371112 bytes
  ld-2.34.so: new -> 213552 bytes
libc6-x32_2.35-0ubuntu3_i386.so
  ld-2.35.so: new -> 240936 bytes
libc6-amd64_2.35-0ubuntu3_i386.so
  ld-2.35.so: new -> 240936 bytes
libc6-x32_2.35-0ubuntu3.1_amd64.so
  ld-2.35.so: new -> 240936 bytes
libc6_2.36-0ubuntu2_amd64.so
  libc.so.6: 2072888 bytes -> 6416416 bytes
  libc6_2.36-0ubuntu2_amd64.so: 2072888 bytes -> 6416416 bytes
  ld-2.36.so: new -> 228720 bytes
Total: 30 new files, 0 deleted files, 20 modified files
Warnings:
warning: failed detecting libc version (is the libc an Ubuntu glibc?): failed finding version string: 153 
warning: failed unstripping libc: eu-unstrip exited with failure: exit status: 1: 163 
warning: failed fetching ld: libc deb error: failed to find file in data.tar: 0 -30
warning: failed unstripping libc: libc deb error: failed to find file in data.tar: 26 -10
Total: 342 -40

TL;DR

Testing with how2pwn from CSAW CTF 2022 Qualifiers, which uses glibc 2.34

Before

dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ pwninit --version
pwninit 3.2.0
dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ ll
total 2188
drwxrwxr-x 2 dario dario    4096 set 15 13:13 ./
drwxrwxr-x 5 dario dario    4096 set 14 13:02 ../
-rwxr-xr-x 1 dario dario   16280 set 13 16:18 chal1*
-rw-rw-r-- 1 dario dario 2215936 set 15 13:13 libc6_2.34-0ubuntu1_amd64.so
dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ pwninit
bin: ./chal1
libc: ./libc6_2.34-0ubuntu1_amd64.so

fetching linker
https://launchpad.net/ubuntu/+archive/primary/+files//libc6_2.34-0ubuntu1_amd64.deb
warning: failed fetching ld: libc deb error: failed to find file in data.tar
unstripping libc
https://launchpad.net/ubuntu/+archive/primary/+files//libc6-dbg_2.34-0ubuntu1_amd64.deb
warning: failed unstripping libc: libc deb error: failed to find file in data.tar
symlinking ./libc.so.6 -> libc6_2.34-0ubuntu1_amd64.so
copying ./chal1 to ./chal1_patched
running patchelf on ./chal1_patched
writing solve.py stub
dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ ll
total 2212
drwxrwxr-x 2 dario dario    4096 set 15 13:14 ./
drwxrwxr-x 5 dario dario    4096 set 14 13:02 ../
-rwxr-xr-x 1 dario dario   16280 set 13 16:18 chal1*
-rwxr-xr-x 1 dario dario   17192 set 15 13:14 chal1_patched*
-rw-rw-r-- 1 dario dario 2215936 set 15 13:13 libc6_2.34-0ubuntu1_amd64.so
lrwxrwxrwx 1 dario dario      28 set 15 13:14 libc.so.6 -> libc6_2.34-0ubuntu1_amd64.so
-rwxrwxr-x 1 dario dario     666 set 15 13:14 solve.py*

After

dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ ll
total 2188
drwxrwxr-x 2 dario dario    4096 set 15 13:15 ./
drwxrwxr-x 5 dario dario    4096 set 14 13:02 ../
-rwxr-xr-x 1 dario dario   16280 set 13 16:18 chal1*
-rw-rw-r-- 1 dario dario 2215936 set 15 13:15 libc6_2.34-0ubuntu1_amd64.so
dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ ../../pwninit/target/release/pwninit 
bin: ./chal1
libc: ./libc6_2.34-0ubuntu1_amd64.so

fetching linker
https://launchpad.net/ubuntu/+archive/primary/+files//libc6_2.34-0ubuntu1_amd64.deb
unstripping libc
https://launchpad.net/ubuntu/+archive/primary/+files//libc6-dbg_2.34-0ubuntu1_amd64.deb
setting ./ld-2.34.so executable
symlinking ./libc.so.6 -> libc6_2.34-0ubuntu1_amd64.so
copying ./chal1 to ./chal1_patched
running patchelf on ./chal1_patched
writing solve.py stub
dario@dario-laptop:~/Desktop/ctf/tools/pwninit-test/a$ ll
total 6672
drwxrwxr-x 2 dario dario    4096 set 15 13:15 ./
drwxrwxr-x 5 dario dario    4096 set 14 13:02 ../
-rwxr-xr-x 1 dario dario   16280 set 13 16:18 chal1*
-rwxr-xr-x 1 dario dario   17176 set 15 13:15 chal1_patched*
-rwxrwxr-x 1 dario dario  220376 set 15 13:15 ld-2.34.so*
-rw-rw-r-- 1 dario dario 6559304 set 15 13:15 libc6_2.34-0ubuntu1_amd64.so
lrwxrwxrwx 1 dario dario      28 set 15 13:15 libc.so.6 -> libc6_2.34-0ubuntu1_amd64.so
-rwxrwxr-x 1 dario dario     452 set 15 13:15 solve.py*

dp1 · 2022-09-25T11:24:54Z

@io12 is there any progress on this? I'd like to get it merged soon since recent libcs are being used in more and more ctfs. If there's something missing or something that should be changed please let me know

io12 · 2022-09-28T16:39:24Z

Sorry, I've been really busy recently. Can you run cargo fmt and fix the cargo clippy lints? Thanks!

dp1 · 2022-09-28T16:52:29Z

No worries at all! I run cargo fmt and fixed the lint errors. I also took the liberty to fix a lint error in solvepy.rs, it complained with the following:

warning: unnecessary closure used to substitute value for `Option::None`
  --> src/solvepy.rs:51:13
   |
51 | /             patch_bin::bin_patched_path(opts)
52 | |                 .as_ref()
53 | |                 .or_else(|| opts.bin.as_ref()),
   | |__________________----------------------------^
   |                    |
   |                    help: use `or(..)` instead: `or(opts.bin.as_ref())`
   |
   = note: `#[warn(clippy::unnecessary_lazy_evaluations)]` on by default
   = help: for further information visit https://rust-lang.github.io/rust-clippy/master/index.html#unnecessary_lazy_evaluations

dp1 · 2022-10-07T14:18:33Z

Hey @io12, just a ping for this :)

Nicholaz99 · 2022-11-02T16:12:17Z

Would be great if this PR can be merged soon

dp1 · 2022-11-04T13:36:52Z

Indeed, it would be great to have pwninit support modern libcs @io12

toasterpwn · 2022-11-09T09:41:28Z

Can this be merged? using pwninit with 2.35 would be nice. @io12

nicolaipre · 2022-11-18T10:20:05Z

Would be nice to see this merged in.

Exiled1 · 2022-11-29T23:11:52Z

Yeah, this would be amazing to have.

io12 · 2022-12-12T01:43:39Z

Thanks so much for this @dp1 and sorry for the delay! I pushed 3.3.0 just now.

Exiled1 · 2022-12-12T02:05:12Z

LETS GO BOYS!!! Best CTF tool ever ♥️

dp1 · 2022-12-12T09:24:31Z

Thank you! Back to pwning♥️

dp1 added 2 commits September 14, 2022 14:59

Use the ABI soname for extracting ld >= 2.34

661f6f3

Use build id to lookup debug symbols for libc >= 2.34

98712b5

dp1 added 2 commits September 28, 2022 18:49

cargo fmt

248ecb8

Fix clippy warning

63d9922

io12 merged commit 0932ae4 into io12:master Dec 12, 2022

dp1 mentioned this pull request Dec 12, 2022

Use build id to lookup debug symbols for libc >= 2.31 #285

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

glibc >= 2.34 support #282

glibc >= 2.34 support #282

dp1 commented Sep 15, 2022

dp1 commented Sep 25, 2022

io12 commented Sep 28, 2022

dp1 commented Sep 28, 2022

dp1 commented Oct 7, 2022

Nicholaz99 commented Nov 2, 2022

dp1 commented Nov 4, 2022

toasterpwn commented Nov 9, 2022

nicolaipre commented Nov 18, 2022 •

edited

Loading

Exiled1 commented Nov 29, 2022

io12 commented Dec 12, 2022

Exiled1 commented Dec 12, 2022

dp1 commented Dec 12, 2022

glibc >= 2.34 support #282

glibc >= 2.34 support #282

Conversation

dp1 commented Sep 15, 2022

TL;DR

dp1 commented Sep 25, 2022

io12 commented Sep 28, 2022

dp1 commented Sep 28, 2022

dp1 commented Oct 7, 2022

Nicholaz99 commented Nov 2, 2022

dp1 commented Nov 4, 2022

toasterpwn commented Nov 9, 2022

nicolaipre commented Nov 18, 2022 • edited Loading

Exiled1 commented Nov 29, 2022

io12 commented Dec 12, 2022

Exiled1 commented Dec 12, 2022

dp1 commented Dec 12, 2022

nicolaipre commented Nov 18, 2022 •

edited

Loading