Improve format string generator

[bennofs]

This PR improves the format string generator:

• it now should build close to optimal length format string (for example, it will make use of long writes to write long sequences of zeros)
• it works with null bytes in addresses (common on 64 bit) since it writes pointers after instructions (so no null
• it has the ability to generate pointers/instructions separately (see fmt_str_split)
• the interface should be such that it is relatively easy to manually craft all kinds of writes (using the AtomWrite primitive)

[zachriggle]

Really cool stuff! It's missing some function-level documentation and doctests, and there are some general design questions (nested functions vs. an object with methods).

If we can get this cleaned up, it would be cool to merge.

[Arusekk]

A huge +1, this module was pretty useless on 64-bit stuff so far. But there are some minor issues concerning python 3 compatibility or coding style.

[bennofs]

I've done most of the cleanup now (there's still one nested function and some docs remaining).

Remaining todo before this is merge ready:

• test on big-endian
• more docs
• remove one nested function

Optional:

• end-to-end tests for all platforms (a binary with printf compiled for each platform -> verify write works)

[bennofs]

Would it be ok to commit compiled, static binaries for tests to the repo? The binary for mips for example is about 600kb. Requiring a cross-compiler for tests is painful.

[zachriggle]

@bennofs Yes, I believe this was part of my format string branch.

I would greatly prefer a flexible binary that allows lots of tests. You can see what I had in mind here: https://github.com/zachriggle/pwntools/tree/format-string/pwnlib/data/formatstring

If you're interested, you can also see how I planned to perform "automatic" format string exploration and exploitation here: https://github.com/zachriggle/pwntools/blob/format-string/pwnlib/formatstring.py

And look at the documentation here: https://github.com/zachriggle/pwntools/blob/format-string/docs/source/fmtstr.rst

Specifically, I wanted to leverage a prefix stub ("get me to the format string") and then use the Corefile tools to extract state.

[zachriggle]

Ah, there is also an LD_PRELOAD tool in that folder that I was planning to use. Basically it replaces all of the PLT entries for *printf with a trap instruction (int3 or equivalent) so that a corefile is generated when the function gets called.

The intent here is to capture the register state and stack at the time *printf is invoked, with a cyclic() pattern on the stack (or in whatever buffer(s) we control). We can then automate everything else.

The obvious limitation is that this will trap on the first *printf call, but it was all just work-in-progress stuff.

[bennofs]

I now verified that the code works correctly in a basic test for: MIPS, MIPS64, MIPSEL, MIPS64EL, X86, X86_64, PPC, PPC64, ARM, AARCH64. There is a Makefile that uses the cross compilers available on Ubuntu bionic (that's the version I used for tests) to build example vulnerable binaries for each arch.

[bennofs]

Should be ready for merge now from my side.

[Arusekk]

It looks like we must resolve some build problems before merging this, but looks ready to me

[zachriggle]

Yes, I need to do a stable => beta => dev merge.

[zachriggle]

For the sake of being able to run the tests provided, would you mind adding the printf binaries that you compiled? (And removing the .gitiginore)

[bennofs]

I added the binaries. Wasn't sure on whether to add them since they increase the size of the git repo. The command to build them, run in ubuntu:18.04: make install-apt && make

[zachriggle]

They do inflate the size of the repo slightly, but having binaries that don't change due to toolchain / CI / etc. differences is preferable.