YARA patterns in the static-code directory are very large

[timokau]

The static-code directory in the compiled archive is 3.9 GiB big (most of that is in static-code/pe/32/le/x86/delphi). In comparison the static-code directory from the uncompiled yara archive is only 710MiB.

Is that to be expected or is there some kind of issue causing this blowup?

If I understand the wiki correctly, those files are non-essential and only used for statically-linked code removal correct? And it sounds like retdec could even use the uncompiled patterns with some performance penalty?

[PeterMatula]

1. Yes, the files are not essential and decompilation would work without them. They are only used for statically-linked code removal.
2. The size blowup is to be expected. It is caused by their compilation by YARA itself. Nothing we can do about it.
3. Yes, I think we could use text YARA files (not the compiled ones) and it would work. But the parsing and compilation would happen at runtime and there would be some measurable performance penalty.

[timokau]

As I laid out here, shipping uncompiled files and compiling them as-needed would be a good compromise. I'm not sure if it is better to keep the discussion here or there.

Yes, I think we could use text YARA files (not the compiled ones) and it would work. But the parsing and compilation would happen at runtime and there would be some measurable performance penalty.

Is that possible right now without changes to the retdec source? That could be a reasonable alternative to just stripping the files altogether, which is what we currently do by default.

[PeterMatula]

I closed this but forgot to write an explanation, so here it goes.

From now on, YARA rules in the support package are in a text form. They are compiled at installation step by default. You can disable the compilation by setting RETDEC_COMPILE_YARA to OFF (it is ON by default). Installation step will install retdec-yarac binary. It can be used later to compile YARA rules if they were not compiled at installation. If you want, you can call support/install-yara.py script later to compile YARA rules - see the comment in the script on how to use it.

We will release the v3.3 version as it is now, but if you have more suggestions how to make this better, please let us know. I'm aware of the other suggestions about packaging RetDec, but I didn't get around to fix them yet. Pull request are welcomed :-D

Btw, even in the text form, YARA rules are probably much bigger than they need to be. Most of this is caused by Delphi signatures. The current solution is kind of an experiment on how to use YARA to this purpose. Maybe the rules do not have too be so long - it could work just as well with much shorter rules. It could be investigated further, but we do not have resources at the moment, maybe some university student will explore this in the future.

[timokau]

Sounds neat, thank you!

So if RETDEC_COMPILE_YARA is set to OFF, how will that affect decompilation results? Will it somehow use the text version (with a runtime cost), will it compile on-demand or will it perform as if no yara patterns were available?

[PeterMatula]

If RETDEC_COMPILE_YARA is OFF, it will use text YARA rules with a runtime cost - but the decompilaton result should not be affected.
It will NOT compile on demand - in a sense that rules would stay compiled for the subsequent runs - rules get compiled inside YARA every time it is triggered.

FYI, runtime cost of parsing YARA rules on my machine (Linux, 3 runs):

• ELF:
  • compiled [s]: 0.038, 0.037, 0.035
  • text [s]: 0.432, 0.445, 0.423
• PE gcc:
  • compiled [s]: 0.021, 0.020, 0.020
  • text [s]: 0.250, 0.235, 0.244
• PE msvc:
  • compiled [s]: 0.143, 0.141, 0.143
  • text [s]: 1.032, 1.061, 1.040
• PE delphi:
  • compiled [s]: 0.195, 0.172, 0.178
  • text [s]: 1.260, 1.268, 1.273

So as you can see, it is nothing terrible, but it is a little bit annoying - especially for Delphi.

[timokau]

Awesome! Thanks for the benchmarks.

The even-nicer-to-have feature request would be caching of the just-in-time compilation, but it is perfectly workable as it is now :)