Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

moves to centralized and uniform path handling, removes dead code #1278

Merged
merged 1 commit into from
Feb 26, 2021

Conversation

ivg
Copy link
Member

@ivg ivg commented Feb 25, 2021

Introduction

This PR changes how and where we store configuration, read-only data,
and caches. Instead of the previous approach, where each plugin was
receiving its own location via the configure script, we now rely on
the central location from Extension.Configuration module of the
bap-main library.

In addition, we introduce user-specific locations for configuration
files and read-only data that. The user-specific files, unlike the
system-wide, are not overwritten during bap installation. Which is
especially useful for developing lifters in Primus Lisp (it is a very
bad UX when your carefully written lifter is overwritten or removed on
opam update) and for configuring bap for your own taste and
feel (e.g., if you're getting annoyed with at&t syntax and want the
intel's one and don't want to update the configuration files on each
update).

Configuration Files and Locations

The configuration files are stored in two folders, the system-wide,
which is $prefix/etc/bap/ hosts the site and package specific
configuration. The user-specific location follows the XDG Base
Directory Specification and is either $XDG_CONFIG_HOME/bap or,
if $XDG_CONFIG_HOME is not set, to $HOME/.config/bap.

Both locactions are folders that hold an arbitrary number of
configuration files. The files are read and applied in alphabetical
order with the first read file having precedence in case if they
define values for the same parameter. The syntax is the common
ini-style syntax, with comments starting with #. Each line shall
have form <parameter> = <value>, where a parameter matches with the
name of a BAP command-line parameter with leading -- removed, e.g.,
suppose we want to switch to the intel syntax, instead of att for
x86 disassembler. The corresponding command-line parameter is
--llvm-x86-syntax. We create a file in the ~/.config/bap folder
with the following contents,

llvm-x86-syntax = intel

The filename could be anything and you can create a separate file per
each feature, or just have one big file. The configuration system is
also integrated with the cache subsystem, so changes to configuration
will be automatically detected and reflected in the cache digests.

Read-only Data

The read-only data includes various specifications and signatures,
like lisp semantics definitions, stubs, policy specifications, api
files, byteweight and other signatures.

We also store them in two locations, the system-wide,
$prefix/share/bap/, where the data shipped with bap is
installed. And user-specific $XDG_DATA_HOME (defaults to
$HOME/.local/share/bap), which is shared by all switches and is
never touched during bap installation.

Locations of the Lisp Files

The Primus Lisp library is now stored in three places:

  • $prefix/share/bap/primus/lisp
  • $prefix/share/bap/primus/site-lisp
  • $XDG_DATA_HOME/bap/primus/lisp

(where $XDG_DATA_HOME defaults to $HOME/.local/share/).

The $prefix/share/bap/primus/lisp stores the standard library and
language-specific definitions of Primus Lisp. It shall not be touched
by users. The $prefix/share/bap/primus/site-lisp folder is the
folder where libraries should be installed, both provided by bap and
by other packages. This approach follows conventions in Lisp and
Python (and other script languages) commmunities.

Finally, the location for local, user-specific, lisp libraries is
$XDG_DATA_HOME/bap/primus/lisp. This is where you can store your own
lisp codes and common libraries and expect that they will never be
touched by bap installation scripts. And do not forget that the
current folder is also in the search path, as well as that you can add
more paths using --primus-lisp-add to add more folders to the search
path. Together with the configurations files, it is now easy to handle
those paths, e.g., by adding a lisp-paths.conf file to ~/.config/bap
with the following contents.

primus-lisp-add = /home/user/share/primus-lisp

The Cache Location

The cache location wasn't really changed but is made more accessible
for those who would like to store their own bap-specific non-essential
runtime files. Just use Extension.Configuration.cachedir for that
or, even better, use the Cache facility of the regular library.

Breaking Changes

No interfaces were broken, but some files changed their
location. Everything is now stored in the bap folder, and is more
organized. For example, bap-api is now stored in bap/api folder, and
byteweight and stubs signatures are stored in bap/signatures. Also,
byteweight signatures are renamed to byteweight.zip from the old
sigs.zip. The update should pass seamless as it is all handled by
the configuation and installation script. Which will be updated
separately.

Introduction
============

This PR changes how and where we store configuration, read-only data,
and caches. Instead of the previous approach, where each plugin was
receiving its own location via the configure script, we now rely on
the central location from `Extension.Configuration` module of the
bap-main library.

In addition, we introduce user-specific locations for configuration
files and read-only data that. The user-specific files, unlike the
system-wide, are not overwritten during bap installation. Which is
especially useful for developing lifters in Primus Lisp (it is a very
bad UX when your carefully written lifter is overwritten or removed on
`opam update`) and for configuring bap for your own taste and
feel (e.g., if you're getting annoyed with at&t syntax and want the
intel's one and don't want to update the configuration files on each
update).

Configuration Files and Locations
=================================

The configuration files are stored in two folders, the system-wide,
which is `$prefix/etc/bap/` hosts the site and package specific
configuration. The user-specific location follows the XDG Base
Directory Specification and is either `$XDG_CONFIG_HOME/bap` or,
if `$XDG_CONFIG_HOME` is not set, to $HOME/.config/bap.

Both locactions are folders that hold an arbitrary number of
configuration files. The files are read and applied in alphabetical
order with the first read file having precedence in case if they
define values for the same parameter. The syntax is the common
ini-style syntax, with comments starting with `#`. Each line shall
have form `<parameter> = <value>`, where a parameter matches with the
name of a BAP command-line parameter with leading `--` removed, e.g.,
suppose we want to switch to the `intel` syntax, instead of `att` for
x86 disassembler. The corresponding command-line parameter is
`--llvm-x86-syntax`. We create a file in the `~/.config/bap` folder
with the following contents,

```
llvm-x86-syntax = intel
```

The filename could be anything and you can create a separate file per
each feature, or just have one big file. The configuration system is
also integrated with the cache subsystem, so changes to configuration
will be automatically detected and reflected in the cache digests.

Read-only Data
==============

The read-only data includes various specifications and signatures,
like lisp semantics definitions, stubs, policy specifications, api
files, byteweight and other signatures.

We also store them in two locations, the system-wide,
`$prefix/share/bap/`, where the data shipped with bap is
installed. And user-specific `$XDG_DATA_HOME` (defaults to
`$HOME/.local/share/bap`), which is shared by all switches and is
never touched during bap installation.

Locations of the Lisp Files
---------------------------

The Primus Lisp library is now stored in three places:

- `$prefix/share/bap/primus/lisp`
- `$prefix/share/bap/primus/site-lisp`
- `$XDG_DATA_HOME/bap/primus/lisp`

(where `$XDG_DATA_HOME` defaults to `$HOME/.local/share/`).

The `$prefix/share/bap/primus/lisp` stores the standard library and
language-specific definitions of Primus Lisp. It shall not be touched
by users. The `$prefix/share/bap/primus/site-lisp` folder is the
folder where libraries should be installed, both provided by bap and
by other packages. This approach follows conventions in Lisp and
Python (and other script languages) commmunities.

Finally, the location for local, user-specific, lisp libraries is
`$XDG_DATA_HOME/bap/primus/lisp`. This is where you can store your own
lisp codes and common libraries and expect that they will never be
touched by bap installation scripts. And do not forget that the
current folder is also in the search path, as well as that you can add
more paths using `--primus-lisp-add` to add more folders to the search
path. Together with the configurations files, it is now easy to handle
those paths, e.g., by adding a `lisp-paths.conf` file to `~/.config/bap`
with the following contents.

```conf
primus-lisp-add = /home/user/share/primus-lisp
```

The Cache Location
==================

The cache location wasn't really changed but is made more accessible
for those who would like to store their own bap-specific non-essential
runtime files. Just use `Extension.Configuration.cachedir` for that
or, even better, use the Cache facility of the `regular` library.

Breaking Changes
================

No interfaces were broken, but some files changed their
location. Everything is now stored in the bap folder, and is more
organized. For example, bap-api is now stored in bap/api folder, and
byteweight and stubs signatures are stored in `bap/signatures`. Also,
byteweight signatures are renamed to `byteweight.zip` from the old
`sigs.zip`. The update should pass seamless as it is all handled by
the configuation and installation script. Which will be updated
separately.
ivg added a commit to BinaryAnalysisPlatform/opam-repository that referenced this pull request Feb 26, 2021
Syncronizes with
BinaryAnalysisPlatform/bap#1278

In fact, the only necessary change was the bap-signatures update,
everything else is more or less unnecessary as opam should correctly
remove installed data files. We should probably remove the `remove`
field altogether.
@ivg ivg merged commit 40efc12 into BinaryAnalysisPlatform:master Feb 26, 2021
@ivg ivg deleted the straighten-folders branch December 1, 2021 19:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant