PIMO : Private Input, Masked Output

PIMO is a tool for data masking. It can mask data from a JSONline stream and return another JSONline stream thanks to a masking configuration contained in a yaml file.

You can use LINO to extract sample data from a database, which you can then use as input data for PIMO's data masking. You can also generate data with a simple yaml configuration file.

Capabilities

credibility : generated data is not distinguishable from real data
data synthesis : generate data from nothing
data masking, including
- randomization : protect personal or sensitive data by writing over it
- pseudonymization, on 3 levels
  - consistent pseudonymisation : real value A is always replaced by pseudo-value X but X can be attributed to other values than A
  - identifiant pseudonymisation : real value A is always replaced by pseudo-value X and X CANNOT be attributed to other values than A
  - reversible pseudonymisation : real value A can be generated from pseudo-value X

Configuration file needed

PIMO requires a yaml configuration file to works. By default, the file is named masking.yml and is placed in the working directory. The file must respect the following format :

version: "1"
seed: 42
functions:
    # Optional define functions
masking:
  - selector:
      jsonpath: "example.example"
    mask:
      type: "argument"
    # Optional cache (coherence preservation)
    cache: "cacheName"
    # Optional custom seed for this mask
    seed:
      field: "example.example"

  # another mask on a different location
  - selector:
      jsonpath: "example.example2"
    mask:
      type: "argument"
    preserve: "null"
    preserve-list: ["value to preserve"]

caches:
  cacheName:
    # Optional bijective cache (enable re-identification if the cache is dumped on disk)
    unique: true
    # Use reverse cache dictionnary
    reverse: true

version is the version of the masking file. seed is to give every random mask the same seed, it is optional and if it is not defined, the seed is derived from the current time to increase randomness. functions is used to define the functions that can be used in the te mask template, template-each, add, and add-transient. masking is used to define the pipeline of masks that is going to be applied. selector is made of a jsonpath and a mask. jsonpath defines the path of the entry that has to be masked in the json file. mask defines the mask that will be used for the entry defined by selector. cache is optional, if the current entry is already in the cache as key the associated value is returned without executing the mask. Otherwise the mask is executed and a new entry is added in the cache with the orignal content as key and the masked result as value. The cache have to be declared in the caches section of the YAML file. preserve is optional, and is used to keep some values unmasked in the json file. Allowed preserve options are: "null" (null values), "empty" (empty string ""), and "blank" (both empty and null values). Additionally, preserve can be used with mask fromCache to preserve uncached values. (usage: preserve: "notInCache") preserve-list is optional and is used to ignore specific values

Multiple masks can be applied on the same jsonpath location, like in this example :

  - selector:
      jsonpath: "example"
    masks:
      - add: "hello"
      - template: "{{.example}} World!"
      - remove: true

Masks can be applied on multiple selectors, like in this example:

  - selectors:
      - jsonpath: "example"
      - jsonpath: "example2"
    mask:
      add: "hello"

It is possible to define functions and reuse them later in the masks, like in this example:

functions:
  add20:
    params:
      - name: "i"
    body: |-
      return i + 20
  sub:
    params:
      - name: "x"
      - name: "y"
    body: |-
      return x - y
masking:
  - selector:
      jsonpath: "addValue"
    mask:
      template: '{{add20 5}}'
  - selector:
      jsonpath: "subValue"
    mask:
      template: '{{sub 10 5}}'

Possible masks

The following types of masks can be used :

Pure randomization masks
- regex is to mask using a regular expression given in argument.
- randomInt is to mask with a random int from a range with arguments min and max.
- randomDecimal is to mask with a random decimal from a range with arguments min, max and precision.
- randDate is to mask a date with a random date between dateMin and dateMax.
- randomDuration is to mask a date by adding or removing a random time between Min and Max.
- randomChoice is to mask with a random value from a list in argument.
- weightedChoice is to mask with a random value from a list with probability, both given with the arguments choice and weight.
- randomChoiceInUri is to mask with a random value from an external resource.
- randomChoiceInCSV is to mask with a random value from an external CSV resource.
- transcode is to mask a value randomly with character class preservation.
- timeline to generate a set of dates related to each other (by rules and constraints)
K-Anonymization
- range is to mask a integer value by a range of value (e.g. replace 5 by [0,10]).
- duration is to mask a date by adding or removing a certain number of days.
Re-identification and coherence preservation
- hash is to mask with a value from a list by matching the original value, allowing to mask a value the same way every time.
- hashInUri is to mask with a value from an external resource, by matching the original value, allowing to mask a value the same way every time.
- hashInCSV is to mask with a value from an external CSV resource, by matching the original value, allowing to mask a value the same way every time.
- fromCache is a mask to obtain a value from a cache.
- ff1 mask allows the use of FPE which enable private-key based re-identification.
- sha3 masks will apply a variable length cryptographic hash (SHAKE variable-output-length hash function defined by FIPS-202) and then apply a base-conversion to the output.
Formatting
- dateParser is to change a date format.
- template is to mask a data with a template using other values from the jsonline.
- template-each is like template but will apply on each value of an array.
- fromjson is to convert string field values to parsed JSON, e.g. "[1,2,3]" -> [1,2,3].
Data structure manipulation
- remove is to mask a field by completely removing it.
- add is a mask to add a field to the jsonline.
- add-transient same as add but the field is not exported in the output jsonline.
Others
- constant is to mask the value by a constant value given in argument.
- command is to mask with the output of a console command given in argument.
- incremental is to mask data with incremental value starting from start with a step of increment.
- sequence generate sequenced IDs of any format.
- fluxUri is to replace by a sequence of values defined in an external resource.
- replacement is to mask a data with another data from the jsonline.
- pipe is a mask to handle complex nested array structures, it can read an array as an object stream and process it with a sub-pipeline.
- apply process selected data with a sub-pipeline.
- luhn can generate valid numbers using the Luhn algorithm (e.g. french SIRET or SIREN).
- markov can generate pseudo text based on a sample text.
- findInCSV get one or multiple csv lines which matched with Json entry value from CSV files.
- xml can manipulate XML content within JSON values.

A full masking.yml file example, using every kind of mask, is given with the source code.

In case two types of mask are entered with the same selector, the program can't extract the masking configuration and will return an error. The file wrongMasking.yml provided with the source illustrate that error.

Usage

To use PIMO to mask a data.json, use in the following way :

./pimo <data.json >maskedData.json

This takes the data.json file, masks the data contained inside it and put the result in a maskedData.json file. If data are in a table (for example multiple names), then each field of this table will be masked using the given mask. The following flags can be used:

--repeat=N This flag will make pimo mask every input N-times (useful for dataset generation).
--skip-line-on-error This flag will totally skip a line if an error occurs masking a field.
--skip-field-on-error This flag will return output without a field if an error occurs masking this field.
--skip-log-file <filename> Skipped lines will be written to <filename>.
--catch-errors <filename> or -e <filename> Equivalent to --skip-line-on-error --skip-log-file <filename>.
--empty-input This flag will give PIMO a {} input, usable with --repeat flag.
--config=filename.yml This flag allow to use another file for config than the default masking.yml.
--load-cache cacheName=filename.json This flag load an initial cache content from a file (json line format {"key":"a", "value":"b"}).
--dump-cache cacheName=filename.json This flag dump final cache content to a file (json line format {"key":"a", "value":"b"}).
--verbosity <level> or -v<level> This flag increase verbosity on the stderr output, possible values: none (0), error (1), warn (2), info (3), debug (4), trace (5).
--debug This flag complete the logs with debug information (source file, line number).
--log-json Set this flag to produce JSON formatted logs (demo9 goes deeper into logging and structured logging)
--seed <int> Set this flage to declare seed in command line.
--mask Declare a simple masking definition in command line (minified YAML format: --mask "value={fluxUri: 'pimo://nameFR'}", or --mask "value=[{add: ''},{fluxUri: 'pimo://nameFR'}]" for multiple masks). For advanced use case (e.g. if caches needed) masking.yml file definition will be preferred.
--repeat-until <condition> This flag will make PIMO keep masking every input until the condition is met. Condition format is using Template. Last output verifies the condition.
--repeat-while <condition> This flag will make PIMO keep masking every input while the condition is met. Condition format is using Template.
--stats <filename | url> This flag either outputs run statistics to the specified file or send them to specified url (has to start with http or https).
--statsTemplate <string> This flag will have PIMO use the value as a template to generate statistics. Please use go templating format to include statistics. To include them you have to specify them as {{ .Stats }}. (i.e. {"software":"PIMO","stats":{{ .Stats }}})

PIMO Play

The play command will start a local website, where you will find commented examples and a playground to play with the masking configuration.

$ pimo play
⇨ http server started on [::]:3010

Then go to http://localhost:3010/ in your browser.

Examples

This section will give examples for every types of mask.

Please check the demo folder for more advanced examples.

Regex

  - selector:
      jsonpath: "phone"
    mask:
      regex: "0[1-7]( ([0-9]){2}){4}"

This example will mask the phone field of the input jsonlines with a random string respecting the regular expression.

Name		Name	Last commit message	Last commit date
Latest commit History 486 Commits
.devcontainer		.devcontainer
.github		.github
.vscode		.vscode
assets		assets
cmd/pimo		cmd/pimo
demo		demo
doc/img		doc/img
githooks		githooks
internal/app/pimo		internal/app/pimo
pkg		pkg
schema/v1		schema/v1
test		test
wasm/pimo		wasm/pimo
web/play		web/play
.editorconfig		.editorconfig
.env.example		.env.example
.gitattributes		.gitattributes
.gitignore		.gitignore
.goreleaser.template.yml		.goreleaser.template.yml
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
Makefile		Makefile
NOTICE.md		NOTICE.md
README.md		README.md
build.yml		build.yml
go.mod		go.mod
go.sum		go.sum
masking.yml		masking.yml

agency	agency_number	name	account_type	account_number	annual_income
NewYork	0032	Doe	classic	12345	50000
SanFrancisco	7894	Smith	saving	67890	60000

agency	agency_number	name	account_type	account_number	annual_income
NewYork	2308	Rolande	saving	1	50000
SanFrancisco	9724	Matéo	securitie	2	60000

name	description
`nameEN`	english female or male names
`nameENF`	english female names
`nameENM`	english male names
`nameFR`	french female or male names
`nameFRF`	french female names
`nameFRM`	french male names
`surnameFR`	french surnames
`townFR`	french towns names

License

CGI-FR/PIMO

Folders and files

Latest commit

History

Repository files navigation

PIMO : Private Input, Masked Output

Configuration file needed

Possible masks

Usage

PIMO Play

Examples

Regex

Constant

RandomChoice

RandomChoiceInUri

RandomChoiceInCSV

RandomInt

RandomDecimal

Command

WeightedChoice

Hash

HashInUri

HashInCSV

RandDate

Duration

DateParser

RandomDuration

Incremental

Sequence

Replacement

Template

Template each

Fromjson

Remove

Add

Add-Transient

FluxURI

FromCache

FF1

Sha3

Range

Pipe

Apply

Luhn

Markov

Transcode

FindInCSV

Input CSV

Timeline

Constraints

Epsilon

XML

Parsing-XML-files

Parsing Parquet files

Example

Masking Configuration (masking.yml)

Resulting Masked Parquet File

pimo:// scheme

Flow chart

Visual Studio Code

Contributors

Licence

About

Topics

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks

Releases 41

Contributors 9

Languages

Masking Configuration (`masking.yml`)

`pimo://` scheme