File specific rules

Associated directory: 12-file-specific-rules

This section was contributed by josephmturner

Motivation: transactions in the CSV file can't be handled by a general rule

Sometimes you'll have transactions which require very specific rules in order to be categorized -- rules so specific that they would, in fact, match this particular transaction only and nothing else. For example, your bank might provide you with transactions that lack information about where the money came from or went to:

$ cat import/lloyds/in/12345678_20171225_0003.csv

Transaction Date,Transaction Type,Sort Code,Account Number,Transaction Description,Debit Amount,Credit Amount,Balance,
10/04/2017,DEB,'12-34-56,12345678,CHECK #0001523,,100,1600.0

After in2csv cleans the csv, they'll look like:

$ cat import/lloyds/csv/12345678_20171225_0003.csv

Transaction Date,Transaction Type,Sort Code,Account Number,Transaction Description,Debit Amount,Credit Amount,Balance,
10/04/2017,DEB,'12-34-56,assets:Lloyds:savings,CHECK #0001523,,100,1600.0

It doesn't makes sense to write a single rule which categorizes all checks (as they would not, presumably, be written to a single payee), so we'll want to write rules that look approximately like this:

if
10/04/2017,DEB,'12-34-56,assets:Lloyds:savings,CHECK #0001523,,100,1600.0
  account2 income:tutoring

This rule is "hyper-specific" as it should only ever apply to a single transaction. Hyper-specific rules are also useful for categorizing transactions with a vendor which are sometimes -- but not always -- deductible.

Where to store hyperspecific rules

You could store hyperspecific rules in the institution-level rules file (import/lloyds/lloyds.rules), but as time goes on, that rules file will fill up with hyperspecific rules which are not relevant to subsequent years/csv imports.

It might be better to store hyper-specific rules in .rules files which would only be applied to a single CSV file.

Those rules files will be stored in a rules directory alongside csv/, journal/, etc.:

import
└── lloyds
    ├── in
    │   ├── 12345678_20171225_0003.csv       - newly downloaded files
    │   └── ...
    ├── csv
    │   ├── 12345678_20171225_0003.csv       - scrubbed and cleaned
    │   └── ...
    ├── journal
    │   ├── 12345678_20171225_0003.journal   - journals that we expect to generate
    │   └── ...
    ├── in2csv                               - conversion script to produce ./csv/*
    ├── csv2journal                          - conversion script to produce ./journal/*
    ├── lloyds.rules                         - general CSV conversion rules
    └── rules
        ├── 12345678_20171225_0003.rules     - rules which are applied to specific CSVs
        └── ...

Each rules file in import/rules/ will look something like:

$ cat import/lloyds/rules/12345678_20171225_0003.rules

# include the general CSV conversion rules which are applicable to all Lloyds CSVs.
# Put the include statement at the top of the file, so that the rules that
# you write below take precedence over the general rules
include ../lloyds.rules

# Optionally, add hyper-specific rules and any other rules which you want applied only to this CSV
if
10/04/2017,DEB,'12-34-56,assets:Lloyds:savings,CHECK #0001523,,100,1600.0
  account2 income:tutoring

Update the scripts

Update csv2journal with the new locations of the rules files.

$ cat import/lloyds/csv2journal

#!/bin/bash
hledger print --rules-file "./rules/$(basename "$1" .csv).rules" -f "$1"

Update the dependencies inside export.hs:

extraDeps file
  | "//lloyds//*.journal" ?== file   =
      let basename = takeBaseName file
      in
        ["./rules/" ++ basename ++ ".rules", "lloyds.rules", "generated.rules"]
  | otherwise = []

Now all you have to do is put your hyper-specific rules in the specific rules file and your shared rules file (lloyds.rules) will remain clean!

Other ways to handle transactions which require manual categorization

One naive solution might be to modify the journal file directly. However, this option is not available with our setup, as your changes will be overwritten whenever the journal file's corresponding .rules file or CSV file changes (and you run export.sh).

Another option is to create an intermediate account for check transactions, and then have a separate journal file (that you maintain by hand) for proper categorization of those transactions (so that the "checks" journal has a final balance of zero). These intermediate accounts could work just like the transfer accounts described in the chapter on adding more accounts.

See the changes in 12-file-specific-rules.

Next steps

Tax returns