jj

go get github.com/lukechampine/jj

jj implements a JSON transaction journal. It enables efficient ACID transactions on JSON objects. If your disk operations require something faster than rewriting a whole file on each save, but you aren't ready to design a full database schema, jj might be for you.

Each Journal represents a single JSON object. The object is serialized as an "initial object" followed by a series of update sets, one per line. Each update specifies a field and a modification. See the Update section for a full specification.

During operation, the object is first loaded by reading the file and applying each update to the initial object. It is subsequently modified by appending update sets to the file, one per line. At any time, a "checkpoint" may be created, which clears the Journal and starts over with a new initial object. This allows for compaction of the Journal file.

In the event of power failure or other serious disruption, the most recent update set may be only partially written. Partially written update sets are simply ignored when reading the Journal. Individual updates may also be ignored if they are malformed, though other updates in the set may be applied. See the Update section for an explanation of malformed updates.

Example

// create initial object
var obj struct {
	Foo struct {
		Bar struct {
			Baz []int `json:"baz"`
		} `json:"bar"`
		Quux int `json:"quux"`
	} `json:"foo"`
}
obj.Foo.Bar.Baz = []int{1,2,3}
obj.Foo.Quux = 6

// create journal with initial object
// (error handling omitted)
j, _ := jj.OpenJournal("myjournal", obj)

// record a set of updates to the object
_ = j.Update([]jj.Update{
	jj.NewUpdate("foo.bar.baz.2", 7),
	jj.NewUpdate("foo.quux", 7),
})

// close and reopen the journal, applying the updates
_ = j.Close()
j, _ = jj.OpenJournal("myjournal", &obj)
println(obj.Foo.Bar.Baz[2]) // -> 7
println(obj.Foo.Quux)       // -> 7

Updates

An Update is a modification of a path in a JSON object. A "path" in this context means an object or array element. Syntactically, a path is a set of accessors joined by the . character. An accessor is either an object key or an array index. For example, given this object:

{
	"foo": {
		"bars": [
			{"baz":3}
		]
	}
}

The following path accesses the value "3":

foo.bars.0.baz

The path is accompanied by a new object. Thus, to increment the value "3" in the above object, we would use the following Update:

{
	"p": "foo.bars.0.baz",
	"v": 4
}

All permutations of the Update object are legal. However, malformed updates are ignored during application. An Update is considered malformed in three circumstances:

• Its Path references an element that does not exist at application time. This includes out-of-bounds array indices.
• Its Path contains invalid characters (e.g. "). See the JSON spec.
• Value contains invalid JSON or is empty.

Other special cases are handled as follows:

• If Path is "", the entire object is replaced.
• If an object contains duplicate keys, the first key encountered is used.

Finally, to enable efficient array updates, the length of the array (at application time) may be used as a special array index. When this index is the last accessor in Path, Value will be appended to the end of the array. If the index is not the last accessor, the Update is considered malformed (and thus is ignored).

Caveats

An important aspect of jj is that you cannot "read" from the journal; you can only apply updates. The only time you retrieve data from the journal is when loading it from disk, usually during initialization. This means that you must keep your in-memory copy of the data in sync with the journal.