JOQULAR Internals

Introduction

We start with an overview of JOQULAR Database Indexing, followed by a discussion of Prototype Enhancement, and close with commentary on Constructor Re-Writes.

Database Indexing

Every object added to a JOQULAR database is fully indexed using a nested JavaScript object first by its constructor, then by its keys, then value types, then (except in the case of functions) values. The terminal value key stores a final object whose keys are the unique ids of the objects that are instances of the constructor with the referenced keys, value, and type. Indexed functions also store the ids of the objects on which they can be invoked, e.g.

function MyClass(config) {
   for(var key in config) { this[key] = config[key] };
}
(MyClass.prototype.isMinor = function() { return this.age<21; }).predicate = true;
MyClass = JOQULAR.createIndex(MyClass);
new MyClass({name: 'Joe', age: 10});
new MyClass({name: 'Bill' age: 21});
new MyClass({name: 'Mary' age: 10});

effectively creates the index

MyClass.ids = {0: {name: 'Joe', age: 10}, 1: {name: 'Bill' age: 21}, 2: {name: 'Mary' age: 10}};
MyClass.index = {name: {string: {Joe: [0]}}, age: {number: {10: [0,2], 21: [1]}}, isMinor: [1,2,3] }};

Not shown above is the fact that the objects associated with MyClass.ids are actual instances and do support the isMinor function.

When values are inserted, the keys are just added to the index, which is extremely fast. At query time, the value keys below the type keys are coerced into the type, sorted in an ascending manner, cached, and have the operators specified in the JOQULAR match pattern applied to them. The application of operators takes advantage of the sort, e.g. eq is actually tested using a binary search. The sorted values are cached until a new value is added to the index at the named key, e.g. adding new MyClass({name: 'John'}) will flush the cache for the name key but not the age key.

Patterns are walked until their terminal nodes, which will always be a primitive value. Or, until a function is encountered, in which case a non-primitive can be tested. Note, even if you put an object in a pattern, it will ultimately resolve to primitives. For function properties stored as keys, each object stored with the key is de-referenced to the object itself and then the named function is invoked on the object.

If the function length property is 0 and it is marked as a data provider, then the pattern key value is compared to the return value. If the length is greater than 0 and the function is marked as a predicate, then the pattern value is provided as an argument and the return value is tested for true/false. Due to JSON constraints, arguments must always be an array if the predicate takes multiple arguments. Functions that are not marked at providers or predicates are not added to the index.

If an object fails to pass a predicate test, not only is it not added to the result set, its id is cached for the duration of the query process so it does not need to be evaluated again if it is associated with another key in the pattern.

Object deletions are just initially applied to a central id to object map. Hence, they are also very fast. Although this leaves deleted object ids within the index, the cost is marginal. When queries are resolving, a check is made to ensure the object associated with the id still exists prior to testing the predicate. If it does not exist, the index is updated.

Prototype Enhancement

In order to normalize the query language, extensive use of prototype enhancement is made. This ensures that the naming and semantics of such things as set operations can remain consistent across Arrays, Sets, Durations, TimeSpans and other classes of objects.

The risks associated with this are relatively low so long as standards are tracked for potential enhancements to core classes. For example, the presence of forEach on Arrays and Sets, but the lack of every and some on Sets while they are present on Arrays is puzzling. We expect the powers that be will address this at some point. In the mean time, we have added these capabilities to JOQULAR enhanced Sets.

Constructor Re-Writes

Constructor re-writes are pretty tricky and can obviously cause lots of issues. This is part of the reason we did not just adopt Moment.js for our Time, TimeSpan, and Duration capability. However, in order to have real-time synchronization of the JOQULAR index with JavaScript objects we felt we had to re-write constructors for classes being indexed. We are looking an ways to remove these re-writes in a future releases by using some combination of Proxy and Observer capability. The main challenge we have is intercepting calls to object creation so that we can make an initial addition to the index and set an Observer on the object for updates. We welcome suggestions on how to accomplish this in another manner!