Code structure

Read this in conjunction with https://github.com/nsip/nias3

The nias3 code takes in arbitrary XML and JSON files, through file drop box or API, and decomposes them into tuples preserving the structure of the source object. It pushes the tuples into nias3-engine (the legacy version of n3-transport), through an HTTP client. It also has code which reinflates a set of tuples fetched from nias3-engine into XML or JSON objects. For SIF in particular, it contains structs which allow the XML to be serialised in correct order.

The API that the code exposes is given in the README. The SIF XML server endpoints expose the CRUD functionality of a simple SIF server, and will need to be preserved.

test.sh is a test script that confirms that the nias3 code does all SIF CRUD operations. It uses top-level XML files.

nswdig.xml is the test file for the Digital Classroom project, consisting of timetabling and student roster data for four schools, 5 MB in size.

• main.go runs the web server for nias3. It has commented out calls to ingest nswdig.xml as a local file, and to ingest a SIF file from a server through an HTTP GET.

• The in folder is the folder to which input files are added for the file watcher to pick up for ingest

• config/config.go reads the NIAS config file, n3.toml. Currently it only contains one field, the port that nias3-engine reads from (so you won't need it for the port to n3-transport)

• sifxml/*go contains structs for converting tuples back into SIF XML. This code is machine generated, from https://github.com/nsip/sifxml2go, and should not be changed. As an alternative, we could attach the XSLT transforms from https://github.com/nsip/sifxml2pescjson; but I doubt XSLT is going to be more performant than Go structs

• xml2triples/switch.go contains code that selects the correct sifxml/*go method for a given SIF object. It too is machine-generated and not to be changed.

• webserver/webserver.go contains the API interface of NIAS3, which receives JSON and XML objects and sends them to NIAS-engine for storage. It implements basic SIF CRUD behaviour through the /sifxml endpoints; those endpoints post or get tuples through the main NIAS3 code. It also implements hard-coded API calls specific to the Digital Classroom project; these will be replaced eventually by GraphQL queries.

The file also contains:

• directoryWatcher(), which watches a file directory for changes, and posts any updated files to NIAS. This relies on an external library, and seems to be flaky.

The web service tries to speed up its POST/PUT interaction with the back end by batching tuples in batches of 1000, and sending them to the back end asynchronously and concurrently (simultSendTriples, sendTriplesAsync).

The main routine sendReaderToDataStore() reads XML from an io.Reader, breaks the XML down into individual objects, and sends a byte slice of each XML object to a different routine that breaks it down into tuples and processes them, depending on the CRUD operation. sendReaderToDataStoreJSON() does the same for JSON.

All SIF objects have a GUID, and in general all objects ingested by NIAS, JSON or XML, are supposed to be addressable through a unique identifier.

• xml2triples/mxj.go is the code that breaks JSON and XML down into triples and sends them to NIAS-engine, and that receives tuples from NIAS-engine and reassembles them into JSON and XML. It is the code that webserver/webserver.go uses to interact with the tuples store(s).

It interacts with nias3-engine through an HTTP client (initClient), which allows up to 100 simultaneous connections. This code will be superseded.

Triples are generated through the MXJ library, which maps between XML and Go maps (Map2SIFXML); JSON is dealt with as an intermediate stage between XML and Go maps.

This code presupposes not only a mechanism for posting triples to the write model, but also a mechanism for querying triples from a read model. Reading tuples from the read model may not be an immediate priority for implementation (n3-transport does not have it), but it will be needed to allow Read and Update SIF operations. The nias3-engine implementation uses a Hexastore to respond to those queries (key/value store with permutations of SPO as its keys), but in principle Influx can respond to them too (getTuplesInflux communicates with an Influx API exposed by nias3-engine).

In the following, I separate the write queries from the read queries; the write queries are immediate priority.

Write:

• sends triples to NIAS3-engine, synchronously and asynchronously (send_triples, SendTriplesAsync, Sif2Xapi).
• deletes all triples with a given subject from NIAS3-engine (DeleteTriplesForRefId, DeleteTriplesForRefIdAsync): realised as post of triples with empty object.
• posts a SIF object on NIAS-engine, by posting all the tuples from that object to NIAS3-engine (StoreXMLasDBtriples, StoreXMLasDBtriplesAsync)
• posts a JSON object on NIAS3-engine (StoreJSONasDBtriplesAsync)

Read:

• checks whether a triple, or part of a triple, exists on NIAS3-engine: given S, SP, or SPO, it returns whether a matching triple exists (hasKey, hasKeyAsync)
• checks whether a triple exists on NIAS3-engine with the given S and a non-empty object (existsRefId)
• retrieve a triple from NIAS3-engine: given S, SP, or SPO, it retrieves the entire matching triple
• updates a SIF object on NIAS3-engine, by deleting all existing tuples with the same ID as the object, then posting all the tuples from that object to NIAS3-engine. (UpdateFullXMLasDBtriples, UpdateFullXMLasDBtriplesAsync). It is critical to preserve the order of deletes and creates for tuples on NIAS3-engine.
• patches a SIF object on NIAS3-engine, deleting only the tuples with the same Subject and Predicate (XPath) as are included in the provided object (UpdatePartialXMLasDBtriples, UpdatePartialXMLasDBtriplesAsync)
• retrieve all tuples for a given subject from NIAS3-engine, and create the corresponding SIF object (DbTriples2XML)
• gets all SIF objects of the given object class from NIAS3-engine (GetAllXMLByObject: done by matching on the XPath of tuples on NIAS3-engine)
• convert a slice of tuples into a JSON object (Tuples2JSON)