SIS-IMPORTER

Backlog

What is importer?

Importer fetches data from Sisu using the export apis. See example of export api, importer uses only since (modification ordinal) and limit parameters. The fetched data is processed in mankelis and saved to PostgreSQL db.

New data from Sisu is fetched once an hour.

Architecture of importer

For step by step overview of how the importer works, see the following READMEs

• Fetcher's README
• Mankeli's README

Caveats

• Document state: documentState field present in all data defines should the data be used or not. Mainly importer ignores any other data than ACTIVE (DRAFT and DELETED in most cases should be ignored).

• Snapshot vs regular data: Snapshot data is described as follows:

  Start of validity for the snapshot. End of validity is defined by the snapshotDateTime of a possible later snapshot

  Meaning one needs to find the most recent non-future snapshot date time and active document state to find the correct instance of the given object. For normal data, the same is greatest modification ordinal (=newest version of the object) with active document state.

  With snapshot data in the db is multiple rows (versions) of the object whereas with the regular data there is only present the latest version.

• Manage production and staging environments: use scripts run_scaled.sh and wipe_ordinals.sh located in the server to manage importer. It is important that mankelis are scaled properly in production.

Tricks & Tips

• https://importer.cs.helsinkif/exploder/reset/:table?token= deletes a single table and triggers fetch. See tables here and token from the server.
• https://importer.cs.helsinkif/exploder/force_update?token= triggers a fetch for all tables
• To add new fields to be fetched from Sisu: Modify message handlers. Remember to add any new columns to models importer-mankeli models.
• To fetch new model from Sisu: Create new message handler and service. Finally add new service to index.js and test locally that importing works.
• Debug data coming from specific channel:
  • First, comment out any other channels in the serviceIds folder in this file.
  • Then, go to importer-mankeli/src/debug. Add a custom debug handler into the customHandlers-folder: view the existing custom handlers for how-to.
  • Add your handler to the index.js file in the debug-folder (in the array).
  • Run with "npm start". It will log stuff into console. If you want, make it write the relevant logs to a file for much better experience. (And please push that code to repo too)
  • Remember to remove any sensitive identifiers in your matcher before pushing code. Also remove the handler from the array, but you can leave the file in the repository if it may be useful in the future.

API catalogs

• ORI (student data) https://sis-helsinki.funidata.fi/ori/docs/index.html
• KORI (course data) https://sis-helsinki.funidata.fi/kori/docs/index.html
• ILMO (course enrolments) https://sis-helsinki.funidata.fi/ilmo/docs/index.html
• OSUVA (study plans) https://sis-helsinki.funidata.fi/osuva/docs/index.html
• ARTO (assessments) https://sis-helsinki.funidata.fi/arto/docs/index.html

Local development

1. See initial setup in https://version.helsinki.fi/toska/dokumentaatio/-/blob/ma