Open Data should be easier.
This project generates an Open Data catalog as a static html site. It can be hosted on Github, S3, or any generic server. The schema is declared using JSON Schema in Yaml files. The metadata and all content are stored in JSON files.
- CKAN / DKAN style data catalog
- Full-text search using Elasticlunr
- Easy to update and override schema
- Simple data workflows
- Harvesting from outside catalogs and data sources
- Easy-to-update or override React front end
- Full administrative interfeace using Interra Catalog Admin
- Low-cost hosting using S3, github pages, or other static file server
- Additional features as microservices
Traditional catalogs use a SQL database (MySQL, PostgreSQL), webserver (Nginx, Apache), and search engine (SOLR, ElasticSearch) to provide a simple catalog. Many catalogs consist of metadata for less than 1,000 datasets with some additional features such as a datastore or visualization library. Interra drastically reduces the overhead for publishing data, especially for catalogs with under 1,000 records, because a database, webserver (if published to S3), and search engine are not necessary. Interra uses Elasticlunr as a search index which offers some features search engines such as boosts and query tokens without the overhead of that service.
Storing data as static files, or optionally in a NoSQL store like MongoDB, drastically reduces the overhead for managing and publishing data. Catalogs using SQL databases rely on complex abstractions for translating data from open formats to an internal state and then back to open formats. For example, many data workflows involve ingesting JSON files, writing them to rows and tables in a SQL database using an ORM, changing the data, then exporting them back to JSON files. These catalogs also require a web server and scripting language to receive requests and serve JSON.
Interra Catalog takes a different approach. Data is saved in a document store which is by default a file system. The only change in the way the data is stored versus consumed are internal references between docs. Updating data between harvesting, storing, and publishing is done with a small set of javascript functions. There is little state to understand and maintain.
Documents are exported as JSON files and rendered using a React app. Search is provided using Elasticlunr.
Additional catalog features are added as microservices. Interra Catalog Admin offers a user-interface for creating, editing, and updating data. A datastore is coming soon using CARTO. Data visualization dashboards will be added with React Dash.
The schema is extremely flexible and easy to update. Adding a new field is as simple as adding lines to a configuration file. Example schema's include Project Open Data's schema however any schema that has the concept of a "primary collection" such as a "dataset". See the test suite for an example.
Roadmap soon to be shared. This project is still in early development.
Currently have an early POC here: http://healthdata-example.interra.io.
To get started install node modules:
npm installThe default search uses ElasticLunr. We require the latest 0.9.6 version however the current version on npm is 0.9.5. To get the latest version:
rm -r node_modules/elasticlunr; git clone git@github.com:weixsong/elasticlunr.js.git node_modules/elasticlunr;To create a new site start the plop generator:
node_modules/.bin/plopDocumentation can be found at Read the docs.
Requires node v6+. This is built to run using Linux. Windows support will not be provided.