-
Notifications
You must be signed in to change notification settings - Fork 4
/
Copy pathREADME.md
2202 lines (1746 loc) · 77.5 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
# JSKOS Server
[](https://github.com/gbv/jskos-server/actions/workflows/test.yml)
[](https://github.com/gbv/jskos-server)
[](https://stats.uptimerobot.com/qZQx1iYZY/780815088)
[](https://github.com/RichardLitt/standard-readme)
> Web service to access [JSKOS] data.
JSKOS Server implements the JSKOS API web service and storage for [JSKOS] data such as controlled vocabularies, concepts, and concept mappings.
## Table of Contents <!-- omit in toc -->
- [Install](#install)
- [Docker](#docker)
- [Dependencies](#dependencies)
- [Clone and Install](#clone-and-install)
- [Configuration](#configuration)
- [Access control](#access-control)
- [Authentication](#authentication)
- [Data Import](#data-import)
- [Usage](#usage)
- [Run Server](#run-server)
- [Run Tests](#run-tests)
- [Run Supplemental Scripts](#run-supplemental-scripts)
- [API](#api)
- [GET /status](#get-status)
- [GET /checkAuth](#get-checkauth)
- [POST /validate](#post-validate)
- [GET /validate](#get-validate)
- [GET /concordances](#get-concordances)
- [GET /concordances/:\_id](#get-concordances_id)
- [POST /concordances](#post-concordances)
- [PUT /concordances/:\_id](#put-concordances_id)
- [PATCH /concordances/:\_id](#patch-concordances_id)
- [DELETE /concordances/:\_id](#delete-concordances_id)
- [GET /mappings](#get-mappings)
- [GET /mappings/suggest](#get-mappingssuggest)
- [GET /mappings/voc](#get-mappingsvoc)
- [GET /mappings/infer](#get-mappingsinfer)
- [GET /mappings/:\_id](#get-mappings_id)
- [POST /mappings](#post-mappings)
- [PUT /mappings/:\_id](#put-mappings_id)
- [PATCH /mappings/:\_id](#patch-mappings_id)
- [DELETE /mappings/:\_id](#delete-mappings_id)
- [GET /voc](#get-voc)
- [POST /voc](#post-voc)
- [PUT /voc](#put-voc)
- [DELETE /voc](#delete-voc)
- [GET /voc/top](#get-voctop)
- [GET /voc/concepts](#get-vocconcepts)
- [DELETE /voc/concepts](#delete-vocconcepts)
- [GET /voc/suggest](#get-vocsuggest)
- [GET /voc/search](#get-vocsearch)
- [GET /data](#get-data)
- [POST /data](#post-data)
- [PUT /data](#put-data)
- [DELETE /data](#delete-data)
- [GET /narrower](#get-narrower)
- [GET /ancestors](#get-ancestors)
- [GET /suggest](#get-suggest)
- [GET /search](#get-search)
- [GET /annotations](#get-annotations)
- [GET /annotations/:\_id](#get-annotations_id)
- [POST /annotations](#post-annotations)
- [PUT /annotations/:\_id](#put-annotations_id)
- [PATCH /annotations/:\_id](#patch-annotations_id)
- [DELETE /annotations/:\_id](#delete-annotations_id)
- [Errors](#errors)
- [Deployment](#deployment)
- [Notes about depolyment on Ubuntu](#notes-about-depolyment-on-ubuntu)
- [Update an instances deployed with PM2](#update-an-instances-deployed-with-pm2)
- [Daily Import](#daily-import)
- [Running Behind a Reverse Proxy](#running-behind-a-reverse-proxy)
- [Related works](#related-works)
- [Maintainers](#maintainers)
- [Contribute](#contribute)
- [Publish](#publish)
- [License](#license)
## Install
### Docker
The easiest way to install and use JSKOS Server is with Docker and Docker Compose. Please refer to the [documentation on Docker Hub](https://hub.docker.com/r/coliconc/jskos-server) for more information and instructions.
### Dependencies
You need Node.js 12 or later to run jskos-server (14 or 16 recommended). You need to have access to a [MongoDB database](https://docs.mongodb.com/manual/installation/).
### Clone and Install
```bash
git clone https://github.com/gbv/jskos-server.git
cd jskos-server
npm ci
```
### Configuration
You can customize the application settings via a configuration file. By default, this configuration file resides in `config/config.json`. However, it is possible to adjust this path via the `CONFIG_FILE` environment variable. Note that the given path has to be either absolute (i.e. starting with `/`) or relative to the `config/` folder (i.e. it defaults to `./config.json`). **Note** that the path to the configuration file needs to be valid and writable because a `namespace` key will be generated and written to the file if it doesn't currently exist.
Currently, there are only two environment variables:
- `NODE_ENV` - either `development` (default) or `production`; currently, the only difference is that in `production`, HTTPS URIs are forced for entities created on POST requests.
- `CONFIG_FILE` - alternate path to a configuration file, relative to the `config/` folder; defaults to `./config.json`.
You can either provide the environment variables during the command to start the server, or in a `.env` file in the root folder.
It is also possible to have more specific configuration based on the environment. These are set in `config/config.development.json` or `config/config.production.json`. Values from these files have precedent over the user configuration.
All missing keys will be defaulted from `config/config.default.json`:
```json
{
"verbosity": "warn",
"baseUrl": null,
"title": "JSKOS Server",
"version": null,
"port": 3000,
"proxies": [],
"mongo": {
"user": "",
"pass": "",
"host": "localhost",
"port": 27017,
"db": "jskos-server",
"options": {
"reconnectTries": 5,
"reconnectInterval": 1000,
"useNewUrlParser": true
}
},
"auth": {
"algorithm": "RS256",
"key": null
},
"anonymous": false,
"schemes": true,
"concepts": true,
"mappings": {
"read": {
"auth": false
},
"create": {
"auth": true
},
"update": {
"auth": true,
"crossUser": false
},
"delete": {
"auth": true,
"crossUser": false
},
"fromSchemeWhitelist": null,
"toSchemeWhitelist": null,
"cardinality": "1-to-n"
},
"concordances": true,
"annotations": {
"read": {
"auth": false
},
"create": {
"auth": true
},
"update": {
"auth": true,
"crossUser": false
},
"delete": {
"auth": true,
"crossUser": false
}
},
"identityProviders": null,
"identities": null,
"ips": null
}
```
The provided configuration files (user config and environment config) will be validated with the provided [JSON Schema](https://json-schema.org) file under `config/config.schema.json` (public URI: https://gbv.github.io/jskos-server/status.schema.json). If validation fails, **JSON Server will refuse to start!** Please check whether your configuration is correct after each change. If there is something wrong, the console output will try to provide you with enough detail to fix the issue.
If you are [running jskos-server behind a reverse proxy](#running-behind-a-reverse-proxy), it is necessary to provide the `baseUrl` key as well as the `proxies` key in your configuration (example for our production API):**
See also:
```json
{
"baseUrl": "https://coli-conc.gbv.de/api/",
"proxies": ["123.456.789.101", "234.567.891.011"]
}
```
With the keys `schemes`, `concepts`, `mappings`, `concordances`, and `annotations`, you can configure whether endpoints related to the specific functionality should be available. A minimal configuration file to just server read-only vocabulary and concept information could look like this:
```json
{
"mappings": false,
"annotations": false,
"concordances": false
}
```
Available actions for `schemes`, `concepts`, `mappings`, and `annotations` are `read`, `create`, `update`, and `delete`. By default, all types can be read, while `mappings` and `annotations` can be created, updated, and deleted with authentication. Explanantions for additional options:
- **`auth`**: Boolean. Can be defined only on actions. Defines whether access will require [authentication via JWT](#authentication). By default `false` for `read`, and `true` for all other actions.
- **`crossUser`**: Boolean or list of URI strings. Can be defined only on `update` and `delete` actions when `auth` is `true`. Defines whether it is possible to edit an entity from a different user than the authenticated one (`true` = allowed for all users, list = allowed for specified user URIs). `false` by default.
- **`anonymous`**: Boolean. Can be defined on any level (deeper levels will take the values from higher levels if necessary\*). If set, no creator and contributor is saved. `false` by default.
- **`cardinality`**: String. Can be defined only on type `mappings`. Currently possible values: `1-to-n` (default), `1-to-1`. If `1-to-1` is configured, mappings with multiple concepts in `to` will be rejected.
- **`identities`**: List of URI strings. Can be defined on any level (deeper levels will take the values from higher levels if necessary\*). If set, an action with `auth` set to `true` can only be used by users with an URI given in the list. `null` by default (no restrictions).
- **`identityProviders`**: List of strings. Can be defined on any level (deeper levels will take the values from higher levels if necessary\*). If set, an action can only be used by users who have that identity associated with them. `null` by default (no restrictions).
- **`ips`**: List of strings. Strings can be IPv4 addresses (e.g. `127.0.0.1`, `123.234.123.234`) or CIDR ranges (e.g. `192.168.0.1/24`). Can be defined on any level (deeper levels will take the values from higher levels if necessary\*). If set, an action can only be used by clients with a whitelisted IP address. `null` by default (no restrictions). Note: An empty array will allow all IPs. Note: This property will be removed for security reasons when accessing [GET /status](#get-status) (meaning that clients will not be able to see the whitelisted IP addresses).
- **`fromSchemeWhitelist`/`toSchemeWhitelist`**: Can be defined only on type `mappings`. List of scheme objects that are allowed for `fromScheme`/`toScheme` respectively. `null` allows all schemes.
\* Only applies to actions `create`, `update`, and `delete`.
Note that any properties not mentioned here are not allowed!
### Access control
The rights to `read`, `create`, `update` and `delete` entities via API can be controlled via several configuration settings described above ([data import](#data-import) is not limited by these restrictions):
* Restricted access via `ips` is always applied *in addition* to other settings
* Without [authentication](#authentication) (`auth` set to `false`) the server does not know about user accounts. In this case the `creator` and `contributor` fields of an object can be set without limitations (default) or they are ignored when `anonymous` is set to `true`.
* With authentication an action can be limited to accounts listed in `identities` (if set). Rights to `create`, `update`, and `delete` entities can further depend on two controls:
1. value of `creator` and `contributor` of a superordinated object. Concepts always belong to vocabularies via `inScheme` or `topConceptOf` and mappings can belong to concordances via `partOf`.
2. settings of `crossUser` together with value of `creator` and `contributor` of the object
The first control is only checked if it has a superordinated object with `contributor` and/or `creator`. This can only be the case for mappings and concepts. The connection to a superordinated object is checked on both the stored object and its modified value, so moving a mapping from one concordance to another is only allowed if access is granted for both. The authenticated user must be listed as `creator` or `contributor` of the superordinated object to pass this control.
The second control is only checked when the first control cannot be applied and only on authenticated actions `update` or `delete` where `anonymous` is set to `false` (this is the default). With `crossUser` set to `false`, the authenticated user must be listed as `creator` of the stored object. With `crossUser` set to `true` any authenticated user (optionally limited to those listed in `identities`) can `update` or `delete` the object.
For authenticated actions with `anonymous` being `false` creation of a new object will always set its initial `creator` to the autenticated user and `update` of an object will always add the user to `contributor` unless it is already included as `creator` or `contributor`. Further modification of `creator` and `contributor` (removal and addition of entries) is limited to vocabularies and concordance by authenticated users listed as `creator` of the object.
Here are some helpful example presets for configuration of "concordances, "mappings", or "annotations".
**Read-only access (does not make sense for annotations):**
```json
{
"read": {
"auth": false
}
}
```
**Anyone can create, but only logged-in users can update and delete (and only their own items):**
```json
{
"read": {
"auth": false
},
"create": {
"auth": false
},
"update": {
"auth": true,
"crossUser": false
},
"delete": {
"auth": true,
"crossUser": false
}
}
```
**Anyone can create, logged-in users can update (independent of creator), logged-in users can delete their own items:**
```json
{
"read": {
"auth": false
},
"create": {
"auth": false
},
"update": {
"auth": true,
"crossUser": true
},
"delete": {
"auth": true,
"crossUser": false
}
}
```
**Anyone can create, as well as update and delete, independent of creator:**
```json
{
"read": {
"auth": false
},
"create": {
"auth": false
},
"update": {
"auth": false,
"crossUser": true
},
"delete": {
"auth": false,
"crossUser": true
}
}
```
If write access for concept schemes and/or concepts is necessary, it is recommended that they are secured by only allowing certain users (via `identities`) or only allowing certain IP addresses (via `ips`):
**Only user with URI `https://coli-conc.gbv.de/login/users/c0c1914a-f9d6-4b92-a624-bf44118b6619` can write:**
```json
{
"read": {
"auth": false
},
"create": {
"auth": true,
"identities": ["https://coli-conc.gbv.de/login/users/c0c1914a-f9d6-4b92-a624-bf44118b6619"]
},
"update": {
"auth": true,
"identities": ["https://coli-conc.gbv.de/login/users/c0c1914a-f9d6-4b92-a624-bf44118b6619"]
},
"delete": {
"auth": true,
"identities": ["https://coli-conc.gbv.de/login/users/c0c1914a-f9d6-4b92-a624-bf44118b6619"]
}
}
```
**Only localhost can write:**
```json
{
"read": {
"auth": false
},
"create": {
"auth": false,
"ips": ["127.0.0.1"]
},
"update": {
"auth": false,
"ips": ["127.0.0.1"]
},
"delete": {
"auth": false,
"ips": ["127.0.0.1"]
}
}
```
Note that `auth` is set to `false` because it refers to authentication via JWT. The IP filter is separate from that. An even more secure way would be to use both JWT authentication with an `identities` filter as well as an IP filter.
**Only user with URI `https://coli-conc.gbv.de/login/users/c0c1914a-f9d6-4b92-a624-bf44118b6619` can create, but others can update/delete if they are creator/contributor of an entity:**
```json
{
"read": {
"auth": false
},
"create": {
"auth": true,
"identities": ["https://coli-conc.gbv.de/login/users/c0c1914a-f9d6-4b92-a624-bf44118b6619"]
},
"update": {
"auth": true
},
"delete": {
"auth": true
}
}
```
A configuration like this will be used to handle concordances in Cocoda. Only selected accounts will be able to create new concordances, but they will be able to add other accounts as creator/contributor so that those accounts will be able to assign mappings to the concordance and edit mappings that belong to the concordance.
### Authentication
It is possible to limit certain actions to authenticated users, indicated by the `auth` option (see [example configurations above](#access-control)). Authorization is performed via JWTs ([JSON Web Tokens](https://jwt.io/)). To configure authentication, you need to provide the JWT algorithm and the key/secret in the configuration file, like this:
```json
"auth": {
"algorithm": "RS256",
"key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA57ZWRoOjXYTQ9yujxAu7\ne3k4+JRBAqGdDVIRRq5vXB2D5nJBIhQjVjylumn+QnTX/MdZx8qn7X96npUwHwIh\nylCgUmsYXcjP08X/AXEcP5bPOkgBBCKjWmcm+p01RQSOM0nSptyxpyXzr2ppWe1b\nuYdRYDWj+JV7vm+jJA4NiFv4UnAhoG5lRATADzu0/6wpMK3dVMBL7L0jQoV5xBAb\nLADOy5hD9XEII3VPkUqDGIKM+Z24flkCIf0lQ7FjsoZ2mmM1SZJ5vPDcjMKreFkX\ncWlcwGHN0PUWZWLhb7c8yYa1rauMcwFwv0d2XyOEfgkqEJdCh8mVT/5jR48D2PNG\ncwIDAQAB\n-----END PUBLIC KEY-----\n"
}
```
The JWT has to be provided as a Bearer token in the authorization header, e.g. `Authorization: Bearer <token>`. Currently, all authorized endpoints will be accessible (although `PUT`/`PATCH`/`DELETE` are limited to the user who created the object by default), but later it will be possible to set scopes for certain users (see [#47](https://github.com/gbv/jskos-server/issues/47)).
The authentication is designed to be used together with an instance of [login-server], but it is also possible to use your own JWTs.
#### JWT Example
The recommended Node.js library for creating JWTs is [jsonwebtoken](https://github.com/auth0/node-jsonwebtoken). Note that for simplicity, we are using the HS256 algorithm which is symmetrical. In most cases, it would be better to use RS256 with a libarary like [node-rsa](https://github.com/rzcoder/node-rsa) instead.
Simple config, restricting the `/mappings` endpoint with authentication:
```json
{
"auth": {
"algorithm": "HS256",
"key": "yoursecret"
},
"mappings": {
"read": {
"auth": true
}
}
}
```
Creating a JWT:
```js
const jwt = require("jsonwebtoken")
// Payload is an object containing the user object with an URI:
const data = {
user: { uri: "urn:test:hallo" }
}
// Sign the token with our secret
const token = jwt.sign(data, "yoursecret", {
algorithm: "HS256",
expiresIn: "7d" // valid for 7 days
})
```
Using the token in a request (using curl):
```bash
# Request without header should return ForbiddenAccessError (code 403)
curl localhost:3000/mappings
# Request with header should return JSON data (insert your own token and jskos-server URL of course)
curl -H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJ1c2VyIjp7InVyaSI6InRlc3Q6aGFsbG8ifSwiaWF0IjoxNTg5NTMyNDU3LCJleHAiOjE1OTAxMzcyNTd9.fXIxgS0QyFk9Lvz7Z-fkb4tAueMTSNZ4zAuB6iwePq4" localhost:3000/mappings
```
If you are the only user that is supposed to be authenticated for your instance of jskos-server, you could in theory use something like this to create a token with a long lifetime and use it for all your requests. Please consider the security implications before doing this though.
#### Login Server Example
If you have multiple users using your instance of jskos-server, it is recommended to use [login-server] for authentication. login-server uses the asymmetrical RS256 algorithm by default and will create a public/private key pair on first launch. The public key will be in `./public.key` and you will need that for the configuration:
```json
{
"auth": {
"algorith": "RS256",
"key": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA57ZWRoOjXYTQ9yujxAu7\ne3k4+JRBAqGdDVIRRq5vXB2D5nJBIhQjVjylumn+QnTX/MdZx8qn7X96npUwHwIh\nylCgUmsYXcjP08X/AXEcP5bPOkgBBCKjWmcm+p01RQSOM0nSptyxpyXzr2ppWe1b\nuYdRYDWj+JV7vm+jJA4NiFv4UnAhoG5lRATADzu0/6wpMK3dVMBL7L0jQoV5xBAb\nLADOy5hD9XEII3VPkUqDGIKM+Z24flkCIf0lQ7FjsoZ2mmM1SZJ5vPDcjMKreFkX\ncWlcwGHN0PUWZWLhb7c8yYa1rauMcwFwv0d2XyOEfgkqEJdCh8mVT/5jR48D2PNG\ncwIDAQAB\n-----END PUBLIC KEY-----\n"
}
}
```
After that, you can use [login-client](https://github.com/gbv/login-client) to interact with your login-server instance and receive JWTs. When using WebSockets, login-server will periodically send a new JWT before the previous one expires. You can then use that to authenticate your requests to jskos-server. (An example on how to use login-client can be found in the [source code of login-server](https://github.com/gbv/login-server/blob/master/views/api.ejs).)
For testing your authentication without a full-fledged solution using login-client, you can use http://localhost:3004/token (where `localhost:3004` is your instance of login-server) to request a JWT.
---
Note about previous additional options for `auth`:
- `postAuthRequired`: now covered by `mappings.create.auth`
- `whitelist`: now covered by `identities`
- `allowCrossUserEditing`: now covered by `mappings.update.crossUser` and `mappings.delete.crossUser`
### Data Import
JSKOS Server provides scripts to import JSKOS data into the database or delete data from the database. Right now, mappings, terminologies (concept schemes), concepts, concordances, and annotations, in JSON (object or array of objects) or [NDJSON](http://ndjson.org) format are supported.
#### Import Notes
**About hierarchies within concepts:** Hierarchies are supported. However, only the `broader` field will be used during import. Both `ancestors` and `narrower` will be removed and the respective endpoints ([GET /ancestors](#get-ancestors) and [GET /narrower](#get-narrower)) will dynamically rebuild these properties. That means that when converting your data, please normalize it so that the hierarchy is expressed via the `broader` field in JSKOS.
Example scheme (as JSON object) with concepts in a hierarchy (as NDJSON):
```json
{
"uri": "urn:test:scheme",
"notation": [
"TEST"
],
"uriPattern": "^urn:test:concept-(.+)$"
}
```
```json
{ "topConceptOf": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-a" }
{ "inScheme": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-a.1", "broader": [{ "uri": "urn:test:concept-a" }] }
{ "inScheme": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-a.2", "broader": [{ "uri": "urn:test:concept-a" }] }
{ "topConceptOf": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-b" }
{ "inScheme": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-b.1", "broader": [{ "uri": "urn:test:concept-b" }] }
{ "inScheme": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-b.1.1", "broader": [{ "uri": "urn:test:concept-b.1" }] }
{ "inScheme": [{ "uri": "urn:test:scheme" }], "uri": "urn:test:concept-b.1.2", "broader": [{ "uri": "urn:test:concept-b.1" }] }
```
(Note that a notation for the concepts can be omitted because we have defined `uriPattern` on the concept scheme. Also, we don't need to define `inScheme` for concepts with `topConceptOf`.)
**About the `created` property for concept schemes:** The import script uses the bulk write endpoints to import data. For concept schemes, this means that any existing data for imported schemes will be **overwritten** and replaced with the new data. This includes especially the `created` property which might not exist in your source data and will be set on import if necessary. If you need a consistent `created` date, make sure that your source data already includes this field.
#### Import Script
Examples of using the import script:
```bash
# Create indexes for all types
npm run import -- --indexes
# Import RVK scheme (from coli-conc API)
npm run import -- schemes https://coli-conc.gbv.de/rvk/api/voc
# Import RVK concepts (this will take a while)
npm run import -- concepts https://coli-conc.gbv.de/rvk/data/2019_1/rvko_2019_1.ndjson
# Import coli-conc concordances
npm run import -- concordances https://coli-conc.gbv.de/api/concordances
# Batch import multiple files or URLs
npm run import-batch -- mappings files.txt
# files.txt should contain one file or URL per line with the full path and no escaping.
# You can, for example, store these batch import files in folder `imports` which is ignored in git.
```
**Note: If you have concepts in your database, make sure to run `npm run import -- --indexes` at least once. This will make sure all necessary indexes are created. Without this step, the `/suggest` and `/search` endpoints will not work.**
For more information about the import script, run `npm run import -- --help`.
#### Reset Script
It is also possible to delete entities from the server via the command line. Running the command will first determine what exactly will be deleted and ask you for confirmation:
```bash
# Will delete everything from database
npm run reset
# Will delete mappings from database
npm run reset -- -t mappings
# Will delete all concepts that belong to a certain concept scheme URI
npm run reset -- -s http://uri.gbv.de/terminology/rvk/
# Will delete all mappings that belong to a certain concordance URI
npm run reset -- -c https://gbv.github.io/jskos/context.json
# Will delete entities with certain URIs
npm run reset -- http://rvk.uni-regensburg.de/nt/A http://rvk.uni-regensburg.de/nt/B
# Will show help for more information
npm run reset -- --help
```
For scripting, you can use the `yes` command to skip confirmation. **Make sure you know what you're doing!** Example: `yes | npm run reset -- urn:test:uri`.
## Usage
### Run Server
```bash
# Development server with hot reload and auto reconnect at localhost:3000 (default)
npm run start
# To run the server in production, run this:
NODE_ENV=production node ./server.js
```
### Run Tests
Tests will use the real MongoDB with `-test-${namespace}` appended to the database name.
```bash
npm test
```
### Run Supplemental Scripts
There are some supplemental scripts that were added to deal with specific sitatuations. These can be called with `npm run extra name-of-script`. The following scripts are available:
- `supplementNotationsInMappings`: This will look for mappings where the field `notation` is missing for any of the concepts, and it will attempt to supplement those notations. This only works for vocabularies which are also imported into the same jskos-server instance and where either `uriPattern` or `namespace` are given.
## API
Unless otherwise specified:
- `GET` requests will return code 200 on success.
- `POST` requests will return code 201 on success.
- `DELETE` requests will return code 204 on success.
- `POST`/`PUT`/`PATCH` requests require a JSON body.
- Alternatively, `POST` can also receive the following inputs:
- any kind of JSON stream
- mutlipart/form-data with the file in `data`
- a URL with JSON data as `url` in the request params
- Note: The `type` request param might be required (either `json`, `ndjson`, or `multipart`)
- `POST`/`PUT`/`PATCH` endpoints will override `creator` and `contributor` of submitted objects (see [this comment](https://github.com/gbv/jskos-server/issues/122#issuecomment-723029967) for more details)
- `POST`/`PUT`/`PATCH`/`DELETE` requests require authentication via a JWT from [login-server](https://github.com/gbv/login-server) in the header. Exception: Authentication for certain actions on certain endpoints can be disabled (see [configuration](#configuration)).
- `PUT`/`PATCH`/`DELETE` requests are required to come from the owner of the entity that is being modified.
- All URL parameters are optional.
- All `GET` endpoints (except for `/status` and those with `:_id`) offer pagination via `limit=[number]` (default: 100) and `offset=[number]` (default: 0) parameters. In the response, there will be a `Link` header like described in the [GitHub API documentation](https://developer.github.com/v3/#pagination), as well as a `X-Total-Count` header containing the total number of results.
- For possible errors, see [Errors](#errors).
### GET /status
Returns a status object.
There is a [JSON Schema](https://json-schema.org) for the format of this endpoint. It is available under `/status.schema.json` for every jskos-server installation (starting from version 1.0.0). The most recent schema can be accessed here: https://gbv.github.io/jskos-server/status.schema.json
Note that certain properties from the actual configuration will not be shown in the result for `/status`:
- `verbosity`
- `port`
- `mongo`
- `namespace`
- `proxies`
- `ips` (including inside of actions)
- `auth.key` if a symmetrical algorithm is used (HS256, HS384, HS512)
* **Success Response**
```json
{
"config": {
"env": "development",
"baseUrl": "http://localhost:3000/",
"version": "1.1",
"auth": {
"algorithm": "RS256",
"key": null
},
"schemes": {
"read": {
"auth": false
}
},
"concepts": {
"read": {
"auth": false
}
},
"mappings": {
"read": {
"auth": false
},
"create": {
"auth": true
},
"update": {
"auth": true,
"crossUser": false
},
"delete": {
"auth": true,
"crossUser": false
},
"fromSchemeWhitelist": null,
"toSchemeWhitelist": null,
"anonymous": false,
"cardinality": "1-to-n"
},
"concordances": {
"read": {
"auth": false
}
},
"annotations": {
"read": {
"auth": false
},
"create": {
"auth": true
},
"update": {
"auth": true,
"crossUser": false
},
"delete": {
"auth": true,
"crossUser": false
}
},
"identityProviders": null,
"identities": null
},
"schemes": "http://localhost:3000/voc",
"top": "http://localhost:3000/voc/top",
"concepts": "http://localhost:3000/voc/concepts",
"voc-suggest": "http://localhost:3000/voc/suggest",
"voc-search": "http://localhost:3000/voc/search",
"data": "http://localhost:3000/data",
"narrower": "http://localhost:3000/narrower",
"ancestors": "http://localhost:3000/ancestors",
"suggest": "http://localhost:3000/suggest",
"search": "http://localhost:3000/search",
"concordances": "http://localhost:3000/concordances",
"mappings": "http://localhost:3000/mappings",
"annotations": "http://localhost:3000/annotations",
"types": null,
"validate": "http://localhost:3000/validate",
"ok": 1
}
```
* **Error Response**
```json
{
"ok": 0
}
```
(other properties omitted)
### GET /checkAuth
Endpoint to check whether a user is authorized. If `type` or `action` are not set, it will use `identities`/`identityProviders` that are defined directly under config.
* **URL Params**
`type=[type]` one of "schemes", "concepts", "mappings", "annotations" (optional)
`action=[action]` one of "read", "create", "update", "delete" (optional)
### POST /validate
Endpoint to validate JSKOS objects via [jskos-validate].
* **URL Params**
`type=[type]` a [JSKOS object type](https://gbv.github.io/jskos/jskos.html#object-types) that all objects must have (optional)
`unknownFields=[boolean]` with `1` or `true` to allow unknown fields inside objects (by default, unknown fields do not pass validation)
`knownSchemes=[boolean]` with `1` or `true` to use concept schemes available in this jskos-server instance for validation of concepts. Implies `type=concept` and all concept must reference a known concept scheme via `inScheme`.
If neither `type` nor `knownSchemes` are specified, concept schemes in the data to be validated can be used to validate following concepts in the same request array (see last example below).
* **Success Response**
Array with the JSON response provided by [jskos-validate]. The indices of the array correspond to the order of the given data. An element is `true` when the object passed validation, or an array of errors when the object failed validation. Data format of error objects may change in future versions but there is always at least field `message`.
* **Sample Call**
In the following example, an empty object is validated. Since no type is specified, it is validated as a Resource which does not have required field names and therefore passes validation.
```bash
curl -X POST "https://coli-conc.gbv.de/dev-api/validate" -H 'Content-Type: application/json' -d '{}'
```
```json
[
true
]
```
In the following example, the same call is given, but the parameter `type` is set to `mapping`. Mappings require the fields `from` and `to`, therefore the empty object fails validation and errors are returned.
```bash
curl -X POST "https://coli-conc.gbv.de/dev-api/validate?type=mapping" -H 'Content-Type: application/json' -d '{}'
```
```json
[
[
{
"instancePath": "",
"schemaPath": "#/required",
"keyword": "required",
"params": {
"missingProperty": "from"
},
"message": "must have required property 'from'"
},
{
"instancePath": "",
"schemaPath": "#/required",
"keyword": "required",
"params": {
"missingProperty": "to"
},
"message": "must have required property 'to'"
}
]
]
```
In this example, an array of mixed typed objects is validated (given in file `example.json`):
```json
[
{
"type": [ "http://www.w3.org/2004/02/skos/core#ConceptScheme" ],
"uri": "http://example.org/voc",
"notationPattern": "[a-z]+"
},
{
"type": [ "http://www.w3.org/2004/02/skos/core#Concept" ],
"uri": "http://example.org/1",
"notation": [ "abc" ],
"inScheme": [ { "uri": "http://example.org/voc" } ]
},
{
"type": [ "http://www.w3.org/2004/02/skos/core#Concept" ],
"uri": "http://example.org/2",
"notation": [ "123" ],
"inScheme": [ { "uri": "http://example.org/voc" } ]
}
]
```
The first object is a concept scheme with `notationPattern`. Since the other two elements are concepts of that concept scheme (see `inScheme`), the concepts must additionally pass tests related to URI or notation patterns of the given schemes. Since the last concept has a notation that does not match the pattern, it fails the validation. Note that only object with appropriate `type` field are included in this additional part of validation.
```bash
curl -X POST "https://coli-conc.gbv.de/dev-api/validate" -H 'Content-Type: application/json' -d @example.json
```
```json
[
true,
true,
[
{
"message": "concept notation 123 does not match [a-z]+"
}
]
]
```
### GET /validate
Same as [POST /validate](#post-validate) but JSKOS data to be validated is passed via URL.
* **URL Params**
`url=[url]` URL to load JSKOS data from
`type=[type]` see [POST /validate](#post-validate)
`unknownFields=[boolean]` see [POST /validate](#post-validate)
`knownSchemes=[boolean]` see [POST /validate](#post-validate)
### GET /concordances
Lists all concordances for mappings.
* **URL Params**
`uri=[uri]` URIs for concordances separated by `|`
`fromScheme=[uri|notation]` only show concordances from concept scheme (URI or notation) (separated by `|`)
`toScheme=[uri|notation]` only show concordances to concept scheme (URI or notation) (separated by `|`)
`creator=[creator]` only show concordances from creator (separated by `|`)
`mode=[mode]` specify the mode for the parameters above, one of `and` (default) and `or`
`download=[type]` returns the whole result as a download (available types are `json` and `ndjson`), ignores `limit` and `offset`
* **Success Response**
JSON array of [JSKOS Concordances](https://gbv.github.io/jskos/jskos.html#concordances)
* **Sample Call**
```bash
curl https://coli-conc.gbv.de/api/concordances?limit=1
```
```json
[
{
"@context": "https://gbv.github.io/jskos/context.json",
"creator": [
{
"prefLabel": {
"de": "VZG"
}
}
],
"distributions": [
{
"download": "https://coli-conc.gbv.de/api/mappings?partOf=http://coli-conc.gbv.de/concordances/ddc_rvk_recht&download=ndjson",
"format": "http://format.gbv.de/jskos",
"mimetype": "application/x-ndjson; charset=utf-8"
}
],
"extent": "2267",
"fromScheme": {
"notation": [
"DDC"
],
"uri": "http://bartoc.org/en/node/241"
},
"notation": [
"ddc_rvk_recht"
],
"scopeNote": {
"de": [
"Recht"
]
},
"toScheme": {
"notation": [
"RVK"
],
"uri": "http://bartoc.org/en/node/533"
},
"type": [
"http://rdfs.org/ns/void#Linkset"
],
"uri": "http://coli-conc.gbv.de/concordances/ddc_rvk_recht"
}
]
```
### GET /concordances/:_id
Returns a specific concordance.
* **URL Params**
None
* **Success Response**
JSKOS object for concordance.
* **Error Response**
If no concordance with `_id` could be found, it will return a 404 not found error.
### POST /concordances
Saves one or more concordances in the database. Note that `fromScheme` and `toScheme` must be supported by the jskos-server instance.
* **URL Params**
None
* **Success Reponse**
JSKOS Concordance object(s) as were saved in the database.
* **Error Response**
When a single concordance is provided, an error can be returned if there's something wrong with it (see [errors](#errors)). When multiple concordances are provided, the first error will be returned.
### PUT /concordances/:_id
Overwrites a concordance in the database.
* **Success Reponse**
JSKOS Concordance object as it was saved in the database.
Note that any changes to the `uri`, `notation`, `fromScheme`, `toScheme`, `extent`, `distributions`, and `created` properties will be ignored. (No error will be thrown in this case.)
### PATCH /concordances/:_id
Adjusts a concordance in the database.
* **Success Reponse**
JSKOS Concordance object as it was saved in the database.
Note that changes to the properties `uri`, `notation`, `fromScheme`, `toScheme`, `created`, `extent`, and `distributions` are currently not allowed and will result in an [InvalidBodyError](#InvalidBodyError).
### DELETE /concordances/:_id
Deletes a concordance from the database.
* **Success Reponse**
Status 204, no content.
**Note that only concordances which have no mappings associated can be deleted.**
### GET /mappings
Returns an array of mappings. Each mapping has a property `uri` under which the specific mapping can be accessed.
* **URL Params**
`identifier=[identifier1|identifier2|...]` specify mapping identifiers separated by `|`
`from=[uriOrNotation1|uriOrNotation2|...]` specify the source URI or notation (truncated search possible by appending a `*`, multiple URIs/notations separated by `|`)
`to=[uriOrNotation1|uriOrNotation2|...]` specify the target URI or notation (truncated search possible by appending a `*`, multiple URIs/notations separated by `|`)
`mode=[mode]` specify the mode for `from`, `to`, and `identifier`, one of `and` (default) and `or`
`direction=[direction]` specify the direction of the mapping. Available values are: `forward` (default), `backward` (essentially swaps `from` and `to`), `both` (combines forward and backward).
`fromScheme=[uriOrNotation1|uriOrNotation2|...]` only show mappings from concept scheme (URI or notation, multiple URIs/notations separated by `|`)
`toScheme=[uriOrNotation1|uriOrNotation2|...]` only show mappings to concept scheme (URI or notation, multiple URIs/notations separated by `|`)
`type=[uri1|uri2|...]` only show mappings that conform to a certain type or types (see [JSKOS Concept Mappings]) (URIs separated by `|`)
`partOf=[uri1|uri2|...]` only show mappings that are part of certain concordances (URIs separated by `|`); value `none` returns mappings that are not part of a concordance, value `any` returns mappings that are part of any concordance
`creator=[string1|string2|...]` only show mappings that have a certain creator (separated by `|`)
`annotatedBy=[uri1|uri2|...]` has annotations by user with URI(s)
`annotatedFor=[motivation]` has annotations with a certain motivation (e.g. `assessing`); value `none` returns mappings that have no annotations at all, value `any` returns mappings that have any kind of annotation, values starting with `!` (e.g. `!assessing`) filter out annotations with that motivation. Note that to mitigate performance issues with negative assertions (`none` or `!xyz`), jskos-server will return the number 9999999 in the `X-Total-Count` header (see [this](https://github.com/gbv/jskos-server/issues/176#issuecomment-1167188606)).
`annotatedWith=[body]` has annotations with a certain body value (e.g. `+1`) OR has a sum of assessment annotations that conforms to the given comparison operation; for the latter, either `from` or `to` must be given, `annotatedFor` must be either not set or set to `assessing`, and the value of this parameter needs to consist of a comparison operator (`=`, `<`, `>`, `<=`, or `>=`) followed by a number. Example: `annotatedWith=>0` returns mappings with a positive assessment sum (equivalent to `annotatedWith=>=1`).
`properties=[list]` with `[list]` being a comma-separated list of properties (currently supporting only `annotations` for mappings)
`download=[type]` returns the whole result as a download (available types are `json`, `ndjson`, `csv`, and `tsv`), ignores `limit` and `offset`; **note**: `csv` and `tsv` are restricted (and fixed) to 5 target concepts, meaning that if the data set includes a mapping with more than 5 target concepts, only the first 5 will appear in the export
`sort=[sort]` sorts by a specific field. Available are `created`, `modified`, and `mappingRelevance` (default). Results will always be additionally sorted by `from.memberSet.uri` and `_id` in order to create a stable and sensible sort.
`order=[order]` order to use for sorting. Available are `asc` and `desc` (default).
`cardinality=[cardinality]` cardinality of the mapping. Available are `1-to-n` (default) and `1-to-1`.
* **Success Response**
JSON array of [JSKOS Concept Mappings]
* **Sample Call**
```bash
curl https://coli-conc.gbv.de/api/mappings?from=http://dewey.info/class/612.116/e23/
```
```json
[
{
"from": {
"memberSet": [
{
"uri": "http://dewey.info/class/612.116/e23/",
"notation": [
"612.116"
]
}
]
},
"to": {
"memberSet": [
{
"uri": "http://rvk.uni-regensburg.de/nt/WW_8800-WW_8839",