Create new dataset (entity list) or property via the API (#1105)

* starter code to publish new dataset from api

* dataset property via api

* More tests

* renamed dataset query helper functions

* Log new dataset and properties from api in audit log

* add dataset.create verb

* code review: timestamps, validation, responses, bugs

* Part of docs: new dataset endpoint

* Removed console log

* small fixes

* Updated docs

docs/api.yaml

@@ -43,6 +43,7 @@ info:
     - Bulk Entity Creation!
       * The existing [Entity Create](/central-api-entity-management/#creating-entities) endpoint now also accepts a list of Entities to append to a Dataset.
       * The `uuid` property is no longer required and Central will generate a UUID for each new Entity if needed.
+    - [Datasets (Entity Lists)](/central-api-dataset-management/#creating-datasets) and [Properties](/central-api-dataset-management/#adding-properties) can now be added via the API instead of only through Forms.
     - OData Data Document for requests of Submissions and Entities now allow use of `$orderby`.
     - ETag headers on all Blobs.
 
@@ -64,7 +65,7 @@ info:
         * `hard`. The Entity has a version that was based on a version other than the version immediately prior to it. Further, that version updated the same property as another update.
         * If an Entity has a conflict, it can be marked as resolved. After that, the conflict field will be null until a new conflicting version is received.
       * Each Entity Version (`currentVersion` or list of versions) has new fields `baseVersion`, `dataReceived`, and `conflictingProperties`
-    - [Datasets](/central-api-dataset-management/#datasets) extended metadata now has a `conflicts` field, which counts number of Entities currently in conflict
+    - [Datasets](/central-api-dataset-management/#listing-datasets) extended metadata now has a `conflicts` field, which counts number of Entities currently in conflict
     - Entity conflicts can be resolved using the existing [PATCH](/central-api-entity-management/#updating-an-entity) endpoint, with or without new data
     - New `relevantToConflict` query parameter on [GET entities/:uuid/versions](/central-api-entity-management/#listing-versions)
 
@@ -129,7 +130,7 @@ info:
 
     **Changed**:
 
-    - [GET /projects/:id/datasets](/central-api-dataset-management/#datasets) now supports `X-Extended-Metadata` header to retrieve number of Entities in the Dataset and timestamp of the last Entity
+    - [GET /projects/:id/datasets](/central-api-dataset-management/#listing-datasets) now supports `X-Extended-Metadata` header to retrieve number of Entities in the Dataset and timestamp of the last Entity
 
     - `$select` in OData now supports selecting complex type(groups)
 
@@ -146,7 +147,7 @@ info:
     * Introducing [Datasets](/central-api-dataset-management) as the first step of Entity-Based Data Collection! Future versions of Central will build on these new concepts. We consider this functionality experimental and subject to change in the next release.
 
       * Forms can now create Datasets in the project, see [Creating a New Form](/central-api-form-management/#creating-a-new-form) and the [ODK XForms specification](https://getodk.github.io/xforms-spec) for details.
-      * New endpoint [GET /projects/:id/datasets](/central-api-dataset-management/#datasets) for listing Datasets of a project.
+      * New endpoint [GET /projects/:id/datasets](/central-api-dataset-management/#listing-datasets) for listing Datasets of a project.
       * New endpoint [GET /projects/:id/datasets/:name/entities.csv](/central-api-dataset-management/#download-dataset) to download the Dataset as a CSV file.
       * New endpoints for [Related Datasets](/central-api-form-management/#related-datasets) to see the Datasets affected by published and unpublished Forms.
       * New endpoint [PATCH .../attachments/:name](/central-api-form-management/#linking-a-dataset-to-a-draft-form-attachment) to link/unlink a Dataset to a Form Attachment.
@@ -9697,9 +9698,9 @@ paths:
     get:
       tags:
       - Dataset Management
-      summary: Datasets
+      summary: Listing Datasets
       description: The Dataset listing endpoint returns all published Datasets in a Project. If a Draft Form defines a new Dataset, that Dataset will not be included in this list until the Form is published.
-      operationId: Datasets
+      operationId: Listing Datasets
       parameters:
       - name: projectId
         in: path
@@ -9752,6 +9753,78 @@ paths:
                 code: "403.1"
                 message: The authenticated actor does not have rights to perform that
                   action.
+    post:
+      tags:
+      - Dataset Management
+      summary: Creating Datasets
+      description: |- 
+        You can create a Dataset with a specific `name` within a Project. This Dataset can then be populated via the API or via Forms and Submissions, and then used by other Forms. This endpoint allows a Dataset to be created programatically without an input form.
+
+        By default, the Dataset will have no properties, but each Entity will have a `label` and a unique ID (`uuid`). You can add additional properties with this [related endpoint](/central-api-dataset-management/#adding-properties).
+
+        The Dataset name must follow the the same rules as XML identifiers and not start with `.` or `__`. See the [Entities XForms Specification](https://getodk.github.io/xforms-spec/entities.html#declaring-that-a-form-creates-entities) for more information.
+      operationId: Creating Datasets
+      parameters:
+      - name: projectId
+        in: path
+        description: The numeric ID of the Project
+        required: true
+        schema:
+          type: number
+        example: "16"
+      requestBody:
+          content:
+            '*/*':
+              schema:
+                required:
+                - name
+                type: object
+                properties:
+                  name:
+                    type: string
+                    description: The desired name of the Dataset. Must be a valid 
+                  approvalRequired:
+                    type: boolean
+                    description: Control whether a Submission should be approved before an Entity is created from it.
+                    required: false
+                example:
+                  name: Trees
+                  approvalRequired: false
+          required: false
+      responses:
+        200:
+          description: OK
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/DatasetMetadata'
+              example:
+                name: people
+                createdAt: '2018-01-19T23:58:03.395Z'
+                projectId: 1
+                approvalRequired: false
+                sourceForms: []
+                linkedForms: []
+                properties: []
+        403:
+          description: Forbidden
+          content:
+            application/json:
+              schema:
+                required:
+                - code
+                type: object
+                properties:
+                  code:
+                    type: string
+                  message:
+                    type: string
+              example:
+                code: "403.1"
+                message: The authenticated actor does not have rights to perform that
+                  action.
   /projects/{projectId}/datasets/{name}:
     get:
       tags:
@@ -9899,6 +9972,69 @@ paths:
                 code: "403.1"
                 message: The authenticated actor does not have rights to perform that
                   action.
+  /projects/{projectId}/datasets/{name}/properties:
+    post:
+      tags:
+      - Dataset Management
+      summary: Adding Properties
+      description: |-
+        Creates a new Property with a specified name in the Dataset.
+
+        Property names follow the same rules as form field names (valid XML identifiers) and cannot use the reserved names of `name` or `label`, or begin with the reserved prefix `__`. 
+      operationId: Adding Properties
+      parameters:
+      - name: projectId
+        in: path
+        description: The numeric ID of the Project
+        required: true
+        schema:
+          type: number
+        example: "16"
+      - name: name
+        in: path
+        description: Name of the Property
+        required: true
+        schema:
+          type: string
+        example: people
+      requestBody:
+          content:
+            '*/*':
+              schema:
+                required:
+                - name
+                type: object
+                properties:
+                  name:
+                    type: string
+                    description: The desired name of the Property.
+                example:
+                  name: circumference
+          required: false
+      responses:
+        200:
+          description: OK
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Success'
+        403:
+          description: Forbidden
+          content:
+            application/json:
+              schema:
+                required:
+                - code
+                type: object
+                properties:
+                  code:
+                    type: string
+                  message:
+                    type: string
+              example:
+                code: "403.1"
+                message: The authenticated actor does not have rights to perform that
+                  action.
   /projects/{projectId}/datasets/{name}/entities.csv:
     get:
       tags:

lib/model/migrations/20240312-01-add-dataset-create-verb.js

@@ -0,0 +1,23 @@
+// Copyright 2024 ODK Central Developers
+// See the NOTICE file at the top-level directory of this distribution and at
+// https://github.com/getodk/central-backend/blob/master/NOTICE.
+// This file is part of ODK Central. It is subject to the license terms in
+// the LICENSE file found in the top-level directory of this distribution and at
+// https://www.apache.org/licenses/LICENSE-2.0. No part of ODK Central,
+// including this file, may be copied, modified, propagated, or distributed
+// except according to the terms contained in the LICENSE file.
+
+const up = (db) => db.raw(`
+UPDATE roles
+SET verbs = verbs || '["dataset.create"]'::jsonb
+WHERE system in ('admin', 'manager')
+`);
+
+const down = (db) => db.raw(`
+UPDATE roles
+SET verbs = (verbs - 'dataset.create')
+WHERE system in ('admin', 'manager')
+`);
+
+module.exports = { up, down };
+

lib/model/query/datasets.js

@@ -8,7 +8,7 @@
 // except according to the terms contained in the LICENSE file.
 
 const { sql } = require('slonik');
-const { extender, QueryOptions, equals, updater } = require('../../util/db');
+const { extender, insert, QueryOptions, equals, updater } = require('../../util/db');
 const { Dataset, Form, Audit } = require('../frames');
 const { validatePropertyName } = require('../../data/dataset');
 const { isEmpty, isNil, either, reduceBy, groupBy, uniqWith, equals: rEquals, map } = require('ramda');
@@ -122,6 +122,30 @@ const createOrMerge = (parsedDataset, form, fields) => async ({ one, Actees, Dat
   return partial.with({ action: result.action, fields: dsPropertyFields });
 };
 
+// Insert a Dataset when there is no associated form and immediately publish
+const createPublishedDataset = (dataset, project) => async ({ one, Actees }) => {
+  const actee = await Actees.provision('dataset', project);
+  const dsWithId = await one(insert(dataset.with({ acteeId: actee.id })));
+  return new Dataset(await one(sql`UPDATE datasets SET "publishedAt" = "createdAt" where id = ${dsWithId.id} RETURNING *`));
+};
+
+createPublishedDataset.audit = (dataset) => (log) =>
+  log('dataset.create', dataset, { properties: [] });
+createPublishedDataset.audit.withResult = true;
+
+// Insert a Dataset Property when there is no associated form and immediately publish
+const _createPublishedProperty = (property) => sql`
+INSERT INTO ds_properties ("name", "datasetId", "publishedAt")
+VALUES (${property.name}, ${property.datasetId}, clock_timestamp())
+RETURNING *`;
+
+// eslint-disable-next-line no-unused-vars
+const createPublishedProperty = (property, dataset) => ({ one }) =>
+  one(_createPublishedProperty(property));
+
+createPublishedProperty.audit = (property, dataset) => (log) =>
+  log('dataset.update', dataset, { properties: [property.name] });
+
 ////////////////////////////////////////////////////////////////////////////
 // DATASET (AND DATASET PROPERTY) PUBLISH
 
@@ -477,6 +501,7 @@ const getUnprocessedSubmissions = (datasetId) => ({ all }) =>
     .then(map(construct(Audit)));
 
 module.exports = {
+  createPublishedDataset, createPublishedProperty,
   createOrMerge, publishIfExists,
   getList, get, getById, getByActeeId,
   getMetadata, getAllByAuth,

lib/resources/datasets.js

@@ -10,7 +10,8 @@
 const sanitize = require('sanitize-filename');
 const { getOrNotFound } = require('../util/promise');
 const { streamEntityCsv } = require('../data/entity');
-const { contentDisposition, withEtag } = require('../util/http');
+const { validateDatasetName, validatePropertyName } = require('../data/dataset');
+const { contentDisposition, success, withEtag } = require('../util/http');
 const { md5sum } = require('../util/crypto');
 const { Dataset } = require('../model/frames');
 const Problem = require('../util/problem');
@@ -51,6 +52,31 @@ module.exports = (service, endpoint) => {
     return Datasets.getMetadata(updatedDataset);
   }));
 
+  service.post('/projects/:id/datasets', endpoint(async ({ Projects, Datasets }, { auth, body, params }) => {
+    const project = await Projects.getById(params.id).then(getOrNotFound);
+    await auth.canOrReject('dataset.create', project);
+
+    const { name } = body;
+    if (name == null || !validateDatasetName(name))
+      throw Problem.user.unexpectedValue({ field: 'name', value: name, reason: 'This is not a valid dataset name.' });
+
+    const dataset = Dataset.fromApi(body).with({ name, projectId: project.id });
+
+    return Datasets.getMetadata(await Datasets.createPublishedDataset(dataset, project));
+  }));
+
+  service.post('/projects/:projectId/datasets/:name/properties', endpoint(async ({ Datasets }, { params, body, auth }) => {
+    const dataset = await Datasets.get(params.projectId, params.name, true).then(getOrNotFound);
+    await auth.canOrReject('dataset.update', dataset);
+
+    const { name } = body;
+    if (name == null || !validatePropertyName(name))
+      throw Problem.user.unexpectedValue({ field: 'name', value: name, reason: 'This is not a valid property name.' });
+
+    await Datasets.createPublishedProperty(new Dataset.Property({ name, datasetId: dataset.id }), dataset);
+    return success;
+  }));
+
   service.get('/projects/:projectId/datasets/:name/entities.csv', endpoint(async ({ Datasets, Entities, Projects }, { params, auth, query }, request, response) => {
     const project = await Projects.getById(params.projectId).then(getOrNotFound);
     await auth.canOrReject('entity.list', project);