new: OData: implement $expand=* getodk#84

As discussed here https://forum.getodk.org/t/extend-api-to-retrieve-plain-json/32204, $expand is now implemented with "*" as only available option for now.

README.md

@@ -36,6 +36,8 @@ CREATE DATABASE jubilant with owner=jubilant encoding=UTF8;
 \c jubilant;
 CREATE EXTENSION IF NOT EXISTS CITEXT;
 CREATE EXTENSION IF NOT EXISTS pg_trgm;
+CREATE DATABASE jubilant_test with owner=jubilant encoding=UTF8;
+\c jubilant;
 ```
 
 Then, go to the repository root in a command line (where this README is) and run `make` with no arguments. This will install all npm dependencies and run all necessary migrations on the database; see the [makefile](Makefile) for details.

docs/api.md

@@ -2894,15 +2894,15 @@ While the latest 4.01 OData specification adds a new JSON EDMX CSDL format, most
 + Response 406 (application/json)
     + Attributes (Error 406)
 
-### Data Document [GET /v1/projects/{projectId}/forms/{xmlFormId}.svc/{table}{?%24skip,%24top,%24count,%24wkt,%24filter}]
+### Data Document [GET /v1/projects/{projectId}/forms/{xmlFormId}.svc/{table}{?%24skip,%24top,%24count,%24wkt,%24filter,%24expand}]
 
 The data documents are the straightforward JSON representation of each table of `Submission` data. They follow the [corresponding specification](http://docs.oasis-open.org/odata/odata-json-format/v4.01/odata-json-format-v4.01.html), but apart from the representation of geospatial data as GeoJSON rather than the ODK proprietary format, the output here should not be at all surprising. If you are looking for JSON output of Submission data, this is the best place to look.
 
 The `$top` and `$skip` querystring parameters, specified by OData, apply `limit` and `offset` operations to the data, respectively. The `$count` parameter, also an OData standard, will annotate the response data with the total row count, regardless of the scoping requested by `$top` and `$skip`. While paging is possible through these parameters, it will not greatly improve the performance of exporting data. ODK Central prefers to bulk-export all of its data at once if possible.
 
 As of ODK Central v1.1, the [`$filter` querystring parameter](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) is partially supported. In OData, you can use `$filter` to filter by any data field in the schema. In ODK Central, the only fields you can reference are `__system/submitterId` and `__system/submissionDate`. These refer to the numeric `actorId` and the timestamp `createdAt` of the submission overall. The operators `lt`, `lte`, `eq`, `neq`, `gte`, `gt`, `not`, `and`, and `or` are supported. The built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second` are supported. These supported elements may be combined in any way, but all other `$filter` features will cause an error. Please see the [OData documentation](http://docs.oasis-open.org/odata/odata/v4.01/odata-v4.01-part1-protocol.html#_Toc31358948) on `$filter` for more information.
 
-In this release of Central, `$expand` is not yet supported. This will likely change in the future, once we can instate Navigation Properties.
+If you want to expand all repetitions, you can use `%24expand=*`. This might be helpful if you want to get a full dump of all submissions within a single query.
 
 The _nonstandard_ `$wkt` querystring parameter may be set to `true` to request that geospatial data is returned as a [Well-Known Text (WKT) string](https://en.wikipedia.org/wiki/Well-known_text) rather than a GeoJSON structure. This exists primarily to support Tableau, which cannot yet read GeoJSON, but you may find it useful as well depending on your mapping software. **Please note** that both GeoJSON and WKT follow a `(lon, lat, alt)` coördinate ordering rather than the ODK-proprietary `lat lon alt`. This is so that the values map neatly to `(x, y, z)`. GPS accuracy information is not a part of either standards specification, and so is presently omitted from OData output entirely. GeoJSON support may come in a future version.
 
@@ -2916,6 +2916,7 @@ As the vast majority of clients only support the JSON OData format, that is the
     + `%24count`: `true` (boolean, optional) - If set to `true`, an `@odata.count` property will be added to the result indicating the total number of rows, ignoring the above paging parameters.
     + `%24wkt`: `true` (boolean, optional) - If set to `true`, geospatial data will be returned as Well-Known Text (WKT) strings rather than GeoJSON structures.
     + `%24filter`: `year(__system/submissionDate) lt year(now())` (string, optional) - If provided, will filter responses to those matching the query. Only the fields `__system/submitterId` and `__system/submissionDate` are available to reference. The operators `lt`, `lte`, `eq`, `neq`, `gte`, `gt`, `not`, `and`, and `or` are supported, and the built-in functions `now`, `year`, `month`, `day`, `hour`, `minute`, `second`.
+    + `%24expand`: `*` (string, optional) - Repetitions, which should get expanded. Currently, only `&#42` is implemented, which expands all repetitions.
 
 + Response 200 (application/json)
     + Body

lib/data/json.js

@@ -180,9 +180,8 @@ const submissionToOData = (fields, table, { xml, encHasData, submission, submitt
           // the appropriate visible structure, and if we are in state 1 we are at the visible
           // structure so we need to build the list. but if we are past that ignore unless we
           // are $expand'd into the object.
-          // TODO: check for $expand.
           const branchState = getBranchState(schemaStack, table);
-          if (branchState < 1) { // TODO: check for $expand
+          if (branchState < 1 || options.expand === '*') {
             // verify that we have an array to push into in our data obj.
             if (dataPtr[outname] == null) dataPtr[outname] = [];
 

lib/http/endpoint.js

@@ -266,7 +266,7 @@ const isJsonType = (x) => /(^|,)(application\/json|json)($|;|,)/i.test(x);
 const isXmlType = (x) => /(^|,)(application\/(atom(svc)?\+)?xml|atom|xml)($|;|,)/i.test(x);
 
 // various supported odata constants:
-const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt' ];
+const supportedParams = [ '$format', '$count', '$skip', '$top', '$filter', '$wkt', '$expand' ];
 const supportedFormats = {
   json: [ 'application/json', 'json' ],
   xml: [ 'application/xml', 'atom' ]

lib/outbound/odata.js

@@ -70,8 +70,8 @@ const nextUrlFor = (limit, offset, count, originalUrl) =>
     : urlWithQueryParams(originalUrl, { $skip: (offset + limit), $top: null }));
 
 // Given a querystring object, returns an object of relevant OData options. Right
-// now that is only { wkt: Bool }
-const extractOptions = (query) => ({ wkt: isTrue(query.$wkt) });
+// now that is only { wkt: Bool, expand: String }
+const extractOptions = (query) => ({ wkt: isTrue(query.$wkt), expand: query.$expand });
 
 // Given a tableParts: [ String ] and a schemaLookup: Object given by schemaAsLookup(),
 // ensures that the table implied by tableParts actually exists in schemaLookup.

test/unit/data/json.js

@@ -407,6 +407,32 @@ describe('submissionToOData', () => {
       });
     }));
 
+  it('should expand all repeat tables for $expand=*', () =>
+    fieldsFor(testData.forms.withrepeat).then((fields) => {
+      const submission = mockSubmission('two', testData.instances.withrepeat.two);
+      return submissionToOData(fields, 'Submissions', submission, { expand: '*' }).then((result) => {
+        result.should.eql([{
+          __id: 'two',
+          __system,
+          meta: { instanceID: 'two' },
+          name: 'Bob',
+          age: 34,
+          children: {
+            'child@odata.navigationLink': "Submissions('two')/children/child",
+            child: [{
+              __id: 'cf9a1b5cc83c6d6270c1eb98860d294eac5d526d',
+              age: 4,
+              name: 'Billy'
+            }, {
+              __id: 'c76d0ccc6d5da236be7b93b985a80413d2e3e172',
+              age: 6,
+              name: 'Blaine'
+            }]
+          }
+        }]);
+      });
+    }));
+
   it('should extract subtable rows within repeats', () =>
     fieldsFor(testData.forms.withrepeat).then((fields) => {
       const row = { submission: { instanceId: 'two' }, xml: testData.instances.withrepeat.two };