From 7fb03f87d5af1dcb0fbf42198a3836d906a476c8 Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Tue, 21 Feb 2023 12:32:48 +1300 Subject: [PATCH 1/7] Updated spec define document paths & field names resolves #123 --- docs/jsonapi-spec.textile | 106 +++++++++++++++++++++++++++++++++++++- 1 file changed, 104 insertions(+), 2 deletions(-) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index 755d78ce9d..932404302f 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -60,10 +60,106 @@ h2(#collection). Collection *TODO:* define the collection, it's properties etc . Do we define how to create one ? -h2(#document). Document +h2(#document). Documents *TODO:* Describe how do we define a document and then refer to the parts of a JSON document, e.g. key or path etc ? +h3(#documentFieldNames). Field Names + +Fields names in the JSON document have additional restrictions not imposed on the "member names" by the "JSON Specification":https://www.json.org/. These restrictions are to ensure compatiblity with the underlaying storage system. + +Fields names must adhere to the ASCII character set (though may be encoded using UTF-8) and follow the regular experession below. The field name @_id@ is the only field name that can be used that does not match the rules for user defined field names. + +_Syntax:_ + +bc(syntax).. + ::= | + + + = _id + ::= [a-zA-Z][a-zA-Z0-9_]* + + +Other than the special @_id@ field there are no "reserved names" for field names that documents must not use. Any reserved words used by the system should be defined to not match the rules for @@ and so avoid collisions. + +p. _Sample:_ + +bc(sample).. +{ + "_id_" : "my_doc", + "a_field" : "the value", + "sub_doc" : { + "inner_field" : 1 + } +} + +h3(#documentArrayIndexes). Array Indexes + +Array indexes are used in query paths to identify elements in a array. They are zero based, and should not include leading zeros (.e.g. prefer @1@ and avoid @0001@). If leading zeros are included the server will ignore them. + +_Syntax:_ + +bc(syntax).. + ::= [0-9]+ + +p. _Sample:_ + +bc(sample).. +// Second element of the `tags` array is equal to "foo" +{"find" : { + "filter" : {"tags.2": "foo"} + } +} +// Equivalent to above +{"find" : { + "filter" : {"tags.002": "foo"} + } +} + +h3(#documentPaths). Document Paths + +Document paths are used to identify fields in a JSON document in multiple contexts, such as a filter for a query or specifying fields to update. + +@@ in this specification are a "JSON Path":https://jsonpath.com/ like syntax. + +"Clauses":#clauses specify where they support the use of @@ to filter, update, sort, or project documents. + +_Syntax:_ + +bc(syntax).. + ::= + +h4(#documentPathsDottedNotation). Dotted Notation Paths + +Dotted Notation Paths support specifying top level fields, fields in embedded documents (nested fields), array elements, and nested structures in arrays. + +In general the dotted notation is one or more @@ or @@ concatenated with a period characher @.@ and no additional whitespace. + +_Syntax:_ + +bc(syntax).. + ::= (. | )* + +p. _Sample:_ + +bc(sample).. +// _id is equal to 1 +{"find" : { + "filter" : {"_id": 1} + } +} +// suburb field in the address sub document is equal to "banana" +{"find" : { + "filter" : {"address.suburb": "banana"} + } +} +// Second element of the `tags` array is equal to "foo" +{"find" : { + "filter" : {"tags.2": "foo"} + } +} + + h2(#commands). Commands Commands are included in a request and are executed against a single collection. @@ -620,6 +716,8 @@ h3(#clauseSort). Sort Clause A sort clause is applied to a set of documents to order them. Examples of this include ordering a large result set that will returned as pages, or sorting a candidate set of documents to determine which document will be returned when only a single document is required. +*TODO* THis is wrong, it needs to be a map + p. _Syntax:_ bc(syntax).. @@ -649,4 +747,8 @@ The @@ is evaluated using the following order of operations: h4(#clauseSortOrder). Sort Order -*TODO:* type coercion, missing fields \ No newline at end of file +*TODO:* type coercion, missing fields + +h3(#clauseUpdate). Update Clause + +*TODO* BNF for update commands, need to use \ No newline at end of file From 3335c4879f2f149b17852f1a1d845a55e5cc0dec Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Tue, 21 Feb 2023 14:10:16 +1300 Subject: [PATCH 2/7] Added limits for namespace, collection & docs resolves #135 --- docs/jsonapi-spec.textile | 55 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index 932404302f..4f09cfac24 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -56,10 +56,36 @@ h2(#namespace). Namespace *TODO:* define the namespace, it's properties etc . Do we define how to create one ? +h3(#namespaceLimits). Namespace Limits + +h4(#namespaceLimitsName). Namespace Name Limit + +Namespace names must follow the regular expression pattern below: + +_Syntax:_ + +bc(syntax).. + ::= [a-zA-Z][a-zA-Z0-9_]* + +p. The maximum length of a namespace name is 48 characters. + h2(#collection). Collection *TODO:* define the collection, it's properties etc . Do we define how to create one ? +h3(#collectionLimits). Collection Limits + +h4(#collectionLimitsName). Collection Name Limit + +Collection names must follow the regular expression pattern below: + +_Syntax:_ + +bc(syntax).. + ::= [a-zA-Z][a-zA-Z0-9_]* + +p. The maximum length of a collection name is 48 characters. + h2(#document). Documents *TODO:* Describe how do we define a document and then refer to the parts of a JSON document, e.g. key or path etc ? @@ -159,6 +185,35 @@ bc(sample).. } } +h3(#documentLimits). Document Limits + +JSON documents must ahere to the following limits. Attempting to insert or modify a document beyond these limits will result in the command failing. + +h4(#documentLimitsSize). Document Size Limit + +The maximum size of a single document is 1 megabyte. The size is determined by the JSON serialisation of the document. + +h4(#documentLimitsDepth). Document Depth Limit + +The maximum nesting depth for a single document is 8 levels deep. Each nested docment or array adds another level. + +h4(#documentLimitsFields). Document Field Limits + +The maximum length of a "field name":#documentFieldNames is 48 characters. + +The maximum number of fields allowed in a single JSON object is 32. + +h4(#documentLimitsValues). Document Value Limits + +The maxium size of field values are: + +|_. JSON Type |_. Maximum Value | +| @string@ | Maximum length of 4096 unicode characters | +| @number@ | Arbitrary sized number described using "BigDecimal in Java":https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#BigDecimal-java.lang.String- | + +h4(#documentLimitsArray). Document Array Limits + +The maximum length of an array is 100 elements. h2(#commands). Commands From 1fa3f7b11eb578a8591146ff7c212c9574031280 Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Tue, 21 Feb 2023 16:48:05 +1300 Subject: [PATCH 3/7] documenting options for all existing commands resolves #108 --- docs/jsonapi-spec.textile | 105 +++++++++++++++++++++++++++++++------- 1 file changed, 86 insertions(+), 19 deletions(-) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index 4f09cfac24..e0d02724fa 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -288,17 +288,32 @@ bc(sample).. p.*TODO:* check this multi doc return format -h3(#commandCountDocuments). Count Documents Command +h3(#commandCountDocuments). countDocuments Command -h4(#commandCountDocumentsOptions). Count Documents Command Options +h4(#commandCountDocumentsOptions). countDocuments Command Options -h3(#commandDeleteMany). Delete Many Command +The @countDocuments@ command does not support any options. -h3(#commandDeleteOne). Delete One Command +h3(#commandDeleteMany). deleteMany Command -h3(#commandEstimatedDocumentCount). Estimated Document Count Command +h4(#commandDeleteManyOptions). deleteMany Command Options -h3(#commandFind). Find Command +@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. + +|_. Option |_. Type |_. Description | +| @limit@ | Positive Integer | Limits the number of documents that will be deleted, defaults to @20@ and the maximum value is @100@. If the provided value is greater than the maximum an error is returned. | + +h3(#commandDeleteOne). deleteOne Command + +h4(#commandCountDocumentsOptions). deleteOne Command Options + +The @deleteOne@ command does not support any options. + +h3(#commandEstimatedDocumentCount). estimatedDocumentCount Command + +*TODO* TBC + +h3(#commandFind). find Command @find@ selects zero or more documents from a collection that match a selection filter, and returns either the complete documents or a partial projection of each document. @@ -329,7 +344,7 @@ bc(sample).. } } -h4(#commandFindOrderOfOperation). Find Command Order of Operations +h4(#commandFindOrderOfOperation). find Command Order of Operations p. TODO: add how cursor state fits into the processing @@ -340,14 +355,14 @@ p. TODO: add how cursor state fits into the processing # The @limit@ option from @@ is applied to reduce the number of candidate documents to no more than @limit@, if no @limit@ is supplied all candidate documents are included. # @@ is applied to each candidate document to create the *result set* documents to be included in the @@, if no @@ is specified the complete candidate documents are included in the response. -h4(#commandFindOptions). Find Command Options - -@@ is a map of key-value pairs that modify the behavior of the find command. All options are optional, with default behavior applied when not provided. +h4(#commandFindOptions). find Command Options -|_. Find Option |_. Type |_. Description | -| @limit@ | Positive Integer | Limits the number of documents to be returned by the command. ==
== If unspecified, or 0, there is no limit on number of documents returned. Note that results may still be broken into pages.| -| @page-state@ | ASCII String | The page state of the previous page, when supplied the @find@ command will return the next page from the result set. ==
== If unspecified, null, or an empty string the first page is returned.| +@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. +|_. Option |_. Type |_. Description | +| @limit@ | Positive Integer | Limits the number of documents to be returned by the command. If unspecified, or 0, there is no limit on number of documents returned. Note that results may still be broken into pages.| +| @pageState@ | ASCII String | The page state of the previous page, when supplied the @find@ command will return the next page from the result set. If unspecified, null, or an empty string the first page is returned.| +| @skip@ | Positive Integer | Skips the specified number of documents, in sort order, before returning any documents.| h3(#commandFindOneAndUpdate). findOneAndUpdate Command @@ -386,17 +401,69 @@ bc(sample).. } } -h3(#commandFindOne). Find One Command +h4(#commandFindOneAndUpdateOptions). findOneAndUpdate Command Options + +@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. + +|_. Option |_. Type |_. Description | +| @returnDocument@ | String Enum | Specifies which document to perform the projection on. If @"before"@ the projection is performed on the document before the update is applied, if @"after"@ the document projection is from the document after the update. Defaults to @"before"@.| +| @upsert@ | Boolean | When @true@ if no documents match the @filter@ clause the command will create a new _empty_ document and apply the @update@ clause to the empty document. If the @_id@ field is included in the @filter@ the new document will use this @_id@, otherwise a random value will be used see "Upsert Considerations":#considerationsUpsert for details. When false the command will only update a document if one matches the filter. Defaults to @false@.| + +h3(#commandFindOne). findOne Command + +h4(#commandFindOneOptions). findOne Command Options + +The @findOne@ command does not support any options. + +h3(#commandInsertMany). insertMany Command + +h4(#commandInsertManyOptions). insertMany Command Options + +@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. + +|_. Option |_. Type |_. Description | +| @ordered@ | Boolean | When @true@ the server will insert the documents in sequential order, ensuring each doucument is succesfully inserted before starting the next. Additionally the command will "fail fast", failing the first document that fails to insert. When @false@ the server is free to re-order the inserts for performance including running multiple inserts in parallel, in this mode more than one document may fail to be inserted (using the "fail silently" mode). See "Multi Document Failure Considerations":#considerationsMultiDocumentFailure for details. Defaults to @true@.| + +h3(#commandInsertOne). insertOne Command + +h4(#commandInsertOneOptions). insertOne Command Options + +The @insertOne@ command does not support any options. + +h3(#commandUpdateMany). updateMany Command + +h4(#commandUpdateManyOptions). updateMany Command Options + +@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. + +|_. Option |_. Type |_. Description | +| @limit@ | Positive Integer | Limits the number of documents that will be updated, defaults to @20@ and the maximum value is @100@. If the provided value is greater than the maximum an error is returned. | +| @upsert@ | Boolean | When @true@ if no documents match the @filter@ clause the command will create a new _empty_ document and apply the @update@ clause to the empty document. If the @_id@ field is included in the @filter@ the new document will use this @_id@, otherwise a random value will be used see "Upsert Considerations":#considerationsUpsert for details. When false the command will only update a document if one matches the filter. Defaults to @false@.| + +h3(#commandUpdateOne). updateOne Command + +h4(#commandUpdateOneOptions). updateOne Command Options + +@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. + +|_. Option |_. Type |_. Description | +| @upsert@ | Boolean | When @true@ if no documents match the @filter@ clause the command will create a new _empty_ document and apply the @update@ clause to the empty document. If the @_id@ field is included in the @filter@ the new document will use this @_id@, otherwise a random value will be used see "Upsert Considerations":#considerationsUpsert for details. When false the command will only update a document if one matches the filter. Defaults to @false@.| + +h2(#commandConsierations). Command Considerations + +Additional considerations that apply to multiple commands. + +h3(#considerationsUpsert). Upsert Considerations -h3(#commandFindOneAndUpdate). Find One And Update Command +Multiple concurrent requests that use upsert may result in multiple documents created when the intention was to only create one. This is because the default behaviour of upsert is to create a new document with a random, server assigned, @_id@ and then apply the updates to that document. -h3(#commandInsertMany). Insert Many Command +If the @filter@ clause of the command includes the @_id@ field when performing an upsert the new document will use the supplied @_id@. This will ensure a single document is created, as the @_id@ is unique. Overlapping commands may read the new document and update it. -h3(#commandInsertOne). Insert One Command +Fields included in the filter clause in addition to the @_id@ are ignored. -h3(#commandUpdateMany). Update Many Command +h3(#considerationsMultiDocumentFailure). Multi Document Failure Considerations -h3(#commandUpdateOne). Update One Command +*TODO* multi document failure considerations h2(#clauses). Clauses From 4f557eb9aa42f5d7221c44dba35180fdce13c9ef Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Wed, 22 Feb 2023 15:19:14 +1300 Subject: [PATCH 4/7] tweaks for fields and arrays resolves #123 --- docs/jsonapi-spec.textile | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index e0d02724fa..2abcc536d2 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -103,30 +103,30 @@ bc(syntax).. = _id - ::= [a-zA-Z][a-zA-Z0-9_]* + ::= [a-zA-Z0-9_]+ - -Other than the special @_id@ field there are no "reserved names" for field names that documents must not use. Any reserved words used by the system should be defined to not match the rules for @@ and so avoid collisions. +p. The @_id@ field is a "reserved name" and is always interpreted as the document identity field. See *TODO DOC ID LINK* for restrictions on it's use. p. _Sample:_ bc(sample).. { - "_id_" : "my_doc", + "_id" : "my_doc", "a_field" : "the value", "sub_doc" : { "inner_field" : 1 - } + }, + "__v" : 1 } h3(#documentArrayIndexes). Array Indexes -Array indexes are used in query paths to identify elements in a array. They are zero based, and should not include leading zeros (.e.g. prefer @1@ and avoid @0001@). If leading zeros are included the server will ignore them. +Array indexes are used in query paths to identify elements in a array. They are zero based, and *must* not include leading zeros, i.e. @1@ is a legal array index and @0001@ is not. Leading zeros are not legal numbers in the "JSON Specification":https://www.json.org/ and should only be used as a property of an object. Not that the first element in the array is indexed using a single zero @0@. _Syntax:_ bc(syntax).. - ::= [0-9]+ + ::= ^0$|^[1-9][0-9]*$ p. _Sample:_ @@ -136,9 +136,9 @@ bc(sample).. "filter" : {"tags.2": "foo"} } } -// Equivalent to above +// Filter on the first element of the tags array {"find" : { - "filter" : {"tags.002": "foo"} + "filter" : {"tags.0": "foo"} } } From 0800fd947b011deec12971e3d3afcc97b926e0d1 Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Thu, 23 Feb 2023 11:49:39 +1300 Subject: [PATCH 5/7] add TODO for expanding field names later resolves #123 --- docs/jsonapi-spec.textile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index 2abcc536d2..1f278c0369 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -107,6 +107,8 @@ bc(syntax).. p. The @_id@ field is a "reserved name" and is always interpreted as the document identity field. See *TODO DOC ID LINK* for restrictions on it's use. +*TODO* add support for allowing "." and "$" in the field names, not needed for the intitial demo release. + p. _Sample:_ bc(sample).. From 4f097e4c8f3017d6ad53193874057c32b468b6e9 Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Thu, 23 Feb 2023 11:56:21 +1300 Subject: [PATCH 6/7] remove limit for deleteMany updateMany this will just be a server side setting resolves #108 --- docs/jsonapi-spec.textile | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index 1f278c0369..bcce1e7449 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -300,10 +300,7 @@ h3(#commandDeleteMany). deleteMany Command h4(#commandDeleteManyOptions). deleteMany Command Options -@@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. - -|_. Option |_. Type |_. Description | -| @limit@ | Positive Integer | Limits the number of documents that will be deleted, defaults to @20@ and the maximum value is @100@. If the provided value is greater than the maximum an error is returned. | +The @deleteMany@ command does not support any options. h3(#commandDeleteOne). deleteOne Command @@ -439,7 +436,6 @@ h4(#commandUpdateManyOptions). updateMany Command Options @@ is a map of key-value pairs that modify the behavior of the command. All options are optional, with default behavior applied when not provided. |_. Option |_. Type |_. Description | -| @limit@ | Positive Integer | Limits the number of documents that will be updated, defaults to @20@ and the maximum value is @100@. If the provided value is greater than the maximum an error is returned. | | @upsert@ | Boolean | When @true@ if no documents match the @filter@ clause the command will create a new _empty_ document and apply the @update@ clause to the empty document. If the @_id@ field is included in the @filter@ the new document will use this @_id@, otherwise a random value will be used see "Upsert Considerations":#considerationsUpsert for details. When false the command will only update a document if one matches the filter. Defaults to @false@.| h3(#commandUpdateOne). updateOne Command From 6edc9b1bf4b5ff0603fadc9b866f4c6f118fe351 Mon Sep 17 00:00:00 2001 From: Aaron Morton Date: Thu, 23 Feb 2023 12:07:36 +1300 Subject: [PATCH 7/7] tweak doc limits from team meeting resolves #135 --- docs/jsonapi-spec.textile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/jsonapi-spec.textile b/docs/jsonapi-spec.textile index bcce1e7449..c117b1be7e 100644 --- a/docs/jsonapi-spec.textile +++ b/docs/jsonapi-spec.textile @@ -203,15 +203,15 @@ h4(#documentLimitsFields). Document Field Limits The maximum length of a "field name":#documentFieldNames is 48 characters. -The maximum number of fields allowed in a single JSON object is 32. +The maximum number of fields allowed in a single JSON object is 64. h4(#documentLimitsValues). Document Value Limits The maxium size of field values are: |_. JSON Type |_. Maximum Value | -| @string@ | Maximum length of 4096 unicode characters | -| @number@ | Arbitrary sized number described using "BigDecimal in Java":https://docs.oracle.com/javase/8/docs/api/java/math/BigDecimal.html#BigDecimal-java.lang.String- | +| @string@ | Maximum length of 16384 unicode characters | +| @number@ | TODO: define max num that is well handled by BigDecimal | h4(#documentLimitsArray). Document Array Limits