Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement M1 Content Hosting Provisioning API #18

Closed
rjb1000 opened this issue Nov 17, 2022 · 41 comments
Closed

Implement M1 Content Hosting Provisioning API #18

rjb1000 opened this issue Nov 17, 2022 · 41 comments
Assignees
Labels
feature A new high-level feature

Comments

@rjb1000
Copy link
Contributor

rjb1000 commented Nov 17, 2022

Specification

Because there can only be zero or one Content Hosting Configuration associated with a parent Provisioning Session, the Content Hosting Configuration is identified by a fixed subresource path rather than by a UUID.

Additional implementation specification:

  • M1_ContentHostingProvisioning_purgeContentHostingCache: The 5GMS AF does not need to validate the optional ECMA regular expression request parameter prior to invoking the purge operation of the 5GMS AS instance(s) current serving the Content Hosting Configuration. However, the 5GMS AF should check for errors returned by each 5GMS AS instance when the M3 purge operation is invoked, and return a suitably non-specific error to the M1 invoker (e.g. title "Application Server encountered a problem when purging its cache" and description "Application Server encountered a problem when purging its cache for the regular expression 'XXXX'."). Any errors returned at M1 must be generated locally by the 5GMS AF so as not to leak any information about the number and nature of the 5GMS AS instances managed by the 5GMS AF.

The following operations are out of scope for the present implementation:

  • M1_ContentHostingProvisioning_patchContentHostingConfiguration. (This operation is underspecified in Rel-17.) This operation should return a 501 Not Implemented HTTP response with a suitably descriptive human-readable error message in the ProblemDetails JSON response document.
    • Note that it will therefore only be possible to modify an existing Content Hosting Configuration using the updateContentHostingConfiguration operation.

Relevant specifications

  • TS 26.512 clauses 4.3.3 and 7.6.
@rjb1000 rjb1000 added the feature A new high-level feature label Nov 17, 2022
@rjb1000 rjb1000 moved this from Backlog to Ready for development in 5GMS: M1d Provisioning Dec 7, 2022
@devbbc devbbc moved this from Ready for development to In Progress in 5GMS: M1d Provisioning Dec 12, 2022
@devbbc
Copy link
Contributor

devbbc commented Dec 14, 2022

TS 26.512 CR0028r2 (Rel-17) went to SA#98-e as SP-221043. On the assumption that this will be approved this week by SA Plenary, have started to develop the M1 Content Hosting Provisioning API against this pre-publication V17.3.0 defintion.

Created openAPI model source files based on this pre-published YAML definition. The newly generated code files have different signatures for functions to create contentHostingConfigurations and distributionConfigurations to the previous release. This means the AF has to be refactored accordingly to call these functions and store the populated contentHostingConfiguration in its data structure.
To support this development, @davidjwbbc has created example contentHostingConfigurations.

@rjb1000
Copy link
Contributor Author

rjb1000 commented Dec 14, 2022

@devbbc : Full details of the M1 uplift in TS 26.512 V17.3.0 can be found on issue #19.

I have marked this in progress since you are effectively working on this at the same time as this issue.

@davidjwbbc
Copy link
Contributor

I have added a comment to #19 noting that the entryPointPath should be checked for conformance using a regex compiled with the GNU C Library regex function regcomp(..., REG_EXTENDED). The compiled regex can later be used with regexec() to check a string for a match.

@devbbc
Copy link
Contributor

devbbc commented Dec 20, 2022

Completed the uplift on #19. Now need to implement Create, Retrieve, Update and Destroy operations for contentHostingConfiguration resource on the M1 interface .

@devbbc
Copy link
Contributor

devbbc commented Dec 21, 2022

Have implemented the the Create, Retrieve, Update and Destroy operations for the contentHostingConfiguration resource on the M1 interface.

The following examples show that the operations are working as specified. However, the code still needs tidying up.

Examples illustrating the operations

createContentHostingConfiguration

To create a Content Hosting Configuration under a previously created Provisioning Session:

$ curl -H 'User-Agent: AF' -H 'Content-Type: application/json' --data @ContentHostingConfiguratioon_v17.3.0_Big-Buck-Bunny_pull-ingest_http_and_https.json -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/json
> Content-Length: 659
> 
* upload completely sent off: 659 out of 659 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Connection: Keep-Alive
< Content-Length: 636
< Content-Type: application/json
< Location: /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
< Date: Wed, 21 Dec 2022 17:56:14 GMT
< 
{
	"distributionConfigurations":	[{
			"baseURL":	"http://localhost/m4d/provisioning-session-a4a6297c-8158-41ed-8dbe-97f9dc07e15d/",
			"canonicalDomainName":	"localhost"
		}, {
			"baseURL":	"https://localhost/m4d/provisioning-session-a4a6297c-8158-41ed-8dbe-97f9dc07e15d/",
			"canonicalDomainName":	"localhost",
			"certificateId":	"testcert1"
		}],
	"entryPointPath":	"BigBuckBunny_4s_onDemand_2014_05_09.mpd",
	"ingestConfiguration":	{
		"baseURL":	"https://ftp.itec.aau.at/datasets/DASHDataset2014/BigBuckBunny/4sec/",
		"protocol":	"urn:3gpp:5gms:content-protocol:http-pull-ingest",
		"pull":	true
	},
	"name":	"Big Buck Bunny"
* Connection #0 to host 127.0.0.22 left intact
}

retrieveContentHostingConfiguration

To retrieve the Content Hosting Configuration of a Provisioning Session:

ubuntu@ogs:~$ curl -H 'User-Agent: AF' -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> GET /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 636
< Content-Type: application/json
< Location: /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
< Date: Wed, 21 Dec 2022 17:57:17 GMT
< 
{
	"distributionConfigurations":	[{
			"baseURL":	"http://localhost/m4d/provisioning-session-a4a6297c-8158-41ed-8dbe-97f9dc07e15d/",
			"canonicalDomainName":	"localhost"
		}, {
			"baseURL":	"https://localhost/m4d/provisioning-session-a4a6297c-8158-41ed-8dbe-97f9dc07e15d/",
			"canonicalDomainName":	"localhost",
			"certificateId":	"testcert1"
		}],
	"entryPointPath":	"BigBuckBunny_4s_onDemand_2014_05_09.mpd",
	"ingestConfiguration":	{
		"baseURL":	"https://ftp.itec.aau.at/datasets/DASHDataset2014/BigBuckBunny/4sec/",
		"protocol":	"urn:3gpp:5gms:content-protocol:http-pull-ingest",
		"pull":	true
	},
	"name":	"Big Buck Bunny"
* Connection #0 to host 127.0.0.22 left intact
}

updateContentHostingConfiguration

Update the entryPointPath of the Content Hosting Configuration to BigBuckBunny_4s_onDemand_2014_05_10.mpd by sending a replacement JSON document to the 5GMS AF:

$ curl -H 'User-Agent: AF' -X PUT -H 'Content-Type: application/json' --data @ContentHostingConfiguration_v17.3.0_Big-Buck-Bunny_pull-ingest_http_and_https.json -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration

*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> PUT /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/json
> Content-Length: 659
> 
* upload completely sent off: 659 out of 659 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Connection: Keep-Alive
< Content-Type: application/json
< Location: /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
< Date: Wed, 21 Dec 2022 17:57:45 GMT
< 
* Connection #0 to host 127.0.0.22 left intact

To verify if the Content Hosting Configuration has been successfully updated:

$ curl -H 'User-Agent: AF' -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> GET /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 636
< Content-Type: application/json
< Location: /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
< Date: Wed, 21 Dec 2022 17:57:58 GMT
< 
{
	"distributionConfigurations":	[{
			"baseURL":	"http://localhost/m4d/provisioning-session-a4a6297c-8158-41ed-8dbe-97f9dc07e15d/",
			"canonicalDomainName":	"localhost"
		}, {
			"baseURL":	"https://localhost/m4d/provisioning-session-a4a6297c-8158-41ed-8dbe-97f9dc07e15d/",
			"canonicalDomainName":	"localhost",
			"certificateId":	"testcert1"
		}],
	"entryPointPath":	"BigBuckBunny_4s_onDemand_2014_05_10.mpd",
	"ingestConfiguration":	{
		"baseURL":	"https://ftp.itec.aau.at/datasets/DASHDataset2014/BigBuckBunny/4sec/",
		"protocol":	"urn:3gpp:5gms:content-protocol:http-pull-ingest",
		"pull":	true
	},
	"name":	"Big Buck Bunny"
* Connection #0 to host 127.0.0.22 left intact
}

destroyContentHostingConfiguration

To destroy an existing Content Hosting Configuration:

$curl -H 'User-Agent: AF' -X DELETE -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> DELETE /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Connection: Keep-Alive
< Date: Wed, 21 Dec 2022 17:58:23 GMT
< 
* Connection #0 to host 127.0.0.22 left intact

Querying on the deleted resource:

$ curl -H 'User-Agent: AF' -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> GET /3gpp-m1/v2/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Connection: Keep-Alive
< Content-Length: 251
< Content-Type: application/problem+json
< Date: Wed, 21 Dec 2022 17:58:42 GMT
< 
{
	"type":	"/3gpp-m1/v2",
	"title":	"The Content Hosting Configuration is not found",
	"status":	404,
	"detail":	"Unable to retrieve the Content Hosting Configuration",
	"instance":	"/provisioning-sessions/a4a6297c-8158-41ed-8dbe-97f9dc07e15d"
}
* Connection #0 to host 127.0.0.22 left intact

@devbbc
Copy link
Contributor

devbbc commented Dec 29, 2022

Started tidying up the code. Meanwhile, realised that I haven't tested the effect of the above M1 interface operations on the recent Application server (i.e Application Server from the "feature/uplift-to-ts26-512-v17-3-0" branch).

So, performed tests by initiating operations through the M1 interface. The AF processed the operations that came through the M1 interface, populated its internal data structure and triggered corresponding operations through the M3 interface to the Application Server. So, tests worked as expected.

Note that M1_ContentHostingProvisioning_purgeContentHostingCache is not yet implemented.

@devbbc
Copy link
Contributor

devbbc commented Dec 30, 2022

Have implemented part of the M1_ContentHostingProvisioning_purgeContentHostingCache. The Application Function can now handle purge requests but lacks proper handling of the response.

Sending back response to the M1 client is work in progress.

@devbbc
Copy link
Contributor

devbbc commented Jan 4, 2023

Implemented the M1_ContentHostingProvisioning_purgeContentHostingCache operations. The Application Function can now handle purge requests from the M1 client, send requests to the Application Server and handle responses through the M3 interface and send response back to the M1 client.

The example below demonstrates a successful purge operation on an already provisioned content Hosting Configuration resource:
(Here the regular expression \.mpd$ is sent in the POST body with the Content-Type: application/x-www-form-urlencoded

curl -H 'User-Agent: AF'-X POST -H 'Content-Type: application/x-www-form-urlencoded' --data "pattern=\.mpd$" -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/1b3b4e28-8c30-41ed-bd24-f3baa7d5e878/content-hosting-configuration/purge
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/1b3b4e28-8c30-41ed-bd24-f3baa7d5e878/content-hosting-configuration/purge HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 14
> 
* upload completely sent off: 14 out of 14 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 0
< Date: Wed, 04 Jan 2023 13:03:23 GMT
< 

Below is an example of purge operation without regular expression:

 curl -H 'User-Agent: AF' -X POST -H 'Content-Type: application/x-www-form-urlencoded' -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/1b3b4e28-8c30-41ed-bd24-f3baa7d5e878/content-hosting-configuration/purge
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/1b3b4e28-8c30-41ed-bd24-f3baa7d5e878/content-hosting-configuration/purge HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/x-www-form-urlencoded
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 0
< Date: Wed, 04 Jan 2023 13:04:51 GMT
< 
* Connection #0 to host 127.0.0.22 left intact

(works without -H 'Content-Type: application/x-www-form-urlencoded' option as well).

@devbbc
Copy link
Contributor

devbbc commented Jan 4, 2023

Implementation design of purge operation

The M1 client sends a Purge request as a POST message. On receipt of this request, the Application Function populates a list that holds the context related to the purge operation such as the regular expression and resource identifier.

It then sends a Purge request to the Application Server through the M3 interface. On receipt of a response from an Application Server, the Application Function checks the HTTP status code.

  • If the status code indicates an error, the Application Function sends a response back to the M1 client with a suitably non-specific error. The Application Function still continues to handle responses from other Application Server instances.
  • On the other hand if the status code indicates a successful operation, the Application Function withholds sending response to the M1 client until it has either received responses from all Application Servers it is managing or receives an error from one of them, per the previous bullet.

@devbbc
Copy link
Contributor

devbbc commented Jan 4, 2023

As the Application Function does not need to validate the optional ECMA regular expression request parameter prior to invoking the purge operation of the 5GMS AS instance(s), it is left to the Application Server to validate this expression and return an error if the expression is invalid.

Currently, if the purge regex pattern is invalid, the Application Server raises an exception and produces a local error log. However, the Application Server returns a response with status code 200 to the Application Function. As a result, the Application Function sends a response with status 200 at M1.

Raised a 5G-MAG/rt-5gms-application-server#52 so that the Application Server returns a 422 (Unprocessable Entity) response in the case of an invalid regex. On receipt of the response with status 422 at M3, the Application Function will trigger a new response at M1 indicating an error with status code 422.

@devbbc
Copy link
Contributor

devbbc commented Jan 5, 2023

The Application Function is modified to handle the 422 response at M3 and trigger a new response at M1 indicating an error with status code 422.

Haven't tested this with the AS as it is not yet modified to send 422 response.

@devbbc devbbc moved this from In Progress to Ready for review in 5GMS: M1d Provisioning Jan 5, 2023
@devbbc devbbc assigned davidjwbbc and unassigned devbbc Jan 5, 2023
@devbbc
Copy link
Contributor

devbbc commented Jan 18, 2023

Tested the AF's handling of 422 response at M3 with the modified AS. The AF handles the 422 response provided by the AS at M3 and triggers a new response at M1 indicating an error with status code 422.

Sample output:

$ curl -H 'User-Agent: AF'-X POST -H 'Content-Type: application/x-www-form-urlencoded' --data "pattern=^[^/#?:]{1,}(/[^#?/]{1,})*(?[^#]*)?(#.*)?$" -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/b63927fa-9758-41ed-a7e7-a7d2e27a6f4a/content-hosting-configuration/purge
* Could not resolve host: POST
* Closing connection 0
curl: (6) Could not resolve host: POST
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#1)
> POST /3gpp-m1/v2/provisioning-sessions/b63927fa-9758-41ed-a7e7-a7d2e27a6f4a/content-hosting-configuration/purge HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF-X
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 50
> 
* upload completely sent off: 50 out of 50 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 422 Unprocessable Entity
< Connection: Keep-Alive
< Content-Length: 328
< Content-Type: application/problem+json
< 
{
	"type":	"/3gpp-m3/v1",
	"title":	"Application Server encountered a problem when purging its cache",
	"status":	422,
	"detail":	"Application Server possibly encountered problem with regex pattern=^[^/#?:]{1,}(/[^#?/]{1,})*(?[^#]*)?(#.*)?$",
	"instance":	"/content-hosting-configurations/b63927fa-9758-41ed-a7e7-a7d2e27a6f4a"
* Connection #1 to host 127.0.0.22 left intact

@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 19, 2023

"type": "/3gpp-m3/v1"

This looks wrong for an M1 error message from the Application Function. Looks like you're basing the M1 response on the M3 response object instead of the M1 request.

@davidjwbbc has provided details of the parse error in ProblemDetail.invalidParams in the M3 response. This should be copied from the M3 response to the M1 response.

@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 19, 2023

@davidjwbbc also noticed that the Server HTTP response header is missing from the response message per clause 6.2.3.3.1 of TS 26.512. (This applies to all response messages you are generating from the AF.)

Also clauses 6.2.1.1 and 6.2.3.4 of TS 26.512 specify Last-Modified and ETag response headers on all responses. But I interpret this clause as only applying RESTful resources (i.e. the M1_*_create*, M1_*_retrieve* and M1_*_update* service operations in the context of this issue) and not to error messages.

  • Last modification time: The AF needs to store the server's timestamp of when it processed the last change to the RESTful entity.
  • Entity tag: @davidjwbbc: Is there are convenient hashing function in Open5GS (or one of its dependent libraries) that @devbbc can use to generate an entity tag value?

@davidjwbbc
Copy link
Contributor

There's a simple hash function which hashes a buffer or string to an unsigned int called ogs_hashfunc_default. But if we want a better hash then we could use something like:

#include <gnutls/gnutls.h>

...
  char result[128];
  size_t result_len = sizeof(result);
  gnutls_datum_t data;

  data.data = input_buffer;
  data.size = input_size;
  gnutls_fingerprint(GNUTLS_DIG_SHA256, &data, result, &result_len);
  /* SHA256 hash is in result and is result_len chars long */
...

or

#include <openssl/crypto.h>
#include <openssl/sha.h>

...
  char result[SHA256_DIGEST_LENGTH];
  char *hexstr;

  SHA256(input_buffer, input_length, result);
  hexstr = OPENSSL_buf2hexstr(result, SHA256_DIGEST_LENGTH);
  /* SHA256 is in hexstr as a nul terminated string */
...
  OPENSSL_free(hexstr);
...

@davidjwbbc
Copy link
Contributor

Passing back to @devbbc to deal with the Server, ETag, Last-Modified and Cache-Control headers.

@davidjwbbc davidjwbbc moved this from Ready for review to In Progress in 5GMS: M1d Provisioning Jan 19, 2023
@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 24, 2023

This would be an improvement to the way in which open5gs handles its request headers, preventing headers which should only appear once from being included multiple times.

Something at the back of my mind suggests that:

Header: value1
Header: value2

and

Header: value1 value2

are considered equivalent in HTTP.

So maybe the duplication permitted by the current Open5GS SBI library code is actually fine, @davidjwbbc?

@davidjwbbc
Copy link
Contributor

This would be an improvement to the way in which open5gs handles its request headers, preventing headers which should only appear once from being included multiple times.

Something at the back of my mind suggests that:

Header: value1
Header: value2

and

Header: value1 value2

are considered equivalent in HTTP.

So maybe the duplication permitted by the current Open5GS SBI library code is actually fine, @davidjwbbc?

Multi-value headers (ones which in the spec are defined as #somevaluetype) are comma separated when grouped into one header value. These (plus a limited set of exceptions, e.g. Set-Cookie) are the only headers permitted to appear more than once. So

Header: value1
Header: value2

...would be interpreted as...

Header: value1,value2

@devbbc
Copy link
Contributor

devbbc commented Jan 25, 2023

Modified the AF so that M1 response is based on the M1 request as opposed to the M3 response object.
Also, can parse the error details in the M3 response, but haven't implemented copying it to the M1 error response. This will become handy once I have custom problem/error response routines, which I will be working on next.

Sample parsed error details shown below:

	{
	"title":	"Unprocessable Entity",
	"detail":	"Bad regex: unknown extension at position 28",
	"type":	"/3gpp-m3/v1",
	"status":	422,
	"instance":	"/content-hosting-configurations/88fd5d18-9ccd-41ed-8f35-adaccac7d059/purge",
	"invalidParams":	[{
			"param":	"pattern",
			"reason":	"unknown extension at position 28"
		}]
}

Sample M1 response for correct regular expression:

$ curl -H 'User-Agent: AF' -H 'Content-Type: applicationx-www-form-urlencoded' --data "pattern=^\.mpd$" -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 15
> 
* upload completely sent off: 15 out of 15 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: Keep-Alive
< Content-Length: 0
< Date: Wed, 25 Jan 2023 16:30:41 GMT
< 
* Connection #0 to host 127.0.0.22 left intact

Sample M1 response for incorrect regular expression

$ curl -H 'User-Agent: AF' -H 'Content-Type: application/x-www-form-urlencoded' --data "pattern=^[^/#?:]{1,}(/[^#?/]{1,})*(?[^#]*)?(#.*)?$" -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 50
> 
* upload completely sent off: 50 out of 50 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 422 Unprocessable Entity
< Connection: Keep-Alive
< Content-Length: 279
< Content-Type: application/problem+json
< Date: Wed, 25 Jan 2023 16:30:58 GMT
< 
{
	"type":	"/3gpp-m1/v2",
	"title":	"Problem with purge operation",
	"status":	422,
	"detail":	"Problem with regex pattern=^[^/#?:]{1,}(/[^#?/]{1,})*(?[^#]*)?(#.*)?$",
	"instance":	"/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059"
* Connection #0 to host 127.0.0.22 left intact

Other headers such as Server will be part of custom response routines.

@davidjwbbc
Copy link
Contributor

The invalidParams for param == "pattern" should also be copied through to the M1 response.

I also can't decide if the instance should be /provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration or not, @rjb1000 your thoughts?

@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 25, 2023

I also can't decide if the instance should be /provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration or not, @rjb1000 your thoughts?

That looks like the correct M1 path, @davidjwbbc. What are your doubts?

@davidjwbbc
Copy link
Contributor

@devbbc has coded

"instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059"

...but I'm thinking it should probably be...

"instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration"

...or even...

"instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge"

... to indicate the instance that caused the error.

@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 26, 2023

..but I'm thinking it should probably be...

"instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration"

...or even...

"instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge"

... to indicate the instance that caused the error.

Oh, I see now. Yes, I agree that the instance property of the error at M1 needs to indicate the path of the original M1 request, and not simply a copy of the instance property in the M3 error.

So I agree that in this particular case it should be /provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059/content-hosting-configuration/purge.

I think the only property that can safely be copied from the M3 error to the M1 error is probably invalidParams. Everything else needs to be created anew using a fresh ProblemDetails JSON object. Even title, details and status shouldn't be copied blindly. In this case, it happens to be the same 422 (Unprocessable Entity) status at M3 and M1, but that's just a happy coincidence, and should be part of some conditional code structure (if/then/else or case/switch). In general, we may need to expose a different status at M1 than the one received at M3.

@devbbc
Copy link
Contributor

devbbc commented Jan 26, 2023

For the purge operation, the only response code specified in TS26512_M1_ContentHostingProvisioning.yaml is 200. No error codes specified.

@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 26, 2023

For the purge operation, the only response code specified in TS26512_M1_ContentHostingProvisioning.yaml is 200. No error codes specified.

That may be correct as of TS 26.512 V17.3.0. But we are discussing possible changes to that.

@devbbc
Copy link
Contributor

devbbc commented Jan 30, 2023

There are a couple of limitations in using Open5GS functions to send error messages:

  • Instance components are limited to two components. Hence, the sample output in the above comment had
    "instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059".
  • There is no way to copy invalidParams from the M3 error to the M1 error.

Hence, wrote custom functions, based on the original functions to handle error messages. With these functions, the error response now contains full path for instance and can include invalidParams copied from the M3 error to the M1 error as shown below:

$  curl  -H 'User-Agent: AF' -H 'Content-Type: applicationx-www-form-urlencoded' --data "pattern=^[^/#?:]{1,}(/[^#?/]{1,})*(?[^#]*)?(#.*)?$" -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a0d20aa2-a0a5-41ed-9dd3-25c4ddc6fd7c/content-hosting-configuration/purge
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/a0d20aa2-a0a5-41ed-9dd3-25c4ddc6fd7c/content-hosting-configuration/purge HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/x-www-form-urlencoded
> Content-Length: 50
> 
* upload completely sent off: 50 out of 50 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 422 Unprocessable Entity
< Connection: Keep-Alive
< Content-Length: 424
< Content-Type: application/problem+json
< Date: Mon, 30 Jan 2023 13:55:26 GMT
< 
{
	"type":	"/3gpp-m1/v2",
	"title":	"Problem occured during cache purge",
	"status":	422,
	"detail":	"Application Server possibly encountered problem with regex pattern=^[^/#?:]{1,}(/[^#?/]{1,})*(?[^#]*)?(#.*)?$",
	"instance":	"/provisioning-sessions/a0d20aa2-a0a5-41ed-9dd3-25c4ddc6fd7c/content-hosting-configuration/purge",
	"invalidParams":	[{
			"param":	"pattern",
			"reason":	"unknown extension at position 28"
		}]
* Connection #0 to host 127.0.0.22 left intact
}

@rjb1000
Copy link
Contributor Author

rjb1000 commented Jan 30, 2023

There are a couple of limitations in using Open5GS functions to send error messages:

  • Instance components are limited to two components. Hence, the sample output in the above comment had
    "instance": "/provisioning-sessions/88fd5d18-9ccd-41ed-8f35-adaccac7d059".

Thinking about it, @devbbc, the Open5GS implementation is probably correct in the strictest sense of identifying the RESTful resource that was being manipulated by the operation. On reflection, I think we maybe ought not to change that aspect. Instead, we could make the human-readable detail more informative. For example:

Syntax error in supplied regular expression "^[^/#?:]{1,}(/[^#?/]{1,})(?[^#])?(#.*)?$" when attempting to purge Content Hosting cache.

We can chat separately about the problem you encountered populating invalidParams in a new ProblemDetails JSON body.

@davidjwbbc
Copy link
Contributor

davidjwbbc commented Jan 31, 2023

By taking a copy of the ogs_sbi_server_send_error and ogs_sbi_server_send_problem functions we can solve a few of the issues we've been having.

  1. Instance name returned in a problem did not indicate the the correct instance path. This is because ogs_sbi_server_send_error will only use up to the second path component after the initial protocol and version indicators. For example a path of /3gpp-m1/v2/a/path/to/an/instance would only return /a/path as the instance path.
  • This could be circumvented by creating an OpenAPI_problem_details_t structure with the correct details and passing that to ogs_sbi_server_send_problem.
  • For example, the instance field will need to contain 4 path elements for problems with certificates, e.g. /provisioning-session/12345678-9abc-def0-123456789abc/certificate/def01234-5678-9abc-def012345678, so fixing this now will help with Implement M1 Server Certificates Provisioning API #17.
  1. No invalidParams field in the problem. The ogs_sbi_server_send_error function does not populate the invalidParams field and takes no parameters to do so.
    • Just like 1, this can be circumvented by building our own OpenAPI_problem_details_t and passing to ogs_sbi_server_send_problem.
    • (At the moment this is only an issue for the cache purge operation.)
  2. No custom header for Server. The error routines, ogs_sbi_server_send_error and ogs_sbi_server_send_problem do not provide a way to pass custom headers.
    • ogs_sbi_server_send_problem would need to be passed custom headers to include in the response message.

So, to fix 1 & 2 we need a similar routine to ogs_sbi_server_send_error, but one that copies the correct number of path elements for the instance identifier and can populate the invalidParams field. This should be called msaf_server_send_error and should reside in new source files called server.c and server.h.

To fix 3 we will need a similar routine to ogs_sbi_server_send_problem, called msaf_server_send_problem, which uses a msaf_server_new_response function to obtain a new response structure with the custom headers already populated (this same function can be used to create the new response structure for normal 2xx responses in the 5GMS AF too, but those parts of the code will also have to add ETag, Last-Modified and Cache-Control headers when body contents are also to be returned). Again, the msaf_server_send_problem and msaf_server_new_response functions should be defined in server.c and server.h.

@rjb1000
Copy link
Contributor Author

rjb1000 commented Feb 1, 2023

@davidjwbbc: Please can we generalise the names of the three new functions to nf_server_send_error, nf_server_send_problem and nf_server_new_response with a view to reuse in future Network Functions that are implemented as part of the 5G-MAG Reference Tools?

@devbbc
Copy link
Contributor

devbbc commented Feb 1, 2023

I think these functions can be defined in already existing sbi-path.c and sbi-path.h rather than creating new files server.c and server.h. sbi-path.c has only two functions defined in it and they are related to server and client handlers. These functions could fit there as well.

@rjb1000
Copy link
Contributor Author

rjb1000 commented Feb 1, 2023

Putting the new functions in their own new source files might be better for reuse elsewhere, though.

@davidjwbbc
Copy link
Contributor

The sbi-path.[ch] are for dealing with the requests arriving and posting the events for them to the right SM. They are not the place for generic server functions, please put the new routines in server.[ch] as requested. As @rjb1000 points out this will be better for re-use. The server.[ch] files will be expanded with more server functions when issue #12 is implemented and we stop relying on the open5gs sbi library.

@davidjwbbc
Copy link
Contributor

I've created a separate issue, #37, for the extra headers as they will be needed for more than just ContentHostingConfiguration responses.

@devbbc
Copy link
Contributor

devbbc commented Feb 1, 2023

As I had previously implemented most of the functions, had to restructure them according to the recent comments. Now, the application function has server.c and server.h. Completed functions nf_server_send_error, nf_server_send_problem. nf_server_new_response is still in progress.

@devbbc
Copy link
Contributor

devbbc commented Feb 6, 2023

Implemented createContentHostingConfiguration operation using the custom nf_server_new_response developed on #37.

The below sample output shows the response generated by this function:

 $ curl -H 'User-Agent: AF' -H 'Content-Type: application/json' --data @/home/ubuntu/git0-2912/rt-5gms-application-function/examples/ContentHostingConfiguration_Big-Buck-Bunny_pull-ingest_http_and_https.json -v http://127.0.0.22:7777/3gpp-m1/v2/provisioning-sessions/a98a09ca-a6d5-41ed-8b6c-edeafa546a93/content-hosting-configuration
*   Trying 127.0.0.22:7777...
* TCP_NODELAY set
* Connected to 127.0.0.22 (127.0.0.22) port 7777 (#0)
> POST /3gpp-m1/v2/provisioning-sessions/a98a09ca-a6d5-41ed-8b6c-edeafa546a93/content-hosting-configuration HTTP/1.1
> Host: 127.0.0.22:7777
> Accept: */*
> User-Agent: AF
> Content-Type: application/json
> Content-Length: 375
> 
* upload completely sent off: 375 out of 375 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 201 Created
< Connection: Keep-Alive
< Content-Length: 636
< Cache-Control: 60
< Last-Modified: Tue, 07 Feb 2023 10:53:47 GMT
< ETag: 1f05ec542503481e293beb2d824903b08f146264c9ae9d5e47b399dde94ffa1e
< Content-Type: application/json
< Location: /3gpp-m1/v2/provisioning-sessions/a98a09ca-a6d5-41ed-8b6c-edeafa546a93/content-hosting-configuration
< Server: 5GMSdAF-af.localdomain/17.3.0 (info.title=M1_ContentHostingProvisioning;info.version=2.1.0) rt-5gms-application-function/1.1.0
< Date: Tue, 07 Feb 2023 10:53:47 GMT
< 
{
	"distributionConfigurations":	[{
			"baseURL":	"http://localhost/m4d/provisioning-session-a98a09ca-a6d5-41ed-8b6c-edeafa546a93/",
			"canonicalDomainName":	"localhost"
		}, {
			"baseURL":	"https://localhost/m4d/provisioning-session-a98a09ca-a6d5-41ed-8b6c-edeafa546a93/",
			"canonicalDomainName":	"localhost",
			"certificateId":	"testcert1"
		}],
	"entryPointPath":	"BigBuckBunny_4s_onDemand_2014_05_09.mpd",
	"ingestConfiguration":	{
		"baseURL":	"https://ftp.itec.aau.at/datasets/DASHDataset2014/BigBuckBunny/4sec/",
		"protocol":	"urn:3gpp:5gms:content-protocol:http-pull-ingest",
		"pull":	true
	},
	"name":	"Big Buck Bunny"
* Connection #0 to host 127.0.0.22 left intact
}

@devbbc devbbc closed this as completed Feb 6, 2023
@github-project-automation github-project-automation bot moved this from In Progress to Done in 5GMS: M1d Provisioning Feb 6, 2023
@davidjwbbc davidjwbbc self-assigned this Feb 14, 2023
@davidjwbbc davidjwbbc moved this from Done to Ready for review in 5GMS: M1d Provisioning Feb 14, 2023
@davidjwbbc
Copy link
Contributor

Reopening for review

@davidjwbbc davidjwbbc reopened this Feb 14, 2023
@davidjwbbc
Copy link
Contributor

There was no error handling when the ContentHostingConfiguration didn't pass verification (i.e. certificate Ids used in the distributionConfiguration match existing certificates for the provisioning session). I've modified the code to feed back when the verification fails and issue a 400 Bad Request response to the M1 client.

@davidjwbbc
Copy link
Contributor

Fixed a bug where the output of the JSON parser was not being checked, so malformed JSON caused the AF to crash.

Tested this fixed version and the ContentHostingProvisioning interface looks good, closing this issue.

@github-project-automation github-project-automation bot moved this from Ready for review to Done in 5GMS: M1d Provisioning Mar 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature A new high-level feature
Projects
Development

No branches or pull requests

3 participants