[Request] Long-running API operations should return Accepted response and notify by WebSocket

[andrewazores]

Describe the feature

API operations like archiving active recordings, generating reports (from active or archived recordings), "View in Grafana" (transfer file to jfr-datasource) etc. can take a long time to complete depending on external factors such as the size of a JFR file, the speed of network connections between Cryostat and target applications or report generator or jfr-datasources, etc. Currently these API actions are implemented such that the client sends a request to Cryostat, then Cryostat performs the action completely, and then Cryostat sends a response to the client based on the result of the performed action. Instead, Cryostat should receive the client's request, perform some initial validations, and send either a 4xx response if the client's request failed the validations, or a 202 response if it will proceed. Then, the final result of the action should be emitted as a WebSocket notification.

Taking active recording archiving as an example, currently the flow is like this:

1. Client sends a PATCH request to Cryostat
2. Cryostat receives the PATCH and creates an S3 file upload request
3. Cryostat opens a connection (JMX or Agent HTTP) to the target JVM and begins to receive JFR bytes
4. Cryostat pipes the target JFR bytes to the S3 upload
5. Cryostat finishes reading the JFR bytes from the target and finalizes the S3 upload
6. Cryostat responds to the client's PATCH with some information about the uploaded file

Instead, it should go like this:

1. Client sends a PATCH request to Cryostat
2. Cryostat receives the PATCH and checks that the specified active recording exists and that the target JVM is reachable (ex. try to open a connection and do something relatively lightweight like compute its JVM ID). If this check succeeds respond to the PATCH with 202, if it fails respond with a 404 (recording not found) or 502 (target not reachable) etc.
3. In the background, Cryostat creates the S3 file upload request, opens a target connection, pipes the bytes, etc. - same as steps 2-5 above
4. Cryostat emits a WebSocket notification, either indicating task successful completion or task failure.

Anything other information?

No response

[andrewazores]

I had some thoughts swimming in my brain last night while trying to fall asleep.

1. Client sends a PATCH request to Cryostat
2. Cryostat receives the PATCH and checks that the specified active recording exists and that the target JVM is reachable (ex. try to open a connection and do something relatively lightweight like compute its JVM ID). If this check succeeds respond to the PATCH with 202, if it fails respond with a 404 (recording not found) or 502 (target not reachable) etc. An OK response (ie. HTTP 202) should include some job identifier, for example a UUID generated in this request handler.
3. In the background, Cryostat creates the S3 file upload request, opens a target connection, pipes the bytes, etc. - same as steps 2-5 above. This background request should not be kicked off (job added to work queue, or worker thread kicked off, etc.) until the HTTP response from the previous step has been fully written out to and received by the client. This ensures that the client knows the ID of the job before the job can possibly have been completed.
4. Cryostat emits a WebSocket notification, either indicating task successful completion or task failure, including the job ID in the message.
5. The client has been listening for a message of this type with this ID, receives it, and acts upon it.