io.camunda.zeebe.broker.client.api.BrokerErrorException: Received error from broker (INTERNAL_ERROR): Processing paused for partition '3'

[korthout]

Describe the bug

io.camunda.zeebe.broker.client.api.BrokerErrorException: Received error from broker (INTERNAL_ERROR): Processing paused for partition '3'

This error was logged about 40 times in a short timespan.

To Reproduce

Not sure, but likely it involves pausing a partition during processing.

Expected behavior

This is probably just noise. We should log that the partition is paused at INFO level and:

• no longer accept requests for a paused partition
• log at WARN level information about the requests that can no longer be handled

Log/Stacktrace

Full logs are available on Google Drive.

Full Stacktrace

io.camunda.zeebe.broker.client.api.BrokerErrorException: Received error from broker (INTERNAL_ERROR): Processing paused for partition '3'

at io.camunda.zeebe.broker.client.impl.BrokerRequestManager.handleResponse ( io/camunda.zeebe.broker.client.impl/BrokerRequestManager.java:195 )
at io.camunda.zeebe.broker.client.impl.BrokerRequestManager.lambda$sendRequestInternal$2 ( io/camunda.zeebe.broker.client.impl/BrokerRequestManager.java:144 )
at io.camunda.zeebe.scheduler.future.FutureContinuationRunnable.run ( io/camunda.zeebe.scheduler.future/FutureContinuationRunnable.java:28 )
at io.camunda.zeebe.scheduler.ActorJob.invoke ( io/camunda.zeebe.scheduler/ActorJob.java:85 )
at io.camunda.zeebe.scheduler.ActorJob.execute ( io/camunda.zeebe.scheduler/ActorJob.java:42 )
at io.camunda.zeebe.scheduler.ActorTask.execute ( io/camunda.zeebe.scheduler/ActorTask.java:122 )
at io.camunda.zeebe.scheduler.ActorThread.executeCurrentTask ( io/camunda.zeebe.scheduler/ActorThread.java:130 )
at io.camunda.zeebe.scheduler.ActorThread.doWork ( io/camunda.zeebe.scheduler/ActorThread.java:108 )
at io.camunda.zeebe.scheduler.ActorThread.run ( io/camunda.zeebe.scheduler/ActorThread.java:227 )

Environment:

• OS:
• Zeebe Version:
• Configuration:

[npepinpe]

INTERNAL_ERROR is also likely the wrong error code, honestly. We probably want to return something like UNAVAILABLE, indicating the system is currently unavailable, but may become so eventually. This, or something like INVALID_STATE, or FAILED_PRECONDITION. I would opt for unavailable, personally.

I would propose introducing a new error code, which is mapped to 503/SERVICE UNAVAILABLE (for REST) and 14/UNAVAILABLE (for gRPC), and not logged as an error but debug (as with other temporarily unavailable things).

I would also argue this is more of a rejection than an error. There is no error here, and the command may be well formed, we're simply rejecting to process it. However, I understand we only have an ErrorResponseWriter in the command API, and it would be quite a bit of refactoring to return a rejection here 🤷

So acceptance criteria are:

• Add a new PARTITION_UNAVAILABLE error code (as in, to the SBE generated ErrorCode enum), which is documented as meaning that the command cannot be processed because the processor is temporarily unavailable.
• Update the CommandApiRequestHandler to return this error, instead of the current INTERNAL_ERROR.
• Map PARTITION_UNAVAILABLE error code in GrpcErrorMapper such that the error is logged as debug, and the mapped gRPC error is UNAVAILABLE.
• Map PARTITION_UNAVAILABLE error code in RestErrorMapper such that the error is logged as debug, and the mapped HTTP code is 503 (SERVICE_UNAVAILABLE).

As far as tests go, you should write a (parameterized) integration/QA regression test which verifies the above behavior (e.g. create node, pause processing, send request, make sure you get appropriate code), and some unit tests for the mappers. Unit tests alone are likely not enough here as we want to ensure that pausing the processing causes such errors to return. Use the RegressionTest annotation :)

[ana-vinogradova-camunda]

Happened again here
Please feel free to let me know if you think it is a different issue.