error tracing + backtrace

[jondot]

when an error occurs and we're doing the handling bit in controller,
we need to manually capture backtrace (only in development), and add it to the trace.

[jondot]

some of the research:

1. we convert a result into a response, handling errors in src/controllers/mod.rs. in there we manually report a warn trace
2. there are some recommendations that if you want to see errors from a handler, you need to instrument(err) it
3. there's a possibility that error reporting needs to be done via the logging middleware by adding .on_failure

we need to survey various other services/products and see how they report errors that may occur in handlers to the operators (this deals only with the developer and not user of the service)

[jondot]

mas is capturing with sentry https://github.com/matrix-org/matrix-authentication-service/blob/main/crates/axum-utils/src/fancy_error.rs#L65

[jondot]

done

[jondot]

for reference, this is how it looks like now: