Add docs best effort responses

[oggy-dfin]

DO NOT MERGE YET - this should be merged only once best-effort responses go live on mainnet.

This PR contains the documentation for the new best-effort response feature. In the process, I noticed that we were lacking a good introductory page on inter-canister calls, addressing topics such as:

• how do I find which endpoints to call?

• what's the role of Candid in inter-canister calls?

• why do I care about the desugaring of async/await

• attaching cycles to calls

• Follows the developer docs style guide.

• Follows the best practices and guidelines.

• New documentation pages include document tags.

• New documentation pages include SEO keywords.

• New documents are in the .mdx file format to support the previous two components.

• New documents are registered in /sidebars.js, otherwise, it will not appear in the
  side navigation bar.

• Make sure that the .github/CODEOWNERS file is
  filled with new documents that you added. This way we can ensure that future Pull Requests are reviewed by the right people.

[github-actions]

🤖 Here's your preview: https://tai3p-vaaaa-aaaam-abema-cai.icp0.io

[alin-at-dfinity]

Only looked at message-execution-properties.mdx for now and added some nitpicks.

However, I can't help thinking that the topic could be explained more clearly. E.g. start with calls/tasks vs. message executions as we already do, but before we list any properties; just a couple of pictures with simplified sequence diagrams. That way, the difference between call/task and message execution would be immediately clear. It would also immediately make it clear why two requests Req1 and Req2 delivered in that order may produce two responses Rep2 and Rep1, in reverse order (even before the protocol may decide to deliver them in an arbitrary order).

Very likely not for this PR, though.

[alin-at-dfinity]

Not part of your change, but apart from mentioning updates and queries here and in the paragraph below, we never clarify anything about either of them.

I haven't followed the link, but by the looks of the URL (involving HTTP endpoints), it refers to "replicated updates" and "non-replicated queries". Which is even more confusing, as further down we say "Each downstream call that a canister makes, query or update, triggers a message." Which may imply that a replicated message execution can somehow make a non-replicated query call.

Maybe it's necessary to explicitly state that this page is all about replicated execution (which I guess it is, but I'm not 100% sure, because there's also composite queries, which behave somewhat similarly to replicated execution). And briefly explain the difference between replicated updates and replicated queries.

[oggy-dfin]

Thanks - I agree with your points.

I think for this document it doesn't matter if we're talking about replicated or non-replicated execution, except for the part where we talk about committing state at the end of a message handler.

[alin-at-dfinity]

My concern is that to the extent that we define updates and queries on this page, updates are replicated and queries are non-replicated. So that later when we say that "a canister makes [a downstream] query call", the uninformed reader would interpret it as "a replicated update call can make a non-replicated query call".

But yeah, it's a nitpick, so feel free to ignore.

[oggy-dfin]

I've removed all the details about HTTP calls and focused on the endpoints - not sure if you're somehow looking at an old version of the PR...

[alin-at-dfinity]

I don't think this is broad enough. I would put it in terms of "for guaranteed response calls there is always exactly one globally unique response to a call (whether from the callee or from the protocol)".

Or even "there is always one single globally unique response for every canister call (originating from either the callee or the protocol); if the call is a guaranteed response call, then this unique response is guaranteed to be delivered to the caller; if it was a best-effort call, then a SYS_UNKNOWN response may be delivered instead (and potentially before)". Or something along those lines.

[oggy-dfin]

Overall, I'm not sure how useful it is to talk about the "real response generated by the system" in case of SYS_UNKNOWN, as this real response is not something observable by the user. What is observable is that the callee produced a response but it didn't get delivered. So we can introduce the internal requests and responses (including system-generated responses) if it's the simplest mental model for the user of what's happening, but if there's a simple way to explain everything using just what's observable, I'd prefer that.

Not entirely related, but my original idea was to have a property that only applies to guaranteed response calls, and we could just say "best-effort responses lack this property". But thinking about it, it's probably useful to have a separate property about losing responses that we can refer to.

[alin-at-dfinity]

I meant that this property (of the caller being guaranteed to learn of the outcome of the call) applies more broadly than just for "reply or reject responses [produced by] the callee-canister". The system itself may also produce a reject response without making the call at all and this is also guaranteed to be delivered to the caller. And, importantly, there is always exactly one outcome for every call. So it's never the case that e.g. the system produces and delivers a reject response to the caller, but then somehow still delivers the request to the callee, which then produces a second response (that the system simply chooses to not deliver).

The difference in the best-effort response case is that the system may at any time, including before the call completes or even begins execution, produce a SYS_UNKNOWN reject response and deliver it to the caller instead of the actual outcome (which is what would always be delivered for a guaranteed response call).

I.e. the former is always a closed loop, with no branches. Whereas the latter is an interrupted loop, where a SYS_UNKNOWN reject response can be delivered at any time, while the call is potentially still ongoing or in the future (meaning that the caller can never find out the outcome in this way).

[oggy-dfin]

The system itself may also produce a reject response without making the call at all and this is also guaranteed to be delivered to the caller. And, importantly, there is always exactly one outcome for every call.

This is covered by Property 7. This property just says that, for guaranteed response calls, if the caller actually produces a response, the user will get that response.

The difference in the best-effort response case is that the system may at any time, including before the call completes or even begins execution, produce a SYS_UNKNOWN reject response

Of course it's also true that for guaranteed response calls, if the system produces some response, only that response will be delivered to the user, which is not true for best-effort responses, and what you write is an entirely accurate description how the system works. What I'm saying is that in general, the user doesn't need to know how the system works. We just have to give them the simplest correct description of the behavior that they can observe. It might well be that the simplest explanation is to tell them how the system really works. I tried providing a simple description without going into the system details but it's quite possible that I failed. I hope my intention is clear now - if it was clear all along and you feel that it's too complicated or not precise enough, I will switch to what you proposed, otherwise take another look and LMK what you think.

[oggy-dfin]

Maybe what's missing is to say something like "the call is only deliver at most once" in Property 5?

[alin-at-dfinity]

I just found it unnecessarily specific. IMHO there is no need to qualify this with "if the call is delivered to the callee and the callee responds without trapping".

But yeah, if you want to describe strictly the observable properties of the system, then this is more accurate. So it's entirely your call.

[derlerd-dfinity]

I left feedback on everything but message-execution-properties.md as Alin already made a pass over this one. I'll make a pass once you've addressed Alin's comments.

[derlerd-dfinity]

Re "true" response from the callee: Strictly speaking, it is also not guaranteed in the current messaging model that the response will come from the callee. The response could also be generated by the system if the request couldn't be delivered to the callee.

The property the current messaging model provides is that the response is globally unique, i.e., if the system produces a reject one can conclude that the callee never saw the request.

[oggy-dfin]

Re "true" response from the callee: Strictly speaking, it is also not guaranteed in the current messaging model that the response will come from the callee. The response could also be generated by the system if the request couldn't be delivered to the callee.

True, I dropped the "from the callee" part.

The property the current messaging model provides is that the response is globally unique, i.e., if the system produces a reject one can conclude that the callee never saw the request.

Well technically a CANISTER_ERROR is also produced by the system but it means that the callee has seen the request (whether any mutation was made on the callee side is anyone's guess, though)

[alin-at-dfinity]

True, I dropped the "from the callee" part.

FWIW, I still see

in this mode the "true" response from the callee is not guaranteed to be delivered

Maybe you just didn't push your change yet.

[oggy-dfin]

I have pushed the change, I see below:

but in this mode the "true" response is not guaranteed to be delivered, and can be masked by a system generated error response instead.

Not sure what's going on 🤷

[derlerd-dfinity]

I think here we should already mention that with best effort messaging the cycles are dropped if the message is dropped. It might be too early to mention this in detail, but maybe one can provide a link to the next section where the difference between best-effort and guaranteed response calls is discussed.

[oggy-dfin]

Good idea, I added an info box that cycles may be dropped and a forward pointer to the section where it's discussed in more detail

[derlerd-dfinity]

Maybe mention here that the system ensures that there is only ever one response.

[alin-at-dfinity]

+1

The other way I think of it (and I've mentioned this in an earlier comment) is that guaranteed response calls are closed loops: the request goes out (potentially all the way to the callee, potentially not very far); one response is produced from it (whether by the callee or by the system); that response is always returned to the caller (whether this takes until the heat death of the universe or just before).

[derlerd-dfinity]

Suggested change

	* For endpoints that change the state of the callee, the best practice is to make such endpoints amenable to [safe retries](/docs/current/developer-docs/smart-contracts/best-practices/idempotency), since this is also required for safely handling external user calls. Use best-effort responses calls for such endpoints, and handle the additional edge cases.
	* For endpoints that change the state of the callee, the best practice is to make such endpoints amenable to [safe retries](/docs/current/developer-docs/smart-contracts/best-practices/idempotency), similar to the one that is required for safely handling external user calls. Use best-effort responses calls for such endpoints, and handle the additional edge cases.

[oggy-dfin]

I reworded this now more thoroughly to make it clear that we recommend doing that for ingress messages anyways.

[derlerd-dfinity]

I would also be very explicit here that dropping a message means dropping the attached cycles as well.

[oggy-dfin]

Added

[derlerd-dfinity]

Here I would not say that best effort calls can lose cycles in edge cases as it makes it sound as if it was a bug in the system. I'd rather use the formulation that if a message is dropped also the attached cycles will be dropped.

[oggy-dfin]

Good idea, changed.

[derlerd-dfinity]

Should we also point to the security best practices for best-effort responses?

[oggy-dfin]

We have a single security best practices doc for all types of inter-canister calls (as most points are shared between both call types), and we already link to it above.

[alin-at-dfinity]

My concern is that to the extent that we define updates and queries on this page, updates are replicated and queries are non-replicated. So that later when we say that "a canister makes [a downstream] query call", the uninformed reader would interpret it as "a replicated update call can make a non-replicated query call".

But yeah, it's a nitpick, so feel free to ignore.

[alin-at-dfinity]

I guess I just find it awkward that we talk about a call or a callback without an await. If there's no await, nothing is called, you've merely created a future even though it's not obvious from the syntax.

IOW, leaving it at "the code after the await is executed as a separate message execution (callback)" (which the paragraph above already says) is IMHO cleaner than adding "but if there's no await, there's no callback". I would rather change this paragraph to say "note that it is actually the await, not the thing that looks like a function call that triggers the canister call and marks the separation point between message executions". Or something to that effect.

[alin-at-dfinity]

I meant that this property (of the caller being guaranteed to learn of the outcome of the call) applies more broadly than just for "reply or reject responses [produced by] the callee-canister". The system itself may also produce a reject response without making the call at all and this is also guaranteed to be delivered to the caller. And, importantly, there is always exactly one outcome for every call. So it's never the case that e.g. the system produces and delivers a reject response to the caller, but then somehow still delivers the request to the callee, which then produces a second response (that the system simply chooses to not deliver).

The difference in the best-effort response case is that the system may at any time, including before the call completes or even begins execution, produce a SYS_UNKNOWN reject response and deliver it to the caller instead of the actual outcome (which is what would always be delivered for a guaranteed response call).

I.e. the former is always a closed loop, with no branches. Whereas the latter is an interrupted loop, where a SYS_UNKNOWN reject response can be delivered at any time, while the call is potentially still ongoing or in the future (meaning that the caller can never find out the outcome in this way).

[alin-at-dfinity]

True, I dropped the "from the callee" part.

FWIW, I still see

in this mode the "true" response from the callee is not guaranteed to be delivered

Maybe you just didn't push your change yet.

[alin-at-dfinity]

+1

The other way I think of it (and I've mentioned this in an earlier comment) is that guaranteed response calls are closed loops: the request goes out (potentially all the way to the callee, potentially not very far); one response is produced from it (whether by the callee or by the system); that response is always returned to the caller (whether this takes until the heat death of the universe or just before).

[andrew-lee-work]

LGTM, left some comments.