Routing for agents behind a aca-py based mediator is broken

[codespree]

• Run a mediator using aca-py version 0.10.3
• Connect to mediator from https://github.com/hyperledger/aries-framework-kotlin
• Run a cloud agent accessible through a public URL
• Add invite in mobile agent to the above cloud agent (connection succeeds)
• Send a basic message using the aca-py cloud agent API

Routing fails, see:
openwallet-foundation/didcomm-mediator-service#79
Indicio-tech/pydid#76

[codespree]

In my mediator config, I set:

enable-undelivered-queue: true

When the mobile agent is not connected via web sockets and my web agent tries to send a basic message, I see in the logs:

 DEBUG Fetched doc DIDDoc(5P8H9BCrRWQQNXPw9hTteh)
2023-10-08 21:27:22,530 aries_cloudagent.resolver.did_resolver DEBUG Valid resolvers for DID did:sov:5P8H9BCrRWQQNXPw9hTteh: [<aries_cloudagent.resolver.default.legacy_peer.LegacyPeerDIDResolver object at 0x118a54a90>]
2023-10-08 21:27:22,530 aries_cloudagent.resolver.did_resolver DEBUG Resolving DID did:sov:5P8H9BCrRWQQNXPw9hTteh with <aries_cloudagent.resolver.default.legacy_peer.LegacyPeerDIDResolver object at 0x118a54a90>
2023-10-08 21:27:22,531 aries_cloudagent.connections.base_manager DEBUG Resolved DID document: context=['https://w3id.org/did/v1'] id='did:sov:5P8H9BCrRWQQNXPw9hTteh' also_known_as=None controller=None verification_method=[Ed25519VerificationKey2018(id='did:sov:5P8H9BCrRWQQNXPw9hTteh#1', type='Ed25519VerificationKey2018', controller='did:sov:5P8H9BCrRWQQNXPw9hTteh', public_key_hex=None, public_key_base58='3PWQeLD6uJkyQc82do8FkstuaUEPemhTTH8tyKXP8ph1', public_key_pem=None, public_key_multibase=None, blockchain_account_id=None, ethereum_address=None, public_key_jwk=None)] authentication=['did:sov:5P8H9BCrRWQQNXPw9hTteh#1'] assertion_method=None key_agreement=None capability_invocation=None capability_delegation=None service=[DIDCommV1Service(id='did:sov:5P8H9BCrRWQQNXPw9hTteh#didcomm-0', type='did-communication', service_endpoint='didcomm:transport/queue', recipient_keys=['did:sov:5P8H9BCrRWQQNXPw9hTteh#1'], routing_keys=[], accept=None, priority=0)]
2023-10-08 21:27:22,531 aries_cloudagent.connections.base_manager DEBUG Resolved DIDComm services: [DIDCommV1Service(id='did:sov:5P8H9BCrRWQQNXPw9hTteh#didcomm-0', type='did-communication', service_endpoint='didcomm:transport/queue', recipient_keys=['did:sov:5P8H9BCrRWQQNXPw9hTteh#1'], routing_keys=[], accept=None, priority=0)]
2023-10-08 21:27:22,531 aries_cloudagent.connections.base_manager DEBUG Caching connection targets
2023-10-08 21:27:22,532 aries_cloudagent.messaging.base_handler INFO Forwarding message to connection: e2c60179-bf5a-4c88-8ee4-340e83794d61
2023-10-08 21:27:22,532 aries_cloudagent.core.conductor WARNING Cannot queue message for delivery, no supported transport
2023-10-08 21:27:22,532 aries_cloudagent.core.event_bus DEBUG Notifying subscribers: <Event topic=acapy::outbound-message::waiting_for_pickup, payload=<OutboundMessage(connection_id='e2c60179-bf5a-4c88-8ee4-340e83794d61', enc_payload=b'{"protected": "eyJlbmMiOiAieGNoYWNoYTIwcG9seTEzMDVfaWV0ZiIsICJ0eXAiOiAiSldNLzEuMCIsICJhbGciOiAiQXV0aGNyeXB0IiwgInJlY2lwaWVudHMiOiBbeyJlbmNyeXB0ZWRfa2V5IjogImZwT19sZEE3ZnFSc2JMaE44T05hSWtHYmtPTjFzcXFBWkxGYllDQ0t4SFZqWVdUenRCNVROVDRhdFNoRTdiTDgiLCAiaGVhZGVyIjogeyJraWQiOiAiRVR6RTJZZkFlMzFDTUdDd0FUV3g3dHp2c29YdW5tbXVHVzFWdXhVVW91UGEiLCAic2VuZGVyIjogIkhhc0lkT0ZoYW1sd25UWXQ3LU9JMEpjeVlCaTRpemo5ZFZtMDV2QzVfQ3h1d29CcGJKNmVubmJUZTFMQk5yUU5QNUFaQ09rNURUMVlRN2JMSjYtcjdyNEVHZU40a2swOXhKRHdNS2xfQUY0Rk1jdTV0dTBUbWdmdEFzbyIsICJpdiI6ICJyQlpyWDFHd01rTlFvNEd6RWRCR0E5SEtMZmJoM2FYSiJ9fV19", "iv": "B5USrSAP3hWL0DTc", "ciphertext": "jhzfb3mj4b60FIr6O0DSvKG1brRfF7tIca5l18zwhHleMhN_oPxk1grlBty4cIE5wyKnf7VAno0kORx-YE1_rMnbxSoFsBMHH8t6S_rO3yrk6vWq9cxK-yCoyqWV8a0-Sd5p-rlQ58mPeR2nr-8KTaja7sp6gRULQMbhW8b8XrMhMfomSq2a8h5WU120ZR9cy0YlyTU1rcHsNaOq0Fs_nUlaJVz7ou59Vdeg8YGGOo4", "tag": "19GN2PB-JkeNW35C03nSrg"}', _endpoint=None, payload=None, reply_session_id=None, reply_thread_id=None, reply_to_verkey='3PWQeLD6uJkyQc82do8FkstuaUEPemhTTH8tyKXP8ph1', reply_from_verkey='EXVvtzkzUjzYxmjHLCrcXUZtefJyPhCoYPNzhufArscT', target=None, target_list=[<ConnectionTarget(did='did:sov:5P8H9BCrRWQQNXPw9hTteh', endpoint='didcomm:transport/queue', label='SenseCrypt Wallet', recipient_keys=['3PWQeLD6uJkyQc82do8FkstuaUEPemhTTH8tyKXP8ph1'], routing_keys=[], sender_key='AjCy6rrqjEHgtuuW8evS6rxdRAsP8pFFyMTEoR2sKh3H')>], to_session_only=False)>>
2023-10-08 21:27:22,532 aries_cloudagent.core.event_bus DEBUG Notifying subscribers: <Event topic=acapy::forward::received, payload={'connection_id': 'e2c60179-bf5a-4c88-8ee4-340e83794d61', 'status': 'waiting_for_pickup', 'recipient_key': 'ETzE2YfAe31CMGCwATWx7tzvsoXunmmuGW1VuxUUouPa'}>

Later, when my mobile agent connects over ws and sends a Trust Ping message with the transport decorator all, I get a Trust Ping response back.

What is the implicit pickup mechanism for an agent to pick these queued messages up from the mediator?

@dbluhm @swcurran

[dbluhm]

The trust ping message must also set response_requested to false

[swcurran]

One other note for those that come across this. If you are upgrading a mediator from pre-0.9.0 and use a public DID to establish the connection, a new startup parameter has been added --requests-through-public-did. See the Changelog “breaking changes” notes for 0.9.0.

Likely not the change in this case, but left for others. We’ve seen this one before.

[codespree]

@dbluhm With the trust ping message setting response_requested to false on the mobile agent, messages are not getting delivered to my mobile agent from my web agent (through a mediator) even while the socket (mobile to mediator) is connected.

With response_requested set to True, the messages to my mobile agent are delivered while my mobile agent is connected to the mediator. If it is not connected, the message is supposedly queued, but they are not delivered later (when my mobile agent reconnects).

I will investigate this further and edit this comment tomorrow.

[codespree]

@dbluhm @swcurran the bug was on the mobile agent. It was not setting the transport decorator when response was not requested. I will file a separate issue for the mobile agent in:
https://github.com/hyperledger/aries-framework-kotlin

Everything is working as expected.

[codespree]

@dbluhm @swcurran, please see the following PR that fixes things on the mobile agent side.

hyperledger/aries-framework-kotlin#10