Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

local endpoints are gone after network outage #13

Closed
tworec opened this issue Mar 15, 2021 · 6 comments · Fixed by #21
Closed

local endpoints are gone after network outage #13

tworec opened this issue Mar 15, 2021 · 6 comments · Fixed by #21

Comments

@tworec
Copy link
Contributor

tworec commented Mar 15, 2021

My local computer (iMac) fell asleep after my inactivity.
Net was disconnected (probably due to energy saving policy).
After awakening, I have such a piece in my log

image

seems like central net was reconnected

$ yagna app-key list
Error: Bad request: No service registered under given address '/local/appkey/List'.
$ yagna payment accounts
Error: Bad request: No service registered under given address '/local/payment/GetAccounts'.

but in yagna service logs I can see:

[2021-03-15T12:53:13Z ERROR r2d2] database is locked
[2021-03-15T12:53:17Z DEBUG ya_sb_router] Accepted connection from unbound-socket-2a3f290a-e00e-4a36-8856-7119cd3943ad
[2021-03-15T12:53:17Z DEBUG ya_sb_router] Received CallRequest from unbound-socket-2a3f290a-e00e-4a36-8856-7119cd3943ad. caller = local, address = /local/appkey/List, request_id = 8660597915919397
[2021-03-15T12:53:17Z DEBUG ya_sb_router] No service registered under given address '/local/appkey/List'.
[2021-03-15T12:53:17Z DEBUG ya_sb_router] Closing connection with unbound-socket-2a3f290a-e00e-4a36-8856-7119cd3943ad
[2021-03-15T12:53:17Z DEBUG ya_sb_router] Removed instance for: unbound-socket-2a3f290a-e00e-4a36-8856-7119cd3943ad
[2021-03-15T12:53:17Z DEBUG ya_sb_router] Connection with unbound-socket-2a3f290a-e00e-4a36-8856-7119cd3943ad closed.


[2021-03-15T12:53:20Z DEBUG ya_market::protocol::discovery] Propagating 1/1 Offers received from [0xb6e1da94d54c4f8f65a17be40e122d44bac3e637].
[2021-03-15T12:53:26Z DEBUG ya_sb_router] Connection with unbound-socket-baa27996-b7f6-4e91-98bb-2265c65ae8b0 closed.
[2021-03-15T12:53:20Z DEBUG ya_market::matcher::resolver] Resolving incoming Offer [bac5aa8f2b5f4af19e40e9daa0799a9a-1aeb0d25950a75bf92cd11638efe273634e22e27cd8718c9b008ec829e583339]


[2021-03-15T12:53:26Z DEBUG ya_sb_router] Accepted connection from unbound-socket-baa27996-b7f6-4e91-98bb-2265c65ae8b0
[2021-03-15T12:53:26Z DEBUG ya_sb_router] Received CallRequest from unbound-socket-baa27996-b7f6-4e91-98bb-2265c65ae8b0. caller = local, address = /local/payment/GetAccounts, request_id = 6840613581803634
[2021-03-15T12:53:26Z DEBUG ya_sb_router] No service registered under given address '/local/payment/GetAccounts'.
[2021-03-15T12:53:26Z DEBUG ya_sb_router] Closing connection with unbound-socket-baa27996-b7f6-4e91-98bb-2265c65ae8b0
[2021-03-15T12:53:26Z DEBUG ya_sb_router] Removed instance for: unbound-socket-baa27996-b7f6-4e91-98bb-2265c65ae8b0
@etam
Copy link
Contributor

etam commented Mar 15, 2021 

[... not doing much ...]
[2021-03-14T22:39:54Z WARN  ya_provider::market::provider_market] Can't query agreement events. Error: AWC error requesting GET http://0.0.0.0:17596/market-api/v1/agreementEvents?timeout=20&afterTimestamp=2021-03-13T19%3A56%3A17.973660305%2B00%3A00&maxEvents=15&appSessionId=collossus.b1-1422785: Failed to connect to host: Internal error: connector has been disconnected
[2021-03-14T22:39:55Z ERROR ya_provider::payments::payments] Can't query invoice events: AWC error requesting GET http://0.0.0.0:17596/payment-api/v1/invoiceEvents?timeout=50&afterTimestamp=2021-03-13T12%3A21%3A24.424Z&appSessionId=collossus.b1-1422785: Failed to connect to host: Internal error: connector has been disconnected
[2021-03-14T22:39:55Z ERROR ya_provider::payments::payments] Can't query debit note events: AWC error requesting GET http://0.0.0.0:17596/payment-api/v1/debitNoteEvents?timeout=50&afterTimestamp=2021-03-13T19%3A54%3A32.070Z&appSessionId=collossus.b1-1422785: Failed to connect to host: Internal error: connector has been disconnected
[2021-03-14T22:39:55Z ERROR ya_provider::execution::task_runner] Can't query activity events: SendRequestError { msg: "Failed to connect to host: Internal error: connector has been disconnected", method: GET, url: "http://0.0.0.0:17596/activity-api/v1/events?afterTimestamp=2021-03-13T19%3A54%3A47.040074105%2B00%3A00&appSessionId=collossus.b1-1422785&timeout=3" }
[2021-03-14T22:39:55Z ERROR ya_provider::execution::task_runner] Can't query activity events: SendRequestError { msg: "Failed to connect to host: Connection refused (os error 111)", method: GET, url: "http://0.0.0.0:17596/activity-api/v1/events?afterTimestamp=2021-03-13T19%3A54%3A47.040074105%2B00%3A00&appSessionId=collossus.b1-1422785&timeout=3" }
[2021-03-14T22:39:59Z ERROR ya_provider::execution::task_runner] Can't query activity events: SendRequestError { msg: "Failed to connect to host: Connection refused (os error 111)", method: GET, url: "http://0.0.0.0:17596/activity-api/v1/events?afterTimestamp=2021-03-13T19%3A54%3A47.040074105%2B00%3A00&appSessionId=collossus.b1-1422785&timeout=3" }
[2021-03-14T22:40:00Z ERROR ya_provider::payments::payments] Can't query invoice events: AWC error requesting GET http://0.0.0.0:17596/payment-api/v1/invoiceEvents?timeout=50&afterTimestamp=2021-03-13T12%3A21%3A24.424Z&appSessionId=collossus.b1-1422785: Failed to connect to host: Connection refused (os error 111)
[2021-03-14T22:40:01Z ERROR ya_provider::payments::payments] Can't query debit note events: AWC error requesting GET http://0.0.0.0:17596/payment-api/v1/debitNoteEvents?timeout=50&afterTimestamp=2021-03-13T19%3A54%3A32.070Z&appSessionId=collossus.b1-1422785: Failed to connect to host: Connection refused (os error 111)
[2021-03-14T22:40:03Z ERROR ya_provider::execution::task_runner] Can't query activity events: SendRequestError { msg: "Failed to connect to host: Connection refused (os error 111)", method: GET, url: "http://0.0.0.0:17596/activity-api/v1/events?afterTimestamp=2021-03-13T19%3A54%3A47.040074105%2B00%3A00&appSessionId=collossus.b1-1422785&timeout=3" }
[2021-03-14T22:40:06Z ERROR ya_provider::payments::payments] Can't query invoice events: AWC error requesting GET http://0.0.0.0:17596/payment-api/v1/invoiceEvents?timeout=50&afterTimestamp=2021-03-13T12%3A21%3A24.424Z&appSessionId=collossus.b1-1422785: Failed to connect to host: Connection refused (os error 111)
[2021-03-14T22:40:06Z ERROR ya_provider::payments::payments] Can't query debit note events: AWC error requesting GET http://0.0.0.0:17596/payment-api/v1/debitNoteEvents?timeout=50&afterTimestamp=2021-03-13T19%3A54%3A32.070Z&appSessionId=collossus.b1-1422785: Failed to connect to host: Connection refused (os error 111)
[2021-03-14T22:40:07Z ERROR ya_provider::execution::task_runner] Can't query activity events: SendRequestError { msg: "Failed to connect to host: Connection refused (os error 111)", method: GET, url: "http://0.0.0.0:17596/activity-api/v1/events?afterTimestamp=2021-03-13T19%3A54%3A47.040074105%2B00%3A00&appSessionId=collossus.b1-1422785&timeout=3" }
[2021-03-14T22:40:07Z ERROR golemsp::service] child yagna exited too early: Ok(ExitStatus(ExitStatus(9)))
[2021-03-14T22:40:38Z WARN  golemsp::service] service exited with: Custom { kind: Other, error: "process exited too early" }
[... process stopped ...]

@mfranciszkiewicz
Copy link
Contributor

To temporarily fix this issue, specify a tcp endpoint for local gsb connection

@etam
Copy link
Contributor

etam commented Mar 16, 2021

Actually my log is from a provider that uses tcp. So I'm not sure if it's the same bug, or something close, but unrelated.

@tworec
Copy link
Contributor Author

tworec commented Mar 16, 2021

Seems related to golemfactory/yagna#1148

This was referenced Mar 16, 2021
Closed
Closed
@mfranciszkiewicz
Copy link
Contributor

Seems related to golemfactory/yagna#1148

Please lets try to narrow the scope of the issue. So far it's limited to entering the sleep state and the following log entries:

  • Connection with unbound-socket-X closed
  • No service registered under given address '/local/Y'

Unless there are signs of disconnection (either with tcp or unix sockets) as above, please create a new issue.

@nieznanysprawiciel
Copy link
Contributor

Related to:
golemfactory/yagna#1172

This was referenced Mar 22, 2021
Closed
Closed
@mfranciszkiewicz mfranciszkiewicz mentioned this issue Apr 12, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Reconnect to local router golemfactory/ya-service-bus
4 participants
@etam @mfranciszkiewicz @nieznanysprawiciel @tworec