Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🩹 Remove _backfill_events method from webhooks service startup #961

Merged
merged 2 commits into from
Aug 8, 2024

Conversation

ff137
Copy link
Collaborator

@ff137 ff137 commented Aug 7, 2024

Backfilling SSE queues with recent events is currently highly inefficient when many wallets exist. Fortunately this is a non-critical feature, and so can just be removed for now. A solution will be implemented as detailed in #899

Backfilling SSE queues with recent events is currently highly inefficient when many wallets exist. Fortunately this is a non-critical feature, and so can just be removed for now. A solution will be implemented as detailed in #899
@ff137 ff137 self-assigned this Aug 7, 2024
@ff137 ff137 added the improvement improving on something that is already existing (not a new feature as such) label Aug 7, 2024
@ff137 ff137 requested a review from wdbasson August 7, 2024 17:26
@ff137 ff137 changed the title 🩹 Remove _backfill_events method from startup 🩹 Remove _backfill_events method from webhooks service startup Aug 7, 2024
Copy link

github-actions bot commented Aug 7, 2024

Coverage

K8s Test Coverage
FileStmtsMissCoverMissing
app/routes
   jsonld.py604722%23–93, 104–137
   messaging.py241154%43–52, 85–95
   oob.py462839%58–99, 131–144, 170–181
   trust_registry.py62297%52–53
   verifier.py1483378%95–97, 102, 162–164, 169, 245, 253–255, 260, 304–309, 315–317, 383–385, 391, 430–432, 437, 473–475, 517–519
   webhooks.py16475%32–36, 57–61
   websocket_endpoint.py23865%24–25, 37–42, 54–57, 70–76
app/routes/wallet
   credentials.py775232%32–45, 54–66, 75–86, 95–109, 120–136, 148–162, 171–183, 192–203
   jws.py422736%35–60, 76–99
   sd_jws.py422736%36–61, 78–101
app/services
   acapy_ledger.py611280%110–111, 124–140, 196–197
   acapy_wallet.py40490%60–61, 98–99
   revocation_registry.py1532882%117–142, 185–189, 356–360, 399–404, 437–441, 469–474
   webhooks.py271833%17–25, 29–39
   websocket.py514120%19–22, 29–36, 47–91
app/services/event_handling
   websocket_manager.py732664%64–68, 87–90, 97–108, 115–125, 139
app/services/issuer
   acapy_issuer_v1.py871484%36, 76–110, 201–202
   acapy_issuer_v2.py1011288%63, 97–106, 110, 126–127, 173, 223–224, 251
app/services/onboarding
   tenants.py594524%30–98, 108–138
   verifier.py37392%67, 83–87
app/services/onboarding/util
   register_issuer_did.py86397%100–102
   set_endorser_metadata.py724044%28–30, 50–52, 76–78, 109–128, 140–151, 163–176, 189–203
app/services/trust_registry
   actors.py107992%95–100, 110, 209–210, 212–217, 282–285
   schemas.py54983%57–63, 89–99
app/services/trust_registry/util
   actor.py31294%26–27
   issuer.py23483%39–40, 48–49
app/tests/e2e
   conftest.py23578%46–54
   test_connections.py129298%277–278
   test_jsonld.py591673%134–169
   test_proof_request_models.py28196%127
   test_tenants.py4031596%203, 247–248, 287, 788, 832–833, 842–843, 852–853, 865–866, 875–876
   test_wallet_credentials.py30197%55
   test_wallet_dids.py691086%93–108
app/tests/e2e/issuer
   test_get_records_paginated.py41295%60–61
   test_save_exchange_record.py72199%178
app/tests/e2e/verifier
   test_get_records_paginated.py41295%58–59
   test_proof_revoked_credential.py451273%130–203
   test_verifier.py2321295%674–745
app/tests/exception_handling
   test_handle_model_with_validation.py23196%22
app/tests/fixtures
   credentials.py1193769%176, 270–351, 364–436
   definitions.py662661%29–50, 64–65, 94–124, 143–144
   member_acapy_clients.py36489%47–50, 73–74
   member_connections.py43881%80–91, 130–141
   member_wallets.py611575%29–34, 49–54, 69–74, 89–94, 111–118
app/tests/routes/issuer
   test_create_offer.py39197%150
   test_send_credential.py36197%129
app/tests/routes/wallet/dids
   test_get_public_did.py29197%40
app/tests/services
   test_revocation_registry.py135299%75, 106
app/tests/services/issuer
   test_issuer.py160299%347–351
app/tests/util
   connections.py852768%104–129, 138–155, 172–181, 236–253
   ledger.py501080%35, 43, 55, 63–67, 77, 83
   regression_testing.py351751%33, 41–68, 74–81
   sse_listener.py41880%50–55, 79–84
   trust_registry.py29390%24, 27–28
   webhooks.py54591%19, 74–77, 100, 124
app/util
   acapy_issuer_utils.py18194%22
   acapy_verifier_utils.py137596%108–111, 191–195, 272
   assert_public_did.py18856%17–33
   credentials.py13285%11, 18
   retry_method.py361558%20–35, 76–77, 91–103
   tenants.py451762%24–29, 44–45, 52–59, 90–91, 97, 124
   transaction_acked.py14286%31–32
endorser/services
   endorsement_processor.py137696%90, 152–162
webhooks/models
   conversions.py63690%70, 83, 91, 112, 149, 168
webhooks/services
   acapy_events_processor.py1862288%98, 161–171, 219–223, 251, 279–280, 308–312, 325–329, 354–367
   billing_manager.py175498%91, 98–99, 132
   sse_manager.py2343685%155, 157–170, 208–218, 251–252, 265–279, 334–338, 344–347, 349–352, 377–378, 421–423, 457, 468–469
   webhooks_redis_service.py1401986%70, 237–238, 354, 378–398, 411–422
webhooks/tests/services
   test_sse_manager.py179398%214–215, 221
webhooks/web/routers
   sse.py102397%168–171
TOTAL1481190594% 

Tests Skipped Failures Errors Time
1069 4 💤 0 ❌ 0 🔥 12m 9s ⏱️

Copy link

github-actions bot commented Aug 7, 2024

Coverage

K8s Regression Test Coverage
FileStmtsMissCoverMissing
app/routes
   jsonld.py604722%23–93, 104–137
   messaging.py241154%43–52, 85–95
   oob.py462839%58–99, 131–144, 170–181
   trust_registry.py62297%52–53
   verifier.py1483378%95–97, 102, 162–164, 169, 245, 253–255, 260, 304–309, 315–317, 383–385, 391, 430–432, 437, 473–475, 517–519
   webhooks.py16475%32–36, 57–61
   websocket_endpoint.py23865%24–25, 37–42, 54–57, 70–76
app/routes/wallet
   credentials.py775232%32–45, 54–66, 75–86, 95–109, 120–136, 148–162, 171–183, 192–203
   jws.py422736%35–60, 76–99
   sd_jws.py422736%36–61, 78–101
app/services
   acapy_ledger.py611969%43–49, 79, 110–111, 124–140, 146–149, 196–197
   acapy_wallet.py401270%49–64, 98–99
   revocation_registry.py1533975%117–142, 185–189, 356–360, 399–404, 437–441, 459–474, 480–489
   webhooks.py271833%17–25, 29–39
   websocket.py514120%19–22, 29–36, 47–91
app/services/event_handling
   websocket_manager.py732664%64–68, 87–90, 97–108, 115–125, 139
app/services/issuer
   acapy_issuer_v1.py871484%36, 76–110, 201–202
   acapy_issuer_v2.py1011288%63, 97–106, 110, 126–127, 173, 223–224, 251
app/services/onboarding
   tenants.py594524%30–98, 108–138
   verifier.py37392%67, 83–87
app/services/onboarding/util
   register_issuer_did.py86397%100–102
   set_endorser_metadata.py724044%28–30, 50–52, 76–78, 109–128, 140–151, 163–176, 189–203
app/services/trust_registry
   actors.py107992%95–100, 110, 209–210, 212–217, 282–285
   schemas.py54983%57–63, 89–99
app/services/trust_registry/util
   actor.py31294%26–27
   issuer.py23483%39–40, 48–49
app/tests/e2e
   conftest.py23578%46–54
   test_connections.py129298%277–278
   test_definitions.py795135%33–56, 68–80, 92–106, 118–146, 164–209
   test_jsonld.py591673%134–169
   test_proof_request_models.py28196%127
   test_revocation.py936134%27–76, 89–106, 117–143, 155–180, 192–206, 218–255, 266–292
   test_tenants.py40334614%29–73, 84–117, 128–172, 185–254, 265–316, 329–458, 467–519, 528–564, 573–622, 631–680, 691–733, 745–794, 805–876, 885–954
   test_wallet_dids.py691578%93–108, 120–131
app/tests/e2e/issuer
   test_get_records_paginated.py413027%24–115
   test_save_exchange_record.py72199%178
app/tests/e2e/verifier
   test_get_records_paginated.py412832%26–111
   test_proof_revoked_credential.py451176%39–116
app/tests/exception_handling
   test_handle_model_with_validation.py23196%22
app/tests/fixtures
   credentials.py1195554%132–207, 216–224, 234–255, 289–349, 384–434
   definitions.py661085%41–48, 57–63, 114–121, 134–139
   member_acapy_clients.py36489%47–50, 73–74
   member_async_clients.py50296%32–33
   member_connections.py43491%73, 91, 123, 141
   member_wallets.py611575%23–27, 43–47, 63–67, 83–87, 103–109
app/tests/routes/issuer
   test_create_offer.py39197%150
   test_send_credential.py36197%129
app/tests/routes/wallet/dids
   test_get_public_did.py29197%40
app/tests/services
   test_revocation_registry.py135299%75, 106
app/tests/services/issuer
   test_issuer.py160299%347–351
app/tests/util
   connections.py851780%154–155, 169, 181, 191–223, 252–253
   ledger.py502648%34–59, 63–67, 73–86
   regression_testing.py35974%33, 60–68, 74–81
   sse_listener.py41880%50–55, 79–84
   tenants.py241250%10, 16–17, 21–26, 30–35, 39–44, 48–53, 57
   trust_registry.py29776%20–28
   webhooks.py54591%19, 74–77, 100, 124
app/util
   acapy_issuer_utils.py18194%22
   acapy_verifier_utils.py137596%108–111, 191–195, 272
   assert_public_did.py181233%14–34
   check_endorser_connection.py10460%14–24
   credentials.py13285%11, 18
   retry_method.py363017%15–36, 68–114
   string.py15193%24
   tenants.py451762%24–29, 44–45, 52–59, 90–91, 97, 124
   transaction_acked.py14750%18–36
endorser/services
   endorsement_processor.py137696%90, 152–162
webhooks/models
   conversions.py63690%70, 83, 91, 112, 149, 168
webhooks/services
   acapy_events_processor.py1862288%98, 161–171, 219–223, 251, 279–280, 308–312, 325–329, 354–367
   billing_manager.py175498%91, 98–99, 132
   sse_manager.py2343685%155, 157–170, 208–218, 251–252, 265–279, 334–338, 344–347, 349–352, 377–378, 421–423, 457, 468–469
   webhooks_redis_service.py1401986%70, 237–238, 354, 378–398, 411–422
webhooks/tests/services
   test_sse_manager.py179398%214–215, 221
webhooks/web/routers
   sse.py102397%168–171
TOTAL14811146290% 

Tests Skipped Failures Errors Time
1069 36 💤 0 ❌ 0 🔥 4m 28s ⏱️

Copy link

sonarcloud bot commented Aug 8, 2024

Copy link

github-actions bot commented Aug 8, 2024

Coverage

K8s Test Coverage
FileStmtsMissCoverMissing
app/routes
   jsonld.py604722%23–93, 104–137
   messaging.py241154%43–52, 85–95
   oob.py462839%58–99, 131–144, 170–181
   trust_registry.py62297%52–53
   verifier.py1483378%95–97, 102, 162–164, 169, 245, 253–255, 260, 304–309, 315–317, 383–385, 391, 430–432, 437, 473–475, 517–519
   webhooks.py16475%32–36, 57–61
   websocket_endpoint.py23865%24–25, 37–42, 54–57, 70–76
app/routes/wallet
   credentials.py775232%32–45, 54–66, 75–86, 95–109, 120–136, 148–162, 171–183, 192–203
   jws.py422736%35–60, 76–99
   sd_jws.py422736%36–61, 78–101
app/services
   acapy_ledger.py611280%110–111, 124–140, 196–197
   acapy_wallet.py40490%60–61, 98–99
   revocation_registry.py1532882%117–142, 185–189, 356–360, 399–404, 437–441, 469–474
   webhooks.py271833%17–25, 29–39
   websocket.py514120%19–22, 29–36, 47–91
app/services/event_handling
   websocket_manager.py732664%64–68, 87–90, 97–108, 115–125, 139
app/services/issuer
   acapy_issuer_v1.py871484%36, 76–110, 201–202
   acapy_issuer_v2.py1011288%63, 97–106, 110, 126–127, 173, 223–224, 251
app/services/onboarding
   tenants.py594524%30–98, 108–138
   verifier.py37392%67, 83–87
app/services/onboarding/util
   register_issuer_did.py86397%100–102
   set_endorser_metadata.py724044%28–30, 50–52, 76–78, 109–128, 140–151, 163–176, 189–203
app/services/trust_registry
   actors.py107992%95–100, 110, 209–210, 212–217, 282–285
   schemas.py54983%57–63, 89–99
app/services/trust_registry/util
   actor.py31294%26–27
   issuer.py23483%39–40, 48–49
app/tests/e2e
   conftest.py23578%46–54
   test_connections.py129298%277–278
   test_jsonld.py591673%134–169
   test_proof_request_models.py28196%127
   test_tenants.py4031596%203, 247–248, 287, 788, 832–833, 842–843, 852–853, 865–866, 875–876
   test_wallet_credentials.py30197%55
   test_wallet_dids.py691086%93–108
app/tests/e2e/issuer
   test_get_records_paginated.py41295%60–61
   test_save_exchange_record.py72199%178
app/tests/e2e/verifier
   test_get_records_paginated.py41295%58–59
   test_proof_revoked_credential.py451273%130–203
   test_verifier.py2321295%674–745
app/tests/exception_handling
   test_handle_model_with_validation.py23196%22
app/tests/fixtures
   credentials.py1193769%176, 270–351, 364–436
   definitions.py662661%29–50, 64–65, 94–124, 143–144
   member_acapy_clients.py36489%47–50, 73–74
   member_connections.py43881%80–91, 130–141
   member_wallets.py611575%29–34, 49–54, 69–74, 89–94, 111–118
app/tests/routes/issuer
   test_create_offer.py39197%150
   test_send_credential.py36197%129
app/tests/routes/wallet/dids
   test_get_public_did.py29197%40
app/tests/services
   test_revocation_registry.py135299%75, 106
app/tests/services/issuer
   test_issuer.py160299%347–351
app/tests/util
   connections.py852768%104–129, 138–155, 172–181, 236–253
   ledger.py501080%35, 43, 55, 63–67, 77, 83
   regression_testing.py351751%33, 41–68, 74–81
   sse_listener.py41880%50–55, 79–84
   trust_registry.py29390%24, 27–28
   webhooks.py54591%19, 74–77, 100, 124
app/util
   acapy_issuer_utils.py18194%22
   acapy_verifier_utils.py137596%108–111, 191–195, 272
   assert_public_did.py18856%17–33
   credentials.py13285%11, 18
   retry_method.py361558%20–35, 76–77, 91–103
   tenants.py451762%24–29, 44–45, 52–59, 90–91, 97, 124
   transaction_acked.py14286%31–32
endorser/services
   endorsement_processor.py137696%90, 152–162
webhooks/models
   conversions.py63690%70, 83, 91, 112, 149, 168
webhooks/services
   acapy_events_processor.py1862288%98, 161–171, 219–223, 251, 279–280, 308–312, 325–329, 354–367
   billing_manager.py175498%91, 98–99, 132
   sse_manager.py2343685%155, 157–170, 208–218, 251–252, 265–279, 334–338, 344–347, 349–352, 377–378, 421–423, 457, 468–469
   webhooks_redis_service.py1401986%70, 237–238, 354, 378–398, 411–422
webhooks/tests/services
   test_sse_manager.py179398%214–215, 221
webhooks/web/routers
   sse.py102397%168–171
TOTAL1484090594% 

Tests Skipped Failures Errors Time
1070 4 💤 0 ❌ 0 🔥 12m 22s ⏱️

Copy link

github-actions bot commented Aug 8, 2024

Coverage

K8s Regression Test Coverage
FileStmtsMissCoverMissing
app/routes
   jsonld.py604722%23–93, 104–137
   messaging.py241154%43–52, 85–95
   oob.py462839%58–99, 131–144, 170–181
   trust_registry.py62297%52–53
   verifier.py1483378%95–97, 102, 162–164, 169, 245, 253–255, 260, 304–309, 315–317, 383–385, 391, 430–432, 437, 473–475, 517–519
   webhooks.py16475%32–36, 57–61
   websocket_endpoint.py23865%24–25, 37–42, 54–57, 70–76
app/routes/wallet
   credentials.py775232%32–45, 54–66, 75–86, 95–109, 120–136, 148–162, 171–183, 192–203
   jws.py422736%35–60, 76–99
   sd_jws.py422736%36–61, 78–101
app/services
   acapy_ledger.py611969%43–49, 79, 110–111, 124–140, 146–149, 196–197
   acapy_wallet.py401270%49–64, 98–99
   revocation_registry.py1533975%117–142, 185–189, 356–360, 399–404, 437–441, 459–474, 480–489
   webhooks.py271833%17–25, 29–39
   websocket.py514120%19–22, 29–36, 47–91
app/services/event_handling
   websocket_manager.py732664%64–68, 87–90, 97–108, 115–125, 139
app/services/issuer
   acapy_issuer_v1.py871484%36, 76–110, 201–202
   acapy_issuer_v2.py1011288%63, 97–106, 110, 126–127, 173, 223–224, 251
app/services/onboarding
   tenants.py594524%30–98, 108–138
   verifier.py37392%67, 83–87
app/services/onboarding/util
   register_issuer_did.py86397%100–102
   set_endorser_metadata.py724044%28–30, 50–52, 76–78, 109–128, 140–151, 163–176, 189–203
app/services/trust_registry
   actors.py107992%95–100, 110, 209–210, 212–217, 282–285
   schemas.py54983%57–63, 89–99
app/services/trust_registry/util
   actor.py31294%26–27
   issuer.py23483%39–40, 48–49
app/tests/e2e
   conftest.py23578%46–54
   test_connections.py129298%277–278
   test_definitions.py795135%33–56, 68–80, 92–106, 118–146, 164–209
   test_jsonld.py591673%134–169
   test_proof_request_models.py28196%127
   test_revocation.py936134%27–76, 89–106, 117–143, 155–180, 192–206, 218–255, 266–292
   test_tenants.py40334614%29–73, 84–117, 128–172, 185–254, 265–316, 329–458, 467–519, 528–564, 573–622, 631–680, 691–733, 745–794, 805–876, 885–954
   test_wallet_dids.py691578%93–108, 120–131
app/tests/e2e/issuer
   test_get_records_paginated.py413027%24–115
   test_save_exchange_record.py72199%178
app/tests/e2e/verifier
   test_get_records_paginated.py412832%26–111
   test_proof_revoked_credential.py451176%39–116
app/tests/exception_handling
   test_handle_model_with_validation.py23196%22
app/tests/fixtures
   credentials.py1195554%132–207, 216–224, 234–255, 289–349, 384–434
   definitions.py661085%41–48, 57–63, 114–121, 134–139
   member_acapy_clients.py36489%47–50, 73–74
   member_async_clients.py50296%32–33
   member_connections.py43491%73, 91, 123, 141
   member_wallets.py611575%23–27, 43–47, 63–67, 83–87, 103–109
app/tests/routes/issuer
   test_create_offer.py39197%150
   test_send_credential.py36197%129
app/tests/routes/wallet/dids
   test_get_public_did.py29197%40
app/tests/services
   test_revocation_registry.py135299%75, 106
app/tests/services/issuer
   test_issuer.py160299%347–351
app/tests/util
   connections.py851780%154–155, 169, 181, 191–223, 252–253
   ledger.py502648%34–59, 63–67, 73–86
   regression_testing.py35974%33, 60–68, 74–81
   sse_listener.py41880%50–55, 79–84
   tenants.py241250%10, 16–17, 21–26, 30–35, 39–44, 48–53, 57
   trust_registry.py29776%20–28
   webhooks.py54591%19, 74–77, 100, 124
app/util
   acapy_issuer_utils.py18194%22
   acapy_verifier_utils.py137596%108–111, 191–195, 272
   assert_public_did.py181233%14–34
   check_endorser_connection.py10460%14–24
   credentials.py13285%11, 18
   retry_method.py363017%15–36, 68–114
   string.py15193%24
   tenants.py451762%24–29, 44–45, 52–59, 90–91, 97, 124
   transaction_acked.py14750%18–36
endorser/services
   endorsement_processor.py137696%90, 152–162
webhooks/models
   conversions.py63690%70, 83, 91, 112, 149, 168
webhooks/services
   acapy_events_processor.py1862288%98, 161–171, 219–223, 251, 279–280, 308–312, 325–329, 354–367
   billing_manager.py175498%91, 98–99, 132
   sse_manager.py2343685%155, 157–170, 208–218, 251–252, 265–279, 334–338, 344–347, 349–352, 377–378, 421–423, 457, 468–469
   webhooks_redis_service.py1401986%70, 237–238, 354, 378–398, 411–422
webhooks/tests/services
   test_sse_manager.py179398%214–215, 221
webhooks/web/routers
   sse.py102397%168–171
TOTAL14840146290% 

Tests Skipped Failures Errors Time
1070 36 💤 0 ❌ 0 🔥 4m 38s ⏱️

@ff137
Copy link
Collaborator Author

ff137 commented Aug 8, 2024

Codacy is currently down with an outage for one of their services. Merging as minor changes don't need codacy review

@ff137 ff137 merged commit 7880d6a into development Aug 8, 2024
36 of 37 checks passed
@ff137 ff137 deleted the patch/remove-sse-backfill-logic branch August 8, 2024 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement improving on something that is already existing (not a new feature as such)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant