Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log-based events #311

Merged
merged 5 commits into from
Feb 14, 2024
Merged

Log-based events #311

merged 5 commits into from
Feb 14, 2024

Conversation

jmendesky
Copy link
Collaborator

@jmendesky jmendesky commented Jan 26, 2024

Closes #283

Replaces the existing setup for VAI run completion events with log-based events directly from Vertex AI/PubSub. This simplifies the architecture as the eventsource server and reduces the amount of API calls as it no longer needs to poll the Vertex AI API for state changes. Instead, events will be emitted b u Vertex AI on pipelne run completion and the eventsource server just needs to query once and populate run events. For resiliency, the eventsource server still NACKs messages in case of errors.

Tasks

  • Change event source server for VAI provider to subscribe to log-based events rather than polling for updates
  • Documentation updated

@jmendesky jmendesky force-pushed the 283-log-based-events branch 5 times, most recently from 757a80f to 6f611a9 Compare January 26, 2024 18:27
@grahamia grahamia force-pushed the 283-log-based-events branch 8 times, most recently from 9e6ff39 to d2ef78a Compare February 1, 2024 19:50
@jmendesky jmendesky force-pushed the 283-log-based-events branch 3 times, most recently from 811e3df to 8f5c333 Compare February 1, 2024 21:30
@grahamia grahamia force-pushed the 283-log-based-events branch 5 times, most recently from 83b6fc4 to 0871140 Compare February 9, 2024 12:23
@sky-uk sky-uk deleted a comment from jmendesky Feb 9, 2024
@jmendesky jmendesky changed the title 283 log based events Log-based events Feb 12, 2024
jmendesky and others added 3 commits February 12, 2024 16:54
Co-authored-by: Ian Graham <ian.graham@sky.uk>
… runconfiguration for provider sa

Co-authored-by: Jonas Mende <jonas.mende@sky.uk>
Co-authored-by: Jonas Mende <jonas.mende@sky.uk>
Co-authored-by: Jonas Mende <jonas.mende@sky.uk>
@grahamia grahamia force-pushed the 283-log-based-events branch 2 times, most recently from ae5888d to 7b2d02a Compare February 13, 2024 13:16
Co-authored-by: Jonas Mende <jonas.mende@sky.uk>
Copy link
Contributor

@grahamia grahamia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit: 🕐

@jmendesky jmendesky marked this pull request as ready for review February 14, 2024 16:51
@jmendesky jmendesky merged commit 1b6feb7 into master Feb 14, 2024
1 check passed
@jmendesky jmendesky deleted the 283-log-based-events branch February 14, 2024 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

VAI Log-based eventing
2 participants