Push flow: how to keep track of data pushed ?

[balasenthil-d]

I'm evaluating OPA for our use and would like to check how I can use push workflow to keep track of documents pushed and push missing documents. I see there are no count APIs for documents or API that report a new OPA service has started (Lifecycle events).

I did try bundle service and its quite straight forward as it always pulls all data and rego policy and update OPA runtime.

[patrick-east]

Using the OPA server as-is you would likely have to query for the documents from/v1/data to get the current state and then reconcile the differences. At best you could have an eventually consistent state.

For lifecycle events that largely depends on the system you are deploying OPA on. For example on kubernetes you would have this type of information available via the kubernetes API's.

[balasenthil-d]

Reconciliation could be difficult to achieve, if we deploy this across different services (as in side cars) and each service will need to synchronize its copy. Run states of all services need to be remotely maintained and handled. I think having life cycle event support in OPA will be helpful - like an API call on post-startup and pre-shutdown will be very helpful in handling these remote states.

[patrick-east]

Reconciliation could be difficult to achieve, if we deploy this across different services (as in side cars) and each service will need to synchronize its copy. Run states of all services need to be remotely maintained and handled.

Right, and that is why bundles are typically the approach people use. Was there a reason that you didn't want to use bundles?

Making a guess based on your comment:

it always pulls all data and rego policy and update OPA runtime.

There is a feature proposed to allow for incremental bundle updates #1055 so the entire thing isn't downloaded each time.

[balasenthil-d]

I am evaluating both approaches. One crucial advantage with Push flow is its ability to push delta changes and only when there is a change. Hence the reason for checking that out. But I think without certain support from OPA (lifecyle events and not everyone is on kubernetes right), push workflow requires considerable additional overhead to implement.

Thanks @patrick-east for clarifying my questions.

[patrick-east]

If you'd like to propose an API for the lifecycle events feel free to open another github issue with the use-case, requirements, proposal, etc. I don't have a good feel for how many other folks would want/need them but opening an RFE/proposal would help start the discussion in the community.

I'm going to go ahead and close this issue out for now, If there is more on this question feel free to re-open it 😄