Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to prevent full refreshes every month #93

Open
pierre opened this issue Apr 23, 2019 · 1 comment
Open

Add option to prevent full refreshes every month #93

pierre opened this issue Apr 23, 2019 · 1 comment

Comments

@pierre
Copy link
Member

pierre commented Apr 23, 2019

As data grows over time, refreshes take more and more time. Can we implement incremental refreshes?

@pierre
Copy link
Member Author

pierre commented May 17, 2019

A real incremental import is quite difficult because of the data denormalization across tables. We can be smarter however to do less work on a per event basis.

Proposed changes follow (new indices needed).

BUNDLE_* and SUBSCRIPTION_* events

  • Refresh only the rows for the bundle associated with the event.
  • Care must be taken however to update bundle_account_rank in analytics_bundles.

PAYMENT_SUCCESS and PAYMENT_FAILED events

  • Refresh invoice row and invoice item rows for the invoice id associated with the event only (if the payment points to an invoice payment).
  • Refresh only the payment tables for the paymentId from the event.

INVOICE_CREATION and INVOICE_ADJUSTMENT events

  • Don't refresh payments.
  • If the event is INVOICE_CREATION and if the invoice doesn't contain any CBA item, only refresh invoice and invoice item rows for that invoice id. Otherwise, we need to still refresh all invoice and invoice items (impact of CBA across invoices). See Send adjustment event after CBA run killbill#1143.

pierre added a commit that referenced this issue Jun 5, 2019
Only refresh invoice and associated invoice item rows for
INVOICE_CREATION and INVOICE_ADJUSTMENT events.

This depends on killbill/killbill#1143 for
accuracy with CBA items.

See #93.

Signed-off-by: Pierre-Alexandre Meyer <pierre@mouraf.org>
pierre added a commit that referenced this issue Nov 4, 2019
In #93,
we stopped refreshing payments for INVOICE_CREATION and INVOICE_ADJUSTMENT events.

Payment rows still need to be updated (if they exist) when CBA is distributed, as
the balance may change.

This fixes #105.

Signed-off-by: Pierre-Alexandre Meyer <pierre@mouraf.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant