Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatically split up each batch into an acceptable request to the Stitch API #17

Merged
merged 1 commit into from
Aug 10, 2022

Conversation

Phanatic
Copy link
Contributor

@Phanatic Phanatic commented Aug 9, 2022

The Stitch API has a few limitations about the request size and contents :

  1. The request includes malformed JSON/Transit, did not provide an array of records, or contains more than 20,000 records.
  2. The size of the request body exceeded 20MB.

We do have a batch-size CLI parameter in the ps-http-tap that allows the caller to pick a batch size for the request to Stitch API. But this requires a lot of manual trial and error to get the value right.
Instead of asking the customer to pick a batch size in the request, we will now automatically detect the size of the request body that we send to Stitch API.

Implemented in :

// getBatchMessages accepts a list of import messages
// and returns a slice of ImportBatch that can be safely uploaded.
// The rules are:
// 1. There cannot be more than 20,000 records in the request.
// 2. The size of the serialized JSON cannot be more than 20 MB.
func getBatchMessages(messages []ImportMessage, stream *Stream, maxObjectsInBatch int, maxBatchSerializedSize int) []ImportBatch {

@Phanatic Phanatic requested review from mdlayher and nickvanw August 9, 2022 21:50
@Phanatic Phanatic merged commit 3836343 into main Aug 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants