Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add basic testing large syncs support #1862

Merged
merged 3 commits into from
Aug 8, 2024

Conversation

erezrokah
Copy link
Member

@erezrokah erezrokah commented Aug 8, 2024

Summary

A very crude and simple way to do https://github.com/cloudquery/cloudquery-issues/issues/1846.
Adds a new hidden fuzz test scheduler that only multiplies the clients (at the moment).
The code is based on the shuffle scheduler then adds then duplicates client based on the multiplier.

See example:

cloudquery sync examples/pagerduty-postgres.yml

Loading spec(s) from examples/pagerduty-postgres.yml
Starting sync for: pagerduty (local@/Users/erezrokah/code/github/cloudquery/cloudquery-private/plugins/source/pagerduty/pagerduty) -> [postgresql (cloudquery/postgresql@v8.2.7)]
Sync completed successfully. Resources: 525, Errors: 0, Warnings: 0, Time: 7s
CQ_DEBUG_SYNC_MULTIPLIER=50 cloudquery sync examples/pagerduty-postgres.yml

Loading spec(s) from examples/pagerduty-postgres.yml
Starting sync for: pagerduty (local@/Users/erezrokah/code/github/cloudquery/cloudquery-private/plugins/source/pagerduty/pagerduty) -> [postgresql (cloudquery/postgresql@v8.2.7)]
Sync completed successfully. Resources: 26385, Errors: 0, Warnings: 0, Time: 2m12s

This has a couple of downsides/tradeoffs

  1. There will be clients with duplicate IDs which breaks the metrics counts
    s.TableClient[tableName][clientID] = &TableClientMetrics{
  2. If a plugin uses the client ID to ensure uniqueness for state client keys, that logic will break too
  3. If a table doesn't have any resources the impact of the multiplier will be lower

However I think this is still useful if we want to artificially make a sync large (e.g. simulate a sync on many AWS accounts)


Use the following steps to ensure your PR is ready to be reviewed

  • Read the contribution guidelines 🧑‍🎓
  • Run go fmt to format your code 🖊
  • Lint your changes via golangci-lint run 🚨 (install golangci-lint here)
  • Update or add tests 🧪
  • Ensure the status checks below are successful ✅

@erezrokah erezrokah requested a review from a team as a code owner August 8, 2024 12:26
@github-actions github-actions bot added the feat label Aug 8, 2024
Copy link
Contributor

@marianogappa marianogappa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think scheduler_fuzz is a good name for this. I did the exercise of looking at the code without looking at the issue first, and it made no sense to me what the point of this scheduler was 🤔

It's more of a load test scheduler 🤔 It doesn't do fuzzing.

I think it would pay to add a small comment in the constructor of what problem it solves.

scheduler/scheduler_fuzz.go Outdated Show resolved Hide resolved
@erezrokah
Copy link
Member Author

I don't think scheduler_fuzz is a good name for this. I did the exercise of looking at the code without looking at the issue first, and it made no sense to me what the point of this scheduler was 🤔

It's more of a load test scheduler 🤔 It doesn't do fuzzing.

💯 Happy to rename it. I don't like the fuzz name as well (though it might do some fuzzing in the future). I'll rename

@erezrokah erezrokah changed the title feat: Add basic fuzzing support feat: Add basic testing large syncs support Aug 8, 2024
@github-actions github-actions bot added feat and removed feat labels Aug 8, 2024
@github-actions github-actions bot added feat and removed feat labels Aug 8, 2024
@github-actions github-actions bot added feat and removed feat labels Aug 8, 2024
@erezrokah
Copy link
Member Author

💯 Happy to rename it. I don't like the fuzz name as well (though it might do some fuzzing in the future). I'll rename

Done in 08bb4a0

@kodiakhq kodiakhq bot merged commit 40a0095 into cloudquery:main Aug 8, 2024
8 checks passed
@erezrokah erezrokah deleted the feat/add_fuzz_scheduler branch August 8, 2024 15:00
kodiakhq bot pushed a commit that referenced this pull request Aug 8, 2024
🤖 I have created a release *beep* *boop*
---


## [4.59.0](v4.58.1...v4.59.0) (2024-08-08)


### Features

* Add basic testing large syncs support ([#1862](#1862)) ([40a0095](40a0095))


### Bug Fixes

* **deps:** Update module github.com/cloudquery/cloudquery-api-go to v1.12.7 ([#1860](#1860)) ([25ed3d2](25ed3d2))

---
This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants