workload: support loading initial data using IMPORT in run and init
#35312
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
|
Nathan's out this week, reassigning |
danhhz
force-pushed
the
workload_cli
branch
2 times, most recently
from
633e2ad
to
9355429
Compare
|
Nathan was the person I originally discussed this with and he's long since back from vacation, so assigning back. |
|
Just noticed there were test failures and fixed them. The merge conflict is something I just introduced in #37080, but it should be easy to resolve. This is ready for a look, but there's also no real hurry |
|
Hey @nvanbenschoten or @tbg, would one of you mind reviewing this? I'd love to get it into 19.2. It has some merge conflicts at this point, but I've already resolved them a couple times on this PR and it's pretty annoying. I'd love to get signoff on the approach before rebasing one last time. |
|
Reviewing now. Sorry to let this sit. |
nvanbenschoten
approved these changes
Jul 19, 2019
There was a problem hiding this comment.
Very nice! LGTM
Reviewed 13 of 13 files at r1.
Reviewable status:complete! 0 of 0 LGTMs obtained (waiting on @tbg)
danhhz
force-pushed
the
workload_cli
branch
3 times, most recently
from
c7f7722
to
209258e
Compare
|
Thanks for the review! No worries on the delay, I did say there was no hurry. I've rebased this one last time and it should be ready to go. |
The --data-loader flag allows using IMPORTs instead of INSERTs to load
the initial table data. IMPORT is much faster but requires a CCL binary.
A few responsibilities changed in this PR. Before:
- Setup runs PostLoad
- ImportFixture conditionally runs PostLoad/splits
- `fixtures import` runs PostLoad/splits via ImportFixture
- MakeFixture doesn't include PostLoad/splits in BACKUP
- RestoreFixture always runs PostLoad/splits
- `fixtures load` runs PostLoad/splits via RestoreFixture
After:
- Setup runs PostLoad
- ImportFixture/RestoreFixture don't run splits
- `fixtures import` runs splits directly and PostLoad via Setup
- MakeFixture still doesn't include PostLoad/splits in BACKUP
- `fixtures load` runs splits directly and PostLoad via Setup
Example output:
$ ./cockroach workload init tpcc
I190417 19:03:15.614640 1 workload/dataload.go:146 imported warehouse (0s, 1 rows)
I190417 19:03:15.619253 1 workload/dataload.go:146 imported district (0s, 10 rows)
I190417 19:03:16.851827 1 workload/dataload.go:146 imported customer (1s, 29835 rows)
I190417 19:03:17.658953 1 workload/dataload.go:146 imported history (1s, 29816 rows)
I190417 19:03:18.318497 1 workload/dataload.go:146 imported order (1s, 29613 rows)
I190417 19:03:18.380737 1 workload/dataload.go:146 imported new_order (0s, 7771 rows)
I190417 19:03:19.607526 1 workload/dataload.go:146 imported item (1s, 99021 rows)
I190417 19:03:22.630372 1 workload/dataload.go:146 imported stock (3s, 99719 rows)
I190417 19:03:28.703662 1 workload/dataload.go:146 imported order_line (6s, 299668 rows)
$ ./cockroach workload init tpcc --data-loader=IMPORT
I190417 19:03:59.048167 1 ccl/workloadccl/fixture.go:453 starting import of 9 tables
I190417 19:03:59.472322 71 ccl/workloadccl/fixture.go:561 imported district (0s, 10 rows, 0 index entries, 1003 B)
I190417 19:03:59.617672 75 ccl/workloadccl/fixture.go:561 imported new_order (1s, 9000 rows, 0 index entries, 114 KiB)
I190417 19:03:59.850071 70 ccl/workloadccl/fixture.go:561 imported warehouse (1s, 1 rows, 0 index entries, 52 B)
I190417 19:04:01.585920 74 ccl/workloadccl/fixture.go:561 imported order (3s, 30000 rows, 60000 index entries, 1.7 MiB)
I190417 19:04:02.567965 73 ccl/workloadccl/fixture.go:561 imported history (4s, 30000 rows, 60000 index entries, 3.8 MiB)
I190417 19:04:02.621967 76 ccl/workloadccl/fixture.go:561 imported item (4s, 100000 rows, 0 index entries, 7.8 MiB)
I190417 19:04:02.703264 72 ccl/workloadccl/fixture.go:561 imported customer (4s, 30000 rows, 30000 index entries, 18 MiB)
I190417 19:04:04.139393 77 ccl/workloadccl/fixture.go:561 imported stock (5s, 100000 rows, 100000 index entries, 32 MiB)
I190417 19:04:05.380937 78 ccl/workloadccl/fixture.go:561 imported order_line (6s, 300343 rows, 300343 index entries, 22 MiB)
I190417 19:04:05.489742 1 ccl/workloadccl/fixture.go:461 imported 84 MiB bytes in 9 tables (took 6.441337s, 13.10 MiB/s)
$ ./cockroach workload fixtures import tpcc
I190417 19:09:35.084985 1 ccl/workloadccl/fixture.go:453 starting import of 9 tables
I190417 19:09:35.582722 24 ccl/workloadccl/fixture.go:561 imported district (0s, 10 rows, 0 index entries, 1008 B)
I190417 19:09:35.657582 28 ccl/workloadccl/fixture.go:561 imported new_order (1s, 9000 rows, 0 index entries, 114 KiB)
I190417 19:09:35.941624 23 ccl/workloadccl/fixture.go:561 imported warehouse (1s, 1 rows, 0 index entries, 53 B)
I190417 19:09:37.821479 27 ccl/workloadccl/fixture.go:561 imported order (3s, 30000 rows, 60000 index entries, 1.7 MiB)
I190417 19:09:38.509619 26 ccl/workloadccl/fixture.go:561 imported history (3s, 30000 rows, 60000 index entries, 3.8 MiB)
I190417 19:09:38.598668 25 ccl/workloadccl/fixture.go:561 imported customer (4s, 30000 rows, 30000 index entries, 18 MiB)
I190417 19:09:38.677361 29 ccl/workloadccl/fixture.go:561 imported item (4s, 100000 rows, 0 index entries, 7.8 MiB)
I190417 19:09:40.245621 30 ccl/workloadccl/fixture.go:561 imported stock (5s, 100000 rows, 100000 index entries, 32 MiB)
I190417 19:09:41.406895 31 ccl/workloadccl/fixture.go:561 imported order_line (6s, 300343 rows, 300343 index entries, 22 MiB)
I190417 19:09:41.532481 1 ccl/workloadccl/fixture.go:461 imported 84 MiB bytes in 9 tables (took 6.447149s, 13.09 MiB/s)
I190417 19:09:41.545086 1 workload/workload.go:325 data is loaded; now running consistency checks (ctrl-c to abort)
I190417 19:09:41.552516 1 workload/tpcc/tpcc.go:322 check 3.3.2.1 took 7.364ms
I190417 19:09:41.593546 1 workload/tpcc/tpcc.go:322 check 3.3.2.2 took 40.99ms
I190417 19:09:41.601818 1 workload/tpcc/tpcc.go:322 check 3.3.2.3 took 8.229ms
I190417 19:09:41.868776 1 workload/tpcc/tpcc.go:322 check 3.3.2.4 took 266.907ms
I190417 19:09:41.958044 1 workload/tpcc/tpcc.go:322 check 3.3.2.5 took 89.214ms
I190417 19:09:42.322464 1 workload/tpcc/tpcc.go:322 check 3.3.2.7 took 364.368ms
I190417 19:09:42.365025 1 workload/tpcc/tpcc.go:322 check 3.3.2.8 took 42.53ms
I190417 19:09:42.412558 1 workload/tpcc/tpcc.go:322 check 3.3.2.9 took 47.501ms
$ ./cockroach workload fixtures load tpcc
I190417 19:18:42.645587 1 ccl/workloadccl/cliccl/fixtures.go:286 starting restore of 9 tables
I190417 19:18:56.447737 97 ccl/workloadccl/fixture.go:647 loaded new_order (14s, 9000 rows, 0 index entries, 114 KiB)
I190417 19:18:56.447870 116 ccl/workloadccl/fixture.go:647 loaded order_line (14s, 299890 rows, 299890 index entries, 20 MiB)
I190417 19:18:57.365546 96 ccl/workloadccl/fixture.go:647 loaded order (15s, 30000 rows, 60000 index entries, 1.7 MiB)
I190417 19:18:57.365630 114 ccl/workloadccl/fixture.go:647 loaded item (15s, 100000 rows, 0 index entries, 7.8 MiB)
I190417 19:18:58.836369 92 ccl/workloadccl/fixture.go:647 loaded warehouse (16s, 1 rows, 0 index entries, 52 B)
I190417 19:18:58.836384 94 ccl/workloadccl/fixture.go:647 loaded customer (16s, 30000 rows, 30000 index entries, 18 MiB)
I190417 19:18:59.315224 93 ccl/workloadccl/fixture.go:647 loaded district (17s, 10 rows, 0 index entries, 1011 B)
I190417 19:18:59.923138 95 ccl/workloadccl/fixture.go:647 loaded history (17s, 30000 rows, 60000 index entries, 3.8 MiB)
I190417 19:19:02.510851 115 ccl/workloadccl/fixture.go:647 loaded stock (20s, 100000 rows, 100000 index entries, 32 MiB)
I190417 19:19:02.632022 1 ccl/workloadccl/cliccl/fixtures.go:293 imported 83 MiB bytes in 9 tables (took 19.986216s, 4.16 MiB/s)
I190417 19:19:02.644449 1 workload/workload.go:325 data is loaded; now running consistency checks (ctrl-c to abort)
I190417 19:19:02.651055 1 workload/tpcc/tpcc.go:322 check 3.3.2.1 took 6.562ms
I190417 19:19:02.691053 1 workload/tpcc/tpcc.go:322 check 3.3.2.2 took 39.967ms
I190417 19:19:02.699790 1 workload/tpcc/tpcc.go:322 check 3.3.2.3 took 8.697ms
I190417 19:19:02.977256 1 workload/tpcc/tpcc.go:322 check 3.3.2.4 took 277.415ms
I190417 19:19:03.070779 1 workload/tpcc/tpcc.go:322 check 3.3.2.5 took 93.462ms
I190417 19:19:03.450045 1 workload/tpcc/tpcc.go:322 check 3.3.2.7 took 379.226ms
I190417 19:19:03.499275 1 workload/tpcc/tpcc.go:322 check 3.3.2.8 took 49.183ms
I190417 19:19:03.552005 1 workload/tpcc/tpcc.go:322 check 3.3.2.9 took 52.678ms
Release note (cli change): `workload run` and `workload init` now
support loading initial table data using IMPORT
nvanbenschoten
approved these changes
Jul 24, 2019
|
bors r=nvanbenschoten |
craig bot
pushed a commit
that referenced
this pull request
Jul 24, 2019
35312: workload: support loading initial data using IMPORT in `run` and `init` r=nvanbenschoten a=danhhz
The --data-loader flag allows using IMPORTs instead of INSERTs to load
the initial table data. IMPORT is much faster but requires a CCL binary.
Example output:
$ ./cockroach workload init tpcc
I190417 19:03:15.614640 1 workload/dataload.go:146 imported warehouse (0s, 1 rows)
I190417 19:03:15.619253 1 workload/dataload.go:146 imported district (0s, 10 rows)
I190417 19:03:16.851827 1 workload/dataload.go:146 imported customer (1s, 29835 rows)
I190417 19:03:17.658953 1 workload/dataload.go:146 imported history (1s, 29816 rows)
I190417 19:03:18.318497 1 workload/dataload.go:146 imported order (1s, 29613 rows)
I190417 19:03:18.380737 1 workload/dataload.go:146 imported new_order (0s, 7771 rows)
I190417 19:03:19.607526 1 workload/dataload.go:146 imported item (1s, 99021 rows)
I190417 19:03:22.630372 1 workload/dataload.go:146 imported stock (3s, 99719 rows)
I190417 19:03:28.703662 1 workload/dataload.go:146 imported order_line (6s, 299668 rows)
$ ./cockroach workload init tpcc --data-loader=IMPORT
I190417 19:03:59.048167 1 ccl/workloadccl/fixture.go:453 starting import of 9 tables
I190417 19:03:59.472322 71 ccl/workloadccl/fixture.go:561 imported district (0s, 10 rows, 0 index entries, 1003 B)
I190417 19:03:59.617672 75 ccl/workloadccl/fixture.go:561 imported new_order (1s, 9000 rows, 0 index entries, 114 KiB)
I190417 19:03:59.850071 70 ccl/workloadccl/fixture.go:561 imported warehouse (1s, 1 rows, 0 index entries, 52 B)
I190417 19:04:01.585920 74 ccl/workloadccl/fixture.go:561 imported order (3s, 30000 rows, 60000 index entries, 1.7 MiB)
I190417 19:04:02.567965 73 ccl/workloadccl/fixture.go:561 imported history (4s, 30000 rows, 60000 index entries, 3.8 MiB)
I190417 19:04:02.621967 76 ccl/workloadccl/fixture.go:561 imported item (4s, 100000 rows, 0 index entries, 7.8 MiB)
I190417 19:04:02.703264 72 ccl/workloadccl/fixture.go:561 imported customer (4s, 30000 rows, 30000 index entries, 18 MiB)
I190417 19:04:04.139393 77 ccl/workloadccl/fixture.go:561 imported stock (5s, 100000 rows, 100000 index entries, 32 MiB)
I190417 19:04:05.380937 78 ccl/workloadccl/fixture.go:561 imported order_line (6s, 300343 rows, 300343 index entries, 22 MiB)
I190417 19:04:05.489742 1 ccl/workloadccl/fixture.go:461 imported 84 MiB bytes in 9 tables (took 6.441337s, 13.10 MiB/s)
$ ./cockroach workload fixtures import tpcc
I190417 19:09:35.084985 1 ccl/workloadccl/fixture.go:453 starting import of 9 tables
I190417 19:09:35.582722 24 ccl/workloadccl/fixture.go:561 imported district (0s, 10 rows, 0 index entries, 1008 B)
I190417 19:09:35.657582 28 ccl/workloadccl/fixture.go:561 imported new_order (1s, 9000 rows, 0 index entries, 114 KiB)
I190417 19:09:35.941624 23 ccl/workloadccl/fixture.go:561 imported warehouse (1s, 1 rows, 0 index entries, 53 B)
I190417 19:09:37.821479 27 ccl/workloadccl/fixture.go:561 imported order (3s, 30000 rows, 60000 index entries, 1.7 MiB)
I190417 19:09:38.509619 26 ccl/workloadccl/fixture.go:561 imported history (3s, 30000 rows, 60000 index entries, 3.8 MiB)
I190417 19:09:38.598668 25 ccl/workloadccl/fixture.go:561 imported customer (4s, 30000 rows, 30000 index entries, 18 MiB)
I190417 19:09:38.677361 29 ccl/workloadccl/fixture.go:561 imported item (4s, 100000 rows, 0 index entries, 7.8 MiB)
I190417 19:09:40.245621 30 ccl/workloadccl/fixture.go:561 imported stock (5s, 100000 rows, 100000 index entries, 32 MiB)
I190417 19:09:41.406895 31 ccl/workloadccl/fixture.go:561 imported order_line (6s, 300343 rows, 300343 index entries, 22 MiB)
I190417 19:09:41.532481 1 ccl/workloadccl/fixture.go:461 imported 84 MiB bytes in 9 tables (took 6.447149s, 13.09 MiB/s)
I190417 19:09:41.545086 1 workload/workload.go:325 data is loaded; now running consistency checks (ctrl-c to abort)
I190417 19:09:41.552516 1 workload/tpcc/tpcc.go:322 check 3.3.2.1 took 7.364ms
I190417 19:09:41.593546 1 workload/tpcc/tpcc.go:322 check 3.3.2.2 took 40.99ms
I190417 19:09:41.601818 1 workload/tpcc/tpcc.go:322 check 3.3.2.3 took 8.229ms
I190417 19:09:41.868776 1 workload/tpcc/tpcc.go:322 check 3.3.2.4 took 266.907ms
I190417 19:09:41.958044 1 workload/tpcc/tpcc.go:322 check 3.3.2.5 took 89.214ms
I190417 19:09:42.322464 1 workload/tpcc/tpcc.go:322 check 3.3.2.7 took 364.368ms
I190417 19:09:42.365025 1 workload/tpcc/tpcc.go:322 check 3.3.2.8 took 42.53ms
I190417 19:09:42.412558 1 workload/tpcc/tpcc.go:322 check 3.3.2.9 took 47.501ms
$ ./cockroach workload fixtures load tpcc
I190417 19:18:42.645587 1 ccl/workloadccl/cliccl/fixtures.go:286 starting restore of 9 tables
I190417 19:18:56.447737 97 ccl/workloadccl/fixture.go:647 loaded new_order (14s, 9000 rows, 0 index entries, 114 KiB)
I190417 19:18:56.447870 116 ccl/workloadccl/fixture.go:647 loaded order_line (14s, 299890 rows, 299890 index entries, 20 MiB)
I190417 19:18:57.365546 96 ccl/workloadccl/fixture.go:647 loaded order (15s, 30000 rows, 60000 index entries, 1.7 MiB)
I190417 19:18:57.365630 114 ccl/workloadccl/fixture.go:647 loaded item (15s, 100000 rows, 0 index entries, 7.8 MiB)
I190417 19:18:58.836369 92 ccl/workloadccl/fixture.go:647 loaded warehouse (16s, 1 rows, 0 index entries, 52 B)
I190417 19:18:58.836384 94 ccl/workloadccl/fixture.go:647 loaded customer (16s, 30000 rows, 30000 index entries, 18 MiB)
I190417 19:18:59.315224 93 ccl/workloadccl/fixture.go:647 loaded district (17s, 10 rows, 0 index entries, 1011 B)
I190417 19:18:59.923138 95 ccl/workloadccl/fixture.go:647 loaded history (17s, 30000 rows, 60000 index entries, 3.8 MiB)
I190417 19:19:02.510851 115 ccl/workloadccl/fixture.go:647 loaded stock (20s, 100000 rows, 100000 index entries, 32 MiB)
I190417 19:19:02.632022 1 ccl/workloadccl/cliccl/fixtures.go:293 imported 83 MiB bytes in 9 tables (took 19.986216s, 4.16 MiB/s)
I190417 19:19:02.644449 1 workload/workload.go:325 data is loaded; now running consistency checks (ctrl-c to abort)
I190417 19:19:02.651055 1 workload/tpcc/tpcc.go:322 check 3.3.2.1 took 6.562ms
I190417 19:19:02.691053 1 workload/tpcc/tpcc.go:322 check 3.3.2.2 took 39.967ms
I190417 19:19:02.699790 1 workload/tpcc/tpcc.go:322 check 3.3.2.3 took 8.697ms
I190417 19:19:02.977256 1 workload/tpcc/tpcc.go:322 check 3.3.2.4 took 277.415ms
I190417 19:19:03.070779 1 workload/tpcc/tpcc.go:322 check 3.3.2.5 took 93.462ms
I190417 19:19:03.450045 1 workload/tpcc/tpcc.go:322 check 3.3.2.7 took 379.226ms
I190417 19:19:03.499275 1 workload/tpcc/tpcc.go:322 check 3.3.2.8 took 49.183ms
I190417 19:19:03.552005 1 workload/tpcc/tpcc.go:322 check 3.3.2.9 took 52.678ms
Release note (cli change): `workload run` and `workload init` now
support loading initial table data using IMPORT
Co-authored-by: Daniel Harrison <daniel.harrison@gmail.com>
Build succeeded |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The --data-loader flag allows using IMPORTs instead of INSERTs to load
the initial table data. IMPORT is much faster but requires a CCL binary.
Example output:
Release note (cli change):
workload runandworkload initnowsupport loading initial table data using IMPORT