Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

workload: support loading initial data using IMPORT in run and init #35312

Merged
merged 1 commit into from
Jul 24, 2019
Merged

workload: support loading initial data using IMPORT in run and init #35312

merged 1 commit into from
Jul 24, 2019

Commits on Jul 23, 2019

  1. workload: support loading initial data using IMPORT in run and init 

    The --data-loader flag allows using IMPORTs instead of INSERTs to load
the initial table data. IMPORT is much faster but requires a CCL binary.

A few responsibilities changed in this PR. Before:
- Setup runs PostLoad
- ImportFixture conditionally runs PostLoad/splits
- `fixtures import` runs PostLoad/splits via ImportFixture
- MakeFixture doesn't include PostLoad/splits in BACKUP
- RestoreFixture always runs PostLoad/splits
- `fixtures load` runs PostLoad/splits via RestoreFixture

After:
- Setup runs PostLoad
- ImportFixture/RestoreFixture don't run splits
- `fixtures import` runs splits directly and PostLoad via Setup
- MakeFixture still doesn't include PostLoad/splits in BACKUP
- `fixtures load` runs splits directly and PostLoad via Setup

Example output:

    $ ./cockroach workload init tpcc
    I190417 19:03:15.614640 1 workload/dataload.go:146  imported warehouse (0s, 1 rows)
    I190417 19:03:15.619253 1 workload/dataload.go:146  imported district (0s, 10 rows)
    I190417 19:03:16.851827 1 workload/dataload.go:146  imported customer (1s, 29835 rows)
    I190417 19:03:17.658953 1 workload/dataload.go:146  imported history (1s, 29816 rows)
    I190417 19:03:18.318497 1 workload/dataload.go:146  imported order (1s, 29613 rows)
    I190417 19:03:18.380737 1 workload/dataload.go:146  imported new_order (0s, 7771 rows)
    I190417 19:03:19.607526 1 workload/dataload.go:146  imported item (1s, 99021 rows)
    I190417 19:03:22.630372 1 workload/dataload.go:146  imported stock (3s, 99719 rows)
    I190417 19:03:28.703662 1 workload/dataload.go:146  imported order_line (6s, 299668 rows)
    $ ./cockroach workload init tpcc --data-loader=IMPORT
    I190417 19:03:59.048167 1 ccl/workloadccl/fixture.go:453  starting import of 9 tables
    I190417 19:03:59.472322 71 ccl/workloadccl/fixture.go:561  imported district (0s, 10 rows, 0 index entries, 1003 B)
    I190417 19:03:59.617672 75 ccl/workloadccl/fixture.go:561  imported new_order (1s, 9000 rows, 0 index entries, 114 KiB)
    I190417 19:03:59.850071 70 ccl/workloadccl/fixture.go:561  imported warehouse (1s, 1 rows, 0 index entries, 52 B)
    I190417 19:04:01.585920 74 ccl/workloadccl/fixture.go:561  imported order (3s, 30000 rows, 60000 index entries, 1.7 MiB)
    I190417 19:04:02.567965 73 ccl/workloadccl/fixture.go:561  imported history (4s, 30000 rows, 60000 index entries, 3.8 MiB)
    I190417 19:04:02.621967 76 ccl/workloadccl/fixture.go:561  imported item (4s, 100000 rows, 0 index entries, 7.8 MiB)
    I190417 19:04:02.703264 72 ccl/workloadccl/fixture.go:561  imported customer (4s, 30000 rows, 30000 index entries, 18 MiB)
    I190417 19:04:04.139393 77 ccl/workloadccl/fixture.go:561  imported stock (5s, 100000 rows, 100000 index entries, 32 MiB)
    I190417 19:04:05.380937 78 ccl/workloadccl/fixture.go:561  imported order_line (6s, 300343 rows, 300343 index entries, 22 MiB)
    I190417 19:04:05.489742 1 ccl/workloadccl/fixture.go:461  imported 84 MiB bytes in 9 tables (took 6.441337s, 13.10 MiB/s)
    $ ./cockroach workload fixtures import tpcc
    I190417 19:09:35.084985 1 ccl/workloadccl/fixture.go:453  starting import of 9 tables
    I190417 19:09:35.582722 24 ccl/workloadccl/fixture.go:561  imported district (0s, 10 rows, 0 index entries, 1008 B)
    I190417 19:09:35.657582 28 ccl/workloadccl/fixture.go:561  imported new_order (1s, 9000 rows, 0 index entries, 114 KiB)
    I190417 19:09:35.941624 23 ccl/workloadccl/fixture.go:561  imported warehouse (1s, 1 rows, 0 index entries, 53 B)
    I190417 19:09:37.821479 27 ccl/workloadccl/fixture.go:561  imported order (3s, 30000 rows, 60000 index entries, 1.7 MiB)
    I190417 19:09:38.509619 26 ccl/workloadccl/fixture.go:561  imported history (3s, 30000 rows, 60000 index entries, 3.8 MiB)
    I190417 19:09:38.598668 25 ccl/workloadccl/fixture.go:561  imported customer (4s, 30000 rows, 30000 index entries, 18 MiB)
    I190417 19:09:38.677361 29 ccl/workloadccl/fixture.go:561  imported item (4s, 100000 rows, 0 index entries, 7.8 MiB)
    I190417 19:09:40.245621 30 ccl/workloadccl/fixture.go:561  imported stock (5s, 100000 rows, 100000 index entries, 32 MiB)
    I190417 19:09:41.406895 31 ccl/workloadccl/fixture.go:561  imported order_line (6s, 300343 rows, 300343 index entries, 22 MiB)
    I190417 19:09:41.532481 1 ccl/workloadccl/fixture.go:461  imported 84 MiB bytes in 9 tables (took 6.447149s, 13.09 MiB/s)
    I190417 19:09:41.545086 1 workload/workload.go:325  data is loaded; now running consistency checks (ctrl-c to abort)
    I190417 19:09:41.552516 1 workload/tpcc/tpcc.go:322  check 3.3.2.1 took 7.364ms
    I190417 19:09:41.593546 1 workload/tpcc/tpcc.go:322  check 3.3.2.2 took 40.99ms
    I190417 19:09:41.601818 1 workload/tpcc/tpcc.go:322  check 3.3.2.3 took 8.229ms
    I190417 19:09:41.868776 1 workload/tpcc/tpcc.go:322  check 3.3.2.4 took 266.907ms
    I190417 19:09:41.958044 1 workload/tpcc/tpcc.go:322  check 3.3.2.5 took 89.214ms
    I190417 19:09:42.322464 1 workload/tpcc/tpcc.go:322  check 3.3.2.7 took 364.368ms
    I190417 19:09:42.365025 1 workload/tpcc/tpcc.go:322  check 3.3.2.8 took 42.53ms
    I190417 19:09:42.412558 1 workload/tpcc/tpcc.go:322  check 3.3.2.9 took 47.501ms
    $ ./cockroach workload fixtures load tpcc
    I190417 19:18:42.645587 1 ccl/workloadccl/cliccl/fixtures.go:286  starting restore of 9 tables
    I190417 19:18:56.447737 97 ccl/workloadccl/fixture.go:647  loaded new_order (14s, 9000 rows, 0 index entries, 114 KiB)
    I190417 19:18:56.447870 116 ccl/workloadccl/fixture.go:647  loaded order_line (14s, 299890 rows, 299890 index entries, 20 MiB)
    I190417 19:18:57.365546 96 ccl/workloadccl/fixture.go:647  loaded order (15s, 30000 rows, 60000 index entries, 1.7 MiB)
    I190417 19:18:57.365630 114 ccl/workloadccl/fixture.go:647  loaded item (15s, 100000 rows, 0 index entries, 7.8 MiB)
    I190417 19:18:58.836369 92 ccl/workloadccl/fixture.go:647  loaded warehouse (16s, 1 rows, 0 index entries, 52 B)
    I190417 19:18:58.836384 94 ccl/workloadccl/fixture.go:647  loaded customer (16s, 30000 rows, 30000 index entries, 18 MiB)
    I190417 19:18:59.315224 93 ccl/workloadccl/fixture.go:647  loaded district (17s, 10 rows, 0 index entries, 1011 B)
    I190417 19:18:59.923138 95 ccl/workloadccl/fixture.go:647  loaded history (17s, 30000 rows, 60000 index entries, 3.8 MiB)
    I190417 19:19:02.510851 115 ccl/workloadccl/fixture.go:647  loaded stock (20s, 100000 rows, 100000 index entries, 32 MiB)
    I190417 19:19:02.632022 1 ccl/workloadccl/cliccl/fixtures.go:293  imported 83 MiB bytes in 9 tables (took 19.986216s, 4.16 MiB/s)
    I190417 19:19:02.644449 1 workload/workload.go:325  data is loaded; now running consistency checks (ctrl-c to abort)
    I190417 19:19:02.651055 1 workload/tpcc/tpcc.go:322  check 3.3.2.1 took 6.562ms
    I190417 19:19:02.691053 1 workload/tpcc/tpcc.go:322  check 3.3.2.2 took 39.967ms
    I190417 19:19:02.699790 1 workload/tpcc/tpcc.go:322  check 3.3.2.3 took 8.697ms
    I190417 19:19:02.977256 1 workload/tpcc/tpcc.go:322  check 3.3.2.4 took 277.415ms
    I190417 19:19:03.070779 1 workload/tpcc/tpcc.go:322  check 3.3.2.5 took 93.462ms
    I190417 19:19:03.450045 1 workload/tpcc/tpcc.go:322  check 3.3.2.7 took 379.226ms
    I190417 19:19:03.499275 1 workload/tpcc/tpcc.go:322  check 3.3.2.8 took 49.183ms
    I190417 19:19:03.552005 1 workload/tpcc/tpcc.go:322  check 3.3.2.9 took 52.678ms

Release note (cli change): `workload run` and `workload init` now
support loading initial table data using IMPORT
    @danhhz
    danhhz committed Jul 23, 2019
    Configuration menu
    Copy the full SHA
    f491ba3 View commit details
    Browse the repository at this point in the history