Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize parms for ingest bench #9999

Merged
merged 12 commits into from
Dec 4, 2024
1 change: 1 addition & 0 deletions .github/workflows/ingest_benchmark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ concurrency:
jobs:
ingest:
strategy:
fail-fast: false # allow other variants to continue even if one fails
matrix:
target_project: [new_empty_project, large_existing_project]
permissions:
Expand Down
10 changes: 5 additions & 5 deletions test_runner/performance/test_perf_ingest_using_pgcopydb.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,13 +60,13 @@ def build_pgcopydb_command(pgcopydb_filter_file: Path, test_output_dir: Path):
"--no-acl",
"--skip-db-properties",
"--table-jobs",
"8",
"4",
"--index-jobs",
"8",
"4",
"--restore-jobs",
"8",
"4",
"--split-tables-larger-than",
"5GB",
"10GB",
"--skip-extensions",
"--use-copy-binary",
"--filters",
Expand Down Expand Up @@ -136,7 +136,7 @@ def run_command_and_log_output(command, log_file_path: Path):
"LD_LIBRARY_PATH": f"{os.getenv('PGCOPYDB_LIB_PATH')}:{os.getenv('PG_16_LIB_PATH')}",
"PGCOPYDB_SOURCE_PGURI": cast(str, os.getenv("BENCHMARK_INGEST_SOURCE_CONNSTR")),
"PGCOPYDB_TARGET_PGURI": cast(str, os.getenv("BENCHMARK_INGEST_TARGET_CONNSTR")),
"PGOPTIONS": "-c maintenance_work_mem=8388608 -c max_parallel_maintenance_workers=16",
"PGOPTIONS": "-c maintenance_work_mem=8388608 -c max_parallel_maintenance_workers=7",
}
# Combine the current environment with custom variables
env = os.environ.copy()
Expand Down
Loading