Skip to content

Commit

Permalink
Update Spark run language/command (#646)
Browse files Browse the repository at this point in the history
* Update Spark run language/command

* Fix print spaces
  • Loading branch information
dfsnow authored Nov 18, 2024
1 parent 69fd57e commit ba9db30
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 9 deletions.
9 changes: 5 additions & 4 deletions dbt/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -890,12 +890,13 @@ You should see output like this, which you can run in the context of the
repository on the server in order to refresh iasWorld tables:

```
Run the following commands on the Data Team server:
Run the following commands on the Data Team server as the shiny-server user:
cd /path/to/service-spark-iasworld/
docker compose up -d
docker exec spark-node-master ./submit.sh --json-string --no-run-github-workflow
'{"aprval": {"table_name": "iasworld.aprval", "min_year": 2024, "cur": ["Y"], ...
docker compose --profile prod up -d
docker exec spark-node-master-prod ./submit.sh
--upload-data --upload-logs --run-glue-crawler
--json-string {"aprval": {"table_name": "iasworld.aprval", "min_year": 2024, "cur": ["Y"], ...
```

#### Running the AHSAP change in value QC report
Expand Down
13 changes: 8 additions & 5 deletions dbt/scripts/export_qc_town_close_reports.py
Original file line number Diff line number Diff line change
Expand Up @@ -181,14 +181,17 @@ def main():

iasworld_deps[table_name] = formatted_dep

print("Run the following commands on the Data Team server:")
print(
"Run the following commands on the Data Team",
"server as the shiny-server user:",
)
print()
print("cd /home/shiny-server/services/service-spark-iasworld")
print("docker compose up -d")
print("docker compose --profile prod up -d")
print(
"docker exec spark-node-master ./submit.sh "
"--no-run-github-workflow "
f"--json-string '{json.dumps(iasworld_deps)}' "
"docker exec spark-node-master-prod ./submit.sh",
"--upload-data --upload-logs --run-glue-crawler",
f"--json-string '{json.dumps(iasworld_deps)}'",
)
else:
models_for_export = query_models_for_export(
Expand Down

0 comments on commit ba9db30

Please sign in to comment.