ETL Water Quality Data from the NWIS System
This ETL is in the process of being transformed to a Java project using Spring Batch. For the moment we will include the README's for both.
Create an application.yml file in the project directory containing the following (shown are example values - they should match the values you used in creating the etlDB):
WQP_DATABASE_ADDRESS: <localhost>
WQP_DATABASE_PORT: <5437>
WQP_DATABASE_NAME: <wqp_db>
WQP_SCHEMA_OWNER_USERNAME: <wqp_core>
WQP_SCHEMA_OWNER_PASSWORD: <changeme>
NWIS_DATABASE_ADDRESS: <localhost>
NWIS_DATABASE_PORT: <5437>
NWIS_DATABASE_NAME: <wqp_db>
NWIS_SCHEMA_OWNER_USERNAME: <nwis_ws_star>
NWIS_SCHEMA_OWNER_PASSWORD: <changeme>
# Currently I have to open a tunnel to the MYSQL db at EROS. That's why
# the address is localhost. The tunnel is opened as follows: ssh -L 1234:localhost:3306 <actual_database_address
MYSQL_DATABASE_ADDRESS: <localhost>
MYSQL_DATABASE_PORT: <1234>
MYSQL_DATABASE_NAME: <nwisweb>
MYSQL_USERNAME: <nwis_user>
MYSQL_PASSWORD: <changeme>
WQP_SCHEMA_NAME: <wqp>
ETL_OWNER_USERNAME: <nwis_ws_star>
GEO_SCHEMA_NAME: <nwis>
ETL_DATA_SOURCE_ID: <2>
ETL_DATA_SOURCE: <NWIS>
QWPORTAL_SUMMARY_ETL: <true>
NWIS_OR_EPA: N
This project contains JUnit tests. Maven can be used to run them (in addition to the capabilities of your IDE).
To run the unit tests of the application use:
mvn packageTo additionally start up a Docker database and run the integration tests of the application use:
mvn verify -DTESTING_DATABASE_PORT=5437 -DTESTING_DATABASE_ADDRESS=localhost -DTESTING_DATABASE_NETWORK=wqpEtlCore