Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Create DB Migration for ETL Tool monitoring tables #955

Conversation

marcionemec-daitan
Copy link
Contributor

@marcionemec-daitan marcionemec-daitan commented Apr 1, 2024

Description

Create monitoring tables (ETL_EXECUTION_LOG_HIST, ETL_EXECUTION_LOG and ETL_EXECUTION_MAP) to store interface executions of batch processes.

Refers to issue 35 of old spar-data-sync repo.

Changelog

New

Changed

Removed

How was this tested?

  • 🧠 Not needed
  • 👀 Eyeball
  • 🤖 Added tests

What gif/image best describes this PR or how it makes you feel?


Thanks for the PR!

Any successful deployments (not always required) will be available below.
Frontend
Backend
Oracle-API

Once merged, code will be promoted and handed off to following workflow run.
Main Merge Workflow

Copy link
Member

@DerekRoberts DerekRoberts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Progress. :)

@DerekRoberts
Copy link
Member

@marcionemec-daitan Backend deployment is failing with the message below.

2024-04-01 17:17:33.888 �[31mWARN �[0;39m [�[34mmain�[0;39m] �[33morg.springframework.boot.web.servlet.context.ServletWebServerApplicationContext�[0;39m: Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flywayInitializer': Migration V29__create_etl_tool_monitoring_tables.sql failed
-----------------------------------------------------------
SQL State  : 42703
Error Code : 0
Message    : ERROR: column "last_run_ts" of relation "spar.etl_execution_map" does not exist
Location   : db/migration/V29__create_etl_tool_monitoring_tables.sql (/db/migration/V29__create_etl_tool_monitoring_tables.sql)
Line       : 40
Statement  : comment on column spar.ETL_EXECUTION_MAP.last_run_ts                is 'Last timestamp this interface was executed for batch execution'

Copy link
Collaborator

@craigyu craigyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQL State  : 42703
Error Code : 0
Message    : ERROR: column "last_run_ts" of relation "spar.etl_execution_map" does not exist
Location   : db/migration/V29__create_etl_tool_monitoring_tables.sql (/db/migration/V29__create_etl_tool_monitoring_tables.sql)
Line       : 40
Statement  : comment on column spar.ETL_EXECUTION_MAP.last_run_ts                is 'Last timestamp this interface was executed for batch execution'

build fails

@craigyu
Copy link
Collaborator

craigyu commented Apr 2, 2024

oops, didn't see Derek's message, but yea same issue.

Also could you provide a link to this old repo's issue? @marcionemec-daitan

Refers to issue 35 of old spar-data-sync repo.

@marcionemec-daitan
Copy link
Contributor Author

Old repo issue link: bcgov/nr-spar-data#49

@DerekRoberts
Copy link
Member

Old repo issue link: bcgov/nr-spar-data#49

I just saw that and migrated it here.

bcgov/nr-spar-data#49

@marcionemec-daitan marcionemec-daitan merged commit 2b8f94b into main Apr 3, 2024
17 checks passed
@marcionemec-daitan marcionemec-daitan deleted the feature/marcio/35-create-load-control-tables-for-pipelines-monitoring branch April 3, 2024 16:39
DerekRoberts pushed a commit that referenced this pull request May 13, 2024
* #35 - Create DB Migration for ETL Tool monitoring tables

* #35 Fixing Typo for table comments
DerekRoberts pushed a commit that referenced this pull request May 14, 2024
* #35 - Create DB Migration for ETL Tool monitoring tables

* #35 Fixing Typo for table comments
DerekRoberts pushed a commit that referenced this pull request May 14, 2024
* #35 - Create DB Migration for ETL Tool monitoring tables

* #35 Fixing Typo for table comments
DerekRoberts pushed a commit that referenced this pull request May 14, 2024
* #35 - Create DB Migration for ETL Tool monitoring tables

* #35 Fixing Typo for table comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create Load control tables for Pipelines Monitoring
3 participants