ISSUE-82 Address ingestion issue(s) with eTUFF-tuna590051-debug.txt

[lewismc]

WIP to address #82
Ingestion of same file now results in following error

tagbase_server_1  | ERROR:tornado.access:500 GET /v0.5.0/ingest?file=file%3A%2F%2F%2Fusr%2Fsrc%2Fapp%2Fdata%2Ficcat_gbyp0001_eTUFF.txt&type=etuff (192.168.160.1) 1967.94ms
postgres_1        | 2022-05-09 00:23:59.372 UTC [61] ERROR:  null value in column "variable_id" of relation "data_histogram_bin_data" violates not-null constraint
postgres_1        | 2022-05-09 00:23:59.372 UTC [61] DETAIL:  Failing row contains (2, 2, 2, 1, 2005-04-15 21:00:00+00, 1.00, null, null).
postgres_1        | 2022-05-09 00:23:59.372 UTC [61] CONTEXT:  SQL statement "WITH moved_rows AS
postgres_1        | 	      ( DELETE
postgres_1        | 	       FROM proc_observations a USING observation_types b,
postgres_1        | 	                                      submission c
postgres_1        | 	       WHERE a.variable_id = b.variable_id
postgres_1        | 	         AND b.variable_name LIKE 'TimeAt%'
postgres_1        | 	         AND a.submission_id = c.submission_id RETURNING a.submission_id,
postgres_1        | 	                                                          c.tag_id,
postgres_1        | 	                                                          a.submission_id AS bin_id,
postgres_1        | 	                                                          cast(substring(variable_name, '(\d+)') AS int) AS bin_class,
postgres_1        | 	                                                          a.date_time,
postgres_1        | 	                                                          variable_value)
postgres_1        | 	    INSERT INTO data_histogram_bin_data
postgres_1        | 	    SELECT *
postgres_1        | 	    FROM moved_rows"
postgres_1        | 	PL/pgSQL function execute_data_migration() line 119 at SQL statement
postgres_1        | 2022-05-09 00:23:59.372 UTC [61] STATEMENT:  INSERT INTO proc_observations (date_time, variable_id, variable_value, submission_id, tag_id, final_value) VALUES ('2005-04-16T23:59:00+00:00'::timestamptz, 8, '145.00', 2, '2', 'TRUE')
tagbase_server_1  | ERROR:__main__:Exception on /v0.5.0/ingest [GET]
tagbase_server_1  | Traceback (most recent call last):
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 2077, in wsgi_app
tagbase_server_1  |     response = self.full_dispatch_request()
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1525, in full_dispatch_request
tagbase_server_1  |     rv = self.handle_user_exception(e)
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1523, in full_dispatch_request
tagbase_server_1  |     rv = self.dispatch_request()
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/flask/app.py", line 1509, in dispatch_request
tagbase_server_1  |     return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/decorator.py", line 68, in wrapper
tagbase_server_1  |     response = function(request)
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/uri_parsing.py", line 149, in wrapper
tagbase_server_1  |     response = function(request)
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/validation.py", line 399, in wrapper
tagbase_server_1  |     return function(request)
tagbase_server_1  |   File "/usr/local/lib/python3.10/site-packages/connexion/decorators/parameter.py", line 120, in wrapper
tagbase_server_1  |     return function(**kwargs)
tagbase_server_1  |   File "/usr/src/app/tagbase_server/controllers/ingest_controller.py", line 195, in ingest_get
tagbase_server_1  |     cur.execute(
tagbase_server_1  | psycopg2.errors.NotNullViolation: null value in column "variable_id" of relation "data_histogram_bin_data" violates not-null constraint
tagbase_server_1  | DETAIL:  Failing row contains (2, 2, 2, 1, 2005-04-15 21:00:00+00, 1.00, null, null).
tagbase_server_1  | CONTEXT:  SQL statement "WITH moved_rows AS
tagbase_server_1  |       ( DELETE
tagbase_server_1  |        FROM proc_observations a USING observation_types b,
tagbase_server_1  |                                       submission c
tagbase_server_1  |        WHERE a.variable_id = b.variable_id
tagbase_server_1  |          AND b.variable_name LIKE 'TimeAt%'
tagbase_server_1  |          AND a.submission_id = c.submission_id RETURNING a.submission_id,
tagbase_server_1  |                                                           c.tag_id,
tagbase_server_1  |                                                           a.submission_id AS bin_id,
tagbase_server_1  |                                                           cast(substring(variable_name, '(\d+)') AS int) AS bin_class,
tagbase_server_1  |                                                           a.date_time,
tagbase_server_1  |                                                           variable_value)
tagbase_server_1  |     INSERT INTO data_histogram_bin_data
tagbase_server_1  |     SELECT *
tagbase_server_1  |     FROM moved_rows"
tagbase_server_1  | PL/pgSQL function execute_data_migration() line 119 at SQL statement
tagbase_server_1  |
tagbase_server_1  | ERROR:tornado.access:500 GET /v0.5.0/ingest?file=file%3A%2F%2F%2Fusr%2Fsrc%2Fapp%2Fdata%2FeTUFF-tuna590051-debug.txt&type=etuff (192.168.160.1) 4210.37ms

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
9 Code Smells

48.7% Coverage
0.0% Duplication