Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update delimited table handling to allow for empty fields #137

Closed
rchenatjpl opened this issue Oct 15, 2019 · 3 comments · Fixed by #138
Closed

Update delimited table handling to allow for empty fields #137

rchenatjpl opened this issue Oct 15, 2019 · 3 comments · Fixed by #138
Labels
bug Something isn't working

Comments

@rchenatjpl
Copy link
Contributor

In the attached one-row .csv file, fields 6,7,9 are totally blank, and field 8 is a few blanks. The first three should definitely not draw an error, since that's the point of using table_delimited vs. a regular table. I opine that field 8 should not be an error, but maybe it should - I haven't thought deeply about it. I guess I would vote for no error since that's what earlier versions of validate did, and maybe nodes have submitted such data.
pit_test_icy_soil_dig1.csv.txt
pit_test_icy_soil_dig1.xml.txt

Separate issue: what's up with the "com.sun.xml.bind..." message - can you suppress that? It's acceptable, just annoying.

Richard

% validate -t pit_test_icy_soil_dig1.xml
Oct 15, 2019 3:22:10 PM com.sun.xml.bind.v2.runtime.reflect.opt.AccessorInjector
INFO: The optimized code generation is disabled
PDS Validate Tool Report
Configuration:
Version 1.17.1
Date 2019-10-15T22:22:07Z
Parameters:
Targets [file:/Users/rchen/Desktop/pit_test_icy_soil_dig1.xml]
Severity Level WARNING
Recurse Directories true
File Filters Used [.xml, .XML]
Data Content Validation on
Product Level Validation on
Max Errors 100000
Registered Contexts File /PDS4tools/validate/resources/registered_context_products.json
Product Level Validation Results
FAIL: file:/Users/rchen/Desktop/pit_test_icy_soil_dig1.xml
Begin Content Validation: file:/Users/rchen/Desktop/pit_test_icy_soil_dig1.csv
ERROR [error.table.field_value_data_type_mismatch] table 1, record 1, field 6: Value does not match its data type 'ASCII_Real': '' does not match the pattern '(+|-)?([0-9]+(.[0-9]
)?|.[0-9]+)(Ee?[0-9]+)?'
ERROR [error.table.field_value_data_type_mismatch] table 1, record 1, field 7: Value does not match its data type 'ASCII_Real': '' does not match the pattern '(+|-)?([0-9]+(.[0-9]
)?|.[0-9]+)(Ee?[0-9]+)?'
ERROR [error.table.field_value_data_type_mismatch] table 1, record 1, field 8: Value does not match its data type 'ASCII_Real': '' does not match the pattern '(+|-)?([0-9]+(.[0-9])?|.[0-9]+)(Ee?[0-9]+)?'
ERROR [error.table.field_value_data_type_mismatch] table 1, record 1, field 9: Value does not match its data type 'ASCII_Real': '' does not match the pattern '(+|-)?([0-9]+(.[0-9]
)?|.[0-9]+)(Ee?[0-9]+)?'
End Content Validation: file:/Users/rchen/Desktop/pit_test_icy_soil_dig1.csv
Validation complete.
Summary:
4 error(s)
0 warning(s)
Message Types:
4 error.table.field_value_data_type_mismatch
End of Report

@rchenatjpl rchenatjpl added bug Something isn't working triage-needed labels Oct 15, 2019
@jordanpadams
Copy link
Member

Most likely relates to #9

jordanpadams added a commit that referenced this issue Oct 16, 2019
Updates made per #9 sufficed for character tables, but caused an issue for delimited tables.

This update allows for blank fields, but still catches issues when a space character is used instead.

According the PDS DSV Standard in the SR (SR 4C.1) "Leading and trailing spaces in a field are considered part of the field" so that is not an issue.

Resolves #137
@jordanpadams jordanpadams changed the title for table_delimited, don't check validity of blank values Update delimited table handling to allow for empty fields Oct 16, 2019
@jordanpadams jordanpadams added this to the PDS.02 (ends 2019-10-23) milestone Oct 16, 2019
@jordanpadams
Copy link
Member

@rchenatjpl the issue with empty fields is fixed per #138.

per your comment regarding allowing space characters, i would disagree. a space character is an actual ASCII character and should be considered as such. the SR DSV standard (4C.1 Delimiter Separated Value Format Description) does say "Leading and trailing spaces in a field are considered part of the field" but it does not explicitly call out fields with only spaces are allowed. my two cents anyways...

@jordanpadams
Copy link
Member

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants