v.in.ogr: skip columns with unsupported data type instead of failing to import #2630

ninsbl · 2022-11-05T19:58:09Z

Before this PR, vector data failed to import if it contained columns in the attribute table that are of a data type that is not supported by GRASS` s DBMI - like binary data (BLOB). This PR changes the behavior of v.in.ogr to drop such columns from import instead of failing to import the data in total.

nilason

Looks good to me. Just had some minor remarks.

vector/v.in.ogr/main.c

metzm · 2022-11-07T21:35:42Z

Instead of adding a special case for Ogr_ftype == OFTBinary, handling of unsupported field types should be fixed in
https://github.com/OSGeo/grass/blob/main/vector/v.in.ogr/main.c#L1177

metzm · 2022-11-07T21:38:43Z

vector/v.in.ogr/main.c

 		Ogr_field = OGR_FD_GetFieldDefn(Ogr_featuredefn, i);
 		Ogr_ftype = OGR_Fld_GetType(Ogr_field);

 		G_debug(3, "Ogr_ftype: %i", Ogr_ftype);	/* look up below */

+            /* skip columns with unsupported data type */
+            if (Ogr_ftype == OFTBinary) {
+                G_warning(_("Column <%s> is omitted, binary data type is not supported."),


Not supported data types are handled below at L1184

Good point! And there all columns with unsupported data types may be omitted. However, that would be a break with present behaviour (keep column, but omit rows). Perhaps a new flag for “omit columns with unsupported data types”?

OK, but isn't the binary column then included in the output table creation struct grass_col_info already in L1091:

grass/vector/v.in.ogr/main.c

Line 1091 in 5d64de2

col_info[i_out].idx = i_out;

So, I guess if we fix it in L1184, it would have to be removed from col_info?
How would I do that? A web-search unfortunately did not give me a straight forward solution...

vector/v.in.ogr/main.c

metzm · 2022-11-08T21:24:24Z

There is a deeper bug in v.in.ogr regarding not supported column types: the behaviour of creating columns and filling columns does not match if there is a not supported column type. See following suggestions to this PR.

vector/v.in.ogr/main.c

metzm · 2022-11-08T21:31:23Z

vector/v.in.ogr/main.c

-		    G_warning(_("Column type (Ogr_ftype: %d) not supported (Ogr_fieldname: %s)"),
-			      Ogr_ftype, Ogr_fieldname);
-		    buf[0] = 0;
-		    col_info[i_out].type = G_store(buf);


Keep the current behaviour, add

i_out--;

remove

col_info[i_out].type = G_store(buf);

metzm · 2022-11-08T21:32:56Z

vector/v.in.ogr/main.c

+                /* handle columns of unsupported data type */
+                G_warning(_("Column type (Ogr_ftype: %d) not supported (Ogr_fieldname: %s)\n"),
+                          Ogr_ftype, Ogr_fieldname);
+                if (flag.drop->answer) {


As above, the drop flag does not make sense: unsupported column types can not be transferred in any reasonable way.

metzm · 2022-11-08T21:43:21Z

When filling attributes, the code block at needs to be removed:

			else {
			    /* column type not supported */
			    G_rasprintf(&sqlbuf, &sqlbufsize, "%c", '\0');
			}

this is the hidden bug: should be ", %c" to create an empty entry for a not supported column type, but dropping any not supported column type avoids confusion later on.

Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com>

ninsbl · 2022-11-08T23:05:02Z

Thanks again. Suggestions implemented in 5cf030a. I hope I got those right...
At least my testcase as well as the unit-test work fine...

ninsbl · 2022-11-08T23:08:31Z

P.S.: I modified the warning message a bit in an attempt to make it a bit more clear what is happening. Happy to change it back if preferred...

nilason · 2022-11-09T11:08:47Z

P.S.: I modified the warning message a bit in an attempt to make it a bit more clear what is happening. Happy to change it back if preferred...

With latest changes, may I suggest the following warning message:

                G_warning(_("Column <%s> is of unsupported \"%s\" data type, "
                            "it is omitted from import."),
		          Ogr_fieldname, OGR_GetFieldTypeName(Ogr_ftype));

which will result in a message like:

WARNING: Column <blob> is of unsupported "Binary" data type, it is omitted
         from import.

General note: G_warning will automatically reflow the printed message to fit in a 80 col terminal, no need to break \n (not at the end either).

vector/v.in.ogr/main.c

wenzeslaus · 2022-11-09T21:20:08Z

@ninsbl Can you please update the PR description to reflect the current intention and changes done? (will be needed for the merge/final commit anyway)

ninsbl · 2022-11-09T21:38:58Z

@ninsbl Can you please update the PR description to reflect the current intention and changes done? (will be needed for the merge/final commit anyway)

Done.

nilason

Great!

vector/v.in.ogr/main.c

wenzeslaus

That looks good! I'm wondering if the line is not too long, but formatting in this file needs fixing anyway, so maybe not a big deal as long as we are planning to fix it soon.

…to import (#2630) * skip binary columns * remove copied comment * indent + warning message * handle all unsupported column types with d-flag * remove second warning * Update vector/v.in.ogr/main.c Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com> * implement suggestion from review * cleanup from review * simplify warning Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com>

…to import (OSGeo#2630) * skip binary columns * remove copied comment * indent + warning message * handle all unsupported column types with d-flag * remove second warning * Update vector/v.in.ogr/main.c Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com> * implement suggestion from review * cleanup from review * simplify warning Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com>

ninsbl added 2 commits November 5, 2022 20:36

skip binary columns

8d5ba2f

remove copied comment

0b80747

ninsbl added bug Something isn't working backport_needed C Related code is in C database Related to database management labels Nov 5, 2022

ninsbl added this to the 8.3.0 milestone Nov 5, 2022

ninsbl requested review from nilason and metzm November 5, 2022 19:58

ninsbl linked an issue Nov 5, 2022 that may be closed by this pull request

[Bug] v.in.ogr: handle columns with unsupported datatype in GRASS DBMI #2628

Closed

nilason requested changes Nov 7, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

nilason mentioned this pull request Nov 7, 2022

[Bug] Attribute Table Manager fails with TEXT data column containing newline #2635

Open

indent + warning message

6bfaba3

ninsbl requested a review from nilason November 7, 2022 20:54

metzm reviewed Nov 7, 2022

View reviewed changes

handle all unsupported column types with d-flag

47b4efd

wenzeslaus requested changes Nov 8, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

remove second warning

d8c62e0

metzm reviewed Nov 8, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

metzm reviewed Nov 8, 2022

View reviewed changes

ninsbl and others added 2 commits November 8, 2022 23:30

Update vector/v.in.ogr/main.c

0d087b6

Co-authored-by: Markus Metz <33666869+metzm@users.noreply.github.com>

implement suggestion from review

5cf030a

ninsbl requested review from wenzeslaus and metzm November 8, 2022 23:05

ninsbl changed the title ~~v.in.ogr: skip binary columns (instead of failing to import the data)~~ v.in.ogr: skip columns with unsupported data type instead of failing to import Nov 9, 2022

nilason reviewed Nov 9, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

nilason reviewed Nov 9, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

nilason reviewed Nov 9, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

cleanup from review

cd67af7

ninsbl requested a review from nilason November 9, 2022 21:17

nilason approved these changes Nov 9, 2022

View reviewed changes

wenzeslaus reviewed Nov 10, 2022

View reviewed changes

vector/v.in.ogr/main.c Outdated Show resolved Hide resolved

simplify warning

ba86d84

ninsbl requested a review from wenzeslaus November 10, 2022 07:41

wenzeslaus approved these changes Nov 10, 2022

View reviewed changes

ninsbl merged commit 726fb2c into OSGeo:main Nov 10, 2022

ninsbl removed the backport_needed label Nov 10, 2022

This was referenced Nov 17, 2022

[Bug] v.in.ogr: handle columns with unsupported datatype in GRASS DBMI #2628

Closed

v.in.ogr: handle skipped, not empty columns #2658

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v.in.ogr: skip columns with unsupported data type instead of failing to import #2630

v.in.ogr: skip columns with unsupported data type instead of failing to import #2630

ninsbl commented Nov 5, 2022 •

edited

Loading

nilason left a comment

metzm commented Nov 7, 2022

metzm Nov 7, 2022

nilason Nov 7, 2022

ninsbl Nov 7, 2022

metzm commented Nov 8, 2022

metzm Nov 8, 2022

metzm Nov 8, 2022

metzm commented Nov 8, 2022

ninsbl commented Nov 8, 2022

ninsbl commented Nov 8, 2022

nilason commented Nov 9, 2022

wenzeslaus commented Nov 9, 2022

ninsbl commented Nov 9, 2022

nilason left a comment

wenzeslaus left a comment

v.in.ogr: skip columns with unsupported data type instead of failing to import #2630

v.in.ogr: skip columns with unsupported data type instead of failing to import #2630

Conversation

ninsbl commented Nov 5, 2022 • edited Loading

nilason left a comment

Choose a reason for hiding this comment

metzm commented Nov 7, 2022

metzm Nov 7, 2022

Choose a reason for hiding this comment

nilason Nov 7, 2022

Choose a reason for hiding this comment

ninsbl Nov 7, 2022

Choose a reason for hiding this comment

metzm commented Nov 8, 2022

metzm Nov 8, 2022

Choose a reason for hiding this comment

metzm Nov 8, 2022

Choose a reason for hiding this comment

metzm commented Nov 8, 2022

ninsbl commented Nov 8, 2022

ninsbl commented Nov 8, 2022

nilason commented Nov 9, 2022

wenzeslaus commented Nov 9, 2022

ninsbl commented Nov 9, 2022

nilason left a comment

Choose a reason for hiding this comment

wenzeslaus left a comment

Choose a reason for hiding this comment

ninsbl commented Nov 5, 2022 •

edited

Loading