v.db.join: speed up processing by using fewer db.execute commands

[griembauer]

This PR speeds up v.db.join for large tables. Previously, individual v.db.addcolumn and db.execute commands were run for each column to join.
This PR alters v.db.addcolumn and v.db.join such that only one db.execute command is executed for each, using a temporary sql_file instead of stdin as input for the SQL statement.

Previous approach (deprecated):
~~In this PR, one single v.db.addcolumn and db.execute command is executed for a chunk of max 100 columns at once.
Here, all individual v.db.addcolumn commands are grouped into one single command. Additionally, all db.execute UPDATE table... commands to add data to the new columns are grouped into a single command. As the SQL command may become very long this way (and incomplete input in sqlite3_prepare() errors may be encountered, see also #3273), commands with a total string length of > ~10.000 are again split into individual SQL UPDATE commands.~~

[tmszi]

The problem with db.execute input="-" is the statically defined character buffer size if the SQL string is read from stdin input="-".

grass/db/db.execute/main.c

Lines 189 to 191 in ea8f3ea

	int get_stmt(FILE *fd, dbString *stmt)
	{
	char buf[DB_SQL_MAX], buf2[DB_SQL_MAX];

grass/include/grass/dbmi.h

Line 142 in ea8f3ea

#define DB_SQL_MAX 65536

This line truncates the SQL string to 8206 chars.

grass/db/db.execute/main.c

Line 196 in ea8f3ea

if (G_getl2(buf, sizeof(buf), fd) == 0)

Example of long SQL string:

1. Generate long SQL string with 30145 chars (create table) with Python

sql = "CREATE TABLE soils_test (cat integer, soiltype varchar(10)"
for i in range(1200):
    sql += f", soiltype{i} varchar(10)"
sql += ")"

2. Try to execute this SQL string with db.execute input="-" command
3. db.execute command fails because SQL string is truncated and not valid

For this long SQL string, it is better to use read SQL string from the file db.execute input=/tmp/create_table.sql

[griembauer]

The problem with db.execute input="-" is the statically defined character buffer size if the SQL string is read from stdin input="-".

grass/db/db.execute/main.c

Lines 189 to 191 in ea8f3ea

	int get_stmt(FILE *fd, dbString *stmt)
	{
	char buf[DB_SQL_MAX], buf2[DB_SQL_MAX];

grass/include/grass/dbmi.h

Line 142 in ea8f3ea

#define DB_SQL_MAX 65536

This line truncates the SQL string to 8206 chars.

grass/db/db.execute/main.c

Line 196 in ea8f3ea

if (G_getl2(buf, sizeof(buf), fd) == 0)

Example of long SQL string:

1. Generate long SQL string with 30145 chars (create table) with Python

sql = "CREATE TABLE soils_test (cat integer, soiltype varchar(10)"
for i in range(1200):
    sql += f", soiltype{i} varchar(10)"
sql += ")"

2. Try to execute this SQL string with `db.execute input="-"` command

3. `db.execute` command fails because SQL string is truncated and not valid

For this long SQL string, it is better to use read SQL string from the file db.execute input=/tmp/create_table.sql

Thanks! I now updated both v.db.join and v.db.update so that they both use db.executewith a temporary sql file as input instead of stdin. For joining 500 INT columns to a test vector with ~750 objects this brings calculation time down from 110s to 9s on my local machine. However, in the sql_files there are still separate sql statements:

[...]
UPDATE builtup_vectorized_base SET test_column_491 = (SELECT test_column_491 FROM builtup_vectorized_attributes WHERE builtup_vectorized_attributes.cat=builtup_vectorized_base.cat);
UPDATE builtup_vectorized_base SET test_column_492 = (SELECT test_column_492 FROM builtup_vectorized_attributes WHERE builtup_vectorized_attributes.cat=builtup_vectorized_base.cat);
UPDATE builtup_vectorized_base SET test_column_493 = (SELECT test_column_493 FROM builtup_vectorized_attributes WHERE builtup_vectorized_attributes.cat=builtup_vectorized_base.cat);
UPDATE builtup_vectorized_base SET test_column_494 = (SELECT test_column_494 FROM builtup_vectorized_attributes WHERE builtup_vectorized_attributes.cat=builtup_vectorized_base.cat);
[...]

instead of having just one UPDATE statement. I am not sure if this would significantly improve the processing time again, but I couldn't manage to put together a single working UPDATE statement - input is welcome!

[neteler]

Probably a set of SQL commands should be wrapped into a TRANSACTION?

    sql = ["BEGIN TRANSACTION"]
   ...
    sql.append("END TRANSACTION")

Random example:

grass/scripts/v.dissolve/v.dissolve.py

Line 200 in 24365e1

sql = ["BEGIN TRANSACTION"]

[griembauer]

Probably a set of SQL commands should be wrapped into a TRANSACTION?

    sql = ["BEGIN TRANSACTION"]
   ...
    sql.append("END TRANSACTION")

Random example:

grass/scripts/v.dissolve/v.dissolve.py

Line 200 in 24365e1

sql = ["BEGIN TRANSACTION"]

Thanks! Adding this to both v.db.addcolumn and v.db.join seems to double the processing speed

[tmszi]

First I appologize for long delay.

Looks good to me. I tested this fix with adding 500 columns and time was reduced from approximately 1 minute to 0.721s.

[metzm]

Looks good to me, impressive speed-up!