Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update build boundaries #418

Open
wants to merge 7 commits into
base: master
Choose a base branch
from
Open

Update build boundaries #418

wants to merge 7 commits into from

Conversation

lacardonap
Copy link
Contributor

@lacardonap lacardonap commented Feb 19, 2021

Actualización del algoritmo que permite construir los linderos

  • Implementar modelo de procesamiento
  • Actualizar tests
  • Soporte para linderos seleccionados
  • Remover dependencia de los linderos con las tablas de topolofía almacenada (puntosccl, masccl y menosccl)
  • Actualizar capa de linderos
  • Resolver inquietudes.
  • Agregar tests para comprobar llenado de tablas topológicas (con y sin selección)
  • Corregir ETL model_build_boundaries.model3
  • Agregar barra de progreso para la toolbar (construir linderos y llenar tablas de topología) Resuelve issue Comunicar estado en la herramienta construir linderos #333

add_progress_bar

@lacardonap lacardonap changed the base branch from master to lev_cat_1_1 February 19, 2021 16:12
@lacardonap
Copy link
Contributor Author

Se esta trabajando en la implementación del nuevo algoritmo de construcción de linderos, en un principio se desea conservar el comportamiento que tenia el algoritmo existente en el cual se le permitia al usuario realizar una selección de linderos antes de proceder a construir los linderos.

Realizar la selección de linderos tiene la ventaja de que si el conjunto de datos es muy grande no es necesario correr todo el algoritmo para toda la base de datos sino unicamente para los linderos seleccionados. Parece una buena estragería, el problema que actualmente existe al adoptarla es que al no seleccionar todos los linderos que estan conectados el resultado obtenido no es el esperado.

Por ejemplo, se puede apreciar en el siguiente gif que al generar los linderos para la zona seleccionada el total de linderos es 58602. Pero si lo genero para toda la base de datos el total de linderos es 58613.

linderos_seleccionados

A partir de los linderos seleccionados por el usuario se realiza una selección por localización y se seleccionan los linderos que estan conectados pero puede ocurrir lo siguiente: los linderos seleccionados no incluyen todos los linderos vecinos razón por la cual la construcción no es la correcta.

construccion_linderos

@lacardonap
Copy link
Contributor Author

Actualmente si doy clic en el botón de construir linderos y la capa de linderos no esta cargada se desplega un mensaje en el cual se le dice a la persona que primero se debe cargar la capa de linderos. Propongo que este mensaje no se muestre al usuario y se cargue automáticamente la capa
pregunta_cargar_linderos

Copy link
Member

@gacarrillor gacarrillor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we rebase this PR (on lev_cat_1_1) to make it ready to be reviewed/merged?
That would also show us the unit tests status.

@lacardonap lacardonap self-assigned this May 25, 2021
@lacardonap
Copy link
Contributor Author

lacardonap commented May 25, 2021

Procedere con la actualización 👍

@lacardonap
Copy link
Contributor Author

lacardonap commented May 26, 2021

This PR was rebase using lev_cat_1_1. Now this one is ready to be reviewed/merged.

@gacarrillor gacarrillor added the Under Review Don't add new commits, this is under review. label Jun 2, 2021
Copy link
Member

@gacarrillor gacarrillor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We've never used [feat] in our commits. For your information, inside the square brackets we are get used to add the module name.

asistente_ladm_col/gui/toolbar.py Outdated Show resolved Hide resolved
asistente_ladm_col/gui/toolbar.py Outdated Show resolved Hide resolved
asistente_ladm_col/gui/toolbar.py Outdated Show resolved Hide resolved
asistente_ladm_col/gui/toolbar.py Outdated Show resolved Hide resolved
asistente_ladm_col/gui/toolbar.py Outdated Show resolved Hide resolved
asistente_ladm_col/gui/toolbar.py Outdated Show resolved Hide resolved
asistente_ladm_col/lib/geometry.py Outdated Show resolved Hide resolved
asistente_ladm_col/utils/decorators.py Outdated Show resolved Hide resolved
asistente_ladm_col/utils/decorators.py Outdated Show resolved Hide resolved
asistente_ladm_col/utils/decorators.py Outdated Show resolved Hide resolved
gacarrillor added a commit that referenced this pull request Jun 2, 2021
gacarrillor added a commit that referenced this pull request Jun 2, 2021
@gacarrillor
Copy link
Member

gacarrillor commented Jun 2, 2021

I'm getting errors on GNU/Linux, using QGIS v3.16.5 and master:

The result of the Build Boundaries model is a layer which has its field names truncated, like in the image below:

nombres_recortados

Because of that, the ETL cannot run well, and I get errors like the following in the console:

../src/core/qgsmessagelog.cpp:29 : (logMessage) [4ms] 2021-06-02T16:56:21 Messages[2] Layer Lindero : OGR error creating feature 40: failed to execute insert : NOT NULL constraint failed: lc_lindero.comienzo_vida_util_version
ERROR 1: failed to execute insert : NOT NULL constraint failed: lc_lindero.comienzo_vida_util_version
../src/core/vector/qgsvectordataprovider.cpp:756 : (pushError) [0ms] OGR error creating feature 41: failed to execute insert : NOT NULL constraint failed: lc_lindero.comienzo_vida_util_version

We cannot merge until we know what's happening and fix it.

This should be done on top of the latest commit (7830fec). Testing on your latest commit (392b326) gave me the same results.

@gacarrillor
Copy link
Member

Apart from that, it would be required to have a test for the toolbar's method called build_boundary(). The only test added by this PR is testing the model, but we also need to test all the logic involved in build_boundary(), that is:

  1. Are topology tables finally updated?
  2. Is the boundary layer updated (number of features should be the same before and after the process)?

Ideally the test should act in two modes: using a selection and using all the features.

@gacarrillor gacarrillor added Not ready for a Merge This PR has errors or pending tasks that must be addressed and removed Under Review Don't add new commits, this is under review. labels Jun 2, 2021
@lacardonap
Copy link
Contributor Author

Thank you very much for the review and adjustments made. I will proceed to review the observations. I would like to confirm if the dataset with which the error was generated corresponds to the structured data for the tutorial?

@gacarrillor
Copy link
Member

I would like to confirm if the dataset with which the error was generated corresponds to the structured data for the tutorial?

Yes, but migrated to Levantamiento Catastral v1.1.

@lacardonap
Copy link
Contributor Author

I'm getting errors on GNU/Linux, using QGIS v3.16.5 and master:

Build_Boundaries model was updated to prevent truncated data.
It can be tested in the commit ea0f7c4

@lacardonap
Copy link
Contributor Author

I have designed the test to validate the entire build_boundaries function. But I am having a problem trying to run the test because the iface object is null.

Do you think it is a good idea to validate the whole build_boundaries function?

https://gist.github.com/lacardonap/157b1aa7f36b0e178758d2f00a3e75e5

I am getting this error when running the test:

======================================================================
ERROR: test_build_boundaries (asistente_ladm_col.tests.test_toolbar.TestToolbar)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 359, in startPlugin
    plugins[packageName].initGui()
  File "/usr/share/qgis/python/plugins/processing/ProcessingPlugin.py", line 181, in initGui
    iface.registerOptionsWidgetFactory(self.options_factory)
AttributeError: 'NoneType' object has no attribute 'registerOptionsWidgetFactory'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/asistente_ladm_col/tests/test_toolbar.py", line 64, in test_build_boundaries
    self.toolbar.build_boundary(self.db_gpkg)
  File "/usr/src/asistente_ladm_col/gui/toolbar.py", line 121, in build_boundary
    self.app.core.run_etl_model_in_backgroud_mode(db, build_boundaries_layer, db.names.LC_BOUNDARY_T)
  File "/usr/src/asistente_ladm_col/utils/decorators.py", line 209, in decorated_function
    startPlugin('processing')
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 362, in startPlugin
    _unloadPluginModules(packageName)
  File "/usr/lib/python3/dist-packages/qgis/utils.py", line 448, in _unloadPluginModules
    mods = _plugin_modules[packageName]
KeyError: 'processing'

Any advice is welcome. Thanks

@gacarrillor
Copy link
Member

Do you think it is a good idea to validate the whole build_boundaries function?

Do you think it's a good idea to delete features from the user layer and do not try to recover those deleted features if any failure occurs while running the model?


Could you commit your test to another branch (created from update_build_boundaries), so that I can inspect it?

@lacardonap
Copy link
Contributor Author

build_boundaries function must be tested, what I meant is if we test the whole function or check it by parts. After thinking about it, it is better to check it all.

I appreciate the support in reviewing the test which was left in the branch https://github.com/SwissTierrasColombia/Asistente-LADM-COL/tree/test_toolbar_build_boundaries, which never ever worked for me locally. I am attentive to what is required.

gacarrillor added a commit that referenced this pull request Jul 8, 2021
Copy link
Member

@gacarrillor gacarrillor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test test_build_boundaries does not fail anymore due to processing plugin. But it's throwing other errors now, that I'll pass for you to adjust.

Additionally, the toolbar.build_boundary() method should be adjusted:

@lacardonap
Copy link
Contributor Author

Thank you for your support. I continue with the requested adjustments.

@lacardonap
Copy link
Contributor Author

lacardonap commented Jul 12, 2021

The requested adjustments are made. Please continue with the review. I stay tuned for what is required

@@ -118,7 +118,7 @@ def build_boundary(self, db):
layers[db.names.LC_BOUNDARY_T].deleteSelectedFeatures()

# Bring back the features we deleted before, but this time, with the boundaries fixed
self.app.core.run_etl_model_in_backgroud_mode(db, build_boundaries_layer, db.names.LC_BOUNDARY_T)
self.app.core.run_etl_model_in_backgroud_mode(db, build_boundaries_layer, db.names.LC_BOUNDARY_T, False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If something has to change, it has to be the call from the tests, not the one from core...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for the feedback. The reply is late but I prefer to respond.
You are right, the suggested adjustment was made and currently in the commit 513f31a

# Bring back the features we deleted before, but this time, with the boundaries fixed
self.app.core.run_etl_model_in_backgroud_mode(db, build_boundaries_layer, db.names.LC_BOUNDARY_T, False)
# Build boundaries should have generated at least one boundary.
if selected_boundaries_count > 0 and build_boundaries_count > 0:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And what if not? Any message for users?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a

@@ -1152,7 +1152,7 @@ def copy_csv_to_db(self, csv_layer, db, target_layer_name):
return True

@_activate_processing_plugin
def run_etl_model_in_backgroud_mode(self, db, input_layer, ladm_col_layer_name):
def run_etl_model_in_backgroud_mode(self, db, input_layer, ladm_col_layer_name, force_reprojection=True):
Copy link
Member

@gacarrillor gacarrillor Jul 12, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't get it. If you need to adjust something here is to set the proper CRS to your layer before calling this method. The method run_etl_model_in_backgroud_mode can remain without changes.

Is there any problem trying to set a proper CRS to your layer?
You should use get_ctm12_crs().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the suggested adjustment was made and currently in the commit 513f31a

layers[db.names.LC_BOUNDARY_T].dataProvider().truncate()

# Original data is restored because an error occurred while trying to build the boundaries
self.app.core.run_etl_model_in_backgroud_mode(db, copy_boundary_layer, db.names.LC_BOUNDARY_T, False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just restore selected features, no need to do it for the whole layer...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wooow thanks for the comment, you are absolutely right, adjustment made in the commit 513f31a

with edit(layers[db.names.LC_BOUNDARY_T]):
# Delete selected features as they will be imported again from a newly created layer after processed
layers[db.names.LC_BOUNDARY_T].deleteSelectedFeatures()
copy_boundary_layer = self.app.core.get_layer_copy(layers[db.names.LC_BOUNDARY_T])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I disagree. Imagine a layer of 1000 boundaries and you select 4 of them before clicking on "Build Boundaries"...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a

# Bring back the features we deleted before, but this time, with the boundaries fixed
self.app.core.run_etl_model_in_backgroud_mode(db, build_boundaries_layer, db.names.LC_BOUNDARY_T, False)

# check if features were insert successful
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

successfully

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a


else:
# Clean layer because wrong data could be insert previously
layers[db.names.LC_BOUNDARY_T].dataProvider().truncate()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're removing all features?

What if the user just selected 1% of them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the change was made, now only the selected boundaries are deleted and if an error occurs they are restored. Available in commit 513f31a

"{} feature(s) was(were) analyzed generating {} boundary(ies)!").format(num_boundaries, build_boundaries_layer.featureCount()))

else:
# Clean layer because wrong data could be insert previously
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could have been inserted previously

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a

'Features count does not match for the layer {}'.format(layer_name))

def test_build_boundaries_with_empty_geom(self):
print('\nINFO: Validating build boundaries with features with null geometry...')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unclear what you're testing.

What is the result you expect? The NULL geometry is ignored?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the information given in the tests has been updated. This test validates that if there is a boundary without geometry it must be removed. vailable in commit 513f31a

'Null features count does not match for the layer boundary layer')

boundary_layer.selectAll()
self.toolbar.build_boundary(db_gpkg)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method build_boundaries should be called like this:

build_boundary(db_gpkg, False)  # where False is do not use selection

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a

@gacarrillor
Copy link
Member

We are wasting too much time in this review. Please double (and triple) check before submitting to a review.

Copy link
Member

@gacarrillor gacarrillor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • The test is not passing.
  • To keep the commit history clean, squash all commits since 34a5e0f on.

@@ -45,7 +45,14 @@ def __init__(self, iface):
self.app = AppInterface()
self.geometry = GeometryUtils()

def build_boundary(self, db):
def build_boundary(self, db, use_user_selection=True):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • False by default, since the expected and recommended way to run this method is without a selection.
  • Change use_user_selection by use_selection like in other methods in the same class and like in GeometryUtils().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Change use_user_selection by use_selection like in other methods in the same class and like in GeometryUtils().

The build_boundary method is designed to display a dialog box to the user indicating that no selection has been made and asking if the user wants to use all boundaries. When there are selected boundaries, no dialog box is shown and the selected boundaries are used. The purpose of using the parameter use_user_selection in the function is to omit the user selection and use all the records of the layer.

It is proposed to change use_user_selection to skip_selection where:

skip_selection (Boolean): True if we omit the boundaries selected by the user, False if we validate the user's selection.

@@ -58,6 +65,9 @@ def build_boundary(self, db):
}
self.app.core.get_layers(db, layers, load=True)

if not use_user_selection:
layers[db.names.LC_BOUNDARY_T].selectAll()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem to make sense.

If use_selection is False, you shouldn't alter selected layer features nor use them in any way. Instead you should use always all layer features (e.g., when creating a copy, when counting features, etc.).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, the suggested adjustment was made and currently in the commit 513f31a

layers[db.names.LC_BOUNDARY_T].dataProvider().truncate()
if layers[db.names.LC_BOUNDARY_T].featureCount() != boundaries_count - selected_boundaries_count:
# Clean layer because wrong data could have been inserted previously
expr = "{} IN ('{}')".format(db.names.T_ILI_TID_F, "','".join([str(t_ili_tid) for t_ili_tid in boundary_t_ili_tids]))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if you actually need str(t_ili_tid), since t_ili_tids are already strings.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

data type of t_ili_tid is string, no conversion is necessary. Adjustment made, available in commit 513f31a

# Clean layer because wrong data could have been inserted previously
expr = "{} IN ('{}')".format(db.names.T_ILI_TID_F, "','".join([str(t_ili_tid) for t_ili_tid in boundary_t_ili_tids]))
layers[db.names.LC_BOUNDARY_T].selectByExpression(expr)
layers[db.names.LC_BOUNDARY_T].invertSelection()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you need to select and then invert the selection, then the expression is wrong.

Perhaps you need expr = "{} NOT IN ('{}')"..., don't you?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the observation, the selection is much more optimal. Adjustment made, available in commit 513f31a


# Original data is restored because an error occurred while trying to build the boundaries
self.app.core.run_etl_model_in_backgroud_mode(db, copy_boundary_layer, db.names.LC_BOUNDARY_T, False)
# the previously deleted boundaries are restored because an error occurred when trying to insert the building boundaries
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any chance to write a unit test for that case?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am trying to generate a data set that will allow us to test this part of the cosw. It has not been possible, but I would also like to have a test that checks this.

return processing.run("model:Build_Boundaries", params)['native:refactorfields_2:built_boundaries']
build_boundary_layer = processing.run("model:Build_Boundaries", params)['native:refactorfields_2:built_boundaries']

# For CTM12 the output layer remains without projection. The projection of the input layer is assigned
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it necessary?

Wasn't it only an issue for unit tests?

If you really need to set the CRS here, just use layer.setCrs().

@@ -105,8 +105,7 @@ def test_build_boundaries_with_empty_geom(self):
self.assertEqual(boundary_layer.selectedFeatureCount(), 1,
'Null features count does not match for the layer boundary layer')

boundary_layer.selectAll()
self.toolbar.build_boundary(db_gpkg)
self.toolbar.build_boundary(db_gpkg, False)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This call should be simply self.toolbar.build_boundary(db_gpkg).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a

@@ -45,7 +45,14 @@ def __init__(self, iface):
self.app = AppInterface()
self.geometry = GeometryUtils()

def build_boundary(self, db):
def build_boundary(self, db, use_user_selection=True):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can rename the method to build_boundaries(), since that's what the method does.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adjustment made, available in commit 513f31a

@@ -22,6 +22,7 @@ def setUpClass(cls):
import_qgis_model_baker()
import_asistente_ladm_col() # Import plugin
cls.app = AppInterface()
cls.app.core.initialize_ctm12() # We need to initialize CTM12
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if you initialize CTM12 after this line instead (and remove any other call to such method in unit tests)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What a good idea, adjustment made. initialize CTRM12 from other classes were removed. Available in commit 513f31a

@lacardonap
Copy link
Contributor Author

@gacarrillor thank you very much for the review, I am checking why if I run only the toolbar tests it works correctly but if I run all the tests it fails.

Screenshot from 2021-07-21 14-52-51

Screenshot from 2021-07-21 14-49-54

I will proceed with the requested adjustments

@lacardonap
Copy link
Contributor Author

The problem was solved by moving the test test_build_boundaries from the test_geometry_utils class to test_build_boundaries (Note: the test_toolbar class was renamed).

@lacardonap
Copy link
Contributor Author

I am currently trying to generate test data to extend the test coverage. The last commit will correspond to the squash all commits since 34a5e0f on. Again, thanks for the review.

@lacardonap
Copy link
Contributor Author

lacardonap commented Jul 27, 2021

In trying to generate a dataset that would allow to extend the coverage of the tests, some improvements were identified and implemented and the corresponding tests were developed. The suggested adjustments are already implemented but to date I have not been able to generate a test data set that fails (the grass algorithm is very good).

You can now proceed with the revision. Thanks

Base automatically changed from lev_cat_1_1 to master February 27, 2022 01:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Not ready for a Merge This PR has errors or pending tasks that must be addressed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants