Skip to content

Commit

Permalink
Improve: qdt files reliability (#514)
Browse files Browse the repository at this point in the history 
This PR improve qdt-files.json (HTTP handler) parsing reliability
handling cases where `qdt-files.json` generated with tree without the
`--prune` option and containing empty folders makes crash the reader.

```python
2024-05-29 09:50:42||ERROR||bouncer||exit_cli_error||43||'NoneType' object is not iterable
Traceback (most recent call last):
  File "qgis_deployment_toolbelt/commands/deployment.py", line 205, in run
  File "qgis_deployment_toolbelt/jobs/job_profiles_downloader.py", line 130, in run
  File "qgis_deployment_toolbelt/profiles/remote_http_handler.py", line 120, in download
  File "qgis_deployment_toolbelt/profiles/remote_http_handler.py", line 158, in tree_to_download_list
  File "qgis_deployment_toolbelt/profiles/remote_http_handler.py", line 158, in tree_to_download_list
  File "qgis_deployment_toolbelt/profiles/remote_http_handler.py", line 150, in tree_to_download_list
TypeError: 'NoneType' object is not iterable
2024-05-29 09:50:42||ERROR||bouncer||exit_cli_error||44||Please, read the full detailed log: /home/florian.bocquet/.cache/qgis-deployment-toolbelt/logs/QGISDeploymentToolbelt_0.34.2.log
'NoneType' object is not iterable
```

Spotted by @Niarolf
  • Loading branch information
@Guts
Guts committed May 30, 2024
2 parents e456316 + 00faae8 commit 53b5d44
Show file tree
Hide file tree
Showing 9 changed files with 13,173 additions and 102 deletions.
17 changes: 12 additions & 5 deletions docs/guides/howto_publish_http.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,27 @@

## Generate the `qdt-files.json` index file

> Typically on Ubuntu
> Typically on Ubuntu 22.04
Install tree:
Install tree > 2:

```sh
sudo apt install tree
```

Check version:

```sh
tree --version
```

Run it:

```sh
# move to your QDT profiles folder. Here we take the QDT repository as example:
cd examples/
# generate the qdt-files.json
tree --gitignore -D --timefmt="%Y-%m-%dT%H:%M:%S%Z" -s -J -o qdt-files.json .
tree --gitignore -D -J --prune -s --timefmt="%Y-%m-%dT%H:%M:%S%Z" -o qdt-files.json .
```

Detailed explanation:
Expand All @@ -25,8 +31,9 @@ Detailed explanation:
<!-- - `-f`: display the full path for each file and directory. -->
- `--gitignore`: apply gitignore-style rules to exclude files and directories.
- `-D`: print the modification time for each file or directory.
- `--timefmt="%Y-%m-%dT%H:%M:%S%Z"`: specify the time format as ISO8601 with UTC (Coordinated Universal Time).
- `-s`: print the size of each file.
- `-J`: output the directory tree in JSON format.
- `--prune`: do not include empty folders
- `-s`: print the size of each file.
- `--timefmt="%Y-%m-%dT%H:%M:%S%Z"`: specify the time format as ISO8601 with UTC (Coordinated Universal Time).
- `-o qdt-files.json`: save the output to a file named 'qdt-files.json'.
- `.`: specify the current directory as the starting point for the tree.
237 changes: 197 additions & 40 deletions examples/qdt-files.json
View file
Original file line number Diff line number Diff line change
@@ -1,42 +1,199 @@
[
{"type":"directory","name":".","size":4096,"time":"2023-12-29T11:03:36CET","contents":[
{"type":"directory","name":"profiles","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"directory","name":"demo","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"file","name":"bookmarks.xml","size":1582,"time":"2023-11-14T17:43:43CET"},
{"type":"directory","name":"images","size":4096,"time":"2023-06-13T17:02:08CEST","contents":[
{"type":"file","name":"logo_qdt.ico","size":227102,"time":"2023-06-13T17:02:08CEST"},
{"type":"file","name":"splash.png","size":354451,"time":"2023-06-13T17:02:08CEST"}
]},
{"type":"file","name":"profile.json","size":594,"time":"2023-06-13T17:02:08CEST"},
{"type":"directory","name":"QGIS","size":4096,"time":"2023-11-15T11:05:26CET","contents":[
{"type":"file","name":"QGIS3.ini","size":10605,"time":"2023-11-15T11:05:26CET"},
{"type":"file","name":"QGISCUSTOMIZATION3.ini","size":144530,"time":"2023-11-14T08:47:47CET"}
]}
]},
{"type":"file","name":"profiles.ini","size":34,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"Viewer Mode","size":4096,"time":"2023-12-29T10:13:27CET","contents":[
{"type":"file","name":"bookmarks.xml","size":1582,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"images","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"file","name":"logo_qdt.ico","size":227102,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"splash.png","size":211808,"time":"2023-12-22T16:14:45CET"}
]},
{"type":"file","name":"profile.json","size":1481,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"project_default_attachments.zip","size":1125,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"project_default.qgs","size":80961,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"QGIS","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"file","name":"QGIS3.ini","size":119124,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"QGISCUSTOMIZATION3.ini","size":144627,"time":"2023-12-22T16:14:45CET"}
]},
{"type":"file","name":"startup_project.qgz","size":13321,"time":"2023-12-22T16:14:45CET"}
]}
]},
{"type":"file","name":"qdt-files.json","size":0,"time":"2023-12-29T11:05:41CET"},
{"type":"file","name":"README.md","size":1139,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"scenarios","size":4096,"time":"2023-12-29T10:09:25CET","contents":[
{"type":"file","name":"demo-scenario-http.qdt.yml","size":1441,"time":"2023-12-29T11:04:38CET"},
{"type":"file","name":"demo-scenario.qdt.yml","size":1540,"time":"2023-12-22T20:07:44CET"}
]}
]}
,
{"type":"report","directories":8,"files":20}
{
"type": "directory",
"name": ".",
"size": 4096,
"time": "2023-12-29T11:03:36CET",
"contents": [
{
"type": "directory",
"name": "profiles",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "directory",
"name": "demo",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "file",
"name": "bookmarks.xml",
"size": 1582,
"time": "2023-11-14T17:43:43CET"
},
{
"type": "directory",
"name": "images",
"size": 4096,
"time": "2023-06-13T17:02:08CEST",
"contents": [
{
"type": "file",
"name": "logo_qdt.ico",
"size": 227102,
"time": "2023-06-13T17:02:08CEST"
},
{
"type": "file",
"name": "splash.png",
"size": 354451,
"time": "2023-06-13T17:02:08CEST"
}
]
},
{
"type": "file",
"name": "profile.json",
"size": 594,
"time": "2023-06-13T17:02:08CEST"
},
{
"type": "directory",
"name": "QGIS",
"size": 4096,
"time": "2023-11-15T11:05:26CET",
"contents": [
{
"type": "file",
"name": "QGIS3.ini",
"size": 10605,
"time": "2023-11-15T11:05:26CET"
},
{
"type": "file",
"name": "QGISCUSTOMIZATION3.ini",
"size": 144530,
"time": "2023-11-14T08:47:47CET"
}
]
}
]
},
{
"type": "file",
"name": "profiles.ini",
"size": 34,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "Viewer Mode",
"size": 4096,
"time": "2023-12-29T10:13:27CET",
"contents": [
{
"type": "file",
"name": "bookmarks.xml",
"size": 1582,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "images",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "file",
"name": "logo_qdt.ico",
"size": 227102,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "splash.png",
"size": 211808,
"time": "2023-12-22T16:14:45CET"
}
]
},
{
"type": "file",
"name": "profile.json",
"size": 1481,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "project_default_attachments.zip",
"size": 1125,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "project_default.qgs",
"size": 80961,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "QGIS",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "file",
"name": "QGIS3.ini",
"size": 119124,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "QGISCUSTOMIZATION3.ini",
"size": 144627,
"time": "2023-12-22T16:14:45CET"
}
]
},
{
"type": "file",
"name": "startup_project.qgz",
"size": 13321,
"time": "2023-12-22T16:14:45CET"
}
]
}
]
},
{
"type": "file",
"name": "qdt-files.json",
"size": 0,
"time": "2023-12-29T11:05:41CET"
},
{
"type": "file",
"name": "README.md",
"size": 1139,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "scenarios",
"size": 4096,
"time": "2023-12-29T10:09:25CET",
"contents": [
{
"type": "file",
"name": "demo-scenario-http.qdt.yml",
"size": 1441,
"time": "2023-12-29T11:04:38CET"
},
{
"type": "file",
"name": "demo-scenario.qdt.yml",
"size": 1540,
"time": "2023-12-22T20:07:44CET"
}
]
}
]
},
{
"type": "report",
"directories": 8,
"files": 20
}
]
63 changes: 6 additions & 57 deletions qgis_deployment_toolbelt/profiles/remote_http_handler.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from shutil import rmtree
from typing import TypedDict

# 3rd party
import requests
Expand All @@ -30,6 +29,7 @@
from qgis_deployment_toolbelt.utils.file_downloader import download_remote_file_to_local
from qgis_deployment_toolbelt.utils.formatters import url_ensure_trailing_slash
from qgis_deployment_toolbelt.utils.proxies import get_proxy_settings
from qgis_deployment_toolbelt.utils.tree_files_reader import tree_to_download_list

# #############################################################################
# ########## Globals ###############
Expand All @@ -38,16 +38,11 @@
# logs
logger = logging.getLogger(__name__)


class Treeitem(TypedDict):
type: str
name: str
contents: list[dict] | None


# #############################################################################
# ########## Classes ###############
# ##################################


class HttpHandler(RemoteProfilesHandlerBase):
"""Handle remote HTTP repositories without git protocol.
Expand Down Expand Up @@ -117,7 +112,7 @@ def download(self, destination_local_path: Path):
)
destination_local_path.mkdir(parents=True)

li_files_to_download = self.tree_to_download_list(tree_array=qdt_tree)
li_files_to_download = tree_to_download_list(tree_array=qdt_tree)
logger.info(f"{len(li_files_to_download)} files to download")

success, fails = self.download_files_to_local(
Expand All @@ -131,54 +126,17 @@ def download(self, destination_local_path: Path):
f"{len(fails)} download failed. Check the above log messages."
)

def tree_to_download_list(
self, tree_array: list[Treeitem], rel_path: str = ""
) -> list:
"""Parse tree structure and return a list of files to download with relative
paths to the base URL. It's meant to be used as recursive funciton to iter
through the tree structure.
Args:
tree_array (list[TreeItem]): input array from tree JSON structure.
rel_path (str, optional): relative path to resolve from. Defaults to "".
Returns:
list: list of files paths relative to the base URL.
"""
li_files = []

for item in tree_array:
if item.get("type") == "directory":
if item.get("name") != ".":
new_rel_path = f"{rel_path}/{item.get('name')}"
else:
new_rel_path = f"{item.get('name')}"

li_files.extend(
self.tree_to_download_list(
tree_array=item.get("contents"),
rel_path=new_rel_path,
)
)
elif item.get("type") == "file":
li_files.append(f"{rel_path}/{item.get('name')}")
else:
logger.debug(f"Unsupported item type: {item.get('type')}")

return li_files

def download_files_to_local(
self, li_files_to_download: list[str], target_folder: Path
) -> tuple[list[tuple[str, Path]], list[tuple[str, Path]]]:
) -> tuple[list[tuple[str, Path]], list[tuple[str, str]]]:
"""Download list of files relative to remote base URL to local target folder.
Args:
li_files_to_download (list[str]): list of files to download.
target_folder (Path): local folder where to download
Returns:
tuple[list[tuple[str, Path]], list[tuple[str, Path]]]: (list of success \
download, list of failed download)
(list of success download, list of failed download)
"""
base_url = self.SOURCE_REPOSITORY_PATH_OR_URL
downloaded_files: list[tuple[str, Path]] = []
Expand Down Expand Up @@ -207,12 +165,3 @@ def download_files_to_local(
failed_files.append((file_to_download, f"{err}"))

return downloaded_files, failed_files


# #############################################################################
# ##### Stand alone program ########
# ##################################

if __name__ == "__main__":
"""Standalone execution."""
pass
Loading

0 comments on commit 53b5d44

Please sign in to comment.