Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve: qdt files reliability #514

Merged
merged 5 commits into from
May 30, 2024
Merged

Improve: qdt files reliability #514

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7659897
improve(docs): add --prune option to tree for excluding empty folders
Guts May 30, 2024
a92f8cf
refacto(http-handler): modularize tree files parser
Guts May 30, 2024
51c041e
feature(tree-files): handle files with empty folders
Guts May 30, 2024
4a6a708
improve(test): add fixtures and dedicated test to qdt-files.json parsing
Guts May 30, 2024
00faae8
improve(test): fake a case with an empty directory but still with con…
Guts May 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions docs/guides/howto_publish_http.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,27 @@

## Generate the `qdt-files.json` index file

> Typically on Ubuntu
> Typically on Ubuntu 22.04

Install tree:
Install tree > 2:

```sh
sudo apt install tree
```

Check version:

```sh
tree --version
```

Run it:

```sh
# move to your QDT profiles folder. Here we take the QDT repository as example:
cd examples/
# generate the qdt-files.json
tree --gitignore -D --timefmt="%Y-%m-%dT%H:%M:%S%Z" -s -J -o qdt-files.json .
tree --gitignore -D -J --prune -s --timefmt="%Y-%m-%dT%H:%M:%S%Z" -o qdt-files.json .
```

Detailed explanation:
Expand All @@ -25,8 +31,9 @@ Detailed explanation:
<!-- - `-f`: display the full path for each file and directory. -->
- `--gitignore`: apply gitignore-style rules to exclude files and directories.
- `-D`: print the modification time for each file or directory.
- `--timefmt="%Y-%m-%dT%H:%M:%S%Z"`: specify the time format as ISO8601 with UTC (Coordinated Universal Time).
- `-s`: print the size of each file.
- `-J`: output the directory tree in JSON format.
- `--prune`: do not include empty folders
- `-s`: print the size of each file.
- `--timefmt="%Y-%m-%dT%H:%M:%S%Z"`: specify the time format as ISO8601 with UTC (Coordinated Universal Time).
- `-o qdt-files.json`: save the output to a file named 'qdt-files.json'.
- `.`: specify the current directory as the starting point for the tree.
237 changes: 197 additions & 40 deletions examples/qdt-files.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,199 @@
[
{"type":"directory","name":".","size":4096,"time":"2023-12-29T11:03:36CET","contents":[
{"type":"directory","name":"profiles","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"directory","name":"demo","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"file","name":"bookmarks.xml","size":1582,"time":"2023-11-14T17:43:43CET"},
{"type":"directory","name":"images","size":4096,"time":"2023-06-13T17:02:08CEST","contents":[
{"type":"file","name":"logo_qdt.ico","size":227102,"time":"2023-06-13T17:02:08CEST"},
{"type":"file","name":"splash.png","size":354451,"time":"2023-06-13T17:02:08CEST"}
]},
{"type":"file","name":"profile.json","size":594,"time":"2023-06-13T17:02:08CEST"},
{"type":"directory","name":"QGIS","size":4096,"time":"2023-11-15T11:05:26CET","contents":[
{"type":"file","name":"QGIS3.ini","size":10605,"time":"2023-11-15T11:05:26CET"},
{"type":"file","name":"QGISCUSTOMIZATION3.ini","size":144530,"time":"2023-11-14T08:47:47CET"}
]}
]},
{"type":"file","name":"profiles.ini","size":34,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"Viewer Mode","size":4096,"time":"2023-12-29T10:13:27CET","contents":[
{"type":"file","name":"bookmarks.xml","size":1582,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"images","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"file","name":"logo_qdt.ico","size":227102,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"splash.png","size":211808,"time":"2023-12-22T16:14:45CET"}
]},
{"type":"file","name":"profile.json","size":1481,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"project_default_attachments.zip","size":1125,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"project_default.qgs","size":80961,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"QGIS","size":4096,"time":"2023-12-22T16:14:45CET","contents":[
{"type":"file","name":"QGIS3.ini","size":119124,"time":"2023-12-22T16:14:45CET"},
{"type":"file","name":"QGISCUSTOMIZATION3.ini","size":144627,"time":"2023-12-22T16:14:45CET"}
]},
{"type":"file","name":"startup_project.qgz","size":13321,"time":"2023-12-22T16:14:45CET"}
]}
]},
{"type":"file","name":"qdt-files.json","size":0,"time":"2023-12-29T11:05:41CET"},
{"type":"file","name":"README.md","size":1139,"time":"2023-12-22T16:14:45CET"},
{"type":"directory","name":"scenarios","size":4096,"time":"2023-12-29T10:09:25CET","contents":[
{"type":"file","name":"demo-scenario-http.qdt.yml","size":1441,"time":"2023-12-29T11:04:38CET"},
{"type":"file","name":"demo-scenario.qdt.yml","size":1540,"time":"2023-12-22T20:07:44CET"}
]}
]}
,
{"type":"report","directories":8,"files":20}
{
"type": "directory",
"name": ".",
"size": 4096,
"time": "2023-12-29T11:03:36CET",
"contents": [
{
"type": "directory",
"name": "profiles",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "directory",
"name": "demo",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "file",
"name": "bookmarks.xml",
"size": 1582,
"time": "2023-11-14T17:43:43CET"
},
{
"type": "directory",
"name": "images",
"size": 4096,
"time": "2023-06-13T17:02:08CEST",
"contents": [
{
"type": "file",
"name": "logo_qdt.ico",
"size": 227102,
"time": "2023-06-13T17:02:08CEST"
},
{
"type": "file",
"name": "splash.png",
"size": 354451,
"time": "2023-06-13T17:02:08CEST"
}
]
},
{
"type": "file",
"name": "profile.json",
"size": 594,
"time": "2023-06-13T17:02:08CEST"
},
{
"type": "directory",
"name": "QGIS",
"size": 4096,
"time": "2023-11-15T11:05:26CET",
"contents": [
{
"type": "file",
"name": "QGIS3.ini",
"size": 10605,
"time": "2023-11-15T11:05:26CET"
},
{
"type": "file",
"name": "QGISCUSTOMIZATION3.ini",
"size": 144530,
"time": "2023-11-14T08:47:47CET"
}
]
}
]
},
{
"type": "file",
"name": "profiles.ini",
"size": 34,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "Viewer Mode",
"size": 4096,
"time": "2023-12-29T10:13:27CET",
"contents": [
{
"type": "file",
"name": "bookmarks.xml",
"size": 1582,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "images",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "file",
"name": "logo_qdt.ico",
"size": 227102,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "splash.png",
"size": 211808,
"time": "2023-12-22T16:14:45CET"
}
]
},
{
"type": "file",
"name": "profile.json",
"size": 1481,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "project_default_attachments.zip",
"size": 1125,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "project_default.qgs",
"size": 80961,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "QGIS",
"size": 4096,
"time": "2023-12-22T16:14:45CET",
"contents": [
{
"type": "file",
"name": "QGIS3.ini",
"size": 119124,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "file",
"name": "QGISCUSTOMIZATION3.ini",
"size": 144627,
"time": "2023-12-22T16:14:45CET"
}
]
},
{
"type": "file",
"name": "startup_project.qgz",
"size": 13321,
"time": "2023-12-22T16:14:45CET"
}
]
}
]
},
{
"type": "file",
"name": "qdt-files.json",
"size": 0,
"time": "2023-12-29T11:05:41CET"
},
{
"type": "file",
"name": "README.md",
"size": 1139,
"time": "2023-12-22T16:14:45CET"
},
{
"type": "directory",
"name": "scenarios",
"size": 4096,
"time": "2023-12-29T10:09:25CET",
"contents": [
{
"type": "file",
"name": "demo-scenario-http.qdt.yml",
"size": 1441,
"time": "2023-12-29T11:04:38CET"
},
{
"type": "file",
"name": "demo-scenario.qdt.yml",
"size": 1540,
"time": "2023-12-22T20:07:44CET"
}
]
}
]
},
{
"type": "report",
"directories": 8,
"files": 20
}
]
63 changes: 6 additions & 57 deletions qgis_deployment_toolbelt/profiles/remote_http_handler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from shutil import rmtree
from typing import TypedDict

# 3rd party
import requests
Expand All @@ -30,6 +29,7 @@
from qgis_deployment_toolbelt.utils.file_downloader import download_remote_file_to_local
from qgis_deployment_toolbelt.utils.formatters import url_ensure_trailing_slash
from qgis_deployment_toolbelt.utils.proxies import get_proxy_settings
from qgis_deployment_toolbelt.utils.tree_files_reader import tree_to_download_list

# #############################################################################
# ########## Globals ###############
Expand All @@ -38,16 +38,11 @@
# logs
logger = logging.getLogger(__name__)


class Treeitem(TypedDict):
type: str
name: str
contents: list[dict] | None


# #############################################################################
# ########## Classes ###############
# ##################################


class HttpHandler(RemoteProfilesHandlerBase):
"""Handle remote HTTP repositories without git protocol.

Expand Down Expand Up @@ -117,7 +112,7 @@ def download(self, destination_local_path: Path):
)
destination_local_path.mkdir(parents=True)

li_files_to_download = self.tree_to_download_list(tree_array=qdt_tree)
li_files_to_download = tree_to_download_list(tree_array=qdt_tree)
logger.info(f"{len(li_files_to_download)} files to download")

success, fails = self.download_files_to_local(
Expand All @@ -131,54 +126,17 @@ def download(self, destination_local_path: Path):
f"{len(fails)} download failed. Check the above log messages."
)

def tree_to_download_list(
self, tree_array: list[Treeitem], rel_path: str = ""
) -> list:
"""Parse tree structure and return a list of files to download with relative
paths to the base URL. It's meant to be used as recursive funciton to iter
through the tree structure.

Args:
tree_array (list[TreeItem]): input array from tree JSON structure.
rel_path (str, optional): relative path to resolve from. Defaults to "".

Returns:
list: list of files paths relative to the base URL.
"""
li_files = []

for item in tree_array:
if item.get("type") == "directory":
if item.get("name") != ".":
new_rel_path = f"{rel_path}/{item.get('name')}"
else:
new_rel_path = f"{item.get('name')}"

li_files.extend(
self.tree_to_download_list(
tree_array=item.get("contents"),
rel_path=new_rel_path,
)
)
elif item.get("type") == "file":
li_files.append(f"{rel_path}/{item.get('name')}")
else:
logger.debug(f"Unsupported item type: {item.get('type')}")

return li_files

def download_files_to_local(
self, li_files_to_download: list[str], target_folder: Path
) -> tuple[list[tuple[str, Path]], list[tuple[str, Path]]]:
) -> tuple[list[tuple[str, Path]], list[tuple[str, str]]]:
"""Download list of files relative to remote base URL to local target folder.

Args:
li_files_to_download (list[str]): list of files to download.
target_folder (Path): local folder where to download

Returns:
tuple[list[tuple[str, Path]], list[tuple[str, Path]]]: (list of success \
download, list of failed download)
(list of success download, list of failed download)
"""
base_url = self.SOURCE_REPOSITORY_PATH_OR_URL
downloaded_files: list[tuple[str, Path]] = []
Expand Down Expand Up @@ -207,12 +165,3 @@ def download_files_to_local(
failed_files.append((file_to_download, f"{err}"))

return downloaded_files, failed_files


# #############################################################################
# ##### Stand alone program ########
# ##################################

if __name__ == "__main__":
"""Standalone execution."""
pass
Loading