-
Notifications
You must be signed in to change notification settings - Fork 20
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add Client for File Service #65
Merged
Merged
Changes from all commits
Commits
Show all changes
46 commits
Select commit
Hold shift + click to select a range
743bc74
Draft of key APIs and tests
santhoshramaraj e6a5e41
Merge branch 'users/sramaraj/file-client' of https://github.com/ni/ni…
santhoshramaraj b0107a0
feat: Use logically grouped models into modules
santhoshramaraj 600c0ba
docs: add rst files for FileClient
santhoshramaraj 134fc30
refactor: Refer to the correct models
santhoshramaraj 7e68d9e
feat: MVP for File Client to get, delete and download files
santhoshramaraj 8946c8f
refactor: comment out unused import
santhoshramaraj 27747b9
refactor: linting
santhoshramaraj 7d3b38b
refactor: type checking
santhoshramaraj 567a312
refactor: delete unused models
santhoshramaraj a686c57
refactor: sort imports
santhoshramaraj 66743f6
feat: Example to download a file using File Client
santhoshramaraj d06eb7b
feat: Use file name from metadata
santhoshramaraj 714f3a6
feat: Use untitled as filename if Name is not available
santhoshramaraj f6a319f
refactor: types and linting
santhoshramaraj 8e5dfcb
docs: reformat docstrings to google format without types
santhoshramaraj ac21dd7
refactor: wrapper to handle special bool serialization of `get_files`
santhoshramaraj 899eca6
refactor: fix mypy types error on Literal vs str
santhoshramaraj 15a6579
refactor: Serialize types unsupported by Uplink to a wrapper method a…
santhoshramaraj 345a89f
refactor: fix linting errors
santhoshramaraj c4c8c6b
docs: Update getting_started.rst to include FileClient
santhoshramaraj 7b30f75
refactor: change dict.get to subscript
santhoshramaraj ebb5726
refactor: add spacing between field definitions
santhoshramaraj 1c603a9
feat: utilities for FileClient
santhoshramaraj 161fe20
refactor: lint changes
santhoshramaraj a446df8
feat: File Upload method
santhoshramaraj 3109fe7
tests: extended test cases to cover test file upload and cleanup
santhoshramaraj 0ea3a09
tests: refactor to avoid too many requests error
santhoshramaraj 26f7e7d
docs: update rst docs for FileClient
santhoshramaraj cc67186
refactor: linting
santhoshramaraj 1ce8c31
feat: Add support for update-metadata endpoint
santhoshramaraj b915552
feat: Utility to rename files
santhoshramaraj 7146e5b
tests: remove redundant tests to reduce too many request error
santhoshramaraj b05a79d
refactor: linting
santhoshramaraj fe687aa
refactor: change args from `file_id` to `id` for consistency with oth…
santhoshramaraj 40713d5
refactor: make `ids` arg of `get_files` a List for ease of use.
santhoshramaraj c3fe3ac
refactor: simplify tests to eliminate duplicate test cases and reduce…
santhoshramaraj 2aead9d
feat: basic back-off-retry for status 429
santhoshramaraj 6de62e5
refactor: make the file-like-response-handler part of core reusables
santhoshramaraj 9006ad8
refactor: eliminate DeleteMultipleRequest and inline the List of IDs …
santhoshramaraj 9059ffb
refactor: Simplify upload_file to return file_id instead of a pydanti…
santhoshramaraj 33d93e6
refactor: replace @returns.json with a response handler to extract id…
santhoshramaraj 611f54c
tests: Prefix files created for test
santhoshramaraj 9227426
refactor: reuse existing `Operation` model
santhoshramaraj 62ba935
refactor: simplify methods with query params with fixed value
santhoshramaraj 614551c
refactor: remove use of uplink.Field when JsonModel works
santhoshramaraj File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
.. _api_tag_page: | ||
|
||
nisystemlink.clients.file | ||
====================== | ||
|
||
.. autoclass:: nisystemlink.clients.file.FileClient | ||
:exclude-members: __init__ | ||
|
||
.. automethod:: __init__ | ||
.. automethod:: api_info | ||
.. automethod:: get_files | ||
.. automethod:: delete_file | ||
.. automethod:: delete_files | ||
.. automethod:: upload_file | ||
.. automethod:: download_file | ||
.. automethod:: update_metadata | ||
|
||
.. automodule:: nisystemlink.clients.file.models | ||
:members: | ||
:imported-members: |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,30 @@ | ||
"""Example to download a file from SystemLink.""" | ||
|
||
from shutil import copyfileobj | ||
|
||
from nisystemlink.clients.file import FileClient | ||
|
||
client = FileClient() | ||
|
||
file_id = "a55adc7f-5068-4202-9d70-70ca6a06bee9" | ||
|
||
# Fetch the file metadata to get the name | ||
files = client.get_files(ids=[file_id]) | ||
|
||
if not files.available_files: | ||
raise Exception(f"File ID {file_id} not found.") | ||
|
||
|
||
file_name = "Untitled" | ||
|
||
file_properties = files.available_files[0].properties | ||
|
||
if file_properties: | ||
file_name = file_properties["Name"] | ||
|
||
# Download the file using FileId with content inline | ||
content = client.download_file(id=file_id) | ||
|
||
# Write the content to a file | ||
with open(file_name, "wb") as f: | ||
copyfileobj(content, f) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
from nisystemlink.clients.core.helpers import IteratorFileLike | ||
from requests.models import Response | ||
|
||
|
||
def file_like_response_handler(response: Response) -> IteratorFileLike: | ||
"""Response handler for File-Like content.""" | ||
return IteratorFileLike(response.iter_content(chunk_size=4096)) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
from ._file_client import FileClient | ||
|
||
# flake8: noqa |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,260 @@ | ||
"""Implementation of FileClient.""" | ||
|
||
import json | ||
from typing import BinaryIO, Dict, List, Optional | ||
|
||
from nisystemlink.clients import core | ||
from nisystemlink.clients.core._uplink._base_client import BaseClient | ||
from nisystemlink.clients.core._uplink._file_like_response import ( | ||
file_like_response_handler, | ||
) | ||
from nisystemlink.clients.core._uplink._methods import ( | ||
delete, | ||
get, | ||
post, | ||
response_handler, | ||
) | ||
from nisystemlink.clients.core.helpers import IteratorFileLike | ||
from requests.models import Response | ||
from uplink import Body, Field, params, Part, Path, Query, retry | ||
|
||
from . import models | ||
|
||
|
||
def _file_uri_response_handler(response: Response) -> str: | ||
"""Response handler for File URI response. Extracts ID from URI.""" | ||
resp = response.json() | ||
uri: str = resp["uri"] | ||
# Split the uri by '/' and get the last part | ||
parts = uri.split("/") | ||
return parts[-1] | ||
|
||
|
||
@retry(when=retry.when.status(429), stop=retry.stop.after_attempt(5)) | ||
class FileClient(BaseClient): | ||
def __init__(self, configuration: Optional[core.HttpConfiguration] = None): | ||
"""Initialize an instance. | ||
|
||
Args: | ||
configuration: Defines the web server to connect to and information about | ||
how to connect. If not provided, the | ||
:class:`HttpConfigurationManager <nisystemlink.clients.core.HttpConfigurationManager>` | ||
is used to obtain the configuration. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
if configuration is None: | ||
configuration = core.HttpConfigurationManager.get_configuration() | ||
|
||
super().__init__(configuration, "/nifile/v1/") | ||
|
||
@get("") | ||
def api_info(self) -> models.V1Operations: | ||
"""Get information about available API operations. | ||
|
||
Returns: | ||
Information about available API operations. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
|
||
@get( | ||
"service-groups/Default/files", | ||
args=[ | ||
Query, | ||
Query, | ||
Query(name="orderBy"), | ||
Query(name="orderByDescending"), | ||
Query(name="id"), | ||
], | ||
) | ||
def __get_files( | ||
self, | ||
skip: int = 0, | ||
take: int = 0, | ||
order_by: Optional[str] = None, | ||
order_by_descending: Optional[str] = "false", | ||
ids: Optional[str] = None, | ||
) -> models.FileQueryResponse: | ||
"""Lists available files on the SystemLink File service. | ||
Use the skip and take parameters to return paged responses. | ||
The orderBy and orderByDescending fields can be used to manage sorting the list by metadata objects. | ||
|
||
Args: | ||
skip: How many files to skip in the result when paging. Defaults to 0. | ||
take: How many files to return in the result, or 0 to use a default defined by the service. | ||
Defaults to 0. | ||
order_by: The name of the metadata key to sort by. Defaults to None. | ||
order_by_descending: The elements in the list are sorted ascending if "false" | ||
and descending if "true". Defaults to "false". | ||
ids: Comma-separated list of file IDs to search by. Defaults to None. | ||
|
||
Returns: | ||
File Query Response | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
|
||
def get_files( | ||
self, | ||
skip: int = 0, | ||
take: int = 0, | ||
order_by: Optional[models.FileQueryOrderBy] = None, | ||
order_by_descending: Optional[bool] = False, | ||
ids: Optional[List[str]] = None, | ||
) -> models.FileQueryResponse: | ||
"""Lists available files on the SystemLink File service. | ||
Use the skip and take parameters to return paged responses. | ||
The orderBy and orderByDescending fields can be used to manage sorting the list by metadata objects. | ||
|
||
Args: | ||
skip: How many files to skip in the result when paging. Defaults to 0. | ||
take: How many files to return in the result, or 0 to use a default defined by the service. | ||
Defaults to 0. | ||
order_by: The name of the metadata key to sort by. Defaults to None. | ||
order_by_descending: The elements in the list are sorted ascending if False | ||
and descending if True. Defaults to False. | ||
ids: List of file IDs to search by. Defaults to None. | ||
|
||
Returns: | ||
File Query Response | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
# Uplink does not support enum serializing into str | ||
santhoshramaraj marked this conversation as resolved.
Show resolved
Hide resolved
|
||
# workaround as the service expects lower case `true` and `false` | ||
# uplink serializes bools to `True` and `False` | ||
order_by_str = order_by.value if order_by is not None else None | ||
order_by_desc_str = "true" if order_by_descending else "false" | ||
|
||
if ids: | ||
ids_str = ",".join(ids) | ||
else: | ||
ids_str = "" | ||
|
||
resp = self.__get_files( | ||
skip=skip, | ||
take=take, | ||
order_by=order_by_str, | ||
order_by_descending=order_by_desc_str, | ||
ids=ids_str, | ||
) | ||
|
||
return resp | ||
|
||
@params({"force": True}) # type: ignore | ||
@delete("service-groups/Default/files/{id}", args=[Path]) | ||
def delete_file(self, id: str) -> None: | ||
"""Deletes the file indicated by the `file_id`. | ||
|
||
Args: | ||
id: The ID of the file. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
|
||
@params({"force": True}) # type: ignore | ||
@post("service-groups/Default/delete-files", args=[Field]) | ||
def delete_files(self, ids: List[str]) -> None: | ||
"""Delete multiple files. | ||
|
||
Args: | ||
ids: List of unique IDs of Files. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
|
||
@params({"inline": True}) # type: ignore | ||
@response_handler(file_like_response_handler) | ||
@get("service-groups/Default/files/{id}/data", args=[Path]) | ||
def download_file(self, id: str) -> IteratorFileLike: | ||
"""Downloads a file from the SystemLink File service. | ||
|
||
Args: | ||
id: The ID of the file. | ||
|
||
Yields: | ||
A file-like object for reading the exported data. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
|
||
@response_handler(_file_uri_response_handler) | ||
@post("service-groups/Default/upload-files") | ||
def __upload_file( | ||
self, | ||
file: Part, | ||
metadata: Part = None, | ||
id: Part = None, | ||
workspace: Query = None, | ||
) -> str: | ||
"""Uploads a file using multipart/form-data headers to send the file payload in the HTTP body. | ||
|
||
Args: | ||
file: The file to upload. | ||
metadata: JSON Dictionary with key/value pairs | ||
id: Specify an unique (among all file) 24-digit Hex string ID of the file once it is uploaded. | ||
Defaults to None. | ||
workspace: The id of the workspace the file belongs to. Defaults to None. | ||
|
||
Returns: | ||
ID of uploaded file. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
|
||
def upload_file( | ||
self, | ||
file: BinaryIO, | ||
metadata: Optional[Dict[str, str]] = None, | ||
id: Optional[str] = None, | ||
workspace: Optional[str] = None, | ||
) -> str: | ||
"""Uploads a file to the File Service. | ||
|
||
Args: | ||
file: The file to upload. | ||
metadata: File Metadata as dictionary. | ||
id: Specify an unique (among all file) 24-digit Hex string ID of the file once it is uploaded. | ||
Defaults to None. | ||
workspace: The id of the workspace the file belongs to. Defaults to None. | ||
|
||
Returns: | ||
ID of uploaded file. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" | ||
if metadata: | ||
metadata_str = json.dumps(metadata) | ||
else: | ||
metadata_str = None | ||
|
||
file_id = self.__upload_file( | ||
file=file, | ||
metadata=metadata_str, | ||
id=id, | ||
workspace=workspace, | ||
) | ||
|
||
return file_id | ||
|
||
@post("service-groups/Default/files/{id}/update-metadata", args=[Body, Path]) | ||
def update_metadata(self, metadata: models.UpdateMetadataRequest, id: str) -> None: | ||
"""Updates an existing file's metadata with the specified metadata properties. | ||
|
||
Args: | ||
metadata: File's metadata and options for updating it. | ||
id: ID of the file to update Metadata. | ||
|
||
Raises: | ||
ApiException: if unable to communicate with the File Service. | ||
""" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,8 @@ | ||
from ._file_metadata import FileMetadata | ||
from ._file_query_order_by import FileQueryOrderBy | ||
from ._file_query_response import FileQueryResponse | ||
from ._link import Link | ||
from ._operations import V1Operations | ||
from ._update_metadata import UpdateMetadataRequest | ||
|
||
# flake8: noqa |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@spanglerco Do you know if we have guidance/conventions for HTTP resilience strategies? How many times should we retry?