Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Client for File Service #65

Merged
merged 46 commits into from
Sep 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
743bc74
Draft of key APIs and tests
santhoshramaraj Nov 21, 2023
e6a5e41
Merge branch 'users/sramaraj/file-client' of https://github.com/ni/ni…
santhoshramaraj Jun 19, 2024
b0107a0
feat: Use logically grouped models into modules
santhoshramaraj Jul 1, 2024
600c0ba
docs: add rst files for FileClient
santhoshramaraj Jul 1, 2024
134fc30
refactor: Refer to the correct models
santhoshramaraj Jul 1, 2024
7e68d9e
feat: MVP for File Client to get, delete and download files
santhoshramaraj Jul 1, 2024
8946c8f
refactor: comment out unused import
santhoshramaraj Jul 1, 2024
27747b9
refactor: linting
santhoshramaraj Jul 1, 2024
7d3b38b
refactor: type checking
santhoshramaraj Jul 1, 2024
567a312
refactor: delete unused models
santhoshramaraj Jul 5, 2024
a686c57
refactor: sort imports
santhoshramaraj Jul 5, 2024
66743f6
feat: Example to download a file using File Client
santhoshramaraj Jul 5, 2024
d06eb7b
feat: Use file name from metadata
santhoshramaraj Jul 5, 2024
714f3a6
feat: Use untitled as filename if Name is not available
santhoshramaraj Jul 5, 2024
f6a319f
refactor: types and linting
santhoshramaraj Jul 5, 2024
8e5dfcb
docs: reformat docstrings to google format without types
santhoshramaraj Aug 10, 2024
ac21dd7
refactor: wrapper to handle special bool serialization of `get_files`
santhoshramaraj Aug 12, 2024
899eca6
refactor: fix mypy types error on Literal vs str
santhoshramaraj Aug 12, 2024
15a6579
refactor: Serialize types unsupported by Uplink to a wrapper method a…
santhoshramaraj Aug 12, 2024
345a89f
refactor: fix linting errors
santhoshramaraj Aug 12, 2024
c4c8c6b
docs: Update getting_started.rst to include FileClient
santhoshramaraj Aug 12, 2024
7b30f75
refactor: change dict.get to subscript
santhoshramaraj Aug 14, 2024
ebb5726
refactor: add spacing between field definitions
santhoshramaraj Aug 14, 2024
1c603a9
feat: utilities for FileClient
santhoshramaraj Sep 3, 2024
161fe20
refactor: lint changes
santhoshramaraj Sep 3, 2024
a446df8
feat: File Upload method
santhoshramaraj Sep 3, 2024
3109fe7
tests: extended test cases to cover test file upload and cleanup
santhoshramaraj Sep 3, 2024
0ea3a09
tests: refactor to avoid too many requests error
santhoshramaraj Sep 3, 2024
26f7e7d
docs: update rst docs for FileClient
santhoshramaraj Sep 3, 2024
cc67186
refactor: linting
santhoshramaraj Sep 3, 2024
1ce8c31
feat: Add support for update-metadata endpoint
santhoshramaraj Sep 3, 2024
b915552
feat: Utility to rename files
santhoshramaraj Sep 3, 2024
7146e5b
tests: remove redundant tests to reduce too many request error
santhoshramaraj Sep 3, 2024
b05a79d
refactor: linting
santhoshramaraj Sep 3, 2024
fe687aa
refactor: change args from `file_id` to `id` for consistency with oth…
santhoshramaraj Sep 9, 2024
40713d5
refactor: make `ids` arg of `get_files` a List for ease of use.
santhoshramaraj Sep 9, 2024
c3fe3ac
refactor: simplify tests to eliminate duplicate test cases and reduce…
santhoshramaraj Sep 10, 2024
2aead9d
feat: basic back-off-retry for status 429
santhoshramaraj Sep 10, 2024
6de62e5
refactor: make the file-like-response-handler part of core reusables
santhoshramaraj Sep 11, 2024
9006ad8
refactor: eliminate DeleteMultipleRequest and inline the List of IDs …
santhoshramaraj Sep 11, 2024
9059ffb
refactor: Simplify upload_file to return file_id instead of a pydanti…
santhoshramaraj Sep 11, 2024
33d93e6
refactor: replace @returns.json with a response handler to extract id…
santhoshramaraj Sep 11, 2024
611f54c
tests: Prefix files created for test
santhoshramaraj Sep 11, 2024
9227426
refactor: reuse existing `Operation` model
santhoshramaraj Sep 16, 2024
62ba935
refactor: simplify methods with query params with fixed value
santhoshramaraj Sep 17, 2024
614551c
refactor: remove use of uplink.Field when JsonModel works
santhoshramaraj Sep 17, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/api_reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ API Reference
api_reference/testmonitor
api_reference/dataframe
api_reference/spec
api_reference/file

Indices and tables
------------------
Expand Down
20 changes: 20 additions & 0 deletions docs/api_reference/file.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
.. _api_tag_page:

nisystemlink.clients.file
======================

.. autoclass:: nisystemlink.clients.file.FileClient
:exclude-members: __init__

.. automethod:: __init__
.. automethod:: api_info
.. automethod:: get_files
.. automethod:: delete_file
.. automethod:: delete_files
.. automethod:: upload_file
.. automethod:: download_file
.. automethod:: update_metadata

.. automodule:: nisystemlink.clients.file.models
:members:
:imported-members:
27 changes: 27 additions & 0 deletions docs/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -155,5 +155,32 @@ Create and Query Specifications
Update and Delete Specifications

.. literalinclude:: ../examples/spec/update_and_delete_specs.py
:language: python
:linenos:


File API
-------

Overview
~~~~~~~~

The :class:`.FileClient` class is the primary entry point of the File API.

When constructing a :class:`.FileClient`, you can pass an
:class:`.HttpConfiguration` (like one retrieved from the
:class:`.HttpConfigurationManager`), or let :class:`.FileClient` use the
default connection. The default connection depends on your environment.

With a :class:`.FileClient` object, you can:

* Get the list of files, download and delete files

Examples
~~~~~~~~

Get the metadata of a File using its Id and download it.

.. literalinclude:: ../examples/file/download_file.py
:language: python
:linenos:
30 changes: 30 additions & 0 deletions examples/file/download_file.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
"""Example to download a file from SystemLink."""

from shutil import copyfileobj

from nisystemlink.clients.file import FileClient

client = FileClient()

file_id = "a55adc7f-5068-4202-9d70-70ca6a06bee9"

# Fetch the file metadata to get the name
files = client.get_files(ids=[file_id])

if not files.available_files:
raise Exception(f"File ID {file_id} not found.")


file_name = "Untitled"

file_properties = files.available_files[0].properties

if file_properties:
file_name = file_properties["Name"]

# Download the file using FileId with content inline
content = client.download_file(id=file_id)

# Write the content to a file
with open(file_name, "wb") as f:
copyfileobj(content, f)
7 changes: 7 additions & 0 deletions nisystemlink/clients/core/_uplink/_file_like_response.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
from nisystemlink.clients.core.helpers import IteratorFileLike
from requests.models import Response


def file_like_response_handler(response: Response) -> IteratorFileLike:
"""Response handler for File-Like content."""
return IteratorFileLike(response.iter_content(chunk_size=4096))
3 changes: 3 additions & 0 deletions nisystemlink/clients/file/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
from ._file_client import FileClient

# flake8: noqa
260 changes: 260 additions & 0 deletions nisystemlink/clients/file/_file_client.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,260 @@
"""Implementation of FileClient."""

import json
from typing import BinaryIO, Dict, List, Optional

from nisystemlink.clients import core
from nisystemlink.clients.core._uplink._base_client import BaseClient
from nisystemlink.clients.core._uplink._file_like_response import (
file_like_response_handler,
)
from nisystemlink.clients.core._uplink._methods import (
delete,
get,
post,
response_handler,
)
from nisystemlink.clients.core.helpers import IteratorFileLike
from requests.models import Response
from uplink import Body, Field, params, Part, Path, Query, retry

from . import models


def _file_uri_response_handler(response: Response) -> str:
"""Response handler for File URI response. Extracts ID from URI."""
resp = response.json()
uri: str = resp["uri"]
# Split the uri by '/' and get the last part
parts = uri.split("/")
return parts[-1]


@retry(when=retry.when.status(429), stop=retry.stop.after_attempt(5))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@spanglerco Do you know if we have guidance/conventions for HTTP resilience strategies? How many times should we retry?

class FileClient(BaseClient):
def __init__(self, configuration: Optional[core.HttpConfiguration] = None):
"""Initialize an instance.

Args:
configuration: Defines the web server to connect to and information about
how to connect. If not provided, the
:class:`HttpConfigurationManager <nisystemlink.clients.core.HttpConfigurationManager>`
is used to obtain the configuration.

Raises:
ApiException: if unable to communicate with the File Service.
"""
if configuration is None:
configuration = core.HttpConfigurationManager.get_configuration()

super().__init__(configuration, "/nifile/v1/")

@get("")
def api_info(self) -> models.V1Operations:
"""Get information about available API operations.

Returns:
Information about available API operations.

Raises:
ApiException: if unable to communicate with the File Service.
"""

@get(
"service-groups/Default/files",
args=[
Query,
Query,
Query(name="orderBy"),
Query(name="orderByDescending"),
Query(name="id"),
],
)
def __get_files(
self,
skip: int = 0,
take: int = 0,
order_by: Optional[str] = None,
order_by_descending: Optional[str] = "false",
ids: Optional[str] = None,
) -> models.FileQueryResponse:
"""Lists available files on the SystemLink File service.
Use the skip and take parameters to return paged responses.
The orderBy and orderByDescending fields can be used to manage sorting the list by metadata objects.

Args:
skip: How many files to skip in the result when paging. Defaults to 0.
take: How many files to return in the result, or 0 to use a default defined by the service.
Defaults to 0.
order_by: The name of the metadata key to sort by. Defaults to None.
order_by_descending: The elements in the list are sorted ascending if "false"
and descending if "true". Defaults to "false".
ids: Comma-separated list of file IDs to search by. Defaults to None.

Returns:
File Query Response

Raises:
ApiException: if unable to communicate with the File Service.
"""

def get_files(
self,
skip: int = 0,
take: int = 0,
order_by: Optional[models.FileQueryOrderBy] = None,
order_by_descending: Optional[bool] = False,
ids: Optional[List[str]] = None,
) -> models.FileQueryResponse:
"""Lists available files on the SystemLink File service.
Use the skip and take parameters to return paged responses.
The orderBy and orderByDescending fields can be used to manage sorting the list by metadata objects.

Args:
skip: How many files to skip in the result when paging. Defaults to 0.
take: How many files to return in the result, or 0 to use a default defined by the service.
Defaults to 0.
order_by: The name of the metadata key to sort by. Defaults to None.
order_by_descending: The elements in the list are sorted ascending if False
and descending if True. Defaults to False.
ids: List of file IDs to search by. Defaults to None.

Returns:
File Query Response

Raises:
ApiException: if unable to communicate with the File Service.
"""
# Uplink does not support enum serializing into str
santhoshramaraj marked this conversation as resolved.
Show resolved Hide resolved
# workaround as the service expects lower case `true` and `false`
# uplink serializes bools to `True` and `False`
order_by_str = order_by.value if order_by is not None else None
order_by_desc_str = "true" if order_by_descending else "false"

if ids:
ids_str = ",".join(ids)
else:
ids_str = ""

resp = self.__get_files(
skip=skip,
take=take,
order_by=order_by_str,
order_by_descending=order_by_desc_str,
ids=ids_str,
)

return resp

@params({"force": True}) # type: ignore
@delete("service-groups/Default/files/{id}", args=[Path])
def delete_file(self, id: str) -> None:
"""Deletes the file indicated by the `file_id`.

Args:
id: The ID of the file.

Raises:
ApiException: if unable to communicate with the File Service.
"""

@params({"force": True}) # type: ignore
@post("service-groups/Default/delete-files", args=[Field])
def delete_files(self, ids: List[str]) -> None:
"""Delete multiple files.

Args:
ids: List of unique IDs of Files.

Raises:
ApiException: if unable to communicate with the File Service.
"""

@params({"inline": True}) # type: ignore
@response_handler(file_like_response_handler)
@get("service-groups/Default/files/{id}/data", args=[Path])
def download_file(self, id: str) -> IteratorFileLike:
"""Downloads a file from the SystemLink File service.

Args:
id: The ID of the file.

Yields:
A file-like object for reading the exported data.

Raises:
ApiException: if unable to communicate with the File Service.
"""

@response_handler(_file_uri_response_handler)
@post("service-groups/Default/upload-files")
def __upload_file(
self,
file: Part,
metadata: Part = None,
id: Part = None,
workspace: Query = None,
) -> str:
"""Uploads a file using multipart/form-data headers to send the file payload in the HTTP body.

Args:
file: The file to upload.
metadata: JSON Dictionary with key/value pairs
id: Specify an unique (among all file) 24-digit Hex string ID of the file once it is uploaded.
Defaults to None.
workspace: The id of the workspace the file belongs to. Defaults to None.

Returns:
ID of uploaded file.

Raises:
ApiException: if unable to communicate with the File Service.
"""

def upload_file(
self,
file: BinaryIO,
metadata: Optional[Dict[str, str]] = None,
id: Optional[str] = None,
workspace: Optional[str] = None,
) -> str:
"""Uploads a file to the File Service.

Args:
file: The file to upload.
metadata: File Metadata as dictionary.
id: Specify an unique (among all file) 24-digit Hex string ID of the file once it is uploaded.
Defaults to None.
workspace: The id of the workspace the file belongs to. Defaults to None.

Returns:
ID of uploaded file.

Raises:
ApiException: if unable to communicate with the File Service.
"""
if metadata:
metadata_str = json.dumps(metadata)
else:
metadata_str = None

file_id = self.__upload_file(
file=file,
metadata=metadata_str,
id=id,
workspace=workspace,
)

return file_id

@post("service-groups/Default/files/{id}/update-metadata", args=[Body, Path])
def update_metadata(self, metadata: models.UpdateMetadataRequest, id: str) -> None:
"""Updates an existing file's metadata with the specified metadata properties.

Args:
metadata: File's metadata and options for updating it.
id: ID of the file to update Metadata.

Raises:
ApiException: if unable to communicate with the File Service.
"""
8 changes: 8 additions & 0 deletions nisystemlink/clients/file/models/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
from ._file_metadata import FileMetadata
from ._file_query_order_by import FileQueryOrderBy
from ._file_query_response import FileQueryResponse
from ._link import Link
from ._operations import V1Operations
from ._update_metadata import UpdateMetadataRequest

# flake8: noqa
Loading
Loading