Introduce filtering columns when fetching tables (#986)

CHANGELOG.md

@@ -1,12 +1,17 @@
-## [UNRELEASED] neptune-client 0.16.6
+## [UNRELEASED] neptune-client 0.16.7
 
-## Features
+### Features
+- Introduce filtering columns when fetching run, model and model_version tables ([#986](https://github.com/neptune-ai/neptune-client/pull/986))
+
+## neptune-client 0.16.6
+
+### Features
 - Added support for Prophet integration ([#978](https://github.com/neptune-ai/neptune-client/pull/978))
 - Log argparse.Namespace objects as dicts ([#984](https://github.com/neptune-ai/neptune-client/pull/984))
 
 ## neptune-client 0.16.5
 
-## Features
+### Features
 - Added `NEPTUNE_MODE` environment variable ([#928](https://github.com/neptune-ai/neptune-client/pull/928))
 - Added support of Service account management ([#927](https://github.com/neptune-ai/neptune-client/pull/927))
 - More informational exception due to plotly and matplotlib incompatibility ([#960](https://github.com/neptune-ai/neptune-client/pull/960))
@@ -18,69 +23,69 @@
 
 ## neptune-client 0.16.4
 
-## Fixes
+### Fixes
 - Fix uploading in-memory files lager than 5MB ([#924](https://github.com/neptune-ai/neptune-client/pull/924))
 - fetch_extension added to Handler ([#923](https://github.com/neptune-ai/neptune-client/pull/923))
 
-## Changes
+### Changes
 - Force jsonschema version < 4.0.0 ([#922](https://github.com/neptune-ai/neptune-client/pull/922))
 
 - Rename and copy update for UnsupportedClientVersion and DeprecatedClientLibraryVersion ([#917](https://github.com/neptune-ai/neptune-client/pull/917))
 
 ## neptune-client 0.16.3
 
-## Features
+### Features
 - Added fetching Models method to Project ([#916](https://github.com/neptune-ai/neptune-client/pull/916))
 
-## Fixes
+### Fixes
 - Fix computing of a multipart upload chunk size ([#897](https://github.com/neptune-ai/neptune-client/pull/897))
 - Matching all listed tags instead of any when calling `fetch_runs_table` ([#899](https://github.com/neptune-ai/neptune-client/pull/899))
 - Fix invalid processing of delete followed by file upload in a single batch ([#880](https://github.com/neptune-ai/neptune-client/pull/880))
 - Stop hanging indefinitely on wait when async data synchronization process is dead ([#909](https://github.com/neptune-ai/neptune-client/pull/909))
 - Finish stop() faster when async data synchronization process dies ([#909](https://github.com/neptune-ai/neptune-client/pull/909))
 
-## Changes
+### Changes
 - `click.echo` replaced with `logging` ([#903](https://github.com/neptune-ai/neptune-client/pull/903))
 
 ## neptune-client 0.16.2
 
-## Features
+### Features
  - Sync only offline runs inside '.neptune' directory CLI flag ([#894](https://github.com/neptune-ai/neptune-client/pull/894))
 
-## Fixes
+### Fixes
 - Fix handling of server errors ([#896](https://github.com/neptune-ai/neptune-client/pull/896))
 
 ## neptune-client 0.16.1
 
-## Features
+### Features
 - Print metadata url on stop ([#883](https://github.com/neptune-ai/neptune-client/pull/883))
 
-## Fixes
+### Fixes
 - Fix handling Internal Server Error ([#885](https://github.com/neptune-ai/neptune-client/pull/885))
 
 ## neptune-client 0.16.0
 
-## Features
+### Features
 - Added python 3.10 support ([#879](https://github.com/neptune-ai/neptune-client/pull/879))
 - Dropped official support for python 3.6 ([#879](https://github.com/neptune-ai/neptune-client/pull/879))
 
-## Fixes
+### Fixes
 - restart upload when file changes during ([#877](https://github.com/neptune-ai/neptune-client/pull/877))
 
 ## neptune-client 0.15.2
 
-## Features
+### Features
 - Added support for workspace visibility in Management API ([#843](https://github.com/neptune-ai/neptune-client/pull/843))
 - Exposed container with a property of Handler ([#864](https://github.com/neptune-ai/neptune-client/pull/864))
 
 ## neptune-client 0.15.1
 
-## Fixes
+### Fixes
 - Restore __version__ in neptune.new ([#860](https://github.com/neptune-ai/neptune-client/pull/860))
 
 ## neptune-client 0.15.0
 
-## Features
+### Features
 - Methods for creating and manipulating Model Registry objects ([#794](https://github.com/neptune-ai/neptune-client/pull/794))
 
 ### Changes
@@ -91,10 +96,10 @@
 
 ## neptune-client 0.14.3
 
-## Features
+### Features
 - Stripping whitespaces from Neptune API Token ([#825](https://github.com/neptune-ai/neptune-client/pull/825))
 
-## Fixes
+### Fixes
 - Raise proper exception when invalid token were provided ([#825](https://github.com/neptune-ai/neptune-client/pull/825))
 - Make status error-handling in legacy client consistent with neptune.new ([#829](https://github.com/neptune-ai/neptune-client/pull/829))
 
@@ -103,7 +108,7 @@
 ### Features
 - Use new file upload API ([#789](https://github.com/neptune-ai/neptune-client/pull/789))
 
-## Fixes
+### Fixes
 - Fixed listing available workspaces when invalid name was provided ([#818](https://github.com/neptune-ai/neptune-client/pull/818))
 - Added proper docstrings for Project-Level Metadata ([#812](https://github.com/neptune-ai/neptune-client/pull/812))
 - Fixed backward compatibility when syncing old offline data ([#810](https://github.com/neptune-ai/neptune-client/pull/810))
@@ -113,7 +118,7 @@
 
 ## neptune-client 0.14.1
 
-## Fixes
+### Fixes
 - Fixed legacy url in NVML information ([#795](https://github.com/neptune-ai/neptune-client/pull/795))
 - Make init_project accepting kwargs only ([#805](https://github.com/neptune-ai/neptune-client/pull/805))
 

neptune/new/internal/backends/hosted_neptune_backend.py

@@ -1026,10 +1026,16 @@ def search_leaderboard_entries(
         project_id: UniqueId,
         types: Optional[Iterable[ContainerType]] = None,
         query: Optional[NQLQuery] = None,
+        columns: Optional[Iterable[str]] = None,
     ) -> List[LeaderboardEntry]:
-        query_params = {}
         if query:
             query_params = {"query": {"query": str(query)}}
+        else:
+            query_params = {}
+        if columns:
+            attributes_filter = {"attributeFilters": [{"path": column} for column in columns]}
+        else:
+            attributes_filter = {}
 
         def get_portion(limit, offset):
             return (
@@ -1038,6 +1044,7 @@ def get_portion(limit, offset):
                     type=list(map(lambda container_type: container_type.to_api(), types)),
                     params={
                         **query_params,
+                        **attributes_filter,
                         "pagination": {"limit": limit, "offset": offset},
                     },
                     **DEFAULT_REQUEST_KWARGS,

neptune/new/internal/backends/neptune_backend.py

@@ -297,5 +297,6 @@ def search_leaderboard_entries(
         project_id: UniqueId,
         types: Optional[Iterable[ContainerType]] = None,
         query: Optional[NQLQuery] = None,
+        columns: Optional[Iterable[str]] = None,
     ) -> List[LeaderboardEntry]:
         pass

neptune/new/internal/backends/neptune_backend_mock.py

@@ -540,6 +540,7 @@ def search_leaderboard_entries(
         project_id: UniqueId,
         types: Optional[Iterable[ContainerType]] = None,
         query: Optional[NQLQuery] = None,
+        columns: Optional[Iterable[str]] = None,
     ) -> List[LeaderboardEntry]:
         """Non relevant for mock"""
 

neptune/new/metadata_containers/metadata_container.py

@@ -23,7 +23,7 @@
 from contextlib import AbstractContextManager
 from datetime import datetime
 from functools import wraps
-from typing import Any, Dict, List, Optional, Union
+from typing import Any, Dict, Iterable, List, Optional, Union
 
 from neptune.exceptions import UNIX_STYLES
 from neptune.new.attributes import create_attribute_from_type
@@ -362,11 +362,19 @@ def _startup(self, debug_mode):
     def _shutdown_hook(self):
         self.stop()
 
-    def _fetch_entries(self, child_type: ContainerType, query: NQLQuery) -> Table:
+    def _fetch_entries(
+        self, child_type: ContainerType, query: NQLQuery, columns: Optional[Iterable[str]]
+    ) -> Table:
+        if columns is not None:
+            # always return entries with `sys/id` column when filter applied
+            columns = set(columns)
+            columns.add("sys/id")
+
         leaderboard_entries = self._backend.search_leaderboard_entries(
             project_id=self._project_id,
             types=[child_type],
             query=query,
+            columns=columns,
         )
 
         return Table(

neptune/new/metadata_containers/model.py

@@ -13,6 +13,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 #
+from typing import Iterable, Optional
+
 from neptune.new.internal.backends.nql import (
     NQLAttributeOperator,
     NQLAttributeType,
@@ -55,8 +57,9 @@ def _url(self) -> str:
     def _metadata_url(self) -> str:
         return self._url.rstrip("/") + "/metadata"
 
-    def fetch_model_versions_table(self) -> Table:
+    def fetch_model_versions_table(self, columns: Optional[Iterable[str]] = None) -> Table:
         """Retrieve all model versions of the given model.
+        TODO: columns parameter docs
 
         Returns:
             ``Table``: object containing experiments matching the specified criteria.
@@ -72,4 +75,5 @@ def fetch_model_versions_table(self) -> Table:
                 operator=NQLAttributeOperator.EQUALS,
                 type=NQLAttributeType.STRING,
             ),
+            columns=columns,
         )