- Add mlmd_resolver as a wrapper later upon metadata_store.
- Extend GetLineageSubgraph API to support returning
associationsandattributions.
- N/A
- Deprecate GetLineageGraph API.
- Deprecate OSS support on Windows OS platform.
- Deprecated python 3.8 support.
- Bumped minimum bazel version to 6.1.0.
- Deprecate types.py support.
- Depends on
attrs>=20.3,<24. - Depends on
protobuf>=4.25.2,<5for Python 3.11 and onprotobuf>3.20.3,<5for 3.9 and 3.10.
- Support PostgreSQL database type.
- Support bool_value in (custom_)property filter queries.
- Add masking support for Artifact / Execution / Context updates
- Support using enum names in IN operator in filter queries.
- Support populating ArtifactTypes for GetArtifactByID API.
- Add GetLineageSubgraph API for efficient lineage tracing.
- N/A
- N/A
- Bumped minimum bazel version to 5.3.0.
- Upgrade Microsoft Visual Studio (MSVC) version to 2017.
- Support filtering by parent/child context id in filter queries.
- Add batch queries for retrieving attributions/associations by artifact/execution ids in query executor layer and deprecate the original SelectAttributionByArtifactID and SelectAssociationByExecutionID functions.
- N/A
- N/A
- N/A
- Fix filter query parsing for QueryLineageGraph when using queries with aliases.
- Depends on
numpy~=1.22.0.
- N/A
- Deprecated python 3.7 support.
- N/A
- N/A
- N/A
- Introduce methods to Create and Update Artifacts, Executions and Contexts with custom create and update timestamp.
- Introduce option to always update node's
last_update_time_since_epocheven if update request matches stored node. ## Bug Fixes and Other Changes - Filter support for list Context with Artifact alias
artifacts_0and Execution aliasexecutions_0. - Enclose
FilterQueryBuilder::GetWhereClause()return value in parentheses to ensure filter query will be evaluated in the correct order. - Upgrade SQLite version to 3.39.2 to support more advanced SQL statements, e.g. using tuples with IN operator.
- Adds
external_idfor Type, Artifact, Execution and Context to store unique string ids from other systems. - Implements a fat client that supports v7, v8 and v9 schema for MLMD.
- Upgrades MLMD schema version to 10.
- Add
proto_valueandbool_valuecolumns forArtifactProperty,ExecutionProperty,ContextProperty. Theproto_valuecolumns store protocol buffer types (https://developers.google.com/protocol-buffers)
- Add
- Implement
UpsertTypes()with batch queries in metadata store.
- N/A
- N/A
- N/A
- Adds utils in shared C++ lib to delete ParentContexts by parent_context_id and child_context_ids.
- Depends on
attrs>=20.3,<22.
- N/A
- N/A
- N/A
- N/A
- N/A
- N/A
- From this version we will be releasing python 3.9 wheels.
- Updates Zlib to 1.2.12.
- Adds .bazelversion to support Bazelisk.
- N/A
- N/A
- Upgrade DB schema to V8, including the following changes:
- Add constraint on Event table on (artifact_id, execution_id, type). If duplicated events are created, already exists error will be returned.
- Add index for properties (for both property name and value).
- N/A
- N/A
- N/A
- Apply max_node_size to GetLineageGraphRequest for returned nodes.
- Skip querying type properties when not needed.
- Added type information file py.typed according to PEP-561.
- Depends on
absl-py>=0.9,<2.0.0.
- N/A
- N/A
- N/A
- Remove INFO log when a type is has empty property definition and not a simple type.
- N/A
- N/A
- Introduce MySQLSourceErrorInfo message to metadata_source proto to support adding additional error information. MySQL implementation adds MySQL error numbers as part MySQLSourceErrorInfo payload message.
- Open source GetLineageGraph() to MLMD OSS. MLMD OSS users now can use GetLineageGraph() to access MLMD lineage.
- N/A
- N/A
- Deprecated python 3.6 support.
- Introduces
base_typefield to ArtifactType, ExecutionType and ConteextType messages in metadata_store.proto. - Modifies PutArtifactType, PutExecutionType and PutTypes APIs to support
creation of type inheritance link; modifies GetArtifactType(s) and
GetExecutionType(s) APIs to return
base_typefield values as well.
- Adds support for PutTypes in the gRPC server.
- N/A
- N/A
- Open sources declarative nodes filtering with zetaSQL. It is currently supported on Linux and MacOS 10.14+.
- Extends get_artifacts, get_executions, get_contexts APIs with filtering capabilities on properties and 1-hop neighborhood nodes.
- Supports configure GRPC options
grpc.http2.max_ping_strikesfrom the python client.
- Introduces
database_name_field for MySQL MetadataSource implementation to enable MySQLMetadataSource to switch to different databases within the same SQL server after connection or during reconnection.
- The minimum required OS version for the macOS is 10.14 now.
- Bumped the minimum bazel version required to build
ml_metadatato 3.7.2.
- N/A
- Introduced
skip_db_creationfor MySQL backend. It is useful when db creation is handled by an admin process, while the lib users should not issue db creation clauses. - Adds utils in shared C++ lib to delete Contexts, Artifacts, Executions, and their corresponding edges (e.g., Associations, Attributions, ParentContexts).
- Removes TensorFlow dependency in MLMD OSS C++ and TensorFlow bazel WORKSPACE dependency in MLMD OSS bazel WORKSPACE.
- Upgrades MLMD schema version to 7.
- Add
byte_valuecolumn forArtifactProperty,ExecutionProperty,ContextPropertyfor better storing binary property values. - For MySQL backend, modify
string_valueto useMEDIUMTEXTinstead ofTEXTto persist property value upto 16MB. - Add index
EventPath.idx_eventpath_event_idfor Event retrieval queries.
- Add
- Depends on
protobuf>=3.13,<4. - Fixes a bug of increasing size of
next_page_tokenover pagination calls. - Returns InvalidArgumentError for PutParentContexts when creating a cyclic parent context.
- N/A
- N/A
- Implements ParentContext creation and retrieval in MLMD python and gRPC APIs.
- When
reuse_context_if_already_existis used for PutExecution, returns Aborted instead of AlreadyExists to improve the API contract when there are data races to create the same Context for the first time.
- Predefines a list of types (MLMD SimpleTypes) when connecting a MLMD instance. If the types already exist, it is a no-op, otherwise, it creates the types. This change may break tests which assume the number of types exist in a MLMD instance.
- N/A
- Supports versioned type creation and listing in python APIs for non-backward compatible type evolutions.
- Supports listing nodes by type with version in MLMD APIs.
- N/A
- N/A
- Drops
create_artifact_with_typefrom MetadataStore python class.
- N/A
- Uses Pybind11 instead of SWIG to wrap C++ libraries for MLMD python client.
- Depends on
absl-py>=0.9,<0.13.
- N/A
- N/A
- Extends type identifier to be (name, version). The version is optional and backward compatible with existing usage.
- Supports versioned creation and listing of types in gRPC for non-backward compatible type evolutions.
- N/A
- N/A
- N/A
- Introduce
google.protobuf.Structas an additionalValuetype for storing complex, JSON-objects in MLMD.
- MLMD wheel for MacOS is now built with minimum OS version 10.9. This will improve compatibility for the older MacOS versions.
- N/A
- N/A
- Exposes limit and order_by parameters to get_artifacts, get_executions and get_contexts API in python MetadataStore client to allow users to specify the maximum results to retrieve and the field the results to be ordered by.
- Adds
attrsas py client dependency.
- N/A
- N/A
- Adds
reuse_context_if_already_existoption toput_executionpython API to better support concurrent execution publishing with the same new context. - Supports pagination and ordering options in GetExecutionsByContext and GetArtifactsByContext APIs.
- N/A
- N/A
- N/A
-
Supports MetadataStoreClientConfig options
client_timeout_secfrom the python client. The grpc APIs would return DeadlineExceededError when server does not respond withinclient_timeout_ms. -
From this release MLMD will also be hosting nightly packages on https://pypi-nightly.tensorflow.org. To install the nightly package use the following command:
pip install --extra-index-url https://pypi-nightly.tensorflow.org/simple ml-metadataNote: These nightly packages are unstable and breakages are likely to happen. The fix could often take a week or more depending on the complexity involved for the wheels to be available on the PyPI cloud service. You can always use the stable version of MLMD available on PyPI by running the command
pip install ml-metadata. -
Upgrades MLMD schema version to 6.
- Add
ParentTypetable for supporting type inheritance. - Add
Type.versioncolumn for Type evolution development. - Add
Type.idx_type_nameindex for type lookup APIs. - Add
Type.descriptioncolumn for capturing static information about Type. - Add
ParentContexttable for supporting context parental relationship. - Add
Artifact.idx_artifact_urifor filtering artifacts by uri. - Add
Event.idx_event_artifact_idandidx_event_execution_idfor lineage traversal APIs. - Add indices on
create_time_since_epoch,last_update_time_since_epochforArtifact,ExecutionandContextfor sorted listing queries.
- Add
-
Allows omitting stored properties when using
put_artifact_type,put_execution_type,put_context_type, to help writing forward compatibility MLMD type registration calls.
- Optimizes GetContext*/GetArtifact*/GetExecution* and corresponding List* calls to reduce number of backend queries.
- Documentation fixes for QueryExecutor methods.
- N/A
- Deprecates
all_fields_matchandcan_delete_fieldsfrom python APIsput_artifact_type,put_execution_type,put_context_type. In previous releases these parameters can only be set with default values, otherwise Unimplemented error returns. This change should be no-op for all existing users.
- Improves building wheels from source with
setup.py. - Supports configure GRPC options
max_receive_message_lengthfrom the python client. - Adds python 3.8 support.
- Adds
grpcioas py client dependency. - Replaces the C++ MOCK_METHOD
<n>family of macros with the new MOCK_METHOD - Updates node's
last_update_time_since_epochwhen changing (custom)properties. - Disables incompatible Golang BUILD targets temporarily.
- Support GetArtifactByTypeAndName, GetExecutionByTypeAndName, GetContextByTypeAndName Go API
- N/A
- Deprecated py3.5 support
- GetArtifacts, GetExecutions and GetContexts now supports pagination and ordering results by ID, Create time and last update time fields.
- Python MetadataStore now exposes get_artifact_by_type_and_name and get_execution_by_type_and_name methods.
- Improves query performance of get_events_by_execution_ids and get_events_by_artifact_ids by combing multiple queries.
- Drops python dependency on tensorflow to make ml-metadata be friendly with non-TFX use cases.
- Python MetadataStore APIs return mlmd errors instead of tensorflow errors.
- Note: We plan to remove Python 3.5 support after this release.
- Uses metadata_store per request for grpc server to improve scalability.
- Uses Iterable[int] instead Sequence[int] for listing APIs accepting ids.
- Depends on
tensorflow>=1.15,!=2.0.*,<3
- Drops Python 2 support and stops releasing py2 wheels.
- Upgrades MLMD schema version to 5.
- Added state columns to persistent Artifact.state, Execution.last_known_state
- Added user-given unique name per type column to Artifact and Execution.
- Added create_time_since_epoch, last_update_time_since_epoch to all Nodes.
- Provides GetArtifactByTypeAndName and GetExecutionByTypeAndName API to get artifact/execution by type and name.
- Refactors transaction executions using TransactionExecutor class.
- Supports storing/retrieving Artifact.state and Execution.last_known_state.
- Returns explicit InvalidArgument for get_artifacts_by_uri from 0.15.x clients when using 0.21.0+ server.
- Updates logging level for python mlmd client.
- Refactoring MetadataAccessObject to allow for more flexibility.
- Release a script to generate Python API documentation.
- GetArtifacts/Executions/Contexts returns OK instead of NotFound to align with other listing APIs.
- Handles mysql stale connection error 2006 by client-side reconnection.
- Handles mysql innodb deadlock error (1213) and lock timeout (1205) via client-side retry.
- Avoids update node or properties without changes.
- Adding artifact states.
- Supporting connection retries to gRPC server.
- Allowing the Python API put_execution to update or insert related contexts.
- Adding a new execution state: CANCELED. This indicates an execution being canceled.
- Adding two event types: INTERNAL_INPUT and INTERNAL_OUTPUT indended to be used by mlmd powered systems (e.g., orchestrator).
- Add support to pass migration options as command line parameters to the MLMD gRPC server.
- Adding a new Python API get_context_by_type_and_name to allow querying a context by its type and context name at the same time.
- Refactoring MetadataAccessObject to allow for more flexibility.
- The Python API put_execution will need an extra input argument to pass in contexts and return updated context_ids. Users using the old API could pass in None or an empty list as context and add another variable to hold the returned context_ids to migrate.
- Passes bytes instead of string to grpc.ssl_channel_credentials.
- Align GRPC python client stub error code with swig client error code.
- Add verify_server_cert support to MySQL source SSL options.
- Add migration options to gRPC MetadataStoreServerConfig.
- Disable auto schema migration by default during connection. The user needs to explicitly enable it when connecting an older database.
- Support SSL options when using MySQL metadata source.
- Fixes MySQL errors with concurrent connection to an empty database. Now, MLMD returns Aborted when concurrent connection error happens and the caller can retry appropriately.
- Deprecates proto field MigrationOptions.disable_upgrade_migration.
- Deprecates
disable_upgrade_migrationin python MetadataStore constructor.
- Add Dockerfile for building a containerized version of the MLMD gRPC server.
- Add support for connecting to a MYSQL Metadata Source via Unix sockets.
- Add support to pass mysql connection configuration as command line parameters to the MLMD gRPC server.
- Provides the ability for metadata_store.py to communicate with the MLMD gRPC server, as an alternative to connecting directly with a database.
- Supports Sqlite for Windows and adds scripts to build wheels for python3 in Windows.
- Provides GetContextTypes to list all Context Types.
- MLMD ConnectionConfig provides an option to disable an automatic upgrade.
- Supports downgrade of the database schema version to older versions.
- Depended on
tensorflow>=1.15,<3- Starting from 1.15, package
tensorflowcomes with GPU support. Users won't need to choose betweentensorflowandtensorflow-gpu. - Caveat:
tensorflow2.0.0 is an exception and does not have GPU support. Iftensorflow-gpu2.0.0 is installed before installingml_metadata, it will be replaced withtensorflow2.0.0. Re-installtensorflow-gpu2.0.0 if needed.
- Starting from 1.15, package
- Add Context and ContextType to MLMD data model, which are used for capturing grouping concepts (e.g., Project, Pipeline, Owner, etc) of Artifacts and Executions.
- Add CACHED state to Execution state enum to model an execution that is skipped due to cached results.
- Add the ability to list all instances of ArtifactType and ExecutionType.
- Support Type update and enforce backward compatibility.
- Support atomic creation and publishing of an execution.
- Support building a manylinux2010 compliant wheel in docker so that it works in other linux OS outside of Ubuntu.
- Provide MLMD migration scheme to migrate out-of-date MLMD instance.
- Support creating and querying ContextType.
- Support creating, updating and querying Context instances.
- Support grouping artifact and executions into contexts.
- Support querying related artifacts and executions through contexts.
- Established ML Metadata as a standalone package.
- Provides a way to store information about how each artifact (e.g. file) was generated.
- Provides tools for determining provenance.