diff --git a/docs/src/main/sphinx/connector/iceberg.rst b/docs/src/main/sphinx/connector/iceberg.rst index 20b78192e2ec..07f1e13556f7 100644 --- a/docs/src/main/sphinx/connector/iceberg.rst +++ b/docs/src/main/sphinx/connector/iceberg.rst @@ -42,119 +42,8 @@ To use Iceberg, you need: * Network access from the Trino coordinator to the HMS. Hive metastore access with the Thrift protocol defaults to using port 9083. -Configuration -------------- - -The connector supports multiple Iceberg catalog types, you may use either a Hive -metastore service (HMS), AWS Glue, or a REST catalog. The catalog type is determined by the -``iceberg.catalog.type`` property, it can be set to ``HIVE_METASTORE``, ``GLUE``, ``JDBC``, or ``REST``. - - -.. _iceberg-hive-catalog: - -Hive metastore catalog -^^^^^^^^^^^^^^^^^^^^^^ - -The Hive metastore catalog is the default implementation. -When using it, the Iceberg connector supports the same metastore -configuration properties as the Hive connector. At a minimum, -``hive.metastore.uri`` must be configured, see -:ref:`Thrift metastore configuration`. - -.. code-block:: text - - connector.name=iceberg - hive.metastore.uri=thrift://localhost:9083 - -.. _iceberg-glue-catalog: - -Glue catalog -^^^^^^^^^^^^ - -When using the Glue catalog, the Iceberg connector supports the same -configuration properties as the Hive connector's Glue setup. See -:ref:`AWS Glue metastore configuration`. - -.. code-block:: text - - connector.name=iceberg - iceberg.catalog.type=glue - -.. _iceberg-rest-catalog: - -REST catalog -^^^^^^^^^^^^^^ - -In order to use the Iceberg REST catalog, ensure to configure the catalog type with -``iceberg.catalog.type=rest`` and provide further details with the following -properties: - -============================================== ============================================================ -Property Name Description -============================================== ============================================================ -``iceberg.rest-catalog.uri`` REST server API endpoint URI (required). - Example: ``http://iceberg-with-rest:8181`` - -``iceberg.rest-catalog.warehouse`` Warehouse identifier/location for the catalog (optional). - Example: ``s3://my_bucket/warehouse_location`` - -``iceberg.rest-catalog.security`` The type of security to use (default: ``NONE``). ``OAUTH2`` - requires either a ``token`` or ``credential``. - Example: ``OAUTH2`` - -``iceberg.rest-catalog.session`` Session information included when communicating with the REST Catalog. - Options are ``NONE`` or ``USER`` (default: ``NONE``). - -``iceberg.rest-catalog.oauth2.token`` The Bearer token which will be used for interactions - with the server. A ``token`` or ``credential`` is required for - ``OAUTH2`` security. - Example: ``AbCdEf123456`` - -``iceberg.rest-catalog.oauth2.credential`` The credential to exchange for a token in the OAuth2 client - credentials flow with the server. A ``token`` or ``credential`` - is required for ``OAUTH2`` security. - Example: ``AbCdEf123456`` -============================================== ============================================================ - -.. code-block:: text - - connector.name=iceberg - iceberg.catalog.type=rest - iceberg.rest-catalog.uri=http://iceberg-with-rest:8181 - -REST catalog does not support :doc:`views` or -:doc:`materialized views`. - -.. _iceberg-jdbc-catalog: - -JDBC catalog -^^^^^^^^^^^^ - -.. warning:: - - The JDBC catalog may face the compatibility issue if Iceberg introduces breaking changes in the future. - Consider the :ref:`REST catalog ` as an alternative solution. - -At a minimum, ``iceberg.jdbc-catalog.driver-class``, ``iceberg.jdbc-catalog.connection-url`` and -``iceberg.jdbc-catalog.catalog-name`` must be configured. -When using any database besides PostgreSQL, a JDBC driver jar file must be placed in the plugin directory. - -.. code-block:: text - - connector.name=iceberg - iceberg.catalog.type=jdbc - iceberg.jdbc-catalog.catalog-name=test - iceberg.jdbc-catalog.driver-class=org.postgresql.Driver - iceberg.jdbc-catalog.connection-url=jdbc:postgresql://example.net:5432/database - iceberg.jdbc-catalog.connection-user=admin - iceberg.jdbc-catalog.connection-password=test - iceberg.jdbc-catalog.default-warehouse-dir=s3://bucket - -JDBC catalog does not support :doc:`views` or -:doc:`materialized views`. - General configuration -^^^^^^^^^^^^^^^^^^^^^ +--------------------- These configuration properties are independent of which catalog implementation is used. @@ -175,7 +64,7 @@ is used. * ``AVRO`` - ``ORC`` * - ``iceberg.compression-codec`` - - The compression codec to be used when writing files. + - The compression codec used when writing files. Possible values are * ``NONE`` @@ -186,7 +75,7 @@ is used. - ``ZSTD`` * - ``iceberg.use-file-size-from-metadata`` - Read file sizes from metadata instead of file system. - This property should only be set as a workaround for + This property must only be used as a workaround for `this issue `_. The problem was fixed in Iceberg version 0.11.0. - ``true`` @@ -203,7 +92,7 @@ is used. - Maximum duration to wait for completion of dynamic filters during split generation. - ``0s`` * - ``iceberg.delete-schema-locations-fallback`` - - Whether schema locations should be deleted when Trino can't determine whether they contain external files. + - Whether schema locations are deleted when Trino can't determine whether they contain external files. - ``false`` * - ``iceberg.minimum-assigned-split-weight`` - A decimal value in the range (0, 1] used as a minimum for weights assigned to each split. A low value may improve performance @@ -227,414 +116,129 @@ is used. - Schema for creating materialized views storage tables. When this property is not configured, storage tables are created in the same schema as the materialized view definition. When the ``storage_schema`` materialized - view property is specified, it takes precedence over this catalog property. - - Empty - * - ``iceberg.register-table-procedure.enabled`` - - Enable to allow user to call ``register_table`` procedure - - ``false`` - -ORC format configuration -^^^^^^^^^^^^^^^^^^^^^^^^ - -The following properties are used to configure the read and write operations -with ORC files performed by the Iceberg connector. - -.. list-table:: ORC format configuration properties - :widths: 30, 58, 12 - :header-rows: 1 - - * - Property name - - Description - - Default - * - ``hive.orc.bloom-filters.enabled`` - - Enable bloom filters for predicate pushdown. - - ``false`` - -Parquet format configuration -^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The following properties are used to configure the read and write operations -with Parquet files performed by the Iceberg connector. - -.. list-table:: Parquet format configuration properties - :widths: 30, 50, 20 - :header-rows: 1 - - * - Property Name - - Description - - Default - * - ``parquet.max-read-block-row-count`` - - Sets the maximum number of rows read in a batch. - - ``8192`` - * - ``parquet.optimized-reader.enabled`` - - Whether batched column readers should be used when reading Parquet files - for improved performance. Set this property to ``false`` to disable the - optimized parquet reader by default. The equivalent catalog session - property is ``parquet_optimized_reader_enabled``. - - ``true`` - * - ``parquet.optimized-nested-reader.enabled`` - - Whether batched column readers should be used when reading ARRAY, MAP - and ROW types from Parquet files for improved performance. Set this - property to ``false`` to disable the optimized parquet reader by default - for structural data types. The equivalent catalog session property is - ``parquet_optimized_nested_reader_enabled``. - - ``true`` - -.. _iceberg-authorization: - -Authorization checks -^^^^^^^^^^^^^^^^^^^^ - -You can enable authorization checks for the connector by setting -the ``iceberg.security`` property in the catalog properties file. This -property must be one of the following values: - -.. list-table:: Iceberg security values - :widths: 30, 60 - :header-rows: 1 - - * - Property value - - Description - * - ``ALLOW_ALL`` - - No authorization checks are enforced. - * - ``SYSTEM`` - - The connector relies on system-level access control. - * - ``READ_ONLY`` - - Operations that read data or metadata, such as :doc:`/sql/select` are - permitted. No operations that write data or metadata, such as - :doc:`/sql/create-table`, :doc:`/sql/insert`, or :doc:`/sql/delete` are - allowed. - * - ``FILE`` - - Authorization checks are enforced using a catalog-level access control - configuration file whose path is specified in the ``security.config-file`` - catalog configuration property. See - :ref:`catalog-file-based-access-control` for information on the - authorization configuration file. - -.. _iceberg-table-redirection: - -Table redirection ------------------ - -.. include:: table-redirection.fragment - -The connector supports redirection from Iceberg tables to Hive tables -with the ``iceberg.hive-catalog-name`` catalog configuration property. - -.. _iceberg-sql-support: - -SQL support ------------ - -This connector provides read access and write access to data and metadata in -Iceberg. In addition to the :ref:`globally available ` -and :ref:`read operation ` statements, the connector -supports the following features: - -* :ref:`sql-write-operations`: - - * :ref:`iceberg-schema-table-management` and :ref:`iceberg-tables` - * :ref:`iceberg-data-management` - * :ref:`sql-view-management` - * :ref:`sql-materialized-view-management`, see also :ref:`iceberg-materialized-views` - -.. _iceberg-schema-table-management: - -Schema and table management -^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The :ref:`sql-schema-table-management` functionality includes support for: - -* :doc:`/sql/create-schema` -* :doc:`/sql/drop-schema` -* :doc:`/sql/alter-schema` -* :doc:`/sql/create-table` -* :doc:`/sql/create-table-as` -* :doc:`/sql/drop-table` -* :doc:`/sql/alter-table` -* :doc:`/sql/comment` - -.. _iceberg-create-schema: - -CREATE SCHEMA -~~~~~~~~~~~~~ - -The connector supports creating schemas. You can create a schema with or without -a specified location. - -You can create a schema with the :doc:`/sql/create-schema` statement and the -``location`` schema property. The tables in this schema, which have no explicit -``location`` set in :doc:`/sql/create-table` statement, are located in a -subdirectory under the directory corresponding to the schema location. - -Create a schema on S3:: - - CREATE SCHEMA example.example_s3_schema - WITH (location = 's3://my-bucket/a/path/'); - -Create a schema on a S3 compatible object storage such as MinIO:: - - CREATE SCHEMA example.example_s3a_schema - WITH (location = 's3a://my-bucket/a/path/'); - -Create a schema on HDFS:: - - CREATE SCHEMA example.example_hdfs_schema - WITH (location='hdfs://hadoop-master:9000/user/hive/warehouse/a/path/'); - -Optionally, on HDFS, the location can be omitted:: - - CREATE SCHEMA example.example_hdfs_schema; - -.. _iceberg-create-table: - -Creating tables -~~~~~~~~~~~~~~~ - -The Iceberg connector supports creating tables using the :doc:`CREATE -TABLE ` syntax. Optionally specify the -:ref:`table properties ` supported by this connector:: - - CREATE TABLE example_table ( - c1 integer, - c2 date, - c3 double - ) - WITH ( - format = 'PARQUET', - partitioning = ARRAY['c1', 'c2'], - sorted_by = ARRAY['c3'], - location = 's3://my-bucket/a/path/' - ); - -When the ``location`` table property is omitted, the content of the table -is stored in a subdirectory under the directory corresponding to the -schema location. - -The Iceberg connector supports creating tables using the :doc:`CREATE -TABLE AS ` with :doc:`SELECT ` syntax:: - - CREATE TABLE tiny_nation - WITH ( - format = 'PARQUET' - ) - AS - SELECT * - FROM nation - WHERE nationkey < 10; - -Another flavor of creating tables with :doc:`CREATE TABLE AS ` -is with :doc:`VALUES ` syntax:: - - CREATE TABLE yearly_clicks ( - year, - clicks - ) - WITH ( - partitioning = ARRAY['year'] - ) - AS VALUES - (2021, 10000), - (2022, 20000); - -``NOT NULL`` column constraint -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -The Iceberg connector supports setting ``NOT NULL`` constraints on the table columns. - -The ``NOT NULL`` constraint can be set on the columns, while creating tables by -using the :doc:`CREATE TABLE ` syntax:: - - CREATE TABLE example_table ( - year INTEGER NOT NULL, - name VARCHAR NOT NULL, - age INTEGER, - address VARCHAR - ); - -When trying to insert/update data in the table, the query fails if trying -to set ``NULL`` value on a column having the ``NOT NULL`` constraint. - -DROP TABLE -~~~~~~~~~~ - -The Iceberg connector supports dropping a table by using the :doc:`/sql/drop-table` -syntax. When the command succeeds, both the data of the Iceberg table and also the -information related to the table in the metastore service are removed. -Dropping tables which have their data/metadata stored in a different location than -the table's corresponding base directory on the object store is not supported. - -.. _iceberg-comment: - -.. _iceberg-alter-table-execute: - -ALTER TABLE EXECUTE -~~~~~~~~~~~~~~~~~~~ - -The connector supports the following commands for use with -:ref:`ALTER TABLE EXECUTE `. - -optimize -"""""""" - -The ``optimize`` command is used for rewriting the active content -of the specified table so that it is merged into fewer but -larger files. -In case that the table is partitioned, the data compaction -acts separately on each partition selected for optimization. -This operation improves read performance. - -All files with a size below the optional ``file_size_threshold`` -parameter (default value for the threshold is ``100MB``) are -merged: - -.. code-block:: sql - - ALTER TABLE test_table EXECUTE optimize - -The following statement merges the files in a table that -are under 10 megabytes in size: - -.. code-block:: sql - - ALTER TABLE test_table EXECUTE optimize(file_size_threshold => '10MB') - -You can use a ``WHERE`` clause with the columns used to partition -the table, to apply ``optimize`` only on the partition(s) corresponding -to the filter: - -.. code-block:: sql - - ALTER TABLE test_partitioned_table EXECUTE optimize - WHERE partition_key = 1 - -expire_snapshots -"""""""""""""""" - -The ``expire_snapshots`` command removes all snapshots and all related metadata and data files. -Regularly expiring snapshots is recommended to delete data files that are no longer needed, -and to keep the size of table metadata small. -The procedure affects all snapshots that are older than the time period configured with the ``retention_threshold`` parameter. - -``expire_snapshots`` can be run as follows: - -.. code-block:: sql - - ALTER TABLE test_table EXECUTE expire_snapshots(retention_threshold => '7d') - -The value for ``retention_threshold`` must be higher than or equal to ``iceberg.expire_snapshots.min-retention`` in the catalog -otherwise the procedure will fail with similar message: -``Retention specified (1.00d) is shorter than the minimum retention configured in the system (7.00d)``. -The default value for this property is ``7d``. - -remove_orphan_files -""""""""""""""""""" - -The ``remove_orphan_files`` command removes all files from table's data directory which are -not linked from metadata files and that are older than the value of ``retention_threshold`` parameter. -Deleting orphan files from time to time is recommended to keep size of table's data directory under control. - -``remove_orphan_files`` can be run as follows: - -.. code-block:: sql - - ALTER TABLE test_table EXECUTE remove_orphan_files(retention_threshold => '7d') - -The value for ``retention_threshold`` must be higher than or equal to ``iceberg.remove_orphan_files.min-retention`` in the catalog -otherwise the procedure will fail with similar message: -``Retention specified (1.00d) is shorter than the minimum retention configured in the system (7.00d)``. -The default value for this property is ``7d``. - -.. _drop-extended-stats: - -drop_extended_stats -~~~~~~~~~~~~~~~~~~~ - -The ``drop_extended_stats`` command removes all extended statistics information from -the table. - -``drop_extended_stats`` can be run as follows: - -.. code-block:: sql + view property is specified, it takes precedence over this catalog property. + - Empty + * - ``iceberg.register-table-procedure.enabled`` + - Enable to allow user to call ``register_table`` procedure + - ``false`` - ALTER TABLE test_table EXECUTE drop_extended_stats +Metastores +---------- -.. _iceberg-alter-table-set-properties: +The Iceberg table format manages most metadata in metadata files in the object +storage itself. A small amount of metadata, however, still requires the use of a +metastore. In the Iceberg ecosystem, these smaller metastores are called Iceberg +metadata catalogs, or just catalogs. The examples in each subsection depict the +contents of a Trino catalog file that uses the the Iceberg connector to +configures different Iceberg metadata catalogs. -ALTER TABLE SET PROPERTIES -~~~~~~~~~~~~~~~~~~~~~~~~~~ +The connector supports multiple Iceberg catalog types; you may use +either a Hive metastore service (HMS), AWS Glue, or a REST catalog. The catalog +type is determined by the ``iceberg.catalog.type`` property. It can be set to +``HIVE_METASTORE``, ``GLUE``, ``JDBC``, or ``REST``. -The connector supports modifying the properties on existing tables using -:ref:`ALTER TABLE SET PROPERTIES `. +.. _iceberg-hive-catalog: -The following table properties can be updated after a table is created: +Hive metastore catalog +^^^^^^^^^^^^^^^^^^^^^^ -* ``format`` -* ``format_version`` -* ``partitioning`` -* ``sorted_by`` +The Hive metastore catalog is the default implementation. +When using it, the Iceberg connector supports the same metastore +configuration properties as the Hive connector. At a minimum, +``hive.metastore.uri`` must be configured, see +:ref:`Thrift metastore configuration`. -For example, to update a table from v1 of the Iceberg specification to v2: +.. code-block:: text -.. code-block:: sql + connector.name=iceberg + hive.metastore.uri=thrift://localhost:9083 - ALTER TABLE table_name SET PROPERTIES format_version = 2; +.. _iceberg-glue-catalog: -Or to set the column ``my_new_partition_column`` as a partition column on a table: +Glue catalog +^^^^^^^^^^^^ -.. code-block:: sql +When using the Glue catalog, the Iceberg connector supports the same +configuration properties as the Hive connector's Glue setup. See +:ref:`AWS Glue metastore configuration`. - ALTER TABLE table_name SET PROPERTIES partitioning = ARRAY[, 'my_new_partition_column']; +.. code-block:: text -The current values of a table's properties can be shown using :doc:`SHOW CREATE TABLE `. + connector.name=iceberg + iceberg.catalog.type=glue -COMMENT -~~~~~~~ +.. _iceberg-rest-catalog: -The Iceberg connector supports setting comments on the following objects: +REST catalog +^^^^^^^^^^^^^^ -- tables -- views -- table columns +In order to use the Iceberg REST catalog, ensure to configure the catalog type with +``iceberg.catalog.type=rest`` and provide further details with the following +properties: -The ``COMMENT`` option is supported on both the table and -the table columns for the :doc:`/sql/create-table` operation. +.. list-table:: Iceberg REST catalog configuration properties + :widths: 40, 60 + :header-rows: 1 -The ``COMMENT`` option is supported for adding table columns -through the :doc:`/sql/alter-table` operations. + * - Property name + - Description + * - ``iceberg.rest-catalog.uri`` + - REST server API endpoint URI (required). + Example: ``http://iceberg-with-rest:8181`` + * - ``iceberg.rest-catalog.warehouse`` + - Warehouse identifier/location for the catalog (optional). + Example: ``s3://my_bucket/warehouse_location`` + * - ``iceberg.rest-catalog.security`` + - The type of security to use (default: ``NONE``). ``OAUTH2`` requires + either a ``token`` or ``credential``. Example: ``OAUTH2`` + * - ``iceberg.rest-catalog.session`` + - Session information included when communicating with the REST Catalog. + Options are ``NONE`` or ``USER`` (default: ``NONE``). + * - ``iceberg.rest-catalog.oauth2.token`` + - The bearer token used for interactions with the server. A + ``token`` or ``credential`` is required for ``OAUTH2`` security. + Example: ``AbCdEf123456`` + * - ``iceberg.rest-catalog.oauth2.credential`` + - The credential to exchange for a token in the OAuth2 client credentials + flow with the server. A ``token`` or ``credential`` is required for + ``OAUTH2`` security. Example: ``AbCdEf123456`` -The connector supports the command :doc:`COMMENT ` for setting -comments on existing entities. +.. code-block:: text -.. _iceberg-data-management: + connector.name=iceberg + iceberg.catalog.type=rest + iceberg.rest-catalog.uri=http://iceberg-with-rest:8181 -Data management -^^^^^^^^^^^^^^^ +REST catalog does not support :doc:`views` or +:doc:`materialized views`. -The :ref:`sql-data-management` functionality includes support for ``INSERT``, -``UPDATE``, ``DELETE``, and ``MERGE`` statements. +.. _iceberg-jdbc-catalog: -.. _iceberg-delete: +JDBC catalog +^^^^^^^^^^^^ -Deletion by partition -~~~~~~~~~~~~~~~~~~~~~ +.. warning:: -For partitioned tables, the Iceberg connector supports the deletion of entire -partitions if the ``WHERE`` clause specifies filters only on the identity-transformed -partitioning columns, that can match entire partitions. Given the table definition -from :ref:`Partitioned Tables ` section, -the following SQL statement deletes all partitions for which ``country`` is ``US``:: + The JDBC catalog may face the compatibility issue if Iceberg introduces breaking changes in the future. + Consider the :ref:`REST catalog ` as an alternative solution. - DELETE FROM example.testdb.customer_orders - WHERE country = 'US' +At a minimum, ``iceberg.jdbc-catalog.driver-class``, ``iceberg.jdbc-catalog.connection-url`` and +``iceberg.jdbc-catalog.catalog-name`` must be configured. +When using any database besides PostgreSQL, a JDBC driver jar file must be placed in the plugin directory. -A partition delete is performed if the ``WHERE`` clause meets these conditions. +.. code-block:: text -Row level deletion -~~~~~~~~~~~~~~~~~~ + connector.name=iceberg + iceberg.catalog.type=jdbc + iceberg.jdbc-catalog.catalog-name=test + iceberg.jdbc-catalog.driver-class=org.postgresql.Driver + iceberg.jdbc-catalog.connection-url=jdbc:postgresql://example.net:5432/database + iceberg.jdbc-catalog.connection-user=admin + iceberg.jdbc-catalog.connection-password=test + iceberg.jdbc-catalog.default-warehouse-dir=s3://bucket -Tables using v2 of the Iceberg specification support deletion of individual rows -by writing position delete files. +JDBC catalog does not support :doc:`views` or +:doc:`materialized views`. Type mapping ------------ @@ -751,155 +355,150 @@ this table: No other types are supported. -.. _iceberg-tables: - -Partitioned tables ------------------- - -Iceberg supports partitioning by specifying transforms over the table columns. -A partition is created for each unique tuple value produced by the transforms. -Identity transforms are simply the column name. Other transforms are: - -===================================== ==================================================================== -Transform Description -===================================== ==================================================================== -``year(ts)`` A partition is created for each year. The partition value is the - integer difference in years between ``ts`` and January 1 1970. - -``month(ts)`` A partition is created for each month of each year. The partition - value is the integer difference in months between ``ts`` and - January 1 1970. +Security +-------- -``day(ts)`` A partition is created for each day of each year. The partition - value is the integer difference in days between ``ts`` and - January 1 1970. +The Iceberg connector allows you to choose one of several means of providing +authorization at the catalog level. -``hour(ts)`` A partition is created hour of each day. The partition value - is a timestamp with the minutes and seconds set to zero. +.. _iceberg-authorization: -``bucket(x, nbuckets)`` The data is hashed into the specified number of buckets. The - partition value is an integer hash of ``x``, with a value between - 0 and ``nbuckets - 1`` inclusive. +Authorization checks +^^^^^^^^^^^^^^^^^^^^ -``truncate(s, nchars)`` The partition value is the first ``nchars`` characters of ``s``. -===================================== ==================================================================== +You can enable authorization checks for the connector by setting +the ``iceberg.security`` property in the catalog properties file. This +property must be one of the following values: -In this example, the table is partitioned by the month of ``order_date``, a hash of -``account_number`` (with 10 buckets), and ``country``:: +.. list-table:: Iceberg security values + :widths: 30, 60 + :header-rows: 1 - CREATE TABLE example.testdb.customer_orders ( - order_id BIGINT, - order_date DATE, - account_number BIGINT, - customer VARCHAR, - country VARCHAR) - WITH (partitioning = ARRAY['month(order_date)', 'bucket(account_number, 10)', 'country']) + * - Property value + - Description + * - ``ALLOW_ALL`` + - No authorization checks are enforced. + * - ``SYSTEM`` + - The connector relies on system-level access control. + * - ``READ_ONLY`` + - Operations that read data or metadata, such as :doc:`/sql/select` are + permitted. No operations that write data or metadata, such as + :doc:`/sql/create-table`, :doc:`/sql/insert`, or :doc:`/sql/delete` are + allowed. + * - ``FILE`` + - Authorization checks are enforced using a catalog-level access control + configuration file whose path is specified in the ``security.config-file`` + catalog configuration property. See + :ref:`catalog-file-based-access-control` for information on the + authorization configuration file. -Sorted tables -------------- +.. _iceberg-sql-support: -The connector supports sorted files as a performance improvement. Data is -sorted during writes within each file based on the specified array of one -or more columns. +SQL support +----------- -Sorting is particularly beneficial when the sorted columns show a -high cardinality and are used as a filter for selective reads. +This connector provides read access and write access to data and metadata in +Iceberg. In addition to the :ref:`globally available ` +and :ref:`read operation ` statements, the connector +supports the following features: -The sort order is configured with the ``sorted_by`` table property. -Specify an array of one or more columns to use for sorting -when creating the table. The following example configures the -``order_date`` column of the ``orders`` table in the ``customers`` -schema in the ``example`` catalog:: +* :ref:`sql-write-operations`: - CREATE TABLE example.customers.orders ( - order_id BIGINT, - order_date DATE, - account_number BIGINT, - customer VARCHAR, - country VARCHAR) - WITH (sorted_by = ARRAY['order_date']) + * :ref:`iceberg-schema-table-management` and :ref:`iceberg-tables` + * :ref:`iceberg-data-management` + * :ref:`sql-view-management` + * :ref:`sql-materialized-view-management`, see also :ref:`iceberg-materialized-views` -Sorting can be combined with partitioning on the same column. For example:: +Basic usage examples +^^^^^^^^^^^^^^^^^^^^ - CREATE TABLE example.customers.orders ( - order_id BIGINT, - order_date DATE, - account_number BIGINT, - customer VARCHAR, - country VARCHAR) - WITH ( - partitioning = ARRAY['month(order_date)'], - sorted_by = ARRAY['order_date'] - ) +The connector supports creating schemas. You can create a schema with or without +a specified location. -You can disable sorted writing with the session property -``sorted_writing_enabled`` set to ``false``. +You can create a schema with the :doc:`/sql/create-schema` statement and the +``location`` schema property. The tables in this schema, which have no explicit +``location`` set in :doc:`/sql/create-table` statement, are located in a +subdirectory under the directory corresponding to the schema location. -Rolling back to a previous snapshot ------------------------------------ +Create a schema on S3:: -Iceberg supports a "snapshot" model of data, where table snapshots are -identified by a snapshot ID. + CREATE SCHEMA example.example_s3_schema + WITH (location = 's3://my-bucket/a/path/'); -The connector provides a system table exposing snapshot information for every -Iceberg table. Snapshots are identified by ``BIGINT`` snapshot IDs. -For example, you could find the snapshot IDs for the ``customer_orders`` table -by running the following query:: +Create a schema on a S3 compatible object storage such as MinIO:: - SELECT snapshot_id - FROM example.testdb."customer_orders$snapshots" - ORDER BY committed_at DESC + CREATE SCHEMA example.example_s3a_schema + WITH (location = 's3a://my-bucket/a/path/'); -.. _iceberg-time-travel: +Create a schema on HDFS:: -Time travel queries -^^^^^^^^^^^^^^^^^^^ + CREATE SCHEMA example.example_hdfs_schema + WITH (location='hdfs://hadoop-master:9000/user/hive/warehouse/a/path/'); -The connector offers the ability to query historical data. -This allows you to query the table as it was when a previous snapshot -of the table was taken, even if the data has since been modified or deleted. +Optionally, on HDFS, the location can be omitted:: -The historical data of the table can be retrieved by specifying the -snapshot identifier corresponding to the version of the table that -needs to be retrieved:: + CREATE SCHEMA example.example_hdfs_schema; - SELECT * - FROM example.testdb.customer_orders FOR VERSION AS OF 8954597067493422955 +The Iceberg connector supports creating tables using the :doc:`CREATE +TABLE ` syntax. Optionally specify the +:ref:`table properties ` supported by this connector:: -A different approach of retrieving historical data is to specify -a point in time in the past, such as a day or week ago. The latest snapshot -of the table taken before or at the specified timestamp in the query is -internally used for providing the previous state of the table:: + CREATE TABLE example_table ( + c1 integer, + c2 date, + c3 double + ) + WITH ( + format = 'PARQUET', + partitioning = ARRAY['c1', 'c2'], + sorted_by = ARRAY['c3'], + location = 's3://my-bucket/a/path/' + ); - SELECT * - FROM example.testdb.customer_orders FOR TIMESTAMP AS OF TIMESTAMP '2022-03-23 09:59:29.803 Europe/Vienna' +When the ``location`` table property is omitted, the content of the table +is stored in a subdirectory under the directory corresponding to the +schema location. -Rolling back to a previous snapshot -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +The Iceberg connector supports creating tables using the :doc:`CREATE +TABLE AS ` with :doc:`SELECT ` syntax:: -Use the ``$snapshots`` metadata table to determine the latest snapshot ID of the table like in the following query:: + CREATE TABLE tiny_nation + WITH ( + format = 'PARQUET' + ) + AS + SELECT * + FROM nation + WHERE nationkey < 10; - SELECT snapshot_id - FROM example.testdb."customer_orders$snapshots" - ORDER BY committed_at DESC LIMIT 1 +Another flavor of creating tables with :doc:`CREATE TABLE AS ` +is with :doc:`VALUES ` syntax:: -The procedure ``system.rollback_to_snapshot`` allows the caller to roll back -the state of the table to a previous snapshot id:: + CREATE TABLE yearly_clicks ( + year, + clicks + ) + WITH ( + partitioning = ARRAY['year'] + ) + AS VALUES + (2021, 10000), + (2022, 20000); - CALL example.system.rollback_to_snapshot('testdb', 'customer_orders', 8954597067493422955) +Procedures +^^^^^^^^^^ -Schema evolution ----------------- +Use the :doc:`/sql/call` statement to perform data manipulation or +administrative tasks. Procedures are available in the system schema of each +catalog. The following code snippet displays how to call the +``example_procedure`` in the ``examplecatalog`` catalog:: -Iceberg supports schema evolution, with safe column add, drop, reorder -and rename operations, including in nested structures. -Table partitioning can also be changed and the connector can still -query data created before the partitioning change. + CALL examplecatalog.system.example_procedure() .. _iceberg-register-table: Register table --------------- +"""""""""""""" The connector can register existing Iceberg tables with the catalog. The procedure ``system.register_table`` allows the caller to register an @@ -921,7 +520,7 @@ The procedure is enabled only when ``iceberg.register-table-procedure.enabled`` .. _iceberg-unregister-table: Unregister table ----------------- +"""""""""""""""" The connector can unregister existing Iceberg tables from the catalog. The procedure ``system.unregister_table`` allows the caller to unregister an @@ -929,8 +528,8 @@ existing Iceberg table from the metastores without deleting the data:: CALL example.system.unregister_table(schema_name => 'testdb', table_name => 'customer_orders') -Migrating existing tables -------------------------- +Migrate table +""""""""""""" The connector can read from or write to Hive tables that have been migrated to Iceberg. An SQL procedure ``system.migrate`` allows the caller to replace @@ -946,39 +545,221 @@ if nested directory exists:: CALL iceberg.system.migrate(schema_name => 'testdb', table_name => 'customer_orders', recursive_directory => 'true') -.. _iceberg-table-properties: +.. _iceberg-data-management: + +Data management +^^^^^^^^^^^^^^^ + +The :ref:`sql-data-management` functionality includes support for ``INSERT``, +``UPDATE``, ``DELETE``, and ``MERGE`` statements. + +.. _iceberg-delete: + +Deletion by partition +""""""""""""""""""""" + +For partitioned tables, the Iceberg connector supports the deletion of entire +partitions if the ``WHERE`` clause specifies filters only on the identity-transformed +partitioning columns, that can match entire partitions. Given the table definition +from :ref:`Partitioned Tables ` section, +the following SQL statement deletes all partitions for which ``country`` is ``US``:: + + DELETE FROM example.testdb.customer_orders + WHERE country = 'US' + +A partition delete is performed if the ``WHERE`` clause meets these conditions. + +Row level deletion +"""""""""""""""""" + +Tables using v2 of the Iceberg specification support deletion of individual rows +by writing position delete files. + +.. _iceberg-schema-table-management: + +Schema and table management +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The :ref:`sql-schema-table-management` functionality includes support for: + +* :doc:`/sql/create-schema` +* :doc:`/sql/drop-schema` +* :doc:`/sql/alter-schema` +* :doc:`/sql/create-table` +* :doc:`/sql/create-table-as` +* :doc:`/sql/drop-table` +* :doc:`/sql/alter-table` +* :doc:`/sql/comment` + +Schema evolution +"""""""""""""""" + +Iceberg supports schema evolution, with safe column add, drop, reorder +and rename operations, including in nested structures. +Table partitioning can also be changed and the connector can still +query data created before the partitioning change. + +.. _iceberg-alter-table-execute: + +ALTER TABLE EXECUTE +""""""""""""""""""" + +The connector supports the following commands for use with +:ref:`ALTER TABLE EXECUTE `. + +optimize +~~~~~~~~ + +The ``optimize`` command is used for rewriting the active content +of the specified table so that it is merged into fewer but +larger files. +In case that the table is partitioned, the data compaction +acts separately on each partition selected for optimization. +This operation improves read performance. + +All files with a size below the optional ``file_size_threshold`` +parameter (default value for the threshold is ``100MB``) are +merged: + +.. code-block:: sql + + ALTER TABLE test_table EXECUTE optimize + +The following statement merges the files in a table that +are under 10 megabytes in size: + +.. code-block:: sql + + ALTER TABLE test_table EXECUTE optimize(file_size_threshold => '10MB') + +You can use a ``WHERE`` clause with the columns used to partition +the table, to apply ``optimize`` only on the partition(s) corresponding +to the filter: + +.. code-block:: sql + + ALTER TABLE test_partitioned_table EXECUTE optimize + WHERE partition_key = 1 -Iceberg table properties ------------------------- +expire_snapshots +~~~~~~~~~~~~~~~~ + +The ``expire_snapshots`` command removes all snapshots and all related metadata and data files. +Regularly expiring snapshots is recommended to delete data files that are no longer needed, +and to keep the size of table metadata small. +The procedure affects all snapshots that are older than the time period configured with the ``retention_threshold`` parameter. + +``expire_snapshots`` can be run as follows: + +.. code-block:: sql + + ALTER TABLE test_table EXECUTE expire_snapshots(retention_threshold => '7d') + +The value for ``retention_threshold`` must be higher than or equal to ``iceberg.expire_snapshots.min-retention`` in the catalog +otherwise the procedure fails with a similar message: +``Retention specified (1.00d) is shorter than the minimum retention configured in the system (7.00d)``. +The default value for this property is ``7d``. + +remove_orphan_files +~~~~~~~~~~~~~~~~~~~ + +The ``remove_orphan_files`` command removes all files from table's data directory which are +not linked from metadata files and that are older than the value of ``retention_threshold`` parameter. +Deleting orphan files from time to time is recommended to keep size of table's data directory under control. + +``remove_orphan_files`` can be run as follows: + +.. code-block:: sql + + ALTER TABLE test_table EXECUTE remove_orphan_files(retention_threshold => '7d') + +The value for ``retention_threshold`` must be higher than or equal to ``iceberg.remove_orphan_files.min-retention`` in the catalog +otherwise the procedure fails with a similar message: +``Retention specified (1.00d) is shorter than the minimum retention configured in the system (7.00d)``. +The default value for this property is ``7d``. + +.. _drop-extended-stats: + +drop_extended_stats +~~~~~~~~~~~~~~~~~~~ + +The ``drop_extended_stats`` command removes all extended statistics information from +the table. + +``drop_extended_stats`` can be run as follows: + +.. code-block:: sql + + ALTER TABLE test_table EXECUTE drop_extended_stats + +.. _iceberg-alter-table-set-properties: + +ALTER TABLE SET PROPERTIES +"""""""""""""""""""""""""" + +The connector supports modifying the properties on existing tables using +:ref:`ALTER TABLE SET PROPERTIES `. + +The following table properties can be updated after a table is created: + +* ``format`` +* ``format_version`` +* ``partitioning`` +* ``sorted_by`` + +For example, to update a table from v1 of the Iceberg specification to v2: + +.. code-block:: sql + + ALTER TABLE table_name SET PROPERTIES format_version = 2; + +Or to set the column ``my_new_partition_column`` as a partition column on a table: + +.. code-block:: sql + + ALTER TABLE table_name SET PROPERTIES partitioning = ARRAY[, 'my_new_partition_column']; + +The current values of a table's properties can be shown using :doc:`SHOW CREATE TABLE `. -================================================== ================================================================ -Property name Description -================================================== ================================================================ -``format`` Optionally specifies the format of table data files; - either ``PARQUET``, ``ORC`` or ``AVRO``. Defaults to ``ORC``. +.. _iceberg-table-properties: -``partitioning`` Optionally specifies table partitioning. - If a table is partitioned by columns ``c1`` and ``c2``, the - partitioning property would be - ``partitioning = ARRAY['c1', 'c2']`` +Table properties +~~~~~~~~~~~~~~~~ -``location`` Optionally specifies the file system location URI for - the table. +Table properties supply or set metadata for the underlying tables. This +is key for :doc:`/sql/create-table-as` statements. Table properties are passed +to the connector using a :doc:`WITH ` clause:: -``format_version`` Optionally specifies the format version of the Iceberg - specification to use for new tables; either ``1`` or ``2``. - Defaults to ``2``. Version ``2`` is required for row level deletes. + CREATE TABLE tablename + WITH (format='CSV', + csv_escape = '"') -``orc_bloom_filter_columns`` Comma separated list of columns to use for ORC bloom filter. - It improves the performance of queries using Equality and IN predicates - when reading ORC file. - Requires ORC format. - Defaults to ``[]``. +.. list-table:: Iceberg table properties + :widths: 40, 60 + :header-rows: 1 -``orc_bloom_filter_fpp`` The ORC bloom filters false positive probability. - Requires ORC format. - Defaults to ``0.05``. -================================================== ================================================================ + * - Property name + - Description + * - ``format`` + - Optionally specifies the format of table data files; either ``PARQUET``, + ``ORC`` or ``AVRO``. Defaults to ``ORC``. + * - ``partitioning`` + - Optionally specifies table partitioning. If a table is partitioned by + columns ``c1`` and ``c2``, the partitioning property is + ``partitioning = ARRAY['c1', 'c2']``. + * - ``location`` + - Optionally specifies the file system location URI for the table. + * - ``format_version`` + - Optionally specifies the format version of the Iceberg specification to + use for new tables; either ``1`` or ``2``. Defaults to ``2``. Version + ``2`` is required for row level deletes. + * - ``orc_bloom_filter_columns`` + - Comma separated list of columns to use for ORC bloom filter. It improves + the performance of queries using Equality and IN predicates when reading + ORC file. Requires ORC format. Defaults to ``[]``. + * - ``orc_bloom_filter_fpp`` + - The ORC bloom filters false positive probability. Requires ORC format. + Defaults to ``0.05``. The table definition below specifies format Parquet, partitioning by columns ``c1`` and ``c2``, and a file system location of ``/var/example_tables/test_table``:: @@ -1005,51 +786,33 @@ fpp is 0.05, and a file system location of ``/var/example_tables/test_table``:: orc_bloom_filter_columns = ARRAY['c1', 'c2'], orc_bloom_filter_fpp = 0.05) -.. _iceberg_metadata_columns: - -Metadata columns ----------------- - -In addition to the defined columns, the Iceberg connector automatically exposes -path metadata as a hidden column in each table: - -* ``$path``: Full file system path name of the file for this row - -* ``$file_modified_time``: Timestamp of the last modification of the file for this row - -You can use these columns in your SQL statements like any other column. This -can be selected directly, or used in conditional statements. For example, you -can inspect the file path for each record:: - - SELECT *, "$path", "$file_modified_time" - FROM example.web.page_views; - -Retrieve all records that belong to a specific file using ``"$path"`` filter:: - - SELECT * - FROM example.web.page_views - WHERE "$path" = '/usr/iceberg/table/web.page_views/data/file_01.parquet' - -Retrieve all records that belong to a specific file using ``"$file_modified_time"`` filter:: - - SELECT * - FROM example.web.page_views - WHERE "$file_modified_time" = CAST('2022-07-01 01:02:03.456 UTC' AS timestamp with time zone) - .. _iceberg-metadata-tables: Metadata tables ---------------- +""""""""""""""" The connector exposes several metadata tables for each Iceberg table. These metadata tables contain information about the internal structure of the Iceberg table. You can query each metadata table by appending the metadata table name to the table name:: - SELECT * FROM "test_table$properties" + SELECT * FROM "test_table$data" + +``$data`` table +~~~~~~~~~~~~~~~ + +The ``$data`` table is an alias for the Iceberg table itself. + +The statement:: + + SELECT * FROM "test_table$data" + +is equivalent to:: + + SELECT * FROM test_table ``$properties`` table -^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~ The ``$properties`` table provides access to general information about Iceberg table configuration and any additional metadata key/value pairs that the table @@ -1067,7 +830,7 @@ table ``test_table`` by using the following query:: write.format.default | PARQUET | ``$history`` table -^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~ The ``$history`` table provides a log of the metadata changes performed on the Iceberg table. @@ -1106,9 +869,8 @@ The output of the query has the following columns: - ``boolean`` - Whether or not this snapshot is an ancestor of the current snapshot - ``$snapshots`` table -^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~ The ``$snapshots`` table provides a detailed view of snapshots of the Iceberg table. A snapshot consists of one or more file manifests, @@ -1127,7 +889,6 @@ You can retrieve the information about the snapshots of the Iceberg table 2022-01-10 08:11:20 Europe/Vienna | 8667764846443717831 | | append | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/snap-8667764846443717831-1-100cf97e-6d56-446e-8961-afdaded63bc4.avro | {changed-partition-count=0, total-equality-deletes=0, total-position-deletes=0, total-delete-files=0, total-files-size=0, total-records=0, total-data-files=0} 2022-01-10 08:11:34 Europe/Vienna | 7860805980949777961 | 8667764846443717831 | append | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/snap-7860805980949777961-1-faa19903-1455-4bb8-855a-61a1bbafbaa7.avro | {changed-partition-count=1, added-data-files=1, total-equality-deletes=0, added-records=1, total-position-deletes=0, added-files-size=442, total-delete-files=0, total-files-size=442, total-records=1, total-data-files=1} - The output of the query has the following columns: .. list-table:: Snapshots columns @@ -1162,9 +923,8 @@ The output of the query has the following columns: - ``map(varchar, varchar)`` - A summary of the changes made from the previous snapshot to the current snapshot - ``$manifests`` table -^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~ The ``$manifests`` table provides a detailed overview of the manifests corresponding to the snapshots performed in the log of the Iceberg table. @@ -1180,7 +940,6 @@ You can retrieve the information about the manifests of the Iceberg table ----------------------------------------------------------------------------------------------------------------+-----------------+----------------------+-----------------------+-------------------------+------------------+-----------------------------+---------------------+-----------------------------+--------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------- hdfs://hadoop-master:9000/user/hive/warehouse/test_table/metadata/faa19903-1455-4bb8-855a-61a1bbafbaa7-m0.avro | 6277 | 0 | 7860805980949777961 | 1 | 100 | 0 | 0 | 0 | 0 | {{contains_null=false, contains_nan= false, lower_bound=1, upper_bound=1},{contains_null=false, contains_nan= false, lower_bound=2021-01-12, upper_bound=2021-01-12}} - The output of the query has the following columns: .. list-table:: Manifests columns @@ -1224,9 +983,8 @@ The output of the query has the following columns: - ``array(row(contains_null boolean, contains_nan boolean, lower_bound varchar, upper_bound varchar))`` - Partition range metadata - ``$partitions`` table -^^^^^^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~~~~~~ The ``$partitions`` table provides a detailed overview of the partitions of the Iceberg table. @@ -1243,7 +1001,6 @@ You can retrieve the information about the partitions of the Iceberg table {c1=1, c2=2021-01-12} | 2 | 2 | 884 | {c3={min=1.0, max=2.0, null_count=0, nan_count=NULL}} {c1=1, c2=2021-01-13} | 1 | 1 | 442 | {c3={min=1.0, max=1.0, null_count=0, nan_count=NULL}} - The output of the query has the following columns: .. list-table:: Partitions columns @@ -1270,7 +1027,7 @@ The output of the query has the following columns: - Partition range metadata ``$files`` table -^^^^^^^^^^^^^^^^ +~~~~~~~~~~~~~~~~ The ``$files`` table provides a detailed overview of the data files in current snapshot of the Iceberg table. @@ -1284,8 +1041,6 @@ To retrieve the information about the data files of the Iceberg table ``test_tab ----------+-------------------------------------------------------------------------------------------------------------------------------+-----------------+---------------+----------------------+----------------------+-------------------+--------------------+-------------------+-----------------------------+-----------------------------+----------------+----------------+--------------- 0 | hdfs://hadoop-master:9000/user/hive/warehouse/test_table/data/c1=3/c2=2021-01-14/af9872b2-40f3-428f-9c87-186d2750d84e.parquet | 1 | PARQUET | 442 | {1=40, 2=40, 3=44} | {1=1, 2=1, 3=1} | {1=0, 2=0, 3=0} | | {1=3, 2=2021-01-14, 3=1.3} | {1=3, 2=2021-01-14, 3=1.3} | | | - - The output of the query has the following columns: .. list-table:: Files columns @@ -1387,10 +1142,237 @@ The output of the query has the following columns: - ``bigint`` - For branch only, the max snapshot age allowed in a branch. Older snapshots in the branch will be expired. +.. _iceberg_metadata_columns: + +Metadata columns +"""""""""""""""" + +In addition to the defined columns, the Iceberg connector automatically exposes +path metadata as a hidden column in each table: + +* ``$path``: Full file system path name of the file for this row + +* ``$file_modified_time``: Timestamp of the last modification of the file for this row + +You can use these columns in your SQL statements like any other column. This +can be selected directly, or used in conditional statements. For example, you +can inspect the file path for each record:: + + SELECT *, "$path", "$file_modified_time" + FROM example.web.page_views; + +Retrieve all records that belong to a specific file using ``"$path"`` filter:: + + SELECT * + FROM example.web.page_views + WHERE "$path" = '/usr/iceberg/table/web.page_views/data/file_01.parquet' + +Retrieve all records that belong to a specific file using ``"$file_modified_time"`` filter:: + + SELECT * + FROM example.web.page_views + WHERE "$file_modified_time" = CAST('2022-07-01 01:02:03.456 UTC' AS timestamp with time zone) + +DROP TABLE +"""""""""" + +The Iceberg connector supports dropping a table by using the :doc:`/sql/drop-table` +syntax. When the command succeeds, both the data of the Iceberg table and also the +information related to the table in the metastore service are removed. +Dropping tables which have their data/metadata stored in a different location than +the table's corresponding base directory on the object store is not supported. + +.. _iceberg-comment: + +COMMENT +""""""" + +The Iceberg connector supports setting comments on the following objects: + +- tables +- views +- table columns + +The ``COMMENT`` option is supported on both the table and +the table columns for the :doc:`/sql/create-table` operation. + +The ``COMMENT`` option is supported for adding table columns +through the :doc:`/sql/alter-table` operations. + +The connector supports the command :doc:`COMMENT ` for setting +comments on existing entities. + +.. _iceberg-tables: + +Partitioned tables +"""""""""""""""""" + +Iceberg supports partitioning by specifying transforms over the table columns. +A partition is created for each unique tuple value produced by the transforms. +Identity transforms are simply the column name. Other transforms are: + +.. list-table:: Iceberg column transforms + :widths: 40, 60 + :header-rows: 1 + + * - Transform + - Description + * - ``year(ts)`` + - A partition is created for each year. The partition value is the integer + difference in years between ``ts`` and January 1 1970. + * - ``month(ts)`` + - A partition is created for each month of each year. The partition value + is the integer difference in months between ``ts`` and January 1 1970. + * - ``day(ts)`` + - A partition is created for each day of each year. The partition value is + the integer difference in days between ``ts`` and January 1 1970. + * - ``hour(ts)`` + - A partition is created hour of each day. The partition value is a + timestamp with the minutes and seconds set to zero. + * - ``bucket(x, nbuckets)`` + - The data is hashed into the specified number of buckets. The partition + value is an integer hash of ``x``, with a value between 0 and + ``nbuckets - 1`` inclusive. + * - ``truncate(s, nchars)`` + - The partition value is the first ``nchars`` characters of ``s``. + +In this example, the table is partitioned by the month of ``order_date``, a hash of +``account_number`` (with 10 buckets), and ``country``:: + + CREATE TABLE example.testdb.customer_orders ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) + WITH (partitioning = ARRAY['month(order_date)', 'bucket(account_number, 10)', 'country']) + +Sorted tables +""""""""""""" + +The connector supports sorted files as a performance improvement. Data is +sorted during writes within each file based on the specified array of one +or more columns. + +Sorting is particularly beneficial when the sorted columns show a +high cardinality and are used as a filter for selective reads. + +The sort order is configured with the ``sorted_by`` table property. +Specify an array of one or more columns to use for sorting +when creating the table. The following example configures the +``order_date`` column of the ``orders`` table in the ``customers`` +schema in the ``example`` catalog:: + + CREATE TABLE example.customers.orders ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) + WITH (sorted_by = ARRAY['order_date']) + +Sorting can be combined with partitioning on the same column. For example:: + + CREATE TABLE example.customers.orders ( + order_id BIGINT, + order_date DATE, + account_number BIGINT, + customer VARCHAR, + country VARCHAR) + WITH ( + partitioning = ARRAY['month(order_date)'], + sorted_by = ARRAY['order_date'] + ) + +You can disable sorted writing with the session property +``sorted_writing_enabled`` set to ``false``. + +Using snapshots +""""""""""""""" + +Iceberg supports a "snapshot" model of data, where table snapshots are +identified by a snapshot ID. + +The connector provides a system table exposing snapshot information for every +Iceberg table. Snapshots are identified by ``BIGINT`` snapshot IDs. +For example, you can find the snapshot IDs for the ``customer_orders`` table +by running the following query:: + + SELECT snapshot_id + FROM example.testdb."customer_orders$snapshots" + ORDER BY committed_at DESC + +.. _iceberg-time-travel: + +Time travel queries +~~~~~~~~~~~~~~~~~~~ + +The connector offers the ability to query historical data. +This allows you to query the table as it was when a previous snapshot +of the table was taken, even if the data has since been modified or deleted. + +The historical data of the table can be retrieved by specifying the +snapshot identifier corresponding to the version of the table to be retrieved:: + + SELECT * + FROM example.testdb.customer_orders FOR VERSION AS OF 8954597067493422955 + +A different approach of retrieving historical data is to specify +a point in time in the past, such as a day or week ago. The latest snapshot +of the table taken before or at the specified timestamp in the query is +internally used for providing the previous state of the table:: + + SELECT * + FROM example.testdb.customer_orders FOR TIMESTAMP AS OF TIMESTAMP '2022-03-23 09:59:29.803 Europe/Vienna' + +Rolling back to a previous snapshot +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Use the ``$snapshots`` metadata table to determine the latest snapshot ID of the table like in the following query:: + + SELECT snapshot_id + FROM example.testdb."customer_orders$snapshots" + ORDER BY committed_at DESC LIMIT 1 + +The procedure ``system.rollback_to_snapshot`` allows the caller to roll back +the state of the table to a previous snapshot id:: + + CALL example.system.rollback_to_snapshot('testdb', 'customer_orders', 8954597067493422955) + +Migrating existing tables +""""""""""""""""""""""""" + +The connector can read from or write to Hive tables that have been migrated to Iceberg. +There is no Trino support for migrating Hive tables to Iceberg, so you must either use +the Iceberg API or Apache Spark. + +``NOT NULL`` column constraint +"""""""""""""""""""""""""""""" + +The Iceberg connector supports setting ``NOT NULL`` constraints on the table columns. + +The ``NOT NULL`` constraint can be set on the columns, while creating tables by +using the :doc:`CREATE TABLE ` syntax:: + + CREATE TABLE example_table ( + year INTEGER NOT NULL, + name VARCHAR NOT NULL, + age INTEGER, + address VARCHAR + ); + +When trying to insert/update data in the table, the query fails if trying +to set ``NULL`` value on a column having the ``NOT NULL`` constraint. + +View management +^^^^^^^^^^^^^^^ + +Trino allows reading from Iceberg materialized views. + .. _iceberg-materialized-views: Materialized views ------------------- +"""""""""""""""""" The Iceberg connector supports :ref:`sql-materialized-view-management`. In the underlying system each materialized view consists of a view definition and an @@ -1408,7 +1390,7 @@ for the data files and partition the storage per day using the column By default, the storage table is created in the same schema as the materialized view definition. The ``iceberg.materialized-views.storage-schema`` catalog configuration property or ``storage_schema`` materialized view property can be -used to specify the schema where the storage table will be created. +used to specify the schema where the storage table is created. Updating the data in the materialized view with :doc:`/sql/refresh-materialized-view` deletes the data from the storage table, @@ -1430,8 +1412,14 @@ has no information whether the underlying non-Iceberg tables have changed. Dropping a materialized view with :doc:`/sql/drop-materialized-view` removes the definition and the storage table. +Performance +----------- + +The connector includes a number of performance improvements, detailed in the +following sections. + Table statistics ----------------- +^^^^^^^^^^^^^^^^ The Iceberg connector can collect column statistics using :doc:`/sql/analyze` statement. This can be disabled using ``iceberg.extended-statistics.enabled`` @@ -1441,7 +1429,7 @@ catalog configuration property, or the corresponding .. _iceberg_analyze: Updating table statistics -^^^^^^^^^^^^^^^^^^^^^^^^^ +""""""""""""""""""""""""" If your queries are complex and include joining large data sets, running :doc:`/sql/analyze` on tables may improve query performance @@ -1460,5 +1448,70 @@ specify a subset of columns to analyzed with the optional ``columns`` property:: This query collects statistics for columns ``col_1`` and ``col_2``. -Note that if statistics were previously collected for all columns, they need to be dropped +Note that if statistics were previously collected for all columns, they must be dropped using :ref:`drop_extended_stats ` command before re-analyzing. + +.. _iceberg-table-redirection: + +Table redirection +^^^^^^^^^^^^^^^^^ + +.. include:: table-redirection.fragment + +The connector supports redirection from Iceberg tables to Hive tables +with the ``iceberg.hive-catalog-name`` catalog configuration property. + +File formats +------------ + +The following file types and formats are supported for the Iceberg connector: + +* ORC +* Parquet + +ORC format configuration +^^^^^^^^^^^^^^^^^^^^^^^^ + +The following properties are used to configure the read and write operations +with ORC files performed by the Iceberg connector. + +.. list-table:: ORC format configuration properties + :widths: 30, 58, 12 + :header-rows: 1 + + * - Property name + - Description + - Default + * - ``hive.orc.bloom-filters.enabled`` + - Enable bloom filters for predicate pushdown. + - ``false`` + +Parquet format configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The following properties are used to configure the read and write operations +with Parquet files performed by the Iceberg connector. + +.. list-table:: Parquet format configuration properties + :widths: 30, 50, 20 + :header-rows: 1 + + * - Property Name + - Description + - Default + * - ``parquet.max-read-block-row-count`` + - Sets the maximum number of rows read in a batch. + - ``8192`` + * - ``parquet.optimized-reader.enabled`` + - Whether batched column readers are used when reading Parquet files + for improved performance. Set this property to ``false`` to disable the + optimized parquet reader by default. The equivalent catalog session + property is ``parquet_optimized_reader_enabled``. + - ``true`` + * - ``parquet.optimized-nested-reader.enabled`` + - Whether batched column readers are used when reading ARRAY, MAP + and ROW types from Parquet files for improved performance. Set this + property to ``false`` to disable the optimized parquet reader by default + for structural data types. The equivalent catalog session property is + ``parquet_optimized_nested_reader_enabled``. + - ``true`` \ No newline at end of file