Iceberg catalog - Unbalanced calls to openTransaction/commitTransaction

[fatihmete]

I working on POC Trino setup. When I try to create table on Iceberg catalog, I get following error:
query:

create table test.test1(
dummy int
)

error:

{'message': 'Failed to create table test.test1: java.lang.RuntimeException: 
commitTransaction was called but openTransactionCalls = 0. This probably indicates that there are unbalanced calls to openTransaction/commitTransaction', 'errorCode': 84148236, 
'errorName': 'ICEBERG_COMMIT_ERROR', 'errorType': 'EXTERNAL', 'failureInfo': {'type': 'io.trino.plugin.iceberg.CreateTableException', 
'message': 'Failed to create table test.test1: java.lang.RuntimeException: commitTransaction was called but openTransactionCalls = 0. This probably indicates that there are unbalanced calls to openTransaction/commitTransaction',
 'cause': {'type': 'io.trino.spi.TrinoException', 'message': 'java.lang.RuntimeException: commitTransaction was called but openTransactionCalls = 0. This probably indicates that there are unbalanced calls to openTransaction/commitTransaction', 'cause': {'type': 'io.trino.hive.thrift.metastore.MetaException', 'message': 'java.lang.RuntimeException: commitTransaction was called but openTransactionCalls = 0. This probably indicates that there are unbalanced calls to openTransaction/commitTransaction', 'suppressed': [], 'stack': 
....

Trino creates the table and writes the data to HDFS. But then deletes metadata files. The table still available in metastore. However, running query on the table is impossible, as metadata was deleted.

Deleted files:
warehouse/external/test.db/test1/metadata/snap-2092755023203536023-1-66c7d0ae-fd3f-4783-8a96-570339284098.avro
/warehouse/external/test.db/test1/metadata/00000-fdf59c76-2895-4d03-834d-93c441ccdd0f.metadata.json

First I think it is related Hive metastore, but when I change trino catalog type to hive, everything works as expected. Maybe It is related Iceberg catalog client settings. Also, other queries SELECT, INSERT, UPDATE, DELETE, CREATE SCHEME, Iceberg commands etc. works.

I tried run the same query with one of the hadoop proxy users (hive), It works. Hive disables authorization checks to proxy users. So problem maybe related with ranger or metastore settings. I am not sure.

Thanks

Standalone Hivemetastore: 4.0.1
Trino: 468
Ranger: 3.0.0 (Only HDFS and Hive plugin -hiveMetastore- activated)
Hadoop 3.4.1
Kerberos enabled

[fatihmete]

I enabled DEBUG mode usin io.trino.plugin.hive.metastore.thrif=DEBUG
trino hive catalog query - success:

DEBUG   transaction-finishing-0 io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient Invocation of 
 createTable(
 tbl=Table(
    tableName:test11, 
    dbName:test, 
    owner:fatih, 
    createTime:0, 
    lastAccessTime:0, 
    retention:0, 
    sd:StorageDescriptor(
        cols:[FieldSchema(name:dummy, type:int, comment:null)], 
        location:hdfs://namenode.company.bigdata.svc.cluster.local:9000/warehouse/external/test.db/test11, 
        inputFormat:org.apache.hadoop.hive.ql.io.orc.OrcInputFormat, 
        outputFormat:org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat, 
        compressed:false, 
        numBuckets:0, 
        serdeInfo:SerDeInfo(name:test11, serializationLib:org.apache.hadoop.hive.ql.io.orc.OrcSerde, parameters:{}), 
        bucketCols:null, 
        sortCols:null, 
        parameters:{}), 
    partitionKeys:[], 
    parameters:{trino_version=468, trino_query_id=20250114_215219_00001_qiwvz, transactional=false, auto.purge=false, numFiles=-1, totalSize=-1}, 
    viewOriginalText:null, 
    viewExpandedText:null, 
    tableType:MANAGED_TABLE, 
    privileges:PrincipalPrivilegeSet(
        userPrivileges:{fatih=[
            PrivilegeGrantInfo(privilege:select, createTime:0, grantor:fatih, grantorType:USER, grantOption:true), 
            PrivilegeGrantInfo(privilege:insert, createTime:0, grantor:fatih, grantorType:USER, grantOption:true), 
            PrivilegeGrantInfo(privilege:update, createTime:0, grantor:fatih, grantorType:USER, grantOption:true), 
            PrivilegeGrantInfo(privilege:delete, createTime:0, grantor:fatih, grantorType:USER, grantOption:true)]}, groupPrivileges:{}, rolePrivileges:{}), ownerType:USER)
    
    ) took 102.42ms and failed with MetaException(message:java.lang.RuntimeException: commitTransaction was called but openTransactionCalls = 0. This probably indicates that there are unbalanced calls to openTransaction/commitTransaction)

trino iceberg catalog query same HMS- fail:

2025-01-14T21:37:48.986Z        DEBUG   Query-20250114_213741_00001_tew35-164   io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient Invocation of 
createTable(
tbl=Table(
    tableName:test10, 
    dbName:test, 
    owner:fatih, 
    createTime:0, 
    lastAccessTime:0, 
    retention:0, 
    sd:StorageDescriptor(
        cols:[FieldSchema(name:dummy, type:int, comment:null)], 
        location:hdfs://namenode.company.bigdata.svc.cluster.local:9000/warehouse/external/test.db/test10, 
        inputFormat:org.apache.hadoop.mapred.FileInputFormat, 
        outputFormat:org.apache.hadoop.mapred.FileOutputFormat, 
        compressed:false, 
        numBuckets:0, 
        serdeInfo:SerDeInfo(name:test10, serializationLib:org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, parameters:{}), 
        bucketCols:null, 
        sortCols:null, 
        parameters:{}), 
    partitionKeys:[], 
    parameters:{EXTERNAL=TRUE, table_type=ICEBERG, metadata_location=hdfs://namenode.company.bigdata.svc.cluster.local:9000/warehouse/external/test.db/test10/metadata/00000-d29859e8-e1a9-44e5-b1c4-a26923022986.metadata.json}, 
    viewOriginalText:null, 
    viewExpandedText:null, 
    tableType:EXTERNAL_TABLE, 
    privileges:
        PrincipalPrivilegeSet(
            userPrivileges:{fatih=[
                PrivilegeGrantInfo(privilege:select, createTime:0, grantor:fatih, grantorType:USER, grantOption:true), 
                PrivilegeGrantInfo(privilege:insert, createTime:0, grantor:fatih, grantorType:USER, grantOption:true), 
                PrivilegeGrantInfo(privilege:update, createTime:0, grantor:fatih, grantorType:USER, grantOption:true), 
                PrivilegeGrantInfo(privilege:delete, createTime:0, grantor:fatih, grantorType:USER, grantOption:true)]}, 
            groupPrivileges:{}, rolePrivileges:{}), ownerType:USER)
    )

Same error exist in success hive log, but query completed.