Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Background iceberg split loader #9404

Closed
liupan664021 opened this issue Sep 28, 2021 · 3 comments
Closed

Background iceberg split loader #9404

liupan664021 opened this issue Sep 28, 2021 · 3 comments

Comments

@liupan664021
Copy link

Hello, is there any plan to put the icebeg split source in bakground? or is anyone working on this?

@Override
    public CompletableFuture<ConnectorSplitBatch> getNextBatch(ConnectorPartitionHandle partitionHandle, int maxSize)
    {
        // TODO: move this to a background thread
        List<ConnectorSplit> splits = new ArrayList<>();
        Iterator<FileScanTask> iterator = limit(fileScanIterator, maxSize);
        while (iterator.hasNext()) {
            FileScanTask task = iterator.next();
            if (!task.deletes().isEmpty()) {
                throw new TrinoException(NOT_SUPPORTED, "Iceberg tables with delete files are not supported: " + schemaTableName);
            }
            splits.add(toIcebergSplit(task));
        }
        return completedFuture(new ConnectorSplitBatch(splits, isFinished()));
    }
@findepi
Copy link
Member

findepi commented Sep 28, 2021

i think @alexjo2144 did it in #9193.
cc @losipiuk

@alexjo2144
Copy link
Member

Not merged yet, but yes

@ebyhr
Copy link
Member

ebyhr commented May 12, 2022

Closing as #9193 is already merged.

@ebyhr ebyhr closed this as completed May 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

4 participants