Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make UDTF execution framework batch processing #12562

Merged
merged 70 commits into from
May 31, 2024
Merged

Make UDTF execution framework batch processing #12562

merged 70 commits into from
May 31, 2024

Commits on Mar 25, 2024

  1. Add .putFloats() to insert multiple rows at the same time.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    e60779e View commit details
    Browse the repository at this point in the history

  2. Complete all batch insert method in BatchData.java.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 25, 2024
    Configuration menu
    Copy the full SHA
    30ab8c8 View commit details
    Browse the repository at this point in the history

Commits on Mar 26, 2024

  1. Add overload methods with indexes to avoid copying.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    aabcc79 View commit details
    Browse the repository at this point in the history

  2. Add Java doc for new methods in BatchData.java.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    7000645 View commit details
    Browse the repository at this point in the history

  3. Merge two BatchDataTest.java files.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 26, 2024
    Configuration menu
    Copy the full SHA
    850d7b2 View commit details
    Browse the repository at this point in the history

Commits on Mar 27, 2024

  1. Add .putColumns() method to SerializableRowRecordList.java.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    464078d View commit details
    Browse the repository at this point in the history

  2. Finish batch insert method prototype in row record list and tv list.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 27, 2024
    Configuration menu
    Copy the full SHA
    7350f03 View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2024

  1. UDTF execution framework can support running really simple UDTF.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 28, 2024
    Configuration menu
    Copy the full SHA
    7184446 View commit details
    Browse the repository at this point in the history

Commits on Mar 31, 2024

  1. Change TVList to list of columns to avoid copying.

    @Sh-Zh-7
    Sh-Zh-7 committed Mar 31, 2024
    Configuration menu
    Copy the full SHA
    15073c0 View commit details
    Browse the repository at this point in the history

Commits on Apr 22, 2024

  1. Compile DataNode without error.

    @Sh-Zh-7
    Sh-Zh-7 committed Apr 22, 2024
    Configuration menu
    Copy the full SHA
    e8627ea View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. Format all files by mvn splotless:apply.

    @Sh-Zh-7
    Sh-Zh-7 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    13ec2ff View commit details
    Browse the repository at this point in the history

  2. Optimize import by using IDEA shortcut.

    @Sh-Zh-7
    Sh-Zh-7 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    7bb2b8d View commit details
    Browse the repository at this point in the history

  3. Fix put data into existing list error.

    @Sh-Zh-7
    Sh-Zh-7 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    fce4efd View commit details
    Browse the repository at this point in the history

  4. Generate data in terminate method.

    @Sh-Zh-7
    Sh-Zh-7 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    db70100 View commit details
    Browse the repository at this point in the history

  5. Format again using mvn spotless:apply.

    @Sh-Zh-7
    Sh-Zh-7 committed May 6, 2024
    Configuration menu
    Copy the full SHA
    6c3b33e View commit details
    Browse the repository at this point in the history

Commits on May 12, 2024

  1. Fix some bugs and implement RowWindow UDTF.

    @Sh-Zh-7
    Sh-Zh-7 committed May 12, 2024
    Configuration menu
    Copy the full SHA
    c1222a7 View commit details
    Browse the repository at this point in the history

Commits on May 13, 2024

  1. Basically pass all UDF-related IT.

    @Sh-Zh-7
    Sh-Zh-7 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    46b6aee View commit details
    Browse the repository at this point in the history

  2. Add license header to all newly-added files.

    @Sh-Zh-7
    Sh-Zh-7 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    895fffb View commit details
    Browse the repository at this point in the history

  3. Add memory control strategy for binary list.

    @Sh-Zh-7
    Sh-Zh-7 committed May 13, 2024
    Configuration menu
    Copy the full SHA
    e16f9c5 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2024

  1. Add memory control strategy for row list that contains binary field.

    @Sh-Zh-7
    Sh-Zh-7 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    69a846b View commit details
    Browse the repository at this point in the history

  2. Copy subregion to avoid memory leakage.

    @Sh-Zh-7
    Sh-Zh-7 committed May 14, 2024
    Configuration menu
    Copy the full SHA
    1c7356b View commit details
    Browse the repository at this point in the history

Commits on May 15, 2024

  1. Update iterator by recording first data index when apply new memory s… 

    …trategy.
    @Sh-Zh-7
    Sh-Zh-7 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    d4f83ee View commit details
    Browse the repository at this point in the history

  2. Unify newXXX series methods name to construct.

    @Sh-Zh-7
    Sh-Zh-7 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    1f5c673 View commit details
    Browse the repository at this point in the history

  3. Basically refactor storage layer.

    @Sh-Zh-7
    Sh-Zh-7 committed May 15, 2024
    Configuration menu
    Copy the full SHA
    079d65e View commit details
    Browse the repository at this point in the history

Commits on May 16, 2024

  1. Add and update copy method to Column based classes.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    46fb020 View commit details
    Browse the repository at this point in the history

  2. Rewrite unit test for ElasticSerializableRowList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    6065cb9 View commit details
    Browse the repository at this point in the history

  3. Rewrite unit test for SerializableRowList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    bb54bf6 View commit details
    Browse the repository at this point in the history

  4. Rewrite unit test for SerializableTVList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    30b93d0 View commit details
    Browse the repository at this point in the history

  5. Rewrite unit test for ElasticSerializableTVList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    875a467 View commit details
    Browse the repository at this point in the history

  6. Sync changes between TVList and RowList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    7537ab8 View commit details
    Browse the repository at this point in the history

  7. Refine UT of ESRowList and ESTVList with random size strings.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    792ffa2 View commit details
    Browse the repository at this point in the history

  8. Do not cache internal list in some situation.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    9ea978f View commit details
    Browse the repository at this point in the history

  9. Add forgotten files.

    @Sh-Zh-7
    Sh-Zh-7 committed May 16, 2024
    Configuration menu
    Copy the full SHA
    701c2a4 View commit details
    Browse the repository at this point in the history

Commits on May 18, 2024

  1. Revert BatchData.java relevant files.

    @Sh-Zh-7
    Sh-Zh-7 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    a998340 View commit details
    Browse the repository at this point in the history

  2. Remove redundant timeHeap.clear() in MultiInputColumnIntermediateLa… 

    …yer.
    @Sh-Zh-7
    Sh-Zh-7 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    22f08b1 View commit details
    Browse the repository at this point in the history

  3. Add constant folding in BinaryTransformer.

    @Sh-Zh-7
    Sh-Zh-7 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    29245e9 View commit details
    Browse the repository at this point in the history

  4. Refactor computational layer.

    @Sh-Zh-7
    Sh-Zh-7 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    d037a3e View commit details
    Browse the repository at this point in the history

  5. Replace origin normal column in constant reader with RLEColumn.

    @Sh-Zh-7
    Sh-Zh-7 committed May 18, 2024
    Configuration menu
    Copy the full SHA
    75fea8f View commit details
    Browse the repository at this point in the history

Commits on May 19, 2024

  1. Batch sliding time row window in SingleInputColumnSingleReferenceInte… 

    …rmediateLayer.
    @Sh-Zh-7
    Sh-Zh-7 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    85c4cd4 View commit details
    Browse the repository at this point in the history

  2. Batch other row windows in SingleInputColumnSingleReferenceIntermedia… 

    …teLayer.
    @Sh-Zh-7
    Sh-Zh-7 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    f65383e View commit details
    Browse the repository at this point in the history

  3. Add forgotten TransformUtils.java file.

    @Sh-Zh-7
    Sh-Zh-7 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    2337bac View commit details
    Browse the repository at this point in the history

  4. Batch all row windows in SingleInputColumnMultiReferenceIntermediateL… 

    …ayer.
    @Sh-Zh-7
    Sh-Zh-7 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    e78a7bd View commit details
    Browse the repository at this point in the history

  5. Batch all row windows in all intermediate layer.

    @Sh-Zh-7
    Sh-Zh-7 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    f9e7fd7 View commit details
    Browse the repository at this point in the history

  6. Remove code with no usage.

    @Sh-Zh-7
    Sh-Zh-7 committed May 19, 2024
    Configuration menu
    Copy the full SHA
    09a0251 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2024

  1. Remove code with no usage agaion.

    @Sh-Zh-7
    Sh-Zh-7 committed May 20, 2024
    Configuration menu
    Copy the full SHA
    263463b View commit details
    Browse the repository at this point in the history

  2. Merge master and resolve conflicts.

    @Sh-Zh-7
    Sh-Zh-7 committed May 20, 2024
    Configuration menu
    Copy the full SHA
    2d7f085 View commit details
    Browse the repository at this point in the history

  3. Remove nonexistent files.

    @Sh-Zh-7
    Sh-Zh-7 committed May 20, 2024
    Configuration menu
    Copy the full SHA
    e4aba5d View commit details
    Browse the repository at this point in the history

  4. Refactor project's structure.

    @Sh-Zh-7
    Sh-Zh-7 committed May 20, 2024
    Configuration menu
    Copy the full SHA
    8a34c66 View commit details
    Browse the repository at this point in the history

  5. Optimize when there is only one transformer.

    @Sh-Zh-7
    Sh-Zh-7 committed May 20, 2024
    Configuration menu
    Copy the full SHA
    244e7bb View commit details
    Browse the repository at this point in the history

  6. Fix losing data bug in benchmarking.

    @Sh-Zh-7
    Sh-Zh-7 committed May 20, 2024
    Configuration menu
    Copy the full SHA
    c16aeca View commit details
    Browse the repository at this point in the history

Commits on May 21, 2024

  1. Refactor UDTF executor.

    @Sh-Zh-7
    Sh-Zh-7 committed May 21, 2024
    Configuration menu
    Copy the full SHA
    6ca7a0d View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. Skip cache update when getting column size.

    @Sh-Zh-7
    Sh-Zh-7 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    2d55b49 View commit details
    Browse the repository at this point in the history

  2. Discard TsBlock when there is only one UDF.

    @Sh-Zh-7
    Sh-Zh-7 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    f7f91b3 View commit details
    Browse the repository at this point in the history

  3. Fix transformer IT errors.

    @Sh-Zh-7
    Sh-Zh-7 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    ef0be45 View commit details
    Browse the repository at this point in the history

  4. Remove unnecessary bitmap in ESRowList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    204d71d View commit details
    Browse the repository at this point in the history

  5. Update TsFile version.

    @Sh-Zh-7
    Sh-Zh-7 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    67fad74 View commit details
    Browse the repository at this point in the history

  6. Fix some IT OOM bugs.

    @Sh-Zh-7
    Sh-Zh-7 committed May 23, 2024
    Configuration menu
    Copy the full SHA
    b0dbc43 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Try to fix UT OOM error.

    @Sh-Zh-7
    Sh-Zh-7 committed May 24, 2024
    Configuration menu
    Copy the full SHA
    891466f View commit details
    Browse the repository at this point in the history

  2. Apply spotless plugin.

    @Sh-Zh-7
    Sh-Zh-7 committed May 24, 2024
    Configuration menu
    Copy the full SHA
    12a8ccb View commit details
    Browse the repository at this point in the history

Commits on May 27, 2024

  1. Evict data in UDF output storage.

    @Sh-Zh-7
    Sh-Zh-7 committed May 27, 2024
    Configuration menu
    Copy the full SHA
    670851b View commit details
    Browse the repository at this point in the history

  2. Revert optimization when there is only one UDTF.

    @Sh-Zh-7
    Sh-Zh-7 committed May 27, 2024
    Configuration menu
    Copy the full SHA
    abe3eef View commit details
    Browse the repository at this point in the history

  3. Fix losing data error.

    @Sh-Zh-7
    Sh-Zh-7 committed May 27, 2024
    Configuration menu
    Copy the full SHA
    739eb63 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. Update internal column count when ESList apply new memory control str… 

    …ategy.
    @Sh-Zh-7
    Sh-Zh-7 committed May 28, 2024
    Configuration menu
    Copy the full SHA
    e316c19 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Resolve conflicts caused by new types.

    @Sh-Zh-7
    Sh-Zh-7 committed May 29, 2024
    Configuration menu
    Copy the full SHA
    361804f View commit details
    Browse the repository at this point in the history

  2. Add new types in SerializableRowList.

    @Sh-Zh-7
    Sh-Zh-7 committed May 29, 2024
    Configuration menu
    Copy the full SHA
    b5f7827 View commit details
    Browse the repository at this point in the history

  3. Apply mvn spotless plugin.

    @Sh-Zh-7
    Sh-Zh-7 committed May 29, 2024
    Configuration menu
    Copy the full SHA
    bb32382 View commit details
    Browse the repository at this point in the history

Commits on May 30, 2024

  1. Fix integration-tests errors.

    @Sh-Zh-7
    Sh-Zh-7 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    8072e22 View commit details
    Browse the repository at this point in the history

  2. Modify code according to committer's advices.[Part One]

    @Sh-Zh-7
    Sh-Zh-7 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    97e0ad8 View commit details
    Browse the repository at this point in the history

  3. Modify code according to committer's advices.[Part Two]

    @Sh-Zh-7
    Sh-Zh-7 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    177a475 View commit details
    Browse the repository at this point in the history

  4. Modify code according to committer's advices.[Part Three]

    @Sh-Zh-7
    Sh-Zh-7 committed May 30, 2024
    Configuration menu
    Copy the full SHA
    06f1694 View commit details
    Browse the repository at this point in the history