Batch Handling Upgrades

The batch handling framework consists of a number of layers that combine to enable Drill to control the size of each record batch, which in turn allows Drill to implement effective memory management and admission control.

The material here starts with concepts, then provides a tour of the various components. Each component is heavily commented, so after reading this material, you should be able to get the details from the code itself.

• Conceptual Overview
• Components
• Code
• Metadata
• Row Set Mechanism
• Column Accessors
• Column Readers

2. Vector readers. Object categories. Vector indexes. Vector accessors. Array accessors. Generated code. Array wrappers for nullable, arrays.

• Column Writers

3. Row-set writers. Top-level writers. Structure. Writing to arrays. Events. Offset vector updates.

4. Row set loader. Concept of overflow. Column states. Vector states. Overflow processing. Vector allocation. Vector cache and multi-reader model.

5. Operator framework. Split of concerns. Protocol adapter. Schema change detection.

6. Projection framework. Concepts. Project lists. Null columns. Implicit columns. Assembling the output batch. Column information in projection list. Recursive projection in maps. Schema smoothing and persistence.

7. Mock reader. CSV reader. Easy format plugin. Concept of Parquet support.

8. JSON concepts. JSON issues. Revised JSON parser. JSON semantics. Open issues. Possible opportunities.

9. Future opportunities. Code generation. Plugin APIs. Reader retrofits. Fixed-size buffers.