Storage Sub Scan

Create the Sub Scan for your Storage Plugin

Sub Scan Overview

The group scan represents the general ("logical plan") idea of "scan table X." The specific scan (also called a "sub scan") contains the information needed to implement the physical scan, including Hadoop splits, database shards or whatever is needed by the storage engine. In our case, we assume a single scan operator that just needs the information gathered above.

The specific scan is serialized to JSON and sent to each Drillbit, where it is deserialized and passed to the scan operator. While the group scan is an active participant in the Calcite planning process, the sub scan is a static container of information passed from the planner (running on the Foreman node) to the execution nodes (running in each Drillbit.) The sub scan is converted to and from JSON even if you run a single Drillbit that contains both the Foreman and execution stages.

Create the Sub Scan Class

Here is an example sub-scan class:

@JsonTypeName("example-sub-scan")
public class ExampleSubScan extends BaseSubScan {

  private final ExampleScanSpec tableSpec;

  public ExampleSubScan(
      @JsonProperty("userName") String userName,
      @JsonProperty("config") StoragePluginConfig config,
      @JsonProperty("scanSpec") DummyScanSpec scanSpec,
      @JsonProperty("columns") List<SchemaPath> columns,
      @JsonProperty("tableSpec") ExampleScanSpec tableSpec) {
    super(userName, config, columns, engineRegistry);
    this.tableSpec = tableSpec;
  }

  public ExampleScanSpec getTableSpec() {
    return tableSpec;
  }

  @Override
  public void buildPlanString(PlanStringBuilder builder) {
    super.buildPlanString(builder);
    builder.field("scanSpec", scanSpec);
  }
}

The BaseSubScan handles much of the boilerplate code you will find in existing SubScan classes.

Notice that we have included the storage plugin config, not the storage plugin itself. The config is Jackson-serializable, the plugin is not.

Create the Sub Scan Instance

We now connect up our sub scan with the group scan:

public class ExampleGroupScan extends AbstractGroupScan {
  ...

  @Override
  public SubScan getSpecificScan(int minorFragmentId) {
    return new ExampleSubScan(config, scanSpec, columns);
  }

Test the Sub Scan

Set a breakpoint in the above method and run the test case. Execution should stop there and you can verify that your sub scan is created as you expect.

With this, we are done with the planner side of the project and are ready to move onto the execution side.