EVF Tutorial Advanced Config

Advanced EVF Configuration

The tutorial thus far showed only those options needed for the log plugin. Here we describe additional options you can use if your needs go beyond the simple cases described thus far.

Define a Row Batch Reader Creator

The above method handles typical cases. If you find you have a special case (such as in the text reader), you can create a gain more control by defining "reader creator" class. (The Easy framework provides a default version that calls the above method.) Here's an example, defined as a nested class within LogFormatPlugin:

  private static class LogReaderFactory extends FileReaderFactory {

    private final LogFormatPlugin plugin;

    public LogReaderFactory(LogFormatPlugin plugin) {
      this.plugin = plugin;
    }

    @Override
    public ManagedReader<? extends FileSchemaNegotiator> newReader() {
       return new LogBatchReader(plugin.getConfig());
    }
  }

This is simple enough: EVF calls the newReader() method when it is ready to read the next file split. The split itself is obtained as shown previously. The advantage of this technique is that we an pass additional information from our plugin into the batch reader if we need more than that provided by the Easy framework.

We then use our class as follows:

  @Override
  protected FileScanBuilder frameworkBuilder(
      FragmentContext context, EasySubScan scan) throws ExecutionSetupException {
    FileScanBuilder builder = new FileScanBuilder();
    builder.setReaderFactory(new LogReaderFactory(this));
    ...

We specify the builder for our batch readers by calling setReaderFactory() with an instance of our reader creator.

If you use this method, then you don't need to implement the newReader() method on the plugin as we did earlier.

Conditionally Select the Traditional or Enhanced Scan Framework

We are now ready to switch over to the new, enhanced (EVF-based) reader. To do so, we simply set one option in our plugin configiruation:

 private static EasyFormatConfig easyConfig(Configuration fsConf, LogFormatConfig pluginConfig) {
    EasyFormatConfig config = new EasyFormatConfig();
    ...
    config.useEnhancedScan = true;
    return config;
  }

With this change, the EVF-based version is now live.

Conditionally Select the Original and EVF-Based Readers

If you are especially cautious, you can leverage the framework builder mechanism to offer both the new and old versions of your reader. Just override the useEnhancedScan() method. By default, the method just returns the option we set above:

  protected boolean useEnhancedScan(OptionManager options) {
    return easyConfig.useEnhancedScan;
  }

But, we could select the framework based on a system/session option as was done with the "v2" and "v3" versions of the text (CSV) reader in Drill 1.16.

  @Override
  protected boolean useEnhancedScan(OptionManager options) {
    return options.getBoolean(ExecConstants.ENABLE_V3_TEXT_READER_KEY);
  }

Customize the Error Context

The EVF does extensive error checking and will throw a UserException if things go wrong. To help your user understand the error, you can add custom context information by defining an "error context". Here's how it is done for the text format plugin:

  @Override
  protected FileScanBuilder frameworkBuilder(
      FragmentContext context, EasySubScan scan) throws ExecutionSetupException {
    ColumnsScanBuilder builder = new ColumnsScanBuilder();
...

    // Provide custom error context
    builder.setContext(
        new CustomErrorContext() {
          @Override
          public void addContext(UserException.Builder builder) {
            builder.addContext("Format plugin:", PLUGIN_NAME);
            builder.addContext("Plugin config name:", getName());
            builder.addContext("Extract headers:",
                Boolean.toString(getConfig().isHeaderExtractionEnabled()));
            builder.addContext("Skip headers:",
                Boolean.toString(getConfig().isSkipFirstLine()));
          }
        });

    return builder;
  }

You define a class which, given a user exception builder, fills in any additional context the user might find helpful. Here we add the name of the plugin and plugin configuration, along with some common setting information.

If you don't provide an error context, the Easy framework provides a very simple one for you with just the plugin name.

---

Next: Special Columns