ParseTreeDispatcher

[ghost]

Currently the ParseTreeWalker notifies one listener of enter and exit events, similar to the following:

ParseTreeWalker walker = new ParseTreeWalker();
walker.walk( parseTreeListener, parseTree );

In some situations having multiple listeners to handle various parts of the grammar would be convenient:

ParseTreeDispatcher dispatcher = new ParseTreeDispatcher();
List<ParseTreeListener> listeners = ...;
dispatcher.walk( listeners, parseTree );

Each listener would receive every enter and exit event, but the entire tree is walked only once.

Note that this is different than parsing the grammar once for each listener, which is trivial. The debug hub could be used to accomplish this, but it has extra code that isn't strictly necessary.

[sharwell]

This would be easily accomplished by defining a ProxyParseTreeListener class following the pattern of ProxyErrorListener.

Also note that you should never need to use new ParseTreeWalker() in code because a default instance is available:

ParseTreeWalker.DEFAULT.walk(parseTreeListener, parseTree);

[ghost]

The ProxyErrorListener appears to delegate via repetitious boilerplate code:

for (ANTLRErrorListener listener : delegates) {
  listener.syntaxError(recognizer, offendingSymbol, line, charPositionInLine, msg, e);
}
// ...
for (ANTLRErrorListener listener : delegates) {
  listener.reportAmbiguity(recognizer, dfa, startIndex, stopIndex, exact, ambigAlts, configs);
}
// ...
for (ANTLRErrorListener listener : delegates) {
  listener.reportContextSensitivity(recognizer, dfa, startIndex, stopIndex, prediction, configs);
}

If the grammar changes, which happens often during development, then the delegate methods must be updated. This is not a solution that is easy to maintain. If the delegate methods can be generated, then so too could a ParseTreeDispatcher (or equivalent).

From a higher vantage point, the ProxyErrorListener and Debug Hub both apply the same concept: they delegate over a list of listeners. If developers must also codify it, then the logic is virtually triplicated.

Ideally, there would be an abstraction generated by ANTLR that the ProxyErrorListener, the Debug Hub, and any application-specific code could inherit. The ParseTreeWalker could also inherit the list-walking behaviour.

(In Smalltalk it is trivial to delegate a method call to a list of objects without needing to know the name of the method being called.)

[ericvergnaud]

Hi,
not sure I understand your point. The code you quote is NOT generated, and NOT grammar dependent. So how does the implementation impact developers?

[sharwell]

@DaveJarvis The final usage might look like this:

ParseTreeListener listener1 = ...;
ParseTreeListener listener2 = ...;
ProxyParseTreeListener proxy = new ProxyParseTreeListener(Arrays.asList(listener1, listener2));
ParseTreeWalker.DEFAULT.walk(proxy, parseTree);

The ProxyParseTreeListener simply implements each method from the ParseTreeListener interface so it passes those calls on to its delegates, which in this case are listener1 and listener2. There is no need to update the implementation in the future unless ParseTreeListener itself changes.

[ghost]

Here's a generic ProxyParseTreeListener, as per the suggestion:

public class ProxyParseTreeListener implements ParseTreeListener {
  private List<ParseTreeListener> listeners;

  public ProxyParseTreeListener() {
    this( new ArrayList<ParseTreeListener>() );
  }

  public ProxyParseTreeListener( List<ParseTreeListener> listeners ) {
    setListeners( listeners );
  }

  public void add( ParseTreeListener listener ) {
    getListeners().add( listener );
  }

  public void remove( ParseTreeListener listener ) {
    getListeners().remove( listener );
  }

  public void enterEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.enterEveryRule( ctx );
    }
  }

  public void exitEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.exitEveryRule( ctx );
    }
  }

  public void visitErrorNode( ErrorNode node ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.visitErrorNode( node );
    }
  }

  public void visitTerminal( TerminalNode node ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.visitTerminal( node );
    }
  }

  private List<ParseTreeListener> getListeners() {
    return this.listeners;
  }

  private void setListeners( List<ParseTreeListener> listeners ) {
    this.listeners = listeners;
  }
}

The code is used as follows:

ParseTreeWalker walker = ParseTreeWalker.DEFAULT;
ProxyParseTreeListener proxy = new ProxyParseTreeListener();
proxy.add( new WhereClause() );
walker.walk( proxy, ctx );

The WhereClause is defined as:

public class WhereClause extends QueryBaseListener {
  public WhereClause() { }
  public void enterWhere( QueryParser.WhereContext ctx ) {
    append( "WHERE " );
  }
}

The enterWhere method is never called, but not using the proxy class (using WhereClause directly) causes enterWhere to be called, as expected:

walker.walk( new WhereClause(), ctx );

Having ProxyParseTreeListener inherit from QueryBaseListener doesn't work. Typecasting the listener doesn't coerce the method calls. Typecasting the ParserRuleContext also doesn't help.

Any suggestions?

[sharwell]

Your implementation of ProxyParseTreeListener.enterEveryRule should look like this. You'll also need to update exitEveryRule in a similar manner.

public void enterEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
        listener.enterEveryRule(ctx);
        ctx.enterRule(listener);
    }
}

[ghost]

Hereby added to the public domain. You are welcome to include it in the ANTLR distribution under any open source license deemed appropriate. Attribution is not required.

import java.util.ArrayList;
import java.util.List;

import org.antlr.v4.runtime.ParserRuleContext;

import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.ParseTreeListener;
import org.antlr.v4.runtime.tree.ErrorNode;
import org.antlr.v4.runtime.tree.TerminalNode;

/**
 * Instances of this class allows multiple listeners to receive events
 * while walking the parse tree. For example:
 *
 * <pre>
 * ProxyParseTreeListener proxy = new ProxyParseTreeListener();
 * ParseTreeListener listener1 = ... ;
 * ParseTreeListener listener2 = ... ;
 * proxy.add( listener1 );
 * proxy.add( listener2 );
 * ParseTreeWalker.DEFAULT.walk( proxy, ctx );
 * </pre>
 */
public class ProxyParseTreeListener implements ParseTreeListener {
  private List<ParseTreeListener> listeners;

  /**
   * Creates a new proxy without an empty list of listeners. Add
   * listeners before walking the tree.
   */
  public ProxyParseTreeListener() {
    // Setting the listener to null automatically instantiates a new list.
    this( null );
  }

  /**
   * Creates a new proxy with the given list of listeners.
   *
   * @param listeners A list of listerners to receive events.
   */
  public ProxyParseTreeListener( List<ParseTreeListener> listeners ) {
    setListeners( listeners );
  }

  @Override
  public void enterEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.enterEveryRule( ctx );
      ctx.enterRule( listener );
    }
  }

  @Override
  public void exitEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
      ctx.exitRule( listener );
      listener.exitEveryRule( ctx );
    }
  }

  @Override
  public void visitErrorNode( ErrorNode node ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.visitErrorNode( node );
    }
  }

  @Override
  public void visitTerminal( TerminalNode node ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.visitTerminal( node );
    }
  }

  /**
   * Adds the given listener to the list of event notification recipients.
   *
   * @param listener A listener to begin receiving events.
   */
  public void add( ParseTreeListener listener ) {
    getListeners().add( listener );
  }

  /**
   * Removes the given listener to the list of event notification recipients.
   *
   * @param listener A listener to stop receiving events.
   * @return false The listener was not registered to receive events.
   */
  public boolean remove( ParseTreeListener listener ) {
    return getListeners().remove( listener );
  }

  /**
   * Returns the list of listeners.
   *
   * @return The list of listeners to receive tree walking events.
   */
  private List<ParseTreeListener> getListeners() {
    return this.listeners;
  }

  /**
   * Changes the list of listeners to receive events. If the given list of
   * listeners is null, an empty list will be created.
   *
   * @param listeners A list of listeners to receive tree walking
   * events.
   */
  public void setListeners( List<ParseTreeListener> listeners ) {
    if( listeners == null ) {
      listeners = new ArrayList<ParseTreeListener>();
    }

    this.listeners = listeners;
  }
}

[ghost]

Version safe for concurrent modifications:

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;

import org.antlr.v4.runtime.ParserRuleContext;

import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.ParseTreeListener;
import org.antlr.v4.runtime.tree.ErrorNode;
import org.antlr.v4.runtime.tree.TerminalNode;

/**
 * Instances of this class allows multiple listeners to receive events
 * while walking the parse tree. For example:
 *
 * <pre>
 * ProxyParseTreeListener proxy = new ProxyParseTreeListener();
 * ParseTreeListener listener1 = ... ;
 * ParseTreeListener listener2 = ... ;
 * proxy.add( listener1 );
 * proxy.add( listener2 );
 * ParseTreeWalker.DEFAULT.walk( proxy, ctx );
 * </pre>
 */
public class ProxyParseTreeListener implements ParseTreeListener {
  private List<ParseTreeListener> listeners;

  /**
   * Creates a new proxy without an empty list of listeners. Add
   * listeners before walking the tree.
   */
  public ProxyParseTreeListener() {
    // Setting the listener to null automatically instantiates a new list.
    this( null );
  }

  /**
   * Creates a new proxy with the given list of listeners.
   *
   * @param listeners A list of listerners to receive events.
   */
  public ProxyParseTreeListener( List<ParseTreeListener> listeners ) {
    setListeners( listeners );
  }

  @Override
  public void enterEveryRule( ParserRuleContext ctx ) {
    Iterator<ParseTreeListener> i = iterator();

    while( i.hasNext() ) {
      ParseTreeListener listener = i.next();
      listener.enterEveryRule( ctx );
      ctx.enterRule( listener );
    }
  }

  @Override
  public void exitEveryRule( ParserRuleContext ctx ) {
    Iterator<ParseTreeListener> i = iterator();

    while( i.hasNext() ) {
      ParseTreeListener listener = i.next();
      ctx.exitRule( listener );
      listener.exitEveryRule( ctx );
    }
  }

  @Override
  public void visitErrorNode( ErrorNode node ) {
    Iterator<ParseTreeListener> i = iterator();

    while( i.hasNext() ) {
      ParseTreeListener listener = i.next();
      listener.visitErrorNode( node );
    }
  }

  @Override
  public void visitTerminal( TerminalNode node ) {
    Iterator<ParseTreeListener> i = iterator();

    while( i.hasNext() ) {
      ParseTreeListener listener = i.next();
      listener.visitTerminal( node );
    }
  }

  /**
   * Adds the given listener to the list of event notification recipients.
   *
   * @param listener A listener to begin receiving events.
   */
  public void add( ParseTreeListener listener ) {
    getListeners().add( listener );
  }

  /**
   * Removes the given listener to the list of event notification recipients.
   *
   * @param listener A listener to stop receiving events.
   * @return false The listener was not registered to receive events.
   */
  public boolean remove( ParseTreeListener listener ) {
    return getListeners().remove( listener );
  }

  /**
   * Returns an iterator of a copy of the current list. This protects
   * against concurrent modifications to the list.
   *
   * @return A non-null, possibly empty, list of ParseTreeListeners that
   * will receive events.
   */
  private Iterator<ParseTreeListener> iterator() {
    List<ParseTreeListener> list = createParseTreeListenerList();
    list.addAll( getListeners() );
    return list.iterator();
  }

  /**
   * Returns the list of listeners.
   *
   * @return The list of listeners to receive tree walking events.
   */
  private List<ParseTreeListener> getListeners() {
    return this.listeners;
  }

  /**
   * Changes the list of listeners to receive events. If the given list of
   * listeners is null, an empty list will be created.
   *
   * @param listeners A list of listeners to receive tree walking
   * events.
   */
  public void setListeners( List<ParseTreeListener> listeners ) {
    if( listeners == null ) {
      listeners = createParseTreeListenerList();
    }

    this.listeners = listeners;
  }

  protected List<ParseTreeListener> createParseTreeListenerList() {
    return new ArrayList<ParseTreeListener>();
  }
}

[parrt]

Thanks. I'll look at this when I can turn to ANTLR again mid May.

[sharwell]

Thank you @DaveJarvis. I'm not sure this will end up in the standard runtime, but it looks like a great contribution. I might add a modified implementation that uses CopyOnWriteArrayList <T> if I get some spare time.

[ghost]

Good call, @sharwell. Using CopyOnWriteArrayList simplifies the code while avoiding concurrency issues. Here's the revised class:

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.CopyOnWriteArrayList;

import org.antlr.v4.runtime.ParserRuleContext;

import org.antlr.v4.runtime.tree.ParseTree;
import org.antlr.v4.runtime.tree.ParseTreeListener;
import org.antlr.v4.runtime.tree.ErrorNode;
import org.antlr.v4.runtime.tree.TerminalNode;

/**
 * Instances of this class allows multiple listeners to receive events
 * while walking the parse tree. For example:
 *
 * <pre>
 * ProxyParseTreeListener proxy = new ProxyParseTreeListener();
 * ParseTreeListener listener1 = ... ;
 * ParseTreeListener listener2 = ... ;
 * proxy.add( listener1 );
 * proxy.add( listener2 );
 * ParseTreeWalker.DEFAULT.walk( proxy, ctx );
 * </pre>
 */
public class ProxyParseTreeListener implements ParseTreeListener {
  private List<ParseTreeListener> listeners;

  /**
   * Creates a new proxy without an empty list of listeners. Add
   * listeners before walking the tree.
   */
  public ProxyParseTreeListener() {
    // Setting the listener to null automatically instantiates a new list.
    this( null );
  }

  /**
   * Creates a new proxy with the given list of listeners.
   *
   * @param listeners A list of listerners to receive events.
   */
  public ProxyParseTreeListener( List<ParseTreeListener> listeners ) {
    setListeners( listeners );
  }

  @Override
  public void enterEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.enterEveryRule( ctx );
      ctx.enterRule( listener );
    }
  }

  @Override
  public void exitEveryRule( ParserRuleContext ctx ) {
    for( ParseTreeListener listener : getListeners() ) {
      ctx.exitRule( listener );
      listener.exitEveryRule( ctx );
    }
  }

  @Override
  public void visitErrorNode( ErrorNode node ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.visitErrorNode( node );
    }
  }

  @Override
  public void visitTerminal( TerminalNode node ) {
    for( ParseTreeListener listener : getListeners() ) {
      listener.visitTerminal( node );
    }
  }

  /**
   * Adds the given listener to the list of event notification recipients.
   *
   * @param listener A listener to begin receiving events.
   */
  public void add( ParseTreeListener listener ) {
    getListeners().add( listener );
  }

  /**
   * Removes the given listener to the list of event notification recipients.
   *
   * @param listener A listener to stop receiving events.
   * @return false The listener was not registered to receive events.
   */
  public boolean remove( ParseTreeListener listener ) {
    return getListeners().remove( listener );
  }

  /**
   * Returns the list of listeners.
   *
   * @return The list of listeners to receive tree walking events.
   */
  private List<ParseTreeListener> getListeners() {
    return this.listeners;
  }

  /**
   * Changes the list of listeners to receive events. If the given list of
   * listeners is null, an empty list will be created.
   *
   * @param listeners A list of listeners to receive tree walking
   * events.
   */
  public void setListeners( List<ParseTreeListener> listeners ) {
    if( listeners == null ) {
      listeners = createParseTreeListenerList();
    }

    this.listeners = listeners;
  }

  /**
   * Creates a CopyOnWriteArrayList to permit concurrent mutative
   * operations.
   *
   * @return A thread-safe, mutable list of event listeners.
   */
  protected List<ParseTreeListener> createParseTreeListenerList() {
    return new CopyOnWriteArrayList<ParseTreeListener>();
  }
}

[parrt]

Hi. Can you add yourself to https://github.com/antlr/antlr4/blob/master/contributors.txt? I can then add your code. thanks!

[ghost]

Updated.

[parrt]

Hi Dave. I thought about this overnight and I decided to be fairly stingy with adding runtime code, versus code in the tool itself, because I have five targets to maintain and keep in sync. For example, I would build some unit tests for this and that would mean porting the code and the unit test to all platforms. I wonder if there's a way to make this code available/visible without increasing my maintenance efforts.

[ghost]

Adding it to the documentation could be useful enough.

[parrt]

Excellent idea. I'll link to this page.

[yaccoff]

Thx very much for the code, its was exactly what I was after and worked out of the box.

[frndlytm]

So to clarify, this is different from registering listeners with a parser through

Parser parser = new Parser();
for (ParseListener listener: getListeners()) {
  parser.addParseListener(listener);
}

how?