Improving the performance of "Rule Expressions"

[abrahamwolk]

I would like to discuss two proposals for improving the performance of "Rule Expressions" in user-specified "Rules".

Alternative 1: Replace "Rule Expressions" with "Formula Expressions" as implemented in core-formula.

Existing work. On the branch JavaRules (see #2786), @shroffk has implemented a replacement of "Rule Expressions" (which on master currently are implemented by a compilation into Python scripts which are subsequently evaluated using the Jython interpreter on the UI thread) with "Formula Expressions" (which are evaluated by an interpreter written in Java that is located in core-formula). The original implementation is from 2019, and I have added some commits to this branch to make it compile when merged with the current master branch. However, the commits I have made were done without understanding the implementation, so they almost certainly break things.

Advantages and disadvantages of replacing "Rule Expressions" with "Formula Expressions". Replacing the current implementation of "Rule Expressions" with "Formula Expressions" implemented in core-formula has several advantages:

• A1. Having a unified language for formulas simplifies the user-experience, since the user only has to learn one language to express formulas.
• A2. Sharing the implementation for formulas simplifies the codebase, since only one formula language has to be implemented.
• A3. The implementation of "Formula Expressions" in core-formula is written in Java, and presumably provides significant performance benefits over the current implementation using Jython for two reasons:
  • A3.1 The evaluation of a rule could be executed on any thread (except for the final writing of the result to the widget); the current Jython implementation always evaluates rule expressions on the UI thread.
  • A3.2 The compilation of rule expressions can take significant time when loading an OPI that contains many rules. (This is in contrast to the execution of scripts that have already been compiled: the execution of compiled scripts doesn't seem to be inefficient except for the aforementioned fact that it always occurs on the UI-thread.)

There are also disadvantages to a replacement of Rule Expressions with Formula Expressions:

• D1. Replacing Rule Expressions with Formula Expressions will break backwards compatibility for the users.
• D2. Formula Expressions currently do not have a mechanism for abbreviating PV names. In contrast, the UI for specifying rules provides a means to give PVs abbreviated names: PVs are given names of the form pv0, pv1, ... . Since PV names tend to be long, formulas that contain PV names are difficult to read and modify.

A further point that has to be taken into account if we were to replace Rule Expressions with Formula Expressions is:

• P1. The execution should be deterministic in the sense that if there is a Rule that contains a list of Formula Expressions, then the first Formula Expression in the list that returns "true" should be the only one that "activates".

Changes to be implemented under alternative 1. The following two changes would be implemented:

• C1. Replace the current implementation of "Rule Expressions" with an implementation based on "Formula Expressions" similar to the implementation on the JavaRules branch, but with a deterministic semantics as described in P1.
• C2. Use the macro-facility to create abbreviations for PV names. Currently, it is not possible to define macro-bindings on most widgets: for instance, it is not possible to define macro bindings on the "Meter" widget. Abbreviations should likely in most cases be defined "locally" to where a Rule is defined that uses the abbreviation in question, and therefore I think we should add the ability to specify macro bindings on individual widgets (not just widgets that contain other widgets). This would to a large extent mitigate D2, I believe.

Breaking the backwards compatibility is of course still a concern (D1). I think that during a transition period, we should allow both "Rule Expressions" and "Formula Expressions", and then remove "Rule Expressions" after the transition period. One idea could be that we add a button "Translate to Formula Expressions" to the "Rules"-dialog, with which a user can automatically convert at least a large subset of "Rule Expressions into "Formula Expressions". After the transition period, the functionality to run "Rule Expressions" would be removed. We should also display warnings in appropriate places, that warn the user that "Rule Expressions" are being deprecated.

Alternative 2: Improving the performance of the current implementation.

It seems plausible that the advantages A3.1 and (to some extent) A3.2 can be obtained by the following two changes, which do not incur the disadvantages D1 and D2 (but also does not result in the advantages A1 and A2):

• If we make sure that "Rule Expressions" never can affect widgets directly, then scripts that are the result of compiling "Rule Expressions" could be executed on any thread (not just on the UI-thread), and only the final update of a widget property would have to be run in the UI-thread. This would mitigate A3.1.
• A cache of compiled versions of scripts could reduce the time spent compiling when re-loading OPIs. Perhaps the cache should be limited to the 100 (or some other number) of most recently run scripts, in order not to use too much RAM. This could perhaps mitigate A3.2 to some extent.

These two changes have the property that they provide advantages without any disadvantages: the current implementation of Rules would presumably be sped up, without any other noticeable changes. At least conceptually, it seems likely that these two changes should be relatively easy to implement.

Relevant GitHub Issues and Pull-Request. The following two GitHub Issues and Pull-Request may be relevant for this discussion:

• Issue Rules implemented in pure Java #893
• Issue Performance tests for pure java based rules #922
• Pull-Request for the branch JavaRules: Java rules #2786

[kasemir]

subsequently evaluated using the Jython interpreter on the UI thread

??
Scripts run in their own thread. For proof, see examples/solitaire where the script is an infinite while True:.. loop.
If that was running on the UI thread, you'd be dead.

There is exactly one jython script thread per display pane so that scripts for multiple displays can run in parallel while having only one per display to avoid issues with 'widget' or 'pvs' getting clobbered.

[shroffk]

Scripts run in their own thread. For proof, see examples/solitaire where the script is an infinite while True:.. loop.
If that was running on the UI thread, you'd be dead.

This might be my fault, I was under the impression that in order to be allowed to access widget properties and modify them the scripts need to run on the UI thread too.

[kasemir]

In BOY, scripts were initialized and executed on the UI thread, which caused 1..3 second freeze when opening a display that contains scripts.
The display builder widget model is thread-safe, so scripts can run in their own thread and still call widget.setPropertyValue(...).

If you want to call JavaFX UI code from the script, you do need to use Platform.runLater as for example in examples/3D.bob.

[abrahamwolk]

@kasemir: Thanks for the correction. Are you aware of any performance problems with the execution of the scripts that rules generate? If so, would it make sense to write a interpreter in Java (avoiding the overhead of Jython) for "Rule Expressions" in order to achieve improved performance?

[georgweiss]

Based on my (not very recent) analysis the performance penalty is with the loading/reloading of the OPIs when rules are converted and compiled.

As for using a cache for rules to avoid the conversion/compilation step... In my view that is a bandaid. Let's spend resources to find a better solution, even at the potential cost of dealing with backwards compatibility. I do not believe a cache would be super trivial, e.g. it would need to update or flush when user has edited an OPI. As for the size of a cache... I have seen cases where a single OPI contains several hundreds of rules.

[abrahamwolk]

I agree that ensuring that the values in the cache are not out-of-date could potentially be tricky.

There are two separate "issues" with the current Rule Expressions, I think:

1. The implementation of Rule Expressions is in terms of compilation into Python and execution using Jython.
2. The language that is used to define Rule Expressions is not the same language that is used to define Formulas ("Formula Expressions").

If we want to solve 1. and 2. by replacing Rule Expressions with Formula Expressions, I don't think we should introduce this change from one release to another, since it would break backwards compatibility. I think that we would need to manage the transition well, due to the number of OPIs in production that use Rule Expressions. If we proceed along this path, then I suggest that we allow for both Rule Expressions and Formula Expressions in Rules during a transition period, during which Rule Expressions are marked as being deprecated, and at the end of which Rule Expressions are finally removed from the code base.

However, we should, I think, also keep in mind that we can fix 1. with no impact on 2. by writing an interpreter for Rule Expressions in Java. (Or, an alternative solution could perhaps be to introduce a cache of compiled objects.) I prefer this option since it preserves backwards compatibility.

[kasemir]

would it make sense to write a interpreter in Java (avoiding the overhead of Jython) ..

Frankly, no. Just like scripts, rules should be the exception. I've not seen the performance of rules or scripts as a showstopper. If your displays rely on rules and scripts, consider switching to PyDM where every display is a custom python program. I wish you the best.

the performance penalty is with the loading/reloading of the OPIs when rules are converted and compiled

I think that's correct.

Here's a screenshot of jProfiler after re-loading examples/09_rules.bob many times.

The context menu for getting to "Reload" and then disposing the existing display take a little time.
Next:

1. openXMLDocument: Opening the display file, parsing the XML
2. compile: Compiling the generated script, but note that loadCode is the actual culprit. More on that below
3. PythonInterpreter.execute is comparably cheap, on par with the resulting formatNumber when the rule-based value is represented on the screen

As for the Jython compilation, I don't remember the full detail, but some evidence is left in the JythonScriptSupport comments. Most of the time was spent in creating and initializing the jython interpreter. This relates to loading the site.py file, which I think we were able to disable via python.import.site=false. Another speedup is possible by leaving the xxx$py.class bytecode files cached, but issues with file permissions caused us to disable that.

So there might be ways to speed up the Jython startup.

Replacing the current rule implementation with something else like a Java implementation runs the risk of breaking existing displays. I'm not sure it's worth it since again rules/scripts should be the exception.

[kasemir]

.. replacing Rule Expressions with Formula Expressions, I don't think we should introduce this change from one release to another, since it would break backwards compatibility

I'd go as far as making the formula expressions a new, separate thing.
Like rules now have the "Value as expression" option, there'd be a new "Formula Expression" option.
Or radio with options

1. Value as text (not sure what to call it, it's what you now get when NOT checking the "Value as expression" option)
2. Value as expression (what you now get when checking the "Value as expression" option)
3. Formula (that new option)

[shroffk]

Ok, I think this might be the way to go.
We avoid the complication of backward compatibility, but at the same time move towards using the formulas instead of rules ( cleaner scripts )

We could also make the formula as the default choice amongst the 3 for new rules

[kasemir]

Yes, offer it as the default choice within the same rules config dialog, and maybe arrange and name it to nudge users in the preferred direction, like options

1. Formula (that new option, default)
2. Legacy Value as text (not sure what to call it, it's what you now get when NOT checking the "Value as expression" option)
3. Legacy Value as expression (what you now get when checking the "Value as expression" option)

[georgweiss]

How about showing legacy options only if there is a legacy rule, i.e. only formula for new rules. Motivation is that user might fall back to the options that he/she is familiar with.

[abrahamwolk]

I suggest the following way to proceed:

1. We implement Kay's suggestion with Formula, Legacy Value as text, and Legacy Value as expression.
2. We add an option of the form deprecation_warnings_for_legacy_rules to the phoebus.ini options that enables deprecation warnings under Legacy Value as text and Legacy Value as expression.
3. We add an option of the form phase_out_legacy_rules that implements a ratchet-like behavior as Georg suggested, where new rules must be written using the Formula option.

The idea is that each site can decide whether or not they want to deprecate Legacy Rules, and also do so at their own pace. Sites could also support the deprecation with other tools, such as adding checks to Git repositories check for the presence of Legacy Rules in OPIs in pull-requests, etc.

I think we should leave the functionality to load, run and edit OPIs that contain Legacy Rules in Phoebus: there may be OPIs that are rarely used that contain Legacy Rules, and there may be sites that see no reason (or don't have the resources to spend) to rewrite Legacy Rules.