Consolidate walker logic in LP/NL representations #3015

jsiirola · 2023-10-12T20:24:31Z

Fixes # .

Summary/Motivation:

This PR reduces repetition in the codebase by consolidating beforeChild and exitNode logic in the LP and NL representation walkers into standard implementations of BeforeChildDispatcher and ExitNodeDispatcher. By consolidating logic we can ensure that all coefficients / values in the walker return data structures are guaranteed to be native_numeric_types, allowing us to simplify (and slightly speed up) the LP and NL walkers.

As part of this, we are making a change in the definition of the native_numeric_types set by removing complex from that set, and registering complex types in a separate native_complex_types set. This change makes for a more consistent definition of native_numeric_types: it contains only data types that are "equivalent to float" and can be emitted as part of standard solver interfaces.

Changes proposed in this PR:

Define BeforeChildDispatcher and ExitNodeDispatcher classes to centralize the definition and management of the multiple dispatch dictionaries used in the representation walkers
Remove complex from native_numeric_types (and put in a separate native_complex_types set
Expand type registration documentation
Update / expand some of the numpy type registrations

Legal Acknowledgement

By contributing to this software project, I have read the contribution guide and agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the BSD license.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

codecov · 2023-10-12T21:35:04Z

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (12be54c) 87.84% compared to head (db38e94) 87.87%.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3015      +/-   ##
==========================================
+ Coverage   87.84%   87.87%   +0.03%     
==========================================
  Files         769      769              
  Lines       89541    89516      -25     
==========================================
+ Hits        78657    78665       +8     
+ Misses      10884    10851      -33

Flag	Coverage Δ
linux	`85.20% <96.61%> (+0.03%)`	⬆️
osx	`75.00% <96.04%> (+0.02%)`	⬆️
other	`85.38% <96.61%> (+0.02%)`	⬆️
win	`82.42% <96.04%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pyomo/common/numeric_types.py	`92.94% <100.00%> (+0.53%)`	⬆️
pyomo/core/kernel/register_numpy_types.py	`100.00% <100.00%> (ø)`
pyomo/repn/plugins/lp_writer.py	`93.65% <ø> (+0.59%)`	⬆️
pyomo/repn/quadratic.py	`93.08% <100.00%> (ø)`
pyomo/repn/util.py	`99.38% <100.00%> (+0.23%)`	⬆️
pyomo/util/calc_var_value.py	`100.00% <100.00%> (ø)`
pyomo/common/dependencies.py	`97.68% <84.61%> (-0.60%)`	⬇️
pyomo/repn/linear.py	`97.75% <97.32%> (+1.28%)`	⬆️
pyomo/repn/plugins/nl_writer.py	`95.47% <95.14%> (+1.45%)`	⬆️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

emma58

This is really cool--I like it! Mostly just questions to make sure I'm understanding...

emma58 · 2023-10-18T19:39:46Z

pyomo/common/numeric_types.py

-    """
-    A utility function for updating the set of types that are
-    recognized to handle numeric values.
+def RegisterNumericType(new_type: type):


Doesn't really bother me (as that would by hypocritical), but wouldn't it be PEP8-ier for these be snake case rather than camel case since they're functions?

Yes - they should be snake case. But as this has been an "official" API for a long time, we would need to rename it with a deprecation path (which I'm OK with, but maybe as a separate PR?).

Ah, I see. A separate PR makes sense.

If we're going to do that... We have these in several places.

emma58 · 2023-10-18T19:41:45Z

pyomo/common/numeric_types.py

+    expressions.  The type should be compatible with :py:class:`float`
+    (that is, store a scalar and be castable to a Python float).


Is this a copy-paste typo? Should they be castable to int rather than float? (I mean, Python is happy to cast floats to ints but...)

Believe it or not, it is not a typo. The assumption in the Numeric expression system is that the value of any expression should be float-like.

This set is actually not used anywhere in Pyomo. I think it was added in the dawn of time on speculation that it might be useful. Maybe the right thing is to deprecate it entirely?

I would be in favor of deprecating it before someone starts using it and we get really confused. :P

emma58 · 2023-10-18T19:46:54Z

pyomo/common/numeric_types.py

+def RegisterLogicalType(new_type: type):
+    """Register the specified type as a "logical type".
+
+    A utility function for registering new types as "native logical
+    types".  Logical types can be leaf nodes in Pyomo logical
+    expressions.  The type should be compatible with :py:class:`bool`
+    (that is, store a scalar and be castable to a Python bool).


What goes in native_logical_types that doesn't go in native_boolean_types? Or is this a backwards compatibility thing?

It's a backwards compatibility thing. native_boolean_types is the deprecated set and was replaced by native_logical_types. I toyed with making them explicit aliases for each other, but elected to not continue mucking with a deprecated API.

emma58 · 2023-10-18T20:04:35Z

pyomo/repn/util.py

+_CONSTANT = ExprType.CONSTANT
+
+
+class BeforeChildDispatcher(collections.defaultdict):


Oh, yeah, this is very cute... I like it.

emma58 · 2023-10-18T20:09:59Z

pyomo/repn/util.py

+    # @staticmethod
+    # def _before_var(visitor, child):
+    #     pass
+
+    # @staticmethod
+    # def _before_named_expression(visitor, child):
+    #     pass


Why are these commented out?

Mostly because I don't like ABCMeta for classes in tight loops. These are two methods that are required by register_dispatcher, but are not implemented in the base class (the LP and NL implementations are completely different). They are commented out to remind developers of an expected API...

But isn't the advantage of ABCMeta that it will actually yell at you when you don't implement the things you expect to be there?

+1 to Emma's comment.

emma58 · 2023-10-18T20:25:05Z

pyomo/repn/plugins/lp_writer.py

-            if offset.__class__ not in int_float:
-                offset = float(offset)


Why don't you have to cast these anymore? Is it a change from this PR, or was it already redundant?

The new walkers are better at catching non-float-like types when processing the leaf nodes, so we no longer have to re-verify the return types from the walker.

mrmundt

One NFC that you can ignore if you want (it's a typo that crate-ci doesn't catch, but I added it to the October list, so it'll be caught soon regardless).

pyomo/common/dependencies.py

mrmundt · 2023-10-23T13:20:29Z

pyomo/common/numeric_types.py

-    """
-    A utility function for updating the set of types that are
-    recognized to handle numeric values.
+def RegisterNumericType(new_type: type):


If we're going to do that... We have these in several places.

mrmundt · 2023-10-23T13:26:54Z

pyomo/repn/util.py

+    # @staticmethod
+    # def _before_var(visitor, child):
+    #     pass
+
+    # @staticmethod
+    # def _before_named_expression(visitor, child):
+    #     pass


+1 to Emma's comment.

jsiirola added 30 commits September 7, 2023 04:49

Add centralized beforeChild/exitNode dispatcher classes

1073c3e

Standardize handling of named expression types

d4434bd

Add docs, improve method naming

1a3e34a

NFC: remove unreachable code; update comments

10a3b95

Fix typos

ea1dc29

Split complex types out from native_numeric_types

8504c05

Update list of numpy types

9143204

Remove repeated constant resolution code

e0729db

NFC: comments

e80bf23

Remove redundant checks for int/float domains

1d89380

Apply black

8876e25

update tests to track dispatcher api changes

0c52ea0

Merge branch 'main' into repn-dry-lp-nl-beforechild

596a45f

Remove unused import

0b60875

Apply black

2640630

Check for platform-specific float/complex types

1c65805

Leverage new native_complex_types set

d122490

Fix base class for detecting named subexpressiosn

431e8bb

relax kernel test to track differences in numpy builds

14def43

relax kernel test to track differences in numpy builds

f31e005

Merge branch 'main' into repn-dry-lp-nl-beforechild

7a552a0

Merge branch 'main' into repn-dry-lp-nl-beforechild

6d3f77c

Add missing import

000cd86

Improve exception/InvalidNumber messages

09c9c2f

Only cache type / unary, binary, and ternary dispatchers

b706fe1

Test BeforeChildDispatcher / ExitNodeDispatcher

5a217cd

Merge branch 'main' into repn-dry-lp-nl-beforechild

9d3b38c

Clarify docstring / exception messages

706d27c

Slotize derived dispatchers

9171886

Merge branch 'main' into repn-dry-lp-nl-beforechild

2b6e111

Apply black

63aba93

jsiirola requested review from mrmundt and emma58 October 16, 2023 15:47

emma58 reviewed Oct 18, 2023

View reviewed changes

emma58 approved these changes Oct 18, 2023

View reviewed changes

mrmundt reviewed Oct 23, 2023

View reviewed changes

NFC: fix comment typo

9219573

mrmundt approved these changes Oct 23, 2023

View reviewed changes

Merge branch 'main' into repn-dry-lp-nl-beforechild

db38e94

jsiirola merged commit 47a9b26 into Pyomo:main Oct 23, 2023
30 checks passed

jsiirola deleted the repn-dry-lp-nl-beforechild branch October 23, 2023 22:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Consolidate walker logic in LP/NL representations #3015

Consolidate walker logic in LP/NL representations #3015

jsiirola commented Oct 12, 2023

codecov bot commented Oct 12, 2023 •

edited

Loading

emma58 left a comment

emma58 Oct 18, 2023

jsiirola Oct 18, 2023

emma58 Oct 18, 2023

mrmundt Oct 23, 2023

emma58 Oct 18, 2023

jsiirola Oct 18, 2023

emma58 Oct 18, 2023

emma58 Oct 18, 2023

jsiirola Oct 18, 2023

emma58 Oct 18, 2023

emma58 Oct 18, 2023

jsiirola Oct 18, 2023 •

edited

Loading

emma58 Oct 18, 2023

mrmundt Oct 23, 2023

emma58 Oct 18, 2023

jsiirola Oct 18, 2023

mrmundt left a comment

mrmundt Oct 23, 2023

mrmundt Oct 23, 2023

		expressions. The type should be compatible with :py:class:`float`
		(that is, store a scalar and be castable to a Python float).

		_CONSTANT = ExprType.CONSTANT


		class BeforeChildDispatcher(collections.defaultdict):

Consolidate walker logic in LP/NL representations #3015

Consolidate walker logic in LP/NL representations #3015

Conversation

jsiirola commented Oct 12, 2023

Fixes # .

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

codecov bot commented Oct 12, 2023 • edited Loading

Codecov Report

emma58 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsiirola Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrmundt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Oct 12, 2023 •

edited

Loading

jsiirola Oct 18, 2023 •

edited

Loading