feat: Dataflow analysis framework #1476

acl-cqc · 2024-08-28T11:22:37Z

Forwards analysis only ATM, parametrized over the abstract domain hence intended to support not only constant folding but (in the future) devirtualization, intergraph-edge-insertion, etc. See #1603 for an "example" use for constant-folding, where it does better than the existing code.

Much complexity is to do with "native" (irrespective of the underlying domain) treatment of Sum types necessary for proper understanding of control flow (e.g. conditionals, loops, CFGs).

Note: we'll be able to separate the DFContext from the HugrView, which'll be neater, if we do #1636 first.

TODO: update with BREAK_TAG / CONTINUE_TAG from feat: Add TailLoop::BREAK_TAG and CONTINUE_TAG #1626

Intended as a development of #1157, with significant changes:

Constant-folding and ValueHandle now stripped out, these will follow in a second PR
Everything is now in hugr-passes
Underlying domain of values abstracted over a trait AbstractValue (ValueHandle will implement this), which represents non-Sum values
datalog uses PartialValue wrapped around the AbstractValue to represent (Partial)Sums and make into a BoundedLattice
The old PV is gone (PartialValue directly implements BoundedLattice)
Interpretation of leaf (extension) ops is handled by the DFContext trait (although MakeTuple, and Untuple are handled by the framework - really prelude MakeTuple is just core Tag and Untuple is a single-Case Conditional with passthrough wires....); the framework handles routing of sums through these ops and all containers, also loading constants (with the DFContext handling non-Sum leaf Values).
Various refactoring of handling values (inc. in datalog) - variant_values+as_sum + more use of rows rather than indexing (this got rid of a bunch of unwraps and so on), significant refactoring of join/meet (and no _unsafe).
I've managed to refactor tests not to use ValueHandle etc. - they are only dealing with sum/loop/conditional routing after all. dataflow/test.rs uses about the simplest possible TestContext which provides zero information after any leaf-op - so we only get the framework-provided handling of Tag/MakeTuple/etc.

propolutate_out_wires largely superceded by passing root-node inputs into Machine::run, but still available for tests.

codecov · 2024-09-02T14:44:48Z

Codecov Report

Attention: Patch coverage is 85.45619% with 161 lines in your changes missing coverage. Please review.

Project coverage is 85.87%. Comparing base (5562c91) to head (da2981c).

Files with missing lines	Patch %	Lines
hugr-passes/src/dataflow/datalog.rs	80.00%	30 Missing and 27 partials ⚠️
hugr-passes/src/dataflow/partial_value.rs	86.72%	47 Missing and 2 partials ⚠️
hugr-passes/src/dataflow.rs	43.18%	25 Missing ⚠️
hugr-passes/src/dataflow/results.rs	74.19%	13 Missing and 3 partials ⚠️
hugr-passes/src/dataflow/value_row.rs	70.83%	14 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1476      +/-   ##
==========================================
+ Coverage   85.79%   85.87%   +0.07%     
==========================================
  Files         136      142       +6     
  Lines       25180    26287    +1107     
  Branches    22092    23199    +1107     
==========================================
+ Hits        21603    22573     +970     
- Misses       2427     2532     +105     
- Partials     1150     1182      +32

Flag	Coverage Δ
python	`92.42% <ø> (ø)`
rust	`84.99% <85.45%> (+0.13%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…t Hash)

* DFContext reinstate fn hugr(), drop AsRef requirement (fixes StackOverflow) * test_tail_loop_iterates_twice: use tail_loop_builder_exts, fix from #1332(?) * Fix only-one-DataflowContext asserts using Arc::ptr_eq

…s what was meant

…g len

…text interprets load_constant

acl-cqc · 2024-10-28T17:45:17Z

I had to recombine DFContext with the Hugr to get round some Rust mutable/immutable reference ownership issues in #1603; I tried AsRef<Hugr> (sufficient to get you the HugrView methods - or indeed just requiring HugrView, as implementing AsRef gets you that anyway), but have gone back to Deref<impl HugrView> as this should allow running analysis on a region (e.g. DescendantsView). I guess I could be persuaded to do something like

trait DFContext<V> {
  type View : HugrView; // as at present
  fn view() -> &Self::View
}

but the extra .view() before every HugrView call seems a bit of a pain (@doug-q do you feel strongly?).

…_value

…pe params

acl-cqc · 2024-10-30T14:25:38Z

hugr-passes/src/dataflow/partial_value.rs

+    /// If this PartialSum had multiple possible tags; or if `typ` was not a [TypeEnum::Sum]
+    /// supporting the single possible tag with the correct number of elements and no row variables;
+    /// or if converting a child element failed via [PartialValue::try_into_value].
+    pub fn try_into_value<VE, SE, V2: TryFrom<V, Error = VE> + TryFrom<Sum<V2>, Error = SE>>(


Yes, this is highly parametric. The first case will be the "easy" one, where V2 is hugr_core::ops::Value, in which case VE = Infallible and SE = ConstTypeError. But I think it's important to keep this open for other analysis domains.

It might be more legible if written <V2, VE, SE> ..... where V2: TryFrom<V, Error=VE> + TryFrom<Sum<V2>, Error=SE>>, happy to do that if anyone shouts.

The bigger issue may be nomenclature - this relates to ExtractValueError so perhaps should be (try_)extract_value? And also to Machine::try_read_wire_value - so, try_extract_wire_value?? However I quite like the word "concrete" so could perhaps use that naming to tie the three together - "try_into_concrete", "try_read_wire_concrete", and, erm, "ConcretizationError" (hmmm, not so good....). Thoughts?

It might be more legible if written <V2, VE, SE> ..... where V2: TryFrom<V, Error=VE> + TryFrom<Sum<V2>, Error=SE>>, happy to do that if anyone shouts.

I think you should!

Done - yes good to standardize the order of the args with try_read_wire_value.

Any thoughts on naming?

acl-cqc · 2024-10-30T14:27:43Z

hugr-passes/src/dataflow/datalog.rs

+
+impl<V: AbstractValue> Machine<V> {
+    /// Provide initial values for some wires.
+    // Likely for test purposes only - should we make non-pub or #[cfg(test)] ?


Not sure quite what to do here. @doug-q may have opinions?

We think this potentially useful (albeit not very powerful, as any prepopulated value will be joined with anything computed). A version that "breaks" the wire (replacing any value computed, without join) would be more effective, also dangerous....

acl-cqc · 2024-10-30T14:29:31Z

hugr-passes/src/dataflow/datalog.rs

+                    .push((root, IncomingPort::from(i), PartialValue::Top));
+            }
+        }
+        // Note/TODO, if analysis is running on a subregion then we should do similar


I'll raise an issue for this. It's not clear how to identify nonlocal edges into a subregion, or to differentiate between nonlocal edges whose source we can't see, and ports with no edge at all, where perhaps "Bottom" is appropriate (it really can't happen, that bit of Hugr can't run....).

croyzor · 2024-11-06T16:44:21Z

hugr-passes/src/dataflow.rs

+        _e: &ExtensionOp,
+        _ins: &[PartialValue<V>],
+        _outs: &mut [PartialValue<V>],
+    ) {


You don't need to provide default implementations for trait methods, you can just write fn f(...); instead of fn f(...) { }

But this impl is totally ok as a default - it means, I know nothing about any leaf op

acl-cqc · 2024-11-08T10:08:32Z

Note we could separate the DFContext from HugrView - which would be significantly neater - if we did #1636 first, but we need that because ValidationLevel::run_validated_pass only gives us a &mut impl HugrMut.

acl-cqc force-pushed the acl/const_fold2 branch 2 times, most recently from 1594e6f to e1c49d7 Compare August 28, 2024 17:31

acl-cqc requested a review from doug-q September 2, 2024 09:09

acl-cqc changed the title ~~DRAFT(v2?) Datalog-style constant-folding skeleton~~ feat: Dataflow analysis framework and use for constant-folding Sep 2, 2024

acl-cqc force-pushed the acl/const_fold2 branch from 718f058 to 6698b1b Compare September 2, 2024 14:42

acl-cqc force-pushed the acl/const_fold2 branch from e54c742 to 6cac41a Compare September 2, 2024 15:46

acl-cqc added 23 commits September 11, 2024 16:15

Just const_fold2 + inside that partial_value (taken from hugr_core)

c7a9d89

merge/update+fmt (ValueName for ConstInt non-compiling as ConstInt no…

ac45e53

…t Hash)

Missing imports / lints. Now running, but failing w/StackOverflow

8adaa6e

Fix tests...

098c735

* DFContext reinstate fn hugr(), drop AsRef requirement (fixes StackOverflow) * test_tail_loop_iterates_twice: use tail_loop_builder_exts, fix from #1332(?) * Fix only-one-DataflowContext asserts using Arc::ptr_eq

ValueKey using MaybeHash

706c892

tag() does not refer to self.is_compound

63bc944

ValueHandle::{is_compound,num_fields,index} => {variant_values, as_sum}

5fa7edb

Rm ValueHandle::tag, use variant_values - inefficient, presume this i…

98bf94a

…s what was meant

add variant_values, rewrite one use of outputs_for_variant

295ec32

...and the other two; remove outputs_for_variant

5c8289e

Rewrite tuple rule to avoid indexing

bf173ab

GC unused (tuple,variant)_field_value, iter_with_ports

0ae4d19

Common up via ValueRow.unpack_first

8608ba9

No DeRef for ValueHandle, just add get_type()

2dca3e9

ValueKey::{Select->Field,index->field}

51e68ea

(join/meet)_mut_unsafe => try_(join/meet)_mut with Err for conflictin…

8635474

…g len

Remove ValueHandle::variant_values - just have as_sum

80d5b86

Optimize as_sum() by returning impl Iterator not Vec

1c8be99

Machine uses PV not PartialValue

b0afa54

Parametrize PartialValue+PV+Machine by AbstractValue/Into<Value>, Con…

d09a1fe

…text interprets load_constant

Move partial_value.rs inside datalog/

af8827b

Hide PartialSum/PartialValue

4b61436

refactor: ValueRow::single_among_bottoms

8b31d8c

ValueRow Debug; ops default to PartialValue::Top less aggressively

a0f2b2c

acl-cqc added 10 commits October 29, 2024 19:52

And back to Deref, should allow using a region view not the whole Hugr

87eb700

fix doclink

a49221b

PartialSum::try_into_value also uses Option<...> as error-type

71ea55d

Proper errors from try_into_value, Option<Extr...> from try_read_wire…

ea9db2e

…_value

fmt

df31523

Add test running on region

a490874

fix: provide PartialValue::Top for unspecified Hugr inputs

3af39aa

try_into_value allows TryFrom by giving ExtractValueError *2* errorty…

dc15999

…pe params

improve docs

2d81264

Parametrize Machine::try_read_wire_value the same way

b61d252

acl-cqc commented Oct 30, 2024

View reviewed changes

tweaks

8cac194

acl-cqc force-pushed the acl/const_fold2 branch from 00ff09b to 8cac194 Compare October 31, 2024 14:32

acl-cqc added 2 commits November 4, 2024 12:52

Massively simplify xor_and_cfg, no need for conditionals

fb3816e

Use tru/fals constants

19571f6

acl-cqc self-assigned this Nov 6, 2024

croyzor reviewed Nov 6, 2024

View reviewed changes

cqc-alec requested review from croyzor and removed request for cqc-alec November 7, 2024 09:37

acl-cqc added 3 commits November 8, 2024 10:00

try_into_value: reorder type params, separate out where clause

69d0f5e

Merge remote-tracking branch 'origin/main' into HEAD

cb60b5b

We don't actually use portgraph, nor downcast-rs

2624ee8

acl-cqc added 3 commits November 8, 2024 10:23

Import RandomState from std::collections::hash_map for rust 1.75

ec526e8

Use BREAK_TAG/CONTINUE_TAG

5650ee4

No, use make_break/make_continue for easy cases

da2981c

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Dataflow analysis framework #1476

feat: Dataflow analysis framework #1476

acl-cqc commented Aug 28, 2024 •

edited

Loading

codecov bot commented Sep 2, 2024 •

edited

Loading

acl-cqc commented Oct 28, 2024 •

edited

Loading

acl-cqc Oct 30, 2024

croyzor Nov 6, 2024

acl-cqc Nov 8, 2024

acl-cqc Nov 8, 2024

acl-cqc Oct 30, 2024

acl-cqc Nov 8, 2024

acl-cqc Oct 30, 2024

croyzor Nov 6, 2024

acl-cqc Nov 8, 2024

acl-cqc commented Nov 8, 2024

feat: Dataflow analysis framework #1476

Are you sure you want to change the base?

feat: Dataflow analysis framework #1476

Conversation

acl-cqc commented Aug 28, 2024 • edited Loading

codecov bot commented Sep 2, 2024 • edited Loading

Codecov Report

acl-cqc commented Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acl-cqc commented Nov 8, 2024

acl-cqc commented Aug 28, 2024 •

edited

Loading

codecov bot commented Sep 2, 2024 •

edited

Loading

acl-cqc commented Oct 28, 2024 •

edited

Loading