Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rethinking goal expansion #2532

Open
triska opened this issue Sep 5, 2024 · 17 comments
Open

Rethinking goal expansion #2532

triska opened this issue Sep 5, 2024 · 17 comments

Comments

@triska
Copy link
Contributor

triska commented Sep 5, 2024

Goal expansion would really benefit from thinking more about it, and maybe changing the overall approach. See also #2433 (comment).

At least the following concerns come to mind:

  1. There are cases where we want goal expansion, and it is very hard to use correctly, i.e., in such a way that the expanded code has exactly the same semantics as the non-expanded code. For a recent example, see Goal expansion in lib reif #2433. Note especially that the goal call(!) is different from a plain ! appearing in a clause, and a correct expansion must be careful not to replace a meta-called goal !/0 by an actual ! in the resulting code.
  2. At the same time, it seems unsatisfactory to wrap expanded goals needlessly in call/1, even though such an "overly cautious" expansion could itself be mitigated by other means that are maybe also worth implementing by themselves, or even also by goal expansion rules.
  3. What about dynamic goals such as maplist(Goal, ...), or even just Goal (i.e., call(Goal)). Should such cases even be goal-expanded? It may seem pointless overhead to produce an expansion at run-time instead of executing the goal directly. On the other hand, this may lead to massive performance differences for instance when timing goals on the toplevel vs. in programs, and that may give casual Prolog programmers a completely wrong impression about the "actual" performance of constructs that occur in programs. Still worth thinking about, and maybe just the toplevel should do such a "dynamic" expansion?
  4. There are cases where we want goal expansion, with exceptions. For instance, nobody benefits from seeing massive amounts of goal-expanded Prolog code appear in reasons produced by library(diadem) (see Porting library(diadem) to Scryer Prolog #840) or even in answers of clause/2. At the same time, there must be a way to inspect the expansion so that we can see it when needed. Is there a way to convey the intent to a goal expansion mechanism, for instance by stating exceptions to the expansion?
  5. Can this all be satisfied while still giving certain guarantees (even if only by stating cases in which the expansion yields unpredictable results and should not be used), instead of resulting in a collection of ad hoc constructs? How does it all relate to meta_predicate/1 declarations?
  6. SICStus Prolog has a sophisticated mechanism for term and goal expansion. Should we aim for compatibility with SICStus? Or other existing mechanisms?
  7. library(lambda) seems to need a very similar or at least closely related mechanism. See also Goal expansion in lib reif #2433 (comment).
@hurufu
Copy link
Contributor

hurufu commented Sep 10, 2024

I've started collecting changes for rethinking goal expansion on top of branch #2433 as a separate draft PR, to make them clearly separated. I've decided to proceed with TDD approach so I started with writing a lot of test cases and benchmarks.

@UWN
Copy link

UWN commented Sep 10, 2024

One remark: The removal of unnecessary call/1 wrappers is particularly important for executions modes with the occurs-check. There, the overheads are substantial (and not just 30% or so) see this for more.

@jjtolton
Copy link

jjtolton commented Oct 6, 2024

Now that I have a better understanding per #2598 (comment), it seems to me that these predicates fill a very similar function to the macro-expansion phase in lisp languages.

I would suggest one possibility that would be more efficient, if you only want a subset of goals/terms in a module rather than ALL terms, would
be something like a directive to indicate that certain predicates/terms are to be expanded.

This is similar to designating something as a macro in a lisp. Also, using a term expansion or goal expansion directive near the predicate it operates on would make code a lot more readable, because it can be very unintuitive by looking at a predicate in isolation, with no other information, metadata, etc, that it is meant to be used in conjunction with an expansion mechanism.

@hurufu
Copy link
Contributor

hurufu commented Oct 6, 2024

[...] something like a directive to indicate that certain predicates/terms are to be expanded.

You can already have it for your own goal expansions:

expansion_enabled(bar(_)).

user:goal_expansion(G, X) :-
    nonvar(G),
    expansion_enabled(G),
    ... .

foo(X) :-
    bar(X).  % bar/1 will be expanded

@triska
Copy link
Contributor Author

triska commented Oct 6, 2024

Note that the main issue is not whether to perform goal expansion at all, or which terms should be subject to goal or term expansion. As I see it, the main issue we are facing is: How do we express that in certain situations, we want to refer to the actual source code, while in others we want to refer to the expanded version? For instance, when showing a listing or program slice, we would like to be able to show the actual source code, and maybe selectively also the expanded version.

@jjtolton
Copy link

Note that the main issue is not whether to perform goal expansion at all, or which terms should be subject to goal or term expansion. As I see it, the main issue we are facing is: How do we express that in certain situations, we want to refer to the actual source code, while in others we want to refer to the expanded version? For instance, when showing a listing or program slice, we would like to be able to show the actual source code, and maybe selectively also the expanded version.

Hmmm. In lisp this problem is addressed by macroexpand-1 vs macroexpand-all, which does a one step expansion vs a recursive expansion.

I could imagine a predicate that returns as leaf answers progressively expanded by goals/terms until the base case expansion.

I don't know enough yet about how term expansion or goal expansion work yet to imagine the internals, however for this and other reasons it makes me wonder if some kind of goal/term expansion with continuation might be desirable.

@notoria
Copy link
Contributor

notoria commented Dec 12, 2024

In the case 3, dynamic goals such as maplist(Goal, ...), or even just call(Goal) should not be expanded at compile-time or run-time.

goal_expansion(a, b).

test :-
   T0 = (G = a,call((G,a)),a),
   expand_goal(T0, T),
   T = (G = a,call((G,a)),b).

At compile-time, the a in the third goal of the conjunction will be expanded to b. If the a in the second goal is expanded to b then at run-time the a must be expanded to b too.

Allowing goal expansion in dynamic goals requires run-time goal expansion.

Now goal expansion can be blocked with call/N.

If the reference implementation of if_/3 is the one without goal expansion which blocks goal expansion then even the expanded version of if_/3 will block goal expansion.

@UWN
Copy link

UWN commented Dec 12, 2024

If the reference implementation of if_/3 is the one without goal expansion which blocks goal expansion then even the expanded version of if_/3 will block goal expansion.

Not sure what you mean. But yes, the reference implementation of if_/3 doe not use goal expansion. Thus if you use goal expansion to improve its efficiency, one has to take into account the very error situations that may happen. Unregulated goal expansion is just a receipt for random behaviour.

@notoria
Copy link
Contributor

notoria commented Dec 12, 2024

Before we continue, a small detour.

Does term-to-body conversion happen before goal expansion?

Can a goal expand into a term that is not a body for execution? (i.e. goal_expansion(integer, 1)).

@UWN
Copy link

UWN commented Dec 12, 2024

Does term-to-body conversion happen before goal expansion?

Goal expansion is something the standard does not define. For good reasons as we see even now, thirty years later, there is no clear convergence.

The standard defines, however, something very much related. This term-to-body conversion. There the following happens: variables-as-goals are wrapped with call/1. Numbers as goals produce an error.

Given all that, it seems most sensible to perform such an expansion prior to the final term-to-body conversion. And at least that is what is done for DCGs in DTR 13211-3. Also, think of it, the purpose of that conversion is to ensure that we do not have numbers or variables as goals, so it seems to me we have to put this last.

Can a goal expand into a term that is not a body for execution? (i.e. goal_expansion(integer, 1)).

Given above considerations, yes, it can, but then immediately the final conversion will catch it and produce an error.

@notoria
Copy link
Contributor

notoria commented Dec 12, 2024

it seems most sensible to perform such an expansion prior to the final term-to-body conversion.

It may make things simpler and goal expansion more troublesome.

The predicate goal_expansion/2 cannot see variable at expansion-time otherwise the variable could be instantiated. Which mean in ?- expand_goal((a,B,c), _), the goal B is invisible.

@UWN
Copy link

UWN commented Dec 12, 2024

Good point. This shows that goal expansion is much more restricted than general term expansion. As for DCGs, the entire body is converted in one fell swoop using the logical expansion and (effectively) replacing variables in place of a non-terminal by phrase//1. Actually, the expansion directly translates it to phrase/3.

@notoria
Copy link
Contributor

notoria commented Dec 12, 2024

Without module, an implementation of if_/3 which doesn't block goal expansion based on source:

Example

:- module(reif0, [if_/3]).

:- use_module(library(dif), [dif/2]).
:- use_module(library(lists), [append/3]).

is_convertible_term(T) :-
    catch((! ; T), error(type_error(callable,_),_), false).

is_body(T) :-
    \+ \+ (
        term_variables(T, Vs),
        append(Vs, _, [0|Vs]),
        is_convertible_term(T)
    ).

has_cut(T) :-
    var(T), !,
    throw(error(instantiation_error,has_cut/1)).
has_cut(!) :-
    !.
has_cut((G0,G)) :-
    !,
    has_cut(G0),
    has_cut(G).
has_cut((G0;G)) :-
    !,
    has_cut(G0),
    has_cut(G).
has_cut((_->G)) :-
    !,
    has_cut(G).
has_cut(_).

user:goal_expansion(if_(If_1, Then_0, Else_0), G_0) :-
    ugoal_expansion(if_(If_1, Then_0, Else_0), G_0).

% Bug?
% :- use_module(library(format)).
% :- use_module(library(debug)).
% user:goal_expansion(G0, G) :-
%     portray_clause(user_error, "almost"),
%     ugoal_expansion(G0, G).

body_expansion(G_0, G) :-
    is_body(G_0), \+ has_cut(G_0), !,
    G = G_0.
body_expansion(G_0, call(G_0)).

=(X, Y, T) :-
    X == Y, !,
    T = true.
=(X, Y, T) :-
    X \= Y, !,
    T = false.
=(X, X, true).
=(X, Y, false) :-
    dif(X, Y).

% % Specialized, inlining of (=)/3
ugoal_expansion(if_(Cond_1, Then0_0, Else0_0), G) :-
    subsumes_term((_=_), Cond_1), !,
    (X=Y) = Cond_1,
    body_expansion(Then0_0, Then_0),
    body_expansion(Else0_0, Else_0),
    G = (   X == Y -> Then_0
        ;   X \= Y -> Else_0
        ;   X = Y,    Then_0
        ;   dif(X,Y), Else_0
        ).
% Many other specialized.
% General
ugoal_expansion(if_(Cond_1, Then0_0, Else0_0), G) :-
    body_expansion(Then0_0, Then_0),
    body_expansion(Else0_0, Else_0),
    G = (   call(Cond_1, T),
            (   T == true
            ->  Then_0
            ;   T == false
            ->  Else_0
            ;   must_be(boolean, T)
            )
        ).

% Reference.
if_(Cond_1, Then_0, Else_0) :-
    (   call(Cond_1, T),
        (   T == true
        ->  call(Then_0)
        ;   T == false
        ->  call(Else_0)
        ;   must_be(boolean, T)
        )
    ).

This implementation doesn't block goal expansion for goal which is a valid body for execution and doesn't have cut. It seems that only one implementation can be reference if dynamic goals don't expand.

@UWN
Copy link

UWN commented Dec 12, 2024

has_cut((G0->G)) :-
    has_cut(G0),
    has_cut(G).

No need to check G0 because the ! remains local here. And as you said, sans modules.

@notoria
Copy link
Contributor

notoria commented Dec 16, 2024

Currently with Scryer:

?- [user].
:- use_module(library(format), [portray_clause/2]).

goal_expansion(G, _) :-
    portray_clause(user_error, goal-G),
    false.

% goal_expansion(((G0,G1),G), (G0,G1,G)).

head :-
    goal0,
    Goal1,
    goal.
% Warning: singleton variables Goal1 at line 9 of user
goal-(goal0,A,goal).
goal-goal0.
goal-(A,goal).
goal-goal.

?-

Since term-to-body conversion isn't done before goal expansion, there are some consequences.

Some goals have to be made invisible (no goal-A. between goal-(A,goal). and goal-goal.) to avoid instantiation.
Some goal expansion can make the system loops (e.g. goal_expansion(((G0,G1),G), (G0,G1,G))).

In general, the same goes for term expansion:
The term that is a variable would be invisible (to avoid instantiation but it would eventually error as an invalid clause).
Some term expansions can throw errors but are silenced (this error is no different from the error of being an invalid clause (current situation with DCG expansion)).

Does term-to-body conversion happen before goal expansion?

it seems most sensible to perform such an expansion prior to the final term-to-body conversion.

Before term_expansion/2 and goal_expansion/2, some processing needs to be done otherwise the expansion system is too fragile, uninformative when thing goes wrong (silent error).

@jjtolton
Copy link

Currently with Scryer:

?- [user].
:- use_module(library(format), [portray_clause/2]).

goal_expansion(G, _) :-
    portray_clause(user_error, goal-G),
    false.

% goal_expansion(((G0,G1),G), (G0,G1,G)).

head :-
    goal0,
    Goal1,
    goal.
% Warning: singleton variables Goal1 at line 9 of user
goal-(goal0,A,goal).
goal-goal0.
goal-(A,goal).
goal-goal.

?-

Since term-to-body conversion isn't done before goal expansion, there are some consequences.

Some goals have to be made invisible (no goal-A. between goal-(A,goal). and goal-goal.) to avoid instantiation. Some goal expansion can make the system loops (e.g. goal_expansion(((G0,G1),G), (G0,G1,G))).

In general, the same goes for term expansion: The term that is a variable would be invisible (to avoid instantiation but it would eventually error as an invalid clause). Some term expansions can throw errors but are silenced (this error is no different from the error of being an invalid clause (current situation with DCG expansion)).

Does term-to-body conversion happen before goal expansion?

it seems most sensible to perform such an expansion prior to the final term-to-body conversion.

Before term_expansion/2 and goal_expansion/2, some processing needs to be done otherwise the expansion system is too fragile, uninformative when thing goes wrong (silent error).

That's interesting. How is it normally done? What alternatives are there?

@notoria
Copy link
Contributor

notoria commented Dec 16, 2024

How is it normally done?

No idea. I'm seeing the flaws of the current implementation (by doing small tests).

What alternatives are there?

This is what this issue is discussing about. I don't know other alternatives (actually SICStus is one but I'm not testing it, there is a documentation though) but I'm proposing that the alternative should fix the raised issues.

In the case of goal expansion, the alternative I'm proposing is one that doesn't expand dynamic goal, only works with valid bodies for execution, doesn't expand at run-time. But if goal expansion is done at run-time then I have to rethink another proposal.

In the case of term expansion, the alternative I'm proposing is one that doesn't silence errors.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants