Meta-ticket: Callable symbolic expressions

[rburing]

The syntax for callable symbolic expressions works in the simplest cases, but beyond that it breaks down and causes much confusion, as can be often seen on Ask SageMath and (in my experience) when trying to teach SageMath to people. Here is a non-exhaustive list of 8 examples:

1. Numbers confusion:

sage: f(x) = x^2
sage: f(2).factor()
4

2. Polynomial confusion:

sage: R.<z,w> = PolynomialRing(QQ)
sage: f(x) = x^2
sage: f(z+w).coefficient({z : 1})
...
TypeError: no canonical coercion from <type 'dict'> to Symbolic Ring

https://ask.sagemath.org/question/47064/how-to-turn-the-function-into-expression/

3. Matrix confusion:

sage: B(x) = matrix([[x, 0], [0, 0]])
sage: B(12)
[x 0]
[0 0]

https://ask.sagemath.org/question/10457/arithmetic-with-matrices-of-formal-functions/

4. List confusion:

sage: f(x) = [x,x]
sage: f(2).parent()
Vector space of dimension 2 over Symbolic Ring

https://ask.sagemath.org/question/10449/how-to-return-a-list-from-callable-symbolic-expression/

5. Derivative confusion (and argument confusion):

sage: f(x) = x.derivative()
sage: f(x^2)
1
sage: f(y) = y.derivative(x)
sage: f(x^2)
0

https://ask.sagemath.org/question/9842/the-difference-between-fx3-and-f3-of-callable-symbolic-expression-f/

6. Matrix argument confusion:

sage: f(x) = x^2
sage: f(2*identity_matrix(2))
...
TypeError: no canonical coercion from Full MatrixSpace of 2 by 2 dense matrices over Integer Ring to Callable function ring with argument x

https://ask.sagemath.org/question/38524/defining-functions-acting-on-matrix-elements/

7. Addition confusion - CallableSymbolicExpressionRing over subrings of SR #32008:

sage: f(x) = x^2
sage: g(x) = x^2
sage: var('t')
sage: h(t) = t^2
sage: f+g
x |--> 2*x^2
sage: f+h
(t, x) |--> t^2 + x^2

https://ask.sagemath.org/question/10782/symbolic-functions-without-named-variables/

8. Non-symbolic function confusion:

sage: tau(n) = len(divisors(n))
...
TypeError: 'sage.symbolic.expression.Expression' object is not iterable

https://ask.sagemath.org/question/47672/is-this-a-bug-or-intended-behavior/

9. Derivative confusion.

sage: f(t) = (t, t^2, t^3)

sage: f
t |--> (t, t^2, t^3)
sage: f(t)
(t, t^2, t^3)

sage: f.parent()
Vector space of dimension 3 over Callable function ring with argument t
sage: f(t).parent()
Vector space of dimension 3 over Symbolic Ring

sage: f1 = f(t).derivative(t)
sage: f1
(1, 2*t, 3*t^2)
sage: f1.parent()
Vector space of dimension 3 over Symbolic Ring

sage: g(t) = f(t).derivative(t)
Traceback (most recent call last)
...
TypeError: unable to convert (1, 2*t, 3*t^2) to a symbolic expression

sage: h = f.derivative()
sage: h
[    t |--> 1]
[  t |--> 2*t]
[t |--> 3*t^2]
sage: h.parent()
Full MatrixSpace of 3 by 1 dense matrices
over Callable function ring with argument t

10. Newline confusion. See Preparser could accomodate multiline input and continuation lines #11621, Multiline doctests fail when using angle notation (preparser) #19088, Implicit line continuation in callable symbolic expression #30953.

sage: f(t) = (t,
....:         t^2,
....:         t^3)
  File "<ipython-input-46-d381d3086d21>", line 2
    t**Integer(2),
    ^
SyntaxError: invalid syntax

Additional issues and tickets related to callable symbolic expressions:

• fast_callable should respect the variable order in callable symbolic expressions (treating them like lambda functions rather than like expressions) #7512 fast_callable should respect the variable order in callable symbolic expressions (treating them like lambda functions rather than like expressions)
• make f(x,y,z)=vector make a vector-valued function #11507 make f(x,y,z)=vector make a vector-valued function
• Create callable matrices in function notation #12075 Create callable matrices in function notation
• Partial evaluation bug for callable vector functions #12302 Partial evaluation bug for callable vector functions
• numerical integral needs an operand for callable symbolic functions #15219 numerical integral needs an operand for callable symbolic functions
• CallableSymbolicExpressionRing over subrings of SR #32008 CallableSymbolicExpressionRing over subrings of SR
• CallableSymbolicExpressionRing: Fix broken element methods #32146 CallableSymbolicExpressionRing: Fix broken element methods
• Improve repr for callable symbolic expressions using unicode #30374 Improve repr for callable symbolic expressions using unicode
• Allow to take the sign of a callable symbolic expression #31282 Allow to take the sign of a callable symbolic expression
• Implicit line continuation in callable symbolic expression #30953 Implicit line continuation in callable symbolic expression

See also:

• Deprecate methods arguments (alias: args), number_of_arguments for non-callable symbolic expressions #32227 Deprecate methods arguments (alias: args), number_of_arguments, _fast_callable_ for non-callable symbolic expressions

CC: @slel @orlitzky @egourgoulhon

Component: symbolics

Keywords: CallableSymbolicExpression, function, callable

Issue created by migration from https://trac.sagemath.org/ticket/28434

[jdemeyer]

comment:2

Let's look at these one by one:

1. The most obvious bug IMHO. Factoring in SR, especially in obvious cases like this, should work. This has nothing to do with callable expressions though.
2. Not really a bug, you shouldn't mix symbolic expressions and polynomials.
3. Bug. Substituting in matrices should work.
4. Not a bug, you cannot make a list symbolic.
5. Not a bug, the RHS x.derivative() equals 1 so f(x) = x.derivative() is just a complicated way to write f(x) = 1.
6. Bug, there is no reason why this shouldn't work.
7. I don't see the problem here, what would you expect?
8. Not a bug, similar to 5: the RHS divisors(n) is meaningless for a symbolic n.

Some of these (in particular 4, 5, 8) are really user mistakes where an ordinary Python function is meant instead of a symbolic function. I agree that this may be confusing, but there really is an important difference between

f(x) = x.derivative()

and

def f(x):
    return x.derivative()

I recommend you to create separate tickets for the bugs that you want to see fixed. I cannot at all promise that they will be fixed. But it will help discussion if we don't have a single issue which is about multiple unrelated things.

[rburing]

comment:3

I have to disagree that these issues are unrelated. The main source of confusion (in my opinion) is that this syntax is reserved for something very particular (callable symbolic expressions), while it looks like something much more general, like a lambda-style definition of a Python function.

1. The most obvious bug IMHO. Factoring in SR, especially in obvious cases like this, should work. This has nothing to do with callable expressions though.

What I meant to show was: the (naive) expectation is that when you put in an integer, you get back an integer. This is false, because what you get is a symbolic expression. So a better example (with a method that isn't available on symbolic expressions) would be f(2).binary().

2. Not really a bug, you shouldn't mix symbolic expressions and polynomials.

They can often be mixed. A "fix" here is to write R(f(z+w)).coefficient({z : 1}). My point is: how does the user know that f(x) = x^2 defines a callable symbolic expression which should get this special treatment (compared to a Python function)? Or, how should a user know not to use these in this context?

4. Not a bug, you cannot make a list symbolic.

How does the user know that this is an issue? It's not an issue with Python functions...

5. Not a bug, the RHS x.derivative() equals 1 so f(x) = x.derivative() is just a complicated way to write f(x) = 1.
   I agree that this may be confusing, but there really is an important difference between

f(x) = x.derivative()

and

def f(x):
    return x.derivative()

We know this and we are used to it, but in my opinion it really makes no sense. I can kind of understand the desire to create a type of function object (which is what happens in the first case), e.g. for prettyprinting, but the fact that these two f's don't give the same output when evaluated is honestly ridiculous. How should a user know?

7. I don't see the problem here, what would you expect?

Some would expect two scalar-valued functions of a scalar variable to always add up to a scalar-valued function of a scalar variable. But I will concede this is the least confusing one.

8. Not a bug, similar to 5: the RHS divisors(n) is meaningless for a symbolic n.

How does the user know that they are defining a symbolic callable expression, and that they shouldn't?

All the questions were rhetorical. In my experience (and in the posted threads), the user doesn't know, because it isn't clear. Sage would be less confusing without this syntax (in its current form).

[defeo]

comment:4

All the questions were rhetorical. In my experience (and in the posted threads), the user doesn't know, because it isn't clear. Sage would be less confusing without this syntax (in its current form).

So, is this ticket suggesting to remove the syntax? If so, I'm not convinced a ticket is the right place to have such a discussion.

Do you realize how much user code it would break? Maybe you should lay out your plan more clearly on sage-devel.

[nbruin]

comment:5

Thanks for putting the list together. It's nice to have a record which issues (often?) arise. A lot of confusion happens because of the mathematics underneath. Confusion is a common state for students who are learning mathematics and I think it is to be expected that they will also encounter it while working with a computer algebra system of significant scope and complexity. It's nice to reduce confusion where possible, but I think it's unrealistic to expect we can do away with it.

Some of your items are indicated above as bugs and can probably be solved. Other items might be better to put in an FAQ document (or the documentation) for people to point to if questions arise. I don't know if these examples are best kept track of on a ticket. The ticket itself isn't really resolvable, but there are other administrative tracker tickets around.

Answer to "how does a user know"?

sage: f(x)=x^2
sage: type(f)
<type 'sage.symbolic.expression.Expression'>
sage: parent(f)
Callable function ring with argument x

Once a user run into issues like this, it's clear that they have to progress beyond the level of "beginner" and start a more "advanced" approach. When they have questions concerning this, they have a motivation to read an "advanced" tutorial that goes more into the nuts and bolts of things. After that tutorial they should be at a level where they can make sense of the reference guide. Perhaps the "advanced" tutorial already exists. Perhaps it needs to be written?

Note that callable symbolic expressions serve a very definite purpose and are therefore not going away. The preparse trick of defining them f(x)=... is magic but incredibly useful for giving compact illustrations in educational settings, and that syntax doesn't seem to be a source for confusion anyway (other than that people don't get much of a cue that this is not a python built-in feature -- but letting people look at preparse("...") quickly illustrates what's going on.
Callable symbolic expressions are necessary to use positional call syntax for symbolically defined functions/expressions, which is very common in math text books.

A general comment: Building a computer algebra system on top of python is necessarily a compromise. This is true for other python-based solutions too, for instance pandas or matplotlib. In most of these cases, a Domain Specific Language would be much more elegant and unified. Although, beware what you ask for: in for instance Maple, they DO have a very clear symbolic expression model that is fundamental to everything. And they found it doesn't address all their needs and had to bolt on other solutions. It makes for a decidedly unpleasant system to program for. They ended up with their compromises in other places.

1. symbolic factor COULD try and use integer factorization on objects that look like an integer or a rational number. That would change the complexity of the routine significantly (but perhaps for SR we don't care), and of course it might make people wonder why 2*I doesn't get factored.

2. doesn't seem to have to do with callable expressions at all. The fact that "derivative" is allowed to try to do something without an argument is necessary because of univariate calculus

3. Bug maybe, but possibly a consequence of design that will be hard to fix: "symbolic matrices" aren't members of the symbolic ring. They are matrices over SR. So filling them into a function that is from and to SR is problematic.

[kcrisman]

comment:6

Note that callable symbolic expressions serve a very definite purpose and are therefore not going away. The preparse trick of defining them f(x)=... is magic but incredibly useful for giving compact illustrations in educational settings, and that syntax doesn't seem to be a source for confusion anyway (other than that people don't get much of a cue that this is not a python built-in feature -- but letting people look at preparse("...") quickly illustrates what's going on.

Thanks for a very well-organized resume of the issues at stake, Nils.

[egourgoulhon]

comment:7

Replying to @kcrisman:

Thanks for a very well-organized resume of the issues at stake, Nils.

+1

[rburing]

comment:8

Thanks, Nils.

I had a look at the implementation. A callable symbolic expression like f(x) = x^2 is mostly just an Expression whose parent is a CallableSymbolicExpressionRing; it keeps track of the list of (symbolic variable) arguments and implements _call_element (for substitution) and _repr_element (for prettyprinting). The notation f(x,y) = (x,y) defines a vector over a CallableSymbolicExpressionRing, and the special element class Vector_callable_symbolic_dense is used to fix the notation with a _repr_ method.

This is quite simple and powerful, and I now agree that the notation should be kept. It seems that the best resolution is to fix the bugs, improve the documentation, improve error handling, and extend the functionality to include a matrix-valued variant. See a detailed proposal below.

I found that the Guided Tour already has a page Some Common Issues with Functions which is a great start for clarifying some of the issues. This should be the page to point people to when they run into any of this.

Let me go over the list again:

1. If f(x) = x^2 then f(2) is not an Integer

The Common Issues page point 2 should explain more precisely what a callable symbolic expression is. In particular, at definition-time, the arguments are symbolic variables (and treated as such on the right-hand side); calling means substitution, and the result (in the scalar-valued case) is a symbolic expression. Also, the "vector-valued" (and possibly "matrix-valued", see 3 below) variant deserves a mention.

2. Polynomial confusion

Covered by the previous point. I would like to have the "solution" to this (explicit conversion) as an example somewhere, but since the Common Issues page is part of the Guided Tour it would be too early there.

3. Matrix confusion

Currently a bug. I believe matrices are not much different from vectors, and a matrix-valued variant could be implemented. It should return a matrix over a CallableSymbolicExpressionRing, and a special element class with _repr_ and _latex_ (like Vector_callable_symbolic_dense) should fix the notation.

4. List confusion

In my opinion the syntax with square brackets for vector-valued functions is an abuse of notation (which has caused confusion with lists); it should be f(x) = (x,x) or f(x) = vector([x,x]) (the latter currently doesn't work, by the way). This is handled in symbolic_expression.

5. Derivative confusion (and argument confusion)

Handled by point 1. In particular, the point is that the arguments in the right-hand side of the definition of a callable symbolic expression are symbolic variables, not arbitrary objects, and in particular they cannot be functions.

6. Matrix argument confusion

From the implementation we know that _call_element is a substitution of symbolic variables. So all arguments must be (coerced to) symbolic expressions. Since this is not possible for a matrix, we can give a much better error here.

7. Addition confusion

I can live with this. Maybe it should be mentioned in the documentation.

8. Non-symbolic function confusion

Can we improve the preparser here? Currently, it translates tau(n) = len(divisors(n)) into

__tmp__=var("n");
tau = symbolic_expression(len(divisors(n))).function(n)

But how about this instead?

__tmp__=var("n");
try:
    tau = symbolic_expression(len(divisors(n))).function(n)
except Exception, e:
    raise TypeError, "Failed to define callable symbolic expression. Is the right-hand side a valid (tuple/vector/matrix of) symbolic expression(s) in symbolic variable(s) " + str(__tmp__) + "? Original exception: " + str(e)

The best part about this is that it informs the user that it is attempting to define a callable symbolic expression. If this is not what they meant, the user will probably do a web search for "Sage function" and will land on the (highly ranked) Common Issues page, which starts with the definition of ordinary Python functions.

[embray]

comment:9

Ticket retargeted after milestone closed

[mkoeppe]

comment:10

Batch modifying tickets that will likely not be ready for 9.1, based on a review of the ticket title, branch/review status, and last modification date.

[mkoeppe]

comment:12

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[slel]

comment:13

Suggestion: make this a meta-ticket and track each item
in its own ticket.

9. Derivative confusion.

sage: f(t) = (t, t^2, t^3)

sage: f
t |--> (t, t^2, t^3)
sage: f(t)
(t, t^2, t^3)

sage: f.parent()
Vector space of dimension 3 over Callable function ring with argument t
sage: f(t).parent()
Vector space of dimension 3 over Symbolic Ring

sage: f1 = f(t).derivative(t)
sage: f1
(1, 2*t, 3*t^2)
sage: f1.parent()
Vector space of dimension 3 over Symbolic Ring

sage: g(t) = f(t).derivative(t)
Traceback (most recent call last)
...
TypeError: unable to convert (1, 2*t, 3*t^2) to a symbolic expression

sage: h = f.derivative()
sage: h
[    t |--> 1]
[  t |--> 2*t]
[t |--> 3*t^2]
sage: h.parent()
Full MatrixSpace of 3 by 1 dense matrices
over Callable function ring with argument t

10. Newline confusion. See Preparser could accomodate multiline input and continuation lines #11621, Multiline doctests fail when using angle notation (preparser) #19088, Implicit line continuation in callable symbolic expression #30953.

sage: f(t) = (t,
....:         t^2,
....:         t^3)
  File "<ipython-input-46-d381d3086d21>", line 2
    t**Integer(2),
    ^
SyntaxError: invalid syntax

[mkoeppe]

comment:14

For item 7, see #32008.