Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom Factors in Python #767

Merged
merged 15 commits into from
Jun 7, 2021
Merged

Custom Factors in Python #767

merged 15 commits into from
Jun 7, 2021

Conversation

ProfFan
Copy link
Collaborator

@ProfFan ProfFan commented May 14, 2021

Per #748 and #764, this is a factor that allow one to create the evaluateError function in Python.

The unit tests document the usage.

It also contains a fix for the proper binding of Point3Pairs. @johnwlambert

Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have to document this. You are the only one* who will do this. It has to also show up in some markdown file, dson't know which one. This is probably worth thinking about re the "readthedocs" change.

Vector CustomFactor::unwhitenedError(const Values& x, boost::optional<std::vector<Matrix>&> H) const {
if(this->active(x)) {
if(H) {
return this->errorFunction(*this, x, H.get_ptr());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document what the mechanism is by which python can access. I expect you'll talk about pybind, types, how it appears in python, what python function should do.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

return this->errorFunction(*this, x, H.get_ptr());
} else {
JacobianVector dummy;
return this->errorFunction(*this, x, &dummy);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document what happens in this case.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
Copy link
Member

@gchenfc gchenfc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool!

See (minor) comments

matlab/CMakeLists.txt Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.cpp Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
gtsam/gtsam.i Outdated Show resolved Hide resolved
@ProfFan ProfFan requested a review from dellaert May 17, 2021 18:59
Copy link
Collaborator

@varunagrawal varunagrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall, some comments and needs formatting. Awesome job @ProfFan!

gtsam/gtsam.i Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
python/gtsam/specializations.h Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Show resolved Hide resolved
Copy link
Collaborator

@varunagrawal varunagrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more stuff I noticed and should have added in my last review.

python/gtsam/tests/test_custom_factor.py Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
@varunagrawal
Copy link
Collaborator

@ProfFan can you please re-request a review after you make the changes? Overall it looks great but I want to do a thorough review again so I can start using this ASAP.

@ProfFan ProfFan requested a review from varunagrawal May 17, 2021 23:19
@ProfFan
Copy link
Collaborator Author

ProfFan commented May 17, 2021

I am thinking about adding an optional constructor param that allows one to specify the name of the custom factor (stored as a string), so people can distinguish different CustomFactor s.

Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MArkdown?

@ProfFan ProfFan requested a review from dellaert May 21, 2021 17:38
Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool ! Starting to get there.

gtsam/gtsam.i Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.cpp Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
gtsam/nonlinear/CustomFactor.h Outdated Show resolved Hide resolved
python/gtsam/preamble.h Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Outdated Show resolved Hide resolved
python/gtsam/tests/test_custom_factor.py Show resolved Hide resolved
@ProfFan ProfFan requested a review from dellaert May 21, 2021 20:12
@ProfFan
Copy link
Collaborator Author

ProfFan commented May 22, 2021

I have addressed all comments @dellaert @varunagrawal

Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After design review, need to expand example to trajopt, and think about autodiff (after that).

@ProfFan ProfFan requested a review from dellaert June 5, 2021 05:18
@ProfFan
Copy link
Collaborator Author

ProfFan commented Jun 5, 2021

Changes:

  • Per @dellaert 's request, I added the example trajectory optimization with CustomFactor
  • Unit test of optimizing an actual graph
  • Fixed linearization of CustomFactor with TBB. Now it will not deadlock trying to acquire the GIL

Copy link
Member

@dellaert dellaert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvement. Not approving yet because we need to talk about sendable

* Should the factor be evaluated in the same thread as the caller
* This is to enable factors that has shared states (like the Python GIL lock)
*/
virtual bool sendable() const {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t understand the name sendable. Since this is a change to the nonlinear factor graph, let’s think about a name that makes sense locally, without the “outside” python context.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I shamelessly borrowed this concept from Rust (here). Basically it indicates whether an object can be shared with another thread. Maybe shareable is a better name?


## Example

The following is a simple `BetweenFactor` implemented in Python.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Either it’s a betweenfactor or it’s not. Pose to compare with still seems to be zero? I thought we had discussed a better, more representative example, but I don’t recall what.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not zero now, if you look at the last unit test.

@@ -0,0 +1,154 @@
import gtsam
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doc block missing...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

from typing import List, Optional
from functools import partial

# Simulate a car for one second
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make a function for this that you call below.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

:param this: gtsam.CustomFactor handle
:param values: gtsam.Values
:param jacobians: Optional list of Jacobians
:param measurement: GPS measurement, to be filled with `partial`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you put measurement first? That makes it like “currying” in functionali languages.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

return error


for k in range(5):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Document what’s happening here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, formatting?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally, maybe assign factor to a variable

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

for k in range(5):
factor_graph.add(gtsam.CustomFactor(gps_model, [unknown[k]], partial(error_gps, measurement=np.array([g[k]]))))

v = gtsam.Values()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove new line at end and Document block

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

python/gtsam/examples/CustomFactorExample.py Show resolved Hide resolved

result = optimizer.optimize()

error = np.array([(result.atVector(unknown[k]) - x[k])[0] for k in range(5)])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What’s happening??? Document

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

:param this: gtsam.CustomFactor handle
:param values: gtsam.Values
:param jacobians: Optional list of Jacobians
:param measurement: Odometry measurement, to be filled with `partial`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Measurement first?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ProfFan ProfFan requested a review from dellaert June 5, 2021 23:33
@ProfFan ProfFan merged commit b1e9167 into develop Jun 7, 2021
@ProfFan ProfFan deleted the feature/custom_factor branch June 7, 2021 18:49
@ProfFan
Copy link
Collaborator Author

ProfFan commented Jun 7, 2021

Merged! 🤯

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants