All: add Stellar operation manageBuyOffer #627

MisterTicot · 2019-10-16T05:43:25Z

I think I covered everything but tests: tests for similar operations manageOffer/createPassiveOffer were missing too.

I did not test the implementation as I don't know about how to proceed. Please let me know if anything is missing or wrong.

tsusanka · 2019-10-16T07:17:36Z

Please run pipenv run make gen to generate protobuf messages.

Yes, you are right that some tests for Stellar operations are missing (see #2). These device tests however are a great start to test your implementation. Have a look on the first test for example and try to write a test for manageBuyOffer, it should be very similar. There is a comment on the top of the file explaining how to get a fixture from Stellar's labrotory.

MisterTicot · 2019-10-16T10:40:35Z

@tsusanka Can you take it from here? I feel like I'm wasting huge amount of time trying to figure out something that you could solve in a dozen of minutes. Meanwhile, nobody is busy writing tests & updates for my own softwares.

tsusanka · 2019-10-16T13:56:02Z

What is the difference between StellarManageBuyOfferOp and the StellarManageOfferOp we already had? Is it completely unrelated op? Or do we use StellarManageOfferOp for "sell"?

matejcik · 2019-10-17T10:30:29Z

What is the difference between StellarManageBuyOfferOp and the StellarManageOfferOp we already had? Is it completely unrelated op? Or do we use StellarManageOfferOp for "sell"?

Correct, "manage offer" is old name for "manage sell offer". We might want to rename "manage offer" to avoid confusion, however, it would break existing Connect users.
OTOH there probably aren't any, so ¯\_(ツ)_/¯.

Another thing i notice is that the messages are identical in terms of fields. The same code could handle them, and we could even fold them into a single message with a BUY/SELL enum flag. Leaving aside the fact that this is what Stellar devs should have done in the first place, this is where the atypical way Stellar is implemented is biting us in the behind.

I mean
we can make three distinct protobuf messages with the same fields (CreatePassiveOffer also has identical fields, except offer id is missing - but that's optional anyway so again ¯\_(ツ)_/¯)
but we should reuse as much existing ManageOffer implementation as possible.
and that is like 98 % of it.

common/protob/messages-stellar.proto

MisterTicot · 2019-10-18T03:26:11Z

Correct, "manage offer" is old name for "manage sell offer". We might want to rename "manage offer" to avoid confusion, however, it would break existing Connect users.
OTOH there probably aren't any, so ¯_(ツ)_/¯.

stellaport.io is one of the major DEX front-end and has integrated Trezor since nearly one year. Naturally, they use the manage(Sell)Offer operation - so I'd assume there are people using it.

MisterTicot · 2019-10-24T15:40:00Z

Note: the latest HEAD passed the checks locally, but when I build T firmware & try to sign createPassiveOffer/manageSellOffer/manageBuyOffer ops I get Firmware error. I've tried to rebase on trezor-firmware master but got the same error.

MisterTicot · 2019-10-25T06:29:43Z

Pull request rebased

Example of transactions leading to firmware error: manageSellOffer.

MisterTicot · 2019-11-01T08:07:55Z

Anybody here???

MisterTicot · 2019-11-25T11:18:18Z

I spent time to play your game: learn how to use your development environment & write some missing code in Trezor support for Stellar. I had to figure out by myself how to make it fits your system.

Yet that PR is being ignored since a month. This is totally unrespectful. 😠

matejcik · 2019-11-25T12:13:06Z

hello, you still haven't addressed this comment #627 (comment):

Another thing i notice is that the messages are identical in terms of fields. The same code could handle them
(...)
but we should reuse as much existing ManageOffer implementation as possible.
and that is like 98 % of it.

I am sorry that i don't have more time to spend on 3rd party contributions in order to provide more detailed guidance. However, this is a low-priority issue for us.

MisterTicot · 2019-11-25T12:40:16Z

That issue was already present in your code. As you noted, there was already a duplicate for passiveSellOffer - I simply followed your ways.

I think there's a difference between me writing an update for your firmware & me refactoring your code base to fix a weakness in it. I'm not proficient at C & I never proposed to go that far.

I agree that this would be better but I fear that if you really want it then you'll have to write it by yourself.

matejcik · 2019-11-25T12:51:21Z

common/protob/messages-stellar.proto

+    optional string source_account = 1;             // (optional) source account address
+    optional StellarAssetType selling_asset = 2;
+    optional StellarAssetType buying_asset = 3;
+    optional sint64 buy_amount = 4;


please rename to amount

You already asked for it the last time then closed it after I replied:

The reason I used buyAmount is because it's what Stellar use (I guess it should be buy_amount by the way?). I know it's not consistent.

Do you want to switch to amount anyway?

I think it's better to stay consistent with the underlying implementation regardless of our opinion about it. (I asked them to change it into amount at a lower level when that operation got merged but they didn't)

Right, sorry about that. Yes, please switch to amount anyway.

core/src/apps/stellar/operations/layout.py

matejcik · 2019-11-25T12:52:14Z

core/src/apps/stellar/operations/serialize.py

@@ -68,6 +69,15 @@ def write_manage_offer_op(w, msg: StellarManageOfferOp):
    writers.write_uint64(w, msg.offer_id)


+def write_manage_buy_offer_op(w, msg: StellarManageBuyOfferOp):


please reuse write_manage_offer_op instead

I'm waiting for your response about amount so I can decide how to rewrite that one.

matejcik · 2019-11-25T12:56:12Z

I pointed out some places where reuse should happen. You're unfortunately right about the legacy code, where we can't use duck typing to save on copy-pasting.

MisterTicot · 2019-12-06T06:35:01Z

@matejcik I'm waiting for your answer.

matejcik · 2019-12-09T11:46:23Z

i'm just now noticing that your code in core serializes selling then buying asset, but the legacy code does the opposite, i.e. buying then selling. which is correct?

i assumed that the code in core is correct, and the serialization for buy-offer and sell-offer is identical.

if it isn't, then please fix what is a bug in write_manage_buy_offer_op, and keep it separate from write_manage_offer_op. (I'm leaving up to you whether to rename buy_amount to amount; I intended the rename to simplify our implementation, but if it turns out that the serialization is different, we can keep the name consistent.)

If it is identical, then please fix the order in legacy code.

The code in python/src/trezorlib/stellar.py might also need the fix? because the assets there are also ordered as selling then buying.

This is the final decision and I'm ready to merge once this is clarified.
I.e., if the serialization is identical between buy and sell, then please change field name to amount (so it is also identical), reuse the serialization function, and fix bug in legacy.

If the serialization is different, please fix the bugs in core/ and python/, and keep the serialization function separate.

matejcik · 2019-12-09T11:46:55Z

also leaving a note for myself to rename "manage offer op" to "manage sell offer op" later

MisterTicot · 2019-12-16T10:44:24Z

i'm just now noticing that your code in core serializes selling then buying asset, but the legacy code does the opposite, i.e. buying then selling. which is correct?

i assumed that the code in core is correct, and the serialization for buy-offer and sell-offer is identical.

Good catch, thanks!

Indeed, both ops are serialized the same way.

MisterTicot · 2019-12-16T10:49:31Z

@matejcik

Just re-read the code... Where did I serialize the wrong way? Couldn't find it.

matejcik · 2019-12-16T10:50:44Z

https://github.com/trezor/trezor-firmware/pull/627/files#diff-771a1c4ded40f4b313fffe1c5aa274d6R554-R558

MisterTicot · 2019-12-16T10:57:43Z

@matejcik Thanks, I'm fixing it. I'm struggling on something: what's the right way to handle typecheck here?

def write_manage_offer_op(w, msg: StellarManageOfferOp):

matejcik · 2019-12-16T10:58:34Z

@matejcik Thanks, I'm fixing it. I'm struggling on something: what's the right way to handle typecheck here?

def write_manage_offer_op(w, msg: StellarManageOfferOp):

Union[StellarManageOfferOp, StellarManageBuyOfferOp]

matejcik · 2019-12-16T11:00:35Z

def write_manage_offer_op(w, msg: StellarManageOfferOp):

that w is untyped here tells me that this code is not typechecked yet, so don't worry about it too much. Properly typing the hashwriters is an ongoing task anyway.

MisterTicot · 2019-12-16T11:27:54Z

@matejcik Is there a command that builds and run all tests at once?

core/src/apps/stellar/consts.py

MisterTicot · 2019-12-20T13:39:45Z

Let me know if tests pass, I'll merge commits together to prepare for merging.

matejcik · 2019-12-20T14:37:54Z

tests are still failing, now both core and legacy produces the same signature IhKLu9XhrrydOlb9U9oBztVO1HMBf1vLi/oc4Okr4gxLDM7oTXzuOccBKfQtxxPv6u4mLtKoFOcjuhHOnXh+AQ==, which is different from what you are asserting.

I seem to be unable to regenerate the transaction in the Stellar TX builder (the signature I get is also completely different), so I'm afraid this is up to you. Let me know if you encounter problems when running the test; the simplest way to run it without the whole test suite is pipenv run pytest tests/device_tests/test_msg_stellar_sign_transaction.py -k buy_offer while an emulator is running

MisterTicot · 2019-12-20T17:16:18Z

I've tried it, but it stays stuck on:

collected 11 items / 10 deselected / 1 selected

tests/device_tests/test_msg_stellar_sign_transaction.py

I fear I already allocated more time than reasonable on that PR & I have no idea what's wrong. There are other matters that need my attention so I'll have to leave it that way for now.

I think your tool suite is great & has a lot of potential, but contributors efficiency would be greatly improved if it where possible run all the important tasks within a few commands.

It was frustrating to have to deal with dozens of commands & to regularly bump into cryptic error messages that required significant amount of time to get solved. I think I spent more time struggling with the dev environment than working on the code itself.

matejcik · 2019-12-27T12:30:17Z

That's unfortunate. Thanks for your work anyway, hopefully someone can pick this up in the future.

We have #769 about running everything with a single command, so that is something that might happen.

MisterTicot · 2019-12-28T21:30:02Z

That would be immensely helpful.
Let me know, and I'll give it another try once my new system is up & running.

overcat · 2021-10-04T11:22:24Z

That's unfortunate. Thanks for your work anyway, hopefully someone can pick this up in the future.

Hello @matejcik, can I try to promote this PR?

matejcik · 2021-10-06T08:40:20Z

@overcat feel free to go ahead

although with the code drift, it might be better to start from scratch.

regarding protobuf design: i'm not sure if it's worth it to introduce a separate ManageBuyOfferOp. ISTM a better choice is add a BUY/SELL enum, which defaults to SELL for backwards compatibility.
The shape of the protobuf doesn't match the shape of Stellar XDR anyway, so this shouldn't matter that much -- but what's your opinion?

Also to be clear, we would currently accept the added ManageBuyOffer, but not other operations that are missing. (reason basically being that this is "pre-approved" because of this pre-existing PR, but we want to wait for the community firmware for the other additions)

overcat · 2021-10-06T11:11:45Z

Hi @matejcik,

The following comments do not consider compatibility issues.

At least for now, the names of the operations in the protobuf file are the same as those in XDR, I don't think we need to break this rule for this special case. If necessary, we can also share code in a lower level of code.
The implementation of various software in the current Stellar ecosystem also follows this point. When other Stellar developers try to contribute code to the firmware, I think keeping the same can reduce their confusion.

The above are my opinions, but I am willing to follow your guidelines.

matejcik · 2021-10-06T13:39:12Z

Very well, you're right that keeping it a separate type is better for understandability of outside contributors. The cost of doing it that way is not significant enough to introduce confusion.

overcat · 2021-10-06T14:28:51Z

Hi @matejcik,

So I will submit a new PR, rename StellarManageOfferOp to StellarManageOfferOp and add a new operation StellarManageBuyOfferOp, sounds great?

Also, StellarPathPaymentOp has been renamed to StellarPathPaymentStrictReceiveOp in XDR and a new operation StellarPathPaymentStrictSendOp has been added, of course, I will not add support for StellarPathPaymentStrictSendOp before I get permission, but would you like me to rename StellarPathPaymentOp to StellarPathPaymentStrictReceiveOp in this PR? This might avoid another broken update in the future, so please let me know if I need to do that, thanks.

P.S.

struct PathPaymentStrictReceiveOp
{
    Asset sendAsset; // asset we pay with
    int64 sendMax;   // the maximum amount of sendAsset to
                     // send (excluding fees).
                     // The operation will fail if can't be met

    MuxedAccount destination; // recipient of the payment
    Asset destAsset;          // what they end up with
    int64 destAmount;         // amount they end up with

    Asset path<5>; // additional hops it must go through to get there
};

struct PathPaymentStrictSendOp
{
    Asset sendAsset;  // asset we pay with
    int64 sendAmount; // amount of sendAsset to send (excluding fees)

    MuxedAccount destination; // recipient of the payment
    Asset destAsset;          // what they end up with
    int64 destMin;            // the minimum amount of dest asset to
                              // be received
                              // The operation will fail if it can't be met

    Asset path<5>; // additional hops it must go through to get there
};

matejcik · 2021-10-06T14:32:28Z

So I will submit a new PR, rename StellarManageOfferOp to StellarManageOfferOp and add a new operation StellarManageBuyOfferOp, sounds great?

i think you mean StellarManageSellOfferOp, and yes 👍

I will not add support for StellarPathPaymentStrictSendOp before I get permission, but would you like me to rename StellarPathPaymentOp to StellarPathPaymentStrictReceiveOp in this PR

Yes please. Also feel free to implement StellarPathPaymentStrictSendOp as well, it really seems symmetrical to the ManageBuyOffer/ManageSellOffer thing so I believe we can subsume it in here.

matejcik · 2021-11-04T10:38:26Z

@MisterTicot thanks again for your work.

this PR is now superseded by #1838

MisterTicot mentioned this pull request Oct 16, 2019

[Feature] Stellar signing / missing operations #579

Closed

matejcik reviewed Oct 17, 2019

View reviewed changes

common/protob/messages-stellar.proto Outdated Show resolved Hide resolved

MisterTicot force-pushed the stellar-manage-buy-offer branch from 6f14b8e to 59b07c8 Compare October 25, 2019 05:45

prusnak added this to the backlog milestone Nov 10, 2019

matejcik requested changes Nov 25, 2019

View reviewed changes

MisterTicot added 3 commits December 16, 2019 19:23

stellar: add operation manageBuyOffer

56ca630

tests: add Stellar operation manageBuyOffer

47f7035

all: update messages

8bca8ce

matejcik reviewed Dec 19, 2019

View reviewed changes

core/src/apps/stellar/consts.py Outdated Show resolved Hide resolved

MisterTicot added 2 commits December 19, 2019 16:08

stellar: fix typo

a8a875a

stellar: Fix price in manageBuyOffer test

8ce11e8

prusnak added the blocked Blocked by external force. Third party inputs required. label Oct 2, 2020

prusnak modified the milestones: backlog, freezer Oct 2, 2020

MisterTicot requested review from prusnak and tsusanka as code owners March 27, 2021 08:23

prusnak removed their request for review May 18, 2021 19:56

tsusanka removed this from the freezer milestone Oct 6, 2021

overcat mentioned this pull request Oct 7, 2021

Stellar: add support for StellarManageBuyOfferOp and StellarPathPaymentStrictSendOp. #1838

Merged

matejcik closed this Nov 4, 2021

		@@ -68,6 +69,15 @@ def write_manage_offer_op(w, msg: StellarManageOfferOp):
		writers.write_uint64(w, msg.offer_id)


		def write_manage_buy_offer_op(w, msg: StellarManageBuyOfferOp):

All: add Stellar operation manageBuyOffer #627

All: add Stellar operation manageBuyOffer #627

Conversation

MisterTicot commented Oct 16, 2019

tsusanka commented Oct 16, 2019

MisterTicot commented Oct 16, 2019

tsusanka commented Oct 16, 2019 • edited Loading

matejcik commented Oct 17, 2019

MisterTicot commented Oct 18, 2019

MisterTicot commented Oct 24, 2019

MisterTicot commented Oct 25, 2019

MisterTicot commented Nov 1, 2019

MisterTicot commented Nov 25, 2019

matejcik commented Nov 25, 2019

MisterTicot commented Nov 25, 2019

matejcik Nov 25, 2019

Choose a reason for hiding this comment

MisterTicot Dec 3, 2019 • edited Loading

Choose a reason for hiding this comment

matejcik Dec 9, 2019

Choose a reason for hiding this comment

matejcik Nov 25, 2019

Choose a reason for hiding this comment

MisterTicot Dec 3, 2019

Choose a reason for hiding this comment

matejcik commented Nov 25, 2019

MisterTicot commented Dec 6, 2019

matejcik commented Dec 9, 2019 • edited Loading

matejcik commented Dec 9, 2019

MisterTicot commented Dec 16, 2019

MisterTicot commented Dec 16, 2019

matejcik commented Dec 16, 2019

MisterTicot commented Dec 16, 2019

matejcik commented Dec 16, 2019 • edited Loading

matejcik commented Dec 16, 2019

MisterTicot commented Dec 16, 2019

MisterTicot commented Dec 20, 2019

matejcik commented Dec 20, 2019 • edited Loading

MisterTicot commented Dec 20, 2019

matejcik commented Dec 27, 2019

MisterTicot commented Dec 28, 2019

overcat commented Oct 4, 2021

matejcik commented Oct 6, 2021

overcat commented Oct 6, 2021

matejcik commented Oct 6, 2021

overcat commented Oct 6, 2021

matejcik commented Oct 6, 2021

matejcik commented Nov 4, 2021

tsusanka commented Oct 16, 2019 •

edited

Loading

MisterTicot Dec 3, 2019 •

edited

Loading

matejcik commented Dec 9, 2019 •

edited

Loading

matejcik commented Dec 16, 2019 •

edited

Loading

matejcik commented Dec 20, 2019 •

edited

Loading