Add options to ignore retweets and boosts (reblogs) #14

bajtos · 2020-04-15T09:06:54Z

Introduce two new config options:

sync_retweets = false ~~ignore_retweets~~ to exclude retweets when sychronizing from Twitter to Mastodon
sync_reblogs = false ~~ignore_boosts~~ to exclude boosts (reblogs) when synchronizing from Mastodon to Twitter

Resolve #11

Caveat: mammut does not support exclude_reblogs option introduced to Mastodon API in Dec 2018 via mastodon/mastodon#9640, we have to filter out reblogs locally at client side. Mammut is no longer maintained and recomment users to migrate elefren. I opened a feature request asking elefren to add support for exclude_reblogs, see https://github.com/pwoolcoc/elefren/issues/134.

~~Discussion point: considering that Mastodon uses the term "reblogs" instead of "boosts", should we perhaps do the same and call the new parameter ignore_reblogs?~~

src/main.rs

klausi

Thanks a lot, I think the changes make sense!

At first I thought we should filter in sync.rs, but of course if we can prevent fetching retweets/reblogs that is even better. The only problem is that we have no test coverage now.

Also we could run into problems with this: What happens if the Mastodon account has 50 reblogs and no original toot? Then the list of Mastodon is empty and there is no latest toot to compare? Then we would try to double post to Mastodon?

Do we run into feedback loops with this when people change their configuration to ignore retweets/reblogs suddenly? At least the double posting prevention should mitigate this somewhat and maybe we don't care?

README.md

src/config.rs

src/main.rs

bajtos · 2020-04-19T08:24:22Z

At first I thought we should filter in sync.rs, but of course if we can prevent fetching retweets/reblogs that is even better. The only problem is that we have no test coverage now.

Yeah, I am worried about the lack of test coverage too.

Initially, I thought there are not tests for the current happy path (syncing retweets/reblogs), because as a new Rust developer coming from Node.js land, I didn't realize I should look for the tests inside the source files. (In Node.js, tests have their own files.) Now I see there is pretty decent test suite in src/sync.rs.

Also we could run into problems with this: What happens if the Mastodon account has 50 reblogs and no original toot? Then the list of Mastodon is empty and there is no latest toot to compare? Then we would try to double post to Mastodon?

Interesting. I didn't look into the details of the sync algorithm, I just tried the simplest thing that seemed to work for me :) Thank you for pointing out edge cases to consider!

Do we run into feedback loops with this when people change their configuration to ignore retweets/reblogs suddenly? At least the double posting prevention should mitigate this somewhat and maybe we don't care?

I am afraid I don't understand what do you mean by a feedback loop and double posting prevention. Would this problem go away if we always fetch retweets/reblogs and filter them out inside the sync algorithm, because that way the sync algorithm always receives the same input data, regardless of whether retweets/reblogs should or should not be synced?

Based on your comments, it seems to me that the easiest (and possibly the most robust) way forward is to rework my pull request to deal with reblogs/retweets inside the sync algorithm. Could you @klausi please confirm?

This is a fun side project for me and I have very limited time budget to work on it, I think it will take me a week or two until I have a new version for review.

klausi · 2020-04-20T14:38:04Z

Confirmed, I agree that we should do this in sync.rs where we can also tests this. Not fully ideal as I said, but a good enough start that should be safe.

No time pressure, thanks a lot for your contribution ❤️

Add two new config options allowing users to disable syncing of Twitter retweets and Mastodon reblogs (boosts). Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

bajtos · 2020-04-26T13:29:15Z

@klausi thank you for a very constructive feedback 👍

I reworked the pull request to implement sync_retweets and sync_reblogs options that are handled inside sync.rs. When I did a dry run, I discovered that there are also quote tweets that are different from retweets. I added a test to verify that they are not affected by sync_retweets option, my idea is that synchronization of quote tweets should be handled by a new option, e.g. sync_quote_tweets. I slightly prefer to leave sync_quote_tweets out of scope of this PR, if you don't mind.

Personally, I would like the following behavior for syncing quote tweets:

The toot contains only the text I wrote in my tweet.
The toot links to the quoted status (tweet)

Consider the following quote tweet: https://twitter.com/bajtos/status/1252469674681544704

At the moment, cargo run -- -n prints the following item:

Posting to Mastodon: LoopBack 4 depends on AJV to perform validations too! Please consider supporting AJV maintenance and development by a donation 🙏 Every little helps!

QT fastifyjs: If you can validate requests in Fastify is thanks to this fantastic project, who is now asking for support.
If you would like to make a donation, please consider it :)
https://github.com/epoberezkin/ajv#please-sponsor-ajv
#opensource #sustainability

What I would like to see instead (conceptually, I am not sure how attachments/links work in Mastodon):

Posting to Mastodon: LoopBack 4 depends on AJV to perform validations too! Please consider supporting AJV maintenance and development by a donation 🙏 Every little helps!
https://twitter.com/fastifyjs/status/1252343103958118402

I briefly considered implementing this behavior by default when sync_retweets is disabled, but then I though it would be too hacky/magical for users, plus the implementation needs to be clever to recognize both "short" (new) form and "long" (current) form of Mastodon toots as equivalent to the original quote tweet. I think this is best to be discussed and implemented in a new pull request, I am thinking about introducing another twitter config flag. Please let me know if you would accept such feature and if yes, what's the best way to go forward? Should I open a new issue to discuss details, or should I go ahead and open a PR?

Anyhow, I wrote all of the above to give you more context about what I am looking for, to better explain why I don't care that much about sync_quote_tweets option. Let's focus on finishing this pull request first 🏁

Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

klausi · 2020-05-02T08:51:44Z

I just recently implemented the quote tweet behavior because I found it really annoying to have twitter links on Mastodon. There is enough post character space on Mastodon to also include the quote tweet then the user does not have to click to get to the content Twitter.

Of course I'm open to move that behavior under a configurable flag, we can discuss in a new issue :)

klausi

Great work, test cases look sufficient! I will merge this now and follow-up with small default value changes in the README.

Thanks for your contribution and feel free to come back any time with further improvements!

klausi · 2020-05-02T08:52:30Z

README.md

@@ -37,6 +37,8 @@ Enable automatic status/favourite deletion with config options. Example:
 delete_older_statuses = true
 # Delete Mastodon favourites that are older than 90 days
 delete_older_favs = true
+# Do not sync reblogs (boosts)
+sync_reblogs = false


Default value should be true also in the docs to have the current behavior.

klausi · 2020-05-02T08:52:41Z

README.md

@@ -56,6 +58,8 @@ user_name = "example"
 delete_older_statuses = true
 # Delete Twitter likes that are older than 90 days
 delete_older_favs = true
+# Do not sync retweets
+sync_retweets = false


Default value should be true also in the docs to have the current behavior.

klausi · 2020-05-02T09:04:24Z

src/sync.rs

@@ -460,7 +484,7 @@ UNLISTED 🔓 ✅ Tagged people

        let tweets = vec![tweet];
        let statuses = vec![status];
-        let posts = determine_posts(&statuses, &tweets);
+        let posts = determine_posts(&statuses, &tweets, &DEFAULT_SYNC_OPTIONS);


this is a bit annoying that we have this API change. I thought about doing a bit of compatibility dance and keep determine_posts() with the same function signature. Then add determine_posts_with_config() which can be called by main(). But probably that is not worth it and just adds one layer of indirection, and refactoring tests like this is fine. mastodon-twitter-sync is not a library, so we can mess up function signatures all we want :)

Yeah, it is annoying. On the other hand, now that we have made this change, adding more sync options should be much easier in the future.

…as it was before (#14)

bajtos · 2020-05-02T09:40:42Z

I just recently implemented the quote tweet behavior because I found it really annoying to have twitter links on Mastodon. There is enough post character space on Mastodon to also include the quote tweet then the user does not have to click to get to the content Twitter.

Makes sense, I see how Twitter links on Mastodon can be annoying. I guess my point of view is that I don't want to (re)post text written by other users and want to treat quoted tweets the same way other links are treated (e.g. when tweeting about a blog post published elsewhere).

Of course I'm open to move that behavior under a configurable flag, we can discuss in a new issue :)

Sounds great 👍

bajtos · 2020-05-02T09:41:51Z

Thank you @klausi for landing my pull request! ❤️

bajtos mentioned this pull request Apr 15, 2020

Feature request: ignore retweets/boosts #11

Closed

bajtos force-pushed the feat/ignore-retweets branch from 3b94672 to 1bf0658 Compare April 15, 2020 09:08

bajtos commented Apr 15, 2020

View reviewed changes

src/main.rs Outdated Show resolved Hide resolved

bajtos marked this pull request as draft April 15, 2020 09:11

bajtos force-pushed the feat/ignore-retweets branch 2 times, most recently from 93edd36 to 425978f Compare April 15, 2020 09:17

bajtos marked this pull request as ready for review April 15, 2020 09:18

klausi requested changes Apr 17, 2020

View reviewed changes

README.md Outdated Show resolved Hide resolved

src/config.rs Outdated Show resolved Hide resolved

src/main.rs Outdated Show resolved Hide resolved

bajtos force-pushed the feat/ignore-retweets branch from 425978f to c4dfc36 Compare April 26, 2020 12:26

feat: disable sync of retweets & reblogs

6e215a6

Add two new config options allowing users to disable syncing of Twitter retweets and Mastodon reblogs (boosts). Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

bajtos force-pushed the feat/ignore-retweets branch from c4dfc36 to 6e215a6 Compare April 26, 2020 13:16

fixup! fix test name

fb857d3

Signed-off-by: Miroslav Bajtoš <mbajtoss@gmail.com>

klausi approved these changes May 2, 2020

View reviewed changes

klausi merged commit 820f058 into klausi:master May 2, 2020

klausi added a commit that referenced this pull request May 2, 2020

docs(README): Set default values for retweet/reblogf syncing to true …

fd2edac

…as it was before (#14)

bajtos deleted the feat/ignore-retweets branch May 2, 2020 09:36

This was referenced May 2, 2020

Configurable handling of quote tweets #17

Open

feat: add an option to ignore sync of quote tweets #18

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add options to ignore retweets and boosts (reblogs) #14

Add options to ignore retweets and boosts (reblogs) #14

bajtos commented Apr 15, 2020 •

edited

Loading

klausi left a comment

bajtos commented Apr 19, 2020

klausi commented Apr 20, 2020

bajtos commented Apr 26, 2020

klausi commented May 2, 2020

klausi left a comment

klausi May 2, 2020

klausi May 2, 2020

klausi May 2, 2020

bajtos May 2, 2020

bajtos commented May 2, 2020

bajtos commented May 2, 2020

Add options to ignore retweets and boosts (reblogs) #14

Add options to ignore retweets and boosts (reblogs) #14

Conversation

bajtos commented Apr 15, 2020 • edited Loading

klausi left a comment

Choose a reason for hiding this comment

bajtos commented Apr 19, 2020

klausi commented Apr 20, 2020

bajtos commented Apr 26, 2020

klausi commented May 2, 2020

klausi left a comment

Choose a reason for hiding this comment

klausi May 2, 2020

Choose a reason for hiding this comment

klausi May 2, 2020

Choose a reason for hiding this comment

klausi May 2, 2020

Choose a reason for hiding this comment

bajtos May 2, 2020

Choose a reason for hiding this comment

bajtos commented May 2, 2020

bajtos commented May 2, 2020

bajtos commented Apr 15, 2020 •

edited

Loading