Fetch images when previewing Twitter

[AndrewRyanChama]

We found that some websites return opengraph information based
on the user agent.

In order to address this, using (bot) in the user agent string
may help the website behave correctly. When testing Twitter, we
found that the correct metadata is returned with the (bot) user
agent while it isn't for the default user agent.

This change additionally updates providers.json, as with the
user agent fix twitter will now return the expected openGraph
metadata.

Pull Request Checklist

• Pull request is based on the develop branch
• Pull request includes a changelog file. The entry should:
  • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
  • Use markdown where necessary, mostly for code blocks.
  • End with either a period (.) or an exclamation mark (!).
  • Start with a capital letter.
  • Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
• Pull request includes a sign off
• Code style is correct
  (run the linters)

Before:

After:

[AndrewRyanChama]

somewhat related #8022

[babolivier]

Hi @AndrewRyanChama, and thanks for your interest in Synapse 🙂

After discussing this PR in #synapse-dev:matrix.org, we've concluded that this is too specific/hacky to be accepted into Synapse. As much as possible, we would like to avoid having to rely on idioms and pseudo-conventions that we would have to change every time a big website changes its behaviour.

The compromise we're willing to make, though, is modifying the user-agent to mention "bot" in its name, so for example "Synapse (bot)", when fetching URL previews. While I still wish that we didn't have to do that (because it's at most a convention where we should be relying on standards), it should be generic enough and existing for long enough that we could accept it in Synapse.

Worth noting the solution of adding "bot" to the UA would fix the specific case of Twitter URL previews in an identical way to your proposed change:

$ curl -s -A "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" "https://twitter.com/matrixdotorg/status/1492207278560268289" | grep -i 'og:'
    <meta  property="og:type" content="article">
    <meta  property="og:url" content="https://twitter.com/matrixdotorg/status/1492207278560268289">
    <meta  property="og:title" content="Matrix on Twitter">
    <meta  property="og:image" content="https://pbs.twimg.com/media/FLVWl6bXEAElabl.jpg:large">
    <meta  property="og:image:user_generated" content="true">
    <meta  property="og:image" content="https://pbs.twimg.com/media/FLVWttFWUAEUQ8v.png:large">
    <meta  property="og:image:user_generated" content="true">
    <meta  property="og:image" content="https://pbs.twimg.com/media/FLVWzSlXEAQwrqi.png:large">
    <meta  property="og:image:user_generated" content="true">
    <meta  property="og:description" content="“🗞️ TWIM is out!&#10;&#10;This week all major homeservers got an update, @onbeeper was released out of beta, @kdecommunity&#39;s NeoChat is now sharing-friendly, @element_hq gets experimental support for threads… and of course much more in the latest TWIM issue!&#10;&#10;https://t.co/ZLD9JaOTLG”">
    <meta  property="og:site_name" content="Twitter">
$ curl -s -A "Synapse (bot)" "https://twitter.com/matrixdotorg/status/1492207278560268289" | grep -i 'og:'
    <meta  property="og:type" content="article">
    <meta  property="og:url" content="https://twitter.com/matrixdotorg/status/1492207278560268289">
    <meta  property="og:title" content="Matrix on Twitter">
    <meta  property="og:image" content="https://pbs.twimg.com/media/FLVWl6bXEAElabl.jpg:large">
    <meta  property="og:image:user_generated" content="true">
    <meta  property="og:image" content="https://pbs.twimg.com/media/FLVWttFWUAEUQ8v.png:large">
    <meta  property="og:image:user_generated" content="true">
    <meta  property="og:image" content="https://pbs.twimg.com/media/FLVWzSlXEAQwrqi.png:large">
    <meta  property="og:image:user_generated" content="true">
    <meta  property="og:description" content="“🗞️ TWIM is out!&#10;&#10;This week all major homeservers got an update, @onbeeper was released out of beta, @kdecommunity&#39;s NeoChat is now sharing-friendly, @element_hq gets experimental support for threads… and of course much more in the latest TWIM issue!&#10;&#10;https://t.co/ZLD9JaOTLG”">
    <meta  property="og:site_name" content="Twitter">

[AndrewRyanChama]

Thanks for the input. I updated the user agent string here to the one proposed

[anoadragon453]

@AndrewRyanChama heads up that synapse maintainers prefer that PR branches aren't rebased after each update, in order to make it easier to see what changed between reviews.

We squash and merge PRs at the end of the review cycle, so it'll all end up in one commit before hitting develop anyhow.

[clokep]

Tested a bunch of URLs with a janky test rig and this seems to have the following changes:

• Tweets now have images (either from the tweet itself or from the profile).
• Twitter "events" now work instead of giving a JavaScript error.
• Some reddit links now include the Reddit logo as an image and include a little bit more description (the first sentence of the post instead of just the vote counts).

Other tested sites had no change (YouTube, GitHub, Flickr, Tenor, Imgur, etc.)

Just a little bit of bookkeeping and I think we can merge this!

[clokep]

Weirdly these URLs don't seem to be used by Twitter anymore? We can leave them for now, but it is a bit annoying they changed their URL scheme at some point...

[clokep]

Thinking through this more, I think we should make this a .feature file which says "Fetch images when previewing Twitter URLs."

You can also credit yourself if you would like "Contributed by (your name or @AndrewRyanChama)."

stale

[clokep]

Thanks for working through this with us! 🎉

[clokep]

For cross-linking, there was previously #7643.