Use bot user agent for openGraph queries

We found that some websites return opengraph information based
on the user agent.

In order to address this, using (bot) in the user agent string
may help the website behave correctly. When testing Twitter, we
found that the correct metadata is returned with the (bot) user
agent while it isn't for the default user agent.

This change additionally updates providers.json, as with the
user agent fix twitter will now return the expected openGraph
metadata.

Signed-off-by: Andrew Ryan <andrewryanchama@clover.club>

changelog.d/11985.misc

@@ -0,0 +1 @@
+Use bot user agent for openGraph queries.

synapse/res/providers.json

@@ -5,13 +5,11 @@
         "endpoints": [
             {
                 "schemes": [
-                    "https://twitter.com/*/status/*",
-                    "https://*.twitter.com/*/status/*",
                     "https://twitter.com/*/moments/*",
                     "https://*.twitter.com/*/moments/*"
                 ],
                 "url": "https://publish.twitter.com/oembed"
             }
         ]
     }
-]
+]

synapse/rest/media/v1/preview_url_resource.py

@@ -326,8 +326,9 @@ async def _do_preview(self, url: str, user: UserID, ts: int) -> bytes:
 
                 # Compile the Open Graph response by using the scraped
                 # information from the HTML and overlaying any information
-                # from the oEmbed response.
-                og = {**og_from_html, **og_from_oembed}
+                # from the oEmbed response. og tags from the original html
+                # have priority over oEmbed data.
+                og = {**og_from_oembed, **og_from_html}
 
                 await self._precache_image_url(user, media_info, og)
             else:
@@ -402,7 +403,12 @@ async def _download_url(self, url: str, output_stream: BinaryIO) -> DownloadResu
                 url,
                 output_stream=output_stream,
                 max_size=self.max_spider_size,
-                headers={"Accept-Language": self.url_preview_accept_language},
+                headers={
+                    b"Accept-Language": self.url_preview_accept_language,
+                    b"User-Agent": [
+                        "Synapse (bot; +https://github.com/matrix-org/synapse)"
+                    ],
+                },
                 is_allowed_content_type=_is_previewable,
             )
         except SynapseError: