Youtube previews no longer available with v1.111.0 #17462
Comments
|
Would you be able to provide more details regarding in what situation this problem occurs? Specifically:
|
|
It happens with any Youtube videos (other than shorts), be it "youtu.be" or full "youtube.com" URLs. Tested clients are (always with the latest version) :
The problem happens with any rooms, public, private, existing, or new rooms. It happens with links sent from me, sent from other people on my homeserver, or over federation. Other members on my homeserver report the same issue of not getting any previews, with the same situations (ie. every link). Here are some logs relating to youtube.com (and the preview_url endpoint) : |
|
The location of the homeserver can affect what Google will show to non-logged in users. At some point we saw different responses between Europe/UK/USA. |
|
The server is located in the EU, however we did get previews before. We did not get a non-logged in preview. |
|
Hmm, very strange. When I use |
|
@devonh For the past month Youtube previews stopped working for me as well, except on synapse:v1.103.0 which is versions before 1.111.0. Years prior Youtube previews have worked fine across multiple versions/upgrades. No changes at all had been done, it just suddenly stopped working one day. I don't think this is version specific? Perhaps homeserver or config specific condition being hit or some kind of google/youtube side changes? I tried upgrading to synapse:v1.112.0 and Youtube URL Previews still do not work, all other URL Previews do work. I am unsure where or how to look at the logging @salixor is showing above, or what relevant logging I could provide to help track down the issue or compare what I am running into. If someone could point me to where best to look, please let me know! Thanks |
|
Logs are usually found at "Glad" to see I'm not the only one! The breakage happened around the time we upgraded to |
|
Thanks @salixor I am using https://github.com/spantaleev/matrix-docker-ansible-deploy, so instead of looking for /var/log/matrix-synapse/homeserver.log, I instead had to perform a I saw some similar things to you: |
|
When did you upgrade to v1.111.0 @salixor? , Looking at logging, it seems my first instances of "Couldn't get dims for" started for me on May 23 2024. With no prior logging mentioning synapse.media.url_previewer before that. |
|
Also, just like @salixor, YouTube Shorts work for me, just not regular videos. |
|
@daemontron It looks like this log line is cut off, what is the rest of the line? (it should print out the content type it read) (Looked at it a little more, not sure that line is relevant to youtube links - maybe it's something about viewing youtube from different regions leading to different redirects or somesuch thing?) |
|
@daemontron Is your server also based in the EU? |
The upgrade was a few days after its release, on July 18th. |
|
Note that previews should mostly work even if downloading the image fails. (This used to break but was fixed at some point...) |
@devonh US Based
|
|
@devonh @salixor I don't think this is a Matrix / Synapse issue, this seems to be regional, or IP Range based like said previously, some kind of slowly rolling out changes or something? Both my server and my home connection are US Based. Maybe It's targeted at popular hosting providers? The server hits the issue below, my own connection at home does not: From the server matrix/synapse is on:
From a linux box on my home connection:
|
|
I am indeed getting the same results. I'll admit since it happened exactly when we upgraded versions, I didn't even consider it could just be something coincidental not related to the upgrade, as nothing else I used got those broken previews. Well, I guess thanks Youtube for nothing, as usual. |
|
Seems like youtube does not ship opengraph data when you are greeted with the "Login to verify that you are not a Bot". But this can be fixed by using OEmbed, Synapse does not ship a default OEmbed provider config contrary to what it states in the Documentation. So I just added the following to my homeserver config: oembed:
disable_default_providers: true
additional_providers:
- providers.jsonI used this file: https://oembed.com/providers.json but it needed to be ""fixed"" because the oembed provider parser that exists in Synapse does not parse this file correctly, even though that is the official oembed provider file, you'll need to remove all entries that don't have "schemes" set in the entrypoints and remove the spotify entry, because Synapse doesn't like the "spotify:*" glob. Or if you just care about youtube you'll only need the following in your providers.json: [
{
"provider_name": "YouTube",
"provider_url": "https://www.youtube.com/",
"endpoints": [
{
"schemes": [
"https://*.youtube.com/watch*",
"https://*.youtube.com/v/*",
"https://youtu.be/*",
"https://*.youtube.com/playlist?list=*",
"https://youtube.com/playlist?list=*",
"https://*.youtube.com/shorts*",
"https://youtube.com/shorts*",
"https://*.youtube.com/embed/*"
],
"url": "https://www.youtube.com/oembed",
"discovery": true
}
]
}
]Related issue: #9877 |
It does, but it includes limited data: https://github.com/element-hq/synapse/blob/develop/synapse/res/providers.json
Please file a separate bug for this. (I would also ask that you use a less dismissive tone -- it isn't like the parser was designed purposefully to not parse this file. It was tested against it when it was originally developed, but the data might have changed or the code might have changed.) |
|
Before this was fixed (for a while) I used to work around this by hacking the HTTP agent for preview requests to be |
|
@clokep ah okay, I tried to find it on the docker image but only searched for oembed and not providers.json! Will do! |
|
@heftig I tried to debug this via curl and it seems to be IP based. Using |
|
I'm not suggesting we shouldn't add YouTube to it, but I believe we didn't initially because it gave worse results if you can properly load the URL. Ideally we could combine the information from both, if available. I don't think that ever got implemented. |
|
@clokep might be worth to mention that the https://oembed.com site also states this:
So the registry isn't really recommended, but the problem is if YouTube blocks you as a "bot", you can't even discover the oembed URL. |
|
#3440 is also a bit related. Just to double check -- does YouTube send the oEmbed discovery via header? I don't think we support that yet. |
|
@clokep yes it does, but as I explained in my last comment, if you get blocked as a bot you don't see the discovery URL, so it wouldn't be useful in this case. |
|
Ah I thought you were specifically referring to the HTML discovery URL. My misunderstanding. |
|
@clokep @devonh
Is it possible/safe to configure providers.json on a currently deployed homeserver? Perhaps to add Youtube itself as Twitter and Youtube Shorts are already there? Thanks |
You should be able to configure via https://element-hq.github.io/synapse/latest/usage/configuration/config_documentation.html#oembed to add additional providers if you'd like. |
|
Here's a script that should output a Synapse-compatible #!/usr/bin/env python3
import json
from collections import OrderedDict
from sys import stdout
from urllib.parse import urlparse
import jsonschema
import requests
from jsonschema.exceptions import ValidationError
output = requests.get("https://oembed.com/providers.json")
output.raise_for_status()
providers = output.json(object_pairs_hook=OrderedDict)
# From synapse/config/oembed.py
_OEMBED_PROVIDER_SCHEMA = {
"type": "array",
"items": {
"type": "object",
"properties": {
"provider_name": {"type": "string"},
"provider_url": {"type": "string"},
"endpoints": {
"type": "array",
"items": {
"type": "object",
"properties": {
"schemes": {
"type": "array",
"items": {"type": "string"},
},
"url": {"type": "string"},
"formats": {"type": "array", "items": {"type": "string"}},
"discovery": {"type": "boolean"},
},
"required": ["schemes", "url"],
},
},
},
"required": ["provider_name", "provider_url", "endpoints"],
},
}
while True:
try:
jsonschema.validate(providers, _OEMBED_PROVIDER_SCHEMA)
except ValidationError as e:
del providers[e.absolute_path[0]]
else:
break
def valid_url(url):
return urlparse(url).scheme in ["http", "https"]
def valid_provider(provider):
for endpoint in provider["endpoints"]:
if not valid_url(endpoint["url"]):
return False
for glob in endpoint["schemes"]:
if not valid_url(glob):
return False
return True
providers = [p for p in providers if valid_provider(p)]
json.dump(providers, stdout, indent=4)
print() |
|
Thanks @clokep and @heftig, Do you think this might warrant a PR to update https://github.com/element-hq/synapse/blob/develop/synapse/res/providers.json? I actually added the following to my own custom_providers.json and most YouTube Preview URLs now successfully parse! This is what I added to my homeserver.yaml: *It's /data/custom_providers.json and not /matrix/synapse/config/custom_providers.json because I use https://github.com/spantaleev/matrix-docker-ansible-deploy/ @salixor FYI incase you want to try it. I don't know if it's best to have disable_default_providers: to false or true, so kept it false and assumed what I added would be added in addition to the default providers. |
|
Hey, thanks a lot for the ping. I just tested it, and it works flawlessly! Thanks a lot for everyone's work ❤️ |
If this is a widespread issue then it is probably worthwhile. If it is only affecting some people then some comparison to what the scraping gets va oEmbed might be warranted? Depends on what the current team wants. |
Description
With v1.111.0 installed, Youtube previews for videos are no longer fetched, and simply show :
Previews were correctly fetched before this version.
Shorts are unaffected and still properly display the preview.
Steps to reproduce
Homeserver
baguette.party
Synapse Version
1.111.0
Installation Method
Debian packages from packages.matrix.org
Database
Single PostgreSQL server
Workers
Single process
Platform
Ubuntu 22.04
Configuration
None.
Relevant log output
Anything else that would be useful to know?
Seems to work perfectly fine for people on v1.110.0, and it worked for me on v1.110.0 as well.
The text was updated successfully, but these errors were encountered: