Youtube shortcode fails on 0.125.0+ on an unlisted video (media type not resolved)

[davidsneighbour]

The following shortcode:

{{< youtube ZVs79y0gjCA >}}

leads to this error:

ERROR The "youtube" shortcode was unable to get remote resource 
"https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZVs79y0gjCA". 
error calling resources.GetRemote: 
failed to resolve media type for remote resource 
"https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZVs79y0gjCA". 
See "/home/patrick/gitlab.com/wonderland/wonderland-engine-website/content/en/news/vertex-color-baked-lighting/index.md:179:1"
You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']

and the server does not start. This has happened since 0.125.0 (0.125.1 also shows the error; it seems unconnected to #12383).

In 0.124.1 it is parsed and shows the video.

hugo v0.124.1-db083b05f16c945fec04f745f0ca8640560cf1ec+extended linux/amd64 BuildDate=2024-03-20T11:40:10Z VendorInfo=gohugoio
hugo v0.125.1-68c5ad638c2072969e47262926b912e80fd71a77+extended linux/amd64 BuildDate=2024-04-18T08:21:19Z VendorInfo=gohugoio

This video is unlisted and other videos on the side do not run into this error.

[jmooring]

I am unable to reproduce the problem with v0.125.1 using:

{{< youtube ZVs79y0gjCA >}}

I tested with another unlisted video as well:

This seems to work fine:

{{< youtube 5I0gZlBdY8M >}}

[vonpupp]

I can confirm this behavior,

I am examining the problem closely.

ERROR The "youtube" shortcode was unable to get remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYCaotPnKU5w". error calling resources.GetRemote: failed to resolve media type for remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYCaotPnKU5w". See "/20220612002812-exploring_my_exo_brain.md:44:1"
You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']
ERROR The "youtube" shortcode was unable to get remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKkF9Sl_NgVc". error calling resources.GetRemote: failed to resolve media type for remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKkF9Sl_NgVc". See "//20220610163147-how_to_install_intersubs_on_mpv_to_learn_hebrew.md:38:1"
You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']
ERROR The "youtube" shortcode was unable to get remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Ds7db59pQruY". error calling resources.GetRemote: failed to resolve media type for remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Ds7db59pQruY". See "/20230822214616-how_to_generate_a_graph_of_a_repo_s_git_history.md:42:1"
You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']
ERROR The "youtube" shortcode was unable to get remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnFAIp6z2YdA". error calling resources.GetRemote: failed to resolve media type for remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnFAIp6z2YdA". See "/20230822214616-how_to_generate_a_graph_of_a_repo_s_git_history.md:47:1"
You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']

[jmooring]

@vonpupp What's the video id in your shortcode call? Please show the markdown.

Also:

• Are these "unlisted" per the OP's report above?
• Which operating system and version?

The YouTube response header has:

Content-Type: application/json

So I'm wondering if your system doesn't understand that content type? I have no problem adding any of the videos above with Ubuntu 22.04.4 LTS.

---

EDIT Sat 20 Apr 2024 08:35:04-08:00

I just tested again with a virgin install (plus Git, Go, and VSCode) of Ubuntu 22.04.2 LTS.

{{< youtube ZVs79y0gjCA >}}

{{< youtube 5I0gZlBdY8M >}}

{{< youtube nFAIp6z2YdA >}}

No problems, so I am still suspicious of media types (AKA MIME types) on each of your systems. I'm not saying we shouldn't address this, but the underlying cause may determine how we address this.

I am disinclined to remove error checking, though maybe we downgrade to a warning. However, if this turns out to be a media type issue on your systems, we may want to look at what resources.GetRemote is doing.

[vonpupp]

Thanks you for your answer @jmooring.

I am using Arch Linux. The content is an org-mode file converted to md using ox-hugo (an exporter for org-mode). Here is a snippet of the code that I use on the org-mode file.

#+html: {{< youtube YCaotPnKU5w >}}

The video is unlisted. It used to work previously. My version of hugo is:

hugo version
hugo` v0.125.1+extended linux/amd64 BuildDate=unknown

Here is the rendering of the md file using ox-hugo:

+++                                                                                                                                                                                                                                                                               
title = "Exploring my [exo]brain"                                                                                                                                                                                                                                                 
author = ["vonpupp"]                                                                                                                                                                                                                                        
date = 2022-06-12                                                                                                                                                                                                                                                                 
lastmod = 2024-04-20T20:35:29-03:00                                                                                                                                                                                                                                               
tags = ["exobrain", "digitalgardens", "research", "graphs", "communities", "clustering"]                                                                                                                                                                                          
categories = ["emacs", "visualization"]                                                                                                                                                                                                                                           
draft = false                                                                                                                                                                                                                                                                     
image = "/images/fm_exploring_exobrain.png"                                                                                                                                                                                                                                       
featured_image = "/images/exploring-exobrain.png"                                                                                                                                                                                                                                
+++                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                  
I really like working with graphs. I remember a project that I worked a while                                                                                                                                                                                                     
ago where I parsed the client requirements relationships into a graph and I                                                                                                                                                                                                       
could determine the priorities of modules development based on that. Good                                                                                                                                                                                                         
souvenirs. This original idea came from an analysis I did on my social                                                                                                                                                                                                            
networks out of curiosity and I could identify and [find communities on graphs]({{< relref "20220109211521-finding_communities_on_graphs.md" >}}).                                                                                                                                
                                                                                                                                                                                                                                                                                  
... text shortened ...

Now lets preview this thing... Most screenshots have been anonimized.  At the                                                                                                                                                                                                     
time of this writing this is how my notes looks like. Keep in mind that out of                                                                                                                                                                                                    
all of my notes, only a few selected ones are shared here on my website.                                                                                                                                                                                                          
                                                                                                                                                                                                                                                                                  
I am going to show first a little video sample                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                  
{{< youtube YCaotPnKU5w >}}                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                  
Figure [1](#figure--fig:full-headings) is how the exobrain looks considering all notes                                                                                                                                                                                            

Thank you very much!

[jmooring]

{{< youtube YCaotPnKU5w >}}  

The above renders for me without error.

On your system, do this:

touch foo.json
mimetype foo.json 

What's the result?

[davidsneighbour]

[10:51:17][patrick@Behemoth]~/gitlab.com/wonderland/wonderland-engine-website(+1|%0|ok)
❯ hugo version
hugo v0.125.2-4e483f5d4abae136c4312d397a55e9e1d39148df+extended linux/amd64 BuildDate=2024-04-20T15:29:44Z VendorInfo=gohugoio
[10:51:21][patrick@Behemoth]~/gitlab.com/wonderland/wonderland-engine-website(+1|%0|ok)
❯ touch foo.json
[10:51:24][patrick@Behemoth]~/gitlab.com/wonderland/wonderland-engine-website(+1|%0|ok)
❯ mimetype foo.json 
foo.json: application/json
[10:51:28][patrick@Behemoth]~/gitlab.com/wonderland/wonderland-engine-website(+1|%0|ok)
❯ 

I will investigate if there are any other pointers surrounding the mystery.

[davidsneighbour]

Going back to the old shortcode helps for now.

I think it might be an issue with resources.GetRemote because I recently tried to use that function with other services (Github REST API, a simple json edge function on Netlify, etc.) and it also resulted in mime type issues. Maybe that goes lost somewhere in this set up.

[vonpupp]

Sure,

mimetype foo.json
foo.json: application/json

I have an older static binary version that works fine:

Hugo Static Site Generator v0.59.0-DEV-5276F1E9:CHROMA-26F03CB linux/amd64 BuildDate: 2019-09-11T08:16:28-0400

[jmooring]

@vonpupp Instead of using an earlier version, for now I suggest you do as the message suggests:

You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']

[sudoCss]

Same issue here with v0.125.1 & v0.125.2, but with any YouTube video.. for example:

{{< youtube id="WmR9IMUD_CY" >}}

which is a public video, results this error:

Start building sites …
hugo v0.125.2+extended linux/amd64 BuildDate=unknown

ERROR The "youtube" shortcode was unable to get remote resource "https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DWmR9IMUD_CY". failed to fetch remote resource: Forbidden. See "/home/sudocss/path/to/my/hugo/project/content/articles/article-name/index.md:52:1"
You can suppress this error by adding the following to your site configuration:
ignoreLogs = ['err-youtube-remote']
Total in 2078 ms
Error: error building site: logged 1 error(s)

I'm on EndeavourOS.

[jmooring]

This is frustrating. I cannot reproduce the problem with any of the video id's referenced in this issue.

{{< youtube ZVs79y0gjCA >}}

{{< youtube 5I0gZlBdY8M >}}

{{< youtube YCaotPnKU5w >}}

{{< youtube KkF9Sl_NgVc >}}

{{< youtube s7db59pQruY >}}

{{< youtube nFAIp6z2YdA >}}

{{< youtube WmR9IMUD_CY >}}

Tested with Ubuntu 22.04 (virgin install plus Git, Go, and VSCode) and Windows 11.

@sudoCss As suggested in the error message, add this to your site config until we can figure out what's causing this:

ignoreLogs = ['err-youtube-remote']

[jmooring]

@davidsneighbour @sudoCss @vonpupp

I would appreciate it if you could post the result of:

curl -I https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZVs79y0gjCA

[sudoCss]

Sure

[jmooring]

@sudoCss And what do you see in the browser if you click this:

https://www.youtube.com/oembed?format=json&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZVs79y0gjCA

[sudoCss]

This:

[davidsneighbour]

❯ HTTP/2 404 
vary: X-Origin
vary: Referer
vary: Origin,Accept-Encoding
content-type: application/json; charset=UTF-8
date: Mon, 22 Apr 2024 08:27:07 GMT
server: scaffolding on HTTPServer2
cache-control: private
x-xss-protection: 0
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
accept-ranges: none

[1]+  Done                    curl -I https://www.youtube.com/oembed?format=json

and the direct link in the browser shows the JSON response for the oembed.

[davidsneighbour]

I added the config option and the error goes away, naturally. The interesting thing is that the video is shown in the resulting page. So either a fallback comes into play or the error message is somehow false?

[bep]

@jmooring looking at

hugo/tpl/tplimpl/embedded/templates/shortcodes/youtube.html

Line 30 in f40f50e

{{- $q := querify "url" (printf "https://www.youtube.com/watch?v=%s" $id) "format" "json" }}

I see that the only data item used from the returned JSON is the title. I briefly tested removing that remote call and I didn't see any difference in the YouTube videos I saw.

I suggest that we make the life a little easier for ourselves and either:

1. Remove that GetRemote block if it doesn't add any obvious value
2. Or silence any error/404.

[jmooring]

I had hoped to use the oEmbed API for other things in the future (e.g., thumbnail for a simple version), but the server response is obviously inconsistent, perhaps by IP range, proxy, etc. For the same video we get 200/403 or 200/404. I'm inclined to rip the whole thing out. I have the ball.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.