Youtube api improvements

[SamantazFox]

This PR is intended to unify the implementation of the next endpoint. The authors of #2217, #2220, #2227 and #2257 will have to rebase their changes against this PR.

Notables changes:

• The youtube API functions have been grouped under the YoutubeAPI namespace and function names have been simplified, for code clarity.
• The next endpoint takes data in Hash or NamedTuple form, so it can be extended in the future, without requiring new function overrides. See documentation in the code for more details.

[unixfox]

Should we also implement the others endpoints in this PR and allow to set others clients?

[SamantazFox]

Should we also implement the others endpoints in this PR and allow to set others clients?

I was wondering that, too. You don't mind if that ends up rewriting half of your PRs?

[unixfox]

Should we also implement the others endpoints in this PR and allow to set others clients?

I was wondering that, too. You don't mind if that ends up rewriting half of your PRs?

No and that's a good thing.

[SamantazFox]

Here we go, I'm done here.

[unixfox]

Related to #2189 (comment), we will need to manipulate the context.

Context needs to be manipulatable as shown by recent information

[SamantazFox]

Related to #2189 (comment), we will need to manipulate the context.

Yeah, I saw that, however, I don't know if that can be provided in any request (We're always using the embed player, so it might fit) or is it required only for age bypass?

[syeopite]

Afaik it should only be required for the age restriction bypass though as shown by recent investigations, using different clients and contexts does provide slightly different functionality.

Either way, the context should be configurable through ClientConfig.

[SamantazFox]

Either way, the context should be configurable through ClientConfig.

That's harder than expected, lol

[unixfox]

We will probably need android embed too in order to solve the throttling issues and directly decrypt the encrypted video streams.

[SamantazFox]

We will probably need android embed too in order to solve the throttling issues and directly decrypt the encrypted video streams.

Sure! I don't know how/where to get those, and probably can't anyway (old, ungoogled phone), so any help is appreciated.

[syeopite]

Have a look at yt-dlp https://github.com/yt-dlp/yt-dlp/blob/c0bc527bca735ee8a8b32857466d75f7d47aa22a/yt_dlp/extractor/youtube.py#L373

[SamantazFox]

But you should be able to directly execute the decryption js.

the problem is that it requires an entire Js engine dependency to be added to invidious, which is something I'd prefer to avoid. See #2222.

[SamantazFox]

@syeopite regarding the ability to modify the context, I tried to implement that, but it's a hell of a nightmare. It seems that crystal (1.0.0, for now) have troubles with merging Hash structures.

I.e, the basic merge()/merge!() function only provide 1 level of merging, so merging {"client": {"foo":1} } and {"client": {"bar":1} } will return {"client": {"bar":1} }, and using a block (merge() { |key,v1,v2| # do_something}) to handle 2nd/3rd level merge will make the compiler complain that the data type doesn't match (the previous example is a Hash(String, String | Hash(String, String), but at level 2 it becomes String | Hash(String, String), which can't be merged.

So for now, I think it's better to keep things like that, and I'll have to find a way of making better data structures (maybe with a JSON::builder?) to hold the context data and allow modification.

[unixfox]

@syeopite regarding the ability to modify the context, I tried to implement that, but it's a hell of a nightmare. It seems that crystal (1.0.0, for now) have troubles with merging Hash structures.

I.e, the basic merge()/merge!() function only provide 1 level of merging, so merging {"client": {"foo":1} } and {"client": {"bar":1} } will return {"client": {"bar":1} }, and using a block (merge() { |key,v1,v2| # do_something}) to handle 2nd/3rd level merge will make the compiler complain that the data type doesn't match (the previous example is a Hash(String, String | Hash(String, String), but at level 2 it becomes String | Hash(String, String), which can't be merged.

So for now, I think it's better to keep things like that, and I'll have to find a way of making better data structures (maybe with a JSON::builder?) to hold the context data and allow modification.

But you can technically merge the second level this way:

hash1 = {"client" => {"foo" => 1} }
hash2 = {"client" => {"bar" => 1} }
hash1["client"].merge!(hash2["client"])
puts hash1 #{"client" => {"foo" => 1, "bar" => 1}}

[syeopite]

It seems that crystal (1.0.0, for now) have troubles with merging Hash structures.

In that case, this is fine for now.

invidious/src/invidious/helpers/youtube_api.cr

Lines 37 to 42 in 56ba453

	ClientType::WebAgeBypass => {
	name: "WEB",
	version: "2.20210721.00.00",
	api_key: "AIzaSyAO_FJ2SlqU8Q4STEHLGCilw_Y9_11qcW8",
	screen: "EMBED",
	},

Everything else looks pretty good!

[SamantazFox]

But you can technically merge the second level this way:

hash1 = {"client" => {"foo" => 1} }
hash2 = {"client" => {"bar" => 1} }
hash1["client"].merge!(hash2["client"])
puts hash1 #{"client" => {"foo" => 1, "bar" => 1}}

I've tried that, and the compiler's not hapy :(

 165 | client_context["client"].merge!(
                                ^-----
Error: undefined method 'merge!' for String (compile-time type is (Hash(String, String) | String))

[SamantazFox]

@syeopite I added more clients. Still approved for you?

@unixfox anything to say?