Use the new youtube api for comments #2217

unixfox · 2021-06-27T11:49:13Z

This PR is related to #1981.

This PR switch the endpoint used for fetching the comments from the old and soon deprecated https://www.youtube.com/comment_service_ajax to the InnerTube API with the endpoint /next.

Issue to fix:

invidious Invalid memory access (signal 11) at address 0xffff811a8158                                                                                                                        │
│ invidious [0x673488] print_backtrace at /usr/lib/crystal/core/exception/call_stack.cr:129:5                                                                                                  │
│ invidious [0x41ac70] __crystal_sigfault_handler at /usr/lib/crystal/core/signal.cr:347:3                                                                                                     │
│ invidious [0xf4e5f4] posix_spawn_file_actions_init at src/process/posix_spawn_file_actions_init.c:7:1                                                                                        │
│ invidious [0xcbd510] ???                                                                                                                                                                     │
│ invidious [0x47f334] -> at /invidious/lib/lsquic/src/lsquic/client.cr:210:11                                                                                                                 │
│ invidious [0x71df28] run at /usr/lib/crystal/core/primitives.cr:255:3                                                                                                                        │
│ invidious [0x41a184] -> at /usr/lib/crystal/core/fiber.cr:92:34                                                                                                                              │
│ invidious [0x0] ???

src/invidious/helpers/youtube_api.cr

SamantazFox · 2021-06-27T20:14:39Z

While you're at it, can you update the hard-coded client version? latest is 2.20210623

unixfox · 2021-06-27T20:49:25Z

While you're at it, can you update the hard-coded client version? latest is 2.20210623

Done ✔️

cloudrac3r · 2021-06-29T11:13:17Z

Why do you say comment_service_ajax will be deprecated?

unixfox · 2021-06-29T13:45:30Z

Why do you say comment_service_ajax will be deprecated?

The desktop interface of Youtube doesn't use it anymore so that's why I think it is going to soon stop working.

SamantazFox · 2021-06-29T21:10:02Z

The desktop interface of Youtube doesn't use it anymore so that's why I think it is going to soon stop working.

Oh, really? The youtube client is still using it in here...

unixfox · 2021-06-29T21:18:46Z

The desktop interface of Youtube doesn't use it anymore so that's why I think it is going to soon stop working.

Oh, really? The youtube client is still using it in here...

Maybe it was an experiment two days ago, unfortunately I didn't record anything, so I can't prove what I said. I really think it was really an experiment because I can't find again the MWEB client name being used on https://m.youtube.com.
m.youtube.com stopped using the innertube API too.

coletdjnz · 2021-06-29T21:33:15Z

We got a report about this too (yt-dlp/yt-dlp#438) - it seems like they are A/B testing it with the new API. For me it's still using the old API, but I'm guessing they'll slowly roll some form of this out in the coming months. The old endpoint probably won't go away for a while.

src/invidious/comments.cr

Perflyst · 2021-07-01T20:49:03Z

In testing on one of my backends

unixfox · 2021-07-07T19:55:56Z

Broken comments on https://yewtu.be/watch?v=Lsuob8Uug50

coletdjnz · 2021-07-08T06:41:45Z

FYI it appears that YouTube is deploying this new API change to desktop web now, and they have changed the response json structure too.

putara · 2021-07-08T10:29:03Z

I’ve been running my instance with #2220 for a week now.
The “View replies” link doesn’t always work and I have to click the link several times to make it work.

For example, right now I can’t open the “View 3 replies” link in JvJVvP5JIgs and just got this:
[info] 500 GET /api/v1/comments/JvJVvP5JIgs?format=html&hl=en-US&thin_mode=false&continuation=EkMSC0p2SlZ2UDVKSWdzyAEA4AEBogINKP___________wFAAMICHQgEGhdodHRwczovL3d3dy55b3V0dWJlLmNvbSIAGAYyTRpLEhpVZ3dRU0plb3BvTEFJY2pjanFsNEFhQUJBZyICCAAqGFVDTExyZ3Y0QmhwbGpJWlBpU0wzUHhMdzILSnZKVnZQNUpJZ3NAAUgK&action=action_get_comment_replies 270.22ms

Also it sometimes crashes the instance:

invidious_1  | Invalid memory access (signal 11) at address 0xffff9f6bd0f8
invidious_1  | [0x6a90f4] ???
invidious_1  | [0x41acd4] ???
invidious_1  | [0xfaa2f4] ???
invidious_1  | [0xd19230] ???
invidious_1  | [0x47f2d0] ???
invidious_1  | [0x753950] ???
invidious_1  | [0x41a1e8] ???
invidious_1  | [0x0] ???

unixfox · 2021-07-15T19:45:57Z

I’ve been running my instance with #2220 for a week now.
The “View replies” link doesn’t always work and I have to click the link several times to make it work.

For example, right now I can’t open the “View 3 replies” link in JvJVvP5JIgs and just got this:
[info] 500 GET /api/v1/comments/JvJVvP5JIgs?format=html&hl=en-US&thin_mode=false&continuation=EkMSC0p2SlZ2UDVKSWdzyAEA4AEBogINKP___________wFAAMICHQgEGhdodHRwczovL3d3dy55b3V0dWJlLmNvbSIAGAYyTRpLEhpVZ3dRU0plb3BvTEFJY2pjanFsNEFhQUJBZyICCAAqGFVDTExyZ3Y0QmhwbGpJWlBpU0wzUHhMdzILSnZKVnZQNUpJZ3NAAUgK&action=action_get_comment_replies 270.22ms

Also it sometimes crashes the instance:
invidious_1  | Invalid memory access (signal 11) at address 0xffff9f6bd0f8
invidious_1  | [0x6a90f4] ???
invidious_1  | [0x41acd4] ???
invidious_1  | [0xfaa2f4] ???
invidious_1  | [0xd19230] ???
invidious_1  | [0x47f2d0] ???
invidious_1  | [0x753950] ???
invidious_1  | [0x41a1e8] ???
invidious_1  | [0x0] ???

Should now be fixed.

@SamantazFox Could you take a look at the new code? Not sure if I wrote it properly.

SamantazFox · 2021-07-16T00:45:09Z

@SamantazFox Could you take a look at the new code? Not sure if I wrote it properly.

I'm not sure what you tried to achieve here: be22e2c#diff-c3b1698911da42ad59caee99b5c324448dfe7c8a79ca6b0995c2e07d3ee8ebe7R86-R99

unixfox · 2021-07-16T08:19:09Z

@SamantazFox Could you take a look at the new code? Not sure if I wrote it properly.

I'm not sure what you tried to achieve here: be22e2c#diff-c3b1698911da42ad59caee99b5c324448dfe7c8a79ca6b0995c2e07d3ee8ebe7R86-R99

Youtube is A/B testing a new format for the comments. This format is derived from the ANDROID client format.
We must support the old format and the new one at the same time because we don't know which one youtube is going to serve.

SamantazFox · 2021-07-16T11:34:27Z

Youtube is A/B testing a new format for the comments. This format is derived from the ANDROID client format.
We must support the old format and the new one at the same time because we don't know which one youtube is going to serve.

I got that, but what about the for loop that overrides the different variables each time?

SamantazFox · 2021-07-28T13:46:28Z

in testing on https://test.invidious.io, along with #2277 and #2220

NB: I had to do some minor changes to make it work

SamantazFox · 2021-08-12T17:15:19Z

Merging as-is, even if in some rare cases comments don't show up. Youtube has moved entirely to that new endpoint, so comments were broken.

unixfox force-pushed the comment-mobile branch from b5ddeb3 to 0545dea Compare June 27, 2021 11:52

unixfox added the need-testing This feature needs to be deployed and tested to see if it's working, and doesn't break something label Jun 27, 2021

unixfox requested a review from SamantazFox June 27, 2021 11:53

SamantazFox reviewed Jun 27, 2021

View reviewed changes

src/invidious/helpers/youtube_api.cr Outdated Show resolved Hide resolved

SamantazFox reviewed Jun 27, 2021

View reviewed changes

src/invidious/helpers/youtube_api.cr Outdated Show resolved Hide resolved

unixfox mentioned this pull request Jun 27, 2021

Switch to innertube API for videos #2220

Merged

3 tasks

unixfox force-pushed the comment-mobile branch from 0545dea to 800390c Compare June 27, 2021 20:48

iv-org deleted a comment from SamantazFox Jun 27, 2021

coletdjnz mentioned this pull request Jun 28, 2021

[Broken] Can't get youtube comments if cookies are provided yt-dlp/yt-dlp#438

Closed

6 tasks

Jerroder mentioned this pull request Jun 29, 2021

[Bug] YouTube throtle speed on some videos #2194

Closed

SamantazFox reviewed Jun 30, 2021

View reviewed changes

src/invidious/comments.cr Outdated Show resolved Hide resolved

unixfox force-pushed the comment-mobile branch 3 times, most recently from db589c9 to 33fd1d4 Compare June 30, 2021 10:50

SamantazFox added in-testing This feature has been deployed and is being tested and removed need-testing This feature needs to be deployed and tested to see if it's working, and doesn't break something labels Jul 2, 2021

iv-org deleted a comment from Akczht Jul 8, 2021

unixfox force-pushed the comment-mobile branch from b8f8ff7 to 33fd1d4 Compare July 10, 2021 15:39

unixfox marked this pull request as draft July 16, 2021 12:22

This was referenced Jul 19, 2021

Playlist starts at the offset (fixes: #1952) #2257

Closed

Youtube api improvements #2277

Merged

unixfox mentioned this pull request Jul 27, 2021

[Bug] Missing Hash key url in /api/v1/comments when reading comment threads #2262

Closed

unixfox added 4 commits August 3, 2021 03:10

use the new youtube api for comments

f124897

useless request

d3c9607

remove PG_DB & action parameter + allow force region

8452a4c

support new comments data with onResponseReceivedEndpoints

80f7b11

SamantazFox force-pushed the comment-mobile branch from be22e2c to 80f7b11 Compare August 3, 2021 01:15

SamantazFox mentioned this pull request Aug 11, 2021

[Bug] Cannot display replies to comments #2318

Closed

TheFrenchGhosty marked this pull request as ready for review August 12, 2021 17:11

TheFrenchGhosty assigned SamantazFox Aug 12, 2021

TheFrenchGhosty removed the in-testing This feature has been deployed and is being tested label Aug 12, 2021

SamantazFox merged commit 88c5e3b into iv-org:master Aug 12, 2021

TheFrenchGhosty mentioned this pull request Aug 12, 2021

[Bug] Edge case: Comments are broken on some videos #2322

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use the new youtube api for comments #2217

Use the new youtube api for comments #2217

unixfox commented Jun 27, 2021 •

edited

Loading

SamantazFox commented Jun 27, 2021

unixfox commented Jun 27, 2021 •

edited

Loading

cloudrac3r commented Jun 29, 2021

unixfox commented Jun 29, 2021

SamantazFox commented Jun 29, 2021

unixfox commented Jun 29, 2021 •

edited

Loading

coletdjnz commented Jun 29, 2021 •

edited

Loading

Perflyst commented Jul 1, 2021

unixfox commented Jul 7, 2021

coletdjnz commented Jul 8, 2021 •

edited

Loading

putara commented Jul 8, 2021

unixfox commented Jul 15, 2021

SamantazFox commented Jul 16, 2021

unixfox commented Jul 16, 2021

SamantazFox commented Jul 16, 2021

SamantazFox commented Jul 28, 2021 •

edited

Loading

SamantazFox commented Aug 12, 2021

Use the new youtube api for comments #2217

Use the new youtube api for comments #2217

Conversation

unixfox commented Jun 27, 2021 • edited Loading

SamantazFox commented Jun 27, 2021

unixfox commented Jun 27, 2021 • edited Loading

cloudrac3r commented Jun 29, 2021

unixfox commented Jun 29, 2021

SamantazFox commented Jun 29, 2021

unixfox commented Jun 29, 2021 • edited Loading

coletdjnz commented Jun 29, 2021 • edited Loading

Perflyst commented Jul 1, 2021

unixfox commented Jul 7, 2021

coletdjnz commented Jul 8, 2021 • edited Loading

putara commented Jul 8, 2021

unixfox commented Jul 15, 2021

SamantazFox commented Jul 16, 2021

unixfox commented Jul 16, 2021

SamantazFox commented Jul 16, 2021

SamantazFox commented Jul 28, 2021 • edited Loading

SamantazFox commented Aug 12, 2021

unixfox commented Jun 27, 2021 •

edited

Loading

unixfox commented Jun 27, 2021 •

edited

Loading

unixfox commented Jun 29, 2021 •

edited

Loading

coletdjnz commented Jun 29, 2021 •

edited

Loading

coletdjnz commented Jul 8, 2021 •

edited

Loading

SamantazFox commented Jul 28, 2021 •

edited

Loading