Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ytm): Re-implement auth with cookie and oauth #236

Merged
merged 11 commits into from
Dec 10, 2024
Merged

Conversation

FoxxMD
Copy link
Owner

@FoxxMD FoxxMD commented Dec 3, 2024

Checklist before requesting a review

Type of change

Please delete options that are not relevant.

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Describe your changes

Youtube TV seems no longer have scope for reading history or account details. Cookies may work and custom oauth seems most stable based on reporting from LuanRT/YouTube.js#803

  • Re-implement cookie auth
  • Implement custom OAuth Client auth based on example

TODO

Issue number and link, if applicable

#195
#229
#203

Youtube TV seems no longer have scope for reading history or account details?

Cookie may work and custom oauth seems most stable based on reporting from LuanRT/YouTube.js#803
@FoxxMD FoxxMD self-assigned this Dec 3, 2024
@FoxxMD FoxxMD added bug Something isn't working enhancement New feature or request safe to test trusted to build image labels Dec 3, 2024
@FoxxMD
Copy link
Owner Author

FoxxMD commented Dec 3, 2024

If you are testing this:

  • Please try both approach, if possible, so I can gather feedback on what needs improvement in MS
  • If one or both works for you please report that it does. I need to hear from users who this fix works for, not just those that it doesn't work for.

Cookie Auth

Works for now but this is likely to be the least stable option.

Use instructions from https://github.com/patrickkfkan/Volumio-YouTube.js/wiki/How-to-obtain-Cookie or https://ytmusicapi.readthedocs.io/en/stable/setup/browser.html#copy-authentication-headers to get the Cookie value.

It is recommended to get the cookie from an Incognito/Private Session to limit the chance the session is invalidated from normal browsing.

Add the cookie to your ytmusic.json config in data:

{
  "type": "ytmusic",
  "enable": true,
  "name": "MyYTM",
  "data": {
    "cookie": "__Secure-1PSIDTS=sidts-CjEB3EgAEvCd-......"
  },
  "options": {
    "logAuthUpdateChanges": true,
    "logDiff": true
  }
}

Custom OAuth

This is likely to be more stable but takes more setup than cookies. If you can, this is the recommended approach.

Base on the instructions from here...

  • Create a google cloud account and a new project
  • Go to APIs and services.
    • Configure the OAuth consent screen
      • Use the old experience if possible, new seems bugged (for me)
        • If new is unavoidable then do not fill out any branding and under Authorized Domains you can delete the empty one (in order to save)
      • Credentials
        • Create Credentials -> choose "OAuth client ID"
          • Application Type is Web Application
          • Name is whatever you want, leave Authorization Javascript origins blank
          • Authorized redirect URIs
            • This must be exactly the same as what is displayed in MS! For now leave it blank so we can generate it from MS first
        • Create
          • In the newly created client popup save the Client ID and Client Secret, then copy them into ytmusic.json
{
  "type": "ytmusic",
  "enable": true,
  "name": "MyYTM",
  "data": {
    "clientId": "8910....6jqupl.apps.googleusercontent.com",
    "clientSecret": "GOCSPX-WGXL6BSuQ343..."
  },
  "options": {
    "logAuthUpdateChanges": true,
    "logDiff": true
  }
}

NOTE: If you have an existing YTM credentials folder in your /config directory rename or delete it now. It would look like yti-MyYTM (or whatever your source name is)

Now, start MS and during the YTMusic startup it will log something like this:

Using Custom OAuth Client with Redirect URI: http://localhost:9078/api/ytmusic/callback?name=MyYTM

If the beginning of the URL (before api) is EXACTLY how you would reach the MS dashboard from your browser (EX http://localhost:9078) then edit your google oauth client section for Authorized redirect URIs and add the URL MS has displayed.

If it is NOT EXACTLY the same you either need to set MS's base url or you can provide a redirect URI for MS to use by setting it in ytmusic.json.

The two parts of the URL that must be the same:

  • it must start with api (after domain or subdirectory IE my.domain.tld/api... or whatever.tld/subDir/api...
  • it must end in ytmusic/callback
  • It must include name=[NameOfSource] in the query string

Remember to add your custom URL to the Authorized redirect URIs section in the google oauth client!

{
  "type": "ytmusic",
  "enable": true,
  "name": "MyYTM",
  "data": {
    "clientId": "8910....6jqupl.apps.googleusercontent.com",
    "clientSecret": "GOCSPX-WGXL6BSuQ343...",
   "redirectUri": "http://my.custom.domain/api/ytmusic/callback?name=MyYTM"
  },
  "options": {
    "logAuthUpdateChanges": true,
    "logDiff": true
  }
}

AFTER setting the google client redirect URI and restarting MS (if you made changes) wait a few minutes. Then restart MS. From the dashboard click (Re)authenticate on the YTmusic source card and follow the auth flow:

  • Make sure to select ALL scopes/permissions/grants it asks you about
  • On the screen about "testing" make sure you hit Continue (not Back To Safety)

Once the flow is finished MS will get the credentials and start polling automatically. You should not need to re-authenticate again after restarting MS as it saves the credentials to the /config folder.

@jrlebbs
Copy link

jrlebbs commented Dec 4, 2024

Good evening,

Moving to this thread, as requested.

First test with the cookie approach: worked as seamless as before.

Second test with the new oauth approach: error once authorizing past the oauth page "No page for /ytmusic/callback%3Fname=MyYTM exists!"

I did initially try the custom URI with my duckdns domain, but it stalled out entirely, so I reverted just to using the localhost to test this configuration.

I do notice that my redirect URI is "...callback%#Fname%" and your example is "callback?name". Maybe that contributes?

Let me know what else I can provide/test help!

@FoxxMD
Copy link
Owner Author

FoxxMD commented Dec 4, 2024

@jrlebbs can you give me more specifics about the conditions these things happened under?

Please provide the redirectUri you are using in ytmusic.json (if any) and also the log output containing the uri (Using Custom OAuth Client with Redirect URI: ...) for both tests that failed, thanks.

EDIT: I have pushed a change to the default redirectUri generation that may fix your issue. Please pull the latest pr-236 image and retry (and include logs/config!)

@xathon
Copy link

xathon commented Dec 4, 2024

Just pulled the latest image.

Can confirm that the cookie authentication works.

Notes while I set up Oauth:

The Google Cloud console tells me that the old experience "will only be available for a few more days. "

Getting a 403 when I use a brand account for YTM that's associated with my Google account. Had to add myself to the test users.

Now the oauth itself seems to work, but the callback gives a 404 page on my multiscrobbler instance. Log output (replaced my domain with example.com):

[2024-12-04 23:18:04.836 +0100] DEBUG  : [App] [API] Server received GET request from ::ffff:172.19.0.5 (192.168.10.20) (UA: Mozilla/5.0 (X11; Linux x86_64; rv:133.0) Gecko/20100101 Firefox/133.0) to unknown route: /api/ytmusic/callback%3Fname=YTMusic?code=4/0AeanS0aF8UsvWVfH7vnjpw67iNw30fccJ9NCmA2xvMHwT2SSZHRWBNwWh634G3FyGM9q1g&scope=https://www.googleapis.com/auth/youtube.force-ssl%20https://www.googleapis.com/auth/youtube-paid-content%20https://www.googleapis.com/auth/youtube%20https://www.googleapis.com/auth/accounts.reauth
[2024-12-04 23:17:50.881 +0100] ERROR  : [App] [Sources] [Ytmusic - YTMusic] Authentication test failed!
Error: Authentication test failed!
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:235:31)
    at YTMusicSource.reauthenticate (CWD/src/backend/sources/YTMusicSource.ts:138:9)
    at <anonymous> (CWD/src/backend/server/auth.ts:56:17)
    at async wrappedMiddleware (CWD/node_modules/@awaitjs/express/index.js:116:7)
caused by: Error: Sign in using https://accounts.google.com/o/oauth2/v2/auth?access_type=offline&scope=http%3A%2F%2Fgdata.youtube.com%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube.force-ssl%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fyoutube-paid-content%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Faccounts.reauth&include_granted_scopes=true&prompt=consent&response_type=code&client_id=547627672513-4t2drr32djjstof7eiqnofif6318lhtm.apps.googleusercontent.com&redirect_uri=https%3A%2F%2Fscrobbler.home.example.com%2Fapi%2Fytmusic%2Fcallback%253Fname%3DYTMusic
    at YTMusicSource.doAuthentication (CWD/src/backend/sources/YTMusicSource.ts:223:27)
    at YTMusicSource.testAuth (CWD/src/backend/common/AbstractComponent.ts:229:27)
    at YTMusicSource.reauthenticate (CWD/src/backend/sources/YTMusicSource.ts:138:9)
    at <anonymous> (CWD/src/backend/server/auth.ts:56:17)
    at async wrappedMiddleware (CWD/node_modules/@awaitjs/express/index.js:116:7)
[2024-12-04 23:17:50.879 +0100] INFO   : [App] [Sources] [Ytmusic - YTMusic] Using Custom OAuth Client with Redirect URI: https://scrobbler.home.example.com/api/ytmusic/callback%3Fname=YTMusic

According to the Google Cloud UI, the OAuth token is being passed. The Redirect URI that the log shows is also what I put in the credentials tab:
image

But in the end, it's not working.

@FoxxMD
Copy link
Owner Author

FoxxMD commented Dec 5, 2024

@xathon thanks for the feedback. Can you try replacing %3F in your Authorized redirect URI with ? and see if that makes a difference?

@xathon
Copy link

xathon commented Dec 5, 2024

Forgot to mention, I actually did that. Google then told me that the redirect URL doesn't match.

@bockbilbo
Copy link

Hey @FoxxMD, I am having the same behavior as @xathon .

After following the steps to set up Oauth, and clicking auth, I got to the Google page for the permissions, then accepted the risk of the testing app, but then got redirected to a url within my MS installation saying "Not Found" (aka 400). I then modified the %3F with ? and now google is complaining about the url not matching the Authorized redirect URIs.

So, I went back to the Google console, and added a second URL under Authorized redirect URIs with the URL using the question mark. After that, the MS page I am redirected to from the Google authorization process says "OK".

I can se the logs now showing tracks discovered in YT Music!

@bockbilbo
Copy link

I can confirm that MS is now scrobbling YTMusic again! Thanks a lot @FoxxMD !

@FoxxMD
Copy link
Owner Author

FoxxMD commented Dec 5, 2024

Google then told me that the redirect URL doesn't match.
...
So, I went back to the Google console, and added a second URL under Authorized redirect URIs with the URL using the question mark.

I did find that after changing the auth URL it takes google cloud at least a few minutes for the change to take affect.

I'm trying to figure out why MS is displaying %3F to ya'll but it should be showing ? (and it shows ? to me in dev).

EDIT: redirect uri query string should be fixed in the latest image.

@xathon
Copy link

xathon commented Dec 5, 2024

Can confirm, that works! Thanks!

@trevornk
Copy link

trevornk commented Dec 5, 2024

I was having the same issues as above users when trying to use oauth yesterday. Today I skipped oauth and went right to cookie auth and everything worked great.

@jrlebbs
Copy link

jrlebbs commented Dec 5, 2024

@jrlebbs can you give me more specifics about the conditions these things happened under?

Please provide the redirectUri you are using in ytmusic.json (if any) and also the log output containing the uri (Using Custom OAuth Client with Redirect URI: ...) for both tests that failed, thanks.

EDIT: I have pushed a change to the default redirectUri generation that may fix your issue. Please pull the latest pr-236 image and retry (and include logs/config!)

Here's my update.

Pulled the latest build of pr-236.

Attempt 1: No custom URI, just used the localhost URI generated by MS. Worked amazing!

Attempt 2: custom domain URI: "http://customdomainhere.duckdns.org:9078/api/ytmusic/callback?name=MyYTM"
clicked "re-authenticate"
re-routes to this URL: http://customdomainhere.duckdns.org:9078/api/source/undefined
Error: "Not Found"

confirmed that i put it in the google cloud redirect uri as well:

image

Verbose logs attached.
log_customdomain.txt

@FoxxMD
Copy link
Owner Author

FoxxMD commented Dec 6, 2024

The newest image should now allow re-authenticating and will also verify the validity of a custom redirect uri

@jrlebbs
Copy link

jrlebbs commented Dec 6, 2024

The newest image should now allow re-authenticating and will also verify the validity of a custom redirect uri

it worked!! no issues at this point!

@FoxxMD
Copy link
Owner Author

FoxxMD commented Dec 6, 2024

Newest image supports ENV config:

Environmental Variable Required? Default Description
YTM_COOKIE No Value for Cookie Authentication
YTM_CLIENT_ID No Client ID for OAuth Athentication
YTM_CLIENT_SECRET No Client Secret for OAuth Athentication
YTM_REDIRECT_URI No A custom redirect URI for OAuth Athentication
YTM_LOG_DIFF No false Log YTM history changes

@tylermiranda
Copy link

What a PIA...That being said..Thank you @FoxxMD for your work on this. I got mine working...

Copy link
Contributor

github-actions bot commented Dec 9, 2024

📦 A new release has been made for this pull request.

To play around with this PR, pull an image:

  • foxxmd/multi-scrobbler:pr-236

Images are available for x86_64 and ARM64.

Latest commit: 9f11d79

@FoxxMD FoxxMD merged commit 1e65abd into master Dec 10, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request safe to test trusted to build image
Projects
None yet
Development

Successfully merging this pull request may close these issues.

bug: YT Music authentication test failed New YTMusic Cookie Format?
6 participants