feat(ytm): Re-implement auth with cookie and oauth

config/ytmusic.json.example

@@ -4,6 +4,16 @@
     "enable": true,
     "clients": [],
     "data": {
+      "cookie": "__Secure-1PSIDTS=sidts-CjEB3EgAEvCd-......",
+      // either cookie or id/secret needs to be provided
+      "clientId": "891098404....apps.googleusercontent.com",
+      "clientSecret": "GOCS..."
+
+      // optional
+      //"redirectUri": "http://my.custom.tld/api/ytmusic/callback?name=MyYTMusic"
+    },
+    "options": {
+      "logDiff": true
     }
   }
 ]

docsite/docs/FAQ.md

@@ -130,15 +130,25 @@ Deezer has discontinued support for their API and the Deezer Source is now [**de
 
 ### Youtube Music fails after some time
 
-The Youtube Music library relies on scraping the YTM site (pretending to be a browser) by using cookies/auth from your actual browser. It does its best to keep these up to date but since this is not an official way to access the service YTM may invalidate your access _to the authenticated session_ at any time. How this is triggered is unknown and not something multi-scrobbler can control.
-
 If you see errors in multi-scrobbler for YTM that contain **401** or **403** like
 
 ```
 Error: Could not send the specified request to browse. Status code: 401
 ```
 
-then YTM has invalidated your access. [Follow the YTM instructions to retrieve a new set of cookies for multi-scrobbler](configuration/configuration.mdx#youtube-music) and then restart MS to potentially resolve the problem. See [this issue](https://github.com/FoxxMD/multi-scrobbler/issues/158) for further discussion of the problem.
+then YTM has invalidated your authentication.
+
+First, ensure you are NOT using [YoutubeTV authentication.](configuration/configuration.mdx?ytmAuth=ytt#youtube-music) If you completed authentication by entering a "User Code" you are using YoutubeTV which has stopped working. You should reauthenticate using **Cookies** or **Custom OAuth.**
+
+#### When using OAuth Client Authentication
+
+Refresh your authentication by using the **(Re)authenticate** link from MS's web dashboard.
+
+#### When using Cookies Authentication
+
+The library MS uses relies on scraping the YTM site by using cookies from your actual browser to pretend it is a browser. It does its best to keep these up to date but since this is not an official way to access the service YTM may invalidate your access _to the authenticated session_ at any time. How this is triggered is unknown and not something multi-scrobbler can control. You can help limit the chance of your session being invalidated by [getting the cookie from an Incognito/Private Session](https://github.com/LuanRT/YouTube.js/issues/803#issuecomment-2504032666) and then immediately closing the browser afterwards.
+
+To re-authenticate MS [follow the YTM instructions to retrieve a new set of cookies for multi-scrobbler](configuration/configuration.mdx?ytmAuth=cookie#youtube-music) and then restart MS to potentially resolve the problem.
 
 ## Configuration Issues
 
@@ -194,10 +204,58 @@ Refer to [Force Media Tracking](configuration/configuration.mdx#forcing-media-tr
 
 Before reporting an issue turn on metadata logging in the MS VLC configuration, [see the VLC documentation.](configuration/configuration.mdx#vlc-information-reporting)
 
-### Youtube Music misses scrobbles
+### Youtube Music misses or duplicates scrobbles
 
 In order for multi-scrobbler to accurately determine if a song has been scrobbled it needs **a source of truth.** For YTM this is a "history" list scraped from the YTM website. Unfortunately, the data in this list can be (often) inconsistent which makes it hard for multi-scrobbler to "trust" that it is correct and determine when/if new scrobbles occur. This inconsistency is not something multi-scrobbler can control -- it is a side-effect of having to use an unofficial method to access YTM (scraping).
 
-In order to compensate for this multi-scrobbler resets when it considers this list the "source of truth" based on if the list changes in an inconsistent way between consecutive checks. New scrobbles can only be detected when this list is "OK" as a source of truth for N+1 checks. Therefore, any new tracks that appear when the list is inconsistent will be ignored.
+To compensate for this multi-scrobbler resets when it considers this list the "source of truth" based on if the list changes in an inconsistent way between consecutive checks. New scrobbles can only be detected when this list is "OK" as a source of truth for N+1 checks. Therefore, any new tracks that appear when the list is inconsistent will be ignored.
+
+Duplicate scrobbles can also occur if the change between two checks is technically consistent. For instance, if you listen to a track twice in some period, separated by other music, YTM will sometimes "remove" the track from the earlier time (further down in your history) and "re-add" it at the top of the history.
+
+#### Reporting YTM scrobble issues
+
+If you experience these behaviors you can help improve MS's YTM heureistic by providing thorough feedback as [an issue.](https://github.com/FoxxMD/multi-scrobbler/issues/new?assignees=&labels=bug&projects=&template=01-bug-report.yml&title=bug%3A+) **Please do the following to provide the most useful report:**
+
+##### Turn on Change Detection
+
+In your YTM configuration (`ytmusic.json`) add `logDiff` under `options` like this:
+
+
+```json
+{
+    "type": "ytmusic",
+    "name": "MyYTM",
+    "data": { ... },
+    "options": {
+        "logDiff": true
+    }
+}
+```
+
+This will cause MS to log YTM history changes similar to this:
+
+```
+[Ytmusic - MyYTM] Changes from last seen list:
+1. (tuhe1CpHRxY) KNOWER - I’m The President --- undefined => Moved -  Originally at 6
+2. (Mtg8V6Xa2nc) Vulfpeck - Romanian Drinking Song --- Schvitz => Moved -  Originally at 1
+3. (rxbCaiyYSXM) Nightmares On Wax - You Wish --- In A Space Outta Sound => Moved -  Originally at 2
+4. (tMt_YXr90AM) Gorillaz - O Green World --- undefined => Moved -  Originally at 3
+...
+```
+
+Which are essential to troubleshooting this behavior.
+
+##### Provide Detail and Context
+
+Provide a detailed account of how you were using YTM when the issue occurred, including things like:
+
+* the platform listening on (desktop, mobile, 3rd party client, etc...)
+* any changes in platform
+  * > I switched from desktop to listening on my phone...
+* how you were listening to music
+  * > I was playing an album start to finish
+  * > I listened to two songs in a row, then browsed for a new song in library by artist, then went back to a song in the queue...
+
+Explain the expected behavior (it should have scrobbled songs x, y, then z) and what actually happened (it scrobbled songs x, then y, then x again, then z)
 
-See [this issue](https://github.com/FoxxMD/multi-scrobbler/issues/156#issuecomment-2312533486) for further discussion and a more detailed explanation of why this is happening and how multi-scrobbler compensates for it.
+Provide ALL logs from the time when the issue occurred including logs from BEFORE (ideally 2-3 minutes of logs) and AFTER the issue.

docsite/docs/configuration/configuration.mdx

@@ -721,36 +721,183 @@ After starting multi-scrobbler with credentials in-place open the dashboard (`ht
 
 ### [Youtube Music](https://music.youtube.com)
 
-<details>
+:::warning
 
-    <summary>Migrating from YT Music cookie-based Source</summary>
+* Communication with YT Music is **unofficial** and not supported or endorsed by Google. This means that **this integration may stop working at any time** if Google decides to change how YT Music works in the browser.
+  * Due to this scrobble history from YTM is often inconsistent and can cause missed scrobbles. [See the FAQ](../FAQ.md#youtube-music-misses-scrobbles) for a more detailed explanation.
 
-    In multi-scrobbler **below v0.9.0** YT Music credentials were extracted from browser cookies. Due to authentication inconsistency and YT service changes this was approach was dropped in favor of [oauth authentication which is more stable.](https://ytjs.dev/guide/authentication.html#youtube-tv-oauth2)
+:::
 
-    Your existing credentials cannot be migrated. However, the oauth approach is quite easy. Continue following the directions below to setup new authentication for your YT Music Source.
+#### Authentication
 
-</details>
+Only one of these methods needs to be used. **Cookies** are easier but **OAuth Client**  may be more stable.
 
-:::note
+<Tabs groupId="ytmAuth" queryString>
+    <TabItem value="cookie" label="Cookies">
+        :::info
 
-* Communication to YT Music is **unofficial** and not supported or endorsed by Google. This means that **this integration may stop working at any time** if Google decides to change how YT Music works in the browser.
-  * Due to this scrobble history from YTM is often inconsistent and can cause missed scrobbles. [See the FAQ](../FAQ.md#youtube-music-misses-scrobbles) for a more detailed explanation.
+        If cookies stop working for you or are being invalidated often try switching to **OAuth Client** authentication.
 
-:::
+        :::
 
-To authenticate simply start multi-scrobbler with an empty YT Music configuration. An authentication URL/code will be logged in additon to being available from the dashboard.
+        Use instructions from 
+
+        * https://github.com/patrickkfkan/Volumio-YouTube.js/wiki/How-to-obtain-Cookie or 
+        * https://ytmusicapi.readthedocs.io/en/stable/setup/browser.html#copy-authentication-headers 
+
+        to get the **Cookie** value from a browser.
+
+        It is highly recommended to [get the cookie from an Incognito/Private Session](https://github.com/LuanRT/YouTube.js/issues/803#issuecomment-2504032666) to limit the chance the session is invalidated from normal browsing.
+
+        Add the cookie to your `ytmusic.json` config in `data`: 
+
+        ```json
+            {
+            "type": "ytmusic",
+            "enable": true,
+            "name": "MyYTM",
+            "data": {
+                "cookie": "__Secure-1PSIDTS=sidts-CjEB3EgAEvCd-......"
+            },
+            "options": {
+                "logAuthUpdateChanges": true,
+                "logDiff": true
+            }
+        }
+        ```
 
-```
-[2024-10-09 15:24:17.358 -0400] INFO   : [App] [Sources] [Ytmusic - MyYTM] ERROR: Sign in with the code 'CLV-KFA-BVKY' using the authentication link on the dashboard or https://www.google.com/device
-```
+        If MS gives you authentication errors (session invalidated) at some point in the future follow the same instructions to get new cookies.
+
+    </TabItem>
+    <TabItem value="oauth" label="OAuth Client">
+        :::note
+
+        This is likely to be the most stable method and least likely to be blocked or have authentication invalidated after an extended period. It requires more setup but is worth the effort.
+
+        :::
+
+        [Based on the instructions from here...](https://github.com/LuanRT/YouTube.js/issues/803#issuecomment-2479689924)
+
+        * Login to [Google Cloud console](https://console.cloud.google.com/) (create an account, if necessary)
+          * [Create a new project](https://console.cloud.google.com/projectcreate)
+        * Go to APIs and services.
+            * Configure the OAuth consent screen
+                * Use the old experience if possible
+                * If new is unavoidable then do not fill out any branding and under Authorized Domains you can delete the empty one (in order to save)
+                * Add yourself as an authorized user
+            * Navigate to Credentials
+                * Create Credentials -> choose "OAuth client ID"
+                    *  Application Type is **Web Application**
+                    * **Name** is whatever you want, leave Authorization Javascript origins blank
+                    * Authorized redirect URIs
+                        * This must be **exactly** the same as what is displayed in MS! For now leave it blank so we can generate it from MS first
+                * Create
+                    * In the newly created client popup save the **Client ID** and **Client Secret**, then copy them into `ytmusic.json`
+
+        ```json
+        {
+            "type": "ytmusic",
+            "enable": true,
+            "name": "MyYTM",
+            "data": {
+                "clientId": "8910....6jqupl.apps.googleusercontent.com",
+                "clientSecret": "GOCSPX-WGXL6BSuQ343..."
+            },
+            "options": {
+                "logAuthUpdateChanges": true,
+                "logDiff": true
+            }
+        }
+        ```
+
+        Now, start MS and during the YTMusic startup it will log something like this:
+
+        ```
+        Using Custom OAuth Client:
+        Client ID: ...
+        Client Secret: ...
+        Redirect URI: http://localhost:9078/api/ytmusic/callback?name=MyYTM
+        ```
 
-Visit the authentication URL and enter the code that was provided (also available on the dashboard). After completing the setup flow MS will log `Auth success` and the YT Music dashboard card will display as **Idle** after refreshing. Click the **Start** link to begin monitoring.
+        If the beginning of the URL (before `api`) is EXACTLY how you would reach the MS dashboard from your browser (EX `http://localhost:9078`) then edit your google oauth client section for `Authorized redirect URIs` and add the URL MS has displayed.
+
+        If it is NOT EXACTLY the same you either need to set MS's [base url](https://foxxmd.github.io/multi-scrobbler/docs/configuration/#base-url) or you can provide your own (Custom) Redirect URI for MS to use by setting it in `ytmusic.json`.
+
+        <details>
+
+            <summary>Using a Custom Redirect URI</summary>
+
+                The three parts of the URL that must be the same:
+
+                * it must start with `api` (after domain or subdirectory IE `my.domain.tld/api...` or `whatever.tld/subDir/api...`
+                * it must end in `ytmusic/callback`
+                * It must include `name=[NameOfSource]` in the query string
+
+                Remember to add your custom URL to the `Authorized redirect URIs`  section in the google oauth client!
+
+                ```json
+                {
+                    "type": "ytmusic",
+                    "enable": true,
+                    "name": "MyYTM",
+                    "data": {
+                        "clientId": "8910....6jqupl.apps.googleusercontent.com",
+                        "clientSecret": "GOCSPX-WGXL6BSuQ343...",
+                        "redirectUri": "http://my.custom.domain/api/ytmusic/callback?name=MyYTM"
+                    },
+                    "options": {
+                        "logAuthUpdateChanges": true,
+                        "logDiff": true
+                    }
+                }
+                ```
+
+        </details>
+
+        AFTER changing the Authorized redirect URIs on Google Cloud console you may need to wait a few minutes for it to take affect. Then restart MS. From the dashboard click `(Re)authenticate` on the YTmusic source card and follow the auth flow:
+
+        * On the screen about "testing" make sure you hit **Continue** (not Back To Safety)
+        * Make sure to select ALL scopes/permissions/grants it asks you about
+          * `Select what [YourAppName] can access` -> Select all
+
+        Once the flow is finished MS will get the credentials and start polling automatically. You should not need to re-authenticate again after restarting MS as it saves the credentials to the `/config` folder.
+
+    </TabItem>
+    <TabItem value="ytt" label="YoutubeTV">
+
+        :::warning
+
+        Using the built-in YoutubeTV authentication is unlikely to work due to [Google restricting what permissions TV clients can have](https://github.com/yt-dlp/yt-dlp/issues/11462#issuecomment-2471703090). This authentication method should not be used.
+
+        :::   
+
+        To authenticate start multi-scrobbler with an empty YT Music configuration. An authentication URL/code will be logged in additon to being available from the dashboard.
+
+        ```
+        ERROR: Sign in with the code 'CLV-KFA-BVKY' using the authentication link on the dashboard or https://www.google.com/device
+        ```
+
+        Visit the authentication URL and enter the code that was provided (also available on the dashboard). After completing the setup flow MS will log `Auth success` and the YT Music dashboard card will display as **Idle** after refreshing. Click the **Start** link to begin monitoring.
+
+    </TabItem>
+</Tabs>
 
 #### Configuration
 
 <Tabs groupId="configType" queryString>
     <TabItem value="env" label="ENV">
-        No ENV support
+
+
+
+| Environmental Variable | Required? | Default |                  Description                  |
+|------------------------|-----------|---------|-----------------------------------------------|
+| YTM_COOKIE             | No        |         | Value for Cookie Authentication               |
+| YTM_CLIENT_ID          | No        |         | Client ID for OAuth Athentication             |
+| YTM_CLIENT_SECRET      | No        |         | Client Secret for OAuth Athentication         |
+| YTM_REDIRECT_URI       | No        |         | A custom redirect URI for OAuth Athentication |
+| YTM_LOG_DIFF           | No        | false   | Log YTM history changes                       |
+
+
     </TabItem>
     <TabItem value="file" label="File">
         <details>