Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor!: better cross runtime support #97

Merged
merged 89 commits into from
Jul 20, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
0569933
refactor: remove dependancies
Wykerd Jul 11, 2022
34cb9da
refactor!: commonjs to es6
Wykerd Jul 11, 2022
1e7a6f0
refactor!: NToken and Signature TS files
Wykerd Jul 11, 2022
6f4d96f
feat: cross platform cache (WIP)
Wykerd Jul 12, 2022
f622448
feat: EventEmitter polyfill
Wykerd Jul 12, 2022
29fe937
refactor: remove events
Wykerd Jul 12, 2022
5035172
feat: HTTPClient based on Fetch API (WIP)
Wykerd Jul 12, 2022
27965ec
refactor!: parsers refactor (WIP)
Wykerd Jul 12, 2022
3726359
refactor!: parsers refactor (WIP)
Wykerd Jul 12, 2022
dd207df
refactor!: parser refactor
Wykerd Jul 12, 2022
37b8f70
fix: some missed parsers while refactoring
Wykerd Jul 12, 2022
189f152
fix: better type inferance for parseResponse
Wykerd Jul 12, 2022
ac83bad
feat(TS): typesafe YTNode casts
Wykerd Jul 13, 2022
53515fb
feat: more type safety in YTNode and Parser
Wykerd Jul 13, 2022
5a2e465
refactor: VideoInfo download with fetch & TS (WIP)
Wykerd Jul 13, 2022
8fb1924
fix: LiveChat in VideoInfo
Wykerd Jul 13, 2022
76d19b4
refactor!: more typesafety in parser
Wykerd Jul 13, 2022
5c49385
refactor!: VideoInfo almost completed
Wykerd Jul 13, 2022
8f873a5
refactor!: player and session refactors
Wykerd Jul 13, 2022
0392751
refactor!: move auth logic to Session (WIP)
Wykerd Jul 13, 2022
23258dc
refactor: TS port for Actions and Innertube
Wykerd Jul 13, 2022
fc1a403
refactor: NavigationEndpoint TS
Wykerd Jul 13, 2022
c9db56f
refactor!: VideoInfo compiles without errors
Wykerd Jul 13, 2022
e265305
chore: delete old player
Wykerd Jul 13, 2022
e1cae96
fix: import errors
Wykerd Jul 13, 2022
882f471
fix: Utils import fixes
Wykerd Jul 13, 2022
093aebc
fix: several runtime errors
Wykerd Jul 13, 2022
42577ae
fix: video streaming
Wykerd Jul 14, 2022
a851516
chore: remove console.log debugging
Wykerd Jul 14, 2022
98d318a
chore: remove old unused dependencies
Wykerd Jul 14, 2022
59be455
fix: typescript errors
Wykerd Jul 14, 2022
297c856
refactor: TS feed
Wykerd Jul 14, 2022
8cb28ad
chore: delete old Feed
Wykerd Jul 14, 2022
40cb10a
refactor: move streamToIterable into Utils
Wykerd Jul 14, 2022
558027b
refactor: AccountManager TS
Wykerd Jul 14, 2022
8586626
refactor: FilterableFeed to TS
Wykerd Jul 14, 2022
1d60d72
refactor: InteractionManager to TS
Wykerd Jul 14, 2022
061903d
refactor: PlaylistManager to TS
Wykerd Jul 14, 2022
ba9e349
refactor: TabbedFeed to TS
Wykerd Jul 14, 2022
29046f7
refactor: Music to TS (WIP)
Wykerd Jul 14, 2022
3f04c3f
fix: getting the tests to pass (6/12)
Wykerd Jul 14, 2022
50d6ee8
fix: tests (7/12)
Wykerd Jul 14, 2022
4a11ea8
fix: download tests (8/12)
Wykerd Jul 14, 2022
21a37ad
fix: tests (9/12)
Wykerd Jul 14, 2022
b0b4117
feat: key based type validation for parsers
Wykerd Jul 14, 2022
e429471
fix: comments tests pass (10/12)
Wykerd Jul 14, 2022
4b7aa96
refactor: type safety checks removing @ts-ignore
Wykerd Jul 14, 2022
21b87a3
fix: playlist tests pass (11/12)
Wykerd Jul 14, 2022
c7240db
fix: all tests pass for node 🎉
Wykerd Jul 14, 2022
4741b3e
build: working Deno bundle
Wykerd Jul 14, 2022
e888cb5
docs: update deno example to download video
Wykerd Jul 14, 2022
e4b62b2
refactor: MusicResponsiveListItem to TS
Wykerd Jul 14, 2022
5c532ce
docs: TSDoc for Parser helpers
Wykerd Jul 14, 2022
10a96ad
docs: Parser documentation for TS
Wykerd Jul 14, 2022
f597f57
docs: add note about parseItem and parseArray
Wykerd Jul 14, 2022
0f64611
test: remove browser tests since they're identical
Wykerd Jul 14, 2022
d051511
feat: browser support and proxy example
Wykerd Jul 15, 2022
4863c1c
fix: merge conflicts
Wykerd Jul 15, 2022
b3b5f21
fix: PlaylistManager TS after merge
Wykerd Jul 15, 2022
9701b92
feat: in-browser video streaming
Wykerd Jul 15, 2022
f39e73f
refactor: cleanup the Dash example
Wykerd Jul 15, 2022
6507ad2
feat: allow custom fetch implementations
Wykerd Jul 16, 2022
0bafe05
feat: fetch debugger
Wykerd Jul 16, 2022
51bf764
fix: OAuth login
Wykerd Jul 16, 2022
cab9b6c
refactor: remove file extensions from imports
Wykerd Jul 16, 2022
559eacd
refactor: build scripts
Wykerd Jul 16, 2022
9124854
fix: CustomEvent on node
Wykerd Jul 17, 2022
f6ff7c2
fix: LiveChat
Wykerd Jul 17, 2022
0a37abb
fix: linting
Wykerd Jul 17, 2022
2eee3e1
fix: liniting in build-parser-json
Wykerd Jul 17, 2022
679b827
chore: update test workflow
Wykerd Jul 17, 2022
a8a4d7f
fix: NToken errors after lint fixes
Wykerd Jul 17, 2022
76b9927
fix: codacy complaints
Wykerd Jul 17, 2022
77d985d
docs: update to reflect changes
Wykerd Jul 17, 2022
1b2570f
refactor: cleanup imports/exports
Wykerd Jul 17, 2022
79c6ca8
fix: browser example
Wykerd Jul 17, 2022
55b983f
fix: cache on node
Wykerd Jul 17, 2022
d51271b
fix: stupid mistake
Wykerd Jul 17, 2022
fa7d414
refactor: Session#signIn to wait untill success
Wykerd Jul 17, 2022
682261a
refactor: freeze Constants
Wykerd Jul 17, 2022
9682676
refactor: cleanup HTTPClient Request
Wykerd Jul 17, 2022
6b8249b
refactor: debugFetch readability
Wykerd Jul 17, 2022
f4be045
chore: lint
Wykerd Jul 17, 2022
1873f5a
refactor: replace jsdoc with tsdoc eslint plugin
Wykerd Jul 18, 2022
14be811
fix: bunch of liniting warnings
Wykerd Jul 18, 2022
3854d81
refactor: better inference on YTNode#is
Wykerd Jul 18, 2022
e772b86
fix: linting warnings
Wykerd Jul 18, 2022
96e3ed7
revert: undici import
Wykerd Jul 20, 2022
4974a06
refactor: rename `list_type` to `item_type`
Wykerd Jul 20, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .eslintrc.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,12 @@ extends: [ eslint:recommended, 'plugin:@typescript-eslint/recommended' ]
parser: '@typescript-eslint/parser'
parserOptions:
ecmaVersion: latest
overrides:
-
files:
- '**/*.js'
rules:
'tsdoc/syntax': 'off'
rules:
max-len:
- error
Expand All @@ -23,6 +29,7 @@ rules:

'@typescript-eslint/ban-types': 'off'
'tsdoc/syntax': 'warn'
'@typescript-eslint/no-explicit-any': 'off'

no-template-curly-in-string: error
no-unreachable-loop: error
Expand Down
2 changes: 1 addition & 1 deletion bundle/browser.d.ts
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export * from "../dist/browser";
export * from '../dist/browser';
2 changes: 1 addition & 1 deletion index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import { getRuntime } from './lib/utils/Utils';

// Polyfill fetch for node
if (getRuntime() === 'node') {
const undici = require('undici');
const undici = Reflect.get(module, 'require')('undici');
Wykerd marked this conversation as resolved.
Show resolved Hide resolved
Reflect.set(globalThis, 'fetch', undici.fetch);
Reflect.set(globalThis, 'Headers', undici.Headers);
Reflect.set(globalThis, 'Request', undici.Request);
Expand Down
8 changes: 4 additions & 4 deletions lib/Innertube.ts
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ class Innertube {
}
/**
* Searches a given query.
* @param string query - search query.
* @param query - search query.
* @param filters - search filters.
*/
async search(query: string, filters: SearchFilters = {}) {
Expand Down Expand Up @@ -139,7 +139,7 @@ class Innertube {
}
/**
* Retrieves watch history.
* Which can also be achieved with {@link getLibrary()}.
* Which can also be achieved with {@link getLibrary}.
*/
async getHistory() {
const response = await this.actions.browse('FEhistory');
Expand Down Expand Up @@ -195,7 +195,7 @@ class Innertube {
* An alternative to {@link download}.
* Returns deciphered streaming data.
*
* @note If you wish to retrieve the video info too, have a look at {@link getBasicInfo} or {@link getInfo}.
* If you wish to retrieve the video info too, have a look at {@link getBasicInfo} or {@link getInfo}.
*/
async getStreamingData(video_id: string, options: FormatOptions = {}) {
const info = await this.getBasicInfo(video_id);
Expand All @@ -204,7 +204,7 @@ class Innertube {
/**
* Downloads a given video. If you only need the direct download link take a look at {@link getStreamingData}.
*
* @note If you wish to retrieve the video info too, have a look at {@link getBasicInfo} or {@link getInfo}.
* If you wish to retrieve the video info too, have a look at {@link getBasicInfo} or {@link getInfo}.
*/
async download(video_id: string, options?: DownloadOptions) {
throwIfMissing({ video_id });
Expand Down
1 change: 0 additions & 1 deletion lib/core/AccountManager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@ import Analytics from '../parser/youtube/Analytics';
import Proto from '../proto/index';
import Actions from './Actions';

/** @namespace */
class AccountManager {
#actions;
channel;
Expand Down
2 changes: 1 addition & 1 deletion lib/core/InteractionManager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ class InteractionManager {
* Translates a given text using YouTube's comment translate feature.
*
* @param target_language - an ISO language code
* @param [args] - optional arguments
* @param args - optional arguments
*/
async translate(text: string, target_language: string, args: {
video_id?: string;
Expand Down
7 changes: 0 additions & 7 deletions lib/core/Music.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,12 +17,9 @@ import NavigationEndpoint from '../parser/classes/NavigationEndpoint';
import MusicDescriptionShelf from '../parser/classes/MusicDescriptionShelf';
import MusicCarouselShelf from '../parser/classes/MusicCarouselShelf';

/** @namespace */
class Music {
#session;
#actions;
constructor(session: Session) {
this.#session = session;
this.#actions = session.actions;
}
/**
Expand Down Expand Up @@ -125,13 +122,9 @@ class Music {
if (!upnext_content)
throw new InnertubeError('Invalid id', video_id);
return {
/** @type {string} */
id: upnext_content.playlist_id,
/** @type {string} */
title: upnext_content.title,
/** @type {boolean} */
is_editable: upnext_content.is_editable,
/** @type {import('../parser/contents/classes/PlaylistPanelVideo')[]} */
contents: observe(upnext_content.contents)
};
}
Expand Down
13 changes: 9 additions & 4 deletions lib/core/OAuth.ts
Original file line number Diff line number Diff line change
Expand Up @@ -179,24 +179,29 @@ class OAuth {
headers: Constants.OAUTH.HEADERS
});
const response_data = await response.text();
const url_body = Constants.OAUTH.REGEX.AUTH_SCRIPT.exec(response_data)![1]; // TODO: probably check this rather than assume.
const url_body = Constants.OAUTH.REGEX.AUTH_SCRIPT.exec(response_data)?.[1];
if (!url_body)
throw new OAuthError('Could not obtain script url.', { status: 'FAILED' });
const script = await this.#session.http.fetch(url_body, {
baseURL: Constants.URLS.YT_BASE
});
const client_identity = (await script.text())
.replace(/\n/g, '')
.match(Constants.OAUTH.REGEX.CLIENT_IDENTITY);
// TODO: check this.
return client_identity!.groups!;
const groups = client_identity?.groups;
if (!groups)
throw new OAuthError('Could not obtain client identity.', { status: 'FAILED' });
return groups;
}
get credentials() {
return this.#credentials;
}
validateCredentials() {
validateCredentials(): this is this & { credentials: Credentials } {
return this.#credentials &&
Reflect.has(this.#credentials, 'access_token') &&
Reflect.has(this.#credentials, 'refresh_token') &&
Reflect.has(this.#credentials, 'expires');
Reflect.has(this.#credentials, 'expires') || false;
}
}
export default OAuth;
10 changes: 6 additions & 4 deletions lib/core/Player.ts
Original file line number Diff line number Diff line change
Expand Up @@ -59,12 +59,14 @@ export default class Player {
url_components.searchParams.set('ratebypass', 'yes');
if (signature_cipher || cipher) {
const signature = this.#signature.decipher(url);
args.get('sp') ?
url_components.searchParams.set(args.get('sp')!, signature) :
const sp = args.get('sp');
sp ?
url_components.searchParams.set(sp, signature) :
url_components.searchParams.set('signature', signature);
}
if (url_components.searchParams.get('n')) {
const ntoken = this.#ntoken.transform(url_components.searchParams.get('n')!);
const n = url_components.searchParams.get('n');
if (n) {
const ntoken = this.#ntoken.transform(n);
url_components.searchParams.set('n', ntoken);
}
return url_components.toString();
Expand Down
4 changes: 2 additions & 2 deletions lib/parser/classes/NavigationEndpoint.ts
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ class NavigationEndpoint extends YTNode {
super();
const name = Object.keys(data || {})
.find((item) => item.endsWith('Endpoint') || item.endsWith('Command'));
this.payload = data?.[name!] || {};
this.payload = name ? Reflect.get(data, name) : {};
if (Reflect.has(this.payload, 'dialog')) {
this.dialog = Parser.parse(this.payload.dialog);
}
Expand Down Expand Up @@ -178,7 +178,7 @@ class NavigationEndpoint extends YTNode {
}
}
/**
* Calls the endpoint. (This is an experiment and may replace {@link call()} in the future.).
* Calls the endpoint. (This is an experiment and may replace {@link call} in the future.).
*/
async callTest(actions: Actions, args = { parse: true, params: {} }) {
if (!actions)
Expand Down
22 changes: 15 additions & 7 deletions lib/parser/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,17 @@ export default class Parser {
static #addToMemo(classname: string, result: YTNode) {
if (!Parser.#memo)
return;
if (!Parser.#memo.has(classname))

const list = Parser.#memo.get(classname);
if (!list)
return Parser.#memo.set(classname, [ result ]);
Parser.#memo.get(classname)!.push(result);

list.push(result);
}
static #getMemo() {
if (!Parser.#memo)
throw new Error('Parser#getMemo() called before Parser#createMemo()');
return Parser.#memo;
}
/**
* Parses InnerTube response.
Expand All @@ -106,24 +114,24 @@ export default class Parser {
this.#createMemo();
// TODO: is this parseItem?
const contents = Parser.parse(data.contents);
const contents_memo = Parser.#memo!;
const contents_memo = this.#getMemo();
// End of memoization
this.#clearMemo();
this.#createMemo();
const on_response_received_actions = data.onResponseReceivedActions ? Parser.parseRR(data.onResponseReceivedActions) : null;
const on_response_received_actions_memo = Parser.#memo!;
const on_response_received_actions_memo = this.#getMemo();
this.#clearMemo();
this.#createMemo();
const on_response_received_endpoints = data.onResponseReceivedEndpoints ? Parser.parseRR(data.onResponseReceivedEndpoints) : null;
const on_response_received_endpoints_memo = Parser.#memo!;
const on_response_received_endpoints_memo = this.#getMemo();
this.#clearMemo();
this.#createMemo();
const on_response_received_commands = data.onResponseReceivedCommands ? Parser.parseRR(data.onResponseReceivedCommands) : null;
const on_response_received_commands_memo = Parser.#memo!;
const on_response_received_commands_memo = this.#getMemo();
this.#clearMemo();
this.#createMemo();
const actions = data.actions ? Parser.parseActions(data.actions) : null;
const actions_memo = Parser.#memo!;
const actions_memo = this.#getMemo();
this.#clearMemo();
return {
actions,
Expand Down
4 changes: 2 additions & 2 deletions lib/parser/youtube/LiveChat.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,8 @@ class LiveChat extends EventEmitter {
throw new InnertubeError('Video has no livechat');
this.#video_info = video_info;
this.#actions = video_info.actions;
this.#continuation = this.#video_info.livechat!.continuation;
this.is_replay = this.#video_info.livechat!.is_replay;
this.#continuation = video_info.livechat.continuation;
this.is_replay = video_info.livechat.is_replay;
this.live_metadata = {
title: null as UpdateTitleAction | null,
description: null as UpdateDescriptionAction | null,
Expand Down
6 changes: 0 additions & 6 deletions lib/parser/youtube/Playlist.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,5 @@ class Playlist extends Feed {
return 'N/A';
return primary_info.stats[index]?.toString() || 'N/A';
}
/**
* @alias videos
*/
get items() {
return this.videos;
}
}
export default Playlist;
2 changes: 1 addition & 1 deletion lib/parser/youtube/Search.js
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ class Search extends Feed {
*/
constructor(actions, data, already_parsed = false) {
super(actions, data, already_parsed);
const contents = this.page.contents.item().as(TwoColumnSearchResults).primary_contents.item().contents.array()
const contents = this.page.contents.item().#as(TwoColumnSearchResults).primary_contents.item().contents.array()
|| this.page.on_response_received_commands[0].contents;
const secondary_contents = this.page.contents?.secondary_contents?.contents;
/** @type {object[]} */
Expand Down
7 changes: 0 additions & 7 deletions lib/parser/youtube/VideoInfo.ts
Original file line number Diff line number Diff line change
Expand Up @@ -145,8 +145,6 @@ class VideoInfo {
this.watch_next_feed = data?.contents;
return this;
}
/** @typedef {import('../classes/CompactVideo')} CompactVideo */
/** @typedef {import('../classes/CompactMix')} CompactMix */
/**
* Retrieves watch next feed continuation.
*/
Expand All @@ -157,11 +155,6 @@ class VideoInfo {
this.#watch_next_continuation = this.watch_next_feed?.pop()?.as(ContinuationItem);
return this.watch_next_feed?.filterType<CompactVideo | CompactMix>([ CompactVideo, CompactMix ]);
}
/**
* API response.
*
* @typedef {{ success: boolean, status_code: number, data: object }} Response
*/
/**
* Likes the video.
*
Expand Down
3 changes: 0 additions & 3 deletions lib/parser/ytmusic/Search.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ import RichShelf from '../classes/RichShelf';
import ReelShelf from '../classes/ReelShelf';
import ChipCloudChip from '../classes/ChipCloudChip';

/** @namespace */
class Search {
#page;
#actions;
Expand Down Expand Up @@ -83,11 +82,9 @@ class Search {
throw new InnertubeError('Endpoint did not return any data');
return new Search(response, this.#actions, { is_continuation: true });
}
/** @type {boolean} */
get has_continuation() {
return !!this.#continuation;
}
/** @type {string[]} */
get filters() {
return this.#header?.key('chips').parsed().array().as(ChipCloudChip).map((chip) => chip.text);
}
Expand Down
22 changes: 10 additions & 12 deletions lib/utils/Cache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,18 +47,16 @@ export default class UniversalCache {
const dir = this.cache_dir;
switch (getRuntime()) {
case 'deno':
{
const Deno: any = Reflect.get(globalThis, 'Deno');
try {
const cwd = await Deno.stat(dir);
if (!cwd.isDirectory)
throw new Error('An unexpected file was found in place of the cache directory');
} catch (e) {
if (e instanceof Deno.errors.NotFound)
await Deno.mkdir(dir, { recursive: true });
else
throw e;
}
const Deno: any = Reflect.get(globalThis, 'Deno');
try {
const cwd = await Deno.stat(dir);
if (!cwd.isDirectory)
throw new Error('An unexpected file was found in place of the cache directory');
} catch (e) {
if (e instanceof Deno.errors.NotFound)
await Deno.mkdir(dir, { recursive: true });
else
throw e;
}
break;

Expand Down
2 changes: 1 addition & 1 deletion lib/utils/HTTPClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ export default class HTTPClient {
if (oauth.validateCredentials()) {
// Check if the access token is valid to avoid authorization errors.
await oauth.checkAccessTokenValidity();
request_headers.set('authorization', `Bearer ${oauth.credentials!.access_token}`);
request_headers.set('authorization', `Bearer ${oauth.credentials.access_token}`);
// Remove API key as it is not required when using oauth.
request_url.searchParams.delete('key');
}
Expand Down
2 changes: 1 addition & 1 deletion lib/utils/Utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ export function timeToSeconds(time: string) {
/**
* Converts strings in camelCase to snake_case.
*
* @param string The string in camelCase.
* @param string - The string in camelCase.
*/
export function camelToSnake(string: string) {
return string[0].toLowerCase() + string.slice(1, string.length).replace(/[A-Z]/g, (letter) => `_${letter.toLowerCase()}`);
Expand Down