feat(dev): visualize network requests in the development environment on Chrome Devtools's network tab

[GrinZero]

Involved Issue / 该 PR 相关 Issue

NOROUTE

Example for the Proposed Route(s) / 路由地址示例

NOROUTE

New RSS Route Checklist / 新 RSS 路由检查表

• New Route / 新的路由
  • Follows Script Standard / 跟随 路由规范
• Anti-bot or rate limit / 反爬/频率限制
  • If yes, do your code reflect this sign? / 如果有, 是否有对应的措施?
• Date and time / 日期和时间
  • Parsed / 可以解析
  • Correct time zone / 时区正确
• New package added / 添加了新的包
• Puppeteer

Note / 说明

RSSHub 是一个非常依赖爬虫的项目，大部分 RSS 源都需要在插件中通过 got(ofetch) 爬取数据后进行加工。这个过程中，会发现调试过程并不显得非常棒，尤其是习惯了能通过 Chrome Devtools 可视化网络请求 payload、body 的开发者来说。

为了解决这个问题，增加开发体验，通过增加node-network-devtools包，这个包极大程度优化了网络调试过程的友好程度。

以下图为例，很容易从可视化的数据中发现某个插件存在部分损坏，而这个过程如果在控制台查看，detail 的大部分日志会被忽略。

预期中还有的额外场景是～ got(ofetch) 遇到 302 请求的自动 follow 行为，不一定可以被及时注意到，而通过可视化的方式或许可以进行提前发现，进而处理成提前中断(当新地址无作用时)来提升性能。

这一版本，我对node-network-devtools做了特制优化，确保不会出现像上次的 PR 一样出现因不存在 Chrome 而导致程序阻塞的问题。同时，请忽略仅会出现一次的日志MainProcess being already running，可以忽略，因为tsx导致其重复运行两次，这实际上是一个警告。

相信我，在安装了Chrome的电脑上调试效果非常好～甚至 node 官方 v22.6.0 实验性的网络调试方案之所以能完成，也是因为我在其中的一些推进。

其次，我在 Follow 的社区找到了一些 RSSHub 贡献者做尝试，他们认为可视化的能力非常有价值！

请不要关闭这个 PR，遇到了任何问题请提交评论，我会进行处理！RSSHub 的主动网络请求之多，可以和node-network-devtools摩擦出非常强大的火花💥。

[GrinZero]

Please do not close this PR. If you encounter any problems, please submit a comment and I will handle it! The high number of active network requests from RSSHub can create a powerful spark of friction with node-network-devtools.

[TonyRL]

The errors are still the same as #17280 (comment) shown. Please make sure it works, at least on GitHub Codespaces, before sending a PR.

[GrinZero]

Okay, I will try to access it on GitHub Codespaces.

[GrinZero]

I have completed the new version and tried it on GitHub Codespaces. This is the current phenomenon:

As I mentioned earlier, the error prompt of the latter is a double running defect from 'tsx' and can be ignored. And the former, on the other hand, dealt with the situation where I couldn't successfully open Chrome. Now, when I can't open the browser, it will throw a warning instead of an error.

[GrinZero]

I am pleased to announce that after several days of fixing, I have found the best debugging method in RSSHub. Now accessing 'node network devtools' will only trigger a single warning and significantly increase support for hot updates.

[TonyRL]

One last thing, request headers are empty.

[GrinZero]

let me cc

[GrinZero]

Thank you for providing the information. I have successfully located and resolved the issue. This is because I overlooked the handling of data that cannot be serialized when dealing with the fetch API. I have submitted a new commit and the issue has been fixed.

[TonyRL]

Headers are now showing but incomplete. All requests from RSSHub should have ua, referer and accept header.

RSSHub/lib/utils/request-rewriter/fetch.ts

Lines 23 to 41 in afcea8f

	// ua
	if (!request.headers.get('user-agent')) {
	request.headers.set('user-agent', config.ua);
	}
	// accept
	if (!request.headers.get('accept')) {
	request.headers.set('accept', '*/*');
	}
	// referer
	if (!request.headers.get('referer')) {
	try {
	const urlHandler = new URL(request.url);
	request.headers.set('referer', urlHandler.origin);
	} catch {
	// ignore
	}
	}

[GrinZero]

After a period of trying, I have identified the root cause. GrinZero/node-network-devtools#28

This is really bad news, that is, due to the packaging of GlobalThis ['fetch'] by RSSHub, and the fact that it ultimately calls the fetch of the unsupported undici library instead of originFetch, I am unable to intercept the headers within it.

I didn't expect there to be any other request methods besides ofetch&got.
I will try to provide support later, but it may take more time.

[GrinZero]

I have an idea. The current way that RSShub overwrites global fetch with undici is really rare, and even if I had undici support calls, it would result in repeated display requests. So, I plan to expose some hooks and allow external modification of some data, which will add some code, but the scalability is better, and I will finish it this weekend

[GrinZero]

Ready again

[GrinZero]

Thank you very much for every communication before the merger. This PR was very meaningful to me and helped me break through some of the limitations of my thinking