Pics Cache Enhancements #389

Revadike · 2022-03-09T03:13:24Z

Allow pics cache to be saved to disk (Custom cache engine for picsCache #188)
Read (and sync) saved pics cache to reduce requesting productinfos (Cache for appOwnershipCached #385)
Refactor code and reduce complexity of getProductInfo() (comments 1 2)
Request product info in bulk (and save progress in between).
~~Request missing tokens by default? (discussion)~~
Use meta_data_only instead of tracking changenumber (reason: changenumber changes can be only requested until a certain point)
Fix README.md

Revadike · 2022-03-09T03:14:14Z

@DoctorMcKay early feedback is welcome and encouraged!

DoctorMcKay · 2022-03-09T03:37:35Z

Request missing tokens by default?

It does make sense to request missing tokens by default, but this would be a breaking change so it's best not to.

resources/default_options.js

DoctorMcKay · 2022-03-09T03:36:37Z

README.md

+
+Defaults to `false`.
+
+**Warning:** This will significantly increase the storage space that is needed!


How much disk space does this need? Might it be worthwhile to gzip the json before we store it?

I should add "potentially", because it all depends on how many products you will be requesting. But it will build up over time, of course.
As for your first question, I once saved the entire SteamUser instance to disk as JSON. This was around 140 MB and the majority of that data consisted of the pics cache JSON. That was 4 years ago and my library has grown from then. Anyway, it will be in the order of hundreds of MB. Maybe getting into the gigabytes eventually, if you are going to monitor all pics changes. Using the custom file engine, one may implement a database hook instead.

So, with the main functionality added now, I got this by just saving my owned products infos to disk:

I think that's reasonable, especially with my unreasonable library size :P
People can always choose to not save pics cache to disk anyway.
The danger lies if people also use picsCacheAll, and potentially save product infos of everything. I've added a warning especially for this.

Revadike · 2022-03-09T04:06:26Z

Request missing tokens by default?

It does make sense to request missing tokens by default, but this would be a breaking change so it's best not to.

Hmm, I'm thinking, maybe just like the global ownershipFilter, the same can be done by globally setting this behavior too.

DoctorMcKay · 2022-03-09T04:20:27Z

Hmm, I'm thinking, maybe just like the global ownershipFilter, the same can be done by globally setting this behavior too.

Ehh, I'm not a big fan of adding a new global option just to avoid having to type true a few times. A global ownership filter makes sense because it affects many different methods and is a complex data type, but a simple bool isn't a burden to type.

Revadike · 2022-03-09T04:21:57Z

Fair enough

Revadike · 2022-05-22T00:54:10Z

It should work as-is, but for this PR to work better, I highly suggest merging this PR (and potentially this PR) as well.

Revadike · 2022-05-22T00:57:14Z

components/apps.js

@@ -137,7 +137,7 @@ class SteamUserApps extends SteamUserAppAuth {
 	 * Get a list of apps or packages which have changed since a particular changenumber.
 	 * @param {int} sinceChangenumber - Changenumber to get changes since. Use 0 to get the latest changenumber, but nothing else
 	 * @param {function} [callback]
-	 * @returns {Promise<{currentChangeNumber: number, appChanges: number[], packageChanges: number[]}>}
+	 * @returns {Promise<{currentChangeNumber: number, appChanges: object[], packageChanges: object[]}>}


This was wrong, btw. I fixed it.

Revadike · 2022-05-22T01:01:51Z

components/apps.js

+
+		for (let appid in apps) {
+			// Public only apps are weird...
+			if (apps[appid].missingToken && !apps[appid].appinfo.public_only) {


I'm really conflicted about this and would love your input. Should I save product infos with missing tokens?
Initially, I would say yes, but that will lead to a difficult to handle situation: when you activate a new product on steam, it will not update your cached product info. Currently, it only updates saved pics cache when changes have been detected since last synced changenumber.
Additionally, there is the case of public_only apps. These are weird, since even if you own it, you cannot seem to request product access tokens for them?

Revadike · 2022-05-22T01:03:50Z

components/apps.js

-		// This one actually can take a while, so allow it to go as long as 60 minutes
-		return StdLib.Promises.timeoutCallbackPromise(3600000, ['apps', 'packages', 'unknownApps', 'unknownPackages'], callback, (resolve, reject) => {
+		// This one actually can take a while, so allow it to go as long as 120 minutes
+		return StdLib.Promises.timeoutCallbackPromise(7200000, ['apps', 'packages', 'unknownApps', 'unknownPackages'], callback, async (resolve, reject) => {


Increased to 2h, because one of the top game owners on steam is trying to use node-steam-user but has issues getting passed this point.

Increased to 2h, because one of the top game owners on steam is trying to use node-steam-user but has issues getting passed this point.

Btw, I can confirm with all my changes, this person was able to finally get his ownership cached. 🎉🎉

Revadike · 2022-06-11T00:15:56Z

New TODO:

Use meta_data_only instead of tracking changenumber (reason: changenumber changes can be only requested until a certain point)

Revadike · 2022-06-17T04:08:43Z

New TODO:

Fix README.md

Revadike · 2022-06-19T17:53:08Z

index.js

+		}
+
+		if (this.options.picsCacheAll && this.options.savePicsCache) {
+			this._warn('Both picsCacheAll and savePicsCache are enabled. Unless a custom storage engine is used, beware that this will cause a lot of disk IO and space usage!');


Maybe it's a good idea to detect custom storage usage, somehow...

Revadike · 2022-06-19T17:53:44Z

README.md

+
+If `enablePicsCache` is enabled, saves all product info from [the PICS cache](#picscache) to disk (in [`dataDirectory`](#dataDirectory)) or to your [Custom Storage Engine](#custom-storage-engine). This will significantly speed up the [`appOwnershipCached`](#appOwnershipCache) event from firing and reduce the amount product info requests to Steam. It will only save product info if it's not missing its access token.
+
+Added in {{TODO}}


Pending version bump

Revadike · 2022-06-19T18:06:25Z

components/apps.js

+					// 	"pkg cache:", 	Object.keys(cached.packages).length,
+					// 	"app to refresh:",	apps.length,
+					// 	"pkg to refresh:",	packages.length,
+					// );


Should I keep this? It's a great log for debugging purposes.

Revadike · 2022-07-01T18:00:08Z

Bug found: requesting product info of subids/appids as string results in deadlock.

Edit:
Or maybe not? :S

Revadike · 2022-07-22T23:54:02Z

Ready to merge?

Added savePicsCache

1240f7d

DoctorMcKay reviewed Mar 9, 2022

View reviewed changes

Revadike added 11 commits March 11, 2022 05:11

update jsdoc

0321d67

fix jsdoc

aa2cc26

issue warning for picsCacheAll+savePicsCache

a9bd0cf

_saveProductInfo jsdoc

c0afe04

savePicsCache functionality

488a74a

current changenumber to disk

24eb205

read changenumber on startup

3d45029

altered warning

ec7aa5d

spelling

84bfec3

implement product info cache

5de1d35

a lot of bugfixes

1a4e46d

Revadike marked this pull request as ready for review May 22, 2022 00:52

Revadike requested a review from DoctorMcKay May 22, 2022 00:52

Revadike commented May 22, 2022

View reviewed changes

Revadike added 5 commits May 22, 2022 03:05

corrected cached object

e514b50

file-manager@2.0.1

61bf5ca

split product info requests

571b5d3

fixed deadlock issue

58b0b70

bugfixes + small changes

2fd5ee0

Prevent any stale data with meta_data_only + sha check

3249f29

Revadike force-pushed the feature-picscache-enhancements branch from b92130b to 3249f29 Compare June 17, 2022 04:06

Revadike added 3 commits June 19, 2022 17:49

improve readme

ff77442

fixed anonymous usage

fd813a7

Merge branch 'master' into feature-picscache-enhancements

395ddeb

Revadike commented Jun 19, 2022

View reviewed changes

Revadike added 4 commits July 5, 2022 00:19

small changes

b8c3791

handle when sha is null

2b0fa8a

minor bugfix

037feac

comments + refactor + handle enablePicsCache disabled

f3a725e

Changed execution order

134ee79

Revadike closed this Oct 12, 2022

Revadike mentioned this pull request Oct 12, 2022

Pics Cache Enhancements #409

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pics Cache Enhancements #389

Pics Cache Enhancements #389

Revadike commented Mar 9, 2022 •

edited

Loading

Revadike commented Mar 9, 2022

DoctorMcKay commented Mar 9, 2022

DoctorMcKay Mar 9, 2022

Revadike Mar 9, 2022

Revadike May 22, 2022

Revadike commented Mar 9, 2022

DoctorMcKay commented Mar 9, 2022

Revadike commented Mar 9, 2022

Revadike commented May 22, 2022

Revadike May 22, 2022

Revadike May 22, 2022

Revadike May 22, 2022

Revadike Jun 3, 2022

Revadike commented Jun 11, 2022 •

edited

Loading

Revadike commented Jun 17, 2022 •

edited

Loading

Revadike Jun 19, 2022

Revadike Jun 19, 2022

Revadike Jun 19, 2022

Revadike commented Jul 1, 2022 •

edited

Loading

Revadike commented Jul 22, 2022


		Defaults to `false`.

		Warning: This will significantly increase the storage space that is needed!


		If `enablePicsCache` is enabled, saves all product info from [the PICS cache](#picscache) to disk (in [`dataDirectory`](#dataDirectory)) or to your [Custom Storage Engine](#custom-storage-engine). This will significantly speed up the [`appOwnershipCached`](#appOwnershipCache) event from firing and reduce the amount product info requests to Steam. It will only save product info if it's not missing its access token.

		Added in {{TODO}}

Pics Cache Enhancements #389

Pics Cache Enhancements #389

Conversation

Revadike commented Mar 9, 2022 • edited Loading

Revadike commented Mar 9, 2022

DoctorMcKay commented Mar 9, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Revadike commented Mar 9, 2022

DoctorMcKay commented Mar 9, 2022

Revadike commented Mar 9, 2022

Revadike commented May 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Revadike commented Jun 11, 2022 • edited Loading

Revadike commented Jun 17, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Revadike commented Jul 1, 2022 • edited Loading

Revadike commented Jul 22, 2022

Revadike commented Mar 9, 2022 •

edited

Loading

Revadike commented Jun 11, 2022 •

edited

Loading

Revadike commented Jun 17, 2022 •

edited

Loading

Revadike commented Jul 1, 2022 •

edited

Loading