Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt to UE4.17+ patch pak reading order #40

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

PaperStrike
Copy link

@PaperStrike PaperStrike commented Mar 24, 2022

This partially resolves iAmAsval/FModel#204, where we only have the "oldest" (closer to "biggest" in my investigation) asset available, even when a newer one is there.
I dug into this and found the problem lies inside CUE4Parse. Currently CUE4Parse mounts paks parallelly and files in the last ready pak override the previous ones. That's the problem.

Two pieces of UE4 source code for reference:

  1. FPakPlatformFile::GetPakOrderFromPakFilePath, basic order of paks in different folders.
    This isn't implemented in this PR.
  2. FPakPlatformFile::Mount#L7239-L7265, order of numbered _P patches.
    This was not so detailed (UE<4.17) and I found some old articles complaining about it can't handle multiple patch paks well. Earlier games should have adapted their own ways (like, split into different parts) and I suppose that's the reason why iAmAsval/FModel#204 is raised only in recent months.

This PR

  1. adds a ReadOrder property to IVfsReader to order readers. Implemented a simple _P pak ordering in PakFileReader.
  2. adds a ReadOrder property to GameFile and adjusted it in VfsEntry to reflect the reader orderings.
    This is mainly to minimize changes to FileProviderDictionary, which receives path-GameFile pairs and I don't know if changing it would be too breaking.
  3. adds a _byPath dictionary property to FileProviderDictionary, which only collects the GameFile of highest priority in each path.

I'm quite noob in C#, and want to have some practice here. If there's anything doesn't seem right, or u have some thoughts on the "basic order" above, let me know then I'll try my best. Or if you're planning a better approach, close this I'm willing to learn ;-)

This tool is amazing! Thank you!

@PaperStrike PaperStrike marked this pull request as draft March 25, 2022 03:40
@PaperStrike
Copy link
Author

It seems the reading order only matters on pak files, so I just reverted the changes to IVfsReader, GameFile, and VfsEntry. Now only PakFileReader and FileProviderDictionary are affected.

@PaperStrike PaperStrike marked this pull request as ready for review March 25, 2022 04:37
@PaperStrike
Copy link
Author

This seems to break the content changes to be reflected on the FileProviderDictionary after being passed to AddFiles.

Is it necessary? 👀
If it is, I will remove _byPath and instead check the priority in every TryGetValue call.
If not, it seems better to require the newFiles passed to AddFiles to be a ReadOnlyDictionary instead of IReadOnlyDictionary, right?

@ryantheleach
Copy link

Would these changes allow FModel to still show both copies, and read both separately?

@PaperStrike
Copy link
Author

PaperStrike commented Apr 15, 2022

@ryantheleach No yet. If I've read it right, FModel only passes the path to CUE4Parse, there's no way for CUE4Parse to know which copy FModel wants to load.

This PR only aims to make CUE4Parse provider to provide the file with the highest priority (as of now almost a random one is provided).


But I managed to made things work locally by combining with these changes:

  1. exposing a static int PakFileReader.GetReadOrder(string pakName).
  2. adding a new GameFile GetGameFile(string path, string? pakName = null) to AbstractFileProvider.
    When pakName is not null, it returns the GameFile with the largest order less than or equal to the order of pakName.
  3. adding an optional parameter string? pakName = null to most members of AbstractFileProvider, use GetGameFile whenever applicable.
  4. In FModel, CU4ParseViewModel.Extract, receive AssetItem asset instead of string fullPath, and then use asset.Package as the pakName passed to the members of Provider.

The problem is the speed. Whenever GetGameFile is called with a pakName string, the FileProviderDictionary would be iterate once. In my practice, it takes about extra 3 seconds when I load a material in a game with only 2 pak file.


Another way I can think of, instead of applying the steps above, would be to use multiple providers in FModel. That is, when a game has N paks, G.pak, G_0_P.pak, G_1_P.pak, ..., G_N-2_P.pak, generate N providers:

  1. provider with G.pak,
  2. provider with G.pak and G_0_P.pak,
  3. provider with G.pak, G_0_P.pak, and G_1_P.pak,
    ...

And then in FModel we can read asset.Package to know which provider to use.

The problem now may be the memory. I'm not so familiar with CUE4Parse and FModel, I have no idea how this method can be applied and how the memory usage can be optimize.


Whichever way we choose, the prioritizing steps inside the provider (like this PR) is necessary.

@PaperStrike
Copy link
Author

Oh, the third method, integrating the second method into the GetGameFile in method 1.
Whatever, the implementations are quite hard for me. It all depends on you guys. Just suggesting some possible ways. 👀

@PaperStrike
Copy link
Author

@ryantheleach Here in PaperStrike/FModel@extract-with-pak-name (with PaperStrike/CUE4Parse@pub-pak-order) has the first method implemented. 👀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

File Duplicates from updates only load oldest version available
2 participants