Add support for CHD format #13810
Add support for CHD format #13810
Conversation
|
How this compresses in regards to CSO? ? |
|
You can check linked issue this fixes for compression tests, CSO has more parameters for compression when you use a software like [Unknown's] MAXCSO, if you go for maximum compression, CSO will be similar, often better(not by much), but max compression is pretty unusable for real PSP due to how much processing power it needs to be read, so using maximum compression kind of makes those formats alike aka useless for real hardware. IMO doesn't matter on the format, there's not much reason in saving a bit of space while loosing compatibility, but at least the microstutters from CHD format mentioned here, shouldn't be a big issue since PPSSPP can cache whole game in RAM. |
|
I don't think adding a custom libchdr to ppsspp is a good long term goal. It will just be technical debt when maintaining and updating it is concerned. |
|
Well, I just followed the beetle-psx-libretro example for a C project importing the libchdr, which just extracts the C files to the dir (it doesn't need to be a fork, it's just that the CHD parent support is something I added to it). I used submodules for the PCSX2 support, so it's possible to use it here as well, but CMake is a PITA, so I had lots of trouble with it. Actually, it's still a problem with its CI right now. If someone can guide me to the correct way of using CMake for this case with submodule, I'd appreciate it. Edit: Regarding the microstutters, yes, it only happens once per "load". For example, when playing The 3rd Birthday, when walking towards a character with a voice clip, with a CISO/CHD, there is microstutter once for this voice, but reloading and playing the same voice, it won't happen again. In regards to compression, the biggest advantage is, by far, CHD's parent/clone support. For example, if you had to keep both the original and a translated game, with CISO, you'd need twice the space, but with CHD, it's actually just the delta hunks. 1.4G Aug 14 04:01 'Digimon Adventure (Japan).chd' In the Digimon games, the English patched versions occupy just 76MB and 30M instead of 1.4G and ~700MB. |
Unfortunately libretro is a good example of how to not do things, but my main point that a custom fork will be hard to maintain down the line. I would suggest waiting for upstream libchdr to merge any required support and then point the submodule to the upstream repo. I believe libchdr supports system wide installs now, it would be also nice if this path was supported. Also needing a modified chdman to use this correctly is not something that many people will have access to. Lastly is CHD support really that useful for ppsspp? |
Submodules are a git feature, not a cmake feature. As far as cmake goes, just point it to the path of the submodule. |
|
I do that with PCSX2, and there is a problem with libFLAC's CMakefile (one of libchdr's dependencies) when it comes to "make install", which installs libFLAC's files as well, even when it's statically linked. I just "include_directories" and "add_subdirectory" the libchdr's project root dir with its CMakefile. |
|
Regarding the fork, it doesn't need to wait for my changes to be merged. My changes are strictly for parent/clone CHD support. We can merge without my changes with vanilla upstream libchdr. System-wide libchdr would be very nice, since I wish every emulator used CHD. But I don't see packages for Debian/Ubuntu/AUR, so I think it'd be better to wait. Or unless you want us to spearhead this initiative, which I'm all for, but I can't be a maintainer for the packages, due to lack of time. In terms of usefulness, #10417 (comment) hrydgard himself is convinced it's a win, though not an enormous one.
But with libraries with mods/hacks, this saves a lot of space. I don't think parent/clone support was added in the conversation with hrydgard, so this comment might change with this. Edit: Nevermind, parent/clone support was added to the conversation already. So his comment will stand. |
This doesn't need the modified chdman. That is only needed in case one is trying to create a grandchild CHD. This PR works normally, with CHDs created by current chdman. |
When programs start supporting this it will probably be added, until then package maintainers might not notice. |
|
Hrydgard's post about it being a win in compression was before testing maxcso with better compression CSO. Everything in noticeable favour of CHD format compression was vs common "lvl 9 cso compression" that's still usable(althrough already not that great in some games) on real hardware, we can achieve better compression within existing CSO format, but it makes the files unusable on real hardware which kind of kills the point for many of us especially since the gains aren't big. CHD format soft patches might be cool, but it still means making them yourself and probably not that useful in many games due to common compression and encryption in PSP games resulting in huge file size of the patches. Fan made translations etc. typically care more about real hardware and PSV where format like CHD does not exist, so it'll probably not be widely used unless someone add's support on their platforms as well. I'm not against adding this, just saying it's not the greatest thing ever that everyone should be using as some fans of this format coming from different more "retro" platforms are claiming it to be mostly due to "owning" a full library of pirated games and having a hard time noticing which of them had missing or corrupted files as that's an use case where I imagine this format and it's features were developed for, it's not a must have for emulating more modern systems. |
|
I'm no longer convinced about the value of this. Would need to hear some more really good arguments. And is anyone out there using CHD format dumps for their UMDs anyway? The natural thing is ISO since that's what you'll get when dumping discs the usual way with custom firmware, plus you can run them on your actual PSP, and then using maxcso to get nearly all of the possible compression benefit. I do not think it's a good idea to dump the whole source of the library into the PPSSPP repo either. Submodule of your own fork with fixes would work. |
And what happens when it never gets updated again? You already have a forked ffmpeg submodule that barely builds and is essentially a time bomb waiting to go off. |
|
The comparison with ffmpeg probably isn't fair. Obviously, because its API is brittle and has undocumented breaking changes regularly, keeping up with updates is a pain. And packagers who have poor understanding of how APIs work and what brittle even means in the context of an API try to force incompatible newer versions of ffmpeg into PPSSPP, making things worse. I've seen no indication whether libchdr's API is as poorly maintained, so it'd probably be much easier to update one way or another. -[Unknown] |
This may have been true at one point, but it does not seem nearly as true now. I follow the ffmpeg git master and everything I have installed that needs ffmpeg works with it excluding ppsspp. Have you tried compiling the ffmpeg submodule lately? It does so as one big warning here and its only a matter of time until it entirely breaks after some compiler or system update. The same could be true for a forked libchdr submodule in a few years... |
|
As for the difference in compression between cso and chd, my experience (for what is worth) is that 2/3 of my psp games compressed better with pbp vs cso. |
|
Well, some of the advantages of CHD:
Well, these were the advantages for me off the top of my head. There may be more. |
|
The CHD on PSP it's been so long that I wait for this. |
|
To quote user hatsunemilku from Reddit, here are some good reasons to adopt CHD: "Standardization: MAME already use this format as it is really friendly for archiving and more projects are adopting it. The more it gets adopted, the easier it is to maintain the archives without the headache of wondering where to get the software to decompress said archive. Ease of use: because tools like CHDMAN are so easy and simple to use, archiving and decompressing a file becomes foolproof. Future proofing: since it is a format tightly related to the emulation scene, the tools and the format can be adapted and improved in a way that can truly answer to the scene’s need. While 7-zip is really good and all, its not a piece of software made only for this case. The scene would need to rely on pushing updates to 7-zip and that may take time in the best of cases or simply not added at worst." |
|
The funny thing is you can sub in CSO for those and it works too. It would be no harder to add zstd to CSO than to CHD (both would require a pull request and simple tooling update.) CSO support is possibly more tied to the emulation scene and just as easy to push updates to. And there are a wealth of tools and CSO implementations, more than CHD as it is a documented format. Anyway, libchdr only supports local files, unlike all our other formats; remote disc streaming probably isn't used by every single user, but it's not as if CHD is without downsides. -[Unknown] |
|
True, but CSO isn't supported by PS1 emulators, for which there is no alternative but CHD. I suppose it's not a big deal if you don't care about having some kind of common standard for PlayStation-type ROMs, but it would be very helpful for users not have to use multiple types of tools for different systems. Or perhaps I should be asking stenzek to support CSO in DuckStation? |
|
@extherian Don't use CSO for PS1 discs. Also, I think CHD support will come eventually but right now you'll just have to use chdman and maxcso for your emulation needs. |
|
PCSX2 has now added support for CHD. Is there any possibility of PPSSPP supporting this one file format for all PlayStation emulators? Or is it still a case of 'not invented here'? |
|
WOAW i discover this issue and yes, i stand for this feature too :) It sill save us a lot of disk space. |
|
It's not a case of NIH, it's a case of extremely tiny benefit while adding a large chunk of code that most users will never have any use for. CHD is not commonly used for PSP games. |
|
I'm not sure what you mean by CHD not being commonly used for PSP games. Clearly it isn't since PPSSPP doesn't support it, but you could change if you merged support for it. And if most users would have no use for it, then I'm guessing that most people don't compress their PSP games at all, then? That is interesting. I thought that CHD would make a good replacement for CSO as a new standard, but if CSO isn't used much either then I can see why you wouldn't want to add it. |
|
Just do everything? Have CSO, CHD, a new decent documentation on formats and compression and a possibly new format thats focused on maximum compression for archive usage specifically for PSP. I mean whats the big matter someone already did the work and put the time in and you just toss it in the bin. In the end its up to the User to decide which format he uses and if he's fine with decompress loading times. Not to mention that CHD support would mean a reason for many people to use the standalone included in frontends instead of retroarch i mean thats a plus for you guys right? |
But, the work wasn't done? As you can see from the Travis checks, it was breaking the build.
On Google Play, PPSSPP has over 100 million installs, and RetroArch has 1 million. I don't really see evidence of the assumption that there's a legion of RetroArch users who would use PPSSPP if only it supported an obscure file format. It seems to be the assertion of a vocal minority who would probably keep using RetroArch. And Libretro's API limitations being designed for weaker systems (even the PS1 is 10% as powerful as the PSP) already means PPSSPP standalone is better. People using RetroArch for PSP emulation are using it even though it's buggier, slower, and crashes more. The 1% of installs RetroArch has on Google Play compared to PPSSPP doesn't make me think a huge number of people are waiting to convert their files to an obscure format that's not even really smaller so they can finally use standalone. -[Unknown] |
|
As someone who has experimented with CSO vs CHD. When it comes to emulation CHD wins hands down. It offers equal to better compression at a fraction of the time and resources to do and stores the hash value for people who audit their stuff with stuff like ReDump. As far as CSO being compatible with PSP, implementing CHD doesn't mean removing CSO support, and PSP is discontinued making that matter less and less by the year as well as they get harder to find and if someone is playing on an emulator, they aren't typically playing it on their console. With the excuse that people don't store their PSP stuff in CHD format, that is likely a chicken and egg scenario due to the fact that PPSSPP is the only PSP emulator worth using and works great and it doesn't support CHD which means there is nothing that plays PSP that can use it at this point. If PPSSPP started supporting CHD, you can bet within a few months most places would start archiving in CHD format as it's size is equal or smaller with lower resources and easier to audit. Just look at RVZ format on Dolphin, it isn't backwards compatible with the Wii or Gamecube hardware but works really well for that purpose. If it didn't do something that CHD did not, then they likely would have just used CHD as well. CSO by comparison functionally offers zero benefits when it comes to emulation and whose only saving grace is its compatibility with the PSP. So, when it comes to people who really only use their computer to store games to trade on and off their system, CSO is the better option, when it comes to those who play their games on the emulator CHD is the better option. And I tested compressing the same ISO with MaxCSO using both the default and max compression settings and CHD beat it with both smaller size and quicker compression and the ability to audit if I wanted without having to decompress it. |
didn't CSO cause problems? Weren't other formats even created specifically to address them? Wouldn't CHD technically load faster as
Anyway from what I gather people want CHD but developers don't want to implement it which makes sense to me. If there was an easy feature request tracker I'd pay for continued CHD support though. |
|
Again and again, I am not really opposed to a CHD implementation, if someone makes a nice, fully working implementation and promises to help maintain it. It's just not really anywhere close to the top of my list of priorities, I prioritize getting games to work and render correctly and with high performance. Storage is solved well-enough by CSO for most purposes. |
|
Oh I see, and you are an actual developer of this, yeah that makes sense. I would agree with your choices as the only reason I continue to use this emulator is I feel it does just that. CSO->CHD should be almost at the bottom of the totem pole although probably not that hard to implement. Anyway keep up the good work we really appreciate it. |
|
The biggest disadvantage of CHD is that it does not support more efficient zstd compression. |
|
Well, it'd be fairly trivial to add zstd to CSO, and PPSSPP already has zstd built in (without any ancient or duplicate libs, and would fold neatly into our remote disc streaming, CRC, etc. support.) I think I did a test run just to see how well it compressed blocks. That said, in theory one could do smarter things with zstd, such as seeing blocks with some uncompressible blocks or something (in CSO or CHD.) Nice to see a real comment about CHD (that link) which addresses some of its challenges while pointing out some of its benefits. Maybe it's finally getting somewhere useful as a format, instead of the hypeware people made unfounded claims about before. -[Unknown] |
Just like how mkv sucks and is only used by a few small anime fansubbers? One of the worst comments I've ever read. You and hrydgard sound like you're funded by big CSO. |
|
The main advantage of CHD is that this is a standard in the emulation scene for cd-based games. And there are many tools for CHD. CSO is a niche for PSP, but not used anywhere else. Since emulation is a great deal about preservation, this would be an important step. It makes me sad that this is closed for those reasons. |
|
The main disadvantage of CHD is that libchdr is a mashup of existing libs with a new container format. We already have FLAC, and we have compression algorithms (zstd) which are better overall for the applications where CHD is used than LZMA. Since few distros package it, it also ends up being vendored with all the caveats that brings. I don't see any compelling reason for it to exist, when zstd exists and for applications where there is redbook audio, we already had FLAC widely available. What we have done as a community is reinvent a wheel needlessly. |
|
Here are some benchmark on 007 - The Russia with Love There is a reason why chd is popular as a format!
iso - 698 MB |
|
I don't understand this at all. Someone did the work to get CHD working with this project. It works alongside other compression formats like CSO just fine. What's the issue causing conflict that this not only didn't get merged, but closed? CHD has specific use cases, such as better compression on certain games and compatibility with RetroAchivements. It comes across that there's a lot of motivated reasoning for arguments against. |
|
Nickkeane, I strongly support CHD as well but my understanding from the developer is that he isn't against it, he just doesn't want to be the one to maintain that part of it and if anyone else wanted to handle that he would be up for it. Also, disregard Lunamoo, they have some irrational hate boner for CHD and it shows every time this topic comes up. |
|
The only hate booner I have is for missinformation which among this format and anything retro-crap related goes on a religious level, like really posting a screenshot of "great benefits" under which re-typing the same data but reducing chd by 200 mb is just one of the many bs here yay! Plus all those great examples aren't even posted for PSP games which on it's own commonly are encrypted and compressed and not give that much of a difference from another layer of compression, but on some completely different platform like nintendo. Compression we have thanks to maxCSO or even PBP - which neither is ever used by people trying to enforce chd agenda, but fully usable in PPSSPP is comparable to chd sometimes going slightly worse, sometimes slightly better and does not require adding tons of new dependencies to this project so as a matter of fact it's main selling point is D-E-A-D. Retro-crap and mame related features of CHD are completely useless for PPSSPP or PSP emulation in general so yeah there's no reason adding tons of hard to maintain code and no amount of missinformation will ever change that. |
|
LunaMoo, me and you have gone around on this in the past. You literally tried to use the fact it is used by mame is say it was too associated with piracy to use. I noticed the number being off on his, but remember when I did the tests myself using 2 different games with you. And I have given examples with PSP games using both CHD and MaxCSO using the placebo level compression and CHD still beat it while using a fraction of the time to do so. You do have some irrational hate boner towards the format for some reason. Hrydgard's response saying it just isn't a priority given the other stuff and him not wanting to maintain it is a legitimate reason. You saying it was associated with piracy because of mame is not a legitimate reason. You claiming that CSO is just as good or better is provably false. And sorry, the reasons for asking for CHD isn't dead because of CSO as CHD matches and beats it in compression level with a fraction of the resources to do it and is still active. PPSSPP is the only thing giving CSO any relevancy at this point. |
That is an exceptionally rude way to put it, the problems with CHD have already been explained in this thread and your willing dismissal of the points doesn't make them magically go away. |
|
Orbea, I have had this go around with him before, this is the most accurate way to describe the exchanges. The fact is CHD meets or beats CSO in compression while using a fraction of the resources to do so, it is still being actively maintained and is being actively used by almost every other disc based emulator except for Nintendo which has their own format that is better for their discs and Xbox. His responses of it being about piracy is not really an issue as CSO and ISO could also be claimed the same. His claim that CSO is good enough is doable for now but it isn't better and is only relevant because of PPSSPP a this point. Nothing I say will make his points magically go away because the functional uses of both CSO And CHD and the continuing uses of both formats already do that for us. Hrydgard's response in the previous exchanges was the perfect answer when he said he has no issues adding it later but he didn't want to be the one to maintain that part given all the other stuff he was working on. It was completely reasonable. LunaMoo's was not. No offense to you Orbea, I was just going by mine and his previous exchanges on it in the past where he made it seem pretty apparent that he had an issue with the format or a major love of CSO. I have no issues with being stuck using CSO overall but it isn't the superior format at all with its ONLY advantages over CHD being the fact the PSP can run it natively which is becoming less and less of a selling point for the format as the systems die out and zero selling point to people who are playing their games exclusively on their PC. |
|
@LunaMoo Has not been the only person that has voiced concerns. Regardless even if you disagree with him I very much appreciate it when you voice it in a constructive manner. I believe @carmiker summed it up well. For me personally the fundamental issues are the technical debt and vendored code. |
|
I was responding how I did as how he was on here where even hrydgard told him to chill and even reopened it after Lunamoo had closed it. Having concerns and issues with something is one thing, but he acted like it was something personal to the point he called Mame "The King of Piracy" and didn't want to be related to it as part of his reasons for not wanting to adopt it. Using CSO isn't a huge issue, saying you don't want to deal with the increased code base and if anyone else wants to manage that part they can volunteer as Hrydgard said is completely understandable. But the route LunaMoo has gone on this singular topic goes beyond and seems like something personal to him. |
|
Finding dedicated and effective maintainers is not easy. Look at libretro support in ppsspp, there are ~40 open issues including some major usability bugs which I have not seen anyone spend much time investigating. And vendoring thousands of lines of a foreign code base needlessly bloats up a repo and I suppose most people would not be happy with requiring a system version of libchdr? Its been said before, but CHD doesn't solve problems that haven't already been solved in better already supported ways. I don't intend to be mean, but trying to force CHD support in every project is tiresome and not at all productive. |
|
It's no problem Orbea, I can understand the reasoning given all the work involved. While many (probably most) see CHD as the better option, it isn't a big enough issue to make or break anything, it isn't like WUA or RVZ levels of difference or anything. My jab wasn't directed over it not being adopted at all so much as it was how personally he seems to take this one. Him just saying, "The benefits of providing support for the format isn't great enough to take our attention away from what we are already working on, if anyone wants to volunteer to provide and maintain it, we will gladly accept the help but we have enough on our plate already" would be a great answer to such suggestions. Either I am getting mixed up but I could have sworn that I saw a recent post from him before mine which was why I brought him up to begin with. |
It's been said before, but no other format supports softpatching. Of course you can say that this is not an enourmous advantage, that people can still hardpatch. It would be one thing to say that this format doesn't have enough benefits for your use case. But saying that it wouldn't offer any benefits at all would be dishonest. Specially if you want to avoid misinformation. |
Soft patching should be a frontend feature, not a feature in the rom format. Do one thing and do it well. |
|
We are talking about frontend features, CHD support is a frontend feature too. Even if you choose to implement soft patching differently you still need a format for the patch, soft patching compressed data wouldn't work very well, so you'd be opening a whole new can of worms. The way CHD is designed, with the hunks of data being compressed separatelly, makes it very efficient for this since you can work with compressed data without having to decompress anything that is being replaced by the patch.
|
|
Soon it's gonna reach a level where implementing and maintaining CHD support is less work than reading this thread :P |
|
I've been following this thread myself and have been wondering what is even going on in here. I've considered posting this before but didn't know if it would be relevant or not, as it sounds like maintenance is the biggest block for CHD being added. I guess I'll add my two cents anyway. In all honesty I found out that CSO is a genuine problem if you don't have a powerful enough pc. I'm not talking about being powerful enough to play, but rather being powerful enough to convert to CSO. Because (in what was a very unexpected surprise to me) for some reason converting to CSO is so much more intensive than CHD that my pc was getting too hot, and I had to stop converting for fear of the heat possibly causing damage. Unfortunately I didn't think to make a note of specific numbers. But never have I seen converting to CHD cause the pc to get so hot as converting to CSO did. I also tested converting some PSP isos to CHD, even though the CHDs would be useless, just to see if there was something about converting PSP isos themselves that caused excessive heating. However converting PSP isos to CHD did not cause the abnormally high temperatures that converting to CSO was getting to. I doubt this will be a factor in whether CHD gets added, because of maintenance being the big issue for adding it, but maybe the information is of use anyway. I still don't understand why converting to CSO is so much more intense than CHD that it requires more powerful hardware to safely do it. |
|
It's not really, the default parameters of maxcso just use multiple compression algorithms in a similar way to pngcrush. That said, if this makes your device unstable, you probably need better cooling. Anyway, you can use If your device is unstable when using all threads, you can pass -[Unknown] |
A few caveats in this pull request.
Original -> Child -> Grandchild
or even:
Disc 1 -> Mod of Disc 1
\-> Disc 2 -> Mod of Disc 2
For grandchildren, a small modification is needed for chdman. It currently only loads one child via command line argument. mamedev/mame#7594
It was mostly tested with commit 3403e28 on a x86_64 Arch Linux machine and Android 10. Rebasing and compiling on latest master worked great, and no problems detected from the rebase.
One small issue with CHD with CDLZ is decompression latency, as it widens micro stutters compared to CDZL/ciso.
I'm not sure if it's related to this PR, but I can't seem to load a savestate from Linux to Android. Saves work OK, just not savestates.
Sorry, this was done in July, but I hadn't had time to clean this for a pull request. Hope it's a good Christmas gift to all.
Edit: Fixes #10417