Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Option to prevent sets with excess files when raw-copying #941

Open
PhasecoreX opened this issue Feb 20, 2024 · 7 comments · Fixed by #1034
Open

Option to prevent sets with excess files when raw-copying #941

PhasecoreX opened this issue Feb 20, 2024 · 7 comments · Fixed by #1034
Assignees
Labels
enhancement New feature or request

Comments

@PhasecoreX
Copy link
Contributor

Is your feature request related to a problem?

I'm sorry, this is probably the last request!

If I have a folder with this fbneo file in it:

dmnfrnt.zip

And I run igir like this:

npx --yes igir@latest move --dat "https://raw.githubusercontent.com/libretro/FBNeo/master/dats/FinalBurn%20Neo%20(ClrMame%20Pro%20XML%2C%20Arcade%20only).dat" --input . --output . --no-bios --no-device

I get an extra file created:

dmnfrnt.zip
dmnfrntpcba.zip

I don't want that extra one being made. I tried using the --single argument, which does prevent the new one being created, but in my actual directory of roms it deletes some of the clones that I actually want to keep around with their parents.

Describe the solution you'd like

Either some way to specify roms to ignore in the --single logic, or have a command line argument to disallow creating new roms. Moving/renaming and copying can still happen, but new ones can't be built from existing ones.

Additional context

My initial solve for this was the "don't create new roms" thing, and had a whole feature request made for it, until I discovered the --single argument. Using that on my test above, it fixed it, so canceled the feature request before submitting it. After I discovered it messed with some of my roms, I rewrote this.

I'm not sure how "don't create new roms" would work with multiple input sources though. If the input is the same as output, then only move/renames would be allowed, and thus no creations of new roms could occur. But if the input isn't the output, I guess the move or copy would go to whatever rom matches the best?

@PhasecoreX PhasecoreX added the enhancement New feature or request label Feb 20, 2024
@emmercm
Copy link
Owner

emmercm commented Feb 20, 2024

I think there might be some confusion with how arcade ROM sets are merged or split. igir uses a "full, non-merged" strategy by default.

To help diagnose, could you provide the following:

  • The contents of dmnfrnt.zip in your input folder, including checksums & sizes (i.e. unzip -v dmnfrnt.zip)
  • The contents of dmnfrnt.zip in your output folder, including checksums & sizes
  • The contents of dmnfrntpcba.zip in your output folder, including checksums & sizes

@PhasecoreX
Copy link
Contributor Author

Originally when I didn't specify the zip command, igir just copied the zip over, exact same size and contents. I've now run the above command with zip (which is how I will normally be running my main command) and I can see it actually build the zip file.

npx --yes igir@latest move zip --dat "https://raw.githubusercontent.com/libretro/FBNeo/master/dats/FinalBurn%20Neo%20(ClrMame%20Pro%20XML%2C%20Arcade%20only).dat" --input . --output . --no-bios --no-device -vvv

Running the above command with -vvv gives this log output: igir.log

Here are the two files:

Archive:  dmnfrnt.zip
TORRENTZIPPED-34A4E7A4
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
  131072  Defl:X    47083  64% 1996-12-24 23:32 517cf7a2  bios.u42
  524288  Defl:X    48201  91% 1996-12-24 23:32 b3cc5c8f  ddp3_bios.u37
 8388608  Defl:X  2762972  67% 1996-12-24 23:32 9741bea6  igs_a04501w064.u3
 8388608  Defl:X  2573370  69% 1996-12-24 23:32 e104f405  igs_a04502w064.u4
 8388608  Defl:X  2357179  72% 1996-12-24 23:32 bfd5cfe3  igs_a04503w064.u6
 8388608  Defl:X  1782126  79% 1996-12-24 23:32 29320b7d  igs_b04501w064.u9
 2097152  Defl:X   382178  82% 1996-12-24 23:32 578c00e9  igs_b04502w016.u11
 8388608  Defl:X  2824182  66% 1996-12-24 23:32 900eaaac  igs_t04501w064.u29
 8388608  Defl:X  6363074  24% 1996-12-24 23:32 3ab58137  igs_w04501b064.u5
 2097152  Defl:X  1549427  26% 1996-12-24 23:32 45ae7159  pgm_m01s.rom
  131072  Defl:X    47017  64% 1996-12-24 23:32 e42b166e  pgm_p01s.u20
  131072  Defl:X    47325  64% 1996-12-24 23:32 78c15fa2  pgm_p02s.u20
 2097152  Defl:X   450423  79% 1996-12-24 23:32 1a7123a0  pgm_t01s.rom
 2097152  Defl:X   316171  85% 1996-12-24 23:32 bda083bd  v105_16m.u5
 4194304  Defl:X  3538586  16% 1996-12-24 23:32 c798c2ef  v105_32m.u26
--------          -------  ---                            -------
63832064         25089314  61%                            15 files
Archive:  dmnfrntpcba.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
 2097152  Defl:N   321408  85% 2024-02-20 22:16 bda083bd  v105_u43.u43
 8388608  Defl:N  2830642  66% 2024-02-20 22:16 900eaaac  igs_t04501w064.u71
 8388608  Defl:N  2779895  67% 2024-02-20 22:16 9741bea6  igs_a04501w064.u30
 8388608  Defl:N  2590158  69% 2024-02-20 22:16 e104f405  igs_a04502w064.u31
 8388608  Defl:N  2375928  72% 2024-02-20 22:16 bfd5cfe3  igs_a04503w064.u32
 8388608  Defl:N  1785762  79% 2024-02-20 22:16 29320b7d  igs_b04501w064.u40
 2097152  Defl:N   384296  82% 2024-02-20 22:16 578c00e9  igs_b04502w016.u41
 8388608  Defl:N  6418226  24% 2024-02-20 22:16 3ab58137  igs_w04501b064.u8
 4194304  Defl:N  3547859  15% 2024-02-20 22:16 c798c2ef  v105_u62.u62
 2097152  Defl:N   452590  78% 2024-02-20 22:16 1a7123a0  pgm_t01s.rom
 2097152  Defl:N  1571781  25% 2024-02-20 22:16 45ae7159  pgm_m01s.rom
  131072  Defl:N    47895  64% 2024-02-20 22:16 78c15fa2  pgm_p02s.u42
--------          -------  ---                            -------
63045632         25106440  60%                            12 files

The input and output directory is the same, so the contents of dmnfrnt.zip in my input and output folder are the same file (it isn't modified). The clone dmnfrntpcba's entire contents are a subset of dmnfrnt; it has no extra roms that dmnfrnt doesn't have, they're just renamed.

Here's that part of the dat file if you want it:

	<game name="dmnfrnt" romof="pgm" sourcefile="pgm/d_pgm.cpp">
		<description>Demon Front (M105XX, S105XX)</description>
		<year>2002</year>
		<manufacturer>IGS</manufacturer>
		<rom name="v105_16m.u5" size="2097152" crc="bda083bd"/>
		<rom name="igs_t04501w064.u29" size="8388608" crc="900eaaac"/>
		<rom name="igs_a04501w064.u3" size="8388608" crc="9741bea6"/>
		<rom name="igs_a04502w064.u4" size="8388608" crc="e104f405"/>
		<rom name="igs_a04503w064.u6" size="8388608" crc="bfd5cfe3"/>
		<rom name="igs_b04501w064.u9" size="8388608" crc="29320b7d"/>
		<rom name="igs_b04502w016.u11" size="2097152" crc="578c00e9"/>
		<rom name="igs_w04501b064.u5" size="8388608" crc="3ab58137"/>
		<rom name="dmnfrnt_igs027a.bin" size="16384" status="nodump"/>
		<rom name="v105_32m.u26" size="4194304" crc="c798c2ef"/>
		<rom name="pgm_t01s.rom" merge="pgm_t01s.rom" size="2097152" crc="1a7123a0"/>
		<rom name="pgm_m01s.rom" merge="pgm_m01s.rom" size="2097152" crc="45ae7159"/>
		<rom name="pgm_p01s.u20" merge="pgm_p01s.u20" size="131072" crc="e42b166e"/>
		<rom name="pgm_p02s.u20" merge="pgm_p02s.u20" size="131072" crc="78c15fa2"/>
		<rom name="ddp3_bios.u37" merge="ddp3_bios.u37" size="524288" crc="b3cc5c8f"/>
		<rom name="bios.u42" merge="bios.u42" size="131072" crc="517cf7a2"/>
		<video type="raster" orientation="horizontal" width="448" height="224" aspectx="4" aspecty="3"/>
		<driver status="good"/>
	</game>
	<game name="dmnfrntpcba" cloneof="dmnfrnt" romof="dmnfrnt" sourcefile="pgm/d_pgm.cpp">
		<description>Demon Front (VM105XX, S105XX, China, Single PCB Version)</description>
		<year>2002</year>
		<manufacturer>IGS</manufacturer>
		<rom name="v105_u43.u43" merge="v105_16m.u5" size="2097152" crc="bda083bd"/>
		<rom name="igs_t04501w064.u71" merge="igs_t04501w064.u29" size="8388608" crc="900eaaac"/>
		<rom name="igs_a04501w064.u30" merge="igs_a04501w064.u3" size="8388608" crc="9741bea6"/>
		<rom name="igs_a04502w064.u31" merge="igs_a04502w064.u4" size="8388608" crc="e104f405"/>
		<rom name="igs_a04503w064.u32" merge="igs_a04503w064.u6" size="8388608" crc="bfd5cfe3"/>
		<rom name="igs_b04501w064.u40" merge="igs_b04501w064.u9" size="8388608" crc="29320b7d"/>
		<rom name="igs_b04502w016.u41" merge="igs_b04502w016.u11" size="2097152" crc="578c00e9"/>
		<rom name="igs_w04501b064.u8" merge="igs_w04501b064.u5" size="8388608" crc="3ab58137"/>
		<rom name="dmnfrnt_igs027a.bin" size="16384" status="nodump"/>
		<rom name="v105_u62.u62" merge="v105_32m.u26" size="4194304" crc="c798c2ef"/>
		<rom name="pgm_t01s.rom" merge="pgm_t01s.rom" size="2097152" crc="1a7123a0"/>
		<rom name="pgm_m01s.rom" merge="pgm_m01s.rom" size="2097152" crc="45ae7159"/>
		<rom name="pgm_p02s.u42" merge="pgm_p02s.u20" size="131072" crc="78c15fa2"/>
		<video type="raster" orientation="horizontal" width="448" height="224" aspectx="4" aspecty="3"/>
		<driver status="good"/>
	</game>

That is the order they are in the file as well.

Also, I get these warnings on the command output:

✓ Scanning for DATs ·········· | 1 unique DAT found
✓ Scanning for ROMs ·········· | 15 files found
✓ Arcade ····················· | 2/7,578 games, 2/7,578 retail releases written
WARN:  dmnfrnt.zip: not deleting moved file, 3 archive entries were unmatched:
WARN:    dmnfrnt.zip|bios.u42,
WARN:    dmnfrnt.zip|ddp3_bios.u37,
WARN:    dmnfrnt.zip|pgm_p01s.u20
✓ Deleting moved files ······· | 0 moved files deleted

If I run the exact same command again, it leaves the two files alone, no warnings or anything. That's expected though.

I find it weird that igir will copy the zip file over if you don't specify the zip command, keeping the extra 3 files in the resulting dmnfrntpcba.zip, when technically that's not the correct full non-merged result.

@emmercm
Copy link
Owner

emmercm commented Mar 3, 2024

The clone dmnfrntpcba's entire contents are a subset of dmnfrnt; it has no extra roms that dmnfrnt doesn't have, they're just renamed.

Interesting, I didn't know that there were clones that did this. igir is greedy in that it will write every game in DATs that it finds ROMs for. I'll try to reproduce to see why --single didn't work here.

I find it weird that igir will copy the zip file over if you don't specify the zip command, keeping the extra 3 files in the resulting dmnfrntpcba.zip, when technically that's not the correct full non-merged result.

copy and move on their own are guaranteed to not modify the input file's contents - it should be the exact file, byte for byte. Omitting zip or extract is most helpful when you're dealing with 7z input files and want to leave them as-is. You're right that the resulting dmnfrntpcba.zip isn't 100% accurate in that it has no excess files, but it should work in FBNeo since it's full non-merged.

@emmercm
Copy link
Owner

emmercm commented Mar 3, 2024

the --single argument. Using that on my test above, it fixed it

Ignore my other comment, I misread your comment.

@emmercm
Copy link
Owner

emmercm commented Mar 3, 2024

I'm thinking a new igir rename command may be a good option, rather than igir move --exact-matches or some similar option name. Or some kind of igir move --single --no-archive-excess might solve some of your surprise here.

I'll have to think about it because I'm worried igir move and igir rename wouldn't have an obvious distinction.

@PhasecoreX
Copy link
Contributor Author

You're right that the resulting dmnfrntpcba.zip isn't 100% accurate in that it has no excess files, but it should work in FBNeo since it's full non-merged.

Ah, yes that makes sense.

I'll have to think about it because I'm worried igir move and igir rename wouldn't have an obvious distinction.

Yeah I think that would be confusing, and at least the way I use it (and I think you do as well), I expect move to reorganize my collection (input and output are the same directory, basically a rename) as well as move in new files from additional paths at the same time. I think I saw that in your real world use script.

I guess there's two ways of looking at it, either you want to complete the collection, or you want to make sure your current collection is correctly organized. In the former, you want your source of roms (itself and any add on paths) to all come together and make as many complete sets as possible. In the latter, you'd want each existing file to be matched up to the closest matching DAT entry, and updated/renamed to match. By default igir has the former mindset, and --single almost covers the latter one, just that if your collection contains a parent and a clone archive, the clone will always be removed.

So maybe there just needs to be an argument that turns off greedy mode, where igir will evaluate each archive in its current state and see which DAT entry most closely matches it, and then goes with that. Not sure how complicated that would be though.

@emmercm
Copy link
Owner

emmercm commented Mar 8, 2024

I'll have to think about the naming of it, but an option to complement --allow-incomplete-sets could make sense. Maybe something to the effect of --no-superfluous-sets.

@emmercm emmercm changed the title 1G1R Exceptions, or Option to Not Create New Roms Option to prevent sets with excess files when raw-copying Mar 21, 2024
@emmercm emmercm self-assigned this May 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants