random stuff pastebin
A pastebin of sorts.
This was intended to improve the accuracy of parsing when trying to repair corrupt data. The concern was spending a lot of computation reading data from irrelevant files. I want to be able to reject files as quickly as possible. I also didn't want to rely on specific file extensions, as that "whitelist" approach is lazy and requires adding multiple new extensions as they appear.
This whole issue stems from the fact that my checkHeader function is too strict. In rare cases, the "ID" of a Controller Pak becomes corrupt (meaning, its checksum, and the backup checksums, become wrong). When this occurs, there is no reliable way to automatically detect and repair it, without inferring that the rest if the filesystem is still in tact.
And the simplest solution is to just let it fail, and continue to parse the filesystem anyway. But considering how verbose my main parser is, I wasn't comfortable allowing .ZIP files to get that far, I wanted a relatively quick and efficient way to reject files before it got that far.
I came up with a simplified version of the filesystem parser, which only checks whether its basic state is legal. I was pretty happy with this, it seemed short and simple enough.
But then I said what if, and wondered if I could reliably reject files even before the filesystem needs to be parsed. This was the result.
The idea was, I would exploit some facts about how the ID blocks work, mainly the fact that there's intended to be 4 identical copies of the data. I then looked into strengthening it by involving an important byte value. Basically the bank size is stored here, and as to be exactly 0x01, or things blow up.
It was imperfect, as the fact that I'm trying to recover corrupt data means there might be corruptions in this data, and there are. So I decided to make a 'score' system, which gives a confidence level of how similar they are. But again, I found this to be not quite reliable, so I had to add in further checks.
Ultimately I was left with too much code, and not enough assurances. It's a nice function, but its liable to reject a file that can still be parsed by the checking the indextable.
Also the fact that it technically makes the code longer to run, which is exactly what I'm trying to do, and the Index checker itself, is much more a reliable indicator.
const parityCheck = function(data) {
let zum = 0, sum = 0, bank = true;
if(data[0x20+26] != 0x01 ||
data[0x60+26] != 0x01 ||
data[0x80+26] != 0x01 ||
data[0xC0+26] != 0x01) {
bank = false; console.log("BAD");
}
for(let i = 0; i < 32; i++) {
let a = data[0x20+i], b = data[0x60+i],
c = data[0x80+i], d = data[0xC0+i];
zum += a+b+c+d;
switch (true) {
case (a==b && b==c && c==d):
sum++; break;
case (a==b && b==c || a==c && c==d ||
b==c && c==d || a==b && b==d):
sum+=0.75; break;
default:
break;
}
}
console.log(sum, zum);
// if all bytes were zero, we can skip
if(zum === 0) return false;
// some valid files have invalid bank value, but usually sum=32
else if(bank == false && sum == 32) return true;
// some files have a valid bank value, but sum >=29
else if(bank == true && sum >= 29) return true;
return false;
}Update: Rather than rework checkIndexes, I decided to make a new function, checkIndexesFast, which only does the bare minimum required. While this does mean that we spend more time checking if a file is invalid, I think the tradeoff is fair. It's a relatively quick function that doesn't do any expensive operations. It also only kicks in if the header is invalid. And it ultimately gives us the flexibility to confidently load files with a corrupt header, just like N64 hardware can do.
What remains to be done, is the actual ID repair operation, which is a whole other bag of worms.
More insight to this issue might be obtained by a full reveng of the mempak lib.
Two users have shown examples of MPK files which do not load in MPKEdit, yet are able to repair the data through a real game. This happens because MPKEdit rejects any MPK files which fail all four checksums, while the mempak library code prompts the user to attempt a repair, with a warning to indicate that data may be lost.
MPKEdit aims to do everything as correctly as possible, and so blindly taking a risk like that would be against my design philosophy. Not to mention, I don't want to concern the user with making such a choice unless absolutely necessary. There are already parts of MPKEdit that silently repair minor issues; the user doesn't need to be concerned about operations that are necessary and 100% risk-free.
In any case, this issue appears relatively common, and it's not even the first time I've heard of a corrupt ID block getting in the way of valid data. So it looks like this issue should be fixed.
The question then becomes, how can I detect this situation, and silently repair the ID block? It can't be too complicated or deeply nested in parsing logic; this is still the initial line of defense; and it has to be rock solid, as we don't want to let any invalid files in. So what are our options?
I previously considered the parity of the ID blocks, but this doesn't appear universally reliable. The solution would have to be 100% reliable. I do have one possibility:
The IndexTable's structure is specific enough that it follows certain rules, that we may be able to easily exploit here.
All the 00s that are highlighted green must be zero. The adjacent values must be either 1, 3 or 5-127. And because there's a backup of the IndexTable, we can do this test twice. Testing for the existence of 246 zero bytes alone, is potentially a useful sanity check.
Although there is one buggy DexDrive file that will fail this (1 out of ~500, so it's quite rare), so I'd have to look into that as well. Not to mention, if higher capacity MPK files ever catch on, these 'zero bytes' stop being zero. The rule is only possible when all MPK files are 32 KiB, but there's a hidden unused feature that breaks this rule, sadly.
OK so, it's not perfect, but it could be just a few extra lines..
These are notes for the refactor. In general, I'm going to be reviewing every line of code, checking for any errors, or ways to optimize and improve the code's structure and readability. I need to add comments badly, so I will be commenting the code heavily.
Debugging is a mess right now. I want to unify all debugging console.log calls, categorize them, and let the debug output be configured in the settings (Log level).
Also, I'd like to look into a more cleaner solution than curfile in parser.js. It's only used for debug logs, and it's been repurposed for detecting whether the current parsed file is a newly opened file. I'd like to find a better solution for both.
Also, It would be nice to be able to disable debug logs when parsing an already-opened file (i.e. deleting notes, adding notes).
Look into the status with DnD overlays. There is a bug in Firefox still. Is there any new method for this?
Pixicon is still technically not finished and the updated version differs from whatever version I committed last. It's also minified. I need to finalize pixicon, optimize it, and implement the final version to MPKEdit.
The modals (popup windows) need to be cleaned up visually, and there are additional features planned for these. I had already wanted to refactor MPKEdit before implementing modals, so I may have been careless when adding the code - will need attention. Making it look pretty and adding features should be done later, when its implemented properly.
I haven't tested the Chrome App since many updates. I should test and make sure everything's working and visually acceptable. The Chrome App code will eventually need to be ported over to NW.js or something.
Such as fsys.js, codedb.js, FileSaver.min.js...
ES6 now has import and export... I should look into its suitability for updating my current module pattern. Perhaps consider other patterns as well. CommonJS is a big name for this.
See disasm.
The goal of this project was always to understand as much as possible how the MPK filesystem works. With the little information available, I had to fill in many gaps of knowledge. MPKEdit is the result. As it is now, it works very reliably, even being able to recalculate header checksum. But it lacks some repair functions that libultra provides, such as being able to salvage correct data from a corrupted file.
There are other parts that I need to reverse engineer to figure out:
- IndexTable has some empty data before the first
00 03marker. Some DexDrive files strangely stored data in there. I need to figure out if it's ever actually used or not. - Label area and checksum blocks - It would be nice to see where the
0x81comes from, whereFF FF FF FFand the random pakId comes from, as well as any configuration bits in there. Confirmation of whether the label area can be safely written to would be helpful. - NoteEntry table - The strange
0x02bit that is required, and all the other bits that vary wildly. Also of interest is the specific handling of the three null bytes. - The repair functions. I would like to implement my own analogues of libultra's repair functions, perhaps surpassing its ability.
This will require a lot of work, and is a project unto itself. I will be having to use a debugger, setting breakpoints, isolating entire routines in MIPS assembly, and analyzing them. Might have to actually learn MIPS this time and even assemble ROMs. Would also help if I go through all n64 developer files, specifically getting the libultra binary to analyze. I should probably do the Mario XOR thing first.
TODO:
Look through app.js, state.js etc. Find shit to fix.
Clean up JS files.
Do thourough testing before moving on.
Logging is important. Do the console log shit. Separate the GUI, and clean it
up. See what can be done for extensions.
. A visualization of the INODE data, to gain an understanding of
how its doing its thing.
`````````````````. Disable prototype shit, use normal objects.
`````````````````. Move saveAS to its own file.
`````````````````. Rename js files!
`````````````````. Rename State.insert to State.insertNote ?
------------------------------------------------
. Rename o to ofs or offset?
. Rename origin to reorderOrigin? or dragOrigin?
------------------------------------------------
. Hold SHIFT to re-order notes. Allow dragging notes outside browser?
. Create a Publisher ID database... ?
------------------
. Analysis: "Defrag"/Organize/Reorder the IndexTable
OR... Reverse it, or Randomize it.
------------------
. An "info" page about the current MPK file? Statistics? "Reserved bits in file extension in use?"
------------------
.Load RAW save file into a specific Save slot
------------------
.Add full support for DexDrive file features!
.Add support for other note file types?
----POTENTIAL SHIT----
.Make Blue bar into a progress bar? of space remaining etc.
. Confirm Delete? Confirm Save? etc.
.What if a Note file is larger than physically possible? Almost never possible.
-------------------------------
1. Sophisticated and configurable logging system
----
I want to be able to configure the output of console logs,
not just rely on hard-coded logs.
Example:
msg("ERROR: Data invalid: %file", "w", filename);
In the above line, we throw a hard warning that a file couldn't
be opened.
msg("ERROR: No room to import note", "w");
Another example of a hard warning, this would occur when attempting
to import a note that won't fit in the current file.
Ideally, these hard warnings would be enabled by default.
Soft warnings?
msg("Reserved bits of ext used in note: %noteName", "sw", noteName);
This for example would give useful info, or indicate a potential problem.
Verbose Info
msg("Opening: %filename", "v1", filename);
This could provide multiple levels of verbose messages,
displaying file information for debugging purposes.
~~~~~~~~~~~~~~~~~~~
Referenced TODOs:
. Add Message when attempting to load a note that is too large
~~~~~~~~~~~~~~~~~~
2. Retain original file structure of MPK file.
----
Currently the program will 'rewrite' the NoteTable, deleting invalid
data, wiping it clean, and moving note headers to the top, keeping
the general order.
This is done because the app is designed to hide empty rows, and allow
note-reordering.
However, this potentially discards information, such as the original
positions in the note table, as well as ghost entries.
Some options:
1) Do not modify the NoteTable until the last minute, such as when a
user re-orders a note, or saves the file.
2) Retain the original note data and order in a secondary mode
that shows ghost entries.
The second option is probably the best one, but it would take
a lot of work to implement. Still important though.
Other notes:
This secondary mode could also involve or support other advanced features I have been thinking about.
~~~~~~~~~~~~~~~~~~~
Referenced TODOs:
. Advanced Display: Show NoteIDs, note-checksum, etc. Serial code?
Page and Note count could be part of advanced view.
~~~~~~~~~~~~~~~~~~
000h 12 ASCII String "123-456-STD",00h
00Ch 4 Usually zerofilled (or meaningless garbage in some files)
010h 5 Always 00h,00h,01h,00h,01h
015h 16 Copy of Sector 0..15 byte[00h] ;"M", followed by allocation states
025h 16 Copy of Sector 0..15 byte[08h] ;00h, followed by next block values
035h 11 Usually zerofilled (or meaningless garbage in some files)
040h F00h Fifteen Description Strings (each one 100h bytes, padded with 00h)
F40h 128K Memory Card Image (128K) (unused sectors 00h or FFh filled)
// .a64 header should look something like this:
// NBCE 01 0203 0 {} {BLASTCORPS GAME}
// first 4 chars are the first 4 bytes
// next 2 chars are the next 2 bytes
// next 4 chars are bytes 8 and 9, in hex (huh?)
// next character is byte 10, in hex (but only one character this time)
// now we've got two sets of braces... the first one contains byte 12 in encoded form (use ReverseNotesA)
// the second one should contain bytes 16 through 31 (ReverseNotesA)
a64-notes
[Game Save Description (optional, written by user)]
a64-data
[Game Header Data]
[64x230 Character Hex Code]
a64-crc
[8 Character CRC Code]
a64-end
// On a real N64, if the first INODE table fails the checksum it automatically
// uses the second one. I have a couple of files that exhibit this problem
// but my implementation ignores it and still looks for valid notes.
// It seems that despite the first INODE table's checksum being invalid,
// the data seems to still correspond to the note table, and it seems valid.
/*
A lot of DexDrive saves seem to have issues with their INODE tables.
Some match, yet have invalid checksums.
Others don't match, and have only one valid checksum.
This issue prevents the integrity of the data from being easily known.
However, it seems the first INODE table, regardless of state, retains the correct
INODE sectors for note entries.
Workarounds for checksum issues, when note data is present.
One INODE table checksum is valid
One INODE table checksum is invalid
In this situation, the invalid INODE may hold correct data, while
the valid INODE show incorrect data.
The inverse can also be true:
The valid INODE could hold correct data, while the invalid INODE
may show incorrect data.
To counter this, we can cross-reference both INODE tables against
the actual data on the pak, as well as any notes stored on the pak.
If all of the data is found empty, and one of the INODE tables
confirms this, then it should be assumed to be the correct INODE table.
However, if data is found, and one of the INODE tables and note table confirms this,
then it should be assumed to be the correct INODE.
Algorithm:
1. For each Note Entry, check its firstINODE value.
2. Ensure that it correlates to 0x06 or higher in the INODE table.
3. Loop through the data associated to that note.
4. Confirm that data is present in each INODE sector.
Do this for both INODE tables.
Each Note
*/
console.log("Hello! Drag MemPaks into the browser window (not the console) to read out their contents and stuff.");
// N64 font value decoding, japanese values are x'd out because i can't decipher kanji reliably to put them in proper order
var code64 = {
0: "",
3: "",
15: " ",
16: "0",
17: "1",
18: "2",
19: "3",
20: "4",
21: "5",
22: "6",
23: "7",
24: "8",
25: "9",
26: "A",
27: "B",
28: "C",
29: "D",
30: "E",
31: "F",
32: "G",
33: "H",
34: "I",
35: "J",
36: "K",
37: "L",
38: "M",
39: "N",
40: "O",
41: "P",
42: "Q",
43: "R",
44: "S",
45: "T",
46: "U",
47: "V",
48: "W",
49: "X",
50: "Y",
51: "Z",
52: "!",
53: '"',
54: "#",
55: "'",
56: "*",
57: "+",
58: ",",
59: "-",
60: ".",
61: "/",
62: ":",
63: "=",
64: "?",
65: "@",
66: "。",
67: "゛",
68: "゜",
69: "ァ",
70: "ィ",
71: "ゥ",
72: "ェ",
73: "ォ",
74: "ッ",
75: "ャ",
76: "ュ",
77: "ョ",
78: "ヲ",
79: "ン",
80: "ア",
81: "イ",
82: "ウ",
83: "エ",
84: "オ",
85: "カ",
86: "キ",
87: "ク",
88: "ケ",
89: "コ",
90: "サ",
91: "シ",
92: "ス",
93: "セ",
94: "ソ",
95: "タ",
96: "チ",
97: "ツ",
98: "テ",
99: "ト",
100: "ナ",
101: "ニ",
102: "ヌ",
103: "ネ",
104: "ノ",
105: "ハ",
106: "ヒ",
107: "フ",
108: "ヘ",
109: "ホ",
110: "マ",
111: "ミ",
112: "ム",
113: "メ",
114: "モ",
115: "ヤ",
116: "ユ",
117: "ヨ",
118: "ラ",
119: "リ",
120: "ル",
121: "レ",
122: "ロ",
123: "ワ",
124: "ガ",
125: "ギ",
126: "グ",
127: "ゲ",
128: "ゴ",
129: "ザ",
130: "ジ",
131: "ズ",
132: "ゼ",
133: "ゾ",
134: "ダ",
135: "ヂ",
136: "ヅ",
137: "デ",
138: "ド",
139: "バ",
140: "ビ",
141: "ブ",
142: "ベ",
143: "ボ",
144: "パ",
145: "ピ",
146: "プ",
147: "ペ",
148: "ポ"
};
var mp = {};
var file = {};
function extract(noteID) // for extracting a specific note!
{
var A = document.createElement('a');
A.download = file.name + "_" + noteID + "_out.bin";
A.href = "data:application/octet-stream;base64," +
btoa(String.fromCharCode.apply(null, mp.notes[noteID].data));
A.dispatchEvent( new MouseEvent('click') );
}
function extractMPK() // for saving the entire file!
{
var A = document.createElement('a');
A.download = file.name + "_" + "full" + "_out.mpk";
A.href = "data:application/octet-stream;base64," +
btoa(String.fromCharCode.apply(null, mp.data));
A.dispatchEvent( new MouseEvent('click') );
}
function deleteNote(noteID) // for deleting notes
{
// Simple and efficient note deleting:
// It takes the note pointer and iterates
// over all associated inodes, setting
// them to 0x03 "free space".
// It then sets the inode in the note table
// to 0x00 which effectively marks the file
// as free. Checksum is updated in primary
// INODE table.
var noteOffset = 0x300 + (32 * noteID);
var inodes = mp.notes[noteID].inodes;
console.log(inodes);
// iterate inodes and set them to free
for (var i = 0; i < inodes.length; i++)
{
var inodeOffset = 0x100 + (2 * inodes[i] + 1);
mp.data[inodeOffset] = 0x03;
console.log(inodeOffset, mp.data[inodeOffset]);
}
mp.data[noteOffset + 7] = 0x00;
console.log(mp.data[noteOffset + 7]);
for(i = 0x100, chk = 0; i < 0x200; i++, chk &= 0xFF)
{
// Calculate the checksum
if(i >= 0x10A) { chk += mp.data[i]; }
}
mp.data[0x100 + 1] = chk;
}
function checkID(offset)
{
var sum1 = (mp.data[offset + 28] << 8) + mp.data[offset + 29];
var sum2 = (mp.data[offset + 30] << 8) + mp.data[offset + 31];
for(var i = 0, sum = 0; i < 28; i+= 2, sum &= 0xFFFF)
{
sum += (mp.data[offset + i] << 8) + mp.data[offset + i + 1];
}
// Detect DexDrive invalid header
if( (sum == sum1) && ((0xFFF2 - sum) != sum2) && ((0xFFF2 - sum) == sum2 ^ 0x0C ))
{
// Update sum2
mp.data[offset + 31] ^= 0x0C;
sum2 = (mp.data[offset + 30] << 8) + mp.data[offset + 31];
}
return ((sum == sum1) && ((0xFFF2 - sum) == sum2));
}
function readMemPak()
{
console.clear();
console.log("%cReading file: " + file.name, "font:900 14px verdana");
mp.notes = [];
mp.inodes = [];
mp.freePages = 0;
mp.usedNotes = 0;
// header operations
var checkHeader = checkID(0x20);
var ch0 = checkID(0x60);
var ch1 = checkID(0x80);
var ch2 = checkID(0xC0);
if (checkHeader === false)
{
console.log("Not a valid Controller Pak file.");
return false;
} else {
console.log("Header checksum valid : ", checkHeader);
console.log("Other header checksums : ", ch0, ch1, ch2);
}
// INODE table operations
for(var i = 0x100, chk = 0, chk2 = 0, cnt = 0; i < 0x200; i++)
{
// Sum the identical bytes of the INODE table
if(mp.data[i] === mp.data[i + 0x100]) { cnt++; }
// Calculate the checksum
if(i >= 0x10A) { chk += mp.data[i]; }
// Calculate the checksum
if(i >= 0x10A) { chk2 += mp.data[i + 0x100]; }
// Sum the number of free pages
if(i >= 0x10A && mp.data[i] === 0x03) { mp.freePages++; }
// Collect array of INODE values
if((i & 0x01) === 1) { mp.inodes.push(mp.data[i]); }
}
mp.usedPages = (123 - mp.freePages);
mp.inodeIsIdentical = (cnt === 256);
// TODO: Verbose checksum validation. Cannot just simply be a >= 0x51.
// because checksum overflows to 0. Perhaps save the full checksum as well?
mp.inodeChecksumCorrect = (mp.data[0x101] === (chk & 0xFF)) && chk !== 0;
mp.inodeChecksumCorrect2 = (mp.data[0x201] === (chk2 & 0xFF)) && chk2 !== 0;
console.log("Inode tables identical : ", mp.inodeIsIdentical);
console.log("Inode checksum correct : ", mp.inodeChecksumCorrect);
console.log("Inode checksum2 correct: ", mp.inodeChecksumCorrect2);
console.log("Number of used pages : ", mp.usedPages);
console.log("Number of free pages : ", mp.freePages,"\n");
if((mp.inodeIsIdentical && mp.inodeChecksumCorrect && mp.inodeChecksumCorrect2 ) === false)
{
console.warn("Warning: Problems found with INODE data. It may be corrupt or invalid.","\n-----\n\n\n");
}
// Note table operations
for(i = 0x300, j = 0; i < 0x500; i += 32, j++)
{
// The first thing we need to know is the inode value
// of each note entry. It should be between 5 and 127
// to indicate that data is present
var b = mp.data[i + 7];
// Additional things to check
var b1 = mp.data[i + 0x0A];
var b2 = mp.data[i + 0x0B];
var b3 = mp.data[i + 6];
var b4 = mp.data[i + 8] & 0x02;
if(b >= 5 && b <= 127 && b1 === 0x00 && b2 === 0x00 && b3 === 0x00)
{
mp.usedNotes++;
var gameNote = {};
gameNote.inodes = [b];
gameNote.pages = 1;
gameNote.data = [];
gameNote.name = "";
// Game ID, Vendor etc
gameNote.game_id = String.fromCharCode(mp.data[i + 0], mp.data[i + 1], mp.data[i + 2]);
gameNote.region = String.fromCharCode(mp.data[i + 3]);
gameNote.vendor = String.fromCharCode(mp.data[i + 4], mp.data[i + 5]);
// Cycle through the inode table to find subsequent inodes
var nextInode = mp.inodes[b];
while(nextInode >= 5 && nextInode <= 127)
{
// infinite loop here. need a method of validating file format.
if(gameNote.inodes.indexOf(nextInode) > -1 ){break;}
gameNote.inodes.push(nextInode);
nextInode = mp.inodes[nextInode];
gameNote.pages++;
}
var k;
// Let's use our inodes to collect the data
for(var inode in gameNote.inodes)
{
var dataOffset = 256 * gameNote.inodes[inode];
for(k = 0; k < 256; k++)
{
gameNote.data.push(mp.data[dataOffset + k]);
}
}
// Get note name and extension
for(k = 0; k < 16; k++)
{
gameNote.name += code64[ mp.data[i + 16 + k] ];
}
if(mp.data[i + 12] !== 0)
{
gameNote.name += "." + code64[ mp.data[i + 12] ];
}
console.log(
j, gameNote.name, gameNote.game_id + gameNote.region, gameNote.vendor,
"Pages: " + gameNote.pages + " (" + gameNote.data.length + " bytes)"
);
console.log("\t" + gameNote.inodes + "\n");
mp.notes.push(gameNote);
}
}
mp.freeNotes = (16 - mp.usedNotes);
console.log("Number of used notes : ", mp.usedNotes);
console.log("Number of free notes : ", mp.freeNotes);
}
/* Drag handler */
document.ondragover = function () { return false; };
document.ondrop = function (e)
{
//for(var w = 0; w < e.dataTransfer.files.length; w++)
//{
// Grab the file target, then read out 36 KB
var filez = e.dataTransfer.files[0];
var f = new FileReader();
f.filez = filez;
f.readAsArrayBuffer( filez.slice(0, 36928) );
// ----
f.onload = function (evt)
{
// Temporary array for read access
var _data = new Uint8Array(evt.target.result);
// Let's check if it's a DexDrive file and strip the leading 0x1040 bytes
for (var i = 0, dex = ""; i < 0x0B; i++)
{
dex += String.fromCharCode( _data[i] );
}
if (dex === "123-456-STD")
{
_data = _data.subarray(0x1040);
}
// Create a proper byte array of the data
mp.data = new Uint8Array(32768);
for(i = 0; i < 32768; i++)
{
mp.data[i] = _data[i];
}
file.name = evt.target.filez.name;
// Then we start reading the data.
readMemPak();
};
// }
return false;
};#PSX format. maybe in the future!
Memory Card Data Format
-----------------------
Data Size
Total Memory 128KB = 131072 bytes = 20000h bytes
1 Block 8KB = 8192 bytes = 2000h bytes
1 Frame 128 bytes = 80h bytes
The memory is split into 16 blocks (of 8 Kbytes each), and each block is split
into 64 sectors (of 128 bytes each). The first block is used as Directory, the
remaining 15 blocks are containing Files, each file can occupy one or more
blocks.
Header Frame (Block 0, Frame 0)
00h-01h Memory Card ID (ASCII "MC")
02h-7Eh Unused (zero)
7Fh Checksum (all above bytes XORed with each other) (usually 0Eh)
Directory Frames (Block 0, Frame 1..15)
00h-03h Block Allocation State
00000051h - In use ;first-or-only block of a file
00000052h - In use ;middle block of a file (if 3 or more blocks)
00000053h - In use ;last block of a file (if 2 or more blocks)
000000A0h - Free ;freshly formatted
000000A1h - Free ;deleted (first-or-only block of file)
000000A2h - Free ;deleted (middle block of file)
000000A3h - Free ;deleted (last block of file)
04h-07h Filesize in bytes (2000h..1E000h; in multiples of 8Kbytes)
08h-09h Pointer to the NEXT block number (minus 1) used by the file
(ie. 0..14 for Block Number 1..15) (or FFFFh if last-or-only block)
0Ah-1Eh Filename in ASCII, terminated by 00h (max 20 chars, plus ending 00h)
1Fh Zero (unused)
20h-7Eh Garbage (usually 00h-filled)
7Fh Checksum (all above bytes XORed with each other)
Filesize [04h..07h] and Filename [0Ah..1Eh] are stored only in the first
directory entry of a file (ie. with State=51h or A1h), other directory entries
have that bytes zero-filled.
Filename Notes
The first some letters of the filename should indicate the game to which the
file belongs, in case of commercial games this is conventionally done like so:
Two character region code:
"BI"=Japan, "BE"=Europe, "BA"=America
followed by 10 character game code,
in "AAAA-NNNNN" form ;for Pocketstation executables replace "-" by "P"
where the "AAAA" part does imply the region too; (SLPS/SCPS=Japan,
SLUS/SCUS=America, SLES/SCES=Europe) (SCxS=Made by Sony, SLxS=Licensed by
Sony), followed by up to 8 characters,
"abcdefgh"
(which may identify the file if the game uses multiple files; this part often
contains a random string which seems to be allowed to contain any chars in
range of 20h..7Fh, of course it shouldn't contain "?" and "*" wildcards).
Broken Sector List (Block 0, Frame 16..35)
00h-03h Broken Sector Number (Block*64+Frame) (FFFFFFFFh=None)
04h-7Eh Garbage (usually 00h-filled) (some cards have [08h..09h]=FFFFh)
7Fh Checksum (all above bytes XORed with each other)
If Block0/Frame(16+N) indicates that a given sector is broken, then the data
for that sector is stored in Block0/Frame(36+N).
Broken Sector Replacement Data (Block 0, Frame 36..55)
00h-7Fh Data (usually FFh-filled, if there's no broken sector)
Unused Frames (Block 0, Frame 56..62)
00h-7Fh Unused (usually FFh-filled)
Write Test Frame (Block 0, Frame 63)
Reportedly "write test". Usually same as Block 0 ("MC", 253 zero-bytes, plus
checksum 0Eh).
Title Frame (Block 1..15, Frame 0) (in first block of file only)
00h-01h ID (ASCII "SC")
02h Icon Display Flag
11h...Icon has 1 frame (static) (same image shown forever)
12h...Icon has 2 frames (animated) (changes every 16 PAL frames)
13h...Icon has 3 frames (animated) (changes every 11 PAL frames)
Values other than 11h..13 seem to be treated as corrupted file
(causing the file not to be listed in the bootmenu)
03h Block Number (1-15) "icon block count" Uh?
(usually 01h or 02h... might be block number within
files that occupy 2 or more blocks)
(actually, that kind of files seem to HAVE title frames
in ALL of their blocks; not only in their FIRST block)
(at least SOME seem to have such duplicated title frame,
but not all?)
04h-43h Title in Shift-JIS format (64 bytes = max 32 characters)
44h-4Fh Reserved (00h)
50h-5Fh Reserved (00h) ;<-- this region is used for the Pocketstation
60h-7Fh Icon 16 Color Palette Data (each entry is 16bit CLUT)
For more info on entries [50h..5Fh], see
--> Pocketstation File Header/Icons
Icon Frame(s) (Block 1..15, Frame 1..3) (in first block of file only)
00h-7Fh Icon Bitmap (16x16 pixels, 4bit color depth)
Note: The icons are shown in the BIOS bootmenu (which appears when starting the
PlayStation without a CDROM inserted). The icons are drawn via GP0(2Ch)
command, ie. as Textured four-point polygon, opaque, with texture-blending,
whereas the 24bit blending color is 808080h (so it's quite the same as raw
texture without blending). As semi-transparency is disabled, Palette/CLUT
values can be 0000h=FullyTransparent, or 8000h=SolidBlack (the icons are
usually shown on a black background, so it doesn't make much of a difference).
Data Frame(s) (Block 1..15, Frame N..63; N=excluding any Title/Icon Frames)
00h-7Fh Data
Note: Files that occupy more than one block are having only ONE Title area, and
only one Icon area (in the first sector(s) of their first block), the
additional blocks are using sectors 0..63 for plain data.
Shift-JIS Character Set (16bit) (used in Title Frames)
Can contain japanese or english text, english characters are encoded like so:
81h,40h --> SPC
81h,43h..97h --> punctuation marks
82h,4Fh..58h --> "0..9"
82h,60h..79h --> "A..Z"
82h,81h..9Ah --> "a..z"
Titles shorter than 32 characters are padded with 00h-bytes.
Note: The titles are <usually> in 16bit format (even if they consist of raw
english text), however, the BIOS memory card manager does also accept 8bit
characters 20h..7Fh (so, in the 8bit form, the title could be theoretically up
to 64 characters long, but, nethertheless, the BIOS displays only max 32
chars).
For displaying Titles, the BIOS includes a complete Shift-JIS character set,
--> BIOS Character Sets
Shift-JIS is focused on asian languages, and does NOT include european letters
(eg. such with accent marks). Although the non-japanese PSX BIOSes DO include a
european character set, the BIOS memory card manager DOESN'T seem to translate
any title character codes to that character set region.
var t = "00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 7E 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0B 04 00 00 01 00 0A 0A 01 00 00 00 FC D6 81".split(" ");
for(var i = 0, y = 0; i<t.length; i++)
{
y ^= parseInt(t[i], 16);
}
console.log(y.toString(16));
http://www.raphnet.net/electronique/psx_adaptor/Playstation.txt
Code for resetting origin sort:
if(!origin || origin.nodeName !== "TR") {return false;}
var origin_id = parseInt(origin.id);
var origin_slot = parseInt(origin.parentNode.children[origin.id].id);
if(origin_id > origin_slot) {
origin.parentNode.insertBefore(
origin, origin.parentNode.children[origin_id+1]);
}
if (origin_id < origin_slot) {
origin.parentNode.insertBefore(
origin, origin.parentNode.children[origin.id]);
}// after proper feature detection, consider recommending IE users to upgrade:
var upgrade = (!!document.documentMode || (!!!document.documentMode && !!window.StyleMedia));Initially, comments will only be able to be saved/loaded within note files. The problem is that MPK files can be modified by games. You can create new saves, delete/erase saves, and even reset the entire Controller Pak data. If I implement comment metadata as an 'extension' to normal MPK files, it is very possible that the comment<->note association could be lost. Some emulators may choose to overwrite, erase, or crop files to 32KB, losing the data. I will have to do tons of testing and busywork to figure out the best option.
- MPK extension after 32KB: Comments designated to numerical note slots.
- MPK extension after 32KB: Comments designated to hash codes representing unique notes. (This is probably best option)
- An alternative export mode, "extended MPK" which is for archival purposes and not emulator?
We could extend the MPK header or INODE table in such a way that it could be used to detect the extended MPK format unobtrusively. Ideally, the extended MPK format should be robust enough to be supported by most emulators, assuming they don't bother checking the memory size of the MPK data. But if emulators cannot reliably support it, we can just make it an archival format.
Notes:
a) For now I will not be attempting to support *.a64 note files. The only existing software/plugin that uses *.a64 (basically just the N-Rage plugin) has a shoddy implementation, and there is no complete description of the format.
Examples of files created in a64 format (notice missing data that essentially makes them unreadable):
waverace64.a64
a64-notes
description of game save goes here...
a64-data
NWRJ }
5445C863FFFFFFFF0100000000FFFFFF522E484159414D4900FF442E4D415249
4E455200412E53544557415254004D2E4A4554455200FFFF0505050505050505
05050505050505050505050505050505000503000503000503FFFFFFFFFF....
perfect dark.a64
a64-notes
description of game save goes here...
a64-data
NPDE 4Y 0200 0 {} {perfect dark}
284305F2FFFFFFFF0805B080F691200A2B2B2B2B2B2B2B2B2B2B2B2B2B2B2B2B
0003000300030003000300030003000300030003000300030003000300030003
000300030003000300030003000300030003000300030003000300030003....
In fact, the format is flawed, as the publisher code or game code is not necessarily going to be ASCII.
Wave Race 64 actually uses a binary 0x01 as the publisher, which cannot be expressed in printable ASCII characters, so this note would be unreadable by Wave Race 64 upon importing. Furthermore, it is impossible to modify the a64-notes section without manually opening the a64 in a text editor. Essentially meaning no one will ever add a note/comment to their save file.
Another reason for not supporting a64 is that, if you can create an a64 file, 99% of the time you already have the *.MPK file which is compatible with mpkedit. Users can just extract a note using mpkedit. Maybe in the future, the mpkedit .note format can be added to other software.
b) The comments in DexDrive files often become decoupled. For example, comments could still exist for notes that have been erased, or comments no longer point to the correct note. Without manually 'repairing' these files, comments can be incorrect, or lost when loading into mpkedit. There is no way around this.
I believe the best way around this, is to generate a hashcode based on non-changing, identifiable information. This could include the Game Code, Publisher Code, Note Name, and perhaps all the other associated data. We can ignore the INODE number, and even the data itself, because those could possibly change. We should also ignore the order and position of the notes. Essentially > Generate a hashcode of all existing notes > lookup the hashcode in the comment data structure > load comments that match.
Update August 2017: Actually, the 'INODE number' mentioned above may not actually change, and may be more static than the Game Code/Publisher Code and other mentioned possibilities. The theory is, that the N64 MPK doesn't employ any sort of 'defragmentation', so unless you erase a note, the data would essentially occupy the same memory locations that it did when it was first saved, meaning it could be a reliable note identifier. This of course can be a problem if multiple notes use the same INODE offset, but I think that is not possible to occur in a legitimate MPK file. This will need to be tested when the time comes to implement comment support to MPK files, which will be a fun time!
If hashcodes are identical, (which might be possible if we ignore data/inode!), it should load comments in the order of the notes themselves (0,1,2,3...15). Assuming they were saved in that order, they would be retrieved back successfully even if duplicate hashcodes exist.
Detection of MPKEdit-modified file: There are some bits in the header that are possibly always set to 0. If MPKEdit-modified files set these bits, then a real N64 unsets them, it can be detected and any MPKEdit-specific data can be discarded (like comments). Note order can change, breaking location of the comments. Alternatively we can modify the note header itself for comment purposes. Could only store comments in note-file, or custom n64-incompatible MPK format.
Update August 2017: Okay, so this idea may be flawed, if I'm referring to offset 0x1A as the "always 0" bits. The game may behave oddly by messing with that offset too much, even with correct checksums. I think that mpkedit can simply discard comment data that isn't associated with any specific note without resorting to 'nuking' everything if MPKEdit detects a modified MPK file. Ideally we want to see if the comment data can be retained even when modified by emulators, etc.
This document describes how games interpret the Note data, one criteria at a time, including any notable differences between games.
Libultra will erase any Note that has a null game code or publisher code. At least one bit is required in both fields to avoid this. Normally this should never occur in an MPK dump, as it effectively causes save data to be erased without even prompting the user to 'repair Controller Pak'.
Strangely, there is a single occurrence of this in the wild. On GameFAQs there is a DexDrive save for Tony Hawk's Pro Skater 2 that contains both 00 00 00 00 and 00 00. This causes the data to be erased upon boot, but it is repairable if the game code and publisher code is manually replaced with the correct values 4E 54 51 45 and 35 32 respectively.
As another example, the publisher code used in Wave Race 64 is 00 01 and contains only a single bit set. This was a goof on the developers, as the publisher code for Nintendo in other games is 01 in ASCII (30 31 in hex). If the publisher code is changed to 00 00, the save data is erased as expected. Also interesting: Wave Race 64's Note uses the Japanese game code even on the American version.
packNoteInfo was made to encode game codes and indexes into as few bits as possible (packing 4 bytes into 3 bytes), but I ultimately decided it was pointless, might as well use the 4 bytes, but still cool how it works, almost like a subset of ASCII.
MPKEdit.packNoteInfo = function(index, gameCode) {
var map = [
"ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890",
"EPJDFISUBAXY" // append any new lang codes. 3 free slots!
], gcode1, gcode2, region, output;
if(index > 0x7F && !gameCode) {
gcode1 = map[0][index >>> 7 & 0x3F];
gcode2 = map[0][index >>> 13 & 0x3F];
region = map[1][index >>> 19 & 0xF];
output = [index & 0x7F, gcode1+gcode2+region];
} else if(index < 0x80 && 3 === gameCode.length) {
gcode1 = map[0].indexOf(gameCode[0]);
gcode2 = map[0].indexOf(gameCode[1]);
region = map[1].indexOf(gameCode[2]);
output = (region << 19) + (gcode2 << 13) + (gcode1 << 7) + (index & 0x7F);
} else {
console.error("Something went wrong in packNoteInfo");
return false;
}
return output;
};