Detecting invalid forms in lua

[RealAntithesis]

Is there a way to determine whether a form is invalid in lua? When papyrus accesses an invalid JC form (e.g. forms have changed due to a different load order etc), it reads as nil. But that form still appears to have a value (CData) in lua - so testing for nil doesn't work. Essentially, I want to be able to delete invalid forms (or otherwise act on them) from a lua script. Thanks.

[SilverIce]

A possible fix:

     JCToLuaValue JCToLuaValue_fromItem(const item& itm) {
      ...
            void operator ()(const form_ref& val) {
                auto formId = (FormIdUnredlying)val.get();
                if (!formid)
                   value = JCToLuaValue_None();
                 else
                    value.form = { (FormIdUnredlying)formId  };
            }

[ryobg]

That can do... but can't be also compared with Form(0) or something after retrieval?

[RealAntithesis]

Haven't tried that. Will do as soon as I get home.

[SilverIce]

@ryobg yeah, Form(0) should work as well

[ryobg]

@RealAntithesis any news here, all good?

[RealAntithesis]

I'm doing more testing. For some reason, some of my form equality testing in lua doesn't appear to be giving consistent results. e.g. an array of forms testing against an array of objects that contain forms seems to be giving different results every time I test them.
e.g. formIsEqual = JArrayOfForms[i] == JMapOfFormContainingObjects["formIDKey"].theForm
Walking through both these structures and picking out those forms that aren't equal and JArray.inserting them into another JArray (created in lua) seems to be populating with different forms every time I reload. The only thing being done to these structures prior is populating the forms array with the player inventory and checking the JMap objects for forms that don't equal Form(formID) and replacing with Form(formID) if it doesn't match.

I need a more controlled test to see what's going on. It's clear though that 'printing' out both the array of forms and JMap of objects containing forms via papyrus, that the forms actually do appear in both but are equating to 'false' in the above test.

[RealAntithesis]

Is it possible to get the formID from a form in lua? That may help workaround this since a formID is just a number that can be directly compared between lua and papyrus.

[SilverIce]

@RealAntithesis , Lua Form has an ___id field

[RealAntithesis]

Great, thanks. Will look into it.

[RealAntithesis]

I've had a look into it. It appears that theForm.___id contains the correct formID for most forms, except for what looks like crafted potions and possibly forms originating from ESLs and ESPFEs (I need to verify this more). Plugging these formIDs into papyrus and using Game.GetFormEx(formID) returns none. i.e. theForm.___id returns a different number for some forms than theForm.GetFormID().

[ryobg]

Do you use that ESL patched release from here (GitHub)?

[RealAntithesis]

I thought I was. Will double check.

[RealAntithesis]

Yep, this is with version 4.1.10 with the ESL fix. Besides, I ran the test concurrently in lua and papyrus at the same time, printing out the same array of forms. Papyrus successfully provided the formID when using theForm.GetFormID(), so the form itself was valid. The problem was that lua returned a different id from theForm.___id for some forms only.
e.g. Print array function output for Silver Mace of Dismay: note that itemFormID was generated from papyrus theForm.GetFormID() and added to the object as key:value pair. Also, the lua output was printed to the log entirely from lua (so there shouldn't be any funny 32bit to 24bit conversion from lua to papyrus).

Lua: 289) Silver Mace of Dismay: itemForm=cdata<struct _CForm &>: 0x37560518; itemFormID=-1409273042; Form(itemFormID)=cdata: 0x375605a0; theForm.___id=2885694254; Form(itemFormID).___id=2885694254

Papyrus: 288) Item: Silver Mace of Dismay; form=[Form < (AC00332E)>](itemFormID=-1409273042; theForm.GetFormID()=-1409273042)

Note that the form AC00332E originated from the Prvtl_HeavyArmory.esp plugin with no overrides in the load order, and which was a normal esp (no ESL flag).

[RealAntithesis]

I think I figured it out.
The form is AC00 332E.
The hex for -1,409,273,042 is FFFF FFFF AC00 332E
And the hex for 2,885,694,254‬ is AC00 332E
Is this something that should be resolved in JContainers (so it reports the same decimal number)?

[ryobg]

(scrathes head) The test is not okay? Depending on how you convert the decimals you may get these FFFF in front or you may not. Could be that the two number you compare are of differnt sign. One is signed, the other not. This will fail until you compare them bitwise say or convert them safely to proper type.

[RealAntithesis]

The numbers are still coming out wrong though. I suppose I can pad the .___id number in lua so it matches the number in papyrus (edit, or chop out the FFFF FFFF in the papyrus number), but shouldn't they equal each other for the same form rather than converting the lua number to enable an equality test with the papyrus-generated number?

Edit: the papyrus number is an Int. Lua doesn't appear to have a type?

[RealAntithesis]

I don't know how this affects my other issue of forms not equalling each other. All I think I'm doing in that other test is to test the CData Form1 == Form2 (I'm not sure what this is doing in the background - is it just comparing the .___id numbers, or is there another comparison going on?). I'll have a another look at it.

[RealAntithesis]

There may be a 32bit vs 24bit issue here? I tried padding out the .___id number to get to the formID generated by papyrus using BitOr(0xFFFFFFFF00000000,0xAC00332E), but instead got FFFF FFFF FFF0 332E (BitOr is my local lua function for bit.bor). However, BitOr(0xFFFF00000000,0xAC00332E) returned the same decimal number as papyrus, so I can use this to compare the id numbers.

[ryobg]

That Lua does not have integeres, and everything above the 24-bit range starts to mess up, if not an extended type. Check what types you are using here & there. Maybe it will be just easier to do what SilverIce suggested, and zero out the forms at the beginning. I'm not sure what I see in your investigation.

[RealAntithesis]

Yep, grabbing the first 16 bits in the papyrus-derived formID is probably the way I'll go considering this is just a lua issue (I'm assuming the 2nd 16 bits will always be filled with FFFF FFFF and won't ever be anything else).

[ryobg]

I will suggest that you add a check for that assumption. Then let me know how this turned out and whether there is something which can be fixed.

[SilverIce]

That Lua does not have integeres

That's not true. _CForm is a structure containing uint32 ___id field.

and everything above the 24-bit range starts to mess up

Lua has doubles. The things mess up currently (#40) on C++ side due to Lua double -> Papyrus or JC float conversion. Would be cool to switch JC to use doubles. Or detect at least, that double value has no fraction numbers and convert it into Int on JC side

The hex for -1,409,273,042 is FFFF FFFF AC00 332E

How did you got that value? Papyrus Int has 64 bits?

[RealAntithesis]

I have to admit, I'm somewhat at a loss here. I've just done a test to compare forms in an array of forms against each other, printing out the results (and the formID being tested), and iterating this 6 times for each form. The intent of this was to test my earlier observation of getting different results when comparing forms in an array, but using more controlled conditions. In this test, the array of forms remained static (it was populated once in papyrus, and not changed in lua).

The results seem to confirm my earlier observations that something incorrect is indeed happening - the search results for the same forms do appear to change over repeated iterations and the form values themselves (or at least their formIDs) appear to change without being explicitly altered by the user, even over a limited set of 6 observations.

These are some of the results I'm getting (1st value is the form.___id field; 2nd value the hex of the form.___id field); 3rd value is the search result of the entire array to find the form - this should always be true since the form is taken from that same array; repeated 6 times):
Good result:

1. 4278196020(0xff001734)=true; 4278196020(0xff001734)=true; 4278196020(0xff001734)=true; 4278196020(0xff001734)=true; 4278196020(0xff001734)=true; 4278196020(0xff001734)=true;

Bad result (formID has changed for some reason, despite being the same form):

22. 4278196286(0xff00183e)=true; 4278196286(0xff00183e)=true; 4278196286(0xff00183e)=true; 4278196286(0xff00183e)=true; 4278195381(0xff0014b5)=true; 4278195381(0xff0014b5)=true; FormID HAS CHANGED. (ERROR)

Bad result (search result has changed for some reason, but the form and array being searched hasn't changed):

56. 412472(0x64b38)=true; 412472(0x64b38)=true; 412472(0x64b38)=true; 412472(0x64b38)=true; 412472(0x64b38)=true; 412472(0x64b38)=false; Search result HAS CHANGED. (ERROR)

Bad result (both formID and search result have changed during the 6 iterations, but neither the form nor the array being searched have changed):

88. 216278(0x34cd6)=true; 216278(0x34cd6)=true; 216278(0x34cd6)=false; 1026148456(0x3d29c868)=false; 1026148456(0x3d29c868)=false; 1026148456(0x3d29c868)=false; FormID HAS CHANGED. Search result HAS CHANGED. (ERROR)

Lua code: this is self contained, except for the papyrus code that populated the array (and retained it) and passed this through to lua in the objInventoryFormTestArray parameter, and the LogMessage() function which just prints to the log. In this test, there are 224 forms in the array.

function AH_Hotkeys.InventoryFormArrayTest(objInventoryFormTestArray) local function FormsAreEqual(form1, form2) if (form1 == nil) or (form2 == nil) then return false elseif form1 == form2 then return true else return form1.___id == form2.___id end end local function FormIsInFormArray(objFormArray, itemForm) local existingForm for i = 1, #objFormArray do existingForm = objFormArray[i] if FormsAreEqual(existingForm, itemForm) then -- objExistingInventoryItem.itemForm == itemForm then return true end end return false end AH_Hotkeys.LogMessage("InventoryFormArrayTest(): Start:", debugLevels.kNotice) -- objInventoryFormTestArray = formArray local numIterations = 6 local textString = "" local formWasFound, origFormWasFound, iteratedFormIDWasChanged, searchResultChanged, wasError local formToCheck local formToCheckID, previousIterationOfSameFormID for i = 1, #objInventoryFormTestArray do textString = textString..tostring(i)..") " formToCheck = objInventoryFormTestArray[i] formToCheckID = formToCheck.___id for n = 1, numIterations do previousIterationOfSameFormID = formToCheck.___id textString = textString..tostring(previousIterationOfSameFormID).."("..string.format('0x%x', previousIterationOfSameFormID)..")=" formWasFound = AH_Hotkeys.FormIsInFormArray(objInventoryFormTestArray, formToCheck) if n == 1 then origFormWasFound = formWasFound end textString = textString..tostring(formWasFound).."; " if formWasFound ~= origFormWasFound then searchResultChanged = true end if previousIterationOfSameFormID ~= formToCheckID then iteratedFormIDWasChanged = true end end if iteratedFormIDWasChanged == true then textString = textString.."FormID HAS CHANGED. " wasError = true end if searchResultChanged == true then textString = textString.."Search result HAS CHANGED. " wasError = true end if wasError == true then textString = textString.." (ERROR)" end textString = textString.."\n" iteratedFormIDWasChanged = false searchResultChanged = false wasError = false end AH_Hotkeys.LogMessage(textString, debugLevels.kNotice) end

[RealAntithesis]

There may also be something else going on. Sometimes on loading a save, some objects are empty and show up as nil when accessed (so lots of errors in the console about lua indexing nil objects etc), but the issue goes away when reloading the save again (nothing else changed except reloading the save). This doesn't happen very often though. I don't remember this being an issue on earlier versions of Skyrim/JC.

[ryobg]

You mean it was better with the previous, non-ESL patched version?

[RealAntithesis]

I was thinking pre SSE 1.5.80. I'm going to revert back to the earlier version this week and see if similar issues are happening.

[RealAntithesis]

Reverted the exe and SKSE (2.0.15) (and JC 4.1.8) back to 1.5.73 and it still happens. As an even simpler test, I ran the following in lua:

	local origFormIDErrors = 0
	for n = 1, #objInventoryFormTestArray do
		origFormVar = objInventoryFormTestArray[n]
		origFormIDVar = origFormVar.___id
		for i = 1, 1000 do
			if origFormVar.___id ~= origFormIDVar then
				origFormIDErrors = origFormIDErrors + 1
			end
			formVar = objInventoryFormTestArray[1]
			countIterations = countIterations + 1
		end
	end
	AH_Hotkeys.LogMessage("InventoryFormArrayTest(Test): origFormID errors = "..tostring(origFormIDErrors).."; total iterations = "..tostring(countIterations))

All this does is to access the form variable's .___id field over several iterations and report on the number of times it doesn't match the id picked up prior to the iterations. This works fine over 1000 iterations, until the line "formVar = objInventoryFormTestArray[1]" is added. Then it starts detecting errors - i.e. getting the .___id from the form is somehow resulting in a different value from the value at the start. But this only starts happening when the array is accessed in a nested loop.

Maybe this is symptomatic of something else happening, but it's a bit weird. I tried a nest loop of incrementing a number, and that was fine. Accessing the array without a nested loop with the same line "formVar = objInventoryFormTestArray[1]" was fine. But putting this line into the loop suddenly resulted in the .___id field changing for seemingly random forms in the array (different forms would report errors over different runs in the same game session).

Does this point to some sort of cause (memory issue etc)?

[RealAntithesis]

Another test. In this one, I copy all the forms in the JArray to a lua table, and then compare the .___id fields. As expected by now, some compare the same, other don't. Over 3 iterations in the same game session, focusing on one particular index of both arrays, I got the following (the results are from comparing equality for the form and the other for the .___id field:

1st result for i = 67
67) JArray=371940; lua=1612364288; form result=false; formID result=false

2nd result for i = 67
67) JArray=371940; lua=671103073; form result=false; formID result=false

3rd result for i = 67
67) JArray=371940; lua=671102002; form result=false; formID result=false

The code to copy the JArray to a lua table is fairly simple and direct:

function AH_Hotkeys.JArrayToTable(objArray)
	local t = {}
	for i = 1, #objArray do
		t[i] = objArray[i]
	end
	return t
end

[SilverIce]

@RealAntithesis , i'm lost in assumptions. Any chance the array is modified by another Papyrus thread while iterating through values? If so, can you deepcopy and pass the copy into lua via JValue.deepCopy?

[SilverIce]

It's JC feature to not allow Lua to have state, since I don't save Lua state.
You can store the cache in JDb, in lua

[RealAntithesis]

Do you mean accessing the value using JValue.solvePath(optr, path) in jc.lua rather than through a local variable? The JMap objItemFormCache in the above example was already added to the JDB in papyrus (so should be retained). The intent of the above code was just to access this JMap directly within lua without having to pass it through via the function that uses it.

[SilverIce]

There is a JDB variable in Lua witch points to Papyrus' JMap (a JDB.root). So, you can access via JDB.key.key2... You can even retrieve it from JDB, and then cache it

The intent of the above code was just to access this JMap directly within lua without having to pass it through via the function that uses it.

You can't rely on it.

Lua engine isn't thread-safe. There were 2 ways to solve that - have a single engine, protected by lock and have potential performance issues when more users use Lua (back then I though that was possible). The other (current) way was to have a pool of Lua engines. When 2 Papyrus threads call evalLua in the same time, 2 Lua engines are allocated. That's why sometimes objItemFormCache is still nil - as there are few engines. Only one of them has objItemFormCache assigned.

Other reason why in JC its best for Lua to have no state (no global variables) - users will expect that the state will be saveable

[SilverIce]

If the objects are in JDB already, you can

local objItemFormCache, objItemIDCache = JDB.AH_key.key1, JDB.AH_key.key2

Since this code executed in main body of your init.lua file, all Lua engines/contexts will run it, will have these values initialized

[RealAntithesis]

Yep, that's what I've started doing:
Edit: corrected. Looks like the path has to be spelled out every time.

local objAHHotkeysData = JDB.AHHotkeys_Data
local objItemFormCache = JDB.AHHotkeys_Data.itemFormCache
local objItemIDCache = JDB.AHHotkeys_Data.itemIDCache
local objTempArray = JDB.AHHotkeys_Data.tempArray
local objKeyMap = JDB.AHHotkeys_Data.keyMap
local objInventoryItemArray = JDB.AHHotkeys_Data.inventoryItemArray
local objRawFoodArray = JDB.AHHotkeys_Data.rawFoodArray
local objUserSettings = JDB.AHHotkeys_Data.userSettings
local objUnequipItemExclusionsArray = JDB.AHHotkeys_Data.userSettings.unequipItemExclusionsArray
local objItemsToMatchByNameArray = JDB.AHHotkeys_Data.unequipItemsArray
local bDebugScript = JDB.AHHotkeys_Data.bDebugScript
local debugLevel = JDB.AHHotkeys_Data.userSettings.debugLevel or debugLevels.kNotice

The only issue is what to do with 'bDebugScript' and 'debugLevels' - these weren't originally in the JDB. I've put them all into the JDB and it seems fine now.

[RealAntithesis]

Just going back to the original question raised in this thread re: testing for invalid forms in lua, testing for invalid forms using Form(0) doesn't appear to work.

A formID may become invalid due to a change in load order, so Game.GetFormEx(formID) may give a nil form. However, in this case, testing the now invalid form in lua against Form(0) still results in false, meaning it tests as valid (not equal to Form(0)) when it is not.
At the moment, I'm testing for invalid forms in papyrus, which is slower when walking through hundreds of forms, but seems to work fine.

Would the other method suggested at the top of this thread yield a different result?

[ryobg]

I don't think so. The solution appears the same. When the form is created it is created as invalid (kinda Form(0)) it is just done on the C side. Considering all this threading spaghetti, you can consider checking upon usage, clean up regularly and check in similar manner.

[RealAntithesis]

Thanks - doing it in papyrus seems to work fine for now. I was just wondering if the discussion in the other thread about resolving forms and formIDs using the SKSE function would be relevant here since that's what papyrus GetFormEx() would be doing (which works fine).

[ryobg]

That resolving is used only during loading. The retain/release... well you know. These where the only changes. Or you mean like exposing for calling a function?

[RealAntithesis]

More that there should be a way to determine that a form is no longer valid if the load order changes - so if, for example, a Circlet of Archery in one game session has a formID of 1032235, but then the user changes mods around which causes the formID to change in the next game session to 1694524400, presumably the previous formID should no longer resolve to a valid form? If this is detected, then the form should revert to something like nil or Form(0). At the moment, I can't figure out a way to determine that the form is invalid since it still seems to have the old formID (.___id) and otherwise seems normal in lua and doesn't equal Form(0) - but it does read as zero in papyrus (i.e. the form actually is invalid).
Edit: the above is a bad example. The only thing that changes (should change) is the first byte. The following is an actual example: -1326380209 changed to -1343157425. Test in lua for Form(-1326380209) == Form(0) resulted in false, but papyrus Game.GetFormEx(-1326380209) resulted in 0.

[ryobg]

Then I guess some SKSE function can do that, have to see which one. If you use say JFormMap and call its allKeysPArray in such situation (mod removed), do you get different/filtered result?

[RealAntithesis]

I'll test that out.

I suppose I was assuming that jcontainers would be rebuilding the forms on start up anyway, since the form ID should just be the base form ID (00xxxxxx for normal ESPs and FE000xxx for ESLs and ESPFEs) with the first 2 digits being the plugin load order number for ESPs and 3rd to 5th digit for ESLs and ESPFEs.

Then, the only thing needing to be done on load should be detecting if the originating mod's load order has changed, and then changing the load order digits to match.

As a bonus, if the mod has been removed from the load order or the form removed from the mod, it should return nil for the form (and then reconstituting the form if the mod is added back into the load order).

I think I can do all this from papyrus, it will just be a little slower (I'll need to get a list of all loaded mods using the SKSE function, grab the load order digits from the form ID, store the mod name, load order index and base form IDs separately, detect invalid forms in the next load, and then rebuild the form ID from the new mod load order index and base form ID and get the form for the new form ID). Should work...

[SilverIce]

Why do all of this, if JC already fixes form ID during save file load?

[RealAntithesis]

I suppose that's the thing. I'm coming across invalid forms resulting from changing the load order (adding or removing mods that otherwise have nothing to do with those forms). So part of my question was whether JContainers was supposed to be dealing with that anyway.

[RealAntithesis]

Hmm, hold fire on this. It may be something I'm doing. Let me test things out further.

[SilverIce]

@RealAntithesis
I realized that it can be something wrong either on your side, either on SKSE - form resolving implemented by SKSE function.

[RealAntithesis]

Everything works now, so I'm going ahead with the AH Hotkeys update. The issue immediately above is that I had absentmindedly switched to using formIDs rather than forms (due to the earlier esl issue), so of course they don't update when the load order changes (duh). Switching back to using the form map is working fine now.

Is the next version of JContainers due to be released soon, including the fix to the form comparison issue in JC.Lua (I've already made this change in my version, but nobody else would have it)? Thanks.

[ryobg]

Can you tell me now, after 54 comments, is there something to fix here or not? I was delaying that ESL fix release due to your two questions.

[RealAntithesis]

The only outstanding question was the testing for an invalid form, but I think that had been largely resolved with both the esl fix and the form comparison fix discussed above. I think we should be able to go ahead with the release unless you think there is something further that needs looking at in the code based on the above. Thanks.
Edit : and there was also the issue with the number lengths, but I'm not sure what can be done about that. The form comparison fix made that somewhat less of an issue.

[ryobg]

So far I see like 3 possible improvements (all proposed from @SilverIce):

1. Extend the lifetime of Lua object (Detecting invalid forms in lua #62 (comment))
2. Make more visible the null forms in Lua (Detecting invalid forms in lua #62 (comment))
3. (unsure) Present the integers to Lua as reals (JLua.evalLuaInt issue with long numbers #40 (comment))

[RealAntithesis]

#1 I think was resolved with the change in jc.lua?
#2 seems fine, but I'm not sure how this will affect FormMaps which may suddenly have nil keys or arrays of forms with nil entries (are they removed, or will they have 'holes' when searching through them). An equality comparison against Form(0) should be sufficient (will need further testing to see if this actually works as intended).
#3 If this resolves the issue with lua/JC/papyrus number conversions, then that would be great - if there are no side effects.

[ryobg]

1. Was not resolved in the mainstream.
2. Yeah, Form(0) should be the same. Probably no need to fix it.
3. Technically, the storage size will double, but this is unlikely to be a real problem (puns here - lol). Can't say about other side effects, likely not. An option will be a separate test release.

[RealAntithesis]

I can test a test release if you make one.

[ryobg]

Well... it will take much more time. Apparently with that fix proposed all Lua tests broke. I have to research how this whole thing work before even proposing a change.

[SilverIce]

@ryobg IDK, I had zero issues with Lua CForm fix: SilverIce@255f512

[ryobg]

@SilverIce It's about #40 - allowing 32-bit integers. I tried that 1st fix #40 (comment) and it broke.