Drop-in LuaJIT replacement in place of Lua 5.4

[Lyrth]

Description

This PR replaces vanilla Lua 5.4 (from LuaRaw dep) with LuaJIT, hoping to provide large performance improvements, and advanced functionality with FFI to, e.g., access raw game/process memory, interact with C datatypes, and load and use external C libraries (that does not need to be (re-)built for Lua).

A compatibility layer is also made to emplace Lua 5.4 functions that are not present in Lua 5.2/LuaJIT, some implementing similar functionality, some as workarounds, some as no-ops. So being a drop-in replacement, it probably can't ensure 100% compatibility with the Lua 5.4 API, which may cause issues with UE4SS - but so far I have not seen any (yet).

OpenResty's LuaJIT branch is used instead of upstream to make use of tagged releases (upstream rolling release tag does not refer to a single commit throughout time), and this also probably ensures a bit of stability. Xrepo packages for upstream/openresty LuaJIT exist but they seem to be meant for old versions of LuaJIT 2.1.0b3, so I ported over the scripts and modified it to work with current versions.

Also added __tostring metamethods to UE4SS objects, since printing them with the current LuaJIT replacement does not print as it used to anymore (they only appear like userdata: 0x07ffd3adb33f09a0). Now they should print with more detailed type info (subtype, etc. along with address), see screenshots.

Obviously, Lua 5.3+ syntax won't work anymore, like bitwise operators, and 5.4's <const> markers. (Changed ConsoleCommandsMod to use the bit library from LuaJIT instead). Other than that, existing Lua scripts should work fine.

As mentioned, JIT should help with performance, and FFI would allow for easily loading C libraries without the need to do it through the Lua C API, among other things. An example usage is shown under screenshots, where a UE4SS Lua script creates a local web server using libmicrohttpd via FFI.

So far seems to work with two games I tested this on. UE4SS functions work, script reloading works, callbacks seem to be fine too. But I have done extensive testing for everything Lua, hence:

Important

This PR is a DRAFT as it has not seen extensive testing yet and is likely to break obvious and non-obvious things. Testing would be much appreciated.

Type of change

• Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Haven't done detailed testing for compat aside from using the same Lua mods (base and custom) on the JIT build. UE4SS functions seem to work fine. Tested FFI functionality, and the __tostring metamethods for some UE4SS objects.

Checklist

• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• I have added the necessary description of this PR to the changelog, and I have followed the same format as other entries.
• Any dependent changes have been merged and published in downstream modules.

Screenshots

Screenshots

__tostring metamethods addition - Source

webserver FFI usage demo - Source - (video may be flashing a bit)

lyrth_280924_mkEgdWUvGx.mov

[narknon]

Also added __tostring metamethods to UE4SS objects, since printing them with the current LuaJIT replacement does not print as it used to anymore (they only appear like userdata: 0x07ffd3adb33f09a0). Now they should print with more detailed type info (subtype, etc. along with address), see screenshots.

I'm a bit confused by this. What caused the difference? Do we want more detailed type info when printing to string? Don't we just want the string so we can operate on it as an exact string?

[Lyrth]

What caused the difference?

Lua 5.4 introduced the __name metatable key, which "When it contains a string, may be used by tostring and in error messages." This gets set when lua.new_metatable (from LuaMadeSimple) is called with metatable_name, which contains the type string, and it then gets used when the object is passed to Lua print/tostring/(anything that coerces the type to string). It's what puts the 'UObject' in UObject: 07FFD3ADB33F09A0

Of course, LuaJIT being based on 5.2, does not have that; instead, the __tostring metamethod for it is used to produce similar results.

---

Do we want more detailed type info when printing to string?

It's optional, but as I'm already composing the string anyway to bring back the functionality, putting in more info doesn't sound bad - it's mostly used for debugging after all (i.e., printing them). I don't think anyone would be reliant on its tostring behavior, and if they do, they should just use :type() and :GetAddress() (the address in vanilla Lua UE4SS tostring isn't even the underlying Unreal object's address, it's the address of the Lua object itself - i.e.:

local ksl = StaticFindObject("/Script/Engine.Default__KismetSystemLibrary")
print(tostring(ksl))
print(("%016X"):format(ksl:GetAddress()))

will print different addresses for vanilla UE4SS.)

(P.S. I forgot to mention that this __tostring change will also print the underlying object's address, the same as what :GetAddress() gives, for convenience. I don't think the address of the Lua object itself is useful anyway)

---

Don't we just want the string so we can operate on it as an exact string?

I'm not sure what you're referring to here, but if you're talking about UObject :ToString(), no, that's not touched by this. Its behavior should remain the same even with this change.

[igromanru]

Can you please pull latest main into your branch and resolve conflicts, I would like to test it.

[Lyrth]

Can you please pull latest main into your branch and resolve conflicts, I would like to test it.

Rebased.

[igromanru]

In case you're still working on it. Want to report my initial feedback.
First thing that I've noticed that simply using the build from this branch with all my mods crashes my game (Abiotic Factor) pretty fast.
I'm now trying to enable mods one by one too see what causes it.
Obviously Lua 5.3 and up features doesn't work, for example string.pack, which I used in one of my function, but it's not important for any of my mods yet.
The first thing that I've found is that my mod that runs in an infinite LoopAsync and uses json.lua crashes the game randomly after a while without any crash dumps.
But I couldn't pin point it. It happens one time, another time it doesn't happen. I commented something out, no crash, then I comment it back in, also no crash. Then I restart the game again and it crashes.
I tried to restart the game after each of my code changes to be sure.
For now I'll move to other less complex mod to test them.

EDIT: It looks like the problem might be indeed json.lua related or duo to the overall complexity of the mod. None of my other mods seems to cause any issues and one of them uses an infinite loop as well.
https://github.com/igromanru/JagerCorpseRemover-UE4SS-AF/blob/250d2bee6df970355876331265046748b29ea23b/Scripts/main.lua#L68

EDIT2:
Left the game running while I was AFK with a simple Loop beside other things was running.

LoopAsync(250, function()
    ExecuteInGameThread(function() 
        local myPlayer = AFUtils.GetMyPlayer()
        if myPlayer then
            
        end
    end)
    return false
end)

After a while I got error:

[18:01:42.4137924] Error: [Lua::call_function] lua_pcall returned [process_lua_function] The lua state '0x1b130ccb788' has no instance inside lua_instances unordered map
stack traceback:
	[C]: in function '__index'
	...in64\ue4ss\Mods\MyPlayground\Scripts\AFUtils\AFUtils.lua:334: in function 'GetMyPlayer'
	...\Binaries\Win64\ue4ss\Mods\MyPlayground\scripts\main.lua:323: in function <...\Binaries\Win64\ue4ss\Mods\MyPlayground\scripts\main.lua:322>

AFUtils.lua:334 is if PlayerCache and PlayerCache:IsValid() then
This never happened before with normal UE4SS.

The JIT version will need a LOT of testing to make sure that nothing breaks on the lua stack.