From 055ec0a4aff7adb988eaa0288d27a6bbe0e03566 Mon Sep 17 00:00:00 2001 From: Barinade Date: Fri, 22 Oct 2021 19:22:06 -0500 Subject: [PATCH] Update luaJIT to bfcb74ed36d733bae249fe7642b7324a56be88d0 --- extern/luajit/doc/ext_buffer.html | 528 ++++++++++++++++++++--- extern/luajit/doc/extensions.html | 4 +- extern/luajit/doc/faq.html | 18 +- extern/luajit/dynasm/dasm_arm.h | 7 +- extern/luajit/dynasm/dasm_arm.lua | 6 +- extern/luajit/dynasm/dasm_arm64.h | 58 ++- extern/luajit/dynasm/dasm_arm64.lua | 135 ++++-- extern/luajit/dynasm/dasm_mips.h | 7 +- extern/luajit/dynasm/dasm_mips.lua | 6 +- extern/luajit/dynasm/dasm_ppc.h | 7 +- extern/luajit/dynasm/dasm_ppc.lua | 6 +- extern/luajit/dynasm/dasm_proto.h | 4 +- extern/luajit/dynasm/dasm_x86.h | 24 +- extern/luajit/dynasm/dasm_x86.lua | 50 ++- extern/luajit/dynasm/dynasm.lua | 6 +- extern/luajit/src/Makefile | 7 +- extern/luajit/src/Makefile.dep | 44 +- extern/luajit/src/host/buildvm_lib.c | 2 + extern/luajit/src/host/minilua.c | 2 +- extern/luajit/src/jit/dis_arm64.lua | 2 +- extern/luajit/src/jit/dump.lua | 28 +- extern/luajit/src/jit/p.lua | 1 + extern/luajit/src/lib_base.c | 22 +- extern/luajit/src/lib_buffer.c | 330 +++++++++++++- extern/luajit/src/lib_ffi.c | 1 + extern/luajit/src/lib_io.c | 37 +- extern/luajit/src/lib_jit.c | 6 +- extern/luajit/src/lib_package.c | 3 +- extern/luajit/src/lib_string.c | 85 +--- extern/luajit/src/lib_table.c | 2 +- extern/luajit/src/lj_api.c | 8 +- extern/luajit/src/lj_asm.c | 124 ++++-- extern/luajit/src/lj_asm_arm.h | 127 ++++-- extern/luajit/src/lj_asm_arm64.h | 113 +++-- extern/luajit/src/lj_asm_mips.h | 184 +++++--- extern/luajit/src/lj_asm_ppc.h | 128 ++++-- extern/luajit/src/lj_asm_x86.h | 148 ++++--- extern/luajit/src/lj_bcread.c | 16 +- extern/luajit/src/lj_bcwrite.c | 22 +- extern/luajit/src/lj_buf.c | 191 +++++--- extern/luajit/src/lj_buf.h | 140 ++++-- extern/luajit/src/lj_ccall.c | 8 +- extern/luajit/src/lj_ccallback.c | 9 +- extern/luajit/src/lj_cconv.c | 5 + extern/luajit/src/lj_clib.c | 8 +- extern/luajit/src/lj_cparse.c | 8 +- extern/luajit/src/lj_crecord.c | 34 +- extern/luajit/src/lj_crecord.h | 5 + extern/luajit/src/lj_ctype.c | 2 +- extern/luajit/src/lj_ctype.h | 11 + extern/luajit/src/lj_debug.c | 2 +- extern/luajit/src/lj_dispatch.c | 8 +- extern/luajit/src/lj_emit_arm64.h | 2 +- extern/luajit/src/lj_emit_mips.h | 2 +- extern/luajit/src/lj_err.c | 38 +- extern/luajit/src/lj_errmsg.h | 4 + extern/luajit/src/lj_ffrecord.c | 394 ++++++++++++++++- extern/luajit/src/lj_fopen.c | 48 --- extern/luajit/src/lj_fopen.h | 14 - extern/luajit/src/lj_frame.h | 12 +- extern/luajit/src/lj_gc.c | 9 + extern/luajit/src/lj_ir.c | 1 + extern/luajit/src/lj_ir.h | 28 +- extern/luajit/src/lj_ircall.h | 20 + extern/luajit/src/lj_iropt.h | 1 + extern/luajit/src/lj_jit.h | 6 +- extern/luajit/src/lj_lex.c | 16 +- extern/luajit/src/lj_lib.c | 56 +++ extern/luajit/src/lj_lib.h | 6 + extern/luajit/src/lj_load.c | 3 +- extern/luajit/src/lj_mcode.c | 9 +- extern/luajit/src/lj_meta.c | 13 +- extern/luajit/src/lj_obj.h | 19 +- extern/luajit/src/lj_opt_fold.c | 86 +++- extern/luajit/src/lj_opt_mem.c | 33 +- extern/luajit/src/lj_opt_split.c | 2 +- extern/luajit/src/lj_parse.c | 6 +- extern/luajit/src/lj_prng.c | 4 +- extern/luajit/src/lj_profile.c | 5 +- extern/luajit/src/lj_record.c | 173 +++++++- extern/luajit/src/lj_record.h | 2 + extern/luajit/src/lj_serialize.c | 329 +++++++++++--- extern/luajit/src/lj_serialize.h | 11 +- extern/luajit/src/lj_snap.c | 62 ++- extern/luajit/src/lj_strfmt.c | 210 +++++++-- extern/luajit/src/lj_strfmt.h | 5 + extern/luajit/src/lj_strfmt_num.c | 2 +- extern/luajit/src/lj_tab.c | 79 ++-- extern/luajit/src/lj_tab.h | 26 +- extern/luajit/src/lj_target_arm.h | 1 + extern/luajit/src/lj_target_arm64.h | 4 + extern/luajit/src/lj_target_mips.h | 2 + extern/luajit/src/lj_target_x86.h | 3 +- extern/luajit/src/lj_trace.c | 37 +- extern/luajit/src/lj_udata.c | 3 +- extern/luajit/src/lj_vm.h | 2 + extern/luajit/src/msvcbuild.bat | 2 +- extern/luajit/src/msvcbuild_utf8open.bat | 127 ------ extern/luajit/src/ps4build.bat | 2 +- extern/luajit/src/psvitabuild.bat | 2 +- extern/luajit/src/vm_arm.dasc | 111 ++++- extern/luajit/src/vm_arm64.dasc | 316 ++++++++++---- extern/luajit/src/vm_mips.dasc | 139 ++++-- extern/luajit/src/vm_mips64.dasc | 132 ++++-- extern/luajit/src/vm_ppc.dasc | 70 ++- extern/luajit/src/vm_x64.dasc | 134 ++++-- extern/luajit/src/vm_x86.dasc | 165 +++++-- extern/luajit/src/xb1build.bat | 202 ++++----- extern/luajit/src/xedkbuild.bat | 2 +- 109 files changed, 4360 insertions(+), 1571 deletions(-) delete mode 100644 extern/luajit/src/lj_fopen.c delete mode 100644 extern/luajit/src/lj_fopen.h delete mode 100644 extern/luajit/src/msvcbuild_utf8open.bat diff --git a/extern/luajit/doc/ext_buffer.html b/extern/luajit/doc/ext_buffer.html index 455c298d62..63c2efe330 100644 --- a/extern/luajit/doc/ext_buffer.html +++ b/extern/luajit/doc/ext_buffer.html @@ -1,19 +1,30 @@ -String Buffers +String Buffer Library +
Lua

- The string buffer library allows high-performance manipulation of string-like data. -

- Unlike Lua strings, which are constants, string buffers are mutable sequences of 8-bit (binary-transparent) characters. Data can be stored, formatted and encoded into a string buffer and later -converted, decoded or extracted. - +converted, extracted or decoded.

- The convenient string buffer API simplifies common string manipulation tasks, that would otherwise require creating many intermediate strings. String buffers improve performance by eliminating redundant memory copies, object creation, string interning and garbage collection overhead. In conjunction with the FFI library, they allow zero-copy operations. +

+

+The string buffer libary also includes a high-performance +serializer for Lua objects. +

+

Work in Progress

+

+This library is a work in progress. More +functionality will be added soon.

-

Using the String Buffer Library

+

Using the String Buffer Library

The string buffer library is built into LuaJIT by default, but it's not loaded by default. Add this to the start of every Lua file that needs @@ -90,142 +105,476 @@

Using the String Buffer Library

 local buffer = require("string.buffer")
+

+The convention for the syntax shown on this page is that buffer +refers to the buffer library and buf refers to an individual +buffer object. +

+

+Please note the difference between a Lua function call, e.g. +buffer.new() (with a dot) and a Lua method call, e.g. +buf:reset() (with a colon). +

-

Work in Progress

+

Buffer Objects

+

+A buffer object is a garbage-collected Lua object. After creation with +buffer.new(), it can (and should) be reused for many operations. +When the last reference to a buffer object is gone, it will eventually +be freed by the garbage collector, along with the allocated buffer +space. +

+

+Buffers operate like a FIFO (first-in first-out) data structure. Data +can be appended (written) to the end of the buffer and consumed (read) +from the front of the buffer. These operations may be freely mixed. +

+

+The buffer space that holds the characters is managed automatically +— it grows as needed and already consumed space is recycled. Use +buffer.new(size) and buf:free(), if you need more +control. +

+

+The maximum size of a single buffer is the same as the maximum size of a +Lua string, which is slightly below two gigabytes. For huge data sizes, +neither strings nor buffers are the right data structure — use the +FFI library to directly map memory or files up to the virtual memory +limit of your OS. +

+

Buffer Method Overview

+ + +

Buffer Creation and Management

+ +

local buf = buffer.new([size [,options]])
+local buf = buffer.new([options])

+

+Creates a new buffer object. +

+The optional size argument ensures a minimum initial buffer +size. This is strictly an optimization when the required buffer size is +known beforehand. The buffer space will grow as needed, in any case. +

+

+The optional table options sets various +serialization options. +

-This library is a work in progress. More -functions will be added soon. +

buf = buf:reset()

+

+Reset (empty) the buffer. The allocated buffer space is not freed and +may be reused. +

+

buf = buf:free()

+

+The buffer space of the buffer object is freed. The object itself +remains intact, empty and may be reused. +

+

+Note: you normally don't need to use this method. The garbage collector +automatically frees the buffer space, when the buffer object is +collected. Use this method, if you need to free the associated memory +immediately.

-

Serialization of Lua Objects

+

Buffer Writers

+ +

buf = buf:put([str|num|obj] [,…])

+

+Appends a string str, a number num or any object +obj with a __tostring metamethod to the buffer. +Multiple arguments are appended in the given order. +

+

+Appending a buffer to a buffer is possible and short-circuited +internally. But it still involves a copy. Better combine the buffer +writes to use a single buffer. +

+ +

buf = buf:putf(format, …)

+

+Appends the formatted arguments to the buffer. The format +string supports the same options as string.format(). +

+ +

buf = buf:putcdata(cdata, len)FFI

+

+Appends the given len number of bytes from the memory pointed +to by the FFI cdata object to the buffer. The object needs to +be convertible to a (constant) pointer. +

+ +

buf = buf:set(str)
+buf = buf:set(cdata, len)FFI

+

+This method allows zero-copy consumption of a string or an FFI cdata +object as a buffer. It stores a reference to the passed string +str or the FFI cdata object in the buffer. Any buffer +space originally allocated is freed. This is not an append +operation, unlike the buf:put*() methods. +

+

+After calling this method, the buffer behaves as if +buf:free():put(str) or buf:free():put(cdata, len) +had been called. However, the data is only referenced and not copied, as +long as the buffer is only consumed. +

+

+In case the buffer is written to later on, the referenced data is copied +and the object reference is removed (copy-on-write semantics). +

+

+The stored reference is an anchor for the garbage collector and keeps the +originally passed string or FFI cdata object alive. +

+ +

ptr, len = buf:reserve(size)FFI
+buf = buf:commit(used)FFI

+

+The reserve method reserves at least size bytes of +write space in the buffer. It returns an uint8_t * FFI +cdata pointer ptr that points to this space. +

+

+The available length in bytes is returned in len. This is at +least size bytes, but may be more to facilitate efficient +buffer growth. You can either make use of the additional space or ignore +len and only use size bytes. +

+

+The commit method appends the used bytes of the +previously returned write space to the buffer data. +

+

+This pair of methods allows zero-copy use of C read-style APIs: +

+
+local MIN_SIZE = 65536
+repeat
+  local ptr, len = buf:reserve(MIN_SIZE)
+  local n = C.read(fd, ptr, len)
+  if n == 0 then break end -- EOF.
+  if n < 0 then error("read error") end
+  buf:commit(n)
+until false
+
+

+The reserved write space is not initialized. At least the +used bytes must be written to before calling the +commit method. There's no need to call the commit +method, if nothing is added to the buffer (e.g. on error). +

+ +

Buffer Readers

+ +

len = #buf

+

+Returns the current length of the buffer data in bytes. +

+ +

res = str|num|buf .. str|num|buf […]

+

+The Lua concatenation operator .. also accepts buffers, just +like strings or numbers. It always returns a string and not a buffer. +

+

+Note that although this is supported for convenience, this thwarts one +of the main reasons to use buffers, which is to avoid string +allocations. Rewrite it with buf:put() and buf:get(). +

+

+Mixing this with unrelated objects that have a __concat +metamethod may not work, since these probably only expect strings. +

+ +

buf = buf:skip(len)

+

+Skips (consumes) len bytes from the buffer up to the current +length of the buffer data. +

+ +

str, … = buf:get([len|nil] [,…])

+

+Consumes the buffer data and returns one or more strings. If called +without arguments, the whole buffer data is consumed. If called with a +number, up to len bytes are consumed. A nil argument +consumes the remaining buffer space (this only makes sense as the last +argument). Multiple arguments consume the buffer data in the given +order. +

+

+Note: a zero length or no remaining buffer data returns an empty string +and not nil. +

+ +

str = buf:tostring()
+str = tostring(buf)

+

+Creates a string from the buffer data, but doesn't consume it. The +buffer remains unchanged. +

+Buffer objects also define a __tostring metamethod. This means +buffers can be passed to the global tostring() function and +many other functions that accept this in place of strings. The important +internal uses in functions like io.write() are short-circuited +to avoid the creation of an intermediate string object. +

+ +

ptr, len = buf:ref()FFI

+

+Returns an uint8_t * FFI cdata pointer ptr that +points to the buffer data. The length of the buffer data in bytes is +returned in len. +

+

+The returned pointer can be directly passed to C functions that expect a +buffer and a length. You can also do bytewise reads +(local x = ptr[i]) or writes +(ptr[i] = 0x40) of the buffer data. +

+

+In conjunction with the skip method, this allows zero-copy use +of C write-style APIs: +

+
+repeat
+  local ptr, len = buf:ref()
+  if len == 0 then break end
+  local n = C.write(fd, ptr, len)
+  if n < 0 then error("write error") end
+  buf:skip(n)
+until n >= len
+
+

+Unlike Lua strings, buffer data is not implicitly +zero-terminated. It's not safe to pass ptr to C functions that +expect zero-terminated strings. If you're not using len, then +you're doing something wrong. +

+

Serialization of Lua Objects

+

The following functions and methods allow high-speed serialization (encoding) of a Lua object into a string and decoding it back to a Lua object. This allows convenient storage and transport of structured data. -

- The encoded data is in an internal binary format. The data can be stored in files, binary-transparent databases or transmitted to other LuaJIT instances across threads, processes or networks. -

- Encoding speed can reach up to 1 Gigabyte/second on a modern desktop- or server-class system, even when serializing many small objects. Decoding speed is mostly constrained by object creation cost. -

- The serializer handles most Lua types, common FFI number types and -nested structures. Functions, thread objects, other FFI cdata, full -userdata and associated metatables cannot be serialized (yet). - +nested structures. Functions, thread objects, other FFI cdata and full +userdata cannot be serialized (yet).

- The encoder serializes nested structures as trees. Multiple references to a single object will be stored separately and create distinct objects after decoding. Circular references cause an error. - -

-

str = buffer.encode(obj)

-

- -Serializes (encodes) the Lua object obj into the string -str. +

Serialization Functions and Methods

+

str = buffer.encode(obj)
+buf = buf:encode(obj)

+

+Serializes (encodes) the Lua object obj. The stand-alone +function returns a string str. The buffer method appends the +encoding to the buffer.

- obj can be any of the supported Lua types — it doesn't need to be a Lua table. -

- This function may throw an error when attempting to serialize unsupported object types, circular references or deeply nested tables. -

-

obj = buffer.decode(str)

+

obj = buffer.decode(str)
+obj = buf:decode()

- -De-serializes (decodes) the string str into the Lua object -obj. - +The stand-alone function de-serializes (decodes) the string +str, the buffer method de-serializes one object from the +buffer. Both return a Lua object obj.

- The returned object may be any of the supported Lua types — even nil. -

- This function may throw an error when fed with malformed or incomplete -encoded data. The standalone function throws when there's left-over data -after decoding a single top-level object. +encoded data. The stand-alone function throws when there's left-over +data after decoding a single top-level object. The buffer method leaves +any left-over data in the buffer. +

+

Serialization Options

+

+The options table passed to buffer.new() may contain +the following members (all optional): +

+ +

+dict needs to be an array of strings and metatable needs +to be an array of tables. Both starting at index 1 and without holes (no +nil inbetween). The tables are anchored in the buffer object and +internally modified into a two-way index (don't do this yourself, just pass +a plain array). The tables must not be modified after they have been passed +to buffer.new(). +

+

+The dict and metatable tables used by the encoder and +decoder must be the same. Put the most common entries at the front. Extend +at the end to ensure backwards-compatibility — older encodings can +then still be read. You may also set some indexes to false to +explicitly drop backwards-compatibility. Old encodings that use these +indexes will throw an error when decoded. +

+

+Metatables that are not found in the metatable dictionary are +ignored when encoding. Decoding returns a table with a nil +metatable. +

+

+Note: parsing and preparation of the options table is somewhat +expensive. Create a buffer object only once and recycle it for multiple +uses. Avoid mixing encoder and decoder buffers, since the +buf:set() method frees the already allocated buffer space:

+
+local options = {
+  dict = { "commonly", "used", "string", "keys" },
+}
+local buf_enc = buffer.new(options)
+local buf_dec = buffer.new(options)
+
+local function encode(obj)
+  return buf_enc:reset():encode(obj):get()
+end
+
+local function decode(str)
+  return buf_dec:set(str):decode()
+end
+
-

Serialization Format Specification

+

Streaming Serialization

+

+In some contexts, it's desirable to do piecewise serialization of large +datasets, also known as streaming. +

+

+This serialization format can be safely concatenated and supports streaming. +Multiple encodings can simply be appended to a buffer and later decoded +individually: +

+
+local buf = buffer.new()
+buf:encode(obj1)
+buf:encode(obj2)
+local copy1 = buf:decode()
+local copy2 = buf:decode()
+
+

+Here's how to iterate over a stream: +

+
+while #buf ~= 0 do
+  local obj = buf:decode()
+  -- Do something with obj.
+end
+

+Since the serialization format doesn't prepend a length to its encoding, +network applications may need to transmit the length, too. +

+

Serialization Format Specification

+

This serialization format is designed for internal use by LuaJIT applications. Serialized data is upwards-compatible and portable across all supported LuaJIT platforms. -

- It's an 8-bit binary format and not human-readable. It uses e.g. embedded zeroes and stores embedded Lua string objects unmodified, which are 8-bit-clean, too. Encoded data can be safely concatenated for streaming and later decoded one top-level object at a time. -

- The encoding is reasonably compact, but tuned for maximum performance, not for minimum space usage. It compresses well with any of the common byte-oriented data compression algorithms. -

- Although documented here for reference, this format is explicitly not intended to be a 'public standard' for structured data interchange across computer languages (like JSON or MessagePack). Please do not use it as such. -

- The specification is given below as a context-free grammar with a top-level object as the starting point. Alternatives are separated by the | symbol and * indicates repeats. Grouping is implicit or indicated by {…}. Terminals are either plain hex numbers, encoded as bytes, or have a .format suffix. - 

 object    → nil | false | true
           | null | lightud32 | lightud64
-          | int | num | tab
+          | int | num | tab | tab_mt
           | int64 | uint64 | complex
           | string
 
@@ -246,12 +595,14 @@ 
Serialization Format Specification

           | 0x0b a.U a*object h.U h*{object object}      // Mixed
           | 0x0c a.U (a-1)*object                // 1-based array
           | 0x0d a.U (a-1)*object h.U h*{object object}  // Mixed
+tab_mt    → 0x0e (index-1).U tab          // Metatable dict entry
 
 int64     → 0x10 int.L                             // FFI int64_t
 uint64    → 0x11 uint.L                           // FFI uint64_t
 complex   → 0x12 re.L im.L                         // FFI complex
 
 string    → (0x20+len).U len*char.B
+          | 0x0f (index-1).U                 // String dict entry
 
 .B = 8 bit
 .I = 32 bit little-endian
@@ -261,6 +612,73 @@ 
Serialization Format Specification

      0xe0..0x1fdf → (0xe0|(((n-0xe0)>>8)&0x1f)).B ((n-0xe0)&0xff).B
    0x1fe0..       → 0xff n.I
+ +

Error handling

+

+Many of the buffer methods can throw an error. Out-of-memory or usage +errors are best caught with an outer wrapper for larger parts of code. +There's not much one can do after that, anyway. +

+

+OTOH you may want to catch some errors individually. Buffer methods need +to receive the buffer object as the first argument. The Lua colon-syntax +obj:method() does that implicitly. But to wrap a method with +pcall(), the arguments need to be passed like this: +

+
+local ok, err = pcall(buf.encode, buf, obj)
+if not ok then
+  -- Handle error in err.
+end
+
+ +

FFI caveats

+

+The string buffer library has been designed to work well together with +the FFI library. But due to the low-level nature of the FFI library, +some care needs to be taken: +

+

+First, please remember that FFI pointers are zero-indexed. The space +returned by buf:reserve() and buf:ref() starts at the +returned pointer and ends before len bytes after that. +

+

+I.e. the first valid index is ptr[0] and the last valid index +is ptr[len-1]. If the returned length is zero, there's no valid +index at all. The returned pointer may even be NULL. +

+

+The space pointed to by the returned pointer is only valid as long as +the buffer is not modified in any way (neither append, nor consume, nor +reset, etc.). The pointer is also not a GC anchor for the buffer object +itself. +

+

+Buffer data is only guaranteed to be byte-aligned. Casting the returned +pointer to a data type with higher alignment may cause unaligned +accesses. It depends on the CPU architecture whether this is allowed or +not (it's always OK on x86/x64 and mostly OK on other modern +architectures). +

+

+FFI pointers or references do not count as GC anchors for an underlying +object. E.g. an array allocated with ffi.new() is +anchored by buf:set(array, len), but not by +buf:set(array+offset, len). The addition of the offset +creates a new pointer, even when the offset is zero. In this case, you +need to make sure there's still a reference to the original array as +long as its contents are in use by the buffer. +

+

+Even though each LuaJIT VM instance is single-threaded (but you can +create multiple VMs), FFI data structures can be accessed concurrently. +Be careful when reading/writing FFI cdata from/to buffers to avoid +concurrent accesses or modifications. In particular, the memory +referenced by buf:set(cdata, len) must not be modified +while buffer readers are working on it. Shared, but read-only memory +mappings of files are OK, but only if the file does not change. +