LuaJIT FFI Tutorial
This page is intended to give you an overview of the features of the FFI library by presenting a few use cases and guidelines.
This page makes no attempt to explain all of the FFI library, though. You'll want to have a look at the ffi.* API function reference and the FFI semantics to learn more.
Loading the FFI Library
The FFI library is built into LuaJIT by default, but it's not loaded and initialized by default. The suggested way to use the FFI library is to add the following to the start of every Lua file that needs one of its functions:
local ffi = require("ffi")
Please note this doesn't define an ffi variable in the table of globals — you really need to use the local variable. The require function ensures the library is only loaded once.
Note: If you want to experiment with the FFI from the interactive prompt of the command line executable, omit the local, as it doesn't preserve local variables across lines.
Accessing Standard System Functions
The following code explains how to access standard system functions. We slowly print two lines of dots by sleeping for 10 milliseconds after each dot:
local ffi = require("ffi")
ffi.cdef[[
void Sleep(int ms);
int poll(struct pollfd *fds, unsigned long nfds, int timeout);
]]
local sleep
if ffi.os == "Windows" then
function sleep(s)
ffi.C.Sleep(s*1000)
end
else
function sleep(s)
ffi.C.poll(nil, 0, s*1000)
end
end
for i=1,160 do
io.write("."); io.flush()
sleep(0.01)
end
io.write("\n")
Here's the step-by-step explanation:
1.This defines the C library functions we're going to use. The part inside the double-brackets (in green) is just standard C syntax. You can usually get this info from the C header files or the documentation provided by each C library or C compiler.
2.The difficulty we're facing here, is that there are different standards to choose from. Windows has a simple Sleep() function. On other systems there are a variety of functions available to achieve sub-second sleeps, but with no clear consensus. Thankfully poll() can be used for this task, too, and it's present on most non-Windows systems. The check for ffi.os makes sure we use the Windows-specific function only on Windows systems.
3.Here we're wrapping the call to the C function in a Lua function. This isn't strictly necessary, but it's helpful to deal with system-specific issues only in one part of the code. The way we're wrapping it ensures the check for the OS is only done during initialization and not for every call.
4.A more subtle point is that we defined our sleep() function (for the sake of this example) as taking the number of seconds, but accepting fractional seconds. Multiplying this by 1000 gets us milliseconds, but that still leaves it a Lua number, which is a floating-point value. Alas, the Sleep() function only accepts an integer value. Luckily for us, the FFI library automatically performs the conversion when calling the function (truncating the FP value towards zero, like in C).
Some readers will notice that Sleep() is part of KERNEL32.DLL and is also a stdcall function. So how can this possibly work? The FFI library provides the ffi.C default C library namespace, which allows calling functions from the default set of libraries, like a C compiler would. Also, the FFI library automatically detects stdcall functions, so you don't need to declare them as such.
5.The poll() function takes a couple more arguments we're not going to use. You can simply use nil to pass a NULL pointer and 0 for the nfds parameter. Please note that the number 0 does not convert to a pointer value, unlike in C++. You really have to pass pointers to pointer arguments and numbers to number arguments.
The page on FFI semantics has all of the gory details about conversions between Lua objects and C types. For the most part you don't have to deal with this, as it's performed automatically and it's carefully designed to bridge the semantic differences between Lua and C.
6.Now that we have defined our own sleep() function, we can just call it from plain Lua code. That wasn't so bad, huh? Turning these boring animated dots into a fascinating best-selling game is left as an exercise for the reader. :-)
Accessing the zlib Compression Library
The following code shows how to access the zlib compression library from Lua code. We'll define two convenience wrapper functions that take a string and compress or uncompress it to another string:
local ffi = require("ffi")
ffi.cdef[[
unsigned long compressBound(unsigned long sourceLen);
int compress2(uint8_t *dest, unsigned long *destLen,
const uint8_t *source, unsigned long sourceLen, int level);
int uncompress(uint8_t *dest, unsigned long *destLen,
const uint8_t *source, unsigned long sourceLen);
]]
local zlib = ffi.load(ffi.os == "Windows" and "zlib1" or "z")
local function compress(txt)
local n = zlib.compressBound(#txt)
local buf = ffi.new("uint8_t[?]", n)
local buflen = ffi.new("unsigned long[1]", n)
local res = zlib.compress2(buf, buflen, txt, #txt, 9)
assert(res == 0)
return ffi.string(buf, buflen[0])
end
local function uncompress(comp, n)
local buf = ffi.new("uint8_t[?]", n)
local buflen = ffi.new("unsigned long[1]", n)
local res = zlib.uncompress(buf, buflen, comp, #comp)
assert(res == 0)
return ffi.string(buf, buflen[0])
end
-- Simple test code.
local txt = string.rep("abcd", 1000)
print("Uncompressed size: ", #txt)
local c = compress(txt)
print("Compressed size: ", #c)
local txt2 = uncompress(c, #txt)
assert(txt2 == txt)
Here's the step-by-step explanation:
① This defines some of the C functions provided by zlib. For the sake of this example, some type indirections have been reduced and it uses the pre-defined fixed-size integer types, while still adhering to the zlib API/ABI.
② This loads the zlib shared library. On POSIX systems it's named libz.so and usually comes pre-installed. Since ffi.load() automatically adds any missing standard prefixes/suffixes, we can simply load the "z" library. On Windows it's named zlib1.dll and you'll have to download it first from the zlib site. The check for ffi.os makes sure we pass the right name to ffi.load().
③ First, the maximum size of the compression buffer is obtained by calling the zlib.compressBound function with the length of the uncompressed string. The next line allocates a byte buffer of this size. The [?] in the type specification indicates a variable-length array (VLA). The actual number of elements of this array is given as the 2nd argument to ffi.new().
④ This may look strange at first, but have a look at the declaration of the compress2 function from zlib: the destination length is defined as a pointer! This is because you pass in the maximum buffer size and get back the actual length that was used.
In C you'd pass in the address of a local variable (&buflen). But since there's no address-of operator in Lua, we'll just pass in a one-element array. Conveniently it can be initialized with the maximum buffer size in one step. Calling the actual zlib.compress2 function is then straightforward.
⑤ We want to return the compressed data as a Lua string, so we'll use ffi.string(). It needs a pointer to the start of the data and the actual length. The length has been returned in the buflen array, so we'll just get it from there.
Note that since the function returns now, the buf and buflen variables will eventually be garbage collected. This is fine, because ffi.string() has copied the contents to a newly created (interned) Lua string. If you plan to call this function lots of times, consider reusing the buffers and/or handing back the results in buffers instead of strings. This will reduce the overhead for garbage collection and string interning.
⑥ The uncompress functions does the exact opposite of the compress function. The compressed data doesn't include the size of the original string, so this needs to be passed in. Otherwise no surprises here.
⑦ The code, that makes use of the functions we just defined, is just plain Lua code. It doesn't need to know anything about the LuaJIT FFI — the convenience wrapper functions completely hide it.
One major advantage of the LuaJIT FFI is that you are now able to write those wrappers in Lua. And at a fraction of the time it would cost you to create an extra C module using the Lua/C API. Many of the simpler C functions can probably be used directly from your Lua code, without any wrappers.
Side note: the zlib API uses the long type for passing lengths and sizes around. But all those zlib functions actually only deal with 32 bit values. This is an unfortunate choice for a public API, but may be explained by zlib's history — we'll just have to deal with it.
First, you should know that a long is a 64 bit type e.g. on POSIX/x64 systems, but a 32 bit type on Windows/x64 and on 32 bit systems. Thus a long result can be either a plain Lua number or a boxed 64 bit integer cdata object, depending on the target system.
Ok, so the ffi.* functions generally accept cdata objects wherever you'd want to use a number. That's why we get a away with passing n to ffi.string() above. But other Lua library functions or modules don't know how to deal with this. So for maximum portability one needs to use tonumber() on returned long results before passing them on. Otherwise the application might work on some systems, but would fail in a POSIX/x64 environment.
Defining Metamethods for a C Type
The following code explains how to define metamethods for a C type. We define a simple point type and add some operations to it:
①
②
③
④
⑤
⑥local ffi = require("ffi")
ffi.cdef[[
typedef struct { double x, y; } point_t;
]]
local point
local mt = {
__add = function(a, b) return point(a.x+b.x, a.y+b.y) end,
__len = function(a) return math.sqrt(a.x*a.x + a.y*a.y) end,
__index = {
area = function(a) return a.x*a.x + a.y*a.y end,
},
}
point = ffi.metatype("point_t", mt)
local a = point(3, 4)
print(a.x, a.y) --> 3 4
print(#a) --> 5
print(a:area()) --> 25
local b = a + point(0.5, 8)
print(#b) --> 12.5
Here's the step-by-step explanation:
① This defines the C type for a two-dimensional point object.
② We have to declare the variable holding the point constructor first, because it's used inside of a metamethod.
③ Let's define an __add metamethod which adds the coordinates of two points and creates a new point object. For simplicity, this function assumes that both arguments are points. But it could be any mix of objects, if at least one operand is of the required type (e.g. adding a point plus a number or vice versa). Our __len metamethod returns the distance of a point to the origin.
④ If we run out of operators, we can define named methods, too. Here the __index table defines an area function. For custom indexing needs, one might want to define __index and __newindex functions instead.
⑤ This associates the metamethods with our C type. This only needs to be done once. For convenience, a constructor is returned by ffi.metatype(). We're not required to use it, though. The original C type can still be used e.g. to create an array of points. The metamethods automatically apply to any and all uses of this type.
Please note that the association with a metatable is permanent and the metatable must not be modified afterwards! Ditto for the __index table.
⑥ Here are some simple usage examples for the point type and their expected results. The pre-defined operations (such as a.x) can be freely mixed with the newly defined metamethods. Note that area is a method and must be called with the Lua syntax for methods: a:area(), not a.area().
The C type metamethod mechanism is most useful when used in conjunction with C libraries that are written in an object-oriented style. Creators return a pointer to a new instance and methods take an instance pointer as the first argument. Sometimes you can just point __index to the library namespace and __gc to the destructor and you're done. But often enough you'll want to add convenience wrappers, e.g. to return actual Lua strings or when returning multiple values.
Some C libraries only declare instance pointers as an opaque void * type. In this case you can use a fake type for all declarations, e.g. a pointer to a named (incomplete) struct will do: typedef struct foo_type *foo_handle. The C side doesn't know what you declare with the LuaJIT FFI, but as long as the underlying types are compatible, everything still works.
Translating C Idioms
Here's a list of common C idioms and their translation to the LuaJIT FFI:
Idiom C code Lua code
Pointer dereference
int *p; x = *p;
*p = y; x = p[0]
p[0] = y
Pointer indexing
int i, *p; x = p[i];
p[i+1] = y; x = p[i]
p[i+1] = y
Array indexing
int i, a[]; x = a[i];
a[i+1] = y; x = a[i]
a[i+1] = y
struct/union dereference
struct foo s; x = s.field;
s.field = y; x = s.field
s.field = y
struct/union pointer deref.
struct foo *sp; x = sp->field;
sp->field = y; x = s.field
s.field = y
Pointer arithmetic
int i, *p; x = p + i;
y = p - i; x = p + i
y = p - i
Pointer difference
int *p1, *p2; x = p1 - p2; x = p1 - p2
Array element pointer
int i, a[]; x = &a[i]; x = a+i
Cast pointer to address
int *p; x = (intptr_t)p; x = tonumber(
ffi.cast("intptr_t",
p))
Functions with outargs
void foo(int *inoutlen); int len = x;
foo(&len);
y = len; local len =
ffi.new("int[1]", x)
foo(len)
y = len[0]
Vararg conversions
int printf(char *fmt, ...); printf("%g", 1.0);
printf("%d", 1);
printf("%g", 1);
printf("%d",
ffi.new("int", 1))
To Cache or Not to Cache
It's a common Lua idiom to cache library functions in local variables or upvalues, e.g.:
local byte, char = string.byte, string.char
local function foo(x)
return char(byte(x)+1)
end
This replaces several hash-table lookups with a (faster) direct use of a local or an upvalue. This is less important with LuaJIT, since the JIT compiler optimizes hash-table lookups a lot and is even able to hoist most of them out of the inner loops. It can't eliminate all of them, though, and it saves some typing for often-used functions. So there's still a place for this, even with LuaJIT.
The situation is a bit different with C function calls via the FFI library. The JIT compiler has special logic to eliminate all of the lookup overhead for functions resolved from a C library namespace! Thus it's not helpful and actually counter-productive to cache individual C functions like this:
local funca, funcb = ffi.C.funcb, ffi.C.funcb -- Not helpful!
local function foo(x, n)
for i=1,n do funcb(funca(x, i), 1) end
end
This turns them into indirect calls and generates bigger and slower machine code. Instead you'll want to cache the namespace itself and rely on the JIT compiler to eliminate the lookups:
local C = ffi.C -- Instead use this!
local function foo(x, n)
for i=1,n do C.funcb(C.funca(x, i), 1) end
end
This generates both shorter and faster code. So don't cache C functions, but do cache namespaces! Most often the namespace is already in a local variable at an outer scope, e.g. from local lib = ffi.load(...). Note that copying it to a local variable in the function scope is unnecessary.
