Calling Go functions from LUA

A few days ago, I came across an interesting article, Calling Go Functions from Other languages. In it, Vladimir Vivien creates a small shared library in Go, which he then invokes from C, Java using Java Native Access, and from Python, Node, Ruby using Foreign Function Interface libraries. I thought it would be a fun exercise to interface LUA, or more accurately LUAJIT with the library which was written here.

A little background - why LUA? I’ve been using LUA in production for several years, to provide low latency (0-1ms) API endpoints. I’ve used it to add some sort of CGI capability to nginx - to process LESS, SASS, SCSS and even ES6 on server-side. Recently I’ve used it to build image crop/resize API endpoints, using the magick library, which also uses FFI to load imagemagick or magickwand libraries and invoke their functions/methods. I’ve had to fork and add things to my needs, because the original library and forks only exposed a part of the library API which was needed.

After all this work with LUA, I was already aware that LUAJIT has FFI, and that I could now use LUA to invoke Go functions.

After looking at the code of the other FFI interfaces I wasn’t exactly sure what I was in store for. I just started to “hack” it together, and see how far I would get. These are some of the lessons I learned when I was using LUAJIT to interface with Go.

I can’t use the .h file directly

You build your C shared library from Go code like this:

go build -o awesome.so -buildmode=c-shared awesome.go

When go builds your shared library, it also generates a awesome.h header file, which has all the definitions a program needs to invoke your functions and expose your data structures. You can use these definitions with FFI, and declare them in LUA by wrapping it in a ffi.cdef block:

ffi.cdef([[
  // your C definitions here
]]);

So simple! So, is FFI a simple bridge to C in the background? Can we just include the .h file and basically have a one-liner that would expose the Go functions and structures to LUA? No. Unfortunately, no preprocessor is included, so things like #define will not work. That leaves us to copy-paste most of the .h file to LUA.

local ffi = require("ffi")
local awesome = ffi.load("./awesome.so")

ffi.cdef([[
typedef long long GoInt64;
typedef unsigned long long GoUint64;
typedef GoInt64 GoInt;
typedef GoUint64 GoUint;
typedef double GoFloat64;

typedef struct { const char *p; GoInt n; } GoString;
typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;

extern GoInt Add(GoInt p0, GoInt p1);
extern GoFloat64 Cosine(GoFloat64 p0);
extern void Sort(GoSlice p0);
extern GoInt Log(GoString p0);
]]);

I removed all the unused types (8, 16, 32 bit precision ints), and typedef __SIZE_TYPE__ GoUintptr; because it was unrecognized by LUAJIT. It’s enough for what we need.

Simple datatypes, integer and float

Things get a bit more interesting when we’re dealing with data types - as they are returned, and as they are passed to Go. Most of this is just what I could google and find, with more of a trial-and-error approach until everything “fits”.

The first thing, functions invoked from awesome library return cdata (a C data object). By virtue of the fact that this object is not a LUA native type, it needs to be converted to one.

n = tonumber(cdata)

Converts a number cdata object to a double and returns it as a Lua number. This is particularly useful for boxed 64 bit integer values.

So, with our first case, we need to use tonumber to get a LUA number from the cdata object, and then use math.floor to convert it to an integer. With the Cosine function, we don’t need to cast to int.

io.write( string.format("awesome.Add(12, 99) = %f\n", math.floor(tonumber(awesome.Add(12,99)))) )
io.write( string.format("awesome.Cosine(1) = %f\n", tonumber(awesome.Cosine(1))) )

Declaring Go tables

When it comes to tables, this is the C declaration of the data that needs to go into a Slice:

typedef struct { void *data; GoInt len; GoInt cap; } GoSlice;

So, the slice data is referenced by a simple pointer (void *), and the other two values are the slice length and capacity. When it comes to the Node implementation, the declaration of the Slice was quite simple:

nums = LongArray([12,54,0,423,9]);
var slice = new GoSlice();
slice["data"] = nums;
slice["len"] = 5;
slice["cap"] = 5;

But, the complexity is hidden behind the LongArray declaration, which is a construct of the library ref-array. In comparison, the Ruby declaration is more accurate in terms what’s being done here:

nums = [92,101,3,44,7]
ptr = FFI::MemoryPointer.new :long_long, nums.size
ptr.write_array_of_long_long  nums

In Ruby, a long long * pointer is created and allocated to the size of the nums array. The array is copied into this memory by a call to ptr.write_array_of_long_long. So, it seems when it comes to constructing the table, we should do something similar with LUA FFI.

local nums = ffi.new("long long[5]", {12,54,0,423,9})
local numsPointer = ffi.new("void *", nums);
local typeSlice = ffi.metatype("GoSlice", {})
local slice = typeSlice(numsPointer, 5, 5)

I came to this point after realizing that I can’t set a LUA table object to the Slice.data; And then I also realized that I can’t set cast a LUA table to a pointer (ffi.new void * …), and only after reading the other FFI examples I realized that the table type was abstracted away in the Node example. Of course it’s only logical that C code can only consume complex types that are declared in C space.

This is the final state that I came up with. It declares a long long[5] in C space, and sets the values from the LUA table. The declaration is a bit simpler than what we have in Ruby. As the C code from the Go .h file expects a void * pointer, I declared this as well, so I can then pass it to the Sort function.

Declaring Go strings

Thankfully, strings are a much more normal data type.

typedef struct { const char *p; GoInt n; } GoString;

As such, they consist only of a pointer to a string (const char *) and an integer with the length of the string.

local typeString = ffi.metatype("GoString", {})
local logString = typeString("Hello LUA!", 10)
awesome.Log(logString)

Putting it all together

You can review the full source of the LUA FFI interface to the Go shared library on this gist; And of course, here’s the obligatory money shot of the thing in action:

$> luajit client.lua
awesome.Add(12, 99) = 111.000000
awesome.Cosine(1) = 0.540302
awesome.Sort([12,54,9,423,9] = 0, 9, 12, 54, 423
Hello LUA!

Update: I submitted a PR to vladimir’s github repo, so you may now review my LUA example along with all the others.