Interfacing Lua With Templates in C++11 Continued

The code for this project is hosted here

This is the second part of a 3 part series: – part 1part 2part 3

If you’ve read the previous post on interfacing Lua with templates, you saw how variadic templates allowed us to avoid a lot of boilerplate in dealing with the Lua stack when calling Lua functions from C++. We started by imagining an ideal interface and came up with something like lua_state.Call<int>("add", 3, 5) to call a 2-arity Lua function bound to "add" and return the result. The Call function was variadic in both the return type and the arguments. To continue our development of this library (which I am calling Selene), we need to decide how to bind C++ functions to Lua.

Envisioning the interface

First, let’s see how Lua expects us to register a C function.

without_selene.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
/* include necessary Lua headers
    ....
*/

int my_add(lua_State *l) {
    float x = luaL_checknumber(l, 1) // reads a value from the stack,
                                     // verifying it is a number
    float y = luaL_checknumber(l, 2)
    const float result = x + y;
    lua_pushnumber(l, result);
    return 1; // return the arity of this function
}

int main() {
    // open a lua context
    lua_State *l = luaL_newstate();

    // bind the function pointer to global "c_my_add"
    lua_register(l, "c_my_add", &my_add);
}

This doesn’t look too bad, but adhering to this interface is certainly not ideal. Every function we write that can be passed to lua_register must be of the form int (*)(lua_State *l) which is aliased to lua_CFunction. In addition, each function we write to expose to Lua in this way must manually push and read from the stack, in addition to checking types in both directions. Also, because the Lua API requires us to pass in a C style function pointer, we can’t use the newschool std::function, lambdas, or traditional functor objects.

What this amounts to is boilerplate city! What could we envision this code looking like? Personally, I fancy something resembling the following:

with_selene.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
float my_add(float x, float y) {
    return x + y;
}

int main() {
    sel::State state;

    // Register function pointer
    state.Register("my_add", &my_add);
    float answer = state.Call<float>("my_add", 20, 22);
    assert(answer == 42);

    // Register lambda
    const std::string greeting = "hello";
    std::function<std::string(void)>> fun = [greeting]() {
        return greeting;
    }
    state.Register("greet", fun);
    assert("greet" == state.Call<std::string>("greet"));
}

Much better! With an interface like this, we don’t have to worry about interacting with the Lua stack at all. This allows our function signatures to have actual return types and argument types as well. Also, we can register functions with state, like the lambda in the above example or with an std::function created from a functor object. So now we just need to implement it. Hold on tight!

Design Discussion

Because we want our registered functions to possibly capture state as closures or be temporaries even, we’ll want to create a new class that encapsulates it. Let’s call it sel::Fun. For the implementation, we note that using lua_register which in turn calls lua_pushcfunction is a bit tricky because we would need to pass a function pointer with signature int (*)(lua_State *). If we use a lot of template magic, we could create such a function if the actual worker function was known at compile time. However, this would not allow us to use functor objects, lambdas, and the like. Note that std::function is not compatible with ANSI C function pointers and attempting to cast them as such leads to madness (I actually tried this. I’m telling you, don’t try it. But if you figure it out please talk to me).

Fortunately, Lua allows us to pass what is called a C closure. That is, in addition to passing the function, we specify and push to the stack a number of upvalues which each invocation of the function will have access to later. Here is an example of this:

upvalues.cpp
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
int my_counter(lua_State *l) {
    int val = lua_tointeger(l, lua_upvalueindex(1)) // gets the first upvalue
    lua_pushinteger(l, ++val); // return the incremented value
    lua_copy(l, -1, lua_upvalueindex(1)); // update the upvalue
    return 1; // return the arity of this function
}

int main() {
    // open a lua context
    lua_State *l = luaL_newstate();

    lua_pushinteger(l, 0); // initialize counter to 1
    // push the closure with a single upvalue
    lua_pushcclosure(l, &my_counter, 1);
    // name the clsoure
    lua_setglobal(l, "my_counter");
}

With this, we come up with the following strategy:

  1. Accept anything that is a function pointer or instance of std::function.
  2. Create an instance of sel::Fun to house the callable entity, templatized by the return type and arguments of the passed function (call it _fun).
  3. In the constructor of sel::Fun, create a C closure with the address of this object as the upvalue and a static dispatch function as the C closure (call it _lua_dispatcher).
  4. Implement in sel::Fun an Apply method which will be called from _lua_dispatcher
  5. Within the Apply method use variadic templates and template metaprogramming to retrieve arguments from the stack, call _fun, the original function, and push the returned values back onto the stack.

There are a number of other implementation details regarding sel::Fun ownership, the unregistration of functions, properly handling destruction of the parent sel::State object, and more but this should keep us occupied for now.

The dispatcher

The implementation of the dispatcher is very simple.

1
2
3
4
5
6
7
8
namespace sel {
namespace detail {
int _lua_dispatcher(lua_State *l) {
    BaseFun *fun = (BaseFun *)lua_touserdata(l, lua_upvalueindex(1));
    return fun->Apply(l);
}
}
}

The dispatcher retrieves the upvalue which is a pointer to sel::BaseFun which sel::Fun inherits from and invokes the Apply method. Note that the lua_State pointer is created each time this function gets called so it must be threaded through.

The sel::Fun object

Because we expect sel::Fun to be a template class, we should create a base class to allow us to create containers of sel::Funs and for _lua_dispatcher to work. Assume that everything that follows is part of the sel namespace.

Fun.h
1
2
3
4
struct BaseFun {
    virtual ~BaseFun() {}
    virtual int Apply(lua_State *l) = 0;
};

This is our barebones abstract class. Now for the main star:

Fun.h
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
template <int N, typename Ret, typename... Args>
class Fun : public BaseFun {
private:
    using _fun_type = std::function<Ret(Args...)>;
    _fun_type _fun;
    std::string _name;
    lua_State **_l; // Used for destruction

public:
    Fun(lua_State *&l,
        const std::string &name,
        Ret(*fun)(Args...)) : Fun(l, name, _fun_type{fun}) {}
    Fun(lua_State *&l,
        const std::string &name,
        _fun_type fun) : _fun(fun), _name(name), _l(&l) {

        // Add pointer to this object to the closure
        lua_pushlightuserdata(l, (void *)static_cast<BaseFun *>(this));

        // Push our dispatcher with the above upvalue
        lua_pushcclosure(l, &detail::_lua_dispatcher, 1);

        // Bind it to the specified name
        lua_setglobal(l, name.c_str());
    }
    Fun(const Fun &other) = delete;
    Fun(Fun &&other)
        : _fun(other._fun),
          _name(other._name),
          _l(other._l) {
        *other._l = nullptr;
    }
    ~Fun() {
        // This destructor will unbind the name in lua, provided that
        // this object has not been moved and the parent context is
        // still alive (hence the pointer to the pointer)
        if (_l != nullptr && *_l != nullptr) {
            lua_pushnil(*_l);
            lua_setglobal(*_l, _name.c_str());
        }
    }

    // TODO implement Apply
};

This is a big chunk of code to take in all at once but hopefully the inlined comments alleviate some of the potential confusion. The important bit is in the constructor where we pass our generic dispatcher along with the pointer as a closure to lua. The remaining bits are implementation details that take care of what happens if the parent state is destroyed or if this function object is itself moved or destroyed. All that’s left is the Apply method.

Implementing sel::Fun::Apply

The order of operations we’ll need to handle are:

  1. Read all arguments from the stack into a tuple of type std::tuple<Args...>
  2. Call the member function _fun with those arguments by converting the tuple back into a parameter pack.
  3. Push the returned value(s) onto the stack.
  4. Return the number of values returned so Lua knows how many values to pop on its end.

First, let’s implement a helper function called _get_args. We could use recursion like I did in the last post, but I’ll use a different technique here since it’s arguably faster (both at compile and runtime). My implementation of Pop<std::tuple<T...>> has also changed but I will leave the other post unchanged since the type recursion technique is a good one to know.

The technique we will use is a common pattern called tag dispatching, where we pass an empty struct whose type encodes important information. In this case, we want to bind to the nth element of the tuple, the nth element on the stack (counting from the bottom). This, we’d love to expose to our function something with a type containing a parameter pack of the form <std::size_t... N>. There are many ways to do this so what follows is just what I find easiest.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
namespace detail {
// Our "tag"
template <std::size_t... Is>
struct _indices {};

// Recursively inherits from itself until...
template <std::size_t N, std::size_t... Is>
struct _indices_builder : _indices_builder<N-1, N-1, Is...> {};

// The base case where we define the type tag
template <std::size_t... Is>
struct _indices_builder<0, Is...> {
    using type = _indices<Is...>;
}
}

We can create a dummy struct of type _indices<0, 1, 2, 3> for example, by doing typename _indices_builder<4>::type(). The typename keyword is required because scoped entities are always assumed to be values by default. Note also that our index tag is 0-indexed. Now, we can implement our _get_args function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
namespace detail {
template <typename... T, std::size_t... N>
std::tuple<T> _get_args(lua_State *l, _indices<N...>) {
    // Assume existence of function _check_get<T>(l, i) which will
    // read the ith element in the stack, ensure it is of type T, and
    // return it
    return std::make_tuple(_check_get<T>(l, N+1)...);
}

template <typename... T>
std::tuple<T...> _get_args(lua_State *l) {
    constexpr std::size_t num_args = sizeof...(T);
    return _get_args<T...>(l, typename _indices_builder<num_args>::type());
}
}

Notice the N+1 in the make_tuple expansion to account for the fact that Lua stacks are 1-indexed from the bottom. Now that we have a tuple with all the correct arguments read from the stack, we can call the function. Unfortunately, we have a tuple instead of a parameter pack which we need to call the funtion. Let’s write a function that can do that conversion for us.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
namespace detail {
template <typename Ret, typename... Args, std::size_t... N>
Ret _lift(std::function<Ret(Args...)> fun,
          std::tuple<Args...> args,
          _indices<N...>) {
    return fun(std::get<N>(args)...);
}

template <typename Ret, typename... Args>
Ret _lift(std::function<Ret(Args...)> fun,
          std::tuple<Args...> args) {
    return _lift(fun, args, typename _indices_builder<sizeof...(Args)>::type());
}
}

The function std::get<int I> can be used to extract the Ith element of a tuple, so fun(std::get<N>(args)...) will correctly call the function with the contents of the tuple as a comma separated list. Also, as before, we provide an overload which will automatically dispatch the necessary index tagging. Neat!

The last step of our apply function is to push the returned arguments back to the stack. We can use the techniques at our disposal to write a function of the form detail::_push(lua_State *l, std::tuple<T...> values) and overload it for singular values but I will leave this as an exercise to the reader. Finally, we can piece everything together and write sel::Fun::Apply:

1
2
3
4
5
6
7
8
9
class Fun {
// ... as before
int Apply(lua_State *l) {
    std::tuple<Args...> args = detail::_get_args<Args...>(l);
    _return_type value = detail::_lift(_fun, args);
    detail::_push(l, std::forward<_return_type>(value));
    return N;
}
};

Registering the function

We still haven’t exposed our sel::Fun class to the wild yet. For now, we’ll allow the user to call Register, a new public method in sel::State with a name and callable entity. The callable entity can be function pointer or an std::function. I provide the implementation for the std::function below:

1
2
3
4
5
6
7
8
9
template <typename Ret, typename... Args>
void Register(const std::string &name,
              std::function<Ret(Args...)> fun) {
    auto tmp = std::unique_ptr<BaseFun){
        new Fun<1, Ret, Args...>{_l, name, fun}};
    // insert Fun object in a member std::map<std::string,
    // std::unique_ptr<BaseFun>>
    _funs.insert(std::make_pair{name, std::move(tmp)});
}

We’d also need to handle the case when Ret is a tuple type so we can properly pass the first template argument to Fun (the number of return values).

Wrapping up

With this, we have almost everything we need to pass arbitrary functions to Lua and we can do things like call a function from lua which calls a C function which calls a lua function, etc. See the unit tests for some concrete examples. From here, we need to add support for C modules, coroutines, and continuations. We are also still missing an abstraction for dealing with table types. However, a lot is already possible just with what is provided already because std::function is such a powerful abstraction. Feel free to follow the project and let me know if you have any comments, suggestions, or if you found any of this helpful.

Comments