Table of Contents

Introduction

I’ve written a bunch of async Python code over the years, so it’s something I’m intimately familiar with. I’ve implemented all sorts of stuff - DNS servers, HTTP servers, HTTP clients, proxies, etc. I also have similar experience from my time working on Squid, working on non-blocking callback based C/C++ to achieve much of the same thing - cooperative multitasking in a single-threaded environment to achieve extremely scalable IO.

Recently I started taking a closer look at Bjoern. This library aims to provide a very fast WSGI server in Python, and it certainly achieves that goal, so much so that it’s 10x faster than any other WSGI server I tested, for my specific use case. Bjoern is written in C, smashing http-parser and libev together to achieve its goals.

As a result of the aforementioned experience with event loops, I’m extremely comfortable with what I expect one to look like, what primitives I expect, that kind of thing. But I’d never used libev before, and I work best by looking at examples to get started, so off I went. To my surprise, there doesn’t seem to be a lot of concise examples out there. Instead, the expectation seems to be that you read the extremely extensive documentation, which is a completely reasonable take - it’s very good and complete. But sometimes it’s nice to just read code, you know?

So this post aims to provide that - an introduction to libev and concepts by way of example code, rather than written words. In terms of pre-requisites, I would recommend that you understand at minimum what select() does, and why it’s advantageous over typical blocking system calls.

Hello World!

Now let’s get started. Let’s start out with the bare minimum: a program that does nothing. Note: all examples in this post can be compiled with gcc <source-name> -lev. Of course, you’ll need to install libev too.

#include <ev.h>

int main(int argc, char **argv)
{
  ev_run(EV_DEFAULT, 0);

  return 0;
}

If you compile and run this, you’ll find it doesn’t do anything. That’s because libev runs on the concept of a “watcher”, which, well, watches for an event. Once the event loop has no remaining watchers, it stops.

EV_DEFAULT is also shorthand for “use the default event loop”, i.e. “I don’t care”. In my opinion you should actually be allocating your event loop explicitly, and destroying it at the end of your program. Let’s see what that looks like.

Even More Hello World!

#include <ev.h>

int main(int argc, char **argv)
{
  struct ev_loop *mainloop = ev_loop_new(0);

  ev_run(mainloop, 0);
  ev_loop_destroy(mainloop);

  return 0;
}

Again, this does nothing, but it’s nice to allocate your own loop, use it, and then clean it up at the end. Getting into this habit now will ensure you think about it if you ever decide to run multiple event loops in multiple threads, where the expectation is that you would allocate one per thread.

Anyway, let’s start doing something vaguely useful. We’ll start with the most basic kind of watcher, a timer.

One Shot Timers

#include <stdio.h>

#include <ev.h>

void run_later(struct ev_loop *mainloop, ev_timer *watcher, const int events)
{
  printf("Neat!\n");
}

int main(int argc, char **argv)
{

  struct ev_loop *mainloop = ev_loop_new(0);
  
  ev_timer timer;
  
  ev_timer_init(&timer, &run_later, 1.0, 0.0);
  ev_timer_start(mainloop, &timer);
  ev_run(mainloop, 0);
  ev_loop_destroy(mainloop);

  return 0;
}

If you compile and run this, you’ll see that “Neat!” is printed after 1 second. This works by allocating an ev_timer on the stack, initializing it with a function and an amount of time, and then starts it running on our loop.

As I said earlier, when there are no more active watchers, libev’s ev_run function will complete. So after one second our one shot timer will fire, and then there won’t be any more active, so the loop stops.

An important note is that ev_timer being allocated on the stack only works in this specific instance, where we know that the stack variable is going to live for the life of the program. If you allocate your watchers on the stack in callback functions, bad things will happen. In those scenarios, you should allocate your watcher on the heap.

ev_timer_init takes two numbers. The first is used to specify that your timer should run once in a given number of seconds. The first is to specify that you want to repeat your timer, and what the interval should be. So let’s look at a repeating timer.

Repeating Timers

#include <stdio.h>

#include <ev.h>

void repeat_ontick(struct ev_loop *mainloop, ev_timer *watcher, const int events)
{
  printf("Repeat called! %f\n", ev_now(mainloop));
}

int main(int argc, char **argv)
{

  struct ev_loop *mainloop = ev_loop_new(0);
  
  ev_timer repeat_watcher;
  
  ev_timer_init(&repeat_watcher, &repeat_ontick, 3.0, 1.0);
  ev_timer_start(mainloop, &repeat_watcher);

  ev_run(mainloop, 0);
  
  ev_loop_destroy(mainloop);

  return 0;
}

If you compile and run this, you’ll see that after three seconds have passed, we’ll start running our repeat_ontick function every second. This is useful if you want to start a timer, but with a delay. But what if we want to stop our timer after some time? Yup, that’s another timer.

Repeating Timers with Timeouts

#include <stdio.h>

#include <ev.h>

void repeat_ontick(struct ev_loop *mainloop, ev_timer *watcher, const int events)
{
  printf("Repeat called! %f\n", ev_now(mainloop));
}

void timeout_run(struct ev_loop *mainloop, ev_timer *watcher, const int events)
{
  printf("Timeout reached! Shutting down.\n");
  
  ev_timer *repeat_watcher = ev_userdata(mainloop);
  
  ev_timer_stop(mainloop, repeat_watcher);
}

int main(int argc, char **argv)
{

  struct ev_loop *mainloop = ev_loop_new(0);
  
  ev_timer timeout_watcher, repeat_watcher;
  
  ev_set_userdata(mainloop, &repeat_watcher);
  
  ev_timer_init(&repeat_watcher, &repeat_ontick, 0.0, 1.0);
  ev_timer_init(&timeout_watcher, &timeout_run, 3.0, 0.0);
  
  ev_timer_start(mainloop, &repeat_watcher);
  ev_timer_start(mainloop, &timeout_watcher);

  ev_run(mainloop, 0);
  
  ev_loop_destroy(mainloop);

  return 0;
}

This will run repeat_ontick for three seconds, before timeout_run will fire, remove the repeating watcher, and thus shutdown. This is a little more complex, because we need to reference the repeat watcher from the timeout watcher. Storing it in global state via the loop’s userdata isn’t exactly scalable though, so how would we fix this in a real program?

Maintaining Additional State

typedef struct {
  ev_timer timeout_watcher, repeat_watcher;
} TimerState;

void timeout_run(struct ev_loop *mainloop, ev_timer *watcher, const int events)
{
  printf("Timeout reached! Shutting down.\n");
  
  TimerState timer_state = (TimerState*)watcher;
  
  ev_timer_stop(mainloop, &timer_state->repeat_watcher);
}

int main(int argc, char **argv)
{
  struct ev_loop *mainloop = ev_loop_new(0);
  
  TimerState timer_state;
  
  ev_timer_init(&timer_state.repeat_watcher, &repeat_ontick, 0.0, 1.0);
  ev_timer_init(&timer_state.timeout_watcher, &timeout_run, 3.0, 0.0);
  
  ev_timer_start(mainloop, &loop_state.repeat_watcher);
  ev_timer_start(mainloop, &loop_state.timeout_watcher);

  ev_run(mainloop, 0);

  return 0;
}

This works much the same, but takes advantage of the fact that the memory address of a struct and its first member are the same. This means that you should try, whenever possible, to put your watchers at the beginning of your structs.

Or maybe you want to be able to refer to your timeout_watcher from your repeat_watcher’s callback, how could you do it? Like with most things in life, this can be solved with pointer arithmetic. I’m not going to provide an example for it, because helpfully the libev documentation already has a great example.

That said though, you should also note that all watchers also have a void *data member, so if you’d like to avoid pointer arithmetic you can simply shove the memory address of your struct in there instead. Note that the documentation specifically calls this out as not being for “real programmers”, but I figure you’re already writing C, so you do you, my dude. I personally think it’s a lot cleaner.

So let’s recap. We have gone through:

1. The building blocks of the event loop.
2. Setting up watchers.
3. Associating additional state with your watchers to do something vaguely useful.

What should we do now? I figure it’s time to graduate to actually performing IO. Let’s write an echo server!

Echo Server Part One: Accepting Connections

Let’s dive right in.

#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <unistd.h>

#include <ev.h>

int set_non_blocking(int sockfd)
{
  int flags = fcntl(sockfd, F_GETFL, 0);
  if(fcntl(sockfd, F_SETFL, (flags < 0 ? 0 : flags) | O_NONBLOCK) == -1) {
    perror("set_non_blocking");
    return -1;
  }
  return 0;
}

void accept_client(struct ev_loop *mainloop, ev_io *watcher, const int events)
{
  int sockfd;
  struct sockaddr_in sockaddr;
  socklen_t addrlen;
  addrlen = sizeof(struct sockaddr_in);
  sockfd = accept(watcher->fd, (struct sockaddr*)&sockaddr, &addrlen);
  
  if (sockfd < 0) {
    perror("accept");
  } else {
    // FIXME: start listening!
    close(sockfd);
  }
}

int main(int argc, char **argv)
{
  struct ev_loop *mainloop = ev_loop_new(0);
  
  ev_io accept_watcher;
  int sockfd = socket(AF_INET, SOCK_STREAM, 0);
  if (sockfd < 0) {
    perror("socket");
    return 1;
  }

  struct sockaddr_in addr_info;

  addr_info.sin_family = AF_INET;
  addr_info.sin_addr.s_addr = inet_addr("127.0.0.1");
  addr_info.sin_port = htons(8000);
  
  if (bind(sockfd, (struct sockaddr*)&addr_info, sizeof(struct sockaddr_in)) < 0) {
    perror("bind");
    return 1;
  }

  if (listen(sockfd, 1024) != 0) {
    perror("listen");
    return 1;
  }
  
  set_non_blocking(sockfd);

  ev_io_init(&accept_watcher, &accept_client, sockfd, EV_READ);
  ev_io_start(mainloop, &accept_watcher);

  ev_run(mainloop, 0);

  return 0;
}

As with most things written in C, this is loaded with boilerplate. I’m going to do my best to ignore most of it, and only talk about the fun libev parts. We start the program by binding and listening our socket like normal, and then after we’ve made it non-blocking we give it to libev. We do this very much in the same way that we did with the ev_timer. Use ev_io_init to associate a callback, socket and the type of events we want, and then ev_io_start to associate our watcher with the event loop (i.e. start actually monitoring it for read events).

Whenever our listening socket is available for read, this means there are clients to accept, so in the callback we’ll do that. At the moment though you can see that our accept_client is pretty rude though, all it does is immediately disconnect the client. Let’s try to do something a little more useful.

Echo Server Part Two: Echo, Echo, Echo…

#include <arpa/inet.h>
#include <errno.h>
#include <fcntl.h>
#include <netinet/in.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/socket.h>
#include <unistd.h>

#include <ev.h>

int set_non_blocking(int sockfd)
{
  int flags = fcntl(sockfd, F_GETFL, 0);
  if(fcntl(sockfd, F_SETFL, (flags < 0 ? 0 : flags) | O_NONBLOCK) == -1) {
    perror("set_non_blocking");
    return -1;
  }
  return 0;
}

void client_read(struct ev_loop *mainloop, ev_io *watcher, const int events)
{
  char buf[4096];

  ssize_t read_bytes = read(watcher->fd, buf, 4096);
  
  if (read_bytes == 0) {
    printf("%d client closed the connection\n", watcher->fd);
    ev_io_stop(mainloop, watcher);
    close(watcher->fd);
    free(watcher);
  } else if (read_bytes < 0) {
    if (errno == EAGAIN || errno == EWOULDBLOCK) {
      printf("Nothing to read yet\n");
    } else {
      printf("%d something bad happened %d %s\n", watcher->fd, errno, strerror(errno));
      ev_io_stop(mainloop, watcher);
      close(watcher->fd);
      free(watcher);
    }
  } else {
    write(watcher->fd, "-> ", 3);
    write(watcher->fd, buf, read_bytes);
  }
}

void accept_client(struct ev_loop *mainloop, ev_io *watcher, const int events)
{
  int sockfd;
  struct sockaddr_in sockaddr;
  socklen_t addrlen;
  addrlen = sizeof(struct sockaddr_in);
  sockfd = accept(watcher->fd, (struct sockaddr*)&sockaddr, &addrlen);
  
  if (sockfd < 0) {
    perror("accept");
  } else {
    set_non_blocking(sockfd);
    printf("Accepted client %s:%d on fd %d\n", inet_ntoa(sockaddr.sin_addr), ntohs(sockaddr.sin_port), sockfd);
    
    ev_io *client_watcher = calloc(1, sizeof(ev_io));
    
    ev_io_init(client_watcher, client_read, sockfd, EV_READ);
    ev_io_start(mainloop, client_watcher);
  }
}

int main(int argc, char **argv)
{
  struct ev_loop *mainloop = ev_loop_new(0);
  
  ev_io accept_watcher;
  int sockfd = socket(AF_INET, SOCK_STREAM, 0);
  if (sockfd < 0) {
    perror("socket");
    return 1;
  }

  struct sockaddr_in addr_info;

  addr_info.sin_family = AF_INET;
  addr_info.sin_addr.s_addr = inet_addr("127.0.0.1");
  addr_info.sin_port = htons(8000);
  
  if (bind(sockfd, (struct sockaddr*)&addr_info, sizeof(struct sockaddr_in)) < 0) {
    perror("bind");
    return 1;
  }

  if (listen(sockfd, 1024) != 0) {
    perror("listen");
    return 1;
  }
  
  set_non_blocking(sockfd);

  ev_io_init(&accept_watcher, &accept_client, sockfd, EV_READ);
  ev_io_start(mainloop, &accept_watcher);

  ev_run(mainloop, 0);

  return 0;
}

Again, very run of the mill, so ignore main, because accept_client and client_read is where the action is. accept_client will now set up a watcher and register it with the event loop, exactly the same as what we did for the listening socket.

client_read is a little more exciting though - it attempts an actual read from the socket! How thrilling. You can see much of the same standard error handling when you get socket errors, but the really interesting bit is ev_io_stop. It’s important to stop trying to process watchers when the underlying resource goes away, and because we allocated our watcher on the heap, we need to make sure to free that too.

An important thing to note here is that the writes very well might fail, either entirely or partially. I’m not writing that for you, so it’s up to you to figure out how you want to manage that kind of state.

But anyway, you can totally test this yourself now! This is a very simple echo server. Simply fire up netcat to 127.0.0.1:8000 and you can see the magic happen.

Conclusion

So that about covers it! I’ve gone over watchers, setting them up, performing IO, all that jazz. This should be enough for you to know what to look for when you’re reading the libev documentation.

Note that there is a ton of cool stuff I did not cover, like signal handlers, threading, async writing, but they all follow the same basic pattern in libev. Have fun learning about them all!