Beanstalk is a simple, easy-to-use job queue system. I had been using NodeJS with the node-beanstalk package to interface with it but it was causing a performance bottleneck so I decided to write my own client. This gave me a 37x speedup in my application!

Design Goals

The goal of this project was to create a NodeJS client for the Beanstalk server protocol that met the following criteria:

• Fast
• Lightweight
• Easy to use
• Comprehensive
• Robust

I believe that my solution, async-beanstalk, meets all of these goals in only 500 lines of code.

Making it Fast: Asynchronous Commands

The Beanstalk protocol is a simple sequence of command and response. You send a command to the server, and it replies with a response. The easiest way to write a client for this kind of protocol is to send a command, wait for the response, then send the next command, and so-on. This approach is unfortunately very slow, as you have to wait for a whole network round-trip before sending the next command.

A much faster method is to decouple the sending of commands from awaiting their responses. Issue all the commands as fast as possible and then process the responses asynchronously as they arrive. As well as avoiding the wait for a response, working asynchronously like this also allows for multiple commands and responses to be sent in a single networking operation, further improving efficiency.

Internal Design – Command Queue

In order to process commands and responses asynchronously, the client must know which command the response relates to. Responses are always received in the order in which the commands were sent, therefore the client simply maintains a list of outstanding commands. Only the command at the head of the list reads from the network. Once the full response has been received, the command’s promise is resolve with the result and the command is removed from the list.

Internal Design – Reading from the Network

Because multiple commands are issued at once, that also means that multiple responses may be sent at once. When the network socket receives a “read” event, the data coming from the network could be:

• Part of a response
• A full response
• Several responses

The solution is to keep a read buffer. When data is received from the socket it is appended to the read buffer. The client inspects the read buffer to see if it contains an entire response, and if so, removes it from the read buffer and processes it.

When the next command reaches the head of the queue it first checks the read buffer to see if its response is already available. If not, it will block and wait for more network data.

This process is handled by a simple readLine() function. If the data is not yet available then the function will wait until the socket emits a ‘data’ event, indicating that there is more data available in the read buffer. The wait may be interrupted by a socket error, or by a read timeout.

Making it Lightweight

The package has zero dependencies and only 500 lines of code, making it extremely lightweight – around 15kB of JavaScript (excluding comments). It is my belief that this not only improves performance, but it improves reliability and maintainability because the code is simple to understand.

By using just a few composable building blocks, most commands can be implemented using only a single line of code.

It was for this reason that I chose not to use a YAML parsing library, but instead to use simple string manipulation to parse the few Beanstalk responses that are encoded as YAML. Beanstalk uses only a very small subset of YAML, and so using a full YAML parser would be excessive.

Making it Easy to Use

The interface exposed to the user is extremely simple: A single “Client” object with one method for every Beanstalk command. Every method returns a promise that is resolved to the server’s response.

The modern promise-based interface allows the user to use async/await, and to use aggregate promise functions such as Promise.all() to await the results of multiple commands at once.

The client is written in TypeScript, which allows users to ensure at build time that the expected data types are being used, thereby reducing runtime errors and simplifying debugging.

Making it Comprehensive

The Beanstalk protocol is simple and well documented. It is based on a few simple building blocks – reading a line of text and reading a specified number of bytes – therefore once the building blocks are in place, implementing the entire protocol is fairly trivial.

Making it Robust

Currently available package suffer from problems with handling outstanding commands if the connection is unexpectedly terminated. This new package will ensure that all promises are rejected if the connection fails.

Error handling is done via rejecting promises. The client will never throw an unhandled asynchronous exception.

Conclusion

By identifying the core concepts that make up the protocol (read a line of text; read a number of bytes; return a string, number, job, or YAML), the client can be fully implemented with remarkably little code.

The application using this module showed a speedup of 37 times (from 37ms per job to 1ms per job).

The package is open source. The source code is available on GitHub, and the package is hosted on NPM.