In my day job, I am part of a cross-functional team building an app and service in a way that probably the vast majority of apps and services get built these days: A backend service provides an API which clients – Android, iOS, web – interact with.

In the course of development we naturally start off by building the API endpoints for a new feature. We stub them out as quickly as possible so the clients can start integrating while we sort out the implementation details on the backend.

We generate API documentation (via a swagger spec file) and our backend also creates a page where the API can be inspected and tried out. While this does help to get a feel for how an API works I find that often the difficulty with APIs is not how to make single calls but how to instrument call sequences.

For instance, in order to interact with an API that requires authorisation the very first step is acquiring an access token. Immediately, it’s not quite as simple as making a single API call to try something out because you need to make a call to acquire a token first.

Another issue is that it’s harder to talk about API usage patterns when they live in the abstract. For example

“You need to call the login endpoint and then take the token from the response to call the product endpoint. And with the product id you get from that you then call the product/id endpoint.”

This doesn’t communicate all that well nor does it persist. Wouldn’t it be much nicer if you could describe this sequence in a standardised textual format? Like the following, for instance:

requests:
  login:
    url: ${api_url}/login
    method: POST
    body:
      form:
        username: foobar
        password: 123password
    validation:
      status: 200
      json:
        access_token: .regex(.{30})
  product_list:
    url: ${api_url}/product
    headers:
      Authorization: Bearer ${login.access_token}
    validation:
      status: 200
      json:
        0:
          id: .regex(\w{8}-\w{4}-\w{4}-\w{4}-\w{12})
    log: json[0].id
  get_product:
    url: ${api_url}/product/${product_list[0].id}
    headers:
      Authorization: Bearer ${login.access_token}
    validation:
      status: 200
      json:
        id: ${product_list[0].id}
    log: json.id

Not only does this spell out the API sequence in clear terms, it also gives way more detail than the paragraph above. For instance, it shows how parameters are passed via headers or forms and what the expected status codes and return values are.

And this is not a fictional format, this is actually what Rester’s request DSL looks like. Rester takes a YML file with request descriptions like the one above – let’s call it a “Restfile” – and processes it. Here’s what this looks like:

There are a few things worth pointing out here:

The format supports variable substitution. Variables can be defined in a variables: section or can be set via environment variables. References like ${variable} get substituted with these variables at runtime.
Each request can optionally have a validation: section, which ensures the request passes the criteria you define for it. Criteria can be a combination of status code and/or values in the response or headers.
Validations are useful for two reasons: failing to validate stops request execution and avoids running through the remainder of requests when an early failure will cause those requests to fail anyway.
However, they also serve to validate detailed response values that you can then use and reference in subsequent requests. The reference to ${product_list[0].id} in the last request is one such example. (Note that any response values are available for referencing, not just those you have explicitly validated. It’s just usually helpful to validate subsequent inputs.)
You can log response values to the console via the log: keyword for visual inspection.

Being able to reference response values is probably the most important aspect of Rester, because it allows you to create requests and reference their results as if they were objects. ${login.access_token} is really “just” the access_token returned from the login request.

This makes it easy to script all sorts of API actions in a cleaner way than curl snippets could.

The next section talks about some other areas where Rester can be helpful.

What else is Rester useful for?

Smoke testing

We’ve started off with an example of how you can use Rester to describe your API.

The problem with descriptions is that unless they are codified and automatically validated they tend to drift apart from the actual system they describe.

Well, Restfiles already codify the system in a machine readable format and Rester is the machine that can execute them. Rester validates the steps in the Restfile. So all it takes to automate the validation of your API description is to schedule Rester to process your Restfiles.

This is what we do in our high level integration tests for example. We’ve built out a set of API call sequences that instrument all of our basic features and run them periodically and after each deployment as a smoke test.

This serves two purposes:

catch things we break during deployment
catch things that break on their own, due to external events (service interruption etc)

That’s a nice first sets of checks of API behaviour that does not require scaffolding in an app or some other “heavy” setup. All it takes is a light weight request description in the form of a Restfile.

And the minute we have written up a set of requests for a new feature for client development we’ve also written a new smoke test for that feature. We just add the Restfile to the pile and have it covered from then on.

Debugging

Some of the trickiest bugs are those that need complex state to reproduce. And some of the trickiest state to reproduce is complex data in a persistent system.

Image an issue only happens for users with a certain set of items in their shopping cart. Perhaps you have the option of setting that up directly in the database but chances are you need to instrument it via the service’s API.

You could UI script your client to do that but that is probably quite tedious and slow.

Instead, you can set up a Restfile to quickly sketch out an API sequence to populate your state. The additional benefit is that you can easily run this scaffolding against various test APIs - your locally running one, development, staging, production. Well, maybe not production 😅

How can I run Rester?

Rester can be run practically anywhere via docker:

docker run --rm -it -v $PWD:/host -w /host finestructure/rester restfile.yml

If you are using a Mac, you can also install Rester directly on your machine. See the installation instructions on Github. (In a nutshell: mint install finestructure/rester.)

The Github repository goes more into detail regarding the installation instructions and if you prefer to learn by example, you will find the contents of the examples directory illuminating.

Do you find Rester useful? Can you think of other use cases? Please get in touch if you have questions or suggestions! You can find me on Twitter or reach me via email.