Engineering Superpowers with OpenAPI

What options do we have for defining HTTP APIs? It might seem a bit overwhelming – do you stick with a REST-like API? How about GraphQL, that looks neat? Which libraries or frameworks can we use to help us out? How do we handle security? What even is middleware?

From almost our very first service we made a decision on all of this and it's worked so amazingly well that almost nothing's changed since then. We write all of our APIs following the OpenAPI standards and it gives us superpowers that rival any Marvel hero.

With great power comes great RESTponsibilty

OpenAPI

If you're already familiar with OpenAPI (or Swagger) then jump to "Superpowers" below.

A quick primer on OpenAPI. It's a specification that defines the structure of an API – the URL path, query parameters, request bodies, responses, authentication, validation and more. Here's a realistic example of part of a specification for our invoices service:

swagger.yml

swagger: "2.0"
info:
  description: Invoices service for drafting and creating invoices
  title: Invoices
  version: "1.0"
schemes: [http]
produces: [application/json]
consumes: [application/json]
parameters:
  businessID:
    in: path
    name: businessID
    type: string
    required: true
    description: ID of the business
paths:
  /invoices/{businessID}/issue:
    post:
      operationId: issueInvoice
      tags: [invoices]
      parameters:
        - $ref: "#/parameters/businessID"
        - in: body
          name: body
          schema:
            $ref: "#/definitions/InvoiceData"
      responses:
        200:
          description: Invoice ID
          schema:
            type: string
        500:
          description: Internal server error
definitions:
  InvoiceData:
    type: object
    required: [lineItems]
    properties:
      lineItems:
        type: array
        items:
          type: string

This might look a little scary at first, so let's break it down.

Metadata

swagger: "2.0"
info:
  description: Invoices service for drafting and creating invoices
  title: Invoices
  version: "1.0"
schemes: [http]
produces: [application/json]
consumes: [application/json]

This defines the service – its name, protocol and payload format (JSON). There are more options available here but this is all we need.

Paths

paths:
  /invoices/{businessID}/issue:
    post:
      operationId: issueInvoice
      tags: [invoices]
      parameters:
        - $ref: "#/parameters/businessID"
        - in: body
          name: body
          schema:
            $ref: "#/definitions/InvoiceData"
      responses:
        200:
          description: Invoice ID
          schema:
            type: string
        500:
          description: Internal server error

The paths section describes your endpoints. We've only got one example here but in reality this includes many endpoints. Each endpoint has

• URL - A URL template that can include parameterised path segments
• HTTP method - GET, POST, PUT, DELETE
• operationId - You can think of this as a human readable name for the endpoint
• parameters - Path parameters, query parameters and body parameters that the endpoint needs
• responses - Response codes and their corresponding payloads
• security - I've not included this in the example to keep things simple but you can read more in the Swagger documentation.

Definitions

definitions:
  InvoiceData:
    type: object
    required: [lineItems]
    properties:
      lineItems:
        type: array
        items:
          type: string

The final section describes our data models, the reusable data structures that we use to define our API. We can define the structures in place but it often makes for a more readable file when we define them in the definitions section. It also helps with code generation which is our superpower origin story.

Superpowers

Writing a few yaml files doesn't exactly sound like a superpower and if you've done anything with Kubernetes you might think it's quite the opposite. So let me share the superpowers the Countingup engineering team get from this.

• Server-side code generation - Our services are written in Go and we use go-swagger to generate server and endpoint handler code for us.
• Type-safe client code generation - Our app and web products are written in TypeScript. We generate TypeScript code for calling our API using NSwag.
• Service-to-service communication - Our backend services communicate with each other using exactly the same API as our frontend clients.
• Robust third-party integrations - If a third-party we work with has an OpenAPI specification (or a specification in another format that's readily convertible to OpenAPI) we generate Go client code to interface with them. If they don't provide a spec we'll hand-write one into the finest of artisanal yaml files.
• API documentation - A single source of truth for the API that developers can lookup and reference.

Server-side code generation

There are quite a lot of options for generating code from an OpenAPI spec – I guess that's kind of the point of having an open spec. We landed on go-swagger for generating server side code for some of the following reasons:

• The generated code is clean and readable (easier for debugging)
• The generated code and our handler code are separate. Some tools require you to edit the generated files and that can get pretty messy, especially when you make changes to the spec later on.
• Support for middleware. It's essential to have those escape hatches to log access requests, intercept and manipulate requests and responses where the OpenAPI doesn't support what we need etc.

CLI

To generate the server code we run swagger generate server:

swagger generate server -t ./server -f ./swagger.yml --exclude-main

This tells go-swagger to:

• Read the spec from the ./swagger.yml file
• Generate server code (as opposed to client code)
• Output the code to the ./server directory
• Skip generating a main function - we just need the operations and models

The output of this command on the yaml file above is,

server
|- models
  |- invoice_data.go
|- restapi
  |- operations
      |- invoices_api.go
      |- issue_invoice_parameters.go
      |- issue_invoice_responses.go
      |- issue_invoice_urlbuilder.go
      |- issue_invoice.go
  |- configure_invoices.go
  |- doc.go
  |- embedded_spec.go
  |- server.go

I won't detail everything here but I'm sure you can imagine the contents of most of these. Instead I'll show you how to start a service with the generated code.

Usage

To run a server that listens for requests to issue an invoice we need to write a little Go code as below:

package main

import (
	"blog/server/restapi"
	"blog/server/restapi/operations"
	"github.com/go-openapi/loads"
	"github.com/go-openapi/runtime/middleware"
)

func main() {
    // Load the swagger spec into memory
	swaggerSpec, err := loads.Analyzed(restapi.SwaggerJSON, "")
	if err != nil {
		panic(err)
	}

    // Create a new API object
	api := operations.NewInvoicesAPI(swaggerSpec)

    // Set the handlers
	api.IssueInvoiceHandler = operations.IssueInvoiceHandlerFunc(issueInvoice)

    // Start the server
	server := restapi.NewServer(api)
	server.Port = 8080
	server.Serve()
}

func issueInvoice(invoice operations.IssueInvoiceParams) middleware.Responder {
	id, err := db.CreateInvoice(invoice.BusinessID, invoice.Body.LineItems)
	if err != nil {
		return operations.NewIssueInvoiceInternalServerError()
	}

	return operations.NewIssueInvoiceOK().WithPayload(id)
}

That's it! We've got a server that can issue invoices. We've not had to type any URLs, parse Go http.Requests, or json.Unmarshal any request bodies, etc. We get to focus on writing domain code – issuing invoices.

We can issue an invoice with a POST request in cURL or Postman to

http://localhost:8080/invoices/biz-1/issue

and if all goes well it'll respond with 200 OK and the invoice ID.

On to the next superpower, type-safe client code generation!

Type-safe client code generation

Even though we've been generating our APIs with go-swagger from day 1 it took a little while before we utilised our second superpower. The React Native app and React web products used to be written in JavaScript (with some flow.js) so having type-safe client code would have a been hazy concept. When we migrated everything to TypeScript (yay!) it made perfect sense for us to look at client code generation more seriously.

Again, there are plenty of options in this space but we found NSwag to be excellent. We like it because:

• It has a staggering number of configuration options
• The generated code is clean, easy-to-read
• Strong TypeScript support
• We can provide a custom fetch function which we heavily rely on for authentication

CLI

To generate the client code we run,

nswag run ./nswag.json

nswag.json is a configuration file that tells nswag how to generate the code including the location of the swagger.yml file. In our case we use the openApiToTypeScriptClient provider to generate TypeScript and use most of the default options.

Usage

The usage here depends a lot on your nswag.json options so this is more an example than a tutorial.

import { InvoicesClient } from "../client/generated/InvoicesClient";

const issueInvoice = async (businessID: string, lineItems: string[]) => {
  // Create the client - typically done once, globally
  const client = new InvoicesClient("localhost:8080");
  try {
    const id = await client.issueInvoice(businessID, lineItems);
    return id;
  } catch (e) {
    // handle error
  }
};

With this superpower we're able to write client code that perfectly matches our backend API without faffing around with URLs, no JSON.stringify calls and no need for libraries like zod which are often recommended when using TypeScript as NSwag does schema validation for us.

The specification acts as a single source of truth where all the code and other documentation is derived from that source. This means we only need to change our API in 1 place for the server and client code to be updated with the latest parameters, descriptions and responses.

Service-to-service communication

All of our backend services communicate with each other through the same HTTP API as our frontend clients. Not all routes are available to all parties and we manage all of that through OpenAPI.

With an OpenAPI spec to hand we go back to our old friend, go-swagger to generate code - but this time it's client code, not server code. Let's say we have another service that wants to issue an invoice on behalf of a user:

CLI

To generate the client code we run swagger generate client:

swagger generate client -t ./client -f ./swagger.yml

This generates client code:

client
|- client
  |- operations
      |- issue_invoice_parameters.go
      |- issue_invoice_responses.go
      |- operations_client.go
  |- invoices_client.go
|- models
  |- invoice_data.go

Usage

To create a Go client that sends a well-formed HTTP request to the third-party endpoint we need to write a little Go code as below:

package main

import (
	"blog/client/client"
	"blog/client/client/invoices"
	"blog/client/models"
	"fmt"
	"github.com/go-openapi/strfmt"
)

func main() {
	// Create the client
	invoicesClient := client.NewHTTPClientWithConfig(strfmt.Default,
		client.DefaultTransportConfig().
			WithHost("invoices").
			WithSchemes([]string{"http"}))

	// Issue the invoice
	ok, err := invoicesClient.Invoices.IssueInvoice(invoices.
		NewIssueInvoiceParams().
		WithBusinessID("biz-1").
		WithBody(
			&models.InvoiceData{LineItems: []string{"item-1", "items-2"}}))

	if err != nil {
		panic(err)
	}

	fmt.Printf("Invoice id = %s\n", ok.Payload)
}

OK so the first three superpowers covered us for our own code, but what about third-party code?

Robust third-party integrations

If you've ever written code to integrate with someone else's API you'll know that it's either a lovely experience or, well... not. Some folks do an amazing job with their developer experience – you can really sense the amount of effort and pride taken to make it as easy as possible to integrate with them.

In an attempt to normalise this across all of our providers and to offer a consistent developer experience for the Countingup engineering team we use OpenAPI to enable robust integration with supplier APIs, our fourth superpower.

Integrating with suppliers

Countingup is on a mission to enable small business owners to easily manage their business. As you can probably imagine this involves integrating with several supplier APIs to handle payments, perform ID checks, submit Making Tax Digital reports through HMRC, etc.

In just about every case we use OpenAPI to generate client code in Go. The specific scenarios vary by supplier:

• Best case scenario 😄 - A supplier publishes an OpenAPI v2 specification. We can use that to generate client code.
• Second best case 😐 - A supplier publishes a spec under a different standard that's readily convertible to OpenAPI. Most often this is RAML.
• Worst case 🙁 - A supplier publishes a spec in a format that we don't support, or no spec at all. In that case we'll usually modify the spec by hand or write one from scratch. This might seem annoying but it's always been worth it for all the benefits mentioned so far.

Now let's pretend that yaml file for issuing invoices came from a third-party, invoyce.io, and we want to call them. We can use the same code as we used previously but tweak the host:

invoicesClient := client.NewHTTPClientWithConfig(strfmt.Default,
		client.DefaultTransportConfig().
			WithHost("api.invoyce.io").
			WithSchemes([]string{"https"}))

API documentation

The API documentation superpower takes 2 forms depending on who the audience is.

For developers

The swagger.yml and generated code act as an internal source of documentation. It's important to understand the behaviour of an API that you're using in order to know if it's doing the thing you need, how it behaves with optional parameters, understand any non-obvious behaviours etc. We get all of this insight for free in a clear, concise and consistent way.

IDEs are our friend here too – autocomplete allows easy discovery and documentation popups for things like parameter summaries let us do a deeper dive without leaving the file we're currently working in.

For external audiences

Our API isn't public, we don't tend to need to provide documentation for it to external audiences so this is a secondary superpower - like being able to wake up early.

Given OpenAPI is an open standard there are a number of tools for generating beautiful documentation for your API. We've needed this a few times over the years, most notably for our external penetration testing team. We were able to generate extensive documentation that we could share with them using Redoc.

To generate static HTML from your OpenAPI specs you can use the redoc-cli:

redoc-cli bundle ./swagger.yml

And this will generate a static html file that looks like this,

Conclusion

Plenty of folks in the Go community are very happy with the standard library options for writing HTTP APIs, Gorilla Mux is also a really popular, lightweight router and matcher and that obviously works well for them. But given the superpowers mentioned above we've found that relying on OpenAPI for code generation is the best way for us at Countingup. I honestly don't think I could go back to hand crafting routes.