canideploy?...👍

Our custom Slack command for communicating production releases.

The button

Deploying at Countingup is as simple as clicking a button... because it is just that.

Our philosophy is to deploy small changes often, so we have invested a lot in our CI/CD processes to allow for this. We use Semaphore CI to automatically build, test and provide the magic deployment button.

In fact, we could do away with the button and have fully automated deployments, but we choose not to (more on that in a future post). We instead, choose to kick off our deployments manually and this post touches on some of the benefits.

Deployment considerations

Every developer at Countingup has the power to deploy to production. It is very much part of our culture to encourage people to take ownership of their work and see it all the way to the end. As a result, ownership becomes more than ensuring your own code works, it is also about delivering, communicating and supporting each change.

So you've taken ownership, hand hovering over the button, you will likely be asking yourself:

• What changes are being made?
• Is anyone else's work included?
• What is the impact to our customers?
• What manual testing is required?
• Does this require other deployments to be made first?
• Are the right people available if something goes wrong?

/canideploy was made to help answer these questions, but it has been a long time in the making. Processes have been created over time to address the questions raised above, to make collaboration and communication between developers easier.

The history

Slack has been the main form of communication within Countingup since the company was founded back in 2017. As a new joiner, who started (October 2021) after all of these processes were established, it has been fun to look back through the Slack history to unearth the origins of /canideploy. I hope you will find it just as interesting!

In the beginning there was Mike, Rob and Dave.

Communication around deployments was straightforward and announcements simple.

As the company grew, deployments ramped up and became more frequent.

An AWS tech talk by Starling Bank introduced the 'rolling giphy' to the team. Each production announcement in Slack would be accompanied by a gif to signal a deployment was taking place. This was a fun way of keeping track of deployments and is something we still do.

With the number of developers increasing, service changes could have multiple people involved. A developer would have to worry about other people's changes mixing in with theirs. The announcements evolved in turn and started to include a breakdown of the changes, @-ing anyone else involved. If needed, a discussion thread could be started or a simple 👍 was given. The great thing about giving a thumbs up was that it signals your involvement in the deployment, whether or not you're actually pushing the button.

Going back to the deployment considerations, this process ticked all the boxes, but would be time-consuming. Each deployment would require identifying all of the code changes, who made them and then writing it up in the Slack thread.

Ask developers to repeat the same task over and over and they'll just automate it. With a working practice established, one developer took the initiative to solidify it in a Slack command.

The how

/canideploy is a custom slash command. Slash commands simply allow Slack to forward a message onto a third-party app for custom behaviour. Apps can be private single-workspace like ours, or distributed publicly.

The command works in two steps. First we generate an ephemeral message. This is only seen by the person issuing the command and allows them to see a preview of the message. The message is then confirmed/cancelled before it is sent on to the #team-dev Slack channel for everyone to see.

/canideploy tax

Firstly, a developer will type in their Slack client:

/canideploy <repository>

For this example we want to deploy the 'tax' repository. So, <repository> is replaced with tax.

Slack will then look up the app associated with 'canideploy' and forward the message on via a webhook. Every Slack app defines its own HTTP endpoint, somewhere on the web, to listen for messages. We built a Go service called slack-commands to handle the command. It is a first-class service in its own right, hosted along with all our other production services in our Kubernetes cluster.

For anyone interested in creating a Slack integration written in Go, a good reference is evnsio's project Decision.

Once slack-commands has received the message, it will begin to pull in all the relevant information and build the deployment summary message.

We use GitHub's API to work out what commits are new for this release. We can do this because our Semaphore CI build creates a release tag for every deployment, so we look for the commits since the last release tag. Other useful information is included, like links to PRs and the last release.

The @ tagging is done by matching a GitHub user for each commit against a Slack user with a similar name. For this, we grab all of the Countingup Slack users via the Slack API and perform a very naive matching (no need to over-engineer if it works).

The traffic light system uses our own system-verification service to tell us the status of all our environments. Anything other than all-green requires investigating before the deployment can go ahead.

Finally, the ephemeral message is sent back to the developer with a cancel & confirm button. We even have special messages to help developers keep to our 'no deploy Fridays' policy via a subtle reminder.

Closing

/canideploy lives at the heart of the #team-dev channel. It provides the developers the information they need; keeps other teams and code owners in the loop and serves as a useful historic reference. I give it a big 👍.