Brujo has been a programmer since the age of 10 and he has more than 15 years in the industry. He was a VB, .NET, Java and Haskell programmer until he found Erlang 8 years ago. He’s now Inaka's CTO and Erlang Solutions’ Tech Lead and Trainer. He’s an active member of the open-source community and his blog (Erlang Battleground on Medium) was the most active Erlang blog of 2016.
07 February 2017
Deploying Elixir releases
I have seen great posts about Elixir release deployments lately and I would like to share my experience deploying Elixir releases to production. After testing different approaches (from git cloning code in the target machine to building Docker images), I decided to go with, in my opinion, a simpler approach.
Deployment is critical
Deployment is one of the most important things in a project. It needs to be easy, fast and painless to rollback when things go wrong. Although it’s critical part of a project, it is sometimes overlooked. It’s not hard to find teams that deploy to a production-like environment only when they will release the project to the public, after months of development and thousands of commits.
Delaying a project deployment until you need to expose your project to the world, brings many unnecessary risks, like:
- Lack of experience deploying the project
- If something goes wrong, is very likely the team will have no clue about the cause
- The team discover issues that are only present in production
- The team didn’t have the chance to benchmark and understand the system performance
- If not automated, it becomes a tedious and error-prone process
All those risks can be mitigated if the deployment starts early, is automated and it’s performed by everyone throughout the development phase.
The goals when defining the deployment strategy of this project, was to offer a really simple and fast way of getting a new release shipped. The TODO list while researching and implementing it was:
- It needs to be easy to deploy, rollback and maintain
- It’s automated
- Everyone is able to deploy
- It works the same way in different environments
- It avoids new dependencies
Before we get into the deployment, I think it’s important to give you some context about the project development cycle, since it has a direct impact on how the deploy is done. The numbers on the image means:
- The developer pushes her/his local changes to Github;
- CircleCI pulls the changes and run the automated tests. If all tests pass, then an Elixir release is created, using Exrm;
- The release is pushed to a AWS S3 bucket.
Some facts about this cycle:
- The tests take less then 5 seconds to run;
- The release (a tar.gz file) size is 55MB;
- Every commit (with a green CI build) can be deployed;
- The same release can be deployed to any env;
- The release includes everything that is needed to run the app.
The infrastructure that powers this project is quite standard, but let’s take a look on it into more details:
This project is accessed both by users (using a browser) and by the mobile applications (iOS and Android), which consume the API. As highlighted by the item 1 on the image above, both of them can perform requests to the HTTP load balancer.
The load balancer spreads the load to all the machines available at a given time (as highlighted by the item 2 on the image).
The infrastructure has its creation and setup automated by Ansible. All the information about the load balancers, web servers, databases, caches, etc are stored and managed by Ansible.
For that reason, using Ansible also for deployments seemed to be a really good fit for our case.
For a post about “deployment”, we have spent a considerable amount of time talking about different things, but not really getting into the main topic… deployment.
As mentioned earlier, we used Ansible for setting all the machines used by this project. Deploying a new version of the project, means running an Ansible Playbook, that contains all the tasks to be applied to the target machines.
That’s the command that someone (developer, web, Slack bot) runs in order to deploy a new version:
Let’s break this command down into smaller parts:
--ask-vault-passmeans we are going to use information stored inside one of the Ansible vaults that belongs to the project. Ansible vaults allow the storage of confidential information (passwords, keys, tokens) inside our repository, without exposing it;
version=[COMMIT]is the commit version of the project that we would like to deploy;
-i [ENV]is the Ansible inventory option. It contains information about the environment we are going to deploy to. In that project, the inventories available are:
When this command runs, several tasks are applied to the machines present in the environment inventory. Using the image above, let’s see what happens in each stage:
The first thing that Ansible does, is to remove the server where the deployment is going to happen from the load balancer pool (item 1). It prevents incoming requests being handled by that server;
After Ansible successfully removes the server from the load balancer pool, it performs a number of actions (item 2):
- Downloads the release tar.gz file from S3;
- Stops the current running version (
- Extracts the new release and promote it as the current release;
- Starts the new release (
- Runs the migrations (
Migratoris a helper module) (
bin/project rpc Elixir.Project.Migrator migrate);
- Runs a smoke test to check if everything is fine.
As soon as the new version is running and the smoke test has successfully passed, then Ansible puts the server back in the load balancer pool (item 3), making it available to new incoming requests;
It repeats the same process (except the migration part, that is only executed once) to all the machines inside the load balancer pool.
Thanks to Elixir releases (and Exrm), the entire deployment process takes less than 30 seconds per machine (when not dealing with long-running migrations) and it’s pretty straightforward.
Are you deploying Elixir to production? What’s your approach? Feel free to send me a message.
25 Jul 2016
A video exploring the potential of fast simulated...
08 Aug 2016
Gradle did come to stay with us. Although...
22 Aug 2016
Software Testing is not for Attention...
19 Sep 2016
New video from #droidconpl2015 is out!...
19 Sep 2016
Last night I attended a software testing...
21 Oct 2016
A summary of my visit from SystemD Conference...
24 Oct 2016
Up until yesterday, I had only gone...
24 Oct 2016
Updating sources (versions, revisions, tags)...
14 Nov 2016
Elixir is a joy to work with, an easy...
16 Nov 2016
A few weeks ago I attended Mobiconf, one of...
20 Jan 2017
When you can’t bang! This story...
26 Jan 2017
I have been using Ruby/Rails for 8 years and...
07 Feb 2017
I am an old Java man, I never allocated many of...
07 Feb 2017
Earlier today, Dorothy Graham presented...
07 Feb 2017
I have seen great posts about Elixir release...
22 Feb 2017
I first time I came across the term of "dark...
22 Feb 2017
You probably know...
22 Feb 2017
I am quite happy to share that I have...
06 Mar 2017
Who will find this interesting If you’re...
07 Mar 2017
A peer2peer and/or crowdfunding, blockchain...