Eliminate Environment Bottlenecks

Creating a new fully-functional clone of your SaaS production environment used to be very difficult. Installing new servers or writing an automation for provisioning additional cloud resources took a lot of additional time. In order to avoid this extra work, it’s become common practice to have a few “static” environments like staging or QA.

With the maturity of Docker and its ecosystem, the notion that creating a new environment from scratch should be hard is changing.

Just take a look at the most basic example — if your entire application’s stack is “just” a few Docker containers running on a Kubernetes cluster, how hard can it be to create a new cluster and deploy the same containers to it?

There are many benefits to having the ability to run a new complete environment of your SaaS from scratch, and we’ll dive into those in a sec. But first, let’s take a look at the limitations of the “old” model.

Bigger Teams, Bigger Problems

As long as your development team consists of a few developers who are focused on their part of the service, you probably won’t hit the limitations mentioned below. Most of the time, these problems manifest themselves in bigger teams.

But as your team grows and the development processes become complicated, here are a few things you’ll begin to notice:

1. Testing Bottlenecks

What happens if your team needs to test two separate components of the system but you need to test each one of these components against different versions of the rest of the stack?

Usually, you will have one static QA or staging environment that will run a specific “snapshot” for the rest of the components of the system.

Having one shared environment in such case will create a bottleneck in the release process.

Your team will need to test each component separately, one after another, which will slow down the delivery of the new features to the end users.

2. Slower UI Prototyping

A slightly different variation of the same problem happens when your team needs to have a few prototypes of the same component to find the best approach for solving a complex UI or a backend problem.

Since you have only one staging environment, aside from when you’re testing the same component locally on developer’s laptop, it will be hard to have multiple versions of the same component run simultaneously along with the rest of the stack.

Someone will need to grab the staging environment first, test that version and then pass the torch to the next developer in line who is waiting to test his version.

3. Different Availability Requirements

Recently, a developer I know got an angry mail from a sales team member with a precise explanation about how the developer’s latest release introduced a new bug in the UI which made the sales team representative look bad in front of a very promising new customer.

Even with the most elaborate testing frameworks, when you release new features on a daily basis, some minor bugs can slip into production.

Overall, it’s not such a big deal if you identify the issue quickly and can easily revert back to a working state. The problem is that the bug can pop up right in the middle of a very important  presentation.

Wouldn’t it be great if you could snapshot the latest working state of the system and have it run on a separate environment so it can be used in these cases?

Eliminate The Static Environment’s Bottlenecks!

All the problems above and several others could be eliminated if you could create a new functional clone of your production environment with a click of a button.

Fortunately, the advances of Docker in production have not only made an ad-hoc environment possible, it’s become more popular. A few leading PaaS platforms and CI tools recently introduced similar features for their users.

For example, Heroku review apps feature allows you to create a separate version of your stack for every pull request your team has created in GitHub. GitLab’s review apps provide the framework and hooks you need to implement a similar solution to Heroku’s.

One company went even further and packaged this idea into a separate product.

Runnable allows you to create a working environment for every branch or pull request your team creates. The product is very similar to the options I mentioned above but isn’t tied to a specific cloud provider like Heroku or CI service like in the case of GitLab.

I haven’t tested Runnable personally, but they claim to support both Kubernetes and docker-compose formats, which means you can deploy all these dynamic environments directly to your own Docker orchestration infrastructure.

Create Your Own “Review Apps” Clone

If you read my article about choosing to self-host a technology versus paying for a hosting service, you probably already know that I usually suggest finding a service that will solve the problem instead of implementing a custom solution.

This is why I recommend checking Runnable first before spending resources on creating a Runnable clone in your Kubernetes environment.

But if you still want to implement a clone of Heroku’s review apps feature yourself, here are a few things you’ll need to think about.

As I mentioned earlier, in a high level, you’ll need to have a script that will need to do three things:

1. Create a new namespace in an existing Kubernetes cluster
2. Deploy the relevant Docker containers into it based on the versions currently running in production and
3. Eventually expose and assign a new DNS name to the front-facing parts of your infrastructure

This is the basic process, but it’s safe to assume that you’ll stumble upon additional challenges which are unique to each application.

For example, how are you going to clone the data? Most of the tests you’ll want to run on these environments will probably need a clone of a live data to work properly. What happens if your database is bigger than a few GB?

Another problem is third party OAuth services. If you integrate with GitHub’s OAuth to authenticate your users, you’ll need to configure an application in GitHub’s UI which is directly tied to your domain name. This is good for production or staging, since they have fixed DNS names which rarely change, but the situation is more complicated when you need to assign the DNS names dynamically.

But don’t be intimidated by all these problems. The good news is that there are solutions for all of them — you’ll just need to plan to invest the resources to write the relevant automations.

Conclusion

It might be much easier than you think to implement a mechanism that will allow you to quickly create ad-hoc environments. In turn, this will allow your development team to move faster by eliminating the bottlenecks related to staging and testing environments.

There are many options for how you can implement this functionality. If you don’t mind paying a little extra, you can use a service like Runnable, or if you already use Docker and host your own Kubernetes cluster, it’s also possible to implement this idea yourself.

I would love to hear about how you’ve automated the creation of environments per branch in your team. If you’re subscribed to my email list, just hit reply to one of my emails and let me know.

‍