How to Stop Saying “It Worked Fine In Development”

Everywhere you look are pizza boxes and half-empty Red Bulls. Admins, illuminated by the glow of 30″ monitors, sit hunched over keyboards typing furiously. It’s 3 AM, the day of your new software release and right now those admins are desperately trying to rollback a botched deployment.

Suddenly the status widgets, which had to be silenced to stop the flood of emails and text messages, blink green. Disaster averted, you send your team home while you prepare for post mortem. After spending hours trying to figure out what went wrong, you are no closer to the answer. After all, it worked fine in development.

So let’s unpack that last statement a second. What’s so special about Development? Why do applications always seem to work there, but not in production?

The answer is that it’s development. It’s not Production, it is different. We constantly allow things to happen in Dev that we would never allow in production. We relax the root ssh requirement, have less restrictive firewall rules, newer versions of Java, and the latest Ruby gems. Development often times looks nothing like your production environment.

Are you ready to put a stop to that? Here are three ways to get started.

1. Provision Machines From Ground Zero

The cost of the cloud is cheap. The cost of keeping sysadmins at the office until 3 AM to rollback a botched deployment is not. Whether it’s your own internal IaaS offering or Azure’s new hotness, it’s time to stop treating your servers as pets.

Always provision from nothing, or at least a sane baseline, and let your configuration management tool do the rest. Environmental problems should NEVER be fixed by hand. Hand fixes aren’t documented and are a major cause of “It works on my machine.”

When your infrastructure is defined as code you can reliably and repeatedly build the same infrastructure across your enterprise. Fix the problem in your configuration management (CM) code and rebuild.

2. Create Security Requirements for Dev Too

Security shouldn’t be an afterthought. It’s part of your environment and should be treated as a first class citizen.

Stick to your security policy no matter what environment you are in. Don’t allow anyone to bend the rules without a REALLY GOOD and REALLY DOCUMENTED reason. Relaxed security rules in Dev almost always lead to security holes in Prod.

Production isn’t the only place you should be scanning for compliance. Tools like InSpec allow you to describe your security requirements as code and easily assess your security posture no matter what environment you are in. Bonus points for bundling it in with your developers Rake file so that they can validate compliance even on their local machine.

3. Abstract Environments with Containers

Security tends to make developers cringe. The old saying goes that the more secure something is the harder it is to use. This way of thinking is especially prevalent when the developers have to worry about the underlying operating system.

Container technologies like Docker allow you to abstract the underlying infrastructure. How many of you would like to code in the exact same environment on your laptop as is running in production? Docker, in many ways, is the answer to the question of why your code works in Development but not in production.

Docker images function like version control for your applications entire environment. The image contains ALL of the dependencies required for your application to run.

When you deploy a container from that image the environment goes with it. Not only does this prevent the dreaded up until 3 AM rollbacks, but think of all the money you are going to save on Pizza?

Want to learn more? Here are a few articles.

Happy Hacking!

Filled under: Blog, container, DevOps