5 deployment traps we can’t seem to avoid
In this 7-part series we’ll look at some common misconceptions about the process of deploying software in today’s unforgiving world. Over the next few posts we tackle these myths head on and show how there is a better way.
Releasing software into the wild is exciting and terrifying. When it goes well: we party. When it doesn’t: we spend the weekend without sleep, showers, food or sleep. Wait! Did I mention no sleep already?
Too often the reason our deployments fail is because we fall into the same traps over and over again. We never have time to step back and do it right so we keep on doing it the best we can and that is where the errors creep in.
Here are five common traps we fall into that are easy to avoid and inexpensive to solve.
1: Every deployment is unique
There’s your whole problem. It is true that what is being deployed is (or at least should be) different each time you deploy but how it is deployed needs to be standardized and familiar so it becomes repeatable and predictable. Each time you update an application it likely that it has the same topology, the same dependencies, the same footprint and the same risks.
2: Every target is unique
This is a common problem too. In the Internet of Things every device on every hip, in every pocket and buried on our phone is a unique configuration of versions and patches and operating systems. Every server has custom settings and distinctive considerations that need to be accommodated. It is beyond any human’s ability to track and manage the discrepancies amongst so many deployment targets. But yet we try with our spreadsheets and notebooks.
3: Emergency fixes are different
On a Sunday morning, at 3:00 am, no one wants to modify the 27 deployment scripts and the 81 server config files or follow the defined procedures to stop the 14 databases and quiesce the 8 transaction queues just to make a simple change. A skilled developer can do what’s need by writing a simple PERL script, right? But on Monday morning, at 8:00 am, no one wants to explain to the CEO why overnight trading in Tokyo and Hong Kong was down either. In many ways emergency fixes have more risk because they are usually developed under pressure, tested less, by pass approval levels and get deployed in whatever way seems quickest. There is too great a temptation (or expectation) to do what is expedient over what it right because the right way is the long way. Yet we all know the automated way is the right way and the fastest (and safest) way.
4: Each deployment needs me
I remember sitting in a meeting room. Outside I could see two developers peering at a screen and nodding their heads up and down in an erratic and random manner. I asked the client what was happening and she explained they were deploying a release and “watching the script go by in case something bad happened.” If a deployment fails many release engineers will roll up their sleeves and start to unpick the changes and the rest will try to fix it on the fly and keep going. This is why deployments need their release engineers to be close during the process. But what are they going to do if the deployment stops (or worse flags and error and doesn’t stop)? They have to find the problem and decide “fix forward” or “backout”? They have to then work out which is best and easiest, quickest, safest to do. If this was built right into the scripts from the beginning they could devote their energies to finding out why it failed and fixing that.
5: Errors happen: its software
There is a sense that errors are inevitable. That something will always get missed and we can correct for that later. If we are agile and iterating in small increments the risks are low and the impact minimal. In our hearts we know that is bogus. One change to one configuration line in a server can stop it from executing and bring our entire system to a halt.
Conclusion
In the next post we will address these issues and look at what is needed to be effective and efficient in deploying software. As this series continues I would welcome your thoughts and experiences. You can add them right here.
Next week: why neither deployments nor target platforms unique.
