If you have been in the software industry for any length of time, you have heard or even experienced release management practices – the good, the bad, and the ugly.
If you are new to release management practices, it’s the “art” of building and deploying software into a consumer type of environment (yes, release management is an art).
In the words of Wikipedia: “release management is the process of managing software releases from development stage to software release. It is a relatively new but rapidly growing discipline within software engineering”.
The art of release management
I call release management an art because the build and delivery of software might sound simple, but it is not. The coordination of teams, tracking of build versions, and transparency of releases is not for the faint of heart. There is no “cookie cutter” way to achieve a well define and smooth workflow for all teams to follow. The release management process is always in a state of continual improvement.
Why do we need release management?
If release management is so difficult why do we need it? What benefits will come from the effort? Although it is not easy to implement in any organization the benefits of release management and DevOps out weight the effort and cost substantially. The more important questions to ask are:
- What is the cost for your production system to be down due to a rogue release or change?
- What is the cost of wait-time for Operations to release a new feature into production?
- What is the cost of not having transparency of which software version has been deployed to an environment?
- What is the hidden cost of non-collaboration between teams?
- What is the risk factor of manual steps for a release?
It’s about the risk
Release management is all about risk management.
Most companies use some type of agile practices for development and testing. In non-release management there tens to be iteration/sprint cycle-time of 2-4 weeks, along with a large amount of risk in the build, testing, and release process. As the development team create more features at a faster cadence, the release process (and testing) can become a bottleneck.
To make the release process go faster and meet the teams improved cycle-time, the “throw it over the fence” technique is usually used, as in “I’m done and it now over to Ops to deliver it into production”. This non-release management technique often creates even more risk, confusion, and is just plain bad!
As agile pushes the release management to build, deploy, and test faster through each environment, it normally stop short of the great barrier called, Operations. Which is the most costly of the environments for uncertainty and bad practices to live.
Those “Ops” people
What is wrong with those Ops people! Don’t they understand we need to get this simple feature out to production? Why are they so paranoid and tend to over react to everything?
There seems to be an ageless battle, Development does not understand Ops and Ops wishes Development would quit breaking production.
My recent favorite quote that sums up Ops view for Development is from The Phoenix Project: “show me a developer who isn’t crashing a production system, and I’ll show you one who can’t fog a mirror. Or more likely, is on vacation”.
The statement comes from a lack of trust between the two teams. And, from developer’s asking for access into production, while promising they will not break anything.
Here is a link to a great summary of what sometimes happens when Ops gives a developer access to production (this is from a very funny blog post titled “Pictures from a developer’s life” that is worth a visit).
Ops’ main job is to protect production and keep it running. Often they feel that it is a full time job just to defend the production system from “script kitties”, developers who try new commands as they learn them, and the internal threats of just making a small change while “looking around”.
Is it any wonder why Ops people are paranoid? Everyone is out to get the production system! Their main job is to protect production and keep it running.
Bridging the development and Ops gap
This is where release management helps. It bridges the gap and reduces the paranoia of Ops, by using logical policies and reducing the uncertainty of a release and changes.
Transparency is key! The “throw it over the wall” technique gets replace with a known, transparent process. All of the teams can see what version of the software is in an environment and if a feature is ready for a production release. In short the risk in building, deploying, and testing software is reduced.
The “what, when, why, and how” is what Ops is looking for. Knowing this helps them find and fix issues quickly when something goes wrong. In addition, the development and testing teams benefit by having a reliable, repeatable, fast build and deployment system, which is critical to building quality software as features become more complex or cycle-time increases.
Release management benefits:
- Efficiency – Reducing the time it takes to find, fix, and deliver reliable, quality software.
- Productivity – Teams can focus on more important items then trying to figure out what version or steps have caused an outage.
- Reducing risk –The ability to identify the what, when, where, and how for a release.
- Collaboration between teams – Opening a platform for the team to discuss blocking issues by using definite and transparent processes.
- Configuration management – Knowing what environment setting, application requirements, and dependences exist in the production, test, and development environments.
Even a small positive change to any one of these areas can greatly improve a team’s ability to deliver quality software to the customer.
The other change is a faster feedback loop. This is achieved by not having to spend time sloughing through what caused the environment to break or which build the bug came from.
Not having to be in “firefighting” mode and the “patch and pray” policy, the team’s ability to focus on continues improvements and opportunities are greatly increased. Not to mention the reliable, continuous delivery that all customers expect today!
Continuous improvement with release management
Creating, building, and delivering quality product to our customer base is the goal, as well as being agile enough to react to opportunities and changes in our industries is required.
This is why we apply continuous improvement to our team. We, as professionals, naturally strive to do better and improve, as should our software development practices.
Release management is a part of that continuous improvement model. Creating and refining a logical process to delivering products to customers in a fast reliable cycle is one important way we keep our customer and improve our customer base.
With faster cycles of software features being released, it is very costly to not have a solid release management or DevOps process. Release management helps bridge the gap between development and Ops. It breaks down the “us against them” factor and replaces it with the knowledge that we are all involved in the improvement process. Not using a release management strategy has its own internal cost that are not seen and rarely measured. Having high risk releases does not allow for teams to be agile or perform delivery and improvements that drive new innovations and quality into your software product.
What is next
Now that I have pointed out the reasons for release management in your software development process, it’s time to invest in the tools to create the infrastructure and workflow for all teams to use. Managed rollouts are not easy but very few things of worth are easy to obtain. In my next blog post I will cover some tools and techniques that will help you build your release management strategy.
My co-worker Martin Hinshelwood has also written a blog post covering Release Management with Team Foundation Server 2012, which covers the process and the tools (and is simply a good read 🙂 ).
Other recommended resources:
- The Phoenix Project
- Beyond Agile: Tales of Continuous Improvement
- Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
Wrapping it all up in 12 steps
If you simply skipped to the bottom and did not read the above sections, below is the summary of the benefits of release management and having a DevOps practice:
- Release management improves your team’s ability to produce quality software with faster feedback and shorter delivery cycles.
- The Ops team will stop giving you the “stink eye” when you ask them about the release night.
- Ops needs to be involved in the iteration/sprint and there should be no surprises.
- Confidence in the fact that the next release has been tested and not simply “thrown over the wall”.
- The hidden cost of low morale, late night bug fixing, and the blame-game.
- Transparency into the what, when, where, and how.
- Resolving configuration management issues between environments.
- Drive quality in software development.
- Faster more reliable and repeatable releases.
- Getting your team out of the “firefighting” mode and the “patch and pray” practice.
- Breaking down the team barriers and improving awareness across teams.
- The ability and opportunity to reach continuous improvement and delivery for the whole organization.