In previous chapters we discovered the world of DevOps—where it came from and the fundamentals of the approach.
Now you know what DevOps is and you understand its advantages and main pillars, I’m sure you’re starting to ask yourself how you can implement it within your organization. Numerous attempts have been made to adopt it within workplaces, with varying levels of success.
In this chapter, I’m going to use some specific examples to show different ways in which DevOps can be adopted. This will include a number of recurring patterns, where the methodology has been implemented in an efficient and beneficial way, and also antipatterns, to show you what not to do. This will give you a feel for what good practice looks like when adopting the approach and make you aware of the pitfalls to avoid.
As you’ll see, in this chapter I’ll be drawing on some of my own personal experience implementing DevOps. You’ll be shown examples for most types of implementations. I’ll be looking in particular at the DevOps transformation I supported within the large corporation I’ve named Contoso.
The main objective when adopting DevOps in an organization is first and foremost to improve the delivery of added value for business areas and end-users. This doesn't necessarily mean reducing costs, increasing automated deliveries, or making everything fully configurable.
What it instead means is that organizations potentially have different needs when structuring their development and production teams to ensure they work together effectively.
A DevOps organizational structure will depend on a number of factors, such as:
the range of products within the organization. If it has fewer products to maintain, it will be easier to work together because there will be fewer natural silos to contend with, as Conway's law explains:
the scope, strength, and efficiency of technical leadership and whether developers and operations have a common objective, as mentioned in the previous chapter.
alignment of IT operations team practices with the value chain. If the IT operations department is focused on server and hardware configuration rather than operational functionality, some serious work will be required to be able to progress. However, not all organizations have the ability or willingness to progress in this way.
the willingness and ability of an organization to move their IT operations team away from “hardware racking” and “server configuration” toward true alignment with the value chain and for software development teams to take operational considerations seriously.
the skills and abilities that are present within the organization that are needed to take the initiative on operational issues.
Don't Fall Into the Trap! ❌
Let’s start by looking at some antipatterns. These generally start with good intentions, but gradually transform into a counter-productive idea that goes against what we’re supposed to be achieving.
Dev Silo and Ops Silo
The first antipattern we see within organizations—which DevOps aims to resolve—is the separation of Development and Ops teams. These teams are separated by the so-called “wall of confusion”.
On one side, we have developers whose role is to create solutions based on user stories, enhance applications, implement new features, and fix bugs. And on the other, we have Ops, whose role is to keep applications running, repair any servers that aren't working, apply security patches, and upgrade underlying infrastructure.
These two teams operate at different speeds, with one working at pace to maximize change, while the other ensures that the application remains stable. The teams are also given incentives—to fix the most bugs or release the most new features on the Dev side, or to control the number of critical production issues on the Ops side.
These two worlds simply don't understand each other and this ends up slowing down the pace at which applications are delivered into production. DevOps is an approach that aims to break down these silos and streamline interpersonal communication on the one hand, while putting tools in place to enable automated delivery on the other. This is why DevOps has experienced a meteoric rise for the last few years in many organizations throughout the world.
I had the opportunity to support Contoso in transforming their Dev and Ops teams by establishing a shared culture and setting up continuous delivery and deployment tools. In the next part of this course, we’re going to see how we can resolve the Dev/Ops silo problem, for example by establishing collaboration patterns or evangelist teams.
DevOps silo
To alleviate the problems we’ve previously touched upon, organizations might choose to set up a DevOps team in charge of implementing continuous delivery and deployment processes. This is usually a cross-functional team and all applications can use their expertise to help them set up automation tools.
Generally, this idea comes from someone in senior management, who attended a conference where people shared their experience in setting up such a team.
Unfortunately, creating it can generate a third silo, because the team will have to deal with:
developers asking for their help without understanding the fundamentals of DevOps.
Ops staff wanting to retain control of go-lives, closely guarding their tools and treating this new team with suspicion.
When I was supporting Contoso in their journey, one of these cross-functional teams had been created. It was in charge of setting up deployment tools for all applications. The creation of this team was based on good intentions, but they soon found themselves inundated with work because all the application teams wanted help setting up continuous delivery and deployment tools. This meant the team was unable to innovate, as it was managing an ever-growing list of tools based on what the applications needed.
What’s more, the Dev and Ops teams had no understanding of the fundamentals of DevOps and just relied on the cross-functional team to provide them with a service. Basically, they’d created a third silo, when DevOps is supposed to break down silos and help to streamline communication.
We were able to partially resolve this problem at Contoso by giving control back to the developers so that they could install and configure some of the continuous delivery tools themselves.
NoOps
With the arrival of cloud computing—and more recently, serverless—an increasing number of blog articles and conferences have been discussing the NoOps movement, where Ops teams are no longer required within the structure. Essentially, the basis for this movement is that the cloud platform and provider will manage all operations for the day-to-day running of the application (backup, disaster recovery, scalability, resilience, etc.).
Cloud providers even promote their services based on the fact that you just need one developer to deploy and manage the application. Unfortunately, even though DevOps allows us to break down silos and streamline communication, the role and skillset of a developer are very different from those of an Ops engineer. Developers don’t assess the complexity of an application and aren't often able to restore functionality of malfunctioning infrastructure.
A team that only consists of developers will often find themselves in need of an Ops engineer as the application becomes more complex. So, even if the developers have a good handle on what they’ve set up, they’ll need help from a cross-functional DevOps team or Cloud team within the organization.
I've found myself discussing this pattern with many teams. It’s a pattern that's generally used by startups, where the top priority is releasing a product that meets users’ needs. Ops concerns aren’t brushed under the carpet, but are dealt with later. It’s also used by teams who are creating a brand-new product, with a plan to deal with the ops issues further down the line.
Cross-functional tools team
Another sub-optimal idea when implementing DevOps is to create a cross-functional team that is solely concerned with tooling. As we saw earlier, a team like this will create a third silo, whereas DevOps is supposed to be removing silos.
This team is also only concerned with implementing tools to establish a continuous delivery and deployment pipeline and to manage configurations and environments etc. The team does nothing to change the culture of the organization and the Dev and Ops teams will continue to work in their usual way without changing their processes, relying heavily on this cross-functional tooling team to deliver the application.
Of course, the team will be of benefit in the short term, as the other teams won’t need to think about anything that doesn't directly affect the application. But the team will have limited impact because the processes, some of which can be particularly convoluted, won’t change.
When one of these teams was set up at Contoso, it soon became inundated with work because it existed solely to deploy and maintain delivery tools such as Jenkins, Nexus, XLDeploy, Git, etc. It was a key team because all the application teams had become completely reliant on it. To prevent this from happening, one idea was to bring control back to the Dev and Ops teams, giving them permission to install their own plugins and tools.
Ops Within the Dev Team
The organization might decide that it doesn't want to retain a separate Operations team, with Development teams instead taking on responsibility for infrastructure, environment management, monitoring, etc. However, doing this on a project or product basis means that there are resource constraints and changing priorities, all of which lead to tunnel vision and weak solutions.
In this antipattern, the organization shows a lack of appreciation of the skills required for efficient IT operations. In particular, the Operations role is devalued because developers treat it as a hindrance (because the Operations team is managed by a team leader with different priorities).
Ensure That DevOps is Properly Established ✅
Dev and Ops Working Together
To encourage communication between these two teams at Contoso, one of my actions was to invite the Ops team to attend the developers’ daily meeting, so they could be made aware of any new features being implemented within the current sprint.
Ops could also use these meetings to raise any points with developers regarding operational aspects of the application. They could provide their input on user stories to ensure application enhancements would run smoothly, or ask for all logs to be created in JSON format, so that they could be more easily imported into the ELK stack.
During the DevOps implementation at Contoso, I also asked for the application performance monitoring dashboard—including production and all other environments—to be set up in the developers’ working area. This ensured developers would be more engaged with the operational side. Developers gained awareness of the impact of implementing and releasing code on the day-to-day running of the application.
And finally, Ops were made aware of code quality using techniques such as Test-Driven Development (TDD) and Behavior-Driven Development (BDD), which produce regression testing reports for production bugs. This gave the team reassurance concerning the quality of the product they were going to put into production.
All of this requires a significant cultural change within an organization and the support of a senior sponsor, such as the director of IT Systems or Operations.
Dev and Cloud Team
For organizations with a traditional IT department that isn’t willing or able to change quickly, as well as ones that run all applications in the public cloud (e.g. AWS EC2, Azure etc.), it's probably useful to treat Operations as a team that simply provides the infrastructure where applications are deployed and run.
A team set up within the Development team (possibly a virtual team) can provide consultancy on all things operations, such as performance metrics, monitoring, server provisioning, and so on, and be in charge of communicating with the Cloud team. This is still a Development team, however it adopts standard practices such as TDD, CI (continuous integration), iterative development and coaching.
The Cloud topology provides a potentially efficient way of facilitating implementation (without direct collaboration with Ops teams), possibly leveraging added value more quickly than with the first pattern of Dev and Ops working together, which could be attempted at a later date.
External DevOps team
Some organizations, especially smaller ones, might not have the budget, experience, or staff required to manage the operational aspects of the application they’re creating. In this case, the Development team can use a service provider such as Google, Amazon or Azure to help them create test environments and automate their infrastructure and monitoring. This external provider will also be able to advise them on what types of operational features they should put in place during the software development lifecycle.
This is what we’d call DevOps-as-a-Service, and it could be a useful and pragmatic way for a small organization or team to learn more about automation, monitoring and configuration management. They could then move towards a “Dev and Cloud team” model or “Dev and Ops working together” as the team grows and starts to focus more on operations.
Temporary DevOps team
A temporary DevOps team is actually quite similar to the DevOps silo antipattern, but its purpose and longevity are very different. This temporary team’s role is to bring developers and operators together, ideally aiming for a Dev and Ops working together model, after which the team will become obsolete.
Members of this temporary team will discuss problems that are particular to them:
by introducing initiatives such as stand-up meetings and Kanban boards for the Ops teams.
by considering details such as load-balancers, network topology and SSLs for the Dev teams.
If enough people start to see the value of bringing Dev and Ops teams together, the temporary team has a real chance of meeting its objective. The long-term responsibility for deployments and production diagnostics shouldn’t belong to the temporary team, otherwise it risks becoming a DevOps silo.
DevOps evangelist team
One of the most effective patterns I've seen in organizations—and one which I tested and implemented at Contoso—is a team of DevOps evangelists or DevOps coaches. The role I had at Contoso was in fact DevOps coach. I helped the Dev and Ops application teams to implement DevOps and also to adopt continuous integration and delivery.
I worked in partnership with the cross-functional DevOps team so that the tools they’d deployed would be used to implement continuous integration and delivery pipelines. My role was to consult with the Dev and Ops teams for each application to raise awareness of the shared benefits of DevOps and to promote good practice.
My own technique was to invite the Ops team along to daily meetings, as I mentioned earlier in the course. But there are also activities such as the DevOps game, where you put yourself in someone else’s shoes so that everyone understands each other's role and the importance of communicating. I had other tools up my sleeve, such as project team dashboards.
My role was to support these teams for three to six months so that they wouldn't need me anymore and could implement DevOps without my involvement.
Working Together on Containers
One of the claims around using containers (Docker in particular) is that you can run the daemon to execute the container in the same way, no matter which operating environment you’re using. The application is held in a container that has everything it needs to be able to run (libraries, framework, middleware, etc.).
Developers just need to create an image for the application to be able to run correctly.
While operations run this image with all its operational constraints in place.
In this way, the container becomes a universal deliverable acting as a bridge between Dev and Ops teams. Because of the high level of automation and tooling, this pattern promotes seamless application delivery, since the deployment and operational requirements are held within the container.
Unfortunately, if developers start to ignore Ops recommendations around keeping the application running (for example, the logging format), this pattern then becomes an antipattern similar to the Dev silo and Ops silo.
In practice, and having seen it implemented many times in various companies, many teams think that the container is the ultimate solution for delivering applications quickly, and therefore to overcome operational constraints (environment delivery problems, problems installing the right version of the framework, middleware evolution, etc.). But in reality, an operating model must be put in place for these containers—because they must be patched, containers must change servers if they need to be updated, etc.
Let's Recap!
So there we have it. I hope this presentation on good and bad practice will help you better understand how you can implement DevOps in an organization. Let’s summarize these practices once more:
Bad practice ❌:
Dev silo and Ops silo. NoDevOps methodology in place, the traditional model.
DevOps silo. THE classic trap, creating a third silo when attempting to break down the other two.
NoOps. Er, no! The role of the Ops team is still important, even if it changes.
Cross-functional tools team. This fails to recognize a whole raft of DevOps concepts and creates a third silo.
Ops within the Dev team. This has a tendency to show a lack of appreciation of what the Ops team does!
Good practice ✅:
Dev and Ops working together. Collaboration is key to ensure the production team runs smoothly in DevOps mode.
Dev and Cloud team. A team whose role is simply to provide the deployment infrastructure. This could be an interim step on the path to a full DevOps implementation.
DevOps as a service. This is a popular option that suits small teams, where enlisting help from an external DevOps expert can be a good idea.
Temporary DevOps team. A DevOps team that only exists for the time it takes to put the DevOps approach in place and train the Dev and Ops team in good practice.
DevOps evangelists. Similar to the previous pattern, this is a team whose sole purpose is to train teams in the DevOps approach.
Working together on containers. Containers enable developers to put their applications into production more seamlessly.
In the final chapter, we’re going to delve into one ultimate example of good practice, which is the creation of the role of the SRE (Site Reliability Engineer).
