What is DevOps?

Introduction

Hello and welcome to what is DevOps? So in this video we’re going to talk through not just the book definition of DevOps but I’m going to give you some examples of how DevOps is applied in an organization and really what it tries to solve. So by that you’ll end up getting the definition of DevOps but we’ll go a little bit further in depth and just a basic definition.

What Most SDLC Orgs Look Like

So first let’s talk about what most software development life cycle organizations look like internally. All of them do this a little bit differently but the the main premise exists across the board. There’s some type of idea for a new feature or a new product and so that idea then goes to some sort of business analysis, feasibility, return investment understanding right? Whether that’s part of a team or part of product management or where that resides but that’s the analysis it goes through. And then, eventually, once they decide to do the thing, the feature, the product or whatever it’s given to application developers. Whether they’re doing agile or not almost doesn’t matter in this instance. So this applies to organizations that are still doing things in a waterfall method as well. And then some level of quality assurance and testing is done at some point. That could be again part of an agile team or it could be a separate group. And then the release management side now. This could be part of IT operations or a separate team or something like that but qa is going to give that code and say “yeah this can be released and deployed.” And then IT operations is responsible for managing the infrastructure and the coding, where the coding is housed and and really delivery of that that feature or that that piece of software isn’t done until the end of that that pipeline right? So um the problem with this type of concept is that there are handoffs. Just like a game of telephone, there’s pieces of information that are not passed. There’s maybe extra pieces of information that are passed that cause problems down the line. Those problems manifest themselves when it comes to deployments right? And this is one of my favorite views of deployment. You show this to somebody that has done a software deployment in the past and they immediately go “oh I identify with that.” Because deployments don’t really work all the time. And I’m going to give you one small example of why they may not work. And I’m going to call this snowflake but that doesn’t there’s no reflection of age or political preference here. It’s just the the word snowflake and basically showing that there’s a lot of differences in your environment.

Fixing Snowflakes with DevOps

So for example, I’m just going to do a basic example of I’ve got some new code. I’m going to deploy that to our environment. And I have three servers. So there’s really not a lot to to get boiled in the minutia of how many servers are being involved in your infrastructure. So in this case there’s two servers that are windows server 2016 standard and one that is data center edition. So slight differences. It’s still a Windows server. It should still work the same right? Well when you look at their windows update versions you see those are different, which means they’re patched at a different level, which then you know there is major differences in the way that they would operate um or at least very minor differences that may get lost in in troubleshooting or why does something work for one customer not for another. And this is really the reasoning is – if you don’t have standardization across your infrastructure, there could be differences. There could be things that cause an issue for one user and not another or for at one time and not another. And it’s easy to see here in an instance where you’re talking about three servers but what if you have an instance or an environment where you have tons of servers, hundreds or thousands of servers that are providing your production environment. And what about the differences in all of those servers. So this concept is a thing called “configuration management.” It’s actually a tenant of DevOps and one of the things that DevOps solves for. So this is an example of a small thing that DevOps tries to manage and maintain. What do i hear from clients when they have these types of problems that DevOps would be able to solve for.

What companies say that means they need DevOps

“The first is we need to get software releases out faster.” Maybe they’re only doing one or two a year. And they’re trying to figure out how to get you know updates out more quickly more or frequent to the market. “All deployments cause multiple issues.” Again back to the deployment thing. Every time we do a deployment there’s a problem. “Our production environment isn’t the same all the time.” It kind of goes back to that configuration configuration management principle. And then “IT operations is having a hard time keeping up with agile teams.” Agile teams want to quickly produce value and and have that value be seen – getting to the end of that pipeline if you will as quickly as possible, and IT operations sometimes doesn’t want to change. And we’ll talk about why that is. So let’s talk about the the main constituents in DevOps. And it’s pretty simple. It comes from the word right dev and ops. It’s the development teams and IT operations teams. And let’s talk about their motivations for a bit. Soon the development side, as I mentioned, they’re in most cases you know following some type of agile methodology and because of that there’s a lot of value seen in how frequently you deploy or how frequently you can show your customer the value of the thing that you’ve worked on. “Hey here’s the the blood sweat and tears that I’ve put into this project for the last couple weeks. I want you to see it.” Right? I want to show off my changes that the things that I’ve done, to enhance the value of the organization. And then conversely to that, you have IT operations who they don’t want to change anything. They’re penalized for downtime. They’re rewarded or potentially bonus for uptime. They don’t want to change anything because from change comes problems. And that they have to fix and sometimes that’s three in the morning or at another time when they don’t want to be doing that type of work, right? So this creates this battle between IT operations and the development teams. “I want to change this!” “No you’re not allowed.” And because of this you end up having what I like to call pseudo-DevOps. And that pseudo DevOps is really not DevOps and because of that people don’t like the term DevOps because it doesn’t really provide the value right they’re just saying DevOps and and not really doing all the practices and all the things that really mean a DevOps implementation.

Why DevOps?

So what are organizations trying to achieve with DevOps practices? And I think this is a very complex answer. So the definition that I’m using in this video is one that the DevOps community I think would agree with. It’s a not the Webster’s definition of DevOps but I think it highlights the most important parts. And we’ll talk about why those parts are important. So it’s “smaller batches of software work.” So smaller batches go through the system quicker. This is a phrase that lots of methodologists and folks in the DevOps community talk about. So smaller batches going through that that system, “deployed more frequently.” So we don’t want to do one or two deployments a year. We want to do one or two deployments a day. Very small changes deploying that to production and then “with less planning and more adaptability.” That basically says not without a lot of heavy stuff and we’ll talk about lean and the waste and whatnot. And then the last part is “by people working better together.” There is a human component here and we’ll talk more about that in a second. So that’s the definition the kind of the book definition of DevOps. And something we can build on from there.

The beginning and evolution of DevOps

So the early adopters of DevOps and the early part of the DevOps community – Damon Edwards and John Willis – they came up with this acronym called CAMS: culture, automation, measurement and sharing. The culture part is basically “I need to get IT operations and the development teams to see that the world the same to realize that we need to create a positive culture that is ultimately benefiting the customer or the marketplace or the company as a whole.” And if both of those constituents can see the good and doing in action then they should be able to work together better. And so there’s this this concept of culture and I hate that culture is first in this acronym, But i think it’s also showing how important it is and for folks in the technical realm to have to you know realize that culture is more important or one of the most important factors, it’s kind of a puzzler for folks that are really trying to adopt DevOps, which I think is a good thing because it makes them think. The the second part is automation. And that’s automate everything. You want to remove the possibility of a human mistake from almost everything. Whether that’s on the dev side or on the IT Ops side. So automate automate automate! This is where tooling comes in and we’ll talk a lot about tooling in this video as well. Measurement – the phrase with measurement here is if you want to get better at something you have to be able to measure it. And this concept means tons and tons of logs, tons and tons of graphical displays for that data, so that if there is a problem, whether it’s automated or not, you can go back and find out what was the state of the world what were the state of our systems at that time. This measurement allows you to then say “oh this is a problem. We need to get this down to that number. Or we need to increase this to that number.” And as a team, you can see that number and you can aim for that number. So that measurement allows a culture to kind of exist really. So the automation, measurement, sharing really contribute to how the culture is organized and then the last um part of this acronym that the sharing portion. And I think the base concept here “shared vision shared understanding, shared responsibility.” And if everybody shares in that then that kind of fosters the proper culture, if you will. So that was the the original founding concepts of how to deploy or how to provide a DevOps practice in your organization. And there’s a lot of folks that have followed this acronym and over the years.
Jez humble came through and Jez is the author of many books in the CI/CD and DevOps space. But he added the the lean portion to this. And basically to help engineers both on the the development and IT operations side understand that this doesn’t have to be this really hard convoluted large process thing. DevOps can be very scaled down and lean and applying that to a DevOps practice is really important at the end of the day. So that’s CALMS and and the lean portion was a really important addition to that.
And then you have the Scaled Agile view of of DevOps. And rather than the “S” for sharing, they changed that to “R” for recovery and I like that from the standpoint of of to get the culture you have to be able to work well together right and and not that sharing is a bad thing. Sharing is to me one of the most important parts of it. But I think that really could be encapsulated in the “culture” aspect of this acronym. But what the calms acronym did not talk about was if we’re deploying something and we’re deploying very frequently how do we get back to a better state if we deploy a problem? We deploy something we go “uh oh that’s not good!” How do we roll back? And I think having that recovery feature inside of this concept is really important because you have to be able to have that as a founding part of a DevOps implementation.

So the culture thing we talked about you know. “If you just do the culture thing that’d be great!” It’s very hard to do and I almost think that it shouldn’t be the first thing because people start to focus on that alone. But really there is a book. There’s a guide for how to implement a lot of these things and it’s the state of DevOps report. That’s can come out for many many years. The things that came out in 2014 are still true today and it talks about you know things like version control for all the things in production. And if you don’t have that today that’s a practice you should probably implement. Each one of these you know very high performing IT companies have a continuous integration and continuous deployment process which is very defined. They have automated acceptance testing. They have peer reviews of their production changes which I think you know ultimately will create a high trust culture. They have proactive monitoring and measurement in the production environment. And that ultimately creates a win-win environment with dev and ops right? So this not that it’s a playbook per se but you can start to identify things in your organization that you don’t have just by looking at the state of DevOps report. You’ll find out what you don’t have.

Amazon & DevOps success

And a little success story I like to talk about in the DevOps world is Werner Vogel’s at Amazon. He said this in the early 2000’s and it was 2004-2005. “You build it you run it.” And the concept here was each team inside of Amazon would build an API for another team to use. So imagine there was a team that built the login feature and a team that built another feature and another feature. In the end another another team may need to call the login feature so they would just you know use their API and call what they needed to. But the problem was is if they had you know they would just put some documentation up and not really service it. His point was “if you’re going to build something for another team to use you have to be able to be the customer service support desk for all the things that they built.” So that allowed people to have to talk or it forced people to have to talk and communicate and say “oh we built this thing and that’s not as useful for you but if we tweak it maybe that would be better. Ok, cool. I understand what you’re doing and I can help you now. And now I’m going to go back and enhance something and make it even better.” And that culture of kind of shared responsibility and being able to communicate with one another actually made the output better for what they do. And so if you look at some of the stats that are out there for Amazon… 11.6 seconds between deployments during the week. 10,000 hosts receiving a deployment at the same time. Over a thousand deployments in a single hour and over 30,000 hosts receiving a deployment. And this is not now. This is in May of 2011. So yes Amazon is a newer generation business and they didn’t they’re not dragging a lot of legacy technology forward but I think if a company like this can can use those concepts to get these numbers, companies that that do have legacy technology should be able to over the course of you know five or ten years be able to convert a lot of that technology into newer stuff that can be “DevOpsy”. That that can have these types of stats.

Continuous Integration, Continuous Delivery, Continuous Deployment

So I’ve talked about some of the continuous words a little bit in this video. And I want to get into some definitions of that because I think to understand the concept of DevOps and its definition you have to understand these continuous words. What they mean and what their ultimate goal is. So continuous integration you’ll hear this referred to as “CI”. It’s doing a build and a test every time that you you change the system. That could be every day. That’s a good practice. And it also is this concept of making sure that you’re working off of a shared trunk and not feature branches. Feature branches in general nowadays are a bad practice and if you disagree with that statement, go read a book that I’m going to talk to you about here about a transformation with HP and how some small changes in their understanding of continuous integration actually revitalize their business. So we’ll talk about that here in a bit. But continuous integration again builds and tests are done all the time very frequently. Continuously essentially! So that you don’t have these big windows of all everybody has to integrate their code. The next continuous word we’ll talk about is continuous deployment. And this is after a CI build passes, getting it automatically to production or potentially some other validation or you know performance testing environment. But basically this thing is good to deploy. Right? This thing is in a in a production environment after it’s been automatically tested and the build successfully passes. So not every business can do this which is why you have the next continuous word which is continuous delivery. And this is when you hear people reference “CI/CD”, they’re really talking about this concept of continuous delivery. And this is really trying to be able to to get to a continuous deployment state but it may not actually hit production. And so this is a great example of this is there’s updated firmware for a printer or a server or a laptop or a desktop. It’s in an available state. It’s maybe on a support website where people can download it but it’s not actually deployed to the equipment itself. Same with mobile apps. The app is updated in the app store but your phone hasn’t actually downloaded it and updated it quite yet because Apple or Google do not allow you to do that 20 times a day. It’s only done you know once or twice a week or something like that. So continuous delivery is the state that most businesses would like to get to. Continuous deployment is a great state of zen where you know the CI build passes and it gets directly to production. But continuous delivery is the state that a lot of businesses will want to focus on. How do we get it to a state that can be delivered.

And to illustrate this I’ve actually got a graphic here that walks through what this looks like. And I’ve used days but this could be hours – any unit of time that is generally small but I’ll use days for an example assuming that the the CI build only happens once a day. Let’s say you have a large system and that happens at night because it’s gotta run security test or something like that. And that takes time. So the first day is this first section here. A developer will check in the code inversion control that will trigger a build and at night that developer will then get an email saying that it’s successfully built and let’s say it didn’t you know. It’s red here. So that developer on the beginning of day two comes into their inbox. They see an email “hey my build didn’t pass. There was an error.” And that second day the developer can then go “oh I made this mistake. I forgot a semicolon. Let me update that and push it back in,” and let it build again. It builds successfully. It goes the automated acceptance test and it fails. It emails that developer that night. Beginning of day three, the developer comes in they see they haven’t have an error that it didn’t meet the acceptance test and they go “oh it’s something I can fix real quick. It’s right in front of me. I was working on that yesterday.” Day three, they make that change. They check it in. It passes the build and the unit tests it gets through the automated acceptance test. It passes the user acceptance test and it’s ready to be released. Somebody clicks a button says “yes this can be released to production.” So this is an example of a deployment pipeline and what that looks like through the build and test criteria. Now the reason continuous integration is really important is because this is over a short period of time. Now imagine this could be you know hours as well right. Hour one, hour two, hour three. How quickly that feedback comes is based upon how the build is done. Does it take you know 30 seconds or does it take an hour and a half or five hours or whatever. So there’s you know a unit of measure let’s say is a day or less. But think about how quickly a developer can realize “oh I did this yesterday and I can fix that code today.” As opposed to a mistake is made in a waterfall environment…a mistake is made and you got six months before integration happens. That developer may be on a different project or may not be with that company anymore. And then the integration fails and somebody has to figure out why did it fail. And this delay is really adding to the delay of fixing and getting something to be deployable. By not having a continuously integrated system like this. So short intervals you know – no more than a day – something like that that that you can see that feedback and get that change and get it back in the system.

DevOps tools

Tooling is important here. Really really important. But keep in mind you know the CALMR or the CALMS acronym. Culture is the first thing and and even in the CAMS acronym, Sharing is on there too. So don’t get focused only around the tools. The tools should help create that that proper culture and that environment that allows the work to get done right. So I’m going to go into tools now but keep in mind that it’s not the the most important thing. It’s just an enabler to allow the organization to work together better. So if you’ve ever seen this, this is the DevOps periodic table of tools. There’s tons and tons of tools on here. I was showing this to somebody the other day and they went “Wow, I didn’t realize there was this many tools!” And there’s really even more than that. This is just kind of a of a first pass for many years ago. If you’re not a DevOps person, you’re not an IT engineer, you’re not a IT operations engineer, you’re not a software developer or even if you are you don’t have to understand all these tools. I say focus on just the top legend there. That legend is really important to understand the types of tools that are in existence in this chart. And I think that’s really where the power comes from to understand tools better and understand why you would want one tool over another or why these tools exist. So I want to run over just some some quick notes on those categories so that you understand those better and why they’re important so source code management, SCM. This is things stuff like Git, GitHub, Bitbucket. These are tools that will version your code and will also allow multiple people to work on the same source code at the same time, which is great. And then it’ll also help merge those which is continuous integration so that’s stuff like Jenkins or Bamboo or Visual Studio that merges the code together, all those changes from different people and it does you know some basic tests right. It’s getting it all back together and can it build. Deployment – so this is something like smartfrog which actually puts the code the past build that you created in your CI and puts it into production. Cloud – Infrastructure as a service, platform as a service – so this is your Azure, AWS, Google Cloud Platform. Things that you’re used to from a from an infrastructure perspective. BI and monitoring – so this is stuff like Nagios, xabbix, New Relic. This gets to that measurement portion that we talked about in the the CALMS acronym. Being able to monitor and measure those types of things. Database management – like Liquidbase. This is ‘how do you manage your data infrastructure’. Automate from an automated perspective right. This is “my dba doesn’t have to be in the database every day all day.” Repo management – so this is stuff like Artifactory which will allow you to kind of encapsulate all of the things that are artifacts in your environment that you want to make sure that is kind of a little bit of saran wrap is put around so that you know what all goes together. And then configuration management – which is stuff like Chef, Puppet, Ansible. These are from the beginning example that I gave about you have different versions of of windows update. The stuff like this will help you identify those types of problems. The needle in the haystack so to speak. So you can go and fix those things and get them all standardized so that they’re the same across the board and you can actually use this. Those applications for provisioning of new services as well because it’ll understand what your base image is so to speak.
Ok so next is things like really release management like urban code. These are “this is how you deploy something”. How you’re gonna organize it, and how you’re gonna structure it. Logging which are tools like Splunk and Sumo Logic which take a lot of these logs and analyze that and give you smarter dashboards to look at as opposed to trying to understand what’s going on in 50,000 different pieces of equipment and virtual environments. Build – which is stuff like Maven and Broccoli and even “make”. This is just you know compiling the code that’s been provided to it. Testing for automated testing is stuff like Selenium and Cucumber, which because it’s automated tests it allows it to get through the testing phase more quickly as opposed to waiting on a manual tester to test, and do all your test cases. Containerization – stuff like Docker and Kubernetes. This is allowing multiple versions of a certain type of software to sit on an operating system. Just kind of containerizes each one of those and so you can call and use just the ones that you need which is very very beneficial for some legacy, older technology and allowing it to have multiple versions on the same physical or virtual equipment. Collaboration – this is stuff like Jira, Slack, ServiceNow. Ways that you can organize your workflow and multiple people and you can all march to the the beat of the same drum. And then lastly is Security. So tools like Snort that are gonna organize your environment and make sure that you know there’s no outside attacks. And if there are attacks, what are the the procedures and policies that it’s going to go through. So this kind of gives you an idea of the the categories and the tools that are in those categories and why those are important. And they’re they’re the main constituents inside of DevOps from a tooling perspective.

Conclusion

Alright so let’s wrap up. I did mention a book earlier and we’ll talk about that. That’s actually the the middle one. That’s “Leading the Transformation.” So if you disagree with me and my point of feature branches are bad please read that book. Another great book – “The Phoenix Project.” That’s a fiction book and walks through a lot of scenarios that folks in IT operations and engineering probably go through. And it really talks about how DevOps was implemented, almost from the ground up, to solve for a lot of problems. The book on the right it was written by Jez Humble and David Farley on “Continuous Delivery” in 2011 and if that date is important, think about the agile manifesto that was written in the very early 2000s and it talked about the continuous delivery of software and there wasn’t a book written until this book was written in 2011. So pretty interesting book. It does read like VCR instructions so beware but it’s a really good book. And it talks about the infrastructure and the methodology by which you can employ a continuous delivery environment. Another as I mentioned this before another really important read is the State of DevOps report. As of this video, the 2021 State of DevOps report is out. Definitely go read that and read that every year. It has some really good information and very very valuable if you’re planning your DevOps journey. And that’s all we have for this video. So thank you for for watching and I hope you learned something.
Thank you and have a great day!