The tech world loves a good portmanteau—a word created by combining two or more words into one. Whether we’re talking about blogs (web logs), pixels (picture elements), or brogrammers (bros who program), these bundled words confer meaning without worrying about definitions. But things can get a little stickier with phrases describing technical practices, processes, and methodologies.
For example, what comes to mind when you think of the word “DevOps” (development and operations)? It’s been around for well over a decade, but you still might run into debate when you offer up a definition of your own. Lacking a singular, industry-wide definition, we can look to GitHub’s definition: “DevOps is a way of working where development, IT operations, and security teams partner together to build, test, and provide regular feedback throughout the entire software development lifecycle (SDLC).” Applied to other realms, this pattern has spawned a whole lexical branch of its own, with terms like DevSecOps, ChatOps, AIOps, MLOps, and GitOps. But can we apply this general definition of DevOps to its derivative phrases?
For many technical portmanteaus, we have agreed to collectively agree (or disagree) on their meanings without spending too much time in etymological debate. But when it comes to GitOps, which broadly refers to the practice of managing and deploying applications and infrastructure using the distributed version control system git as a source of truth, there has recently been a rare, collective effort to hash out a formal open source definition that lives under the OpenGitOps project.
The time is ripe for discussion. GitHub’s Octoverse 2022 identified infrastructure as code (IaC)—which alongside platform engineering and continuous integration and continuous delivery (CI/CD) form the foundation for GitOps—as one of the three big trends to watch for in the year ahead. To say that GitOps is on the rise would not meet much resistance. The latest Gartner Hype Cycle™ for Software Engineering, 2022 includes GitOps as an emerging technology that “would be transformative.” The report further states that “GitOps can be used to improve version control, automation, collaboration, and compliance.” All of this “translates directly to business agility and faster time to market.”*
Infrastructure as code is not new, but the technology has seen a dramatic increase over the past year. According to Octoverse, the Hashicorp Configuration Language (HCL) was the fastest growing language in 2022 in terms of year-over-year growth percentage, indicating a drastic uptick in the adoption of IaC itself. For companies that want to fully realize the benefits of practices like IaC, GitOps is an obvious next step, but first it might be good to know exactly what GitOps really means.
Defining “GitOps”
Weaveworks CEO Alexis Richardson first used the term GitOps in 2017, with buzz around the idea quickly growing in the world of cloud-native software development, which, according to its own definition, involves the building and running of “scalable applications in modern, dynamic environments such as public, private, and hybrid cloud” that are exemplified by “containers, service meshes, microservices, immutable infrastructure, and declarative APIs.” As quickly as GitOps’ popularity grew as a concept, so did the ways people used the terminology, and so Richardson set out to clarify its meaning by forming the GitOps Working Group in 2020 under the umbrella of the Cloud Native Computing Foundation (CNCF).
“I wanted to make sure that GitOps did not run into the same issues as DevOps or Agile, which lack clear definitions,” says Richardson. “The GitOps Working Group was designed to be super open, and very quickly, everybody joined and we defined, with the smallest possible number of statements, what GitOps is.”
That definition, the product of more than six months of discussion amongst more than 100 individuals and 40 companies, lives today in v1.0.0 of the GitOps principles:
GitOps is a set of principles for operating and managing software systems. These principles are derived from modern software operations, but are also rooted in pre-existing and widely adopted best practices.
The desired state of a GitOps managed system must be:
1. Declarative: A system managed by GitOps must have its desired state expressed declaratively.
2. Versioned and Immutable: Desired state is stored in a way that enforces immutability, versioning and retains a complete version history.
3. Pulled Automatically: Software agents automatically pull the desired state declarations from the source.
4. Continuously Reconciled: Software agents continuously observe actual system state and attempt to apply the desired state.
“We didn’t invent the GitOps principles, we took them from the best practices of the past 20 years—things that haven't necessarily been put together, packaged, and made accessible so anybody could just pick it up off the shelf and use it,” explains Dan Garfield, chief open source officer at Codefresh. “GitOps used to be a dark art, where you had one super genius that knew how to build all of the automation. But now most of that is open source on GitHub, and you can just grab it and deploy it.”
Companies like Codefresh and Weaveworks, both founding members of the OpenGitOps project, created some of those open source tools that are now key to implementing the GitOps principles. While the Kubernetes container orchestrator and git provide a common foundation for the vast majority of GitOps practices and tooling, a variety of other tools comprise the GitOps ecosystem today, including ArgoCD, Flux, Jenkins X, and GitHub Actions. Rather than focus on tooling, Richardson says the definition steers clear to remain open and agnostic.
“There's just enough definition to make it clear that there's more going on than just using Git, while also leaving room for future evolution and different implementations,” says Richardson. “This thing is going to evolve. It isn't supposed to be a standard or a kind of Bible that we all have to carry around or anything like that.”
If you come to GitOps from the world of DevOps—a likely trajectory—you might find yourself looking at this definition and wondering about the culture, a characteristic often associated with most of the other Ops portmanteaus. On this point, Richardson is clear that, at least in its definition, GitOps is not intended to tackle the fuzzier area of social interactions and culture.
"GitOps is ultimately meant to be a technology pattern, not a social practice,” he says. “Fundamentally, all the social aspects should be derived from the technical principles. Git is designed as a team tool and gives you a social context, because you have metadata, committed workflows, and group activities. But GitOps is not supposed to be a social movement."
With GitOps, all changes to infrastructure and application deployment are versioned, immutable, auditable, and reversible, offering guarantees that cannot be made with a cultural shift or by one team taking over the responsibilities of another.
“GitOps is about automating operations and shifting left, so that developers can perform operations tasks without having to always understand it in detail,” he says. “With GitOps, developers use tools like git to cause operational changes in a robust manner, allowing platform teams to focus on core platform engineering and operations, and leaving developers to focus on product, customers, rapid development, user satisfaction, and so on.”
What good is a definition?
The lack of firm, agreed-upon definitions for terms like “DevOps” and “Cloud” (as well as the long debates that ensued to gain industry consensus) was often offered as a reason for why GitOps needed a definition of this sort in the first place. That, and the fact that people use the term interchangeably with other established, but lacking-in firm-definition, terminologies.
“One of the early points of confusion was that people thought that GitOps was just infrastructure as code,” says Garfield. “Once people saw the principles listed in OpenGitOps, they realized that infrastructure as code is the declarative part, that's number one, but there's still three more principles to work out. That really changed the way that people thought about it.”
Richardson says he heard similar confusion early on between GitOps and CI/CD. “A lot of people went around saying GitOps was just a marketing buzzword that was no different from CI/CD, but that’s not true. They're now beginning to realize that GitOps is actually a really nice extension of what they've already been doing into the world of operations, adding capabilities like automated continuous deployment, verification, and operability—even for complex applications,” he says.
While confusion around what constitutes GitOps versus some other approach led to the current effort to define the term, John Allspaw, who is among those credited with popularizing the word “DevOps,” argues that a definition is only valuable if it enables productive conversation.
“How many debates are found in social media about what observability, DevOps, or Agile means? If one of the contenders wins, so what?” says Allspaw. “Is that definition open to interpretation? The answer will be yes, because that's how words work, and therefore, what you thought you were doing by solving a clear and agreed upon definition didn't do the thing that you thought it did.”
For Allspaw, discussing the topic at hand is more important than having a solidified definition. In fact, he argues that having a definition of DevOps early on would have caused more harm than good, because people would have either opted in or out of conversations about DevOps depending on what that definition entailed.
“People say ‘DevOps this, DevOps that,’ but I just ask them to describe what they do. It’s either DevOps or it’s not. They’ll have reasons for why it's DevOps in this way, or it’s not DevOps in that way, but it doesn’t matter. The point wasn’t to coin a term. You need words to refer to things, but I’m more interested in descriptions,” says Allspaw. “One of the best ways to keep from getting sucked into a misunderstanding due to an overburdened phrase is to just talk about the thing that it’s supposed to represent. If you want to say ‘deploying new code’ then say ‘deploying new code.’”
Gabriel Kohen, a principal senior engineer and fellow at Blue Yonder, says he has been performing GitOps for several years now in absence of any formal definition. “GitOps is a way of managing the operation of the organization by known and established processes of git—pull requests, the whole review process, the auditing, the immutability, all those kinds of things,” he says. “To me, that’s GitOps.”
While some, like Allspaw, may prefer discussing details than nailing down a definition, it can be hard to have a productive conversation when different people interpret basic terminology in different ways. Meghan Liese, vice president of product marketing at Hashicorp, which makes IaC tool Terraform and container orchestrator Nomad, has seen the lack of a common definition lead to confusion within organizations. Even different industry analysts have had different interpretations, with some seeing Kubernetes as a requirement.
“I feel like GitOps had become just another buzzword that people liked to pack a lot of things into,” says Liese. “I like that someone has actually said ‘These are the four things we care about.’ It gives us something tangible to respond to. Otherwise, different stakeholders might talk right past each other because they had different ideas of what something means.”
On this point, OpenGitOps project maintainer Christian Hernandez agrees. “People just throw it out as a buzzword. It's okay that it's a buzzword, but let's put some definition around it. Let's demystify it first of all, and second of all, let's define it so that it's accessible to everyone else,” he says.
For Kohen, having a definition can also serve as a tool of accessibility. “People who are very high in engineering think that everybody is like them—into these very difficult, intricate engineering practices—and can make an informed decision,” he says. “But most people are just starting out and looking for guidance. This is where principles come in handy."
Future-proofing for an evolving technology
If you take a closer look at the GitOps definition, you might notice some rather glaring omissions. The primary one is that of git, the source control management system namesake for which the idea (and name) is predicated on. But Hernandez says the absence is absolutely on purpose.
“We wanted to keep the definition open enough to where it offers a clear guide but isn't exclusionary,” he says. “For example, in the OpenGitOps principles, we don't actually mention git at all. We say ‘versioned and immutable.’ Git is a fit, but you don't necessarily have to use git. We aimed to make it open enough to where the definition could fit a lot of paradigms.”
Likewise, Kubernetes is not mentioned, even though most of those involved in creating the definition arrived from the cloud native space. Kyle Ruddy, director of technical product marketing at Hashicorp, agrees with this strategy.
“We see GitOps as a process, not a particular technology,” he says. “While many have put it into practice with Kubernetes operators as the mechanism, the key principle is driving infrastructure state from source code.”
Another thing to note about the current definition is the prominently displayed 1.0.0 version, which inherently implies further versions. For Richardson, the idea of GitOps 2.0 is already a thought process well under way.
“How do we scale beyond the initial limitations of the first implementations? How do we include other kinds of repositories than git, and where do they fit in?” he asks, pointing out that many are already moving to using Open Container Initiative (OCI) image registries in place of git, because of the different advantages they offer. “You can sign, distribute, and cache images, and then you can create hierarchies of OCI registries and use them like a CDN, so you can put them on airplanes and have offline caches. There's so much more you can do.”
And while there has been no formal effort yet, there have been rumblings in GitOps circles around the idea of creating an OpenGitOps compliance standard, that would further codify the principles to identify which technologies are interoperable. In terms of whether or not definitions are valuable, its usefulness in creating an ecosystem of compatible tooling would certainly provide one proof point.
A tool-agnostic definition also allows GitOps to live outside the tools that might otherwise hold it back. For example, the complexity of Kubernetes, and even git, can be seen as a hindrance to GitOps adoption, and Garfield sees this as part of where GitOps will go next, with the introduction of tooling to abstract away some of the finer points.
“Do I really want to generate lots of YAML and manifests by hand? It seems like a system can do that for me,” says Garfield. “I don’t think that the underlying system is going away anytime soon—Kubernetes and the API it provides are going to be around for a very long time—but GitOps and the tools that we use to operate it are going to get a lot more accessible.”
Already, GitOps has been growing rapidly. Codefresh’s GitOps certification has seen more than 13,000 enrollments since the beginning of 2022; GitOpsCon (a portmanteau of GitOps and conference) has already completed its fourth year; and FluxCD has been pulled down from DockerHub more than a billion times. Add an evolving ecosystem to the mix, and GitOps might seem like a bit of a moving target for a definition. But perhaps working to define a term can serve more than one purpose. The community can define the word for use cases like compliance testing and interoperability, in the way of a standard, and that act can simultaneously serve as a vehicle for discussion, bringing us right back to Allspaw’s objections around discussion offering more than definition. Perhaps we can have both.
According to Hernandez, who helped lead months of discussions, sometimes spending hours debating the choice of a single word (across several languages, no less), the message is reaching its intended audience.
“Before, you would search for GitOps and, besides getting stuff from Weaveworks, you might find someone writing a blog, and they would just totally get it wrong. And I don't mean ‘wrong’ disparagingly. They were just gathering the information that was out there,” says Hernandez. “But now, a little over a year later, you see people writing about GitOps and putting the OpenGitOps principles in their blog. I see that as a sign of success.”
But it isn’t full agreement with the principles that defines success here, says Hernandez.
“I've also seen some pushback, where people ask questions and point out the nuances, which is even more valuable,” he says. “That’s why versioning the definition is important, too. People ask, ‘What about this use case?’ and I think they have a point.”
