Let's get to know GitOps

What is GitOps?

GitOps is an operating model for modern application and infrastructures that is built around automation of developer experience.

It takes DevOps best practices (e.g. version control, collaboration, compliance, CI/CD) to automate infrastructure and manage application configurations. GitOps is used to automate the process of provisioning infrastructure.

Today’s large infrastructure demands the implementation of high-degree of automation. However, unlike the software development lifecycle, infrastructure has partially remained a manual process that required specialized teams and knowledge. This is important for the effective management of cloud resources that are needed for continuous deployments.

Many tools can be used together to build a GitOps workflow. For example: Git/Gitlab repositories, Kubernetes, Continuous integration/Continuous delivery (CI/CD) tools as well as Terraform or other IaC.

How does GitOps work?

IaC + MRs + CI/CD = GitOps

IaC (Infrastructure as code)

GitOps uses a Git repository as the single source of truth for declarative infrastructure and applications. GitOps combines the Git version control system with Kubernetes’ convergence properties.

  • Git is an open source version control system that tracks all changes in any set of files.
  • IaC is the process of managing and provisioning IT infrastructure (like networks, virtual machines, load balancers, etc.) using configuration files (manifests) instead of manual provisioning processes.

 

MRs (merge requests)

MR is a Git mechanism that merges development branches into stable (often called main) branch. It is where teams can collaborate via reviews and comments and where formal code review and approvals take place. GitOps uses merge MRs as the change mechanism for all infrastructure updates. A merge commit is typically opened to the main branch and serves as an audit log.

 

Continuous Integration & Continuous Delivery (CI/CD)

Git workflow automates infrastructure updates with continuous integration (CI) and continuous delivery (CD). When new code is merged, an automated process can trigger the descriptive state of a pipeline that adapts the infrastructure accordingly. Manual changes, infrastructure drift or errors are overwritten and fixed by GitOps automation so the environment converges on the desired state defined in Git.

GitOps Features

  • Versatility: Can be applied to all types of infrastructure automation including VMs, containers and even Bare-Metal
  • Developer-centered: 
    • Empowers developers to perform tasks which are typically limited to IT operations
  • Lets developers choose the tools they need or use the same tools and processes as for software development
  • Observability: Ensures that the desired state and the observed state (or actual state) are the same

  • CI/CD: End-to-end pipelines and Git workflows are applied to both operations, and development

GitOps Benefits and Challenges

GitOps Benefits

Organizations can benefit from GitOps that allows them to manage their entire infrastructure and application development lifecycle using the same or similar tools. Let’s take a look at at some of the main benefits of GitOps workflow:

More Productivity

  • Faster Deployment:
    • Continuous deployment automation with integrated feedback control loop 
    • No need to switch tools for deploying
    • You can ship 10-100x times more changes per day

  • Faster Development:
    • Best practices allow developers to use familiar tools and work faster
    • Continuously feature updates increase agility and response speed
    • Experimenting with new infrastructure configurations is easy thanks to the “revert” functionality in Git.

  • Team Efficiency:
    • All team members can review the Git commit history and changes over time
    • Anyone can reproduce it and find examples of how to set up new systems

More Stability

  • Security:
    • Changes are tracked and logged securely
    • Audit trail helps meet SOC 2 and other compliances
    • Comparison tools help to ensure that the tracked and auditable changes match reality
    • Least privilege principle: Separation of responsibility between packaging software and releasing it to a production environment reduces the impact of compromise and provides a smaller attack surface

  • Privacy:
    • Manage deployments completely from inside your environment
    • Minimal access to Git repositories and image registries can be enough
    • No need to give developers direct access to the environment (e.g. Cloud account)

  • Reliability & Recovery:
    • Revert/rollback and fork features ensure stable and reproducible changes
    • Single source of truth facilitates recovery in case of a meltdown
    • Meantime to recovery (MTTR) is reduced from hours to minutes

Consistency and Standardization

  • Having one work model enables consistent end-to-end workflows
  • Continuous Integration and deployment pipelines are all driven by pull requests
  • Operations tasks are fully reproducible through Git
GitOps Challenges

Collaborative processes always come with the challenge of requiring a lot of discipline and commitment from all participants. Let’s take a look at some of the main challenges in GitOps and how to overcome them:

  1. GitOps approval process
👎 Challenge
  • How it works: 
    • 1) developers make changes to the code and create a merge request
    • 3) an approver reviews and merges these changes
    • 4) the change is deployed
  • Engineers used to quick, manual changes might find this process tedious
  • GitOps workflow can be hard at first in some organizations
👍 How to overcome it
  • The fewer “cowboy engineering” cases there are – the better GitOps works and the more stable the infrastructure is.

     

  1. CI and CD separation 
👎 Challenge
  • The CI pipeline handles the code building and testing
  • A separate CD tool used for deployment (push- or pull-based)
  • Deployment is triggered with Git push, separating both functions
  • Difficult practical implementation
    • Access to source code and all Git parameters is needed for Deployment
    • Conducting smoke tests after deployment can be a challenge
👍 How to overcome it
  • For pull-based deployment, the Kubernetes cluster monitors changes and applies them automatically bringing the deployments to desired state

     

3. Multi-environment 

👎 Challenge
  • Multiple environments are difficult to manage:
    • All the environment details are in the Git repository
    • A new Git push has to be executed with each change of environment
    • Creating and tracking different branches in Git becomes troublesome
👍 How to overcome it
  • Automation for the rescue:
    • Use reusable code to replicate environments
    • Create a metasystem that handles the different environments and automates the respective Git push to set up an infrastructure environment (Especially large-scale applications)
    • Dedicated repository per environment can be a good option

       

  1. Automated updates
👎 Challenge
  • Git is designed for code editing and conflict resolution
  • Automation required for Applications in a CD process that get updated frequently
    • The CI process has to create the pull request to release to the test environment at the end of a Continuous Integration process
    • Multiple CI processes can end up writing to the same GitOps repo, causing conflicts
    • Two processes can clone the same repo and one can push changes sooner than the other
👍 How to overcome it
  • You can use more repositories (e.g., one repository per namespace)
    • Or use sequential automation pipelines

       

  1. Limited visibility
👎 Challenge
  • The larger the number of GitOps repositories and/or configuration files, the harder it is to find them
  • Git logs and diffs can only be comprehended by developers
  • Project managers need information at a higher level
👍 How to overcome it
  • Build separate tools that read information from Git logs
  • Draw visualizations and diagrams
  • Design a Grafana dashboard

     

  1. Difficult to implement rollbacks
👎  Challenge
  • Multiple past commits can complicate setting up the rollback process
  • Different software development teams might approach rollbacks differently
  • Automating rollbacks is not well-defined
👍 How to overcome it
  • You can select a stable release as the default rollback option
    • Same state is used even if there are stable working states released later
  • You can revert everything to the last known working state
    • The state to fall back to will be tried and tested
    • Automated rollback procedures can make the system resilient
  • You can package applications as Helm charts
    • Helm provides an easy way to rollback k8s based workloads

       

  1. Difficult promotion of releases
👎 Challenge
  • Git commit offers the path to deploy the code to an environment
  • But there are no standard procedures to promote the code to different environments 
  • Having multiple commits for different environments generates overhead
👍 How to overcome it
  • The general promotion path: DEV → TEST → PROD 
  • Defining pipelines to promote changes automatically across environments makes promotion easier
  • It is possible to use dev/test/prod branches of the same repository and merge code between

     

  1. Cost overruns
👎  Challenge
  • Dynamically scalable architecture for different environments can increase costs 
  • Developers’ concern is performance and ease of use
👍 How to overcome it
  • Monitor costs regularly across all infrastructure and projects
  • Analyze the past spending behavior to identify usage patterns and waste of resources
  • Forecast and budget for cloud spending using past data
  • Scale down resources when they are not used

GitOps Use Cases

  1. Software Delivery lifecycle
  • Whether a multi-tier microservice application or a monolith
  • Can also be packaged as containers deployed on Kubernetes
  • With GitOps you can:
    • configure your applications and track any changes
    • deploy your application versions to different environments
    • release the target version to production with a single commit
    • rollback previous version easily in case of any issues

  1. Infrastructure Management
  • Whether a Cloud based or on-premise infrastructure
  • Any infrastructure that can be described with a code (IaC)
  • With GitOps you can:
    • scale and change your infrastructure setup as needed
    • manage multiple infrastructures with single source of truth
    • replicate infrastructure easily

  1. Static Websites
  • Static sites have made a comeback for its simplicity
  • It might not seem like static sites need GitOps
  • But for complex static websites to be easier to edit than raw HTML, a set of markdown files is needed
  • It needs at least one build to make it publishable
  • With GitOps you can:
    • build up a site 
    • optimize an image
    • copy the deployment to the web server, etc.

4. Writing Docs or even Books

  • Product documentation is text-based and can be created as markdown or Ascii docs and easily be stored in Git.
  • Then a CI tool fetches the changes and deploys the new version of the documentation. The deploy stage can upload the latest version on various platforms as new changes are committed to the documentation.
  • Books are text-centric and can be aligned with a VCS system
  • When the book is finished GitOps pipeline can:
    • run automatic grammar & spelling checks
    • build these into various formats (e.g. doc, pdf, ePUB, etc.)
    • publish it to your cloud drive or a static website

  1. Contracts and statements of work (SOW)
  • Git’s great change tracking ability is ideal for Contracts and SOW
  • These and other legal documents often undergo multiple changes before they’re finalized
  • You can use GitOps to:
    • apply the changes after each revision via git commit

(Series of git commits are easier to look at than change tracking in a word processor)

  • pull requests to ensure there’s a complete record of every change
  • check spelling and grammar
  • export text into a PDF to send to the client or partner

Conclusion

As the technology landscape continues to evolve, GitOps offers the most advanced approach to automate the management of applications and infrastructure today.

GitOps can help organizations enhance their development processes, accelerate deployments, and achieve greater stability in an increasingly dynamic and fast-paced environments.

Are you ready to unlock the benefits of GitOps for your organization? Contact us today!