GitOps¶

GitOps is a modern approach to managing and automating infrastructure and application deployments using Git as the single source of truth. It combines the power of version control with declarative infrastructure and application definitions to provide a reliable and auditable way to manage and deploy changes.

With GitOps, all configuration and deployment instructions are stored in a Git repository, allowing teams to track changes, collaborate, and roll back deployments easily. This approach promotes a Git-centric workflow, where changes made to the repository trigger automated deployments, ensuring consistency and reproducibility across environments.

RAIL is leveraging a PULL-style GitOps model, where the cluster pulls the desired state from a Git repository and applies it to the cluster. This model is in contrast to a PUSH-style model, where a CI/CD system pushes changes to the cluster.

The GitOps Repository¶

The GitOps Repository is a Git repository that contains all the configuration and deployment instructions for a team’s applications. It is the single source of truth for the team’s infrastructure and application deployments. The repository is organized into directories, with a base directory for the application and resources available for all clusters, and a directory for each cluster where the application is deployed. Using Kustomize, Flux will monitor the subdirectories for each cluster and apply the configuration accordingly.

The GitOps Repository is organized into the following directories:

├── .sops.pub.asc                   # A public PGP key you can use to sign secrets that will be decoded in the clusters by SOPS (mozilla Secret   OPerationS)
├── .sops.yaml                      # A configuration file that tells SOPS which values to encrypt
├── base                            # This level contains apps and resources that's available for use in all clusters
│   ├── example-from-docs           # This level represents this app and its set of resources. It must include kuztomization.yaml that includes   the other manifest files
│   │   ├── deployment.yaml
│   │   ├── ingress.yaml
│   │   ├── kustomization.yaml      # This file includes the other manifest files
│   │   └── service.yaml
│   └── README.md                   # Love your READMEs, and feed and exercise them regulary and keep them lean.
├── osl1-test                       # This directory level representes the configuration thats unique for a specific cluster
│   ├── example-from-docs           # This directory level represents configuration for this app in the osl1-test cluster
│   └── RAIL                        # This is where the fluxcd controller finds deployments for this cluster
├── README.md
└── bgo1-prod                       # You guessed it, this is another cluster, for your production workloads
    ├── RAIL                        # ...and this is where the fluxcd controller finds deployments for this specific cluster
    │    └── some_app
    │        └── kustomization.yaml # Define a namespace and a fluxcd kustomization here in order to deploy   some_app
    └── some_app                    #
        └── kustomization.yaml      # This file includes the base directory and the other manifest files,   enabling the resources in the cluster

Kustomizations¶

In order to make sense of the GitOps Repository structure we need to understand what Kubernetes Kustomizations are.

First we need to establish some basic Kubernetes terms:

Resource: A somewhat abstract concept of the stuff might be realized or represented by a Kubernetes cluster.
Resource Object: An JSON-style object that describes the desired state of a Resource as well as attributes that describe its current status. Resource Objects are what the Kubernetes API returns and can mutate from its resource endpoints. Resource Objects are identified by the triplet of attributes (kind, metadata.namespace, metadata.name). The field apiVersion is also mandatory and determines together with kind how the rest of the attributes are to be interpreted.
Resource Definition: The schema that specifies what attributes can be present for a Resource Object of a specific kind and apiVersion.
Resource Configuration: An object that represent the desired state of an identified Resource. It is customary for many resource definitions to express the desired state within the attribute named spec. The Resource Configuration is a partial Resource Object.
Manifest: An YAML file (or stream) that contains one or more YAML documents that each represent a single Resource Configuration. Multiple YAML documents in a YAML file are separated by 3 dashes "---\n".

This is the structure of each YAML document (representing a Resource Configuration) in a Manifest:

apiVersion: v1
kind: ResourceType
metadata:
  name: something
  namespace: adm-it-xxx
spec:
  ...

A Kustomization is a directory in the file system that contains a file called kustomization.yaml that contains a single YAML document with the attributes:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - manifest.yaml
  - kustomization_directory
  - ...

The resources attribute is a list of paths which can either reference a Manifest file (with .yaml extension) or another Kustomization directory. The result of this is a new Manifest containing the concatenations of all the referenced Manifests. There might be other attributes in the kustomization.yaml file that modifies attributes within the resource configurations contained or adds new resource configurations. Read the documentation for kustomize for details on how to kustomize your Manifests.

You can run the command kubectl kustomize <directory> to stream the generated Manifest to stdout.

GitOps Repository Structure¶

Each RAIL Team has an identifier on the form adm-it-xxx and they are given a GitOps repository and a Kubernetes namespace (in each RAIL cluster) with the same name. RAIL clusters synchronizes the resource configurations in the team’s namespace from the kustomization found in the <cluster-name>/RAIL directory in the main branch of the team’s GitOps repository.

The component within RAIL that does this synchronization is called Flux, and more specifically the Flux Kustomization Controller. Its operation and status can be monitored by watching the Flux Kustomization resource objects within your team’s namespace. Run kubectl get ks or kubectl describe ks to list it or display details.

If the <cluster-name>/RAIL directory does not contain a kustomization.yaml file, then Flux works as if it created one by running the equivalent of the command kustomize create --autodetect --recursive in that directory. The effect is that all manifests and kustomizations in that directory and all subdirectories are automatically included.

It is recommended to run each application that the RAIL Team manages in their own sub-namespace. We achieve this by setting up the sub-namespaces with their own Flux Kustomization configurations from the bootstrap kustomization:

<cluster-name>/RAIL # contains configurations for the `adm-it-xxx` namespace
<cluster-name>/RAIL/app1.yaml   # sets up the `adm-it-xxx-app1` namespace
<cluster-name>/RAIL/app2.yaml   # sets up the `adm-it-xxx-app2` namespace
<cluster-name>/app1 # contains configurations for the `adm-it-xxx-app1` namespace
<cluster-name>/app1/ingress.yaml
<cluster-name>/app1/deploy.yaml
<cluster-name>/app2 # contains configurations for the `adm-it-xxx-app2` namespace
<cluster-name>/app2/ingress.yaml
<cluster-name>/app2/deploy.yaml

In addition teams usually want to run multiple instances of the same application on the RAIL clusters without duplicating the configuration. We achieve this by moving the app configuration to the base directory and setting up kustomizations that import it and tweaks some settings for each instance.

For instance we end up with the following structure when we want to run multiple instances of app1:

base
base/app1
base/app1/kustomization.yaml
base/app1/ingress.yaml
base/app1/deploy.yaml
<cluster-name>/RAIL
<cluster-name>/RAIL/app1.yaml
<cluster-name>/RAIL/app1-test.yaml
<cluster-name>/RAIL/app2.yaml
<cluster-name>/app1
<cluster-name>/app1/kustomization.yaml # imports `../../base/app1`
<cluster-name>/app1-test
<cluster-name>/app1-test/kustomization.yaml # imports `../../base/app1`
<cluster-name>/app2
<cluster-name>/app2/ingress.yaml
<cluster-name>/app2/deploy.yaml

Protect secrets with SOPS¶

Secrets can also be added to namespaces by creating encrypted manifests to the GitOps Repository. Secrets encrypted with the SOPS tool using the published public PGP-key can be decryptet inside the RAIL-cluster by Flux.

Preparation:

install the gnupg package –- which gives you access to the gpg-command line utility
install the sops package
run gpg --import .sops.pub.asc from the root of the GitOps Repository

Create a YAML-file (for instance a manifest containing a Secret) with values that you want to encrypt, like:

apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
type: Opaque
stringData:
  auth.txt: |
    app_feide_client_id::...
    app_feide_client_secret::...

run sops --encrypt --in-place myfile.yaml

Check that the secret are now replaced with a string that looks like ENC[...unreadable stuff...] and that a sops attribute was appended to the top level YAML structure of the file. While testing run the command without the --in-place option and inspect that you get the expected output on stdout.

Commit the file to Git in the knowledge that nobody can recover the secret even if they get access to the repository.

Remove an service from RAIL¶

To remove an service from RAIL all that is needed is to remove all YAML manifest files that declare the service and then push the update to the GitOps repository. In actuality only the declaration of the sub-namespace needs to be removed but for clarity it is best to remove all YAML files that belong to the service. If you have some common manifests declared in base for test and production it is best to wait removing them from base until you are ready to remove from both test and production.

If you for some reason want to delete the namespace manually, run kubectl -n adm-it-xxx delete subns adm-it-xxx-app. Be aware that if the corresponding manifest has not been removed in GitOps, then the namespace will be recreated in RAIL after a short while.