GitOps

Czyli konfigurowanie Kubernetesa Gitem

Marcin Jasion

SysOps/DevOps Warszawa MeetUp #48 - 27.02.2020

How we manage Kubernetes today?

Command line

kubectl create/apply/edit/delete ...
kubectl create/apply/edit/delete -f some.yaml

This is Manual

• Human as a source of truth
• Possible situation, when two developers are applying the same object - communication needed
• Hard to figure out diffs

Failure story #1: YAML Overriding

Other problems

• Assume now K8S as objects store(like records in database). And we store object definitions in Git.

  So when we delete file from Git... not everyone remembers to delete the object.

• Hard to compare changes before apply.
• Even when we try to commit changes before apply, often we forget to push the code.

Ok. those are problems

Are there better options?

Yes

But first let's determine our needs.

The entire system is described declaratively

• Configuration is guaranteed by a set of facts instead of by a set of instructions.
• Cluster can be easily reproduced(e.g. business recovery tests; site clone).
• Reduced meantime to recovery(MTTR).

System state is versioned in Git

Changelog and rollbacks are easy

• git log
• git revert

Access to the repository may be restricted to a small group of people.

Changes in state are automatically applied to the system

• Creating, modifying or deleting objects are automatically applied on cluster.
• Why? To increase consistency and standardization of our pipelines.

Operator ensure correctness and alert on divergence

Cluster state is restored to latest revision automatically.

That was Four principles of GitOps

The practical guide to GitOps from Weaveworks

When can I use GitOps?

To deploy core services in your cluster

• Ingress controllers: Nginx, Envoy.
• Service Mesh with configuration: Istio, Consul Connect.
• Logging processors: Fluentbit, Filebeat.
• Cluster tools: kube-state-metrics.
• Other: HPA, Operators, Chaos monkey and much more.

To provide base configuration to all of those services

Maybe I am doing GitOps? I have CI/CD pipelines

Yes...

• They are automated
• And state is applied from Git repository

... but what about deleting the objects?

This is not easy:

• How to track previously created objects?
• What if I have to use different templating tools?

And how to regulary restore the state on cluster?

The pipeline has to be scheduled regularly.

But how to check which object is out of sync with the state?

Example: one of Deployments is misconfigured

ArgoCD

What can be configured

• Repository URL
• Reference: branch/tag/hash
• Synchronization policy
• Manual
• Automatic
• Path in repository
• Recursive - To follow directories recursively
• Namespace, where objects are applied(if namespaced)

Demo

Let's deploy first application

tree demo_1/
demo_1/
├── deployment.yaml
├── pod.yaml
└── service.yaml

0 directories, 3 files

Not only UI

spec:
  source:
    path: demo_1/
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    directory:
      recurse: true
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo1

Synchronization policies

• Manual
• Automatic
• (Optional) With resource pruning
• (Optional) With self healing

Application Panel

Refresh

Compare the latest code in Git with the live state. Figure out what is different.

Sync

The process of making an application move to its target state. E.g. by applying changes to a Kubernetes cluster.

Demo

Application is out of sync

Auto-heal is disabled

Demo

Something is out of sync

Let's figure out the differences with state

Sync options

Let's remove something!

Ok, but what about security

Required priviledges

ArgoCD needs ClusterAdmin permission

RBAC is designed to define allow rules

But this ClusterAdmin can sometimes save your ass :-)

Failure #2: EKS Authentication Configuration

In EKS there is and aws-auth ConfigMap

aws-auth is to map IAM users to K8S users

And add proper permissions

data:
  mapUsers: |
    - "groups":
      - "system:masters" # <---- THIS
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated" # <---- THIS
      "userarn": "arn:aws:iam::xxx:user/other"
      "username": "other"

But Somethimes, when you are doing code refactor

You can not notice the issue...

data:
  mapUsers: |
    - "groups":
      - "system:masters"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"

... and then you are not admin of your cluster 😎

ArgoCD supports templating tools

• Plain directory of YAML/json manifests
• helm charts
• kustomize applications
• ksonnet applications
• jsonnet files
• Custom tools(developing own plugin is required)

Be aware

It is not possible to configure dependencies between Applications

Nope

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  namespace: argocd
  name: argo-root
spec:
  source:
    path: applications/
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    directory:
      recurse: true
  destination:
    namespace: argocd
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Why it is helpful?

To reduce recovery procedure.

To have all things in repository.

Cluster recreation becomes simple!

1. Deploy K8S Cluster
2. Install ArgoCD
3. Deploy argo-root Application which manages other Application definitions
4. WIN WIN WIN

Time to real deployment

Istio in 90 seconds

Istio CRD

kind: Application
metadata:
  name: istio-crd
spec:
  destination:
    namespace: istio-system
    server: 'https://kubernetes.default.svc'
  source:
    path: istio_helm/istio-init
...

Istio

kind: Application
metadata:
  name: istio
spec:
  source:
    path: istio_helm/istio
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    helm:
      valueFiles:
        - values-istio-demo.yaml
  destination:
    namespace: istio-system
...
                   

Usful tips

• Store plain YAML's in single repository.
• Separate them by folders(by namespaces).
• Put other applications in separate repositories.

What to choose for multiple environments: folders, branches?

It is better to use branches: prod & staging etc.

• Easier to track differences.
• Easier to compare deployed versions.

Alternatives

Flux Operator

GitOps precursor - tool developed Weaveworks

Automatically updates container tags

Requires write access to git repository

Poor support for branching model

Support plain objects and Helm

Object pruning was in alpha stage

No UI😢

github.com/weaveworks/flux

But Argo and Flux are joining forces - Argo Flux

Who is adopting GitOps?