Command line

kubectl create/apply/edit/delete ...
kubectl create/apply/edit/delete -f some.yaml
kubectl create/apply/edit/delete ...
kubectl create/apply/edit/delete -f some.yaml

This is Manual

• Human as a source of truth
• Possible situation, when two developers are applying the same object - communication needed
• Hard to figure out diffs

Failure story #1: YAML Overriding

Other problems

• Assume now K8S as objects store(like records in database). And we store object definitions in Git.

  So when we delete file from Git... not everyone remembers to delete the object.

• Hard to compare changes before apply.
• Even when we try to commit changes before apply, often we forget to push the code.

Yes

But first let's determine our needs.

The entire system is described declaratively

• Configuration is guaranteed by a set of facts instead of by a set of instructions.
• Cluster can be easily reproduced(e.g. business recovery tests; site clone).
• Reduced meantime to recovery(MTTR).

System state is versioned in Git

Changelog and rollbacks are easy

• git log
• git revert

Access to the repository may be restricted to a small group of people.

Changes in state are automatically applied to the system

• Creating, modifying or deleting objects are automatically applied on cluster.
• Why? To increase consistency and standardization of our pipelines.

Operator ensure correctness and alert on divergence

Cluster state is restored to latest revision automatically.

To deploy core services in your cluster

• Ingress controllers: Nginx, Envoy.
• Service Mesh with configuration: Istio, Consul Connect.
• Logging processors: Fluentbit, Filebeat.
• Cluster tools: kube-state-metrics.
• Other: HPA, Operators, Chaos monkey and much more.

To provide base configuration to all of those services

Yes...

• They are automated
• And state is applied from Git repository

... but what about deleting the objects?

This is not easy:

• How to track previously created objects?
• What if I have to use different templating tools?

And how to regulary restore the state on cluster?

The pipeline has to be scheduled regularly.

But how to check which object is out of sync with the state?

Example: one of Deployments is misconfigured

What can be configured

• Repository URL
• Reference: branch/tag/hash
• Synchronization policy
• Manual
• Automatic
• Path in repository
• Recursive - To follow directories recursively
• Namespace, where objects are applied(if namespaced)

tree demo_1/
demo_1/
├── deployment.yaml
├── pod.yaml
└── service.yaml

0 directories, 3 files

Not only UI

spec:
  source:
    path: demo_1/
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    directory:
      recurse: true
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo1
spec:
  source:
    path: demo_1/
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    directory:
      recurse: true
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo1
spec:
  source:
    path: demo_1/
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    directory:
      recurse: true
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: demo1

Application Panel

Refresh

Compare the latest code in Git with the live state. Figure out what is different.

Sync

The process of making an application move to its target state. E.g. by applying changes to a Kubernetes cluster.

Demo

Something is out of sync

Let's figure out the differences with state

Sync options

Required priviledges

ArgoCD needs ClusterAdmin permission

RBAC is designed to define allow rules

But this ClusterAdmin can sometimes save your ass :-)

Failure #2: EKS Authentication Configuration

In EKS there is and aws-auth ConfigMap

aws-auth is to map IAM users to K8S users

And add proper permissions

data:
  mapUsers: |
    - "groups":
      - "system:masters" # <---- THIS
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated" # <---- THIS
      "userarn": "arn:aws:iam::xxx:user/other"
      "username": "other"
data:
  mapUsers: |
    - "groups":
      - "system:masters" # <---- THIS
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated" # <---- THIS
      "userarn": "arn:aws:iam::xxx:user/other"
      "username": "other"

But Somethimes, when you are doing code refactor

You can not notice the issue...

data:
  mapUsers: |
    - "groups":
      - "system:masters"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
data:
  mapUsers: |
    - "groups":
      - "system:masters"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
data:
  mapUsers: |
    - "groups":
      - "system:masters"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"
    - "groups":
      - "system:authenticated"
      "userarn": "arn:aws:iam::xxx:user/marcin"
      "username": "marcin"

... and then you are not admin of your cluster 😎

Be aware

It is not possible to configure dependencies between Applications

Nope

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  namespace: argocd
  name: argo-root
spec:
  source:
    path: applications/
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    directory:
      recurse: true
  destination:
    namespace: argocd
    server: 'https://kubernetes.default.svc'
  project: default
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Why it is helpful?

To reduce recovery procedure.

To have all things in repository.

Cluster recreation becomes simple!

1. Deploy K8S Cluster
2. Install ArgoCD
3. Deploy argo-root Application which manages other Application definitions
4. WIN WIN WIN

Istio CRD

kind: Application
metadata:
  name: istio-crd
spec:
  destination:
    namespace: istio-system
    server: 'https://kubernetes.default.svc'
  source:
    path: istio_helm/istio-init
...

Istio

kind: Application
metadata:
  name: istio
spec:
  source:
    path: istio_helm/istio
    repoURL: 'https://gitlab.com/mjasion/argocd-example.git'
    targetRevision: master
    helm:
      valueFiles:
        - values-istio-demo.yaml
  destination:
    namespace: istio-system
...

What to choose for multiple environments: folders, branches?

It is better to use branches: prod & staging etc.

• Easier to track differences.
• Easier to compare deployed versions.

Flux Operator

GitOps precursor - tool developed Weaveworks

Automatically updates container tags

Requires write access to git repository

Poor support for branching model

Support plain objects and Helm

Object pruning was in alpha stage

No UI😢

github.com/weaveworks/flux

But Argo and Flux are joining forces - Argo Flux