GitOps and Argo

Argo CD is a controller that runs inside your cluster, watches a Git repository, and keeps the cluster's resources matching what's committed there. That's GitOps: Git holds the desired state, and an agent in the cluster pulls it in. This Argo CD tutorial takes you from an empty cluster to a working setup. You'll install Argo CD 3.5, log in with the initial admin secret, deploy the guestbook app from the public argocd-example-apps repository as an Application manifest, and then switch on automated sync with self-heal and prune so the cluster corrects itself. The last part of the chapter covers the rest of the Argo family (Workflows, Rollouts, Events) at the depth a GitOps reader needs.

How Argo CD thinks about your cluster#

Everything Argo CD does revolves around one custom resource, the Application. An Application names a source (a repository, a revision, a path inside it) and a destination (a cluster and a namespace). Argo CD renders whatever sits at that path, compares it with what's live, and reports two separate things.

• Sync status says whether the live objects match Git. OutOfSync means they don't.
• Health status says whether those objects are working. A Deployment can be perfectly in sync and still be Progressing while its pods start.

Keep those two apart in your head and the UI stops being confusing. The direction of travel matters too. The controller reaches out to Git from inside the cluster, so your CI system never needs cluster credentials, and nobody has to open the API server to the outside world for deployments.

Before you start#

You need a cluster you can throw away and kubectl pointed at it. A local cluster from the KinD Cluster chapter is ideal; K3s works as well. With kind:

kind create cluster --name argocd

One version note. The Argo CD installation docs list the Kubernetes versions each release is tested against, and for 3.5 that's 1.33 through 1.36. Kubernetes 1.37 isn't in that table yet. A 1.36 cluster keeps you inside the tested range; on 1.37 you're a little ahead of it, which is fine for learning and something to check before production.

It also helps to have read the Kubernetes Deployment chapter, because the guestbook app is just a Deployment and a Service, and you'll be poking at both.

Install Argo CD#

Argo CD lives in its own namespace, called argocd by convention. The official getting-started guide applies the manifest from the floating stable branch and recommends a pinned version for production. Pinning costs nothing, so this chapter pins v3.5.6, the newest 3.x release on the Argo CD releases page at the time of writing:

kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml

Both flags are there for a reason. Some of Argo CD's CRDs are too big for the annotation that client-side kubectl apply writes (the docs mention a 262KB limit), so server-side apply is required, and --force-conflicts lets the apply take ownership of fields another tool may already manage. Leave either off and you'll see errors on the CRDs.

Watch the pods come up:

kubectl get pods -n argocd -w

Give it a minute or two on a laptop. Once every pod reads Running, press Ctrl+C and move on.

Install the argocd CLI#

You can do everything in the web UI, but the CLI makes each step repeatable, and that's the habit GitOps is built on. On Linux or WSL, download the binary from the latest release:

curl -sSL -o argocd-linux-amd64 https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64
sudo install -m 555 argocd-linux-amd64 /usr/local/bin/argocd
rm argocd-linux-amd64

On macOS (Intel or Apple Silicon), Homebrew is simpler:

brew install argocd

Check it without needing a server:

argocd version --client

Log in with the initial admin secret#

The install doesn't expose the API server outside the cluster. For a local setup, port-forward it in a second terminal and leave that running:

kubectl port-forward svc/argocd-server -n argocd 8080:443

At install time Argo CD generated a password for the admin user and stored it in a Secret named argocd-initial-admin-secret. The CLI can read it for you through your kubeconfig (no port-forward needed for this one):

argocd admin initial-password -n argocd

Now log in. The server presents a self-signed certificate, so on this throwaway cluster add --insecure, which skips certificate verification:

argocd login localhost:8080 --username admin --insecure
argocd account update-password

That initial Secret holds the password in clear text and serves no other purpose. The docs say to delete it once you've changed the password, and Argo CD re-creates it if it ever needs to issue a new one:

kubectl -n argocd delete secret argocd-initial-admin-secret

Open https://localhost:8080 in a browser too, accept the certificate warning, and sign in as admin. Keep the tab open. It's the quickest way to see what each of the next commands does.

Deploy your first Application from Git#

The Argo project keeps a public repository of demo apps, argoproj/argocd-example-apps. Its guestbook directory contains two plain manifests: a Deployment called guestbook-ui with one replica, and a Service of the same name on port 80. That's the app you'll deploy.

You could create it with argocd app create, and the getting-started guide does. Writing it as YAML is better practice, because the Application itself then becomes something you can commit. Save this as guestbook-app.yaml:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook
  destination:
    server: https://kubernetes.default.svc
    namespace: guestbook
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Three fields trip people up. metadata.namespace must be the namespace Argo CD runs in, because Applications and AppProjects have to live there. destination.server set to https://kubernetes.default.svc means "the cluster Argo CD itself is running in". And CreateNamespace=true tells Argo CD to create the guestbook namespace on sync; without it the sync fails, since that namespace doesn't exist yet.

kubectl apply -n argocd -f guestbook-app.yaml
argocd app get guestbook

Nothing has been deployed yet. There's no sync policy, so Argo CD only reports the gap. The output looks something like this (trimmed):

Sync Policy:        <none>
Sync Status:        OutOfSync from HEAD (<commit>)
Health Status:      Missing

GROUP  KIND        NAMESPACE  NAME          STATUS     HEALTH
       Service     guestbook  guestbook-ui  OutOfSync  Missing
apps   Deployment  guestbook  guestbook-ui  OutOfSync  Missing

Run the sync:

argocd app sync guestbook
kubectl get deploy,svc -n guestbook

Run argocd app get guestbook again and both resources should read Synced, with health moving to Healthy once the pod is up. In the UI you'll see the same app as a tree: Application, then Deployment and Service, then the ReplicaSet and Pod underneath.

Drift, self-heal and prune#

Here's where GitOps earns its keep. Change the live state by hand, the way someone would during an incident:

kubectl scale --replicas=3 deployment/guestbook-ui -n guestbook
argocd app diff guestbook

argocd app diff prints the difference between Git and the cluster, here the replica count, and exits with code 1 when there is one (0 when there isn't), which makes it handy in scripts. The app is now OutOfSync. Argo CD noticed; with a manual policy it won't touch anything.

Turn on automated sync with both safety switches:

argocd app set guestbook --sync-policy automated --self-heal --auto-prune

In the YAML, the same thing is three keys under syncPolicy. Put them in guestbook-app.yaml as well, so the file stays the source of truth:

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Scale the Deployment to 3 again and watch kubectl get pods -n guestbook -w. The extra pods get terminated within seconds, because self-heal re-syncs after a short timeout (5 seconds by default) whenever the live state drifts.

The two switches cover opposite directions. Self-heal reacts to changes made in the cluster. Prune reacts to Git: when a manifest disappears from Git, Argo CD deletes the matching object. Automated sync leaves removed resources in place unless prune is on, which is a sensible default for a tool that can delete things.

To see prune for yourself you need a repository you can push to. Fork argocd-example-apps on GitHub, change repoURL to your fork, re-apply, then delete guestbook/guestbook-ui-svc.yaml in the fork and commit. Changes in Git aren't instant. Argo CD polls repositories on an interval set by timeout.reconciliation in the argocd-cm ConfigMap, 120 seconds plus up to 60 seconds of jitter by default, so allow about three minutes before the Service disappears.

The automated sync rules in the official auto-sync guide have two consequences worth knowing early. Each combination of commit and parameters is synced once, and a failed sync for that commit isn't retried. And rollback isn't possible while automated sync is on, because Git is the thing you roll back: revert the commit and let Argo CD follow.

App of apps, briefly#

Once you have more than a handful of Applications, applying each YAML by hand defeats the purpose. The app-of-apps pattern fixes that with one parent Application whose path points at a directory of Application manifests. Sync the parent and it creates the children; add a file to that directory and a new app appears.

The cluster bootstrapping guide shows it with the example repository's apps directory. Read that directory before you sync it on this cluster. It's a Helm chart that generates close to twenty child Applications, including several that need config management plugins or extra projects a fresh install doesn't have, and one of them is named guestbook, which collides with the app you already created. For practice, put two Application files in a folder of your fork and point a parent at it.

Two rules from the docs carry over to real setups. Children only take their resources with them on deletion if each child carries the resources-finalizer.argocd.argoproj.io finalizer. And creating Applications in arbitrary projects is an admin-level power, so only admins should be able to push to the parent's repository. For fleets of clusters, the same guide recommends ApplicationSets.

Common mistakes#

• Application in the wrong namespace. An Application applied to default is just an object Argo CD never looks at. It has to sit in argocd.
• Forgetting CreateNamespace=true (or creating the namespace yourself) when the destination namespace doesn't exist, which leaves the first sync failing.
• Installing without --server-side. The CRDs are too large for client-side apply.
• Deleting an Application with kubectl delete and expecting its workload to go too. Without the resources finalizer, the Deployment and Service stay. argocd app delete cascades by default.
• Fixing production with kubectl edit while self-heal is on, then wondering why the fix vanished five seconds later. Commit the fix.

Clean up#

Delete the apps first, while the controller is still running to remove their resources:

argocd app delete guestbook --yes
kubectl delete -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v3.5.6/manifests/install.yaml
kubectl delete namespace argocd

If the cluster was only for this chapter, kind delete cluster --name argocd removes the lot.

The rest of the Argo family#

Argo is four projects under one roof, and only Argo CD is about GitOps directly. The other three install their own controllers and CRDs, and each can itself be deployed by Argo CD from Git.

Argo Workflows#

A workflow engine for Kubernetes: every step runs as a container, and steps can be ordered as a list or as a DAG with dependencies. Teams use it for CI, data pipelines and batch jobs. The quick start installs a minimal, explicitly not-for-production setup from a release asset:

ARGO_WORKFLOWS_VERSION="v4.1.6"
kubectl create namespace argo
kubectl apply --server-side -n argo -f "https://github.com/argoproj/argo-workflows/releases/download/${ARGO_WORKFLOWS_VERSION}/quick-start-minimal.yaml"

With the argo CLI from the same release page, submit the hello-world example and follow it, then read its logs:

argo submit -n argo --watch https://raw.githubusercontent.com/argoproj/argo-workflows/main/examples/hello-world.yaml
argo logs -n argo @latest

The UI is at https://localhost:2746 after kubectl -n argo port-forward service/argo-server 2746:2746. Note the https; you'll get a self-signed certificate warning there too.

Argo Rollouts#

Rollouts replaces the Deployment's rolling update with canary and blue-green strategies, using a Rollout resource that looks like a Deployment plus a strategy block. The getting-started canary moves through steps such as 20% of traffic, an indefinite pause, then 40%, 60% and 80% with ten-second pauses. You install the controller and a kubectl plugin:

kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
brew install argoproj/tap/kubectl-argo-rollouts

The paused canary waits until you run kubectl argo rollouts promote rollouts-demo. That pairs well with Argo CD. Git says which image you want; Rollouts decides how fast it reaches users. The Rollouts documentation walks through the demo, and traffic-weighted canaries tie in with the mesh from the Linkerd chapter.

Argo Events#

Events is the glue that starts things. An EventSource consumes events from outside (webhooks, queues, schedules and the like), converts them to CloudEvents and puts them on the EventBus. A Sensor subscribes to the bus, waits until its dependencies are met, and fires a Trigger: an Argo Workflow, any Kubernetes object, an HTTP call. For the bus, JetStream and Kafka are the supported backends; the original NATS Streaming option is marked deprecated. On its own Events does little, so you'll usually meet it paired with Workflows, for example a Git webhook that starts a build workflow.

Next in the book#

The guestbook here was plain YAML, but Argo CD renders Helm charts and Kustomize overlays from the same source block. Point an Application at a chart from Helm Part 1 or an overlay from Kustomize and the sync, diff and self-heal steps above work unchanged.

Further watching#

• Video by Anais Urlichs (the original 100 Days of Kubernetes series), on GitOps and Argo CD
• Video by Anais Urlichs (the original 100 Days of Kubernetes series), on Argo Workflows and Argo Events