Chapter 03 · Dossier 006
Two delivery pipelines: applications and platform
A GitOps estate can run everything through one reconciler and still keep two separate delivery paths: an application pipeline that ships workloads, and a platform pipeline that ships the shared controls and services those workloads depend on. This chapter breaks the pattern into layers, shows where each boundary is enforced, and labels which parts are checked facts from my homelab and which are illustrative design.
- Pipelines
- 2, one engine
- Layers reviewed
- 8
- Scope
- lab + illustrative
- Personal lab case study
- This chapter describes a personal homelab and a GitOps design pattern. It does not describe an employer, customer, or production environment.
- Lab observed
- Reproduced against the running lab, and published with the date it was checked.
- Lab configuration
- Found in the lab's inspected configuration. Not runtime-tested for this chapter, so it describes what the configuration says rather than a measured result.
- Lab lesson
- A sanitised historical event from this lab. The shape and the fix are published; identifiers and timings are not.
- Illustrative pattern
- Design guidance, including possible multi-team implementations. Not deployed here, and not a universal standard.
- Open
- Not yet checked, and therefore not asserted. Unfinished remediation, credentials, host inventory, internal service names and security-sensitive operational detail are not published here.
A deployment that was healthy and unreachable
LAB LESSONA small read-only web application onboarded onto the lab. Its manifests were sound: a hardened pod, its own namespace, a network policy, a route on the gateway. The reconciler deployed it cleanly. The route was accepted, the endpoints were ready, the pod answered its own health checks. Every request from outside timed out. Identifiers and timings are omitted.
The application resources were healthy; the missing control was the platform-side ingress allowance. The gateway in this lab runs behind a default-deny network policy with an explicit allow-list of backend namespaces, and the new namespace was not yet on that list. The allow-list is not part of the application's manifests; it is delivered by the platform pipeline. The application pipeline had done its whole job, and the deployment still could not serve, because onboarding a service takes both pipelines: the application's manifests, and a reviewed platform change that agrees to carry it.
That gap is the subject of this chapter.
The model
One reconciler - Argo CD in this lab - watches git and applies it to the clusters. What flows through it travels in two lanes with different permissions and different reviewers.
The application pipeline delivers what runs: product repositories holding a site, a tool, or a service. The unit it ships is a service, or a small stack of applications that belong together, each landing in its own namespace with its own network policy, its own secret scope, and its own service accounts.
The platform pipeline (the infrastructure pipeline; "platform pipeline" from here on) delivers what everything runs on: network policy and ingress, storage classes and provisioners, the secrets machinery, certificates, and the reconciler's own baseline configuration. Applications consume these as services; they do not ship them.
| Layer | Application pipeline | Platform pipeline |
|---|---|---|
| Git | Product repositories, one per service | Platform repositories - baselines, policy, storage, tooling |
| Reconciler | Applications constrained by project allow-lists and destination policy | The project definitions, cluster credentials and Application-management permissions themselves |
| Secrets | Read access to the service's own leased scope | The secret system's wiring and the identities it trusts |
| Registry | Pull images by digest | Push and promote images; manage registry credentials |
| Runtime identity | Least-practical-privilege service accounts; no API token where none is needed | Controller identities with scoped, named roles |
| Network | A request for named flows | The policies themselves and the ingress allow-lists |
| Nodes | No access | Host configuration and node lifecycle |
Where each boundary is enforced
A commit becomes running software by passing a chain of controls. Each layer reduces a different failure mode: repository write access, deployment destination, runtime identity, secret scope, registry permission, storage retention, and network reachability. None of them is automatic; each is a configuration that has to be made and reviewed.
Git. Application and platform repositories are separate, with separate write access. The reconciler holds read-only deploy keys: it can watch every repository and write to none of them. Write access to a platform repository is what allows someone to change policy, storage or baseline configuration, so it is held more narrowly than application write access - and repository write access must be limited to the people allowed to affect the project that deploys from it.
The reconciler. Argo CD can enforce source-repository, destination, and resource boundaries through AppProjects and RBAC. Those boundaries are only as strong as the project allow-lists, the Application-management permissions, repository write access, and the controller's target-cluster credentials. A project that can deploy into the Argo CD namespace is administratively sensitive, because it can alter the reconciler itself. The authority available during reconciliation depends on the target-cluster credentials, Kubernetes RBAC and Argo CD configuration. Review that authority separately from AppProject source, destination and resource restrictions. Recent Argo CD releases additionally support application-sync impersonation, which performs a sync under a nominated service account instead of the controller's own identity; it is an available configuration option, and this chapter does not report it as deployed in this lab.
Secrets. A cluster-connected secret system is part of the threat model. The pattern this lab uses: each cluster authenticates to the secret manager with its own machine identity, limited to the secret scope it requires, and the secrets controller delivers each application only its own lease. Limit each machine identity to the required scope, bind it to the intended Kubernetes identity where the secret system supports it, and avoid treating in-cluster access as equivalent to unrestricted vault access. Bootstrap is its own problem. The secret controller needs an authentication path whose initial trust does not depend solely on successfully retrieving its own credentials. This lab provisions that trust manually; other environments may use Kubernetes authentication, workload identity, OIDC, external KMS/HSM integration or a managed secret service. Placing that first credential by hand, once per cluster, is this lab's configuration rather than a recommendation for every environment, and the bootstrap path is recorded rather than left implicit.
The registry. Image pull and image push should use separate identities, and deployments should reference images by digest. In this homelab the registry-routing and node-credential behaviour is documented separately in The Mirror; it is not assumed for every Kubernetes or OpenShift platform - image-pull identity and credential distribution are runtime and platform specific.
Runtime identity. Workloads that do not need Kubernetes API access should set
automountServiceAccountToken: false. Workloads that do need API access should use a
dedicated service account with the smallest practical role. Token automounting defaults to
enabled, so the restrictive setting has to be written into the pod spec or the service account
and reviewed in the platform pipeline. The same applies to pod security settings - run as an
ordinary user, drop what is not needed, and let the namespace's admission profile refuse
regressions.
Storage. Application manifests commonly declare PersistentVolumeClaims, while the platform commonly provides StorageClasses and provisioners. Data retention is a separate, explicit contract: a PVC's deletion can lead to storage deletion depending on the PV reclaim policy, the StorageClass, the controller involved, StatefulSet PVC retention settings, and GitOps pruning configuration. Inspect all of those before deleting an application that owns data.
LAB LESSONAn application stack was removed while automated pruning was enabled, and its volume claims were pruned with it. Backups made the data recoverable, and the gap between the last snapshot and the deletion was still lost. The durable fix was procedural and mechanical at once: pruning behaviour is now reviewed for any PVC-bearing application, and a migration is not complete until the data is verified present at the destination. Identifiers and timings are omitted.
Network. Workload namespaces in this lab use default-deny NetworkPolicy with explicit allowances - enforcement depends on a network plugin that implements NetworkPolicy, so the policy is only as real as the CNI beneath it. Platform namespaces require their own reviewed policy model so critical DNS, CNI, ingress, storage and control-plane traffic remains functional; "exempt" is not the right word for them so much as "differently policied".
LAB CONFIGURATIONNodes. The lab uses Talos Linux, an API-managed node OS without SSH; host access is by certificate to its management API. That is specific to Talos - other Kubernetes and OpenShift environments use different host-management models. In each of them node access belongs to the platform pipeline, held apart from application credentials and granted as narrowly as the host model allows.
Onboarding: what the split is for
A separation that only says no is friction. This one earns its keep when a new service arrives.
The service brings an application-pipeline package: a repository, images, manifests, and a statement of what it needs. The platform pipeline answers with a reviewed onboarding change: a namespace, a network policy naming the service's flows, a secret scope, registry pull access, a storage class its claims may bind to, and the ingress allow-list entry that lets the gateway carry it. That last entry is the control missing in the opening lesson.
Two properties fall out. The service's author never needed platform credentials, and the platform never needed to read the service's code beyond its deployable surface. And offboarding becomes reviewable in the same terms: revoke the namespace, the flows, the scope and the registry access, and treat the data's fate as its own explicit decision under the storage contract above.
The bootstrap loops from chapter 02 belong to this picture too. The reconciler is itself an application in git, reconciled by itself, which is convenient right up until it is broken - so the recovery path for the reconciler must not depend on the reconciler. The secret controller's initial trust has the same shape: it cannot rest solely on the secret system it authenticates to. Both loops are the onboarding contract applied to the platform's own components, and both need the same written-down, reviewed answer as any tenant.
An illustrative multi-team translation
This is one possible extension of the model, not a description of a workplace or a universal production architecture.
| Concern | Possible multi-team control | Decision depends on |
|---|---|---|
| Git separation | Protected repositories and independent review | Ownership and change risk |
| Cluster access | Scoped platform credentials or impersonation | Argo CD and RBAC design |
| Secrets | Separate identities and audited access | Secret-system capabilities |
| Break-glass | Controlled emergency process | Recovery requirements |
| Storage deletion | Retention and restore verification | Data classification and RPO |
In the homelab, one operator performs both roles while repository and permission boundaries remain separate.
Two tests worth running
The close of this chapter is a pair of checks, not a promise.
First, the hostile test. A compromised application repository must be constrained by the controls that are configured: repository write access, the AppProject's source and destination allow-lists, Kubernetes RBAC, admission policy, network policy, and runtime identity. The desired outcome is that the blast radius ends at the application's own cell - and that is an outcome to validate with permissions review and negative tests, not to assume from the diagram.
Second, the friendly test. A new service should be onboardable with two artefacts: its application manifests, and a reviewed platform onboarding change covering the services it depends on. If application maintainers require platform credentials to onboard a service, the intended separation has failed.
A useful boundary supports safe onboarding as well as restriction.
References
The sources behind the design decisions in this chapter.
- Argo CD Projects - source, destination and resource boundaries per project.
- Argo CD declarative setup - how the reconciler itself is configured and why that surface is sensitive.
- Argo CD automated sync and pruning - what prune does and when it runs.
- Argo CD sync options - per-resource and per-application overrides.
- Argo CD application sync using impersonation - syncing under a nominated service account instead of the controller's own identity.
- Kubernetes service accounts - configuring pod identity and token automounting.
- Kubernetes RBAC good practices - least privilege for workload and controller identities.
- Kubernetes persistent volumes - the PVC/PV/StorageClass relationship.
- Changing PV reclaim policy - the setting that governs deletion behaviour.
- Kubernetes network policies - the API and what enforcement requires.
- Talos Linux FAQ - the API-managed, no-SSH host model this lab runs.
- The Mirror (chapter 02 of this series) - the supply-chain half of the story: registry routing, digest preservation and node credentials in this lab. This page is which pipeline may carry them.