AWS Cloud Development Kit for Kubernetes called cdk8s generates Kubernetes manifest files for Kubernetes (k8s) deployments and services. Flux CD deploys new and modified manifests via scanning git repositories. You may not want to store the generated k8s manifests because the cdk8s application may generate different configurations for the same services based on the application state and environment variables. In that case, to help Flux-CD pick the desired configuration, you'll have to use a kustomization overlay. The overlays need to be created every time after the manifests are generated across all of the k8s services. All the repetitive steps above increase the complexity and brittleness of configuration management. This post describes a simple and scalable pattern for the synchronization of multiple k8s services.

High Level Architecture

In the above setup, there are three k8s services: nodezoo-npm-service, nodezoo-github-service, and nodezoo-info-service. Each service's k8s manifests are stored in an eponymous branch in a single manifest repository (nodezoo-k8s-manifests). The flux configuration repository is applied to the k8s cluster whenever a new service is added to nodezoo-k8s-manifests. When changes are detected in the nodezoo-k8s-manifests branches, their associated k8s configuration is changed automatically.

Configuration

The configuration across the various repositories is as follows.

Kubernetes Service

Each of the k8s services such as nodezoo-npm-service have the standard k8s manifests with a service (of kind Service) and deployment.

Manifest Repository

The single manifest repository has a simple kustomization file that references the manifests pushed to the branch.

resources:
  - nodezoo-npm-service.k8s.yaml

When a new k8s service is added, the kustomization file and the service manifests need to be pushed to a new branch with the same name as the service.

Flux Configuration

Flux is configured to watch the manifest repository instead of the service repository.

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: nodezoo-npm-service
  namespace: flux-system
spec:
  interval: 30s
  ref:
    branch: nodezoo-npm-service # branch to watch
  url: https://gitlab.com/n1470/nodezoo-k8s-manifests.git # repo to watch

When a new k8s service is added, the flux configuration needs to be updated with the additional manifest source.

Automation and Scalability

All the above flows can be automated with gitops. For example, when a tag is pushed to any of the k8s services, it can trigger a CI action to generate the manifests and push them to the single manifest repository (nodezoo-k8s-manifests) while the static assets are pulled from a common source such as s3.

This method is scalable because it is constant in terms of repositories needed for synchronizing with flux and the Kubernetes cluster. This setup will always have one repository that can support multiple services with branches.

Source Code

The source for all the examples are open source.

• nodezoo-k8s-manifests. The deployment files are in different branches.
• nodezoo-flux