Skip to content

Commit 3c3874d

Browse files
tjamettjamet
authored
Publish static kro manifests on release (#820)
* Publish static kro manifests on release The industry tends to have different descriptions for helm and plain k8s manifests with kustomize. We can notice it in argoCD where they maintain the [manifests](https://github.com/argoproj/argo-cd/blob/master/manifests/core-install.yaml) as well as the helm [chart](https://github.com/argoproj/argo-helm/tree/main/charts/argo-cd) (even in a different repository). When we first tried to generate static manifests using this technique, during the review, a concern was raised about the additional cognitive load and maintenance burden having 2 different deployment methods would introduce. This commit aims at providing static bundles to benefit from static manifests without the additional maintenance burden. It keeps Helm as the source of truth for deployment description and picks a list of well-known values combination to generate static manifests out of it. * Add make commands to install static manifests in kind * Install static manifests in kro-system chainsaw tests are using kro-system that is a standard practice for operator namespaces. Follow this naming convention * Add make commands to run plain manifests e2e tests * Add cluster-roles for chainsaw tests chainsaw tests are run with unrestricted kro. to tests raw manifests generation, we want to ensure that it works for least-privilege mode. in the least-privilege mode (core-install) kro does not have permission to manage individual resources and hence e2e tests fails. This commits extends Kro's privileges in end to end tests to ensure they pass. It reflects how users should use KRO when using the `rbac.mode=aggregation` in helm * Update installation docs * Address review comments * Run 1 raw manifest in CI Run the least privileged install in CI to ensure raw manifests works as expected * Fix remaining failing chainsaw cleanup * Move raw manifests e2e tests to dedicated script
1 parent cb129b5 commit 3c3874d

File tree

28 files changed

+531
-29
lines changed

28 files changed

+531
-29
lines changed

‎.github/workflows/github-release.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,14 @@ jobs:
1313
runs-on: ubuntu-latest
1414
steps:
1515
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5
16+
- name: Build static manifests
17+
env:
18+
TAG: ${{ github.ref_name}}
19+
run: |
20+
make render-static-manifests RELEASE_VERSION=${TAG}
21+
1622
- uses: softprops/action-gh-release@6da8fa9354ddfdc4aeace5fc48d7f679b5214090 # 2.4.1
1723
with:
1824
generate_release_notes: true
25+
files: |
26+
./manifests/rendered/kro-*.yaml

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,3 +44,4 @@ TODO.*
4444
node_modules/
4545

4646
tools/lsp/client/out/
47+
manifests/rendered

‎Makefile‎

Lines changed: 60 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,15 @@ LDFLAGS="-buildid= -X sigs.k8s.io/release-utils/version.gitVersion=$(GIT_VERSION
4444

4545
WITH_GOFLAGS = GOFLAGS="$(GOFLAGS)"
4646

47+
HELM_STATIC_MANIFESTS_FLAGS ?= --set metadata.includeHelmChart=false --set metadata.includeManagedBy=false --include-crds --namespace kro-system
48+
HELM_STATIC_MANIFEST_IMAGE_FLAGS ?= --set image.tag=${RELEASE_VERSION}
49+
50+
ifeq ($(shell uname -s),Darwin)
51+
SED_INPLACE_FLAGS ?= -i ''
52+
else
53+
SED_INPLACE_FLAGS ?= -i
54+
endif
55+
4756
HELM_DIR = ./helm
4857
WHAT ?= unit
4958

@@ -228,17 +237,33 @@ publish-image: ko ## Publish the kro controller images
228237
$(KO) publish --bare github.com/kubernetes-sigs/kro/cmd/controller \
229238
--tags ${RELEASE_VERSION} --sbom=none
230239

240+
.PHONY: inject-helm-version
241+
inject-helm-version:
242+
sed $(SED_INPLACE_FLAGS) 's/tag: .*/tag: "$(RELEASE_VERSION)"/' helm/values.yaml
243+
sed $(SED_INPLACE_FLAGS) 's/version: .*/version: $(RELEASE_VERSION)/' helm/Chart.yaml
244+
sed $(SED_INPLACE_FLAGS) 's/appVersion: .*/appVersion: "$(RELEASE_VERSION)"/' helm/Chart.yaml
245+
231246
.PHONY: package-helm
232-
package-helm: ## Package Helm chart
233-
sed -i 's/tag: .*/tag: "$(RELEASE_VERSION)"/' helm/values.yaml
234-
sed -i 's/version: .*/version: $(RELEASE_VERSION)/' helm/Chart.yaml
235-
sed -i 's/appVersion: .*/appVersion: "$(RELEASE_VERSION)"/' helm/Chart.yaml
247+
package-helm: inject-helm-version ## Package Helm chart
236248
${HELM} package helm
237249

238250
.PHONY: publish-helm
239251
publish-helm: ## Helm publish
240252
${HELM} push ./kro-${RELEASE_VERSION}.tgz oci://${HELM_IMAGE}
241253

254+
.PHONY: render-static-manifests
255+
render-static-manifests: inject-helm-version
256+
mkdir -p manifests/rendered
257+
@command -v yq >/dev/null 2>&1 || { echo >&2 "yq is required but not installed. Please install yq v4+"; exit 1; }
258+
@variants=$$(yq -r '.variants[].name' manifests/variants.yaml); \
259+
for v in $$variants; do \
260+
echo "Rendering variant: $$v"; \
261+
tmpfile=$$(mktemp); \
262+
yq -r ".variants[] | select(.name==\"$${v}\") | .values" manifests/variants.yaml > $$tmpfile; \
263+
${HELM} template ${HELM_STATIC_MANIFESTS_FLAGS} ${HELM_STATIC_MANIFEST_IMAGE_FLAGS} -f $$tmpfile kro ./helm > manifests/rendered/$${v}.yaml; \
264+
rm -f $$tmpfile; \
265+
done
266+
242267
.PHONY:
243268
release: build-image publish-image package-helm publish-helm
244269

@@ -256,19 +281,31 @@ install: manifests ## Install CRDs into the K8s cluster specified in ~/.kube/con
256281
uninstall: manifests kustomize ## Uninstall CRDs from the K8s cluster specified in ~/.kube/config
257282
$(KUBECTL) delete --ignore-not-found=$(ignore-not-found) -f ./helm/crds
258283

259-
.PHONY: deploy-kind
260-
deploy-kind: export KO_DOCKER_REPO=kind.local
261-
deploy-kind: ko ## Deploy kro to a kind cluster
262-
$(KIND) delete clusters ${KIND_CLUSTER_NAME} || true
284+
.PHONY: start-kind
285+
start-kind:
286+
$(KIND) delete cluster --name ${KIND_CLUSTER_NAME} || true
263287
$(KIND) create cluster --name ${KIND_CLUSTER_NAME}
264-
$(KUBECTL) --context kind-$(KIND_CLUSTER_NAME) create namespace kro-system
288+
$(KUBECTL) --context kind-${KIND_CLUSTER_NAME} create namespace kro-system
289+
290+
.PHONY: deploy-kind-helm
291+
deploy-kind-helm: export KO_DOCKER_REPO=kind.local
292+
deploy-kind-helm: ko start-kind
265293
make install
266294
# This generates deployment with ko://... used in image.
267295
# ko then intercepts it builds image, pushes to kind node, replaces the image in deployment and applies it
268296
${HELM} template kro ./helm --namespace kro-system --set image.pullPolicy=Never --set image.ko=true --set config.allowCRDDeletion=true | $(KO) apply -f -
269297
kubectl wait --for=condition=ready --timeout=1m pod -n kro-system -l app.kubernetes.io/component=controller
270298
$(KUBECTL) --context kind-${KIND_CLUSTER_NAME} get pods -A
271299

300+
.PHONY: deploy-kind-%
301+
deploy-kind-%: export KO_DOCKER_REPO=kind.local
302+
deploy-kind-%: export HELM_STATIC_MANIFEST_IMAGE_FLAGS=--set image.pullPolicy=Never --set image.ko=true
303+
deploy-kind-%: export RELEASE_VERSION=v0.0.0-dev
304+
deploy-kind-%: ko start-kind render-static-manifests ## Apply the static manifests for the given variant
305+
$(KO) apply -f manifests/rendered/kro-$*.yaml
306+
kubectl wait --for=condition=ready --timeout=1m pod -n kro-system -l app.kubernetes.io/component=controller
307+
$(KUBECTL) --context kind-${KIND_CLUSTER_NAME} get pods -A
308+
272309
.PHONY: ko-apply
273310
ko-apply: ko
274311
${HELM} template kro ./helm --namespace kro-system --set image.pullPolicy=Never --set image.ko=true | $(KO) apply -f -
@@ -286,6 +323,18 @@ cli:
286323
test-e2e: chainsaw ## Run e2e tests
287324
$(CHAINSAW) test ./test/e2e/chainsaw
288325

326+
327+
.PHONY: test-e2e-kind-%
328+
test-e2e-kind-%: deploy-kind-%
329+
make test-e2e
330+
331+
332+
# Default deployment uses helm deployments
333+
334+
.PHONY: deploy-kind
335+
deploy-kind: export KO_DOCKER_REPO=kind.local
336+
deploy-kind: ko deploy-kind-helm ## Deploy kro to a kind cluster
337+
338+
# Default end to end tests uses helm deployments
289339
.PHONY: test-e2e-kind
290-
test-e2e-kind: deploy-kind
291-
make test-e2e
340+
test-e2e-kind: deploy-kind-helm

‎helm/templates/_helpers.tpl‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,16 @@ Create the name of the service account to use
4545
Common labels
4646
*/}}
4747
{{- define "kro.labels" -}}
48+
{{- if .Values.metadata.includeHelmChart }}
4849
helm.sh/chart: {{ include "kro.chart" . }}
50+
{{- end }}
4951
{{ include "kro.selectorLabels" . }}
5052
{{- if .Chart.AppVersion }}
5153
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
5254
{{- end }}
55+
{{- if .Values.metadata.includeManagedBy }}
5356
app.kubernetes.io/managed-by: {{ .Release.Service }}
57+
{{- end }}
5458
app.kubernetes.io/component: controller
5559
app.kubernetes.io/part-of: kro
5660
{{- if .Values.additionalLabels }}

‎helm/values.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,10 @@ fullnameOverride: ""
77
additionalLabels: {}
88
# app: kro
99

10+
metadata:
11+
includeHelmChart: true
12+
includeManagedBy: true
13+
1014
image:
1115
# The location of the container image repository
1216
repository: registry.k8s.io/kro/kro

‎manifests/kustomization.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
resources:
2+
- ./bundled/kro-core-install-manifests.yaml

‎manifests/variants.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
variants:
2+
- name: kro-core-install-manifests
3+
values:
4+
rbac:
5+
mode: aggregation
6+
- name: kro-core-install-manifests-with-prometheus
7+
values:
8+
rbac:
9+
mode: aggregation
10+
metrics:
11+
service:
12+
create: true
13+
serviceMonitor:
14+
enabled: true

‎scripts/ci/presubmits/e2e-raw-manifests-tests‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
3+
set -o errexit
4+
set -o nounset
5+
set -o pipefail
6+
7+
REPO_ROOT="$(git rev-parse --show-toplevel)"
8+
cd ${REPO_ROOT}
9+
10+
# Build
11+
go build cmd
12+
13+
mkdir -p ./bin
14+
curl -Lo ./bin/kind https://kind.sigs.k8s.io/dl/v0.30.0/kind-linux-amd64
15+
chmod +x ./bin/kind
16+
17+
# Raw manifests based end to end tests, with minimal installation and "aggregated" mode
18+
make test-e2e-kind-core-install-manifests KIND=./bin/kind

‎scripts/ci/presubmits/e2e-tests‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,4 +14,5 @@ mkdir -p ./bin
1414
curl -Lo ./bin/kind https://kind.sigs.k8s.io/dl/v0.30.0/kind-linux-amd64
1515
chmod +x ./bin/kind
1616

17+
# Helm based end to end tests, with helm defaults
1718
make test-e2e-kind KIND=./bin/kind

‎test/e2e/chainsaw/check-arbitary-objects/chainsaw-test.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,9 @@ spec:
99
- name: install-rgd
1010
try:
1111
#description: Install the RGD that creates an Instance CRD
12+
- apply:
13+
file: cluster-role.yaml
14+
description: Apply Cluster Role
1215
- apply:
1316
file: rgd.yaml
1417
description: Apply RGD

0 commit comments

Comments
 (0)