- Published on
argocd-image-updater didn't work with Local Docker Registry
Table of Contents
Summary
There is a issue that argocd-image-updater cannot update images in Application that has multiple images of Local Docker Registry (e.g. 192.168.0.100:32000). This article is a brief note on the issue and its solution.
In conclusion, don't use registry names containing colon :
.
If you want to know more details, please let me know in the comments. Note that I am still learning kubernetes, so there may be some errors.
Background
- I run my hobby kubernetes environment on a local machine.
- I run Docker Registry locally, not ECR or GCR.
- I use tekton for CI and argocd for CD.
- I wanted to use argocd-image-updater to automate updating images in manifests used by argocd.
Environment
- kubernetes
- created by microk8s
- work on local machine
- Container Registry
- Use local registry created by microk8s (
microk8s enable registry
) - work on NodePort of kubernetes like 192.168.0.100:32000 (default configurations)
- Use local registry created by microk8s (
- argocd
- v2.5.5+fc3eaec
- argocd-image-updater
- v0.12.1
Issue
There is an issue with an Application that uses Kustomize with more than one Local Registry image, and only one image is updated.
This manifest is a configuration to automatically update the following two images to the latest version.
- 192.168.0.100:32000/api-server
- 192.168.0.100:32000/app
When CI pushes an image, argocd-image-updater is expected to automatically deploy the latest image. The argocd-image-updater logs and Events show two images are updated as expected, but the result is that only 192.168.0.100:32000/app is updated.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: myapp
namespace: argocd
annotations:
argocd-image-updater.argoproj.io/image-list: 192.168.0.100:32000/api-server,192.168.0.100:32000/app
argocd-image-updater.argoproj.io/force-update: 'true'
argocd-image-updater.argoproj.io/update-strategy: latest
argocd-image-updater.argoproj.io/write-back-method: git
argocd-image-updater.argoproj.io/git-branch: main
argocd-image-updater.argoproj.io/write-back-target: kustomization
spec:
project: myapp
source:
repoURL: git@github.com:tkat0/xxx.git
targetRevision: HEAD
path: kubernetes/myapp/development
syncPolicy:
automated:
prune: true
selfHeal: true
destination:
server: https://kubernetes.default.svc
namespace: default
Solution
You can't use registry names containing colon :
. Therefore, the following solutions are available.
- Option1: Publish Local Registry with a URI that doesn't include a port, such as 192.168.0.101.
- Used this option in this time.
- Option2: Use GCR, ECR or other registries.
For Option 1, it would be sufficient to simply set the service type of Local Registry to LoadBalancer so that it can be accessed directly with an address such as 192.168.0.101, but I used Ingress to configure the Local Docker Registry as follows.
- (required) Ingress exposes Service at
/
. - (required) nginx.ingress.kubernetes.io/proxy-body-size: "0"
- (optional) Set my domain as host (cr.cluster.tkat0.dev)
- For convenience (note that this host is configured to resolve local cluster in LAN, not public)
- A wild card DNS such as https://sslip.io/ should work fine
- (optional) Quit using microk8s registry and deploy another registry with helm
- To manage manifests with Git as much as possible
Here is the manifest of Local Registry as above. I installed twuni/docker-registry via argocd and helm, plus added Ingress. (Depending on the configuration of the helm chart, it may be possible to complete all of this only in Application resource.)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cr
namespace: ingress
annotations:
nginx.ingress.kubernetes.io/ssl-redirect: 'false'
# NOTE: 413 Request Entity Too Large:
# https://github.com/kubernetes/ingress-nginx/issues/4825#issuecomment-721911893
# https://github.com/kubernetes/ingress-nginx/blob/main/docs/user-guide/nginx-configuration/annotations.md#custom-max-body-size
# 0 means no upper bound
nginx.ingress.kubernetes.io/proxy-body-size: '0'
spec:
ingressClassName: nginx
rules:
- host: cr.cluster.tkat0.dev
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: docker-registry
port:
number: 5000
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: docker-registry
namespace: argocd
spec:
project: default
source:
chart: docker-registry
# https://github.com/twuni/docker-registry.helm
repoURL: https://helm.twun.io
targetRevision: 2.2.2
helm:
releaseName: docker-registry
parameters:
- name: service.type
value: ClusterIP
- name: garbageCollect.enabled
value: 'true'
- name: persistence.size
value: 20Gi
- name: ingress.enabled
value: 'false'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
destination:
server: https://kubernetes.default.svc
namespace: docker-registry
Cause
The cause is due to the implementation of argocd.
When comparing Kustomize image names, argocd checks the image name up to first :
as an identifier, so it incorrectly recognizes 192.168.0.100:32000/api-server and 192.168.0.100:32000/app as the same image (192.168.0.100).
Specifically, argocd-image-updater updates Application in func SetKustomizeImage. A image should be added to Application each time this method is called.
However, func (*ApplicationSourceKustomize) MergeImage and func (KustomizeImage) Match overwrote the same image rather than added.
This is because, as mentioned above, if the image name is the same up to :
, it is recognized as the same image.
In general, images such as gcr.io/project/image-1:v1.0 and gcr.io/project/image-2:v1.0 are recognized as different images, but this issue occurs in Local Registry that contains port number.
I understand that including a colon in registry URI is not common, especially in a production environment, but I could not find any documentation that says it should not be included as a specification (if anyone knows, please let me know).
At least, this documentation says including :8080 is fine.
https://docs.docker.com/engine/reference/commandline/tag/
The hostname must comply with standard DNS rules, but may not contain underscores. If a hostname is present, it may optionally be followed by a port number If a hostname is present, it may optionally be followed by a port number
:8080
.
Conclusion
This article describes the issue I encountered with argocd-image-updater not working with Local Registry.
I will organize the information and report to the community soon.
update(Jan 7): reported here https://github.com/argoproj-labs/argocd-image-updater/issues/516
Thank you for reading.