kyverno updaterequest accumulating
It started with kyverno-background-controller-579cf74476-ts9s5
being in CrashLoopBackOff
.
Investigating this lead to the fact that it is oom
.
Reading this https://github.com/kyverno/kyverno/issues/9568
Checking our ur
(update request), we had:
k get ur -A | wc -l
11572
pending ur
\o/
I deleted them with:
k delete ur -l generate.kyverno.io/policy-name=grafana-datasource-local-loki
k delete ur -l generate.kyverno.io/policy-name=velero-local-minio-creds
I guess we have an RBAC issue we have to investigate to make: https://forge.liiib.re/indiehost/libre.sh/libre.sh/-/blob/develop/cluster/components/backups/velero/gen-policy.yaml?ref_type=heads and: https://forge.liiib.re/indiehost/libre.sh/libre.sh/-/blob/develop/cluster/components/observability/loki/datasource.yaml?ref_type=heads
work.
Looking at the issue, it can also be the synchronize: true
.
I don't want to investigate right now. But we have to fix it at one point.