"error": "Pod \"arc-runner-set-5dmdc-runner-98fqm\" is invalid: metadata.labels: Invalid value: \"“true”\"

[tibers]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.13.1

Deployment Method

Helm

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Upgrade from 0.13.0 to 0.13.1

Describe the bug

After upgrading we originally got bit by #4356 - so I uninstalled the runners completely (the runners and the controller) and deleted their namespaces. This hung at arc-systems and I had to delete it by editing the finalizers.

Reinstalled the 0.13.1 chart and started getting:

2026-02-06T18:28:29Z    ERROR   EphemeralRunner Failed to create pod resource for ephemeral runner.     {"version": "0.13.0", "ephemeralrunner": {"name":"arc-runner-set-5dmdc-runner-98fqm","namespace":"arc-runners"}, "error": "Pod \"arc-runner-set-5dmdc-runner-98fqm\" is invalid: metadata.labels: Invalid value: \"“true”\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*EphemeralRunnerReconciler).createPod
        github.com/actions/actions-runner-controller/controllers/actions.github.com/ephemeralrunner_controller.go:687
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*EphemeralRunnerReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.github.com/ephemeralrunner_controller.go:278
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:216](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:216)
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:461](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:461)
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421)
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296)
2026-02-06T18:28:29Z    ERROR   EphemeralRunner Failed to create a pod due to unrecoverable failure     {"version": "0.13.0", "ephemeralrunner": {"name":"arc-runner-set-5dmdc-runner-98fqm","namespace":"arc-runners"}, "error": "Pod \"arc-runner-set-5dmdc-runner-98fqm\" is invalid: metadata.labels: Invalid value: \"“true”\": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*EphemeralRunnerReconciler).Reconcile
        github.com/actions/actions-runner-controller/controllers/actions.github.com/ephemeralrunner_controller.go:286
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:216](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:216)
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:461](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:461)
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:421)
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
        [sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296](mailto:sigs.k8s.io/controller-runtime@v0.22.3/pkg/internal/controller/controller.go:296)
2026-02-06T18:28:29Z    INFO    EphemeralRunner Updating ephemeral runner status to Failed      {"version": "0.13.0", "ephemeralrunner": {"name":"arc-runner-set-5dmdc-runner-98fqm","namespace":"arc-runners"}}

So then I did the same - I uninstalled the charts, deleted the namespaces, and re-installed 0.13.0 which was our "known good" version. However this did not solve the problem, it's still complaining about labels in the logs.

Not really sure where to go from here. :(

Running K8s version 1.32 on AWS's EKS.

Describe the expected behavior

Don't screw up.

Additional Context

We're using the default here except for the URLs and PATs to the enterprise.

Controller Logs

see above

Runner Pod Logs

nil

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[tibers]

Some additional notes - I eventually cleared the labels error by deleting the CRDs. When I uninstalled everything and did a kubectl get crds I saw that there were some CRDs from 2025 which hadn't been removed. I removed these CRDs. The installation replaced them with fresh copies. However installing 0.13.1 did not work. I did the same song and dance of removing the CRDs and tried 0.13.0 again, this also did not fix the problem. I see in the controller logs there's messages about the ephemeralrunnerset crd but these have current: 20 and desired: 20 (correct for our maxRunners config) but these eventually go into failed. No pods are spawned in the runner namespace.

[tibers]

Further notes: At some point the namespace and autoscalingrunnerset got "stuck" and someone did a patch on the finalizer. I also had to do the patch-the-finalizer trick for actually deleting the CRDs. I suspect that this was preventing the proper lifecycle of the CRDs. I would love to hear from others with this problem if they got painted into a similar corner.

[tibers]

Back to the label error - the true has curly (windows pretty) quotation - but where does this come from?

EDIT: Someone here edited a template in wordpad. :(

[tibers]

I installed this on a fresh cluster and it's working now.

While there's a bunch of nonsense going on, I think the CRD cleanup is something we need help with.

[bhat39a]

I installed this on a fresh cluster and it's working now.

While there's a bunch of nonsense going on, I think the CRD cleanup is something we need help with.

@tibers we are on the same page, faced similar issue while doing the version upgrade. Deleted all the CRD's and ensured everything is cleaned up but still like you said no pods have spawned up in the runners namespace. Surprisingly had to tear down the entire cluster and then it worked. Its really frustrating!

[nikola-jokic]

Hey, how did you upgrade exactly? What are the steps you took to upgrade

[tibers]

Hey, how did you upgrade exactly? What are the steps you took to upgrade

Originally helm upgrade -i ... but then ultimately helm uninstall ... and then having to delete the CRDs. It's likely a stuck finalizer. I eventually figured out that someone had done a patch (like you do when a namespace is stuck in terminating) on the CRD to remove the finalizer. While that's poor form, they're reacting to a problem that's happening, not causing it.

[nikola-jokic]

Oh, so basically, the resource was not finalized properly, that is when you need to manually patch (delete) finalizers. I'm not sure how to exactly reproduce the issue, that is why I'm asking.

[tibers]

Oh, so basically, the resource was not finalized properly, that is when you need to manually patch (delete) finalizers. I'm not sure how to exactly reproduce the issue, that is why I'm asking.

Try deleting the runner group on the GitHub Cloud side via the API and then give that a bit of time for the errors to start coming in on the runner side. Once you see errors in the logs, try uninstalling the chart and see if you're left with CRDs trying to finalize. That's my best guess.