It seems that the solution of https://github.com/spiffe/spire-controller-manager/issues/39 is missing from v0.6.0 therefore there is a restart because the spire-server socket is not ready.