Automatic Retry of Failed state based on retry count

[nagadeesh-nagaraja]

Summary

Introduce a feature to automatically retry failed CRD #X times.

This automatic retry will stop when this counter runs down to 0.

This is re initiated when the Spec of the CRD are updated

Basic example

Spec: 
  FailedAutoRetryCount: 2

Status:
  AutoRetryCountRemaining: 2

when the CRD goes to fail state

Spec: 
  FailedAutoRetryCount: 2

Status:
  AutoRetryCountRemaining: 1

at next failure

Spec: 
  FailedAutoRetryCount: 2

Status:
  AutoRetryCountRemaining: 0

it stops retry at this point.

when Spec of this CRD is updated

Spec: 
  FailedAutoRetryCount: 2

Status:
  AutoRetryCountRemaining: 2

Motivation

to reduce the manual effort needed to deal with flaky errors