[Feat] Add Ray Cron Job #4159

machichima · 2025-10-30T14:12:53Z

Why are these changes needed?

Support cron job scheduling. Following this design docs and implement milestone 1 in this PR

Main changes:

Add RayCronJob CRD and controller
Add feature gate for enabling RayCronJob
Add unit test

Test

Apply the sample YAML ray-operator/config/samples/ray-cronjob.sample.yaml. RayJobs are being scheduled every minute:

Trigger validation error

Related issue number

Following comment: #2426 (comment)

Closes #2426

Checks

I've made sure the tests are passing.
Testing Strategy
- Unit tests
- Manual tests
- This PR is not tested :(

…-poc

ray-operator/apis/ray/v1/raycronjob_types.go

ray-operator/controllers/ray/raycronjob_controller.go

ray-operator/apis/ray/v1/raycronjob_types.go

win5923 · 2025-11-26T17:07:51Z

ray-operator/apis/ray/v1/raycronjob_types.go

+type RayCronJobSpec struct {
+	// JobTemplate defines the job spec that will be created by cron scheduling
+	JobTemplate *RayJobSpec `json:"jobTemplate"`
+	// Schedule is the cron schedule string
+	Schedule string `json:"schedule"`
+}


Not sure if this is alright, but I think we should introduce a separate struct (e.g. RayJobTemplateSpec) to hold both the metadata and the spec for the generated RayJob, similar to how Kubernetes models JobTemplateSpec in CronJob.

This would allow users to specify metadata inside jobTemplate, which we can then propagate to the created RayJob. It also keeps the API aligned with common Kubernetes patterns.

WDYT?

Reference:
https://github.com/kubernetes/kubernetes/blob/af9fb799ef09bbdb0b2b40b4e441f2ffccaffe18/pkg/apis/batch/types.go#L94-L105

Not sure if we want this. In RayJobSpec when we specify the RayClusterSpec, we only set RayClusterSpec itself rather than also letting user set the ObjectMeta.

kuberay/ray-operator/apis/ray/v1/rayjob_types.go

Line 206 in 5a836b6

RayClusterSpec *RayClusterSpec `json:"rayClusterSpec,omitempty"`

cc @rueian for confirmation.

No need for it at the moment, this can be a future work.

The current structure looks good to me already.

Signed-off-by: machichima <[email protected]>

Future-Outlier · 2025-11-29T01:24:23Z

ray-operator/config/samples/ray-cronjob.sample.yaml

+      rayVersion: '2.46.0' # should match the Ray version in the image of the containers
+      # Ray head pod template


can we use 2.52.0 version? thank you!

Updated in 6e3c37a. Thank you!

Signed-off-by: machichima <[email protected]>

win5923 · 2025-12-01T15:55:01Z

ray-operator/controllers/ray/raycronjob_controller.go

+		// This is the only 2 places where we update the RayCronJob status. This will directly
+		// update the ScheduleStatus to ValidationFailed if there's validation error
+		if err = r.updateRayCronJobStatus(ctx, originalRayCronJobInstance, rayCronJobInstance); err != nil {
+			logger.Info("Failed to update RayCronJob status", "error", err)
+			return ctrl.Result{RequeueAfter: RayCronJobDefaultRequeueDuration}, err
+		}


No need to update status for validation failed since we not use ScheduleStatus for RayCronJob.

Sorry this is the left over code, I forgot to delete it. Removed in 612fdb4

win5923 · 2025-12-01T16:33:04Z

ray-operator/controllers/ray/raycronjob_controller_unit_test.go

+	// Create RayCronJob with invalid cron schedule
+	rayCronJob := &rayv1.RayCronJob{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      "invalid-cronjob",
+			Namespace: "default",
+		},
+		Spec: rayv1.RayCronJobSpec{
+			Schedule: "invalid cron string",
+			JobTemplate: &rayv1.RayJobSpec{
+				Entrypoint: "python test.py",
+				RayClusterSpec: &rayv1.RayClusterSpec{
+					HeadGroupSpec: rayv1.HeadGroupSpec{
+						Template: corev1.PodTemplateSpec{
+							Spec: corev1.PodSpec{
+								Containers: []corev1.Container{
+									{
+										Name:  "ray-head",
+										Image: "rayproject/ray:2.9.0",
+									},
+								},
+							},
+						},
+					},
+				},
+			},
+		},
+	}


Can we create a template function for the ut? like this:

kuberay/ray-operator/controllers/ray/raycluster_controller_test.go

Line 45 in 8aa4d10

func rayClusterTemplate(name string, namespace string) *rayv1.RayCluster {

Yes of course! Modified in 9a44ff3

win5923

LGTM, Really cool!

rueian · 2025-12-01T20:28:59Z

ray-operator/controllers/ray/raycronjob_controller.go

+
+	if rayCronJobInstance.Status.LastScheduleTime == nil {
+		// The new RayCronJob, not yet scheduled
+		rayCronJobInstance.Status.LastScheduleTime = &metav1.Time{Time: now}


Why do we only update the LastScheduleTime without creating the rayjob in the first reconciliation?

The first reconciliation is just to prepare the schedule. We don't create the job yet because we need to check if the scheduled time has actually arrived (e.g., if the job runs every minute but it's currently 15:10:30, we shouldn't create it yet)

Therefore, the first reconcile just sets the current time as LastScheduleTime and sets the requeue delay to future schedule time - now."

I have the same question, is this how kuberentes job api works?

I have the same question, is this how kuberentes job api works?

Not really, Kubernetes CronJob uses the CreationTimestamp.Time to compute the next schedule time when evaluating it for the first time:

func mostRecentScheduleTime(cj *batchv1.CronJob, now time.Time, ...) { earliestTime := cj.ObjectMeta.CreationTimestamp.Time if cj.Status.LastScheduleTime != nil { earliestTime = cj.Status.LastScheduleTime.Time } // ... }

https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/controller/cronjob/utils.go#L94-L123

The CronJob also use two schedule times (t1 and t2) calculate the next time to handle missed schedules, if the current time falls between t1 and t2, the controller will create a Job.

t1 := schedule.Next(earliestTime) t2 := schedule.Next(t1) if now.Before(t1) { return earliestTime, nil, missedSchedules, nil } if now.Before(t2) { return earliestTime, &t1, missedSchedules, nil }

https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/controller/cronjob/utils.go#L115-L180

sequenceDiagram participant R as Reconciler participant M as mostRecentScheduleTime() participant K as K8s API Note over R: CronJob created at 10:05<br/>Schedule: "0 * * * *" R->>M: First reconcile (10:05) M->>M: earliestTime = CreationTimestamp (10:05) M->>M: t1 = schedule.Next(10:05) = 11:00 M->>M: now (10:05) < t1 (11:00) ? M-->>R: Return nil (not time yet) Note over R: LastScheduleTime stays nil<br/>Requeue until 11:00 R->>M: Second reconcile (11:00) M->>M: earliestTime = CreationTimestamp (10:05) M->>M: t1 = 11:00, t2 = 12:00 M->>M: now (11:00) >= t1 && now < t2 ? M-->>R: Return &t1 (11:00) ✅ R->>K: CreateJob(scheduledTime: 11:00) K-->>R: Job created ✅ R->>K: Update Status.LastScheduleTime = 11:00 Note over R: First Job created!

Loading

And the LastScheduleTime is only updated after creating a Job.

scheduledTime, err := nextScheduleTime(logger, cronJob, now, sched, jm.recorder) .... cronJob.Status.Active = append(cronJob.Status.Active, *jobRef) cronJob.Status.LastScheduleTime = &metav1.Time{Time: *scheduledTime} updateStatus = true

https://github.com/kubernetes/kubernetes/blob/ec1bf8a4f3a5f054065225dc8275c66b93310d17/pkg/controller/cronjob/cronjob_controllerv2.go#L668-L670

If i was wrong, please correct me.

Thanks @win5923 for giving more details!

@Future-Outlier It's a bit different as Kubernetes use a more complex way of handling the schedule. But basically the first reconcile will not create a job. Details are as below:

In the first reconciliation (when cronjob first created), the LastScheduleTime will be nil, which the
earliestTime will be set to the cron job's creation time (link)

Then, it will calculate the expected schedule time t1 based on the earliestTime, just like what @win5923
mentioned. Note that when the current time is "before" the expected schedule time now < t1, we will "not" create a job. Detailed code trace as below:

if now < t1, return mostRecentTime as nil (second return parameter)

code line

This mostRecentTime will be passed up to nextScheduleTime then syncCronJob -> the reconcile logic lies in this function

here, its value will be passed into scheduledTime -> scheduledTime is nil

If scheduledTime is nil, we will requeue without creating a job

code line

Future-Outlier · 2025-12-02T14:25:20Z

ray-operator/controllers/ray/raycronjob_controller.go

+
+	if rayCronJobInstance.Status.LastScheduleTime == nil {
+		// The new RayCronJob, not yet scheduled
+		rayCronJobInstance.Status.LastScheduleTime = &metav1.Time{Time: now}


I have the same question, is this how kuberentes job api works?

Future-Outlier · 2025-12-02T14:26:42Z

ray-operator/controllers/ray/raycronjob_controller.go

+func (r *RayCronJobReconciler) updateRayCronJobStatus(ctx context.Context, oldRayCronJob *rayv1.RayCronJob, newRayCronJob *rayv1.RayCronJob) error {
+	logger := ctrl.LoggerFrom(ctx)
+	oldRayCronJobStatus := oldRayCronJob.Status
+	newRayCronJobStatus := newRayCronJob.Status
+	if oldRayCronJobStatus.LastScheduleTime != newRayCronJobStatus.LastScheduleTime {
+
+		logger.Info("updateRayCronJobStatus", "old RayCronJobStatus", oldRayCronJobStatus, "new RayCronJobStatus", newRayCronJobStatus)
+		if err := r.Status().Update(ctx, newRayCronJob); err != nil {
+			return err
+		}
+	}
+	return nil
+}


Does this include the first time we initialize LastScheduleTime?

Yes. For the first time the oldRayCronJobStatus.LastScheduleTime will be nil, and we will update the status with the new LastScheduleTime set here

Signed-off-by: machichima <[email protected]>

Future-Outlier

Hi, @alimaazamat
are you interested in reviewing this PR?

Future-Outlier · 2025-12-08T09:49:03Z

ray-operator/controllers/ray/raycronjob_controller.go

+//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs,verbs=get;list;watch;create;update;patch;delete
+//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs/status,verbs=get;update;patch
+//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs/finalizers,verbs=update


Suggested change

//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs,verbs=get;list;watch;create;update;patch;delete

//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs/status,verbs=get;update;patch

//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs/finalizers,verbs=update

//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs,verbs=get;list;watch;create;update;patch;delete

//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs/status,verbs=get;update;patch

//+kubebuilder:rbac:groups=ray.io,resources=raycronjobs/finalizers,verbs=update

//+kubebuilder:rbac:groups=ray.io,resources=rayjobs,verbs=create

//+kubebuilder:rbac:groups=core,resources=events,verbs=create;patch

Fixed in b03a22d

Future-Outlier · 2025-12-08T10:04:53Z

ray-operator/controllers/ray/raycronjob_controller.go

+			Labels: map[string]string{
+				"ray.io/cronjob-name": cronJob.Name,
+			},


Suggested change

Labels: map[string]string{

"ray.io/cronjob-name": cronJob.Name,

},

Labels: map[string]string{

utils.RayCronJobLabelKey: cronJob.Name,

},

fixed in 342e5a9

Future-Outlier · 2025-12-08T10:12:47Z

ray-operator/controllers/ray/raycronjob_controller.go

+// SetupWithManager sets up the controller with the Manager.
+func (r *RayCronJobReconciler) SetupWithManager(mgr ctrl.Manager) error {
+	return ctrl.NewControllerManagedBy(mgr).
+		For(&rayv1.RayCronJob{}).
+		Complete(r)
+}


Suggested change

// SetupWithManager sets up the controller with the Manager.

func (r *RayCronJobReconciler) SetupWithManager(mgr ctrl.Manager) error {

return ctrl.NewControllerManagedBy(mgr).

For(&rayv1.RayCronJob{}).

Complete(r)

}

// SetupWithManager sets up the controller with the Manager.

func (r *RayCronJobReconciler) SetupWithManager(mgr ctrl.Manager, reconcileConcurrency int) error {

return ctrl.NewControllerManagedBy(mgr).

For(&rayv1.RayCronJob{}).

WithOptions(controller.Options{

MaxConcurrentReconciles: reconcileConcurrency,

LogConstructor: func(request *reconcile.Request) logr.Logger {

logger := ctrl.Log.WithName("controllers").WithName("RayCronJob")

if request != nil {

logger = logger.WithValues("RayCronJob", request.NamespacedName)

}

return logger

},

}).

Complete(r)

}

Oh so sorry I forgot to update this. Thank you for pointing this out!

Modified in d29b748

Future-Outlier · 2025-12-08T10:13:32Z

ray-operator/main.go

+	if features.Enabled(features.RayCronJob) {
+		setupLog.Info("RayCronJob feature gate is enabled, starting RayCronJob controller")
+		exitOnError(ray.NewRayCronJobReconciler(mgr).SetupWithManager(mgr),
+			"unable to create controller", "controller", "RayCronJob")
+	} else {
+		setupLog.Info("RayCronJob feature gate is disabled, skipping RayCronJob controller setup")
+	}


Suggested change

if features.Enabled(features.RayCronJob) {

setupLog.Info("RayCronJob feature gate is enabled, starting RayCronJob controller")

exitOnError(ray.NewRayCronJobReconciler(mgr).SetupWithManager(mgr),

"unable to create controller", "controller", "RayCronJob")

} else {

setupLog.Info("RayCronJob feature gate is disabled, skipping RayCronJob controller setup")

}

if features.Enabled(features.RayCronJob) {

setupLog.Info("RayCronJob feature gate is enabled, starting RayCronJob controller")

exitOnError(ray.NewRayCronJobReconciler(mgr).SetupWithManager(mgr, config.ReconcileConcurrency),

"unable to create controller", "controller", "RayCronJob")

} else {

setupLog.Info("RayCronJob feature gate is disabled, skipping RayCronJob controller setup")

}

updated in d29b748

Future-Outlier

overall its good, I haven't had time to review validation files, go.mod, and go.sum.

cc @jiaweijiang @seanlaii for help if you have time

seanlaii

Thanks for the contributions!
Just a reminder, for e2e tests, we need to enable feature gate here and here.

seanlaii · 2025-12-09T00:23:54Z

ray-operator/apis/ray/v1/raycronjob_types.go

+// RayCronJobSpec defines the desired state of RayCronJob
+type RayCronJobSpec struct {
+	// JobTemplate defines the job spec that will be created by cron scheduling
+	JobTemplate *RayJobSpec `json:"jobTemplate"`


Is there a plan to make it optional, such as referring to an existing job? If it will always be required, should we use a pointer type?

I think this will always be required. Let me change it to normal object. Thank you!

Changed in c3f3092

seanlaii · 2025-12-09T00:32:29Z

ray-operator/controllers/ray/raycronjob_controller.go

+			return ctrl.Result{RequeueAfter: nextScheduleTime.Sub(now)}, nil
+		}
+
+		rayJob, err := r.constructRayJob(rayCronJobInstance)


Maybe we could also add something similar to cronjob-scheduled-timestamp to the RayJob annotation for debugging.
Don't have to be in this PR if this is not trivial. Overall LGTM.

Added in bb353fb

Signed-off-by: machichima <[email protected]>

Future-Outlier

LGTM, really nice PR, cc @rueian to do final pass and merge this.

machichima added 17 commits October 30, 2025 22:10

feat: init RayCronJob controller and add CRs

384c0e1

feat: update status

027ccaf

feat: add validate

e99572d

feat: add update status function

3b4a2cb

refactor: udpate LastScheduleTime type

42e6ec2

feat: implement logic for diff ScheduleStatus

f3c5faa

build: regen CRD

e7c36c1

feat: correctly create reconciler and add loggings

93916ec

feat: check if it's time for schedule new rayjob

c2fe872

feat: add raycronjob example yaml

9cc5ecc

Merge branch 'master' of github.com:ray-project/kuberay into cron-job…

11d3cd5

…-poc

fix: remove StatusScheduled

b104f61

build: make sync

eade1ee

refactor: move validate to validation.go

8568e66

test: add validation and raycronjob unit test

ff272e0

feat: add feature gate

aa91293

test: update test

4c053d0

machichima changed the title ~~[POC] Add Ray Cron Job~~ [Feat] Add Ray Cron Job Nov 24, 2025

fix: update helm chart rules

3f04c07

machichima marked this pull request as ready for review November 24, 2025 13:03

machichima requested review from MortalHappiness, andrewsykim, kevin85421 and rueian as code owners November 24, 2025 13:03

CheyuWu self-requested a review November 24, 2025 17:01

win5923 reviewed Nov 24, 2025

View reviewed changes

ray-operator/apis/ray/v1/raycronjob_types.go Show resolved Hide resolved

win5923 reviewed Nov 24, 2025

View reviewed changes

ray-operator/controllers/ray/raycronjob_controller.go Outdated Show resolved Hide resolved

fix: add OwnerReference from cronjob to rayjob

584abd3

win5923 reviewed Nov 26, 2025

View reviewed changes

ray-operator/apis/ray/v1/raycronjob_types.go Outdated Show resolved Hide resolved

win5923 reviewed Nov 26, 2025

View reviewed changes

ray-operator/apis/ray/v1/raycronjob_types.go Outdated Show resolved Hide resolved

win5923 reviewed Nov 26, 2025

View reviewed changes

machichima added 3 commits November 28, 2025 20:45

fix: field order for RayCronJob struct

c8c93ac

Signed-off-by: machichima <[email protected]>

fix: remove schedule status

970c3ab

Signed-off-by: machichima <[email protected]>

build: make generate

a38d13a

Signed-off-by: machichima <[email protected]>

Future-Outlier reviewed Nov 29, 2025

View reviewed changes

fix: update example yaml to use ray 2.52.0 image

6e3c37a

Signed-off-by: machichima <[email protected]>

win5923 reviewed Dec 1, 2025

View reviewed changes

win5923 approved these changes Dec 1, 2025

View reviewed changes

rueian reviewed Dec 1, 2025

View reviewed changes

Future-Outlier reviewed Dec 2, 2025

View reviewed changes

machichima added 2 commits December 3, 2025 20:10

fix: no need to update status for validate fail

612fdb4

Signed-off-by: machichima <[email protected]>

test: use rayCronJobTemplate func to create rayCronJob in test

9a44ff3

Signed-off-by: machichima <[email protected]>

Future-Outlier reviewed Dec 6, 2025

View reviewed changes

Future-Outlier reviewed Dec 8, 2025

View reviewed changes

seanlaii reviewed Dec 9, 2025

View reviewed changes

machichima added 5 commits December 9, 2025 19:22

build: add kubebuilder rbac config

b03a22d

Signed-off-by: machichima <[email protected]>

feat: extract ray cron job name to constant

342e5a9

Signed-off-by: machichima <[email protected]>

fix: update SetupWithManager

d29b748

Signed-off-by: machichima <[email protected]>

fix: JobTemplate to normal object

c3f3092

Signed-off-by: machichima <[email protected]>

feat: add cron job origin expected timestamp annotation

bb353fb

Signed-off-by: machichima <[email protected]>

Future-Outlier approved these changes Dec 10, 2025

View reviewed changes

alimaazamat assigned alimaazamat and unassigned alimaazamat Dec 10, 2025

		rayVersion: '2.46.0' # should match the Ray version in the image of the containers
		# Ray head pod template

[Feat] Add Ray Cron Job #4159

Are you sure you want to change the base?

[Feat] Add Ray Cron Job #4159

Conversation

machichima commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Test

Related issue number

Checks

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

win5923 Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

win5923 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

machichima Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Future-Outlier left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

machichima commented Oct 30, 2025 •

edited

Loading

win5923 Nov 26, 2025 •

edited

Loading

win5923 Dec 1, 2025 •

edited

Loading

win5923 Dec 1, 2025 •

edited

Loading

machichima Dec 3, 2025 •

edited

Loading

machichima Dec 3, 2025 •

edited

Loading

seanlaii Dec 9, 2025 •

edited

Loading

seanlaii Dec 9, 2025 •

edited

Loading