Prevent phantom autoscaler history entries on failed replica updates

[WHOIM1205]

Fix autoscaler history corruption on failed replica updates

Summary

This PR fixes a correctness bug in the Kthena autoscaler where scaling decisions were recorded in stabilization and rate-limiting history before replica updates were successfully applied to the Kubernetes API.

When the API update failed (most commonly due to conflicts with concurrent HPA/KEDA updates), the autoscaler history was left in a corrupted state, causing phantom scale events to influence future scaling decisions.

This resulted in silent and non-deterministic autoscaling behavior.

---

What was wrong

The autoscaler updated its internal history inside Scale() / Optimize(), prior to attempting the API update:

• If the API update failed:
  • History was not rolled back
  • Stabilization windows contained values that were never applied
  • Rate-limiting budgets were artificially consumed
  • Scale-down could become permanently blocked

This issue is especially visible when Kthena runs alongside HPA or KEDA, where update conflicts are expected.

---

What this PR fixes

1. Separates computation from side effects

   • Scale() and Optimize() now return results without mutating history
   • History is committed explicitly only after successful API updates

2. Prevents phantom scale events

   • Autoscaler history now reflects applied state, not attempted state

3. Ensures atomic behavior for heterogeneous scaling

   • History is committed only after all target updates succeed

4. Fixes error masking

   • updateTargetReplicas() now returns real API errors instead of misleading
     "not supported" messages

---

Code changes

Changes are scoped and minimal, focused on correctness:

• pkg/autoscaler/autoscaler/

  • scaler.go
    • Introduced ScaleResult
    • Removed implicit history mutation
    • Added CommitScaleResult() for explicit history commits
  • optimizer.go
    • Introduced OptimizeResult
    • Same commit-after-success pattern for heterogeneous scaling

• pkg/autoscaler/controller/

  • autoscale_controller.go
    • History committed only after successful replica updates
    • Fixed API error propagation in updateTargetReplicas()

---

Impact

• Eliminates silent autoscaling failures caused by phantom history
• Restores deterministic stabilization and rate-limiting behavior
• Improves correctness when running alongside HPA/KEDA
• Makes production debugging significantly easier by surfacing real API errors

No API changes and no behavior change on successful updates — only correctness improvements on failure paths.

---

Test verification

Added comprehensive table-driven tests covering both success and failure paths:

• TestDoScale_HistoryCommitBehavior
  • Verifies history is committed only after successful scale updates
• TestDoScale_ErrorPropagation
  • Verifies real API errors are returned and not masked
• TestDoOptimize_HistoryCommitOnlyAfterAllUpdatesSucceed
  • Verifies heterogeneous scaling commits history only after all updates succeed
• Defensive tests for commit helpers (ScaleResult, OptimizeResult)
• all the test cases are passed locally

Run only the new tests:

go test -v ./pkg/autoscaler/controller/... -run "TestDoScale_HistoryCommitBehavior|TestDoScale_ErrorPropagation|TestDoOptimize_HistoryCommitOnlyAfterAllUpdatesSucceed"

[volcano-sh-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign yaozengzeng for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[gemini-code-assist]

Summary of Changes

Hello @WHOIM1205, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical correctness bug in the Kthena autoscaler that led to corrupted scaling history when replica updates to the Kubernetes API failed. By refactoring the autoscaler to separate scaling computation from history commitment, the system now ensures that history accurately reflects only successfully applied state. This change prevents 'phantom scale events,' restores deterministic stabilization and rate-limiting, and significantly improves the autoscaler's reliability, particularly when operating alongside other scaling mechanisms like HPA or KEDA. Additionally, error reporting for API interactions has been made more precise, aiding in debugging.

Highlights

• Corrected Autoscaler History: Autoscaler history (stabilization and rate-limiting) is now only committed after successful replica updates to the Kubernetes API, preventing corruption from failed updates.
• Separation of Concerns: The Scale() and Optimize() functions now return computation results (ScaleResult, OptimizeResult) without mutating internal history, deferring history commitment to explicit CommitScaleResult() and CommitOptimizeResult() calls.
• Improved Error Propagation: The updateTargetReplicas() function now returns specific API errors for failed ModelServing operations, eliminating misleading 'not supported' messages and improving debuggability.
• Enhanced Test Coverage: New comprehensive table-driven tests have been added to verify the correct history commit behavior for both homogeneous and heterogeneous scaling, as well as accurate error propagation.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This is a high-quality pull request that addresses a critical correctness bug in the autoscaler. The separation of concerns, by moving history updates out of the Scale and Optimize functions and into explicit Commit functions, is a great architectural improvement. This change effectively prevents phantom history entries when API updates fail. The improved error propagation with more context is also a welcome change that will make debugging much easier. The new comprehensive test suite is excellent and provides strong confidence in the fix. I have a few suggestions to further improve the code, mainly around preventing potential integer overflows and improving code clarity.

[FAUST-BENCHOU]

Here's my advice.we dont need to add more struct to store history or else.we can simply change the place of AppendRecommendation and AppendCorrected

for example

kthena/pkg/autoscaler/autoscaler/optimizer.go

Lines 203 to 204 in a8c9193

	optimizer.Status.AppendRecommendation(recommendedInstances)
	optimizer.Status.AppendCorrected(recommendedInstances)

kthena/pkg/autoscaler/autoscaler/scaler.go

Lines 104 to 105 in a8c9193

	autoscaler.Status.AppendRecommendation(recommendedInstances)
	autoscaler.Status.AppendCorrected(correctedInstances)

can be moved to doOptimize and doScale in autoscale_controller
to make sure the controller call Status.AppendRecommendation/AppendCorrected only after all API updates succeed.However, that means we need to also return recommendedInstances in func Optimize and Scale

[WHOIM1205]

Here's my advice.we dont need to add more struct to store history or else.we can simply change the place of AppendRecommendation and AppendCorrected

for example

kthena/pkg/autoscaler/autoscaler/optimizer.go

Lines 203 to 204 in a8c9193

	optimizer.Status.AppendRecommendation(recommendedInstances)
	optimizer.Status.AppendCorrected(recommendedInstances)

kthena/pkg/autoscaler/autoscaler/scaler.go

Lines 104 to 105 in a8c9193

	autoscaler.Status.AppendRecommendation(recommendedInstances)
	autoscaler.Status.AppendCorrected(correctedInstances)

can be moved to doOptimize and doScale in autoscale_controller to make sure the controller call Status.AppendRecommendation/AppendCorrected only after all API updates succeed.However, that means we need to also return recommendedInstances in func Optimize and Scale

thanks for the suggestion
the main goal here was just to make sure we only write to the autoscaler history after the scale actually succeeds splitting the calculation and the history update made that a bit clearer and easier to test across doscale and dooptimize
functionally it’s the same idea as moving the append* calls into the controller just with the success boundary being explicit
happy to tweak this if maintainers prefer a simpler pattern

[WHOIM1205]

/assign @YaoZengzeng

[WHOIM1205]

/assign @hzxuzhonghu

[WHOIM1205]

@hzxuzhonghu is there anything i can change