feat: ai rate limiting redis support #12751

beardnick · 2025-11-16T07:23:15Z

Description

Which issue(s) this PR fixes:

Fixes #12482

Notice

I have updated the limit-count-redis.lua and limit-count-redis-cluster.lua files to ensure they now support rate-limiting during the log phase.
I referred to the limit-conn-redis.lua file as a guide while implementing rate-limiting in the log phase.
To ensure a clean testing environment, I added the require("lib.test_redis").flush_all() function to every Redis rate-limiting test case. Without this addition, the tests could fail unpredictably.
I have adjusted the expected results in limit-req-redis.t and limit-req-redis-cluster.t test cases. After thorough verification, the correct behavior is [200, 403, 403, 403]. Previously, the results appeared as [403, 403, 403, 403] due to Redis not being properly cleaned beforehand.

Checklist

I have explained the need for this PR and the problem it solves
I have explained the changes or the new features added to this PR
I have added tests corresponding to this change
I have updated the documentation to reflect this change
I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

beardnick · 2025-11-16T07:25:54Z

@Baoyuantop PTAL

membphis · 2025-11-17T02:40:54Z

@beardnick many thx for your contribution, some CI tests are failed, you can fix them

apisix/plugins/limit-count/util.lua

apisix/plugins/limit-count/init.lua

nic-6443 · 2025-11-21T01:37:47Z

apisix/plugins/limit-count/limit-count-redis-cluster.lua


-    local remaining = res[1]
-    ttl = res[2]
+local function log_phase_incoming_thread(premature, self, key, cost)


This code should be placed in the ai-rate-limiting plugin, not in the limit-count plugin itself, because this code is only useful for ai-rate-limiting.

I disagree. Similar logic also appears in the limit-conn.

apisix/apisix/plugins/limit-conn/limit-conn-redis.lua

Lines 60 to 81 in 7e907a5

local function leaving_thread(premature, self, key, req_latency)

local conf = self.conf

local red, err = redis.new(conf)

if not red then

return red, err

end

return util.leaving(self, red, key, req_latency)

end

function _M.leaving(self, key, req_latency)

-- log_by_lua can't use cosocket

local ok, err = ngx_timer_at(0, leaving_thread, self, key, req_latency)

if not ok then

core.log.error("failed to create timer: ", err)

return nil, err

end

return ok

end

If I put this logic into ai-rate-limiting plugin, it will make the plugin too complicated. I have to copy a lot of logic from limit-count. Currently, the ai-rate-limiting is just a simple wrapper of limit-count.

Or, do you think I need to rewrite the ai-rate-limiting to something like limit-ai-redis.lua and limit-ai-redis-cluster.lua?

This PR is quite large and hard to review. If you agree, I can split it into 3 separate PRs:

A PR to improve the tests for the rate-limiting plugins.

A PR to add support for the log phase.

A PR to add Redis support to the ai-rate-limiting.

apisix/plugins/limit-count/util.lua

t/plugin/limit-conn-redis-cluster.t

nic-6443

Look like the changes in this PR should be limited to the ai-rate-limiting plugin and minor tweaks to the limit-count plugin. The current PR modifies too much code and test cases for other rate limiting plugins, expanding its scope. We should minimize unnecessary code changes.

apisix/plugins/limit-count/limit-count-redis-cluster.lua

AlinsRan · 2025-11-25T03:59:16Z

t/plugin/ai-rate-limiting.t

+
+
+
+=== TEST 21: set route with Redis policy


It is recommended to use multi worker testing for Redis related tasks in the new test file ai-rate-limiting-redis.t.

AlinsRan · 2025-12-03T05:17:15Z

apisix/plugins/limit-count/limit-count-redis.lua

+    local commit = true
+    if dry_run ~= nil then
+        commit = not dry_run
+    end
+
+    local delay, remaining, ttl = util.redis_incoming(self, red, key, commit, cost)


Suggested change

local commit = true

if dry_run ~= nil then

commit = not dry_run

end

local delay, remaining, ttl = util.redis_incoming(self, red, key, commit, cost)

local delay, remaining, ttl = util.redis_incoming(self, red, key, not commit, cost)

AlinsRan · 2025-12-03T05:18:44Z

apisix/plugins/limit-count/util.lua

+    local requested_cost = cost or 1
+    local script_cost = commit and requested_cost or 0
+    local res, err = red:eval(commit_script, 1, key, limit, window, script_cost)


Suggested change

local requested_cost = cost or 1

local script_cost = commit and requested_cost or 0

local res, err = red:eval(commit_script, 1, key, limit, window, script_cost)

local res, err = red:eval(commit_script, 1, key, limit, window, commit and cost or 0)

AlinsRan · 2025-12-03T05:20:19Z

apisix/plugins/limit-count/util.lua

+    local remaining
+    if commit then
+        remaining = stored_remaining
+    else
+        remaining = stored_remaining - requested_cost
+    end


Suggested change

local remaining

if commit then

remaining = stored_remaining

else

remaining = stored_remaining - requested_cost

end

local remaining = stored_remaining - (commit and 0 or cost)

AlinsRan · 2025-12-03T05:23:17Z

apisix/plugins/limit-count/util.lua

+    return 0, remaining, ttl
+end
+
+function _M.redis_log_phase_incoming(self, red, key, cost)


I think it's redundant. We can call redis_incoming.

AlinsRan · 2025-12-03T05:24:38Z

t/plugin/ai-rate-limiting.t

+--- more_headers
+Authorization: Bearer token
+--- error_code eval
+[200, 200, 200, 503]


I think it's necessary to check the response header:

--- response_headers eval [ "X-AI-RateLimit-Remaining-ai-proxy-openai: 29", "X-AI-RateLimit-Remaining-ai-proxy-openai: 19", "X-AI-RateLimit-Remaining-ai-proxy-openai: 9", "X-AI-RateLimit-Remaining-ai-proxy-openai: 0", ]

AlinsRan · 2025-12-03T05:24:49Z

t/plugin/ai-rate-limiting.t

+--- more_headers
+Authorization: Bearer token
+--- error_code eval
+[200, 200, 200, 503]


AlinsRan · 2025-12-03T05:28:56Z

t/plugin/limit-count-redis.t

 --- config
    location /t {
        content_by_lua_block {
+            require("lib.test_redis").flush_all()


It can be globally initialized and reset without the need to call it in every test chunk, and the same applies to other test files.

add_block_preprocessor(sub { my ($block) = @_; my $extra_init_worker_by_lua = <<_EOC_; require("lib.test_redis").flush_all() _EOC_ $block->set_value("extra_init_worker_by_lua", $extra_init_worker_by_lua); });

AlinsRan · 2025-12-03T05:44:11Z

apisix/plugins/limit-count/limit-count-redis.lua

+end
+
+
+function _M.incoming(self, key, cost, dry_run)


We should be consistent with local and recommend using the commit parameter instead of dry_run.

It seems that the modification of the init.lua file was overlooked, and the caller did not pass the dry_run parameter.

apisix/apisix/plugins/limit-count/init.lua

Line 282 in 896d3c3

delay, remaining, reset = lim:incoming(key, cost)

AlinsRan · 2025-12-03T05:58:31Z

apisix/plugins/limit-count/limit-count-redis.lua

+
+
+function _M.incoming(self, key, cost, dry_run)
+    if get_phase() == "log" then


I think we should use a new function to handle the log phase instead of dealing with it within the incoming function. This is because we need to mock parameters like remaining, which isn't reasonable, and the log phase doesn't require a status code.

apisix/apisix/plugins/limit-count/init.lua

Lines 301 to 322 in 896d3c3

if not delay then

local err = remaining

if err == "rejected" then

-- show count limit header when rejected

if conf.show_limit_quota_header and set_header then

core.response.set_header(set_limit_headers.limit_header, conf.count,

set_limit_headers.remaining_header, 0,

set_limit_headers.reset_header, reset)

end

if conf.rejected_msg then

return conf.rejected_code, { error_msg = conf.rejected_msg }

end

return conf.rejected_code

end

core.log.error("failed to limit count: ", err)

if conf.allow_degradation then

return

end

return 500, {error_msg = "failed to limit count"}

end

Baoyuantop · 2025-12-03T06:08:36Z

Hi @beardnick, please check these comments.

beardnick added 14 commits November 8, 2025 15:48

refactor: align limit-count redis implementation with limit-conn

2194930

feat: stable tests

680af73

feat: redis test util

75b0aa7

feat: review

14e6fce

feat: review

d5552d8

feat: review

82323ae

Merge branch 'apache:master' into feature/ai-rate-limiting-redis-support

3b8e2ad

Merge branch 'apache:master' into feature/ai-rate-limiting-redis-support

5204105

feat: delete useless file

c857713

feat: inline code

73737c1

feat: set keepalive_pool to 0 by default in test_redis.lua

b303401

feat: add keepalive_pool to test_redis.lua

0b76863

test: flush redis before each test

f86b98f

doc: update documentation

706bcb7

dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. enhancement New feature or request labels Nov 16, 2025

beardnick changed the title ~~Feature/ai rate limiting redis support~~ feat: ai rate limiting redis support Nov 16, 2025

membphis reviewed Nov 17, 2025

View reviewed changes

apisix/plugins/limit-count/util.lua Show resolved Hide resolved

AlinsRan reviewed Nov 17, 2025

View reviewed changes

apisix/plugins/limit-count/init.lua Outdated Show resolved Hide resolved

beardnick added 4 commits November 17, 2025 19:37

fix: lint

a10b0e7

feat: rename

ac70f00

fix: limit in log phase

f9ceaeb

feat: add license header

5276e0c

nic-6443 reviewed Nov 21, 2025

View reviewed changes

apisix/plugins/limit-count/util.lua Outdated Show resolved Hide resolved

nic-6443 reviewed Nov 21, 2025

View reviewed changes

t/plugin/limit-conn-redis-cluster.t Show resolved Hide resolved

nic-6443 reviewed Nov 21, 2025

View reviewed changes

apisix/plugins/limit-count/limit-count-redis-cluster.lua Outdated Show resolved Hide resolved

beardnick added 4 commits November 23, 2025 09:33

feat: simplify the code

7f02e0e

feat: simplify the code

a6b71af

feat: simplify the code

a716a34

feat: simplify the code

5c409c6

AlinsRan reviewed Nov 25, 2025

View reviewed changes

AlinsRan reviewed Dec 3, 2025

View reviewed changes

t/plugin/ai-rate-limiting.t

--- more_headers

Authorization: Bearer token

--- error_code eval

[200, 200, 200, 503]

Copy link

Contributor

AlinsRan Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

AlinsRan reviewed Dec 3, 2025

View reviewed changes

	local function leaving_thread(premature, self, key, req_latency)

	local conf = self.conf
	local red, err = redis.new(conf)
	if not red then
	return red, err
	end
	return util.leaving(self, red, key, req_latency)
	end


	function _M.leaving(self, key, req_latency)
	-- log_by_lua can't use cosocket
	local ok, err = ngx_timer_at(0, leaving_thread, self, key, req_latency)
	if not ok then
	core.log.error("failed to create timer: ", err)
	return nil, err
	end

	return ok

	end



		function _M.incoming(self, key, cost, dry_run)
		if get_phase() == "log" then

	if not delay then
	local err = remaining
	if err == "rejected" then
	-- show count limit header when rejected
	if conf.show_limit_quota_header and set_header then
	core.response.set_header(set_limit_headers.limit_header, conf.count,
	set_limit_headers.remaining_header, 0,
	set_limit_headers.reset_header, reset)
	end

	if conf.rejected_msg then
	return conf.rejected_code, { error_msg = conf.rejected_msg }
	end
	return conf.rejected_code
	end

	core.log.error("failed to limit count: ", err)
	if conf.allow_degradation then
	return
	end
	return 500, {error_msg = "failed to limit count"}
	end

feat: ai rate limiting redis support #12751

Are you sure you want to change the base?

feat: ai rate limiting redis support #12751

Uh oh!

Conversation

beardnick commented Nov 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Which issue(s) this PR fixes:

Notice

Checklist

Uh oh!

beardnick commented Nov 16, 2025

Uh oh!

membphis commented Nov 17, 2025

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

nic-6443 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AlinsRan Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlinsRan Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlinsRan Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlinsRan Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Baoyuantop commented Dec 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

beardnick commented Nov 16, 2025 •

edited

Loading

nic-6443 left a comment •

edited

Loading

AlinsRan Nov 25, 2025 •

edited

Loading

AlinsRan Dec 3, 2025 •

edited

Loading

AlinsRan Dec 3, 2025 •

edited

Loading

AlinsRan Dec 3, 2025 •

edited

Loading