How to Rate-Limit Stolen Large-Model API Calls: Stop the Bleeding First, Then Set Tiered Thresholds by Token

When you notice abnormal billing or a clogged GPU queue, act in this order: revoke and rotate keys → set hard spending caps on the provider side → rate-limit by endpoint path at the edge and block abnormal signatures → hide the inference origin → then return to fill in multi-dimensional rate limiting at the gateway layer.

One caveat: large-model endpoints can't simply copy the traditional REST API approach of "how many requests per second per IP." Cost lies in tokens and GPU inference time. A single very long prompt plus a very large max_tokens counts as only 1 request but can cost as much as thousands of normal calls. So the rate-limiting thresholds must switch dimensions.

Spend two minutes first to identify which kind of abuse it is

The three cases require different actions, and misjudging them wastes effort. Look at three columns of data in your provider console or gateway logs: source IP distribution, model distribution, and per-call token consumption.

• Main key leak: The key was committed to Git, bundled into the frontend, written into logs, or exposed in a screenshot. Signs include calls from completely unfamiliar IPs and UAs, chaotic model and parameter choices, and people often using your key for tasks unrelated to your business.
• Frontend endpoint captured and reused: Your own business endpoint lacks strong authentication, or only does Referer checks. Signs include call paths being your own endpoints, but Referer and UA not matching, or the same user ID making far more calls than normal usage.
• Distributed abuse: Rotating IPs through residential proxy pools or botnets. Signs include no single IP having high frequency, but IPs being extremely dispersed and total token volume soaring, often accompanied by very long inputs or streaming long connections held open.

The third is the easiest to misjudge. If you find every IP stays within the threshold while total consumption spikes, stop adjusting the per-IP frequency threshold—that path won't block it. Switch to fingerprint and behavioral dimensions.

First-hour containment sequence

1. Rotate the key. If it's a key leak, revoke the current key in the provider console and generate a new one, storing the new key only in server-side environment variables or a secrets management service. Before revoking, confirm which production services use this key so you don't stop your own business too; if there's no time to distinguish, stop them first by the principle of minimizing loss, then restore one by one—bills grow by the second, and downtime is easier to clean up than overspending.

2. Set hard spending and quota caps. Most commercial large-model providers support account-level or project-level usage limits and alerts; for self-hosted inference clusters, configure daily quotas at the gateway. This step is the fallback: when all the rate-limiting rules below fail, it determines the maximum you lose that day. Many teams only remember this switch was never turned on after being abused.

3. Rate-limit by endpoint path at the edge and block abnormal signatures. Set separate frequency and concurrency limits for inference paths like /v1/chat/completions instead of sharing rules with static assets. At the same time, block illegal Referers, empty or abnormal User-Agents, and known proxy IP ranges; add JS challenges or human verification for suspected bulk sources. The advantage of doing this at the edge is that requests are dropped before reaching the inference backend, so no compute or tokens are spent. If blocked at the application layer, the GPU may already have started.

If you're being abused right now and don't yet have usable edge blocking, first point your domain to WAF and CC protection and hold the line with path rate limiting and bot detection, then gradually add fine-grained rules. In an emergency, you can go directly to emergency onboarding.

4. Hide the inference origin. If the attacker already has your inference server IP and connects directly bypassing the edge, the first three steps are useless. Change origin access to allow only edge node IPs or add an origin authentication header. For details, see How to hide the origin? Five gates from exposed surface to mTLS origin.

The four quantities that LLM rate limiting must limit

After stopping the bleeding, rebuild the rate-limiting rules. For large-model endpoints, cover at least these four items; missing one leaves a gap:

The third item is especially easy to miss. The server should cap and truncate max_tokens rather than passing through user-supplied values unchanged; same for context length—reject anything beyond a reasonable business range.

How to configure gateway-layer rate limiting

Use composite keys, not just IP. The suggested granularity is a combination of 用户ID + API Key + 目标模型, with IP only as an auxiliary dimension. The reason is that under distributed abuse IPs are already unreliable, and the same user's reasonable usage differs across models anyway.

Use token bucket or sliding window algorithms. A token bucket allows some burstiness, suitable for normal users occasionally sending several requests in a row; a sliding window gives smoother statistics, suitable for per-minute token quotas. In multi-instance deployments, you must use Redis for distributed atomic counting; otherwise each instance counts separately and the actual allowed volume is the threshold multiplied by the number of instances.

Return 429 with Retry-After when over limit. Don't use timeouts, disconnects, or 500s to signal rate limiting—clients can't tell whether they were rate-limited or the service is down, and usually retry immediately, worsening pressure. A correct 429 with a retry interval lets normal clients back off.

Set differentiated quotas by model. Flagship models have high unit cost, so keep quotas tight; lightweight models can be looser. Also separate free/trial accounts and paid accounts into different pools so trial traffic doesn't squeeze out paid users' capacity.

Add account-level daily spending circuit breaking. When an account's daily spend hits the threshold, stop its calls and alert. This is the final gate beyond rate limiting. Set the trigger by amount rather than request count, because amount is what you actually care about.

Eliminate the problem at the architecture level

Containment and rate limiting are responses; the following are prerequisites for preventing the same incident from recurring:

• Never put a long-term main key in frontends, mini-programs, or desktop clients. Bundled artifacts can be decompiled and traffic captured; so-called encrypted storage only raises the cost slightly. Route all calls through a business server for relayed authentication.
• Issue short-lived temporary credentials when direct client-side access is needed. Have the server issue time-limited temporary tokens, combined with nonces to prevent replay, so the effective window after a leak is only a few minutes.
• Monitor token rate and per-user spending curves, not just request counts. Set anomaly alerts and push them to a channel you'll see immediately (e.g., Telegram). Abuse often causes significant loss within tens of minutes; waiting until the next day's bill is too late.
• Run normal business once after rate-limiting rules go live. New rules accidentally blocking your own callbacks or batch jobs is a common incident. For troubleshooting ideas, see What to do when WAF blocks payment callbacks: adding a whitelist. Allow precisely by path and source; don't disable the whole rule.

Two premises that change the specific approach

Whether you self-host inference or proxy an external commercial API. When self-hosting a vLLM/Ollama cluster, the loss is GPU compute and availability, so circuit-breaker thresholds should be set by concurrency and VRAM usage, focusing on preserving service quality for normal users. When proxying a commercial API, the loss is directly the bill, so the focus is provider-side spending caps and key management. The containment priorities for these two are exactly opposite.

Your existing gateway form. Gateways like APISIX, Kong, and Higress have built-in rate-limiting plugins, so you only need to configure composite keys and Redis counting. If you currently only have Nginx, limit_req can only do frequency control at the IP or fixed-key dimension; token-dimension rate limiting must be implemented yourself at the application layer, or you need to add an AI gateway layer in front. First confirm what you have, then decide which layer handles which dimension, to avoid configuring the same limit in three places and having them conflict.