On this page 12 sections
Rate limits are a shared resource across everything that calls the same API, so a workflow that behaved fine alone starts failing when a second one is added, or when you scale workers. Budget concurrency per API rather than per workflow, back off exponentially with jitter on 429s, and respect the retry-after header when the provider sends one. The failure to design for is the daily quota, not the per-second one.
Per-second limits produce errors you notice immediately. Daily quotas produce a workflow that stops working at 3pm and starts again at midnight, which is much harder to diagnose.
Key takeaways
- The limit belongs to the API, not to your workflow. Everything you run shares it.
- Scaling workers multiplies your request rate. That is how a capacity change causes 429s.
- Back off exponentially with jitter, or your retries synchronise and re-trigger the limit.
- Honour retry-after when it is sent. Guessing is worse than being told.
- Batch endpoints are usually the real fix. One request for 100 records beats 100 requests.
Who this applies to
You are running automations that call third-party APIs at any volume, particularly if you have recently added workers, added workflows, or started seeing intermittent 429s.
The three limits you are subject to
Most providers apply more than one, and they fail differently.
Per-second or per-minute rate limits. The common one. Exceed it and you get 429s immediately. Recoverable with backoff, and it is the limit people design for.
Daily or monthly quotas. Exceed it and you are done until the window resets. No amount of backoff helps. This is the one that produces the “it worked this morning” report, and it is the one most workflows have no handling for at all.
Concurrency limits. A cap on simultaneous in-flight requests rather than on rate. Adding workers hits this before it hits a rate limit, and the error is often less clear.
Check which apply to each API you use, and where the counter is scoped - per API key, per account, per endpoint. Per-account scoping is what makes this a shared-resource problem.
Why scaling causes this
The sequence is consistent enough to predict.
A workflow runs fine. Volume grows, so you add workers to keep queue depth down - see running n8n in production. Now four workers run the same workflow concurrently, quadrupling your request rate against an API whose limit has not changed.
The symptom presents as intermittent failures that correlate with load rather than with any particular record, and it appears immediately after a change that was supposed to improve things.
The same happens when a second workflow starts using an API the first one was already saturating. Neither workflow is wrong; the budget is shared and nobody was tracking it.
Budget per API, not per workflow
The mental shift that makes this tractable.
For each external API, know its limit and decide how your total request rate is allocated across everything that calls it. Then enforce that allocation somewhere shared, rather than hoping each workflow behaves.
Practical mechanisms, in increasing order of rigour:
Per-workflow concurrency settings. Limit how many items a workflow processes in parallel. Crude, and it works when one workflow dominates the traffic.
A shared token bucket. A counter in Redis that every caller decrements before making a request. This is the correct implementation for multiple workflows or multiple workers sharing a limit, and it is perhaps thirty lines.
A single gateway workflow. All calls to a given API go through one sub-workflow that owns the rate budget. Adds a hop and makes the limit enforceable in one place - the same argument as putting integration logic in a sub-workflow generally, see when to write a custom n8n node.
Whichever you choose, write down the limit and the allocation somewhere findable. The most common cause of this problem recurring is that nobody knew the budget was already spent.
Backoff that works
Three properties.
Exponential. Wait 1s, 2s, 4s, 8s. Retrying immediately against a rate limit is how you stay rate limited.
Jittered. Add randomness to each wait. Without it, twenty workers that all hit the limit at once all retry at the same moment, re-trigger the limit together, and synchronise into a pattern that does not resolve. Jitter is not an optimisation here; it is what stops the thundering herd.
Capped, and bounded in attempts. Three or four attempts, with a ceiling on the wait. Past that, route the record to the dead-letter store and move on - see n8n error handling.
And above all: honour Retry-After when the provider sends it. That header is the API telling you exactly when to come back. Ignoring it in favour of your own schedule is guessing when you have been given the answer.
Backoff does not help when the quota is exhausted for the day. Retrying just burns the retries.
Three responses that do work.
Track consumption yourself. Increment a counter per API per day. When you approach the limit, stop dispatching rather than discovering it through errors. Providers frequently return remaining-quota headers - read and record them rather than counting independently, since their number is authoritative.
Prioritise. If quota is scarce, spend it on the work that matters. A nightly enrichment job should yield to a customer-facing lookup. That requires knowing which workflows are competing, which is the same visibility the budget gives you.
Reschedule rather than fail. Work that hits a daily quota should be parked and retried after the reset, not dropped into the dead-letter queue as a failure. A separate “waiting for quota” state keeps it out of the error path, so a genuine failure is still visible.
Batch endpoints are the real fix
Before engineering around a limit, check whether the provider offers a batch or bulk endpoint.
Fetching 100 records individually is 100 requests against your budget. One bulk request is one. Where a batch endpoint exists, using it usually removes the problem rather than managing it, and it is faster.
The same logic applies to webhooks: if the provider can push changes to you, you stop polling, and polling is where most quota goes. See scheduled or event-driven.
Monitor before it bites
Three signals worth having:
- 429 rate by API. Rising before it becomes visible failure.
- Quota consumption against limit, as a percentage, per day. Alert at 80%.
- Request rate by API, so a new workflow’s contribution is visible when it appears rather than when it breaks something.
The middle one is the useful one. Discovering at 80% that you will run out at 4pm is a manageable problem; discovering it through failures at 4pm is not.
What we build in
A shared rate budget per external API where more than one workflow uses it, exponential backoff with jitter, Retry-After honoured, quota consumption tracked and alerted at 80%, and quota exhaustion handled as a reschedule rather than a failure.
The distinction we spend most time explaining is the last one. Treating a quota exhaustion as an error puts legitimate work into the dead-letter queue, where it competes for attention with genuine failures and where somebody has to decide whether to replay it. Treating it as “not yet” keeps the error path meaningful. It is a small design decision that determines whether anyone still trusts the error queue after a month.
The one we get asked to skip is the shared budget, because with a single workflow a per-workflow concurrency setting works and is simpler. That is fair on day one. It stops being fair the moment a second workflow calls the same API, and the failure at that point is attributed to the new workflow rather than to the missing budget - which sends the investigation in the wrong direction.
Where we have been wrong: we have historically under-used batch endpoints, engineering careful rate management around a per-record loop when the provider offered a bulk call that removed the problem. Check for one before building anything on this page.
When you can ignore this
Low volume against generous limits. If you use 2% of a quota, budget management is premature.
A single workflow, single worker, sequential processing. You cannot exceed much.
Internal APIs you control. Raise the limit or remove it, rather than engineering around your own constraint.
Frequently asked questions
How do we find out an API’s limits?
Documentation first, then response headers - many providers return remaining quota and reset time on every call. Log those headers; they are more reliable than documentation.
Should retries count against the limit?
They do, from the provider’s side. That is why capped retries matter: each attempt spends budget that successful work could have used.
What is a sensible jitter?
Randomising the wait across a range around the target - full jitter, where you pick uniformly between zero and the backoff ceiling, is a common and effective choice. The specific scheme matters less than having one.
Can we just get a higher limit?
Often yes, by asking, particularly on paid tiers. Worth doing before engineering. Also worth doing after, because the engineering does not stop being useful.
How do we handle multiple API keys?
Some teams rotate keys to multiply throughput. Check the terms first - where limits are scoped per account rather than per key, this does not work, and where it is prohibited it risks the account.
Next step
If you have seen 429s after adding capacity, the fix is a shared budget rather than more retries. The business process automation engagement builds rate management and quota monitoring alongside the workflows.
Related: Running n8n in production · n8n error handling · Scheduled or event-driven · Business process automation