Noisy Neighbors in Background Jobs: Fair Queues for Multi-Tenant SaaS

A customer kicks off a CSV import with 400,000 rows. Your worker pool picks it up and chews through it for forty minutes. During those forty minutes, every other tenant on the platform sees their "export report" and "sync inventory" jobs sit in the queue. Support gets messages that the product is down. It is not down. It is working perfectly, for exactly one customer.

This is the noisy neighbor problem, and in background jobs it is almost always a scheduling problem rather than a capacity problem. A single first-in-first-out queue treats work as if all jobs were equal and all tenants were one tenant. The tenant who submits the most jobs gets the most service, which is the opposite of what a shared platform promises.

You do not need a queue per tenant to fix it. You need to decide what "fair" means for your product and put a small amount of logic between the queue and the workers.

Decide what fair means before touching the queue

There are two reasonable definitions and they lead to different designs.

• Equal share. Every tenant with waiting work gets roughly the same slice of workers. A tenant with one job and a tenant with ten thousand jobs both make progress right away.
• Weighted share. Plans buy capacity. A larger plan gets a larger slice, but no tenant can reduce another tenant's slice to zero.

Most teams should start with equal share and a hard ceiling per tenant, then add weights only when a pricing conversation demands it. Weighted fairness is easy to promise on a pricing page and hard to explain when a premium customer's job is still waiting.

The cheap fix: a per-tenant concurrency cap

The smallest change that helps is a ceiling on how many jobs from one tenant can run at the same time. If you have 20 workers and set the cap at 4, one tenant can never occupy more than a fifth of the pool. The rest stays available to everyone else.

Implement it where jobs are claimed, not where they are enqueued. Enqueueing is cheap and you want to accept the work. The question is only whether a worker should start it now.

-- Claim the oldest eligible job whose tenant is under its cap.
WITH running AS (
  SELECT tenant_id, count(*) AS n
  FROM jobs WHERE status = 'running' GROUP BY tenant_id
)
UPDATE jobs SET status = 'running', claimed_at = now()
WHERE id = (
  SELECT j.id FROM jobs j
  LEFT JOIN running r ON r.tenant_id = j.tenant_id
  WHERE j.status = 'queued'
    AND COALESCE(r.n, 0) < 4          -- per-tenant cap
  ORDER BY j.enqueued_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING *;

This is a plain Postgres query and it works comfortably at the scale most SaaS products reach. The cost is that the claim query now does a little more work, so index (status, tenant_id, enqueued_at) and watch its latency.

Round robin when caps are not enough

A cap protects the pool but still lets a flooding tenant be first in line every time a slot frees up. If you want a tenant with three waiting jobs to start ahead of a tenant with forty thousand, claim by tenant rather than by job: pick the tenant that was served least recently among those with queued work, then take that tenant's oldest job.

The tradeoff is ordering. Within a tenant you keep FIFO, but across tenants you give up global arrival order. That is the point, and it is worth telling your team plainly, because someone will file a bug saying "my job was submitted before theirs and ran after."

Separate lanes for different kinds of work

Fairness between tenants is only half the problem. A cheap 200-millisecond job and a 40-minute import should not share a lane at all. Split the pool by job shape:

• An interactive lane for short jobs a user is waiting on, such as sending a receipt or recalculating a total.
• A bulk lane for imports, exports, report generation and migrations, with a lower cap and a longer timeout.
• A scheduled lane for periodic work that can run late without anyone noticing.

Lanes also give you a place to apply backpressure. If the bulk lane is saturated, you can tell the user "your import is queued, position 3" instead of letting it silently delay everything else.

Make the wait visible

Half of the support burden from noisy neighbors is not the delay itself but the silence around it. Surface job state in the product: queued, running, percent complete where you can compute it, and an honest estimate only if you have data to back it. The same principle applies to the admin side. A support agent should be able to see which tenant is occupying the bulk lane without running SQL, which is one of the reasons we recommend a proper admin console for support.

When this is overkill

If your platform has twelve tenants and none of them import anything large, a shared queue with a sensible timeout is fine. Fairness logic adds a failure surface of its own: a bug in the claim query can strand jobs, and a cap set too low leaves workers idle while work waits. Measure first. If p95 queue wait for small jobs stays under a few seconds during your heaviest tenant's heaviest hour, leave the design alone.

It also does not help when the real constraint is downstream, such as a third-party API with its own rate limit. In that case the cap belongs on the integration, not on the tenant. Our own multi-tenant work, covered in the SaaS platforms we build, usually treats those two limits as separate knobs.

A short rollout plan

1. Record queue wait time per job and per tenant. You cannot tune what you cannot see.
2. Add the per-tenant concurrency cap, starting generous, and watch for idle workers.
3. Split bulk work into its own lane.
4. Move to round-robin claiming only if one tenant still dominates the first-served position.
5. Expose job state to users and to support.

Fairness is a product decision with an engineering implementation. If you are planning tenancy for a new platform, our write-up on shared schema versus database per tenant is a good companion, since the isolation model you pick shapes how much of this you need.