The symptom: a 2% error rate that wouldn’t tune away
The client’s monitoring showed a Lambda-backed service with a steady 2% error rate. Not a spike, not an incident — a persistent background failure rate that had been there long enough to be treated as “just how the service behaves.” The failures weren’t random: they clustered around periods of uneven invocation frequency — bursts after quiet stretches — which is the signature of a cold-start problem, not a code bug.
What made this expensive rather than just annoying: the function sat in the call path for 5 services. A failure at this layer didn’t stay contained — it propagated downstream as retries, timeouts, and partial failures in every service that depended on it. The blast radius was much larger than the one function’s own error budget suggested.
Why cold starts were failing outright, not just running slow
Lambda cold starts are usually framed as a latency problem — the first invocation after idle takes longer while the runtime initializes. That’s true, but it understates the failure mode that actually hurts in production: when a cold start’s added latency pushes a request past a caller’s timeout, the caller doesn’t see “slow.” It sees a failed request. If the caller retries into another cold instance, or if the timeout is tight relative to typical cold-start duration, that failure is deterministic under certain traffic patterns — not a fluke.
The usual Lambda tuning options didn’t close the gap here:
- Provisioned Concurrency keeps a fixed number of instances warm, but it’s a cost/coverage tradeoff — under-provision it and you’re still cold-starting during bursts above the provisioned count, which is exactly when the problem showed up.
- Increasing memory (which also increases CPU allocation) reduces cold-start duration somewhat, but doesn’t eliminate the init window — it narrows it.
- Keeping the function warm with a scheduled ping only guarantees one instance stays warm; concurrent bursts still spin up additional cold instances alongside it.
None of these change the fundamental shape of the problem: Lambda’s execution model tears down and re-initializes instances based on traffic and idle time, and no amount of tuning removes that model — it only shifts where the cold starts land.
The fix: move the function off Lambda
For a function sitting in the critical path of 5 services with downstream dependents, the right fix wasn’t to tune around cold starts — it was to remove the condition that causes them. Moving the workload onto an EC2 instance (or a small autoscaled fleet, sized to the service’s actual baseline load) means the process stays running continuously. There is no idle-teardown cycle, so there is no cold start to trigger a timeout in the first place.
This is not a general “Lambda is bad, EC2 is good” conclusion — for genuinely spiky, low-frequency, or event-driven workloads, Lambda’s pay-per-invocation model and zero-idle-cost are still the right tradeoff. The signal that flips the decision is specific: a function with steady baseline traffic, sitting in a synchronous call path with tight downstream timeouts, where cold-start latency is large relative to those timeouts. That combination turns Lambda’s elasticity — normally an advantage — into the direct cause of the failures.
The result
After the migration: the error rate at this layer dropped from 2% to 0.01% — roughly a 200× reduction. Because the function sat upstream of 5 dependent services, the fix improved reliability across the whole call chain, not just the one metric that was being watched. The remaining 0.01% reflects genuine transient network and infrastructure failures — the kind of error rate you’d expect from a healthy, always-on service — rather than a structural, traffic-pattern-driven failure mode.
How to tell if this applies to you
Before assuming your Lambda function has the same problem, check for this specific pattern:
- Correlate errors with invocation gaps. Pull your error timestamps and check whether they cluster after periods of low or zero invocations — that’s the cold-start signature, distinct from errors that are evenly distributed or tied to specific input payloads.
- Compare cold-start duration to your tightest caller timeout. If anything calling this function synchronously has a timeout close to or shorter than observed cold-start latency, you have a structural problem, not an occasional one.
- Map what’s actually downstream. A function’s own error rate can look tolerable in isolation while still being the single largest contributor to failures elsewhere in the system. Trace the call graph before deciding the risk is acceptable.
When NOT to do this
A few honest caveats, because this fix isn’t universal:
- Genuinely spiky or rare workloads. If your function runs a handful of times a day with no latency-sensitive caller, cold starts are irrelevant — don’t pay for an always-on instance to solve a problem you don’t have.
- Workloads that scale far beyond what you’d want to manage on EC2. Lambda’s automatic horizontal scaling is a real advantage for unpredictable, high-fan-out traffic — moving to EC2 means you now own the autoscaling, patching, and capacity planning yourself.
- Provisioned Concurrency may be enough. If your traffic pattern is predictable rather than bursty, sizing Provisioned Concurrency correctly can close the gap without a full migration. Reserve the EC2 move for cases where the traffic pattern itself is the problem.
Related reading
- Are Your AWS Lambda Functions Leaking Money? The Hidden Cost of CloudWatch REPORT Logs — a different client, a different Lambda problem: the platform’s own logging was driving 57% of the CloudWatch bill.
- Autoscaling is Not Capacity Planning: Understanding the Differences for Optimal Performance
- Autoscaling Reaction Time: Why It Matters for Reliable Cloud Systems
