"A good engineer understands both the product and the infrastructure underneath it," Daniela says. "Your AI SRE should too."
Alert Overload
Daniela Miao is Co-Founder and CTO of Momento, a real-time data infrastructure company. Years earlier, she led platform engineering at Lightstep — and the same instinct for finding signal in noise applies to how she approaches alerting today.
Most teams have experienced an alert storm: hundreds of alerts within a minute of a backend service suddenly slowing down. The gateway, the API layer, and every downstream metric all start reporting the same failure, flooding the on-call team. Finding the actual issue under all that noise is tedious and not always obvious. The noise distracts and slows you down.
In order for an engineer to properly handle this alert storm, they need to understand what's at the root of it. That, in turn, requires an accurate understanding of the world. The best engineers have this in their heads, but most need help reconstructing it in real time.
Dependent Much?
Take the gateway-and-backend relationship as an example. "If the backend errors, the gateway in front of it is going to error too, because one depends on the other," Daniela explains. Page on both as independent failures, and you've turned one problem into two.
Her fix isn't blanket suppression, cranking down every alert until the noise goes away. It's placing alerts according to what each part of the system can actually tell you. Alert on customer-facing errors at the gateway. Alert on leading indicators, like tail latency, network pressure, and dropped packets, at the backend components upstream of it. What you don't need is the same error signal duplicated at every layer a failure touches.
Doing that well depends on your tooling. "It requires the observability tool to be quite knowledgeable about that architecture," she says, "and make suggestions based on an understanding of a constructed service graph."
At her own company, Momento, she doesn't have that tool yet but would love to.
Hand-to-Hand Alert Combat
Momento wants per-second or sub-second granularity across its infrastructure, plus visibility into the kernel and the network layer, well beyond the standard machine, CPU, memory, and application-level metrics most teams can afford to keep at that resolution. Today, capturing deep kernel, network, and packet-level telemetry across massive throughput means Momento operates around 10 to 15 second granularity, which Daniela explains is "pretty good, but not enough when your SLAs live in the microseconds."
More telemetry, however, means more places for the same failure to trip an alert. A single incident can now page a dozen components at once, and nothing in that pile tells you which one is the actual cause. "If you're over-alerting during on-calls, the problem is the team spends more time dealing with noise, and it hides the actual signal," she says. "The root cause component sometimes is one alert out of maybe a hundred others that you're getting paged for."
Momento currently continuously prunes alert noise based on real post-incident data, but Daniela wants to replace that iterative operational tuning with automated, dependency-aware decisions from day zero. "That pruning cycle is quite manual and slow," she says. "They kind of happen over a few incidents, rather than in some proactive, first-principles way of doing it." That's the same manual, incident-by-incident pattern-matching Daniela wants replaced with something systematic.
Lead This. Lag That.
Where you instrument matters as much as when you alert. Instrument only at the layer closest to the customer, and you find out after the damage is done. Instrument further upstream, and you catch it before it gets there.
"This is rule number one in the SRE book," Daniela says: "you always set your SLO as close to your customer, if possible." For a consumer-facing product, that's usually enough.
For infrastructure companies, it isn't. "By the time we detect the error that's closest to the customer, it is too late, we've already, maybe, caused an outage for them," she says.
My Kingdom for a Map
Daniela isn't just guessing at how to place alerts. She's watched the same dynamic play out before, in a different form, at Lightstep.
She remembers customers at Lightstep discovering another use for the observability platform: as an onboarding ground for engineers dropping into systems with hundreds, sometimes thousands, of services. "You get lost, you don't actually know what's happening in your application," she says.
What made the tool useful was more than the raw data. It was the way the data gave her a second version of the system to compare against her own mental model.
"This is my understanding of what's happening in my system," Daniela says, "and then this is what my telemetry tools are telling me their understanding is, based on the data that's actually flowing through these graphs. That's the power of instrumentation, the more you instrument, the more you discover about your system what you didn't even know."
What This Means for AI SREs
More telemetry and better-placed thresholds help. But knowing where to point them, and recognizing that most of a hundred alerts are just the same failure showing up again, still depends on understanding how the system was built: which services depend on which, which signals tend to appear together, and which alert actually points to something new.
Daniela has been skillfully solving pieces of that problem manually, one incident at a time, for years. Any AI SRE agent helping take up that challenge will inherit the same thorny problem. To be trusted with deciding what's worth paging someone about at 2 a.m., it needs more than access to telemetry. It needs enough understanding of the system to separate the root cause from the storm.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Daniela Miao is Co-Founder and CTO of Momento. She previously joined Lightstep as employee #9, led platform engineering as Director of Engineering, and launched the company's Metrics product ahead of its acquisition by ServiceNow. Earlier in her career, she was a Tech Lead on AWS DynamoDB.