The cloud promises infinite compute and storage with five-nines reliability. That abstraction is built by a network of interdependent components, each constantly upgraded to meet customer expectations. We'd like each component to upgrade without coordinating with the others, so upgrade velocity isn't compromised. But going fast has its cost — sometimes it causes a regression. When that happens, how do we know which component is the culprit? Let us explore.
Health signals
A complex system like Azure reports its state continuously, saved in easy-to-query logs. There are various indicators of health in these logs — when a component is under stress, for instance, it shows a flurry of exceptions. These indicators let us probe deeper into regressions.
Health and upgrade correlation
Our aim is to build a correlation between a component upgrade event and a worsening of component health. We call this spatio-temporal correlation. The stronger the correlation, the more likely a specific component is causing the regression and should lead the investigation.
Temporal correlation
Say a burglary is reported on a Sunday morning. The police investigate the closed-circuit footage around that time and find a few suspects, which builds a suspect list. The presence of suspects at the time of the burglary is an example of temporal correlation.
Spatial correlation
Now suppose the burglary is more widespread, reported at different times around the city. The police find that a common suspect shows up in the footage. The presence of a suspect at different locations around the city at the times of the burglaries is an example of spatial correlation.
Armed with this intuition, we built software that let us zero in on the component most likely behind a regression. We named it Gandalf.
Metrics
Precision
The percentage of regression detections across all detections. It's important to err on the side of precision, especially early on, so partners begin to trust the system.
Recall
The percentage of regression detections across all regressions. High recall means we catch most of the regressions in the system. Recall improves progressively as more components are onboarded to the Gandalf framework.