A complex system like a cloud needs continuous care. From time to time, we need to access production resources. These touches are necessary but risky. We'd like to assess the riskiness of our production touch points so we can reduce risk over time. Is there a way to get there, given the deluge of touches staring at us? Let us explore.
The Challenge
Typically there's governance around touches to production: you request access with a justification and you're granted a short time window. A few of these justifications give us a window into the risk of the access; for others, we need to look elsewhere. So the problem is twofold. How do we assess risk for the majority of production touches? And how do we group them into a smaller set that's easy to review and act on?
The Solution
Assessing risk
We review historical production touches and build rules on features like the justification to bootstrap a risk score. We then exploit the similarity between requests — for example, by embedding the request trace (building a search index) and finding similar requests. These inherit the rule-based risk score if they're close enough.
Grouping touches
Once indexed, we can find the neighbors of each production touch request. These peer-similarity scores build a similarity matrix, which we feed into a clustering algorithm — and out come clusters of production touches.
Serving
Finally, we publish clusters of production touches with their counts and owners, ordered by risk score. This lets owners start acting on the riskiest touches first.