Traditional three-tiered architecture is built around a data store, a middle tier and a UI layer. One key focus of the middle tier is to cache entities for rapid serving. Caching is typically done in memory or using platforms like Redis. Here we cover how to cache resources like containers. Can we lean on ready-made solutions? To answer that, we need to analyze how close our scenario is to the typical ones. At first glance, containers look just like any key-value pair. But are there attributes that force a custom solution? Let's think.
Cache Hit
We design caching solutions to increase cache hits — a user request served at the middle cache layer without refreshing from the backend store. The cache stays valid and can serve future requests; it only gets invalidated on an update. For containers, a user requests a container backed by a specific image. We can serve it from the cache pool, but once served, it is no longer available for future requests. This smells more like inventory management, where we fill an inventory with specific objects and serve users until it runs out. Let us follow our nose.
Container Inventory Management
Inventory management requires decisions about what objects to insert in the pool, when to evict an unused object, and when to order new inventory. We can define policies for each, keeping in mind the velocity of consumption, the variability of demand, and the time it takes for a reorder to be fulfilled. The insertion policy relies on accurate demand forecasting. We have a notion of safety stock for each cached container: raise it to increase the service level (at higher operational cost), or reduce it to save cost (at the price of higher latency). These considerations define the reorder policy. And if we find a stale container that isn't being requested, we need an eviction policy to manage it.
Metrics
Customer metrics
The customer cares about the time it takes to fulfill her request. A bigger caching layer meets that expectation but raises operational costs — which means we should also look beyond caching to make the system fast.
Platform metrics
The platform maintains a series of metrics to manage inventory efficiently, including hit-rate and eviction-rate. To manage the cost of inventory, we measure the age of objects in the pool. We aim for short-lived containers served quickly to end users. The platform benefits from the flexibility to trade off service level against operational cost.