I’ve spent part of this year building V.I.N.CENT, a fleet-observability tool that watches 327 production sites — named, yes, for the robot in The Black Hole. The dossier lives on the work page; these are the three lessons that turned out to be the actual product.
Archive what the platform discards
Hosting platforms remember less than you’d think — on Pantheon, deploy history runs about 14 days and traffic granularity about 28. But the questions arrive later than the data expires: “when did this start?” is always asked after the window closed. So the first real design call was unglamorous: collect continuously, keep everything. Eleven thousand workflow events and counting — history the platform itself no longer holds. If you’re building any observability tool, this is the cheap decision that pays the longest: tape every show, because you can’t re-record the one that mattered.
Judge each site against itself
Global thresholds lie. “Cache hit rate below 70%” flags a quiet brochure site that’s fine and misses a busy store that fell from 95. V.I.N.CENT instead keeps 90 days of each site’s own history and asks whether this site deviated from its own normal (median/MAD, for the statistically inclined — robust statistics that a weird Tuesday can’t skew). “Dropped” has to mean dropped for this site, or an alert is just weather.
Silence is a feature you build
Four severity tiers, and only one interrupts anybody: emergency means a capability is gone right now — a site that can’t send email, an exposed installer. Everything below that stays calm by design, because an alert channel that cries daily trains everyone to stop reading it, and then the real one drowns. Nothing glows when the fleet is healthy. That restraint took more engineering than the detection did, and it’s worth more.
One craft note from the trenches: platform APIs will hand you paginated results with polite defaults and silently truncate the rest — and a truncated fleet is indistinguishable from a healthy one. Whatever you build on top of someone else’s API, go find out what it isn’t telling you first. The full decision log, including how each call aged, is on the case-study page.