Whose Service Is It Anyway?

The Importance of Ownership

An incident kicks off at 2 a.m. and the on-call engineer gets paged. After some digging, they track it down to a service, but they aren’t sure how to fix it. So they go to escalate to the service owners, only… nobody actually knows who to page. The team that built it got reorged last year, the person listed in the README left the company, and the Slack channel in the runbook has been archived. Now it’s 2:45 a.m. and the incident is still going. At this point, they need to improvise (get it?).

That’s the dramatic version, but you can probably think of plenty of times this has happened on a smaller scale. A security vulnerability sits in a backlog because nobody thinks it’s theirs. A migration stalls at 80% because the last 20% of services don’t have anyone to chase. A dependency gets deprecated and nobody hears about it until it breaks. Gaps in ownership lead to worse accountability, and worse accountability leads to worse outcomes for an engineering organization. In this post, we’ll walk through how we built a system at Cortex that recommends owners for services automatically. That way, when the next page comes in, there’s no scramble to figure out who to call.

For users of Cortex, ownership is the foundation that everything else sits on. Scorecards, initiatives, notifications, and on-call routing all depend on someone being on the hook for each service. Good ownership data means the rest of the platform just works. Bad ownership data means every feature built on top of it inherits the problem.

Why It’s Difficult

If ownership is so important, why doesn’t everyone just have it figured out? There are a lot of reasons it gets messy, and most of them come down to the fact that organizations change much faster than the metadata that describes them.

Reorgs and team renames. A team splits, two teams merge, or a team just gets a new name. The services don’t move on their own, so the ownership data slowly drifts from reality.

People leave. An engineer who was the go-to person for a service leaves the company, and their name stays on it for months.

Monorepos. Lots of hands touch lots of different pieces of the same repository, so “who owns this repo” isn’t really the right question anymore.

Too many sources of truth. Ownership can live in a catalog, in a CODEOWNERS file, in a PagerDuty schedule, in a spreadsheet, or in somebody’s head. When they disagree, which one wins?

The good news is that the hardest part is getting started. Once ownership goes from 0 to 1, it becomes a bit less overwhelming to maintain. A team splits into two subteams? Just reassign the services. A new service gets created? Give it an owner on day one. Keeping ownership accurate is mostly a matter of small, incremental changes. It’s that first pass across hundreds or thousands of unowned services that feels impossible. However, ownership still needs regular upkeep and attention as things change. Bitrot can just as well put ownership in a bad state over time.

Ownership itself is also more complex than it looks at first. A service can be owned by a team, by an individual, or by both. Teams can have parent teams, and entities (anything in the Cortex catalog, like a service or a domain) can have parent entities, so ownership can be transitive. For example, I own a service because I’m a member of a team, my team owns a domain, and that domain contains the service. Here’s what that can look like:

So who owns search-api? Jordan does, through the Search team and the Search domain. Does Platform Eng own it too? Depending on how you ask, maybe. And indexer has two paths to Jordan, one direct and one through the team. Every one of those questions turns into a graph traversal, and we’ve spent a lot of time investing in our core data model to make those traversals fast.

More broadly, consider this a context graph of any of the relationships in our system. Nodes like the team or entity are bound by edges, which are the relationships between them. Going further, entities could have a Slack channel. That’s another relationship. And when you enumerate all of the possible types and relationships, you can imagine how things could become a ball of yarn.

Going back to ownership, to set the graph up for success, we care about which edges get created. Look at that direct line from Jordan to indexer. It’s the most fragile edge in the picture. If we didn’t have the team owner edges, and if Jordan left the company, indexer ends up with nobody, which is exactly the problem from the start of this post. The edge from the Search team to the Search domain is much more durable. People can come and go, and as long as the team exists, everything underneath it still has an owner, and every member of the team still inherits it.

That’s why, when we recommend an owner, we recommend a team and not an individual. The tricky part is that most of the hints about who works on a service point at people, not teams. The context graph is what connects the two. If Jordan works on a service a lot, and Jordan is a member of the Search team, then the Search team is a good guess for the owner. So the problem we’re actually solving is filling in the missing edges in this graph, and filling them in at the team level so they don’t go stale.

Prediction Framework

To address these issues, we’ve been building out a new ownership recommendation system to facilitate both that initial 0 to 1 step and the ongoing maintenance of ownership. The idea is pretty simple: instead of asking someone to research hundreds of unowned services one at a time, we look at the signals we already have, figure out which team most likely owns each one, and hand that over as a recommendation. It also catches some of that bitrot, so if an owning team gets archived, we recommend new owners for the services it left behind. Someone still makes the final call, but reviewing a good guess is a lot faster than starting from a blank field.

Getting the data in

The signals we need live in the same Postgres database that serves the Cortex app, and we don’t want a model training job creating heavy load. So the first piece is a change data capture (CDC) pipeline. Every insert, update, and delete in the product database streams into BigQuery, where we store a log of operations, and merge jobs keep a replica of each table current. The ML side of the house only ever reads from our data warehouse here, as to not impact any production workloads.

From there, two tools do most of the work:

What the model looks at

For every service, we build a list of candidate teams and describe each (service, team) pair with a set of features. These features typically fall into three families:

Code activity. Who writes and reviews pull requests in the service’s repositories, across GitHub, GitLab, Azure DevOps, and Bitbucket. Activity belongs to people, so we use identity and team membership data to credit it to teams.

The neighborhood in the graph. Services that share a repository, a Jira project, a Slack or Microsoft Teams channel, a domain, or a dependency tend to have related owners.

Names and descriptions. We check for similarity in names, descriptions, and tags between the entities owned by a team, and the target entity.

The labels come from ownership that customers have already set up. Every catalog has some services with a team owner, and those teach the model what ownership looks like in that organization. For inference, we then go through each service, surfacing the top few teams, each with a confidence level.

What we learned along the way

Generalize to customers the model has never seen. Most services we score belong to organizations that the model never trained on. One example of this is a new customer that’s starting to onboard and they want to establish ownership. Features that looked great on familiar data, like how many services a team already owns, didn’t hold up on new organizations, so we cut them to make sure the model could generalize.

Some edges in the graph are noise. A channel that half the company uses, or a monorepo that every team touches can link services together without giving us any real information about about who owns them. We learned to ignore those hubs, and monorepos are still one of the hardest cases.

Don’t settle for an easy answer. When a service inherits its owner from a parent domain, the “right answer” is just a copy of the parent. Training on those taught the model a shortcut instead of the real signal, so we leave them out. It’s the same reason a recommendation like “Engineering” isn’t useful. It’s technically right, but it isn’t really valuable information.

Results

Ownership recommendation is a ranking problem, not totally dissimilar from search. For a service, we don’t just want to know whether one team is the owner. We want the right team at the top of the list. So we evaluate the model the way you’d evaluate a search engine. For each service with a known owner, we rank every candidate team and look at:

There are a couple interesting points about how we evaluate the model performance. For example, we hold out whole organizations, not random services. We split customers into folds, train on all but one fold, and test on the customers the model has never seen. A random split would let the model learn one organization’s conventions and then get graded on the same organization.

We also tune specifically for precision precisely because a wrong owner is worse than no suggestion. We don’t want the 2 a.m. page to the wrong team, so we’d rather have fewer high-confidence suggestions, than more low-confidence ones. This is reflected in the metrics above. Precision (0.94) is much higher than recall (0.53).

Every training run records these metrics per customer and in aggregate, and sends them to our monitoring, so that a drop shows up before customers notice it.

In plain terms, results vary between customers, but for the median customer, the first suggestion is the right team about 74% of the time, and the right team is in the top three about 90% of the time.¹

One caveat is that we can only grade the model on services that already have an owner. As with most recommendation systems, offline metrics only tell part of the story. They measure how well we reproduce ownership that already exists. However, we’d like to eventually track what happens to the recommendations in practice, e.g. how often do people really accept them?

Try It Out

Did some of these issues resonate? Well good news, our new ownership recommendations are now in public beta!

If you’re a Cortex customer, you can find recommendations in Tools -> Ownership. It is particularly useful you when you have a lot of services without a team owner.

Since this is a public beta, we’d love your feedback. If you aren’t a Cortex customer yet and the 2 a.m. page from the start of this post sounds a little too familiar, book a demo and we’ll show you around.

Ownership isn’t the most exciting part of a service catalog, but it’s the part that everything else depends on. Our goal is to make that something you can finish setting up in an afternoon, so you can get back to the parts of engineering that are actually fun.