Teams are building their own dev tools more than ever using AI. Honestly, we do too. The reality is that engineering teams, mine included, are just trying to move fast with the least amount of unnecessary dependencies. If I can hack something together over the weekend that does a good enough job to get us unblocked, guess what I'm going to do? Comparing notes with several customers, here's where I come out on when to DIY your own dev tool, and when it's not worth it.
Let's start with when a vibe-coded dev tool is perfectly fine. The short answer: a home-built agent is good at simple knowledge retrieval, with some real caveats I'll get to. One senior reliability lead I talked with built an internal agent that answers questions about their systems. It lives in Slack with read access to most of what they run, and people use it the way they used to use a senior engineer. How is this service deployed? What is it connected to? Who owns this alert? It answers most of these kinds of questions with decent accuracy. Bonus that it started as a hackathon project. Cheap, quick, dirty, and super useful. That's what we call a win. What makes those questions easy is that each one has an answer sitting in a single place, and the agent just has to go find it. That's the easy case. The caveat is that answer accuracy will break down when data has to be pulled from multiple systems and then knowing which source is correct or dealing with data contradiction becomes a real problem.
Now let's look at where it breaks down. The same reliability lead then built a deeper version for incidents, a set of skills that queries logs, pulls metrics, runs a profiler, diffs a good commit against a bad one. Real work went into it. So of course I was eager to hear how it fared with actual incidents. His answer? “It mostly gets it wrong.”
Here's my read on why, and it's a pattern I see everywhere now, in our own systems and in every team I talk to that has built one. It's not that the model is bad at this. After all, it's the same model that handled the simple questions so well. The difference is what you are asking it to do, and more importantly, what it has to work with.
Diagnosing an incident is not a lookup. When the alert fires, the answer does not exist yet. Getting to it means forming a hypothesis, testing it, throwing it out, forming another, the loop a good engineer runs using first principles. To run that loop, the agent needs the right information about your systems at each step, and this is exactly where a homegrown setup falls down. That requires a rigorously constructed knowledge graph with specialized data pipelines, and that is continuously updated. That's a big project. Without it, the agent starts every investigation from a blank slate, with no organized map of your systems to reason from.
When an agent has poor context about your environment, unable to reliably figure out what’s what, it just makes things up. Or you go the other way and wire it to everything, a dozen MCP servers across logs, metrics, traces, and repos. But now giving it everything—again without a specialized, dependable knowledge graph—simply overwhelms it and it blindly guesses.
Here is an example the reliability lead gave me that illustrates the second problem. A new deployment had started making permission checks it should not have, which buried another service in queries. The agent looked at the load and reported the obvious: the database cannot keep up with the volume of queries. That was true. But it was a symptom, one step downstream of the cause, and the agent never connected it back to the deployment that set it off. The question that would have solved it, why did the queries spike, was the one it never asked. It reported the symptom as the cause, because it never understood the system well enough to tie the two together.
So here is where I land. Simple knowledge tends to be easier and debugging tends to be harder. However, I need to warn you that plenty of knowledge work is in fact hard, and parts of debugging can be easy. The real variable is how many sources an answer depends on. One source, and a DIY agent that fetches relatively simple answers is great. When several sources contradict each other, adding more of them doesn't help, because data access was never the limiting factor. What's missing is a nuanced enough understanding of your systems to separate signal from noise.
So why buy anything, if you can build the good part yourself? Claude Code, Cursor, and Codex really are getting better fast. But they also need something an MCP server connection can’t offer: a curated, maintained picture of your systems, the knowledge prepared so it can be searched by meaning, the dependencies mapped so the agent starts with a sense of where to look and what to trust. And even if you manage to build it, it’s never finished because both the underlying model and the systems it reasons about continuously change. So what you need to in fact buy is something that creates and maintains an understanding of complex, interdependent systems and data. That I do not recommend trying to vibe code.
So in a way the decision isn’t really about “build versus buy.” It starts with knowing which kind of problem you're handing the agent. When the answer lives in one place and you need to retrieve simple answers, build it. And, honestly, I'm happy to compare notes and help you get there if you ever want to ping me.
But when the answer has to be harmonized from numerous changing sources that don't agree, that “wrong answer” your agent is giving you (with the utmost confidence) can easily turn an incident into an outage. That’s when I think you should buy.