My VP Eng Told Me to Stop Working on Observability. Here's Why He Was Wrong.

I was a Staff Engineer at a small, quickly growing start-up. Our new VP eng pulled me aside:

it seems like you care a lot about developer experience and observability, but I need you to invest more in the feature work.

I understood where he was coming from, we were in a critical time of growing the business and the features were the clear path to growing revenue. However, the unseen part that he couldn’t see that were directly related to tech debt:

  • we had outages every week or two
  • users were getting issues connecting with SSE
  • AI agents were failing unreliably.

Everytime some issue happened, I had to take at least an hour (or two!) to find the issue then start in on the fix. It was a grind and it wasn’t getting better. Bugs and errors are common and can take a long time to resolve. Anything we can do to significantly cut down on the loop for discovering the issue, isolating the problem, understanding impact and fixing them will at the end of the day allow for a better user experience, and allow for more time to actually build the product.

Observability in my mind is: the holistic picture we have to understand the ongoing health of the system and finding issues. The quality of the observability is a combined measure of:

  1. How quickly a dev can find the source of an issue.
  2. Determine how widely that issue is impacting users.
  3. How proactive issues bubble up by severity vs being brought to us by the users.

One of the first things I do when joining a team is getting the lay of the land of the scope of our responsibility and what our observability looks like. Here is the tier list I’ve seen:

1. Nothing apart from the out of the box tools - entirely complaint driven

  • you’re just using the Vercel logs or kubernetes logs you’re deployed with.
  • you don’t have any alerting because how could you? You aren’t emitting exceptions
  • all your bug reports come in from your users.

2. You added Datadog with integrations (Most teams I’ve been on)

Congratulations! If technology companies exist in part to give Datadog money, you’ve made it!

Bonus

Check out hosted Signoz. Similar functionality, open source, less expensive and Open Telemetry based.

At this stage:

  • you have auto-instrumenting telemetry enabled so you get basic logs and exceptions pushed
  • you added a bunch of your integrations and their default dashboards
  • you added some basic alerts

The good:

  • you have a solid picture when an error is emitted somewhere
  • you have a framework for alerting and paging your team
  • you have a single place for all your data.
  • MCP can do a basic investigation on its own since the data is there. It may miss a lot since the full attribution of a log isn’t there.

The bad:

  • all your services are disconnected: a request from the browser is disconnected from your routes which is disconnected from your db queries which is disconnected from your redis cache which is disconnected from your job runner. Tracking down where a request went wrong for a particular user is a nightmare
  • your logs are a dumpster fire. You haven’t done any curation at all. Lots of duplication and stuff you really don’t need.
  • logs aren’t structured: it’s impossible to able to isolate errors to particular users so when you’re digging into a user, you have no idea you try to correlate based on time. Which sucks. While the search tooling is good, you can only search by free text
  • no scope of impact since you have duplication of errors everywhere - you don’t have any grouping of errors so you can’t tell how often certain issues are happening or user impact.
  • cost - dumping all your data into something like Datadog can get expensive quick, especially if you don’t get the value you were hoping for because everything is still a mess.

3. Curated Distributed traces

Someone on your team has had enough frustration debugging that they bit the bullet and they have done the work to tie together. When a user has an issue you can see the entire path:

The browser request -> nginx/vercel -> router -> business logic and jobs -> database requests

Good:

  • out of the box metrics on what is slow - since all traces have start and end times, you can now have an objective window into what is slow in your system at a granular level
  • find that error with full context - remember that SSE error I mentioned at the beginning? With traces we could see all the requests coming in but we could also see no backend services being triggered. Without connecting the two, we only saw the incoming request of the user just like everyone else. Once we could see the full path, the pattern jumped out: the affected users were all behind corporate firewalls that were blocking the SSE connections. Weeks of confusion, solved in an afternoon.
  • surface issues nobody is reporting - another out of the box win: once traces were in, I made a scatterplot of a sample of 1000 user chat sessions. 15% of users were silently timing out. Comparing the logs of the users who timed out against the ones who didn’t, we found the provider was cutting off the returned prompt midway and never streaming anything back to the user. Not a single one of those users filed a bug report. They just had a bad experience and moved on.
  • MCPs can quickly find the issues since it knows how to go up and down the trace to find the source issue to help with reproducibility to solve the issue.
  • you can create really clean dashboards monitoring the endpoints, services and SLAs that matter to you!

Bad:

  • this requires manual work to actively maintain. You have to decide the shape of your traces and what metadata you want to add to it.
  • You might have additional infrastructure that needs to be stood up: new routes for your front end observability, maybe a collector or two or take the raw logs, traces and filter them. Unless you’re a small team and you can do what you want, you’ll need to allocate time to set it up.
  • you still have a ton of data to wade through and 80% of the data is probably noise.

4. Nirvana - Observability as part of your Development

A few folks on your team are using your dashboards regularly and you’re confident your data is clean and you’re very familiar with the tool.

In advance of building a feature, you’re now thinking how you’re going to measure how many people actually use it and its performance once it’s released. Once you deploy your feature, you have a window into how it’s doing right away accelerating your ability to improve it.

Good:

  • You have a solid platform that has been battle tested and you’re likely very good at knowing exactly where you can see any reported issue
  • Your team is enabled to build really solid best practices for observability since that’s now the established pattern

Bad:

  • it’s taken a lot of work to get here and your org needs to stay on top of it. Some folks will be lazy and not implement best practices creating blind spots when YOU have to dig in to debug an issue.
  • you’ll need an approach for new service. Whenever you add an additional technology layer into your path, that needs to be part of your distributed trace and that takes effort and may not be trivial to figure out.

Was it worth it?

Back to my VP. He wasn’t wrong that features grow the business, but the framing of observability vs feature work turned out to be a false choice. The SSE issue that had been dragging on for weeks was solved in an afternoon once we had traces. The silent timeouts hitting 15% of chat sessions? No amount of feature work would have surfaced that, because not a single affected user told us. They just quietly had a bad experience.

Every hour spent digging through disconnected logs is an hour not spent building. Every issue you can’t see is a user you might be losing. Investing in observability didn’t slow down our feature work. It’s what gave us the time and the confidence to do it.

If you’re at tier 1 or 2 today, the jump to curated distributed traces is the single biggest unlock. It’s real work, but it pays for itself the first time a “weird intermittent issue” turns into a five minute investigation instead of a multi-week mystery.

One of my favourite things with writing these posts is that there are always a few folks who reach out and I get to connect with. Anything here fun?

Hit me up! jevin@quickjack.ca.