Data Observability is dead, Long live Data Operations! Part 1

Data Observability is dead, Long live Data Operations! Part 1

March 5, 2024

Table of contents

Data observability is dead.

Now that I have your attention! 

The obvious: Cloud data warehouses (CDWs) are everywhere and here to stay

In 2023, when Snowflake and Databricks hosted their annual conferences on exactly the same days, the impact on the data community was felt across the board, presenting a classic FOMO dilemma: which one could I not afford to miss? And the choice truly mattered because no one wanted to be left behind on the revolutionary announcements from either company.

With CDWs now rapidly evolving beyond structured data analytics into the world of AI and GenAI, one thing is unequivocally clear: a sound CDW strategy is a must-have for every company, whether an enterprise or SMB, across all sizes and sectors. Building on this...

Contrarian view - data observability is dead

Therefore it’s no surprise that the category of data observability over CDWs has been getting increasing attention in the past 2-3 years from the data community riding the waves of CDW adoption. Earlier, we had seen the highly successful evolution of the APMs (Datadog, and alike) on the tailwinds of cloud adoption.

So why now take a contrarian view for data observability and call the category as dead? 
This point of view stems directly from the industry’s adaptation of the terminology of data observability both from the vantage point of adopters i.e, data leaders who are looking to embrace data observability for their own organization, as well as from the strictly limited scoped offerings from providers aka data observability vendors who have built solutions for this space.

There are 3 fundamental tests that this category fails when it comes to “real” adoption.

  • Lack of clear value proposition. In our discussions with 100+ data leaders over the past two years about what “value” they are looking for from a data observability offering, we found responses that varied all across the board. On the other hand, the value a CDW brings to enterprises is unequivocally clear. So is there a repeatable value here?
  • Incompleteness. Data observability as defined by vendors is today synonymous with data quality. This limited scope doesn’t work when it comes to the larger needs of adopters (data engineering team) who are trying to make sense of their data investment.
  • Absence of automation first approach. Even when scoped to data quality, data observability vendors have taken a manual-first approach; they put the onus on data teams to select what, why, when and how to monitor. What follows post demo, long deployment cycles is little or no adoption, specifically as success is then strongly predicated on the rarest commodity: the data team’s constant time and attention

To contrast, my co-founder Shashank previously spearheaded the area of data quality at Meta where he brought the power of automation-first (1.5M + tables, 10K+ data practitioners). Impact: With automation, coverage for data quality went up from single digits (~7%) to where the entire CDW was getting monitored for data quality. That's a gem, isn’t it!

And we have not even gotten to the larger problem  - yes, data observability when it can, it calls out the issue, but what next?

Test one: Data observability - is it a rope or a trunk …? 

Lack of convergence on a clear value proposition of data observability.

Yes, data observability is a relatively nascent area and came into prominence in the past 2-3 years. In our discussions with 100+ data leaders, we found responses that were all across the board. Some got into data quality issues, some talked about the need for right semantics or normalization across data sources, others got more into lineage (too transactional), or monitoring data flow and more, the list just got bigger. 

Transform your data observability experience with Revefi
Get started for free

TL;DR - What I found missing was convergence from the leaders who were in charge of decision making on a clear crisp value proposition of such a “magical” data observability offering. This ambiguity around why they would want a data observability solution is a huge red flag in itself. Is the problem this category has been trying too hard to solve too vague, is it even repeatable?

Data is a complex ecosystem; even when value propositions are clear (who, why, what), take a BI system as an example, adoption tends to be a long grueling process. However for data observability this problem was clearly compounded - in the absence of knowing the purpose, it's hard to imagine what success looks like for such offerings.

It was no surprise that after spending months in deployment with the current offerings in the data observability category, we found their customers very soon questioning the adoption and ROI that they got from such offerings.

Test two: Focusing on just data quality doesn’t cut it!

For a data leader the strategic question is whether they are investing “right” on their data stack (what's the ROI they are getting from their overall data investment) and how, where they should continue to invest in the future. It's a cliche that every organization wants to have high data adoption, and be on the cutting edge of using data to drive key business decisions. It goes without saying that data quality is key to having trust on data which is a must-have for data adoption. 

Getting the right data at the right time for businesses is imperative to trust and adoption, but where is the critical part around getting this data at the right cost? Especially with the consumption based pricing models that are underpinnings of CDWs.

Passively following a Gmail model of continuing to accumulate data in a CDW without having an exit strategy for the data doesn’t work; the impact on dollars and team’s time is real and huge. 

The cost angle has now been well established in the industry where data leaders are frantically trying to rebalance their budgets as CDW costs are spiraling out of control over time. 

Question: is another data product now needed to manage data spend? To me, any data that no longer (or perhaps never) serves a purpose has no business of being there in the first place. Having yet another “silo” (spend management) offering is not the right approach.

The approach of focusing on fixing the data quality issues only without having a complete picture of the purpose and cost of having the data over which quality issues were raised in the first place falls remarkably short and in fact establishes the wrong data culture. More often than not, the right way to fix a data quality issue is to eliminate the data and save time, dollars and compliance headaches down the road, and this precisely is the need of data teams.

Test three: Automation is the foundation to 100% coverage. No exceptions here!

Are current data observability offerings from the current vendors really delivering on the promise of saving team’s time when it comes to reducing data related escalations?

A FinTech company shared with us that it took them over a year to get their data observability deployed in their company. Question: what’s the ROI? Response: that is yet to be seen!

There are two larger challenges with the current offerings in the data observability category that stems from their primarily manual-first approach. First, long deployment cycles. It's a paradox if a system that is supposed to save teams’ time and efforts itself takes weeks, months to deploy. 

As a data leader, this begs home the question - was the problem even that critical to begin with, what was the ROI of the investment, what's the basis for renewal.

Second, who takes on the hard work of selecting what, how and when to monitor? Does the offering provide a pretty interface but leaves the data teams the tough task of defining the checks, finding and maintaining thresholds, or does the offering really give data teams the peace of mind that everything that's there now, what comes in future is going to be monitored automatically without requiring any active supervision.  

We know that no team has the time and energy to continue to define and manage such data quality rules for the entire warehouse - whether the offering is packaged as a low code system, a pretty UX, or tests in dbt or whether its SQL directly. Time of data teams is best spent bringing new data, new insights to advance the business and not chasing issues or managing data observability systems.

Unless the data observability offering has its DNA in automation-first, and automation at scale approach, it's a recipe for poor adoption, and onto the path of being a shelfware.

On the lighter side, to get a perspective on how industry has advanced when customers were spending months in deploying data observability solutions at their end, in the past 12+ months, the whole tech world has been fundamentally transformed with GenAI + LLM. Fun question - can Nvidia’s market cap beat Apple!!

Stay tuned for part two: Long live Data Operations!

Healthy data adoption falls within the purview of data engineering, it's not one problem. Instead of creating more silos, and continuing to layer more observability above such silos:

                         It's time we cater to the persona and not a problem!

Sanjay Agrawal

Black-and-yellow aim icon