Data Reliability: Maintaining high-quality data across an extensive environment.
Operational Overhead: Reducing manual intervention while ensuring real-time visibility into data quality and pipeline health.
Cost Efficiency: Controlling escalating data warehouse costs.
Real-Time Monitoring: Quickly detecting issues like delayed data loads, schema changes, and performance bottlenecks.
Results at a glance
Cost Savings: Achieved warehouse cost reductions of up to 60% savings.
Spend Optimization: Projected annualized ROI exceeding 100%.
Automated Data Observability: Deployed over 665,000 monitors that comprehensively tracked data freshness, schema changes, and query performance.
Operational Efficiency: Automated, zero-touch approach and integrated notifications minimized manual oversight and accelerated response times.
Introduction
In November 2024, Verisk, a leader in data analytics, began a strategic transformation of its data operations by partnering with Revefi. The goal was to deliver reliable, timely, and cost-efficient data while leveraging automation and intelligent insights to enhance overall performance and reduce operational burdens.
The Challenge
Verisk faced several pressing issues within its data ecosystem:
Data Integrity & Visibility: Ensuring accurate, fresh data across a vast number of tables and queries.
Operational Complexity: Managing detailed monitoring requirements with minimal manual intervention.
Cost Management: Tackling rising costs in data warehousing, specifically in terms of Snowflake query performance and storage inefficiencies.
Timely Detection: Rapid identification of issues like late or missing data, unexpected schema changes, and performance slowdowns was essential for maintaining pipeline health.
Revefi Implementation and Results
Verisk deployed the Revefi AI Data Engineer with key capabilities to address challenges.
Automated Warehouse Management: Dynamically adjusting warehouse sizes and scaling based on real-time performance metrics.
Advanced Monitoring: Continuously tracking data operations across 130,000+ tables, 7 trillion rows, and 10 million queries per month.
Integrated Alerting: Using Microsoft Teams for real-time notifications on data load statuses, freshness, schema changes, and query failures.
Zero-Touch Deployment: Minimized manual efforts and streamlined integration into existing workflows.
The partnership with Revefi led to significant improvements:
Cost Efficiency: Realized average savings of 18-20% on warehouse costs, with some environments achieving up to 60% savings, leading to a projected ROI of over 100%.
Enhanced Data Observability: Comprehensive monitoring enabled proactive identification and resolution of data quality issues.
Improved Operational Agility: The zero-touch approach allowed engineering teams to focus on strategic initiatives while reducing manual monitoring efforts.
After turning on Automated Warehouse Management, Verisk gained the ability to dynamically adjust warehouse sizes and scale based on real-time performance metrics without any human involvement. As shown in Figure 1 below, with this Snowflake warehouse, Revefi identified savings of $36,004, i.e., 60% savings even though the number of queries executed increased by 44% !
Figure 01: Warehouse savings example
The reduction in credits consumed after turning on (“Report Applied”) Automated Warehouse Management is shown in Figure 2:
Figure 02: Reduced Snowflake Credit Consumption
Figure 03: Automated Data Observability Monitors
Conclusion
Verisk’s collaboration with Revefi demonstrates how automation and comprehensive data observability can transform data management practices. By addressing critical challenges in data quality, operational overhead, and cost management, Revefi’s AI Data Engineer enabled Verisk to achieve significant cost savings and enhanced operational performance.
Key Outcomes
60%
Reduce Snowflake Cost by 60%
After turning on Automated Warehouse Management, Verisk gained the ability to dynamically adjust warehouse sizes and scale based on real-time performance metrics without any human involvement.
Case studies
