Storj Cuts Their Cloud Data Warehouse Spend by $10K+ on Google's BigQuery with Revefi

“One of the top advantages that we've found from using Revefi so far is that one of the biggest things is just the peace of mind that we have that comes from knowing Revefi is watching out for unusual activity and letting us know. We are especially grateful for the cost and spend management that Revefi has offered to us.” – Jennifer Johnson, Senior Software Engineering Manager, Storj.

Storj is a cloud storage provider offering distributed nodes for enterprise-level purposes. Thanks to its data encryption and decentralized storage features, the service ensures direct and secure access to data for every business network member.

To empower its data ecosystem with more advanced storage and computing capabilities, Storj decided to create a new data warehouse on Google BigQuery. Alongside this, the company needed to keep its data lean, monitor its usage, and minimize spending on BigQuery resources.

This is exactly where Revefi’s Data Operations Cloud shines. Read how Storj benefited from the straightforward, zero-touch installation of Revefi and optimized their spending on BigQuery resources.

The Lowdown

Storj strived to minimize the cost of their BigQuery usage, making it more insightful about recurring data pipeline issues and resources consumed due to unnecessary queries. However, data engineers didn’t use any automated tools to monitor and send alerts on the data warehouse health issues. So, they primarily looked for an out-of-the-box solution with easily integrable monitors and informative notifications to get started at once.

Storj team also expected to: 

• Reduce time for data troubleshooting and debugging
• Analyze and improve datasets based on actionable insights from monitoring software
• Tackle large queries to Google BigQuery cloud data warehouse to shrink their bills.  
  ‍

The Deep Dive

The Roadblocks

The key concern for Storj data engineers was that they had to address BigQuery excessive resource consumption to align ETL processes with their budget.  Therefore, they primarily strived to implement a monitoring system that could inform effective spend management to make the most out of their cloud data warehouse use.

Before Revefi, the Storj data team was flying blind about the facts of:

• Data behavior during updates and uploads of new assets
• The disproportionate use of data arrays by certain users and applications
• The way BigQuery algorithms append and deduplicate new records in existing data sets
  ‍

The Specifics

Luckily, Storj data management set off once they’ve plugged in Revefi to copilot their cloud data warehouse maintenance. It paid off with:

• Hassle-free installation. The initial configuration didn’t take much from Storj's side, as Revefi’s data monitors and alerts were configured automatically in a couple of steps. Revefi experts curated the entire onboarding. Storj team members have started getting Revefi’s alerts right after their metadata was fully indexed – it took a single day.
• Absolute peace of mind. Automated data health alerts provided Storj data engineers with true peace of mind. Immediate notifications and reports have prevented data specialists from wasting precious time and effort on devising a custom monitoring solution.
• The improved data culture. The company built its own culture of data usage awareness with Revefi. The read and rewrite operations in the cloud data warehouse are no longer a black box. Thanks to clear and understandable prompts, the data team can pinpoint flawed waypoints of the data lifecycle.
• Spend management success. By reducing requests sent from company service accounts and applications, Storj cut their cloud data warehouse spending by $10,000. That’s truly an astonishing return from investing $9.99/mo – $299/mo into a data governance product.
• Greater confidence for future scale-ups. Thanks to complete control over cloud data warehouse spending and a 360-degree view of data quality, the Storj team can turn their BigQuery storage into a strategic tool. Now, they can align data stack productivity with business goals.
  ‍

The Outcome

Storj is happy with Revefi for establishing a comprehensive and autonomous cloud data warehouse monitoring that sets their mind on ease and unburdens them from tons of workload.

“Instead of building metrics and monitoring our data system, we can just spend time to actually work with the data. I would absolutely recommend Revefi to teams like ours. It has not only been useful in our data warehouse build-out, but I also foresee it allowing us to optimize our system and data governance going forward.” – Jennifer Johnson, Senior Software Engineering Manager, Storj.

Taking the first big step to data governance excellence and higher returns from migrating to cloud data services doesn’t take much. The Storj’s data team nailed it effortlessly with Revefi, and you can, too!
‍