Case Study: How Hourglass Cosmetics Validated Meta's True Impact with a Geo Holdout Test

Measuring the real impact of paid social is especially challenging for beauty brands, given brand strength and omnichannel distribution models. Platform-reported metrics often tell one story, but what's the actual incremental lift?

For Hourglass Cosmetics, a luxury beauty brand with significant retail presence through Sephora and other channels, this question was critical — especially when ecommerce represents just a fraction of total revenue.

Working with DV Rockerbox™'s Professional Services team, Hourglass designed and executed a rigorous geo holdout test to measure Meta's true incremental impact on both overall conversions and new customer acquisition. The results provided clear, statistically significant evidence of Meta's impact and validated their investment in the channel.

The Challenge: Understanding Meta's Impact in a Multi-Channel Retail Business

Hourglass came to DV Rockerbox with a specific question: How does Meta advertising affect incremental sales and new customer sales, and how can we measure this impact rigorously?

This type of incrementality testing is especially important for brands like Hourglass, where:

• Attribution Is Complex: Customers interact with multiple touchpoints across paid social, search, email and retail before converting.
• Budget Decisions Require Confidence: With Meta representing one of the largest spend channels, understanding true incremental lift is essential for optimization.
• Previous Testing Showed Promise: A successful geo test during the Canadian market launch had demonstrated positive incremental lift, building confidence in the methodology.

The Approach: Collaborative Test Planning and Design

The path to a successful test began with collaborative planning between Hourglass, its media agency and DV Rockerbox's Professional Services team. The team created a test design using a 10 DMA holdout to measure Meta's overall impact across the country. The ten DMAs selected had, in aggregate, very strong relationships to the control markets, with correlations of 0.96-0.98 for new customer conversions and revenue, and 0.97 to 0.99 for all customer conversions and revenue, resulting in a very high-powered test.

The test ran for six weeks from May 5 to June 16, 2025, during which time all Meta campaigns were paused in the 10 holdout DMAs. DV Rockerbox analyzed Meta's impact during the test using ecommerce transaction data collected directly from the Hourglass site via DV Rockerbox pixels.

The Results: Clear Incremental Lift Across Key Metrics

The geo holdout test revealed statistically significant lift for both overall performance and new customer acquisition.

 During the test period, the results showed:

• Incremental CPA: $130 (10.4% lift at 99% confidence)
• Incremental ROAS: 0.51 (7.9% lift at 90% confidence)

For new customer acquisition specifically, the results were equally compelling:

• Incremental new customer CPA: $190 (10.6% lift at 95% confidence)
• Incremental new customer ROAS: 0.35 (8.6% lift at 90% confidence)

Given that ecommerce is just one of Hourglass's sales channels, the full omnichannel lift from Meta is likely several times higher and above the brand's breakeven threshold.

What This Means: Validated Investment and Strategic Clarity

The geo holdout test provided Hourglass with several critical insights:

1. Meta Drives Measurable Incremental Value

The test proved that Meta advertising generates real incremental conversions and revenue beyond what would occur organically. The 10%+ lift in conversions shows that Meta isn't simply taking attribution credit for conversions that would have happened anyway.

2. New Customer Acquisition is Working

With a 10.6% lift in new customer conversions and a prospecting-specific iCPA of $190, Hourglass validated that its Meta prospecting strategy is effectively reaching and converting new audiences.

3. ecommerce Results Are Just the Beginning

Important context: These results measure ecommerce impact only. Hourglass's Sephora retail revenue is about 8x its ecommerce revenue, meaning the incremental lift observed in ecommerce likely represents only a fraction of Meta's total business impact.

While the ecommerce-only iROAS of 0.51 might appear modest, if Meta drives similar behavior in retail (which is highly likely given the halo effect of digital advertising), the true total iROAS could be 4-5 when retail impact is included.

DV Rockerbox can directly analyze the impact of media on sales in third-party retailers such as Sephora or Amazon when the data is made available. Store-level sales data for retail chains makes an ideal data foundation for testing, and with access to that information, DV Rockerbox could have proven the above hypothesis and understood the impact of media on both ecommerce and Sephora revenue.

4. Statistical Rigor Builds Confidence

The high pre-test correlation (0.96-0.99) and strong statistical significance (90-99% confidence) mean these results are highly reliable. This level of rigor gives the marketing team confidence to make strategic budget decisions based on the findings.

The DV Rockerbox Advantage: First-Party Data Meets Rigorous Testing

This test showcases the power of combining first-party measurement with rigorous incrementality testing:

• DV Rockerbox's ecommerce conversion data provided a clean, unbiased view of performance across DMAs.
• Robust pre-test planning ensured the test was powerful prior to launch.
• Statistical rigor delivered results the team could trust for strategic decision-making.

For brands looking to move beyond platform-reported metrics and understand true incremental impact, geo holdout testing offers a gold-standard approach — and DV Rockerbox makes it accessible and actionable.

Want to validate your channel performance with rigorous incrementality testing? DV Rockerbox's platform makes it easy to design, execute and analyze geo holdout tests that deliver statistically significant insights.