AI Crawlers and Scrapers Are Contributing to an 86% Increase in General Invalid Traffic

New research from the DV Fraud Lab revealed that in late 2024, the internet saw a major surge in general invalid traffic (GIVT) rates — nearly doubling with an 86 percent year-over-year increase in the second half of 2024, to be exact. The fourth quarter of 2024 is also the first time in history we’ve seen monthly GIVT volumes reach over 2 billion ad requests. Our data shows that this growth pattern was sustained throughout H2 2024, with GIVT rates growing by roughly 70 percent in December 2024 compared with December 2023. 

So what’s driving this growth? DV data found that the increase is strongly correlated with the proliferation of AI-powered crawlers and scrapers that collect and analyze web content. While GIVT is not inherently malicious like sophisticated invalid traffic (SIVT) and industry standards and regulations protect advertisers against wasted ad spend from GIVT, the results can be harmful to accurate measurement. Therefore, it’s important for advertisers to measure and avoid both GIVT and SIVT to improve campaign performance and maximize budget allocation.

Let’s take a look at why GIVT deserves more attention, how generative AI has fueled its growth and what steps the industry can take to mitigate its impact.

Understanding GIVT: Why It Matters

GIVT is invalid traffic that can be identified using routine means of filtration, such as search engine crawlers, creative auditing bots, stress testing bots, AI scrapers and other automated tools used for legitimate purposes. Unlike SIVT, which involves fraudulent activities like ad stacking or domain spoofing, GIVT typically lacks malicious intent. In fact, the Interactive Advertising Bureau (IAB) and the Media Rating Council (MRC) ensure robust industry standards and regulations are in place to protect advertisers from wasted ad spend. The MRC establishes guidelines that define standards for identifying and filtering invalid traffic, including GIVT, and the IAB provides detailed documentation outlining how to implement these standards. Publishers, platforms and verification vendors like DV are mandated to comply with these guidelines. As a result, if an ad were to serve to a GIVT bot, the industry standard is that this traffic be removed from reporting and advertisers not charged for that impression.   

However, if left unchecked, GIVT can distort campaign metrics, inflate impression counts and raise concerns about discrepancies. This makes its accurate detection and filtration essential for maintaining trust and transparency in digital advertising. DV offers solutions like pre-bid filtering and post-bid blocking to help advertisers manage this traffic, even though it is excluded from billable activity.

How Generative AI Is Driving GIVT

The rapid adoption of generative AI tools in recent years has led to a notable rise in AI-driven crawlers and scrapers. These tools, designed to simulate user behavior, are widely used by businesses, research institutions and even independent developers to extract valuable data for purposes like training machine learning models and gathering market insights. These activities can inflate traffic metrics by generating impressions that don’t represent genuine human engagement and therefore contribute to the increase in GIVT.  

In fact, DV data showed that a record 16 percent of GIVT from known-bot impressions in 2024 were generated by those that are associated with AI scrapers, such as GPTBot, ClaudeBot and AppleBot. Some GIVT bots declare themselves as fully devoted to the purpose of training AI, such as Meta AI bot and AppleBot, while other crawlers serve a mix of purposes (e.g. GoogleBot, which represents both search and AI). This distinction plays into the relative share that various GIVT bots have in the overall increase (for example, the current impact of AppleBot is impacted by some publishers’ decision to opt out of scraping). 

The following chart shows the relative portion of major crawlers and scrapers that drove GIVT in December 2024, measured on known-bot impressions:

Below, we see the overall increase in AI-based scraping in 2024, which amplifies the need for advertisers to measure and account for GIVT in their campaigns. 

Beyond GIVT: Tackling the Rise of Evasive Crawlers

Alongside transparent, self-declared scrapers that contributed to the rise of GIVT in 2024, there were also more evasive scrapers operated by individuals/organizations who intentionally disguised them as human visitors. DV continuously identifies non-transparent scrapers and crawlers based on their behavior and digital telemetry. False impressions from these scrapers may be categorized as SIVT bot fraud, often referred to as the Scorpio fraud scheme by the DV Fraud Lab.

The charts below illustrate the H2 2024 trends in SIVT and GIVT, with a focus on traffic driven by crawlers and scrapers. These charts demonstrate how both SIVT crawlers (associated with the Scorpio fraud scheme) and GIVT crawlers (operating from headless browsers) increased as the U.S. election approached. Their activity continued to rise in Q4, showing spikes on major U.S. shopping holidays — behavior typical of both benign and malicious crawler/scraper traffic. In the case of SIVT bots, such as those in the Scorpio scheme, Retail Media Networks may also experience traffic impact, as these undeclared bots are harder to avoid.

The chart below shows that SIVT fraud spikes from the Scorpio crawler bot were observed on the dates of the U.S. election.

Here, we see similar spikes observed in GIVT towards the election:

How DV Audits, Measures and Blocks GIVT

At DV, we offer a comprehensive suite of solutions to tackle GIVT both before and after the bidding process. DV is accredited by the MRC for pre-bid IVT detection on all of its integration partners, and for post-bid GIVT and SIVT prevention and measurement on desktop, mobile web, mobile apps and CTV environments. As part of that accreditation, DV participates in annual audits by an independent third party. Here are some of the key steps we take:

Leveraging TAG and IAB Lists

DV actively uses the TAG Data Center List, IAB Known Browsers and IAB Spiders and Bots List, which are widely accepted as a standardized resource for identifying GIVT. As a contributor and collaborator with the teams and Tech Lab of TAG and the IAB, DV helps to enhance its accuracy and relevance, reflecting the latest trends in automated traffic.

Maintaining Proprietary Filtration Methods

In addition to the IAB list, DV applies its own proprietary GIVT filtration methods, built from years of data collection and analysis. This methodology complements our SIVT detection processes, ensuring comprehensive coverage of all types of invalid traffic.

Offering Pre- and Post-Bid Solutions

DV offers a range of solutions to address GIVT both pre-bid and post-bid. 

Post-bid, our tools provide detailed reporting and insights, enabling advertisers to identify and mitigate the impact of GIVT on their campaigns. Every client currently engaged with DV for IVT receives GIVT reporting post-bid. 

Tracking GIVT post-bid through monitoring (Example Screenshot from DV’s Programmatic Analytics UI for Platforms)

While post-bid measurement provides valuable insights, it does not enable dynamic optimization of spend before the purchase. Some verification vendors only offer channel-limited post-bid measurement, which means they cannot proactively optimize ad delivery. Increasingly, sophisticated advertisers seek to reduce waste by avoiding invalid impressions and unsuitable content through pre-bid solutions across channels.

DV set the industry standard as the first verification company to offer pre-bid avoidance for GIVT, helping advertisers avoid GIVT wherever supported by media-buying platforms. As the MRC requires that GIVT be removed from monetized counts and metrics and as part of our compliance with industry guidelines, DV provides advertisers with related disclosure reporting for troubleshooting any discrepancy. The following is a screenshot of DV’s disclosure reports section on GIVT for monitoring and blocking: 

Post-bid GIVT blocking, which is above and beyond MRC requirements, offers additional flexibility. These solutions provide an additional layer of defense for advertisers who prefer to avoid the majority of this traffic, thereby optimizing ad-serving costs and/or eliminating fees from third-party measurement providers. While some GIVT cannot be immediately avoided due to the constant emergence of new crawlers and bots (that may outpace the IAB’s list), our pre-bid services work to prevent the majority of such traffic before bidding occurs.

By leveraging both pre- and post-bid solutions, DV empowers advertisers to maximize their campaign efficiency and effectiveness, ensuring that their ad spend is directed towards genuine, high-quality impressions.

Additional Considerations for Advertisers

By using DV services, which automatically include all industry-standard resources, GIVT can be avoided. However, advertisers must also consider the following.

1. Optimize Your Campaign Settings

While DV provides robust capabilities to avoid GIVT, it’s crucial for you to actively review and optimize your campaign settings. Ensure that GIVT and SIVT blocking is activated where applicable. Not all campaigns are set up to block GIVT pre-bid, even when the option is available. 

2. Understand Platform Limitations

Be aware that some media-buying platforms may not currently support pre-bid avoidance of GIVT. Additionally, while blocking and filtering are available through DV’s integrated partners, the effectiveness of GIVT avoidance depends on the capabilities of each media-buying platform. Platforms that support user-agent lookups are better equipped to preemptively filter GIVT. Demand-side platforms (DSPs) are responsible for following the integration specifications provided by DV and adhering to the defined latency and logic requirements. If a DSP fails to ingest DV data promptly, implements the logic incorrectly, or does not adopt updated logic specifications, it impacts the accuracy of pre-bid avoidance.

The user-agent string provides essential details about the browser, operating system and device making the ad request, which helps identify invalid traffic. If a platform lacks user-agent lookup functionality or does not prioritize it, certain types of GIVT can evade detection. While some platforms achieve sufficient efficacy by relying on ADIDs (Advertising IDs) and IP-only filtering, adding user-agent lookups strengthens pre-bid filtering and ensures more robust protection against GIVT.

Advertisers should proactively inquire with their ad tech partners about these limitations and prioritize working with platforms that fully support pre-bid GIVT avoidance. 

3. Address GIVT Challenges with Inclusion Lists

Some media-buying partners may advocate for an inclusion-list-only approach or rely primarily on post-bid measurement methods like log file analysis. These methods fail to proactively address GIVT, especially as new forms emerge in real time. Even on reputable publishers within an inclusion list, GIVT remains a key part of invalid traffic. While the IAB and TAG lists are valuable resources, they don’t cover the entire spectrum of GIVT bots. This highlights the importance of DV’s proprietary GIVT identification and our pre-bid capabilities that enable advertisers to block GIVT proactively, providing an additional layer of protection.

The Path Forward

As the digital landscape evolves, so too do the challenges associated with invalid traffic. The rise of AI-powered bots and scrapers highlights the need for ongoing vigilance and innovation. DV remains committed to providing advertisers with the tools and insights needed to manage GIVT effectively, ensuring that every campaign delivers accurate, trustworthy results.

This piece is part of DV’s Transparency Center, a dedicated portal designed to educate the industry about DV technology and measurement. By providing detailed explanations, insights and timely statements on key issues, we aim to foster trust and transparency within the digital advertising ecosystem.