Understanding URLScan, Headless Browsers, and DV’s Detection Methods

In recent industry conversations about bot detection and ad verification, URLScan has emerged as a frequently mentioned tool. But what is URLScan, and what does it mean for advertisers?

What is URLScan?

URLScan is a web analysis tool that security researchers, developers, and analysts use to examine website behavior, analyze headers, and review third-party calls. It’s a tool for analysis, and is not inherently malicious.

Is URLScan a User-Agent Bot?

The answer depends on its setting. URLScan allows users to initiate crawls from several types of user-agents, as seen in the screenshot below. If someone chooses the “Googlebot” or “Bingbot” option, then URLScan bots would be detected as known bots, according to the IAB list of known bot User-Agents. 

However, the vast majority of URLScan crawls refrain from the self-declared option. In fact, URLScan defaults to the “Latest Google Chrome,” making it more evasive by design. 

Does URLScan Use a Headless Browser?

Yes, URLScan uses headless browser technology to fully render pages, including scripts, tracking pixels, and ad calls, without a visible interface.

What is a Headless Browser, and Why Does It Matter?

A headless browser operates like a standard browser but lacks a graphical user interface (GUI). This allows automated systems to interact with web content—executing JavaScript, rendering images, and triggering ad requests—essentially mimicking the behavior of human viewers. 

The DV Fraud Lab accurately detects headless browser activity and reports it according to its context. Benign usage of headless browsers is flagged as general invalid traffic (GIVT), while more evasive headless browsers (paired with other signals for manipulation or evasion) can be reported as SIVT. In most cases, since the vast majority of URLScan crawls, specifically, do not opt to self-declare, we often identify it as sophisticated invalid traffic (SIVT).

DV has publicly discussed its innovative tech for identifying various flavors of headless browsers on several occasions, like at conferences dating back to 2022, and more recently in DV’s transparency center GIVT blogpost. 

Is URLScan Easy to Detect?

No, it’s not. According to bot research expert Antoine Vastel, headless browser bots like URLScan are “increasingly” difficult to detect due to their sophisticated evasion tactics. Similarly, in an exchange between a DV representative and URLScan, we asked: 

“Do you see [URLScan] as something that should be self-declared when interacting with websites or ad systems? Is URLScan ‘easy-to-catch’ or is it more complex?”

URLScan’s CEO, Johannes Gilger, responded, “Our scanner does not announce itself, that would defeat the purpose of the tool. Instead, it will look like a regular web-browser.”

With that said, of the traffic DV analyzes, headless browser GIVT impressions are a fraction of a percent and are effectively identified post-bid—preventing advertisers from paying for these impressions. It is industry standard to remove GIVT from monetizable impression counts. 

How Does DV Detect URLScan?

Due to its evasive nature, DV may classify some URLScan traffic as SIVT, not just GIVT. DV has developed world-class technology and expertise to detect and flag this type of bot traffic, and others like it, because their evasiveness and user-agent limitations often prevent pre-bid detection. Again, DV’s systems and experts identify URLScan’s signals during post-bid analysis and remove the resulting impressions from billable counts shared with customers through DV GIVT disclosure reporting. 

Ultimately, DV might detect, identify, and manage URLScan in a variety of ways. However, regardless of the method or process, DV accurately detects URLScan pre- (if permitted) and post-bid (if not permitted pre-bid).

Why Pre-Bid Protection Still Matters

While certain evasive bots like URLScan may be challenging to block outright pre-bid, this doesn’t diminish the critical importance of pre-bid prevention. Pre-bid filtering remains the most effective way to prevent invalid traffic from ever reaching a publisher’s page, minimizing wasted media spend and reducing the risk of exposure to fraudulent or unsuitable environments.

When pre-bid avoidance is enabled, GIVT from known (self-declared) bots appears on less than 0.03% of programmatic impressions. Even then, DV identifies these bots post-bid in virtually all cases. This represents a 75% to 98% reduction—depending on client settings—compared to unfiltered programmatic impressions, which proves that GIVT pre-bid avoidance is highly effective, especially at reducing the need for post-bid reconciliation. 

Advertisers should prioritize working with verification partners that offer robust pre-bid bot detection and filtration. Pre-bid technology provides the first line of defense—ensuring that most GIVT is stopped before an ad is served. Combined with post-bid analysis for GIVT edge cases like headless browser bots, pre-bid solutions help safeguard campaign performance, protect brand investments, and support a healthier digital ecosystem.

–

Gilit Saporta is Vice President of Product Management, Fraud & Quality at DoubleVerify, where she leads the company’s global fraud prevention strategy and product development. With over two decades of experience in fraud detection and cybersecurity, Gilit has held leadership roles at PayPal, Forter, and Simplex, specializing in profiling, identity analytics, and fraud intelligence. She is also the co-chair of FraudCon at CyberWeek, a leading academy and industry conference on cyber fraud.

Gilit is the author of “Practical Fraud Prevention using SQL and Python,” a widely respected guide to online fraud defense, published by O’Reilly (author royalties are donated to UNICEF). At DV, she oversees the company’s Fraud Lab and plays a key role in protecting global brands from invalid traffic and sophisticated ad fraud schemes.