200 MILLION DOMAINS CRAWLED
I found a website. Then I wondered about the network behind it.
A website you come across online might be part of a larger publisher network. Different names, different designs, different countries… So how do you find the connections between them?
I crawled approximately 200 million domains. Using a parallel .NET spider I built, I inspected websites’ ads.txt files. The advertising publisher IDs in those files allowed me to identify connections between domains.
The result was Pubatlas: a research tool where you can enter a domain, discover other websites associated with the same publisher ID, and explore their stored traffic estimates, country breakdowns, and keyword data.
| Source domains crawled | Total crawl duration | Daily average | The tool I built |
|---|---|---|---|
| ≈200 million domains | ≈20 days | ≈10 million domains / day | Parallel .NET spider |
Six domains associated with one publisher ID. The displayed monthly organic traffic estimate of 28,766,342 is the sum for the two sites with traffic data, not all six sites. Screenshot captured on 21 October 2026.
From researching a website to exploring a network
Suppose you find an interesting calculator in the search results. You look at what it does, how it is designed, and the need it addresses.
But where else is the same publisher ID being used? On other tool websites? On content projects in different languages? On domains you have never come across before?
Pubatlas lets you follow that question:
Your research can now extend beyond a single website. You can open related domains, compare traffic estimates where data is available, and explore the topics those sites cover.
That was my reason for building this project: to see the connections beyond individual websites when researching new website and content opportunities.
A shared publisher ID is a research signal. It does not, on its own, prove that the websites belong to the same person or company; shared ad management and intermediary relationships can also create this connection.
A real research path: calculator.net → publisher network → unitconverters.net
Starting point: the calculator.net record provides a path to its publisher ID and six related domains.
The network view lists calculator.net, unitconverters.net, math.net, and other associated domains together. Missing traffic data is not displayed as zero.
Next step: the unitconverters.net record shows a monthly organic traffic estimate of 2,392,247, a country breakdown, and keywords. This stored estimate was recorded on 22 September 2026. A shared publisher ID alone does not prove common ownership.
To find out, I knocked on 200 million doors
I started by purchasing a ready-made dataset containing approximately 200 million domains. I had a large list. I wanted to turn it into a structure where I could connect those domains through their advertising publisher relationships.
To do this, I built a .NET console spider that processed domains in parallel. The spider crawled their ads.txt files. The entire source list of approximately 200 million domains was crawled in around 20 days. Calculated from that scope and duration, the daily average comes to approximately 10 million domains.
Not every domain produced a usable result. Some websites were unreachable; others had no ads.txt file. The publisher relationships I was able to extract formed Pubatlas’s initial data layer.
The 200 million figure refers to the source domains crawled. It does not mean that the current database contains 200 million verified relationships.
How is the data expanded and enriched today?
The initial crawl established the foundation of connections. Two needs then emerged: are those relationships still valid, and how much traffic do the related websites receive?
Today, I develop the dataset in two ways:
- The Chrome/Edge extension I built: It collects traffic estimates, country data, and keyword data across multiple browser tabs. When a tab requires verification, it can wait for the operator while the other workers continue.
- Discovering new domains through searches within Pubatlas: If a domain I am researching is not yet in the database, the system saves it and checks its ads.txt file. When a publisher relationship can be identified, I can follow it to other websites. This means new domains join the research pool as I use the application. This step investigates the publisher relationship; traffic data is collected separately.
The first large crawl created the initial pool. The browser extension enriches that pool with traffic signals, while research within the application brings new domains and connections into view.
The technical challenges
Preventing two workers from claiming the same job
Opening tabs in parallel does not, by itself, solve job coordination. Two workers might claim the same domain, or an old tab might send a delayed result.
The server therefore gives each job an assignment ID with an expiry time. Job ownership is established through conditional database updates. When a result arrives, the assignment is checked to make sure it is still valid; results from expired assignments or assignments that have been replaced are rejected.
Recovering after an interruption
The browser extension stores results locally before sending them. If the connection drops, it can retry delivery. Jobs whose assignments expire become available to claim again.
Preserving existing data when a new request fails
A website failing to respond today does not necessarily mean its previous publisher relationship has ended. Similarly, a missing traffic or country field does not mean the previous value should be reset.
Temporary errors, incomplete results, and relationship changes are therefore handled separately. The current domain status and relationship history are stored, and invalid new results are prevented from erasing the existing snapshot.
The technology behind the interface
| Component | Technology / role |
|---|---|
| Initial large-scale crawl | Parallel .NET console spider |
| Web application | .NET 8, ASP.NET Core MVC |
| Data layer | Entity Framework Core, MySQL |
| Browser collector | JavaScript, Manifest V3 |
| Current data flow | Browser extension, API, and ads.txt checks during searches |
| Validation | Isolated integration tests and extension tests |
The research interface makes this infrastructure usable through domain and publisher searches, website lists, traffic sorting, filters, and CSV exports. Notes and favourites make it possible to return to a research session later.
Traffic figures are estimates from external sources, rather than data taken directly from the websites’ analytics. The current model stores the latest traffic snapshot. Network totals should also be interpreted in the context of which websites have data available.
One domain, new research paths
When I open a website in Pubatlas, I can move on to other websites associated with the same publisher ID and investigate the countries and topics where they have a presence.
For me, this is the value of crawling approximately 200 million domains: finding a single website becomes the starting point for a much broader investigation of the web.