Google is aware of about 300T pages on the net. It’s uncertain they crawl all of these, and no less than in keeping with some paperwork from their antitrust trial we realized they solely listed 400B. That’s round .133% of the pages they learn about, roughly 1 out of each 752 pages.
For Ahrefs, we select to retailer about 340B pages in our index as of December 2023.
At a sure level, the standard of the net turns into unhealthy. There are many spam and junk pages that simply add noise to the information with out including any worth to the index.
Giant elements of the net are additionally duplicate content material, ~60% according to Google’s Gary Illyes. Most of that is technical duplication attributable to completely different programs. Nevertheless, when you don’t account for this duplication, it might probably waste extra assets and create extra noise within the knowledge.
When constructing an index of the net, firms need to make many decisions round crawling, parsing, and indexing knowledge. Whereas there’s going to be a number of overlap between indexes, there’s additionally going to be some variations relying on every firm’s selections.
Evaluating hyperlink indexes is difficult due to all of the completely different decisions the varied instruments have made. I attempt my greatest to make some comparisons extra truthful, however even for a number of websites I’m telling you that I don’t need to put in the entire work wanted to make an correct comparability, a lot much less do it for a whole research. You’ll see why I say this later if you learn what it might take to match the information precisely.
Nevertheless, I did run some assessments on a pattern of web sites and I’ll present you methods to test the information your self. I additionally pulled some pretty giant third celebration knowledge samples for some extra validation.
Let’s dive in.
If you happen to simply checked out dashboard numbers for hyperlinks and RDs in numerous instruments you would possibly see fully various things.
For instance, right here’s what we depend in Ahrefs:
- Reside hyperlinks
- Reside RDs
- 6 months of knowledge
In Semrush, right here’s what they depend:
- Reside + useless hyperlinks
- Reside + useless RDs
- 6 months of information + a bit extra*
*By a bit extra, what I imply is that their knowledge goes again 6 months and to the beginning of the earlier month. So, as an example, if it’s the fifteenth of the month, they might even have about 6.5 months of information as an alternative of 6 months of information. If it’s the final week of the month, they might have near 7 months of information as an alternative of 6.
This will not appear to be so much, however it might probably improve the numbers proven by so much, particularly if you’re nonetheless counting useless hyperlinks and useless RDs.
I don’t assume SEOs need to see a quantity that features useless hyperlinks. I don’t see a superb cause to depend them, both, aside from to have greater and doubtlessly deceptive numbers.
I solely say this as a result of I’ve referred to as Semrush out on making this kind of biased comparability earlier than on Twitter, however I finished arguing once I realized that they actually didn’t need the comparability to be truthful; they simply needed to win the comparability.
There are some methods you’ll be able to evaluate the information to get considerably related time intervals and solely take a look at lively hyperlinks.
If you happen to filter the Semrush backlinks report for “Lively” hyperlinks, you’ll have a considerably extra correct quantity to match towards the Ahrefs dashboard quantity.
Alternatively, when you use the “Present historical past: Final 6 months” possibility within the Ahrefs backlink report, this would come with misplaced hyperlinks and be a fairer comparability to Semrush’s dashboard quantity.
Right here’s an instance of methods to get extra related knowledge:
- Semrush Dashboard: 5.1K = Ahrefs (6-month date comparability): 5.6K
- Semrush All Hyperlinks: 5.1K = Ahrefs (6-month date comparability): 5.6K
- Semrush Lively Hyperlinks: 2.9K = Ahrefs Dashboard: 3.5K = Ahrefs (no date comparability): 3.5K
What you shouldn’t evaluate is Semrush Dashboard and Ahrefs Dashboard numbers. The quantity in Semrush (5.1K) consists of useless hyperlinks. The quantity in Ahrefs (3.5K) doesn’t; it’s solely dwell hyperlinks!
Word that the time intervals will not be precisely the identical as talked about earlier than due to the additional days within the Semrush knowledge. You possibly can take a look at what day their knowledge stops and choose that precise day within the Ahrefs knowledge to get an much more correct, however nonetheless not fairly correct comparability.
I don’t assume the comparability works in any respect with bigger domains due to a problem in Semrush. Right here’s what I noticed for semrush.com:
- Semrush Dashboard: 48.7M = Ahrefs (6 month date comparability): 24.7M
- Semrush All Hyperlinks: 48.7M = Ahrefs (6 month date comparability): 24.7M
- Semrush Lively Hyperlinks: 1.8M = Ahrefs Dashboard: 15.9M = Ahrefs (no date comparability): 15.9M
In order that’s 1.8M lively hyperlinks in Semrush vs 15.9M lively in Ahrefs. However as I stated, I don’t assume it is a truthful comparability. Semrush appears to have a problem with bigger websites. There’s a warning in Semrush that claims, “Because of the measurement of the analyzed area, solely essentially the most related hyperlinks shall be proven.” It’s attainable they’re not exhibiting all of the hyperlinks, however that is suspicious as a result of they are going to present the whole for all hyperlinks which is a bigger quantity, and I can filter these in different methods.
I may type usually by the oldest final seen date and see all of the hyperlinks, however once I do final seen + lively, I see solely 608K hyperlinks. I can’t get greater than 50k rows of their system to analyze this additional, however one thing is fishy right here.
Extra hyperlink variations
The above comparability wouldn’t be sufficient to make an correct comparability. There are nonetheless various variations and issues that make any kind of comparability troublesome.
This tweet is as related because the day I wrote it:
It’s virtually unimaginable to do a good hyperlink comparability
Right here’s how we count links, but it’s worth mentioning that each tool counts links in different ways.
To recap some of the main points, here are some things we do:
- We store some links inserted with JavaScript, no one else does this. We render ~250M pages a day.
- We have a canonicalization system in place that others may not, which means we shouldn’t count as many duplicates as others do.
- Our crawler tries to be intelligent about what to prioritize for crawling to avoid spam and things like infinite crawl paths.
- We count one link per page, others may count multiple links per page.
These differences make a fair link comparison nearly impossible to do.
How to see where the biggest link differences are
The easiest way to see the biggest discrepancies in link totals is to go to the Referring Domains reports in the tools and sort by the number of links. You can use the dropdowns to see what kinds of issues each index may have with overcounting some links. In many cases, you’re likely to see millions of links from the same site for some of the reasons mentioned above.
For example, when I looked in Semrush I found blogspot links that they claimed to have recently checked, but these are showing 404 when I visit them. Semrush still counts them for some reason. I saw this issue on multiple domains I checked. This is one of those pages:
Lots of links counted as live are actually dead
Seeing the dead link above counted in the total made me want to check how many dead links were in each index. I ran crawls on the list of the most recent live links in each tool to see how many were actually still live.
For Semrush, 49.6% of the links they said were live were actually dead. Some churn is expected as the web changes, but half the links in 6 months indicates that a lot of these may be on the spammier part of the web that isn’t as stable or they’re not re-crawling the links often. For some context, the same number for Ahrefs came back as 17.2% dead.
It’s going to get more complicated to compare these numbers
Ahrefs recently added a filter for “Best links” which you can configure to filter out noise. For instance, if you want to remove all blogspot.com blogs from the report, you can add a filter for it.
This means you’ll only see links you consider important in the reports. This can also be applied to the main dashboard numbers and charts now. If the filter is active, people will see different numbers depending on their settings.
You would think this is straightforward, but it’s not.
Solving for all the issues is a lot of work
There are a lot of different things you’d have to solve for here:
- The extra days in Semrush’s data that you’ll have to remove or add to the Ahrefs number.
- Remember that Semrush also includes dead RDs in their dashboard numbers. So you need to filter their RD report to just “Active” to get the live ones.
- Remember that half the links in the test of Semrush live data were actually dead, so I would suspect that a number of the RDs are actually lost as well. You could possibly look for domains with low link counts and just crawl the listed links from those to remove most of the dead ones.
- After all that, you’re still going to need to strip the domains down to the root domain only to account for the differences in what each tool may be counting as a domain.
What is a domain?
Ahrefs currently shows 206.3M RDs in our database and Semrush shows 1.6B. Domains are being counted in extremely different ways between the tools.
According to the major sources who look at these kinds of things, the number of domains on the internet seems to be between 269M–359M and the variety of web sites between 1.1B–1.5B, with 191M–200M of them being lively.
Semrush’s variety of RDs is greater than the variety of domains that exist.
I imagine Semrush could also be complicated completely different phrases. Their numbers match pretty carefully with the variety of web sites on the web, however that’s not the identical because the variety of domains. Plus, lots of these web sites aren’t even dwell.
It’s going to get extra sophisticated to match these numbers
A part of our course of is dropping spam domains, and we additionally deal with some subdomains as completely different domains. We come up near the numbers from different third celebration research for the variety of lively web sites and domains, whereas Semrush appears to come back in nearer to the whole variety of web sites (together with inactive ones).
We’re going to simplify our methodology quickly in order that one area is definitely only one area. That is going to make our RD numbers go down, however be extra correct to what individuals truly contemplate a website. It’s additionally going to make for a fair greater disparity within the numbers between the instruments.
I ran some high quality checks for each the first-seen and last-seen hyperlink knowledge. On each web site I checked, Ahrefs picked up extra hyperlinks first and up to date the hyperlinks extra just lately than Semrush. Don’t simply imagine me, although; test for your self.
Evaluating that is biased irrespective of the way you take a look at it as a result of our knowledge is extra granular and consists of the hours and minutes as an alternative of simply the day. Leaving the hours and minutes creates a biased comparability, and so does eradicating it. You’ll need to match the URLs and test which date is first or if there’s a tie after which depend the totals. There shall be some completely different hyperlinks in every dataset, so that you’ll have to do the lookups on every set of information for comparability.
Semrush declare,s “We replace the backlinks knowledge within the interface each quarter-hour.”
Ahrefs claims, “The world’s largest index of dwell backlinks, up to date with recent knowledge each 15–half-hour.”
I pulled knowledge on the similar time from each instruments to see when the most recent hyperlinks for some in style web sites have been discovered. Right here’s a abstract desk:
| Area | Ahrefs Newest | Semrush newest |
|---|---|---|
| semrush.com | 3 minutes in the past | 7 days in the past |
| ahrefs.com | 2 minutes in the past | 5 days in the past |
| hubspot.com | 0 minutes in the past | 9 days in the past |
| foxnews.com | 1 minute in the past | 12 days in the past |
| cnn.com | 0 minutes in the past | 13 days in the past |
| amazon.com | 0 minutes in the past | 6 days in the past |
That doesn’t appear recent in any respect. Their 15-minute replace declare appears fairly doubtful to me with so many web sites not having updates for a lot of days.
Don’t simply belief me, although; I encourage you to test some web sites your self. Go into the backlinks experiences in each instruments and type by final seen. Be sure you share your outcomes on social media.
Ahrefs now receives knowledge from IndexNow
It will make our knowledge even brisker. That’s ~2.5B URLs / day in March 2024. The web sites inform us about new pages, deleted pages, or any adjustments they make in order that we are able to go crawl them and replace the information. Learn extra here.
Ahrefs crawls 7B+ pages every day. Semrush claims they crawl 25B pages per day. This would be ~3.5x what Ahrefs crawls per day. The problem is that I can’t find any evidence that they crawl that fast.
We saw that around half the links that Semrush had marked as active were actually dead compared to about 17% in Ahrefs, which indicated to me that they may not re-crawl links as often. That and the freshness test both pointed to them crawling slower. I decided to look into it.
Logs of my sites
I checked the logs of some of my sites and sites I have access to, and I didn’t see anything to support the claim that Semrush crawls faster. If you have access to logs of your own site, you should be able to check which bots are crawling the fastest.
80,000 months of log data
I was curious and wanted to look at bigger samples. I used Web Explorer and a few different footprints (patterns) to find log file summaries produced by AWStats and Webalizer. These are often published on the web.
I scraped and parsed ~80,000 log file summaries that contained 1 month of data each and were generated in the last couple of years. This sample contained over 9k websites in total.
I did not see evidence of Semrush crawling many times faster than Ahrefs for these sites, as they claim they do. The only bot that was crawling much faster than Ahrefsbot in this dataset was Googlebot. Even other search engines were behind our crawl rate.
That’s just data from a small-ish number of sites compared to the scale of the web. What about for a larger chunk of the web?
Data from 20%+ of web traffic
At the time of writing, Cloudflare Radar has Ahrefsbot because the #7 most lively bot on the net and Semrushbot at #40.
Whereas this isn’t an entire image of the net, it’s a pretty big chunk. In 2021, Cloudflare was stated to handle ~20% of the web’s traffic, up from ~10% in 2018. It’s probably a lot greater now with that type of progress. I couldn’t discover the numbers from 2021, however in early 2022 they have been dealing with 32 million HTTP requests / second on common and in early 2023 that they had already grown to dealing with 45 million HTTP requests / second on average, over 40% extra in a single yr!
Moreover, ~80% of websites that use a CDN use Cloudflare. They deal with lots of the bigger websites on the net; BuiltWith reveals that Cloudflare is used by ~32% of the Top 1M websites. That’s a major pattern measurement and certain the most important pattern that exists.
How a lot do website positioning instruments crawl?
Among the website positioning instruments share the variety of pages they crawl on their web sites. The one one within the chart beneath that doesn’t have a publicly printed crawl price is AhrefsSiteAudit bot, however I requested our group to drag the information for this. Let me put the rankings in perspective with precise and claimed crawl charges.
| Rating | Bot | Crawl Charge |
|---|---|---|
| 7 | Ahrefsbot | 7B+ / day |
| 27 | DataForSEO Bot | 2B / day |
| 29 | AhrefsSiteAudit | 600M – 700M / day |
| 35 | Botify | 143.3M / day |
| 40 | Semrushbot | 25B / day* claimed |
The mathematics isn’t mathing. How can Semrush declare they’re crawling a number of occasions as quick as these others, however their rating is decrease? Cloudflare doesn’t cowl your entire net, however it’s a big chunk of the net and a greater than consultant pattern measurement.
Once they initially made this 25B declare, I imagine they have been nearer to ninetieth on Cloudflare Radar, close to the underside of the listing on the time. Semrush hasn’t up to date this quantity since then, and I recall a time frame the place they have been within the 60s-70s on Cloudflare Radar as properly. They do appear to be getting sooner, however their claimed numbers nonetheless don’t add up.
I don’t hear SEOs raving about Moz or Sistrix having the perfect hyperlink knowledge, however they’re twenty first and thirty sixth on the listing respectively. Each are greater than Semrush.
Potential explanations of variations
Semrush could also be conflating the time period pages with hyperlinks, which is definitely talked about in a few of their documentation. I don’t need to hyperlink to it, however you could find it with this quote: “Each day, our bot crawls over 25 billion hyperlinks”. However hyperlinks usually are not the identical factor as pages and there might be lots of of hyperlinks on a single web page.
It’s additionally attainable they’re crawling a portion of the net that’s simply extra spammy and isn’t mirrored within the knowledge from both of the sources I checked out. Among the numbers point out this can be the case.
Y’all shouldn’t belief research achieved by a particular vendor when it compares them to others, even this one. I attempt to be as truthful as I might be and comply with the information, however since I work at Ahrefs you’ll be able to hardly contemplate me unbiased. Go take a look at the information yourselves and run your individual assessments.
There are some people within the website positioning neighborhood who attempt to do these assessments each occasionally. The final main 3rd party study was run by Matthew Woodward, who initially declared Semrush the winner, however the conclusion was modified and Ahrefs was finally declared to be the rightful winner. What occurred?
The methodology chosen for the research closely favored Semrush and was investigated by a buddy of mine, Russ Jones, could he relaxation in peace. Right here’s what Russ needed to say about it:
Whereas companies like Majestic and Ahrefs probably retailer a single canonical IP deal with per area, SEMRush appears to retailer per hyperlink, which accounts for why there can be extra IPs that referring domains in some circumstances. I don’t assume SEMRush is deliberately inflating their numbers, I feel they’re storing the information otherwise than rivals which leads to a quantity that’s greater and doubtlessly deceptive, however not resulting from ailing intent.
The response from Matthew indicated that Semrush may need misled him of their favor. Right here’s that remark:
In the long run, Ahrefs received.
Test our present stats on our massive knowledge web page.
Whereas Semrush doesn’t present present {hardware} stats, they did present some up to now once they made adjustments to their hyperlink index.
In June 2019, they made an announcement that claimed that they had the most important index. The check from Matthew Woodward that I talked about occurred after this check, and as you noticed, Ahrefs received that.
In June 2021, they made one other announcement about their hyperlink index that claimed they have been the most important, quickest, and greatest.
These are some stats they launched on the time:
- 500 servers
- 16,128 cpu cores
- 245 TB of reminiscence
- 13.9 PB of storage
- 25B+ pages / day
- 43.8T hyperlinks
The discharge stated they elevated storage, however their earlier launch stated that they had 4000 PBs of storage. They stated the storage was 4x, so I suppose the earlier quantity was presupposed to be 4000 TBs and never 4000 PBs, and so they simply received combined up on the terminology.
I checked our numbers on the time, and that is how we matched up:
- 2400 servers (~5x higher)
- 200,000 cpu cores (~12.5x higher)
- 900 TB of reminiscence (~4x higher)
- 120 PB of storage (~9x higher)
- 7B pages / day (~3.5x much less???)
- 2.8T dwell hyperlinks (I’m unsure the whole measurement, however to this present day it’s not as massive because the quantity they claimed)
They have been claiming extra hyperlinks and sooner crawling with a lot much less storage and {hardware}. Granted, we don’t know the small print of the {hardware}, however we don’t run on dated tech.
They claimed to retailer extra hyperlinks than now we have even now and in much less house than we add to our system every month. It actually doesn’t make sense.
Remaining ideas
Don’t blindly belief the numbers on the dashboards or the overall numbers as a result of they might characterize fully various things. Whereas there’s no excellent solution to evaluate the information between completely different instruments, you’ll be able to run lots of the checks I confirmed to attempt to evaluate related issues and clear up the information. If one thing appears to be like off, ask the instrument distributors for a proof.
If there ever comes a time after we cease successful on issues like tech and crawl pace, go forward and change to a different instrument and cease paying us. However till that point, I’d be extremely skeptical of any claims by different instruments.
When you’ve got questions, message me on X.
