Which search engine optimization bots are blocked probably the most? This will have an effect on the standard of the info the instrument offers.
Blocking these bots has a big affect on the hyperlink index of the instrument. They can not crawl pages, to allow them to’t see the place these pages hyperlink. It doesn’t matter whether or not it is site visitors estimates, key phrase rankings, homepage, and so forth. These are constructed from completely different knowledge sources.
Within the case of AHREFS, it additionally impacts the web page historical past operate. This means modifications to the web page over time. ahrefsbot additionally drives search engine indexes. Yep.comsubsequently blocking ahrefsbot means it will not seem in yep’s search outcomes.
We checked out 140 million web sites to see how usually search engine optimization bots have been blocked. I need to thank the info scientists Xibeiijia Guan To retrieve this knowledge.
That is the highest three most blocked search engine optimization bots.
- MJ12bot (Majestic). Blocked at 6.49% of all web sites.
- semrushbot. Blocked at 6.34% of all web sites.
- ahrefsbot. Blocked at 6.31% of all web sites.
We have seen the full variety of web sites that block bots. There are numerous methods to dam bots with robots.txt.
- Express blockif the bot is talked about and prohibited
- Common blocksall bots could also be blocked
- Cases with instructions Bot allowedafter blocking all bots
Warning: This doesn’t embrace different block sorts resembling firewalls and IP blocks.
As talked about earlier than, probably the most blocked bot is Majestic’s MJ12Bot. I believe there are a number of causes for this.
- They’re distributed crawlers. Which means it isn’t attainable to search for or block on an IPS, which reduces reliability.
- They have been uncooked the net for longer.
- The consumer base is smaller than well-liked search engine optimization instruments, which suggests you will have much less leverage to take away blocks.
That is probably the most blocked search engine optimization bot:
And the full of internet sites that block search engine optimization bots:
This is the info:
| Bot Title | depend | p.c% | Bot Operator |
|---|---|---|---|
| MJ12BOT | 9081205 | 6.49 | Majestic |
| semrushbot | 8868486 | 6.34 | semrush |
| ahrefsbot | 8831316 | 6.31 | ahrefs |
| Dotbot | 8569766 | 6.13 | Mos |
| blexbot | 8374216 | 5.99 | search engine optimization PowerSuite |
| serpstatbot | 7878935 | 5.63 | serpstat |
| DataForSeObot | 7872939 | 5.63 | dataforseo |
| semrushbot-ct | 7855400 | 5.62 | semrush |
| Barkrowler | 7804425 | 5.58 | Babal |
| semrushbot-ba | 7796785 | 5.57 | semrush |
| semrushbot-swa | 7789812 | 5.57 | semrush |
| semrushbot-si | 7789062 | 5.57 | semrush |
| Seokicks | 7758904 | 5.55 | Seokicks |
| Screaming frog search engine optimization spider | 7711108 | 5.51 | Screaming frog |
| linkdexbot | 7704425 | 5.51 | linkdex |
| DomainStatsBot | 7696944 | 5.5 | Area Stats |
| Zoombot | 7669495 | 5.48 | seozoom |
| Sitecheckerbotcrawler | 7666545 | 5.48 | Sitechecker |
| Cocolyse Bot | 7666233 | 5.48 | Cocolize |
| seobilitybot | 7664228 | 5.48 | Sorbility |
| senutobot | 7655145 | 5.47 | Cenuto |
| Excessive Stat | 7648671 | 5.47 | Excessive Stat |
| On-line Webceo-bot | 7648444 | 5.47 | WebCeo |
| Shiny Edge Crawler | 7648139 | 5.47 | Shiny Edge |
| Soliser | 7648112 | 5.47 | Soliser |
Analyzing it turns into a bit extra difficult. For the above, I regarded into the primary robots.txt file on the web site, however all subdomains can have their very own set of directions. Trying on the robots.txt in complete, totalling ~461m, probably the most blocked search engine optimization bot is the 5.76% Semrushbot. That is the highest 5:
- semrushbot: 5.76%
- dotbot (moz):5.34%
- MJ12bot (Majestic): 4.96%
- blexbot: 4.88%
- ahrefsbot: 4.67%
This scale solely considers sure bots when they aren’t permitted. It doesn’t embrace an total ban assertion or the place solely sure bots could also be permitted. In these instances, the web site house owners have made it out of the way in which to particularly block sure bots.
Majestic bots are probably the most focused, adopted by Moz bots.
Essentially the most blocked search engine optimization bots by express point out are:
Listed below are the variety of web sites that explicitly block search engine optimization bots:
This is the info:
| Bot Title | depend | p.c% | Bot Operator |
|---|---|---|---|
| MJ12BOT | 2000372 | 1.43 | Majestic |
| Dotbot | 1402305 | 1 | Mos |
| ahrefsbot | 1350771 | 0.97 | ahrefs |
| semrushbot | 1285857 | 0.92 | semrush |
| blexbot | 861184 | 0.62 | search engine optimization PowerSuite |
| serpstatbot | 354683 | 0.25 | serpstat |
| DataForSeObot | 284694 | 0.2 | dataforseo |
| Barkrowler | 276332 | 0.2 | Babal |
| Seokicks | 219961 | 0.16 | Seokicks |
| semrushbot-ct | 211895 | 0.15 | semrush |
| linkdexbot | 166405 | 0.12 | linkdex |
| DomainStatsBot | 157053 | 0.11 | Area Stats |
| semrushbot-ba | 154349 | 0.11 | semrush |
| semrushbot-si | 147999 | 0.11 | semrush |
| semrushbot-swa | 146261 | 0.1 | semrush |
| Zoombot | 125310 | 0.09 | seozoom |
| Sitecheckerbotcrawler | 122574 | 0.09 | Sitechecker |
| Cocolyse Bot | 121737 | 0.09 | Cocolize |
| seobilitybot | 117558 | 0.08 | Sorbility |
| Screaming frog search engine optimization spider | 87673 | 0.06 | Screaming frog |
| senutobot | 54978 | 0.04 | Cenuto |
| Excessive Stat | 861 | 0 | Excessive Stat |
| senutobot | 54978 | 0.04 | Cenuto |
| Excessive Stat | 861 | 0 | Excessive Stat |
| On-line Webceo-bot | 659 | 0 | WebCeo |
| Shiny Edge Crawler | 289 | 0 | Shiny Edge |
| Soliser | 253 | 0 | Soliser |
I noticed the highest 1M websites by DR. Semrush is probably the most blocked after Majestic and Moz.
This is how particular person bots collapse throughout completely different classes of internet sites: The highest three are as follows:
- autos_and_vehicles: 39%
- books_and_literature: 27%
- actual property: 17%
Go together with a bot request CloudFlare RadarAhrefs is a a lot quicker crawler within the search engine optimization area. It is ~4.6 occasions quicker than Moz and ~6.7x quicker than semrush.
