The BrightEdge Bot is one of the User Agents that BrightEdge uses to discover and collect web data. It is important to note that BrightEdge uses other User Agents for site crawls as well.

Data collected by the BrightEdge Bot is used for:

• Site Audit/Content IQ, which analyzes on-page SEO, technical and usability issues
  • Site Audit Crawls can be scheduled on a desired basis (e.g. weekly/monthly)
• On-page Recommendations, providing prescriptive actions to optimize web pages
  • Recommendation Crawls occur weekly or on-demand
• SearchIQ, identifying areas to prioritize for SEO and analyzing current performance
  • SearchIQ Crawls occur weekly as well

BrightEdge Bot Details

• BrightEdge - BrightEdge Crawler/1.0 (crawler@brightedge.com)

How BrightEdge Bot Crawls Your Site

BrightEdge starts crawls based on start URLs. As the bot visits these URLs, it opens links from each page for further crawling. These URLs are ultimately cached as a list which can be referenced to make future crawls more efficient and act as a basis for comparison between crawls.

How To Block BrightEdge from Crawling Your Site

While many bots are well-intentioned, you may want to prevent certain ones from crawling your site. The most efficient way to do this is through a robots.txt file, containing instructions and rules about which bots are allowed and where they have permission to crawl.

Note, a robots.txt file should exist on each hosted directory – e.g. subdomains should have their own robots.txt files defined as well!

To disallow the BrightEdge User Agent to crawl your site, add the following to your robots.txt file:

User-agent: BrightEdge
Disallow: /

To ensure the BrightEdge User Agent is explicitly allowed to crawl your site, you can add:

User-agent: BrightEdge
Disallow:

Note: If BrightEdge IP ranges are whitelisted, there is no dependency on User Agent selection. The BrightEdge user agent does not necessarily have to be allowed, as we use other User Agents too!

Getting Support

For any questions about whitelisting or BrightEdge crawlers, please email support@brightedge.com