HotSphinx.com is a price comparison shopping website. We display products and services from many sellers. We cite the information and provide direct links back to the seller's website.
HotSphinxBot is the name of our crawler. It simulates a mobile or desktop user. It can be identified by the following user agent string.
Mozilla/5.0 AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; HotSphinxBot/1.0; +https://hotsphinx.com/bot) Chrome/115.0.0.0 Safari/537.36
The content on our website comes from multiple sources such as product data APIs, seller APIs and crawling. We only display content that is publicly available. We never fill out forms, attempt to authenticate or get behind paywall access. We don't crawl your entire website. We crawl product listing pages only. We do not click through other links, such as putting items in the shopping cart.
We use algorithms to determine how often to crawl a site. A crawler will, generally, not crawl the same url more than once every 7 days. A crawler will access your site once every few seconds. We may deploy multiple crawlers to your site simultaneously. We take care to ensure we do not overwhelm your site. We will, generally, not exceed 10 active crawlers on your site at a given time.
In addition to the user agent header, we will send the "From" header and, when available, the "Referer" header.
Add the following to your site's robots.txt to disallow HotSphinxBot:
User-agent: HotSphinxBot
Disallow: /
If you wish to control which paths are crawled, add the following to your robots.txt:
User-agent: HotSphinxBot
Allow: /path-1/
Disallow: /path-2/
Note: We also honor wildcards in your robots.txt. In addition to the robots.txt file, we will honor noindex and nofollow meta tags.
Other crawlers may spoof our user agent. Here is a list of IP ranges used by HotSphinxBot.
If you have any further questions or concerns about HotSphinxBot, please contact us at bot@hotsphinx.com.