Long-rumored AppleBot Web crawler is the real deal
The AppleBot Web crawler has been rumored to exist for quite some time, but Cupertino has been quiet about the service until now. A recently-updated support document has provided some details of AppleBot’s operations and made it clear that Cupertino is not relying solely on existing search engines for Siri and Spotlight.
AppleBot was first spotted in action last November, when a Web developer saw numerous hits from a crawler in the 17.0.0.0/8 IP address block. That block is assigned to Apple, one of just a few companies to control an entire A block of Internet addresses. AppleInsider pointed out that developer Jan Moesen had noticed hits from the Web crawler software dating back to Oct. 15, and revealed that the bot was written in the Go programming language.
Does anyone know why Apple is running a web crawler (written in @golang, no less)? http://t.co/5JDTZuplQp
— Jan Moesen (@janmoesen) November 6, 2014
Apple’s Web crawler only makes requests for HTML pages, ignoring the accompanying CSS, JavaScript, and image files associated with a website. This would be indicative of a search engine crawler, indexing Web pages for later retrieval based on specific keyword queries. The bot honors robots instructions, following directives for Googlebot if there are no details listed for AppleBot.
Cupertino has been enhancing Spotlight in both OS X Yosemite and iOS 8, allowing users to access Web search content directly, without the need for a third-party search tool like Google or Bing. Siri, which is officially partnered with Bing for Web queries, has also seen refinements. Apple has been working to provide specific information directly within the service’s search results. AppleBot might not be robust enough to compete with Google or Bing, but it can certainly help Cupertino reduce its reliance on those search engines.
