Grok AI Bots

Grok behaves very differently from other major LLMs. When prompted, it aggressively crawls sites with dozens of bots, pulls every file type it can find, and does not identify itself in its user agent so its true footprint in the AI ecosystem is largely unknown.

aeo.press editorialAI visibility insights
Published

I kept wondering why Grok never showed up in aeo.press server logs. So I went looking.

The way Grok’s bots behave feels very on-brand for Elon Musk…

I ran a simple test in Grok to better understand. I wrote:

“What does the aeo.press llms.txt file say about itself?”

And then Grok happened.

Grok immediately sent 70+ AI bots to pull content. And they were hungry for everything, including .txt files. Compare that to ChatGPT, which sent 10 bots and mostly stuck with HTML.

Another big difference: every major LLM identifies itself clearly. This is ChatGPT’s user agent (notice the tail end):
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +[https://openai.com/bot](https://openai.com/bot%60)

This is Grok’s:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[142.0.0.0](http://142.0.0.0/) Safari/537.36

Grok doesn’t include anything to help you identify it. All it really tells us is that the site was accessed by an older Mac running Chrome.

So Grok plays a very different game:

  • No self-identification
  • Aggressive parallel crawling
  • Happy with any file type to answer a question

Which makes it fascinating… and hard to measure. We just don’t know how big a role Grok plays in the AI space.

1770074925642.jpg