Track LLM visibility with a small, reliable KPI set: AI bot search hits, LLM referrals, citation share, question coverage, and engagement quality of AI-driven visits.
Why a new KPI set?
Traditional SEO metrics miss AI behavior because chatbots often don’t run JavaScript and may not pass standard referrers. You need edge logs plus selective analytics to see the full picture.
Core KPIs (definitions and formulas)
- AI Bot Search Hits
- What: Count of requests from known AI/LLM crawlers and AI-enabled search agents
- How: Parse edge logs by user-agent patterns (e.g., gptbot, oai-searchbot, claude, perplexitybot, googlebot)
- Output: hits/day, hits by bot, hits by path
- LLM Referrals
- What: Human clicks that originate from AI answers (when referrer is present) or from AI product link-outs
- How: Combine edge referrer parsing with GA4 custom dimensions; annotate likely AI sources
- Output: sessions/week, by landing page and source
- Citation Share (Directional)
- What: Share of your pages cited or summarized across monitored AI answers for a topic cluster
- How: Spot-check in major LLMs, track mentions in logs, and maintain a rolling sample of citation events
- Output: % of cluster pages observed as cited in last 30/90 days (est.)
- Question Coverage (Fan-Out Coverage)
- What: % of priority question variants (your question graph) with a published, chunkable answer
- How: Maintain a question backlog; mark covered vs. uncovered; update monthly
- Output: coverage %, uncovered count by intent tier
- Chunk Retrievability Score
- What: Structural readiness of content chunks (clear H2/H3, concise summaries, FAQs, citations)
- How: Heuristic scoring 0–100 across a sample of pages
- Output: avg score by cluster; target ≥80
- Engagement Quality of AI-Driven Visits
- What: Downstream behavior of AI-qualified traffic
- How: Compare session depth, time on page, and conversion rate vs. baseline organic
- Output: lift vs. baseline (e.g., +35% session depth)
- Freshness Interval
- What: Days since last meaningful update on key pages
- How: Content ops tracker; aim for recency aligned to your domain pace
- Output: median days since update by cluster
Instrumentation blueprint
- Capture at the edge: enable access logs with IP, UA, referer, path, timestamp
- Parse bots: maintain and update a bot UA list; tag events as bot/human
- GA4 overlay: create custom dimensions (is_bot, bot_name); segment reports
- Dashboard: combine edge metrics + GA4 engagement to monitor trends weekly
Sample dashboard (sections)
- AI Bot Activity: hits by bot, top paths, trendline
- LLM Referrals: sessions by source/landing page; conversion vs baseline
- Coverage & Structure: question coverage %, chunk retrievability score
- Freshness: pages due for update; last-updated trend
Benchmarks (directional, vary by site)
- Bot hits: stable growth over time as content expands
- AI-driven engagement: typically higher session depth vs generic organic (monitor lift, not absolute)
- Coverage: aim for 70–90% across priority question sets within a quarter
FAQs
- Can GA4 alone measure LLM performance? No—AI bots don’t execute JS. Use edge logs plus GA4.
- How do I attribute “AI referrals” when referrers are missing? Use landing page patterns, time windows around known AI answer exposure, and annotate campaigns; treat as directional.
- What’s the fastest lever to move these KPIs? Improve question coverage and chunk structure on high-intent clusters.
Sources
- Measurement limitations of GA4 for AI and how to fix them
- Bot capture and parsing approach
- Factors that influence LLM visibility and structure priorities