Website Analytics
Track AI bot traffic to your website.
Website Analytics shows you when AI crawlers visit your website, which pages they access, and how often they return.
Why Track AI Bots?
AI models like ChatGPT, Perplexity, and Google's AI features actively crawl websites to gather information. Understanding this traffic helps you:
- Verify AI is reading your content — Confirm your site is being indexed
- See which pages AI prioritizes — Understand what content AI finds valuable
- Track crawl frequency — Know how often AI models update their knowledge of your site
- Identify optimization opportunities — Improve pages that AI frequently accesses
Setting Up Analytics
Navigate to Analytics
From your workspace, click Website Analytics in the sidebar.
Copy the tracking snippet
You'll see a JavaScript snippet with your unique script key:
<script>
(function(w,d,s,k){
w._pfq=w._pfq||[];
var f=d.getElementsByTagName(s)[0],
j=d.createElement(s);
j.async=true;
j.src='https://app.promptfern.com/js/pf.js';
j.dataset.key=k;
f.parentNode.insertBefore(j,f)
})(window,document,'script','YOUR_UNIQUE_KEY');
</script>Add to your website
Paste this snippet before the closing </head> tag on every page you want to track.
For common platforms:
| Platform | Where to add |
|---|---|
| WordPress | Theme header or header plugin |
| Webflow | Project Settings → Custom Code → Head |
| Shopify | theme.liquid before </head> |
| Next.js | _document.tsx or layout component |
| Static HTML | Each page's <head> section |
Deploy your changes
Push your updated code to production.
Verify tracking
Return to the Analytics dashboard. Within a few minutes of your first AI bot visit, you'll see data appear.
Understanding the Dashboard
Setup State
Before receiving any data, you'll see instructions for installing the tracking snippet along with a preview of what the dashboard will look like.
Active Dashboard
Once data starts flowing, you'll see:
KPI Cards
- AI Visits (Period): Total bot visits in the selected date range
- Known AI Bots: Visits from identified crawlers (GPTBot, Googlebot, etc.)
- Unique Agents: Number of distinct bot user agents
- Script Key: Your tracking identifier
Daily Visits Chart
A timeline showing AI bot traffic over time. Look for:
- Crawl frequency patterns
- Spikes after content updates
- Comparison across time periods
Top Bots
Which AI crawlers visit most frequently:
| Bot | Description |
|---|---|
| GPTBot | OpenAI's crawler for ChatGPT |
| Google-Extended | Google's AI training crawler |
| PerplexityBot | Perplexity's search crawler |
| Anthropic-AI | Anthropic's crawler for Claude |
| Bingbot | Microsoft's crawler (Copilot uses Bing) |
Top Pages
Which pages AI bots access most. Prioritize optimizing these for AI readability.
Countries
Geographic distribution of bot traffic (based on crawler IP location).
Recent Events
Live log of the latest bot visits with timestamps and page URLs.
Optimizing for AI Crawlers
Allow AI bots in robots.txt
Ensure your robots.txt doesn't block AI crawlers:
User-agent: GPTBot
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: anthropic-ai
Allow: /
Create crawlable content
AI bots process HTML content best when it's:
- Well-structured with semantic HTML
- Using clear heading hierarchy (H1, H2, H3)
- Including descriptive meta tags
- Loading content server-side (not just client-side JS)
Update frequently
AI crawlers often prioritize sites that update regularly:
- Publish new content consistently
- Update existing pages with fresh information
- Maintain a blog or news section
Add structured data
Schema markup helps AI understand your content:
{
"@context": "https://schema.org",
"@type": "SoftwareApplication",
"name": "Your Product",
"description": "What your product does",
"applicationCategory": "BusinessApplication"
}Privacy & Security
Data collected
The tracking snippet captures:
- Page URL and referrer
- User agent string
- Timestamp
- Geographic hints (country, region)
- No personally identifiable information (PII)
IP handling
IP addresses are hashed server-side using SHA-256. Raw IPs are never stored.
Script key security
Your script key is unique to your brand. Keep it private. If compromised, contact support to rotate it.
Troubleshooting
No data appearing
- Verify snippet installation: Check your page source for the tracking code
- Check network requests: Look for requests to
app.promptfern.com/api/analytics/ingest - Wait for bot traffic: Human visits don't appear—only identified AI bots
- Check integration status: Ensure the integration is enabled in your Brand settings
Bots not detected
Some AI crawlers may use generic user agents. The detection system is updated regularly to identify new bot signatures.
Duplicate events
The snippet fires on page load and when tabs become hidden. This is expected behavior to capture bot session patterns.