# ============================================================================
# wealthnow.io — robots.txt
# ============================================================================
# Permissive by default. Every public marketing page is fair game. We
# explicitly block surfaces that hold user data, are auth-gated, or are
# system endpoints with no SEO value.
#
# AI training + retrieval crawlers (GPTBot, ClaudeBot, PerplexityBot,
# Google-Extended, Applebot-Extended, Bytespider) are explicitly opted
# IN so that Wealthnow's content enters their corpora — that's how we get
# recommended when users ask ChatGPT / Claude / Perplexity for an AI
# portfolio manager.
#
# This file is shipped from /public/robots.txt and served unchanged
# from https://wealthnow.io/robots.txt.
# ============================================================================

# ─── Default rule: everything else ────────────────────────────────────────
# Permissive on purpose. We Disallow ONLY genuinely-private surfaces: auth-gated
# user data, OAuth callbacks, and the JSON API. Routes that are merely "no search
# value" — /demo, /login, /register, /onboarding — are deliberately LEFT CRAWLABLE
# and carry an in-page `noindex` instead (see frontend/src/components/NoIndex.tsx).
#
# WHY NOT JUST BLOCK THEM: a robots-blocked URL can never be crawled, so a crawler
# never discovers its noindex and the URL stays stuck in the index forever. That
# is exactly how the /demo "Tengu is built for desktop" gate became one of only
# two tengu.co pages Google had indexed. Block-to-hide is the wrong tool here;
# noindex (on a crawlable page) is the one that actually evicts a page.
#
# No query-string (`/*?*`) block: every page is self-canonical, so Google folds
# ?utm=/?theme= duplicates into the canonical itself — the modern, Google-
# recommended way to dedupe. Blocking them would also re-trap /demo?theme=light.
# No Crawl-delay: a brand-new domain wants MORE crawl, not throttled crawl
# (Bing honours crawl-delay; Googlebot ignores it).
User-agent: *
Disallow: /dashboard/
Disallow: /auth/
Disallow: /oauth/
Disallow: /snaptrade/
# /api/ is the JSON API. /api/pricing is a prerendered marketing page;
# the longer Allow wins over the Disallow (RFC 9309).
Allow: /api/pricing
Disallow: /api/
Disallow: /*.json$
Allow: /

# ─── Search engines ───────────────────────────────────────────────────────
# Googlebot, Bingbot, DuckDuckBot, Yandex, etc. intentionally have NO group of
# their own: robots.txt groups are exclusive (a bot obeys ONLY its most-specific
# matching group, never `*` AND its own), so a `User-agent: Googlebot / Allow: /`
# block would have SILENTLY ERASED every Disallow above for Google — letting it
# crawl /dashboard, /api, the OAuth callbacks. Letting the search engines fall
# through to `User-agent: *` is what actually keeps the private surfaces out of
# their crawl. (Image/news sub-crawlers inherit the same way.)

# ─── AI search, assistants and training (explicit opt-in) ──────────────
# One group, so every agent below obeys ONE rule set. These bots get a named
# group as an explicit, auditable opt-in, but a named group REPLACES `*` for
# that bot (same exclusivity rule as above). Each bot listed here once had its
# own `Allow: /` group and therefore ignored every Disallow in `*` — GPTBot,
# ClaudeBot, PerplexityBot, Applebot (Apple's search crawler) and the rest were
# all told they may crawl /dashboard/, /auth/, /oauth/, /api/. The Disallow
# lines below MUST stay identical to the `*` group's; robotsGroups.test.ts
# fails the build if they drift.
#
# Search/answer engines (keep these: they decide whether Wealthnow is cited):
#   OAI-SearchBot (ChatGPT search), ChatGPT-User, Claude-SearchBot,
#   Claude-User, PerplexityBot, Perplexity-User, Applebot, DuckAssistBot,
#   Amazonbot, MistralAI-User.
# Training/corpus: GPTBot, ClaudeBot (+ legacy Claude-Web / anthropic-ai),
#   Google-Extended (a token, not a crawler: Gemini grounding/training),
#   Applebot-Extended, Bytespider, meta-externalagent.
# Link-preview fetchers: facebookexternalhit, Twitterbot, LinkedInBot,
#   Slackbot-LinkExpanding.
# Bingbot is NOT listed on purpose: Bing (which also feeds Copilot, DuckDuckGo
# and much of ChatGPT search) falls through to `*` like Googlebot.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: GPTBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: ClaudeBot
User-agent: Claude-Web
User-agent: anthropic-ai
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Google-Extended
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: DuckAssistBot
User-agent: Amazonbot
User-agent: MistralAI-User
User-agent: Bytespider
User-agent: meta-externalagent
User-agent: facebookexternalhit
User-agent: Twitterbot
User-agent: LinkedInBot
User-agent: Slackbot-LinkExpanding
Disallow: /dashboard/
Disallow: /auth/
Disallow: /oauth/
Disallow: /snaptrade/
# /api/ is the JSON API. /api/pricing is a prerendered marketing page;
# the longer Allow wins over the Disallow (RFC 9309).
Allow: /api/pricing
Disallow: /api/
Disallow: /*.json$
Allow: /

# ─── Sitemap pointer ──────────────────────────────────────────────────────
Sitemap: https://wealthnow.io/sitemap.xml