README

README.md

@@ -96,15 +96,32 @@ The `web_search` block wires in Anthropic's server-side `web_search` (and option
 }
 ```
 
-| Field                | Type     | Description |
+| Field                  | Type     | Description |
-| -------------------- | -------- | ----------- |
+| ---------------------- | -------- | ----------- |
-| `allowed_domains`    | string[] | Search results and fetch targets are restricted to these. Each entry is a **domain with an optional path prefix and no scheme** — `example.com` or `example.com/help`. Mutually exclusive with `blocked_domains`. |
+| `allowed_domains`      | string[] | Restricts **`web_search`** results (and, by default, `web_fetch` targets) to these. Each entry is a **domain with an optional path prefix and no scheme** — `example.com` or `example.com/help`. Mutually exclusive with `blocked_domains`. **The path only scopes `web_search`** — see the asymmetry note below. |
-| `blocked_domains`    | string[] | Excludes these domains instead of allowlisting. Mutually exclusive with `allowed_domains` (setting both fails validation at boot, since the API 400s). |
+| `blocked_domains`      | string[] | Excludes these domains instead of allowlisting. Mutually exclusive with `allowed_domains` (setting both fails validation at boot, since the API 400s). |
-| `max_uses`           | number   | Caps how many searches the model may run per turn. Omit for no cap. |
+| `fetch_allowed_domains`| string[] | **Host-only** allowlist for `web_fetch`, independent of the search list. Omit it and `web_fetch` reuses the *hosts* of `allowed_domains`. Set it to fetch a **narrower** set than you search — e.g. to keep a shared host (a social platform) search-only. Only meaningful with `fetch: true`. |
-| `fetch`              | bool     | Also enable `web_fetch`, which pulls full page content for URLs already surfaced by a search or pasted by the user (it cannot fetch model-invented URLs). Shares the same allow/block list. Citations are always on when fetch is enabled, so fetched passages are sourceable. |
+| `max_uses`             | number   | Caps how many searches the model may run per turn. Omit for no cap. |
-| `max_content_tokens` | number   | Caps the tokens a single `web_fetch` may pull into context. Only meaningful with `fetch: true`. |
+| `fetch`                | bool     | Also enable `web_fetch`, which pulls full page content for URLs already surfaced by a search or pasted by the user (it cannot fetch model-invented URLs). Citations are always on when fetch is enabled, so fetched passages are sourceable. |
+| `max_content_tokens`   | number   | Caps the tokens a single `web_fetch` may pull into context. Only meaningful with `fetch: true`. |
 
-The allowlist is a **hard, server-side sandbox**, not a prompt request — off-list results are dropped and an off-list fetch target returns `url_not_allowed`, which is the exfiltration mitigation Anthropic recommends for bots processing untrusted input. Keep the list tight. Note that `web_fetch` cannot read JavaScript-rendered pages, so for dynamic sites the search snippet (plus citations) is the reliable signal and a full fetch is a bonus, not a dependency.
+**Search/fetch path asymmetry (important).** Anthropic filters the two tools differently: `web_search` honors a path prefix (`example.com/help` matches only `example.com/help/...`), but **`web_fetch` matches on the host only — an entry that includes a path never matches any fetch URL**. The engine bridges this: `web_search` gets your entries verbatim, while `web_fetch` gets the host portion of each entry (path stripped, deduped). So a path-scoped `allowed_domains` still permits fetching across the whole host. If that host isn't wholly trusted, list the fetchable hosts explicitly in `fetch_allowed_domains` instead.
+
+**Search-only pattern for shared hosts (e.g. social).** To let the model *search* a single account on a shared platform without ever *fetching* the wider platform, put the account path in `allowed_domains` and leave its host out of `fetch_allowed_domains`:
+
+```json
+"web_search": {
+  "allowed_domains": ["helpcenter.example/hc", "x.com/youraccount"],
+  "fetch_allowed_domains": ["helpcenter.example"],
+  "fetch": true
+}
+```
+
+Here search is confined to your help center *and* your one social account, but fetch can only ever reach `helpcenter.example` — a pasted `x.com/someone-else` URL is not fetchable. Note the search side only sandboxes cleanly on platforms that keep an account's content under its handle path (X/Twitter, TikTok `@handle`, Facebook); use the `handle/` or `handle/*` form to avoid a prefix bleed. Instagram is an exception — posts live at `instagram.com/p/<code>`, not under the handle — so it can't be account-sandboxed by path.
+
+The allowlist is a **hard, server-side sandbox**, not a prompt request — off-list results are dropped and an off-list fetch target returns `url_not_allowed`, which is the exfiltration mitigation Anthropic recommends for bots processing untrusted input. Keep the list tight. A fetch can still fail for site-side reasons (bot protection such as Cloudflare returns `url_not_accessible`); the search snippet plus citations remain the reliable signal, so a full fetch is a bonus, not a dependency.
+
+`web_fetch` can only open a URL that a `web_search` result surfaced (or the user pasted) — never one the model invents or rebuilds from a page title; those return `url_not_in_prior_context`. So a fetch-enabled bot's prompt should steer it to **search first, then fetch a returned URL**, and to **re-search with a more specific query** (rather than guess a URL) when the exact page it wants isn't in the results.
 
 Web search is **not free** — Anthropic bills per search (plus the tokens the results add) — so scope the allowlist and `max_uses` deliberately, and lean on a prompt that tells the bot to search only when a question actually turns on current or authoritative information. Search activity logs as `[web] ...` lines, and long multi-search turns are handled transparently (the engine follows Anthropic's `pause_turn` continuations up to a small cap, so a slow search doesn't truncate the reply).
 

anthropic.go

@@ -177,9 +177,13 @@ func webSearchTools(cfg *WebSearchConfig) []anthropic.BetaToolUnionParam {
 	tools := []anthropic.BetaToolUnionParam{{OfWebSearchTool20250305: search}}
 
 	if cfg.Fetch {
+		fetchAllowed := cfg.FetchAllowedDomains
+		if len(fetchAllowed) == 0 {
+			fetchAllowed = cfg.AllowedDomains
+		}
 		fetch := &anthropic.BetaWebFetchTool20250910Param{
-			AllowedDomains: cfg.AllowedDomains,
+			AllowedDomains: fetchHosts(fetchAllowed),
-			BlockedDomains: cfg.BlockedDomains,
+			BlockedDomains: fetchHosts(cfg.BlockedDomains),
 			Citations:      anthropic.BetaCitationsConfigParam{Enabled: param.NewOpt(true)},
 		}
 		if cfg.MaxUses > 0 {
@@ -193,6 +197,29 @@ func webSearchTools(cfg *WebSearchConfig) []anthropic.BetaToolUnionParam {
 	return tools
 }
 
+// fetchHosts strips any path from each domain entry. web_fetch matches on host
+// only, so a path-scoped entry (valid for web_search) would otherwise never match
+// any fetch URL. Search keeps the path-scoped entries; fetch gets host-only.
+func fetchHosts(entries []string) []string {
+	if len(entries) == 0 {
+		return nil
+	}
+	seen := make(map[string]bool, len(entries))
+	hosts := make([]string, 0, len(entries))
+	for _, e := range entries {
+		host := e
+		if i := strings.IndexByte(host, '/'); i >= 0 {
+			host = host[:i]
+		}
+		if host == "" || seen[host] {
+			continue
+		}
+		seen[host] = true
+		hosts = append(hosts, host)
+	}
+	return hosts
+}
+
 func buildUserContext(username, firstName, lastName string, isPremium bool, languageCode string, messageTime int) string {
 	name := strings.TrimSpace(firstName + " " + lastName)
 	if name == "" {

anthropic_test.go

@@ -165,8 +165,8 @@ func TestWebSearchTools(t *testing.T) {
 		if search == nil {
 			t.Fatalf("tools[0] is not a web_search tool")
 		}
-		if len(search.AllowedDomains) != 2 {
+		if !sameStrings(search.AllowedDomains, []string{"example.com/hc", "docs.example.com"}) {
-			t.Errorf("search AllowedDomains = %v, want 2 entries", search.AllowedDomains)
+			t.Errorf("search AllowedDomains = %v, want the path-scoped list unchanged", search.AllowedDomains)
 		}
 		if search.MaxUses.Value != 3 {
 			t.Errorf("search MaxUses = %d, want 3", search.MaxUses.Value)
@@ -176,8 +176,8 @@ func TestWebSearchTools(t *testing.T) {
 		if fetch == nil {
 			t.Fatalf("tools[1] is not a web_fetch tool")
 		}
-		if len(fetch.AllowedDomains) != 2 {
+		if !sameStrings(fetch.AllowedDomains, []string{"example.com", "docs.example.com"}) {
-			t.Errorf("fetch AllowedDomains = %v, want 2 entries", fetch.AllowedDomains)
+			t.Errorf("fetch AllowedDomains = %v, want host-only [example.com docs.example.com]", fetch.AllowedDomains)
 		}
 		if fetch.MaxContentTokens.Value != 50000 {
 			t.Errorf("fetch MaxContentTokens = %d, want 50000", fetch.MaxContentTokens.Value)
@@ -187,6 +187,54 @@ func TestWebSearchTools(t *testing.T) {
 		}
 	})
 
+	t.Run("fetch hosts are deduped", func(t *testing.T) {
+		tools := webSearchTools(&WebSearchConfig{
+			AllowedDomains: []string{"a.com/x", "a.com/y", "b.com"},
+			Fetch:          true,
+		})
+		fetch := tools[1].OfWebFetchTool20250910
+		if fetch == nil {
+			t.Fatalf("tools[1] is not a web_fetch tool")
+		}
+		if !sameStrings(fetch.AllowedDomains, []string{"a.com", "b.com"}) {
+			t.Errorf("fetch AllowedDomains = %v, want deduped [a.com b.com]", fetch.AllowedDomains)
+		}
+	})
+
+	t.Run("social host is search-only via fetch_allowed_domains", func(t *testing.T) {
+		tools := webSearchTools(&WebSearchConfig{
+			AllowedDomains:      []string{"helpshift.example/hc", "x.com/thatskygame"},
+			FetchAllowedDomains: []string{"helpshift.example"},
+			Fetch:               true,
+		})
+		search := tools[0].OfWebSearchTool20250305
+		if search == nil {
+			t.Fatalf("tools[0] is not a web_search tool")
+		}
+		var searchHasSocial bool
+		for _, d := range search.AllowedDomains {
+			if d == "x.com/thatskygame" {
+				searchHasSocial = true
+			}
+		}
+		if !searchHasSocial {
+			t.Errorf("search AllowedDomains = %v, want it to include x.com/thatskygame", search.AllowedDomains)
+		}
+
+		fetch := tools[1].OfWebFetchTool20250910
+		if fetch == nil {
+			t.Fatalf("tools[1] is not a web_fetch tool")
+		}
+		if !sameStrings(fetch.AllowedDomains, []string{"helpshift.example"}) {
+			t.Errorf("fetch AllowedDomains = %v, want only [helpshift.example]", fetch.AllowedDomains)
+		}
+		for _, d := range fetch.AllowedDomains {
+			if strings.HasPrefix(d, "x.com") {
+				t.Errorf("fetch AllowedDomains leaked the social host: %v", fetch.AllowedDomains)
+			}
+		}
+	})
+
 	t.Run("wire shape carries allowed_domains", func(t *testing.T) {
 		tools := webSearchTools(&WebSearchConfig{
 			AllowedDomains: []string{"thatgamecompany.helpshift.com/hc"},
@@ -211,6 +259,41 @@ func TestWebSearchTools(t *testing.T) {
 	})
 }
 
+func sameStrings(got, want []string) bool {
+	if len(got) != len(want) {
+		return false
+	}
+	for i := range got {
+		if got[i] != want[i] {
+			return false
+		}
+	}
+	return true
+}
+
+func TestFetchHosts(t *testing.T) {
+	cases := []struct {
+		name string
+		in   []string
+		want []string
+	}{
+		{"nil in nil out", nil, nil},
+		{"empty in nil out", []string{}, nil},
+		{"host passthrough", []string{"example.com"}, []string{"example.com"}},
+		{"strip path", []string{"example.com/hc/en"}, []string{"example.com"}},
+		{"dedup after strip", []string{"a.com/x", "a.com/y"}, []string{"a.com"}},
+		{"preserve order and subdomains", []string{"docs.example.com/a", "example.com"}, []string{"docs.example.com", "example.com"}},
+		{"drop empty leading slash", []string{"/oops", "ok.com"}, []string{"ok.com"}},
+	}
+	for _, tc := range cases {
+		t.Run(tc.name, func(t *testing.T) {
+			if got := fetchHosts(tc.in); !sameStrings(got, tc.want) {
+				t.Errorf("fetchHosts(%v) = %v, want %v", tc.in, got, tc.want)
+			}
+		})
+	}
+}
+
 func TestEmptyStreamError(t *testing.T) {
 	err := emptyStreamError("max_tokens", 3900, 4000)
 	for _, want := range []string{"output budget exhausted", "3900", "4000"} {

config.go

@@ -16,11 +16,12 @@ type MCPServer struct {
 }
 
 type WebSearchConfig struct {
-	AllowedDomains   []string `json:"allowed_domains,omitempty"`
+	AllowedDomains      []string `json:"allowed_domains,omitempty"`
-	BlockedDomains   []string `json:"blocked_domains,omitempty"`
+	BlockedDomains      []string `json:"blocked_domains,omitempty"`
-	MaxUses          int      `json:"max_uses,omitempty"`
+	FetchAllowedDomains []string `json:"fetch_allowed_domains,omitempty"`
-	Fetch            bool     `json:"fetch,omitempty"`
+	MaxUses             int      `json:"max_uses,omitempty"`
-	MaxContentTokens int      `json:"max_content_tokens,omitempty"`
+	Fetch               bool     `json:"fetch,omitempty"`
+	MaxContentTokens    int      `json:"max_content_tokens,omitempty"`
 }
 
 const (
@@ -126,6 +127,19 @@ func loadAllConfigs(dir string) ([]BotConfig, error) {
 					config.ID)
 			}
 
+			if ws := config.WebSearch; ws != nil && len(ws.FetchAllowedDomains) > 0 {
+				if !ws.Fetch {
+					InfoLogger.Printf("[%s] web_search.fetch_allowed_domains is set but fetch is disabled: it has no effect",
+						config.ID)
+				}
+				for _, d := range ws.FetchAllowedDomains {
+					if strings.Contains(d, "/") {
+						InfoLogger.Printf("[%s] web_search.fetch_allowed_domains entry %q includes a path: web_fetch matches host-only, so the whole host is fetchable",
+							config.ID, d)
+					}
+				}
+			}
+
 			config.ConfigFilePath = validPath
 			configs = append(configs, config)
 		}