diff --git a/README.md b/README.md index a05944a..9733456 100644 --- a/README.md +++ b/README.md @@ -96,15 +96,32 @@ The `web_search` block wires in Anthropic's server-side `web_search` (and option } ``` -| Field | Type | Description | -| -------------------- | -------- | ----------- | -| `allowed_domains` | string[] | Search results and fetch targets are restricted to these. Each entry is a **domain with an optional path prefix and no scheme** — `example.com` or `example.com/help`. Mutually exclusive with `blocked_domains`. | -| `blocked_domains` | string[] | Excludes these domains instead of allowlisting. Mutually exclusive with `allowed_domains` (setting both fails validation at boot, since the API 400s). | -| `max_uses` | number | Caps how many searches the model may run per turn. Omit for no cap. | -| `fetch` | bool | Also enable `web_fetch`, which pulls full page content for URLs already surfaced by a search or pasted by the user (it cannot fetch model-invented URLs). Shares the same allow/block list. Citations are always on when fetch is enabled, so fetched passages are sourceable. | -| `max_content_tokens` | number | Caps the tokens a single `web_fetch` may pull into context. Only meaningful with `fetch: true`. | +| Field | Type | Description | +| ---------------------- | -------- | ----------- | +| `allowed_domains` | string[] | Restricts **`web_search`** results (and, by default, `web_fetch` targets) to these. Each entry is a **domain with an optional path prefix and no scheme** — `example.com` or `example.com/help`. Mutually exclusive with `blocked_domains`. **The path only scopes `web_search`** — see the asymmetry note below. | +| `blocked_domains` | string[] | Excludes these domains instead of allowlisting. Mutually exclusive with `allowed_domains` (setting both fails validation at boot, since the API 400s). | +| `fetch_allowed_domains`| string[] | **Host-only** allowlist for `web_fetch`, independent of the search list. Omit it and `web_fetch` reuses the *hosts* of `allowed_domains`. Set it to fetch a **narrower** set than you search — e.g. to keep a shared host (a social platform) search-only. Only meaningful with `fetch: true`. | +| `max_uses` | number | Caps how many searches the model may run per turn. Omit for no cap. | +| `fetch` | bool | Also enable `web_fetch`, which pulls full page content for URLs already surfaced by a search or pasted by the user (it cannot fetch model-invented URLs). Citations are always on when fetch is enabled, so fetched passages are sourceable. | +| `max_content_tokens` | number | Caps the tokens a single `web_fetch` may pull into context. Only meaningful with `fetch: true`. | -The allowlist is a **hard, server-side sandbox**, not a prompt request — off-list results are dropped and an off-list fetch target returns `url_not_allowed`, which is the exfiltration mitigation Anthropic recommends for bots processing untrusted input. Keep the list tight. Note that `web_fetch` cannot read JavaScript-rendered pages, so for dynamic sites the search snippet (plus citations) is the reliable signal and a full fetch is a bonus, not a dependency. +**Search/fetch path asymmetry (important).** Anthropic filters the two tools differently: `web_search` honors a path prefix (`example.com/help` matches only `example.com/help/...`), but **`web_fetch` matches on the host only — an entry that includes a path never matches any fetch URL**. The engine bridges this: `web_search` gets your entries verbatim, while `web_fetch` gets the host portion of each entry (path stripped, deduped). So a path-scoped `allowed_domains` still permits fetching across the whole host. If that host isn't wholly trusted, list the fetchable hosts explicitly in `fetch_allowed_domains` instead. + +**Search-only pattern for shared hosts (e.g. social).** To let the model *search* a single account on a shared platform without ever *fetching* the wider platform, put the account path in `allowed_domains` and leave its host out of `fetch_allowed_domains`: + +```json +"web_search": { + "allowed_domains": ["helpcenter.example/hc", "x.com/youraccount"], + "fetch_allowed_domains": ["helpcenter.example"], + "fetch": true +} +``` + +Here search is confined to your help center *and* your one social account, but fetch can only ever reach `helpcenter.example` — a pasted `x.com/someone-else` URL is not fetchable. Note the search side only sandboxes cleanly on platforms that keep an account's content under its handle path (X/Twitter, TikTok `@handle`, Facebook); use the `handle/` or `handle/*` form to avoid a prefix bleed. Instagram is an exception — posts live at `instagram.com/p/`, not under the handle — so it can't be account-sandboxed by path. + +The allowlist is a **hard, server-side sandbox**, not a prompt request — off-list results are dropped and an off-list fetch target returns `url_not_allowed`, which is the exfiltration mitigation Anthropic recommends for bots processing untrusted input. Keep the list tight. A fetch can still fail for site-side reasons (bot protection such as Cloudflare returns `url_not_accessible`); the search snippet plus citations remain the reliable signal, so a full fetch is a bonus, not a dependency. + +`web_fetch` can only open a URL that a `web_search` result surfaced (or the user pasted) — never one the model invents or rebuilds from a page title; those return `url_not_in_prior_context`. So a fetch-enabled bot's prompt should steer it to **search first, then fetch a returned URL**, and to **re-search with a more specific query** (rather than guess a URL) when the exact page it wants isn't in the results. Web search is **not free** — Anthropic bills per search (plus the tokens the results add) — so scope the allowlist and `max_uses` deliberately, and lean on a prompt that tells the bot to search only when a question actually turns on current or authoritative information. Search activity logs as `[web] ...` lines, and long multi-search turns are handled transparently (the engine follows Anthropic's `pause_turn` continuations up to a small cap, so a slow search doesn't truncate the reply). diff --git a/anthropic.go b/anthropic.go index 1f9c71b..f306edd 100644 --- a/anthropic.go +++ b/anthropic.go @@ -177,9 +177,13 @@ func webSearchTools(cfg *WebSearchConfig) []anthropic.BetaToolUnionParam { tools := []anthropic.BetaToolUnionParam{{OfWebSearchTool20250305: search}} if cfg.Fetch { + fetchAllowed := cfg.FetchAllowedDomains + if len(fetchAllowed) == 0 { + fetchAllowed = cfg.AllowedDomains + } fetch := &anthropic.BetaWebFetchTool20250910Param{ - AllowedDomains: cfg.AllowedDomains, - BlockedDomains: cfg.BlockedDomains, + AllowedDomains: fetchHosts(fetchAllowed), + BlockedDomains: fetchHosts(cfg.BlockedDomains), Citations: anthropic.BetaCitationsConfigParam{Enabled: param.NewOpt(true)}, } if cfg.MaxUses > 0 { @@ -193,6 +197,29 @@ func webSearchTools(cfg *WebSearchConfig) []anthropic.BetaToolUnionParam { return tools } +// fetchHosts strips any path from each domain entry. web_fetch matches on host +// only, so a path-scoped entry (valid for web_search) would otherwise never match +// any fetch URL. Search keeps the path-scoped entries; fetch gets host-only. +func fetchHosts(entries []string) []string { + if len(entries) == 0 { + return nil + } + seen := make(map[string]bool, len(entries)) + hosts := make([]string, 0, len(entries)) + for _, e := range entries { + host := e + if i := strings.IndexByte(host, '/'); i >= 0 { + host = host[:i] + } + if host == "" || seen[host] { + continue + } + seen[host] = true + hosts = append(hosts, host) + } + return hosts +} + func buildUserContext(username, firstName, lastName string, isPremium bool, languageCode string, messageTime int) string { name := strings.TrimSpace(firstName + " " + lastName) if name == "" { diff --git a/anthropic_test.go b/anthropic_test.go index 5827e10..b3fe3cf 100644 --- a/anthropic_test.go +++ b/anthropic_test.go @@ -165,8 +165,8 @@ func TestWebSearchTools(t *testing.T) { if search == nil { t.Fatalf("tools[0] is not a web_search tool") } - if len(search.AllowedDomains) != 2 { - t.Errorf("search AllowedDomains = %v, want 2 entries", search.AllowedDomains) + if !sameStrings(search.AllowedDomains, []string{"example.com/hc", "docs.example.com"}) { + t.Errorf("search AllowedDomains = %v, want the path-scoped list unchanged", search.AllowedDomains) } if search.MaxUses.Value != 3 { t.Errorf("search MaxUses = %d, want 3", search.MaxUses.Value) @@ -176,8 +176,8 @@ func TestWebSearchTools(t *testing.T) { if fetch == nil { t.Fatalf("tools[1] is not a web_fetch tool") } - if len(fetch.AllowedDomains) != 2 { - t.Errorf("fetch AllowedDomains = %v, want 2 entries", fetch.AllowedDomains) + if !sameStrings(fetch.AllowedDomains, []string{"example.com", "docs.example.com"}) { + t.Errorf("fetch AllowedDomains = %v, want host-only [example.com docs.example.com]", fetch.AllowedDomains) } if fetch.MaxContentTokens.Value != 50000 { t.Errorf("fetch MaxContentTokens = %d, want 50000", fetch.MaxContentTokens.Value) @@ -187,6 +187,54 @@ func TestWebSearchTools(t *testing.T) { } }) + t.Run("fetch hosts are deduped", func(t *testing.T) { + tools := webSearchTools(&WebSearchConfig{ + AllowedDomains: []string{"a.com/x", "a.com/y", "b.com"}, + Fetch: true, + }) + fetch := tools[1].OfWebFetchTool20250910 + if fetch == nil { + t.Fatalf("tools[1] is not a web_fetch tool") + } + if !sameStrings(fetch.AllowedDomains, []string{"a.com", "b.com"}) { + t.Errorf("fetch AllowedDomains = %v, want deduped [a.com b.com]", fetch.AllowedDomains) + } + }) + + t.Run("social host is search-only via fetch_allowed_domains", func(t *testing.T) { + tools := webSearchTools(&WebSearchConfig{ + AllowedDomains: []string{"helpshift.example/hc", "x.com/thatskygame"}, + FetchAllowedDomains: []string{"helpshift.example"}, + Fetch: true, + }) + search := tools[0].OfWebSearchTool20250305 + if search == nil { + t.Fatalf("tools[0] is not a web_search tool") + } + var searchHasSocial bool + for _, d := range search.AllowedDomains { + if d == "x.com/thatskygame" { + searchHasSocial = true + } + } + if !searchHasSocial { + t.Errorf("search AllowedDomains = %v, want it to include x.com/thatskygame", search.AllowedDomains) + } + + fetch := tools[1].OfWebFetchTool20250910 + if fetch == nil { + t.Fatalf("tools[1] is not a web_fetch tool") + } + if !sameStrings(fetch.AllowedDomains, []string{"helpshift.example"}) { + t.Errorf("fetch AllowedDomains = %v, want only [helpshift.example]", fetch.AllowedDomains) + } + for _, d := range fetch.AllowedDomains { + if strings.HasPrefix(d, "x.com") { + t.Errorf("fetch AllowedDomains leaked the social host: %v", fetch.AllowedDomains) + } + } + }) + t.Run("wire shape carries allowed_domains", func(t *testing.T) { tools := webSearchTools(&WebSearchConfig{ AllowedDomains: []string{"thatgamecompany.helpshift.com/hc"}, @@ -211,6 +259,41 @@ func TestWebSearchTools(t *testing.T) { }) } +func sameStrings(got, want []string) bool { + if len(got) != len(want) { + return false + } + for i := range got { + if got[i] != want[i] { + return false + } + } + return true +} + +func TestFetchHosts(t *testing.T) { + cases := []struct { + name string + in []string + want []string + }{ + {"nil in nil out", nil, nil}, + {"empty in nil out", []string{}, nil}, + {"host passthrough", []string{"example.com"}, []string{"example.com"}}, + {"strip path", []string{"example.com/hc/en"}, []string{"example.com"}}, + {"dedup after strip", []string{"a.com/x", "a.com/y"}, []string{"a.com"}}, + {"preserve order and subdomains", []string{"docs.example.com/a", "example.com"}, []string{"docs.example.com", "example.com"}}, + {"drop empty leading slash", []string{"/oops", "ok.com"}, []string{"ok.com"}}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := fetchHosts(tc.in); !sameStrings(got, tc.want) { + t.Errorf("fetchHosts(%v) = %v, want %v", tc.in, got, tc.want) + } + }) + } +} + func TestEmptyStreamError(t *testing.T) { err := emptyStreamError("max_tokens", 3900, 4000) for _, want := range []string{"output budget exhausted", "3900", "4000"} { diff --git a/config.go b/config.go index 439350a..d1de3cb 100644 --- a/config.go +++ b/config.go @@ -16,11 +16,12 @@ type MCPServer struct { } type WebSearchConfig struct { - AllowedDomains []string `json:"allowed_domains,omitempty"` - BlockedDomains []string `json:"blocked_domains,omitempty"` - MaxUses int `json:"max_uses,omitempty"` - Fetch bool `json:"fetch,omitempty"` - MaxContentTokens int `json:"max_content_tokens,omitempty"` + AllowedDomains []string `json:"allowed_domains,omitempty"` + BlockedDomains []string `json:"blocked_domains,omitempty"` + FetchAllowedDomains []string `json:"fetch_allowed_domains,omitempty"` + MaxUses int `json:"max_uses,omitempty"` + Fetch bool `json:"fetch,omitempty"` + MaxContentTokens int `json:"max_content_tokens,omitempty"` } const ( @@ -126,6 +127,19 @@ func loadAllConfigs(dir string) ([]BotConfig, error) { config.ID) } + if ws := config.WebSearch; ws != nil && len(ws.FetchAllowedDomains) > 0 { + if !ws.Fetch { + InfoLogger.Printf("[%s] web_search.fetch_allowed_domains is set but fetch is disabled: it has no effect", + config.ID) + } + for _, d := range ws.FetchAllowedDomains { + if strings.Contains(d, "/") { + InfoLogger.Printf("[%s] web_search.fetch_allowed_domains entry %q includes a path: web_fetch matches host-only, so the whole host is fetchable", + config.ID, d) + } + } + } + config.ConfigFilePath = validPath configs = append(configs, config) } diff --git a/go-telegram-bot.exe b/go-telegram-bot.exe index a515f12..5ceef7b 100644 Binary files a/go-telegram-bot.exe and b/go-telegram-bot.exe differ