Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.
gateway: a name a fake-IP resolver answers is fetched through its TUN or a local proxy (#2593)
yetone240f3eb4c5fa
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.mdindex cac121363..77c891d0a 100644--- a/docs/subsystems/gateway-routing.md+++ b/docs/subsystems/gateway-routing.md@@ -611,7 +611,9 @@ its URL for an agent with no fetch tool of its own. The URL is the agent's,so it is downloaded as a caller's image is (`webFetchClient`, which is`publicClient`): public addresses only, every resolved address checked andpinned, through at most five redirects each checked the same, wherever-the request came from. It reads for at most 30 seconds and 5 MiB. An+the request came from (a name a fake-IP resolver answers in 198.18.0.0/15+is read through its TUN or a proxy on this computer, as an image is: see+the guard below, #2593). It reads for at most 30 seconds and 5 MiB. AnHTML page is decoded from its charset and given as Markdown (`htmlText`:headings, lists, links made absolute, code; scripts, styles and hiddenelements dropped), other text as it is, anything else is a tool error, and@@ -764,7 +766,7 @@ the row of the entry that id reaches, with its label, under the bare id.[`bare_ids_test.go`](../../internal/gateway/bare_ids_test.go),`TestBareModelIDsSetting` in `internal/gui`, and`internal/gui/tests/bare-ids.test.cjs`.-- A URL magpie downloads but didn't choose itself is downloaded from public addresses only, wherever the request came from ([`image_fetch_guard.go`](../../internal/gateway/image_fetch_guard.go)). That is an image a caller names (an `/v1/images/edits` or `/v1/images/generations` reference image, for an images API that takes files, Gemini's own API or Code Assist: `fetchPicture` in [`draw.go`](../../internal/gateway/draw.go), `publicClient`), a page `web_fetch` reads (`webFetchClient`), and an image or video a vendor answers by URL, which what the model was asked to draw or do can steer (the inline download for a `b64_json` caller in `draw.go` and `/v1/videos/{id}/content` in [`video.go`](../../internal/gateway/video.go), `vendorClient`; generate_image's own download in [`internal/imagemcp`](../../internal/imagemcp/imagemcp.go), `PublicClient`). Every address the host resolves to must be public (`publicAddr`: not loopback, RFC 1918, fc00::/7, link-local with 169.254.169.254, CGNAT 100.64/10, unspecified, multicast, documentation, reserved, Teredo, nor an IPv4-mapped, NAT64 or 6to4 form of one), and the request goes to the address checked, so a name that resolves elsewhere the second time changes nothing; a direct connection is checked again on the address dialled (`guardDialer`). Through a proxy the proxy is sent that address, never the name (an `https://` CONNECT or SOCKS keeps the name for TLS and Host; a plain `http://` URL through an HTTP proxy is asked for by the address in Host too). Each redirect, up to 5, is checked the same, and a caller's image must sniff as an image. A request from this computer gets nothing more: a reverse proxy or relay in front of magpie on the same host makes every request it carries arrive from loopback, so where a request came from is never a reason to reach a private address. Two things are: settings.json's `fetchPrivate` (`settings.FetchPrivate`, none by default, no Settings page, this computer's own and never brought by a sync or restore), a list of names, `name:port`s, IPs and CIDR ranges the user lets magpie download from (`allowFor`; a settings.json that can't be read lets nothing extra in), and for a vendor's URL the `host:port` of that provider's own endpoints, which the user named themselves (a provider on the LAN that serves its images from itself). A vendor's image that can't be downloaded is answered by its URL, never dropped; its video is a 502 naming why. A URL left for the vendor to fetch (a chat model's message) is the vendor's to fetch.+- A URL magpie downloads but didn't choose itself is downloaded from public addresses only, wherever the request came from ([`image_fetch_guard.go`](../../internal/gateway/image_fetch_guard.go)). That is an image a caller names (an `/v1/images/edits` or `/v1/images/generations` reference image, for an images API that takes files, Gemini's own API or Code Assist: `fetchPicture` in [`draw.go`](../../internal/gateway/draw.go), `publicClient`), a page `web_fetch` reads (`webFetchClient`), and an image or video a vendor answers by URL, which what the model was asked to draw or do can steer (the inline download for a `b64_json` caller in `draw.go` and `/v1/videos/{id}/content` in [`video.go`](../../internal/gateway/video.go), `vendorClient`; generate_image's own download in [`internal/imagemcp`](../../internal/imagemcp/imagemcp.go), `PublicClient`). Every address the host resolves to must be public (`publicAddr`, which is [`internal/netguard`](../../internal/netguard/netguard.go)'s `Public`, the rule a provider's icon uses too: not loopback, RFC 1918, fc00::/7, link-local with 169.254.169.254, CGNAT 100.64/10, unspecified, multicast, documentation, reserved, benchmarking, Teredo, nor an IPv4-mapped, NAT64 or 6to4 form of one; an AAAA record Loon synthesizes for an A answer in the same reply, `fd27:712::/96` with that IPv4 address inside, is left out and the A answer judged in its place, `dropLoon`), and the request goes to the address checked, so a name that resolves elsewhere the second time changes nothing; a direct connection is checked again on the address dialled (`guardDialer`). Through a proxy the proxy is sent that address, never the name (an `https://` CONNECT or SOCKS keeps the name for TLS and Host; a plain `http://` URL through an HTTP proxy is asked for by the address in Host too). A fake-IP resolver's answer is the one shape let past that (#2593; ggbdpq, with h4rk8s's and TomXPRIME's reports): Clash's TUN with fake-ip, Clash Verge, Surge's enhanced mode and Loon answer every name in 198.18.0.0/15 (`netguard.FakeIP`), so with one on every download was refused as a private address. A name (never an address literal) whose every answer is in that range is fetched: with no proxy (pure TUN) its fake address is pinned and dialled, the dial guard letting that one address in, and the TUN carries it; through a proxy on this computer (a loopback address or `localhost`, `onThisComputer`) the proxy is given the name, never the fake address, which a real proxy can't route (h4rk8s: a CONNECT by the name went through where one by its 198.18 address failed), and a plain `http://` request's Host is the checked name, never the caller's own; through a proxy elsewhere it is refused. An answer with any other address beside the fake ones (private or public) is judged as any, so a fake one beside 192.168.x is refused. The trust this asks for is the fake-IP resolver's: which host the name stands for, and so where the TUN or the proxy connects, is decided where magpie can't see, and a name that really points at the user's network is reached if that resolver's rules send it there. A provider's icon was already fetched this way (`publicIP` in [`internal/provider/icon.go`](../../internal/provider/icon.go), #252), and shares the rule. `TestFakeIPNameGoesToALocalProxyByName`, `TestFakeIPNameDialledThroughTheTUN`, `TestFakeIPRefusesTheRest` in [`fakeip_test.go`](../../internal/gateway/fakeip_test.go). Each redirect, up to 5, is checked the same, and a caller's image must sniff as an image. A request from this computer gets nothing more: a reverse proxy or relay in front of magpie on the same host makes every request it carries arrive from loopback, so where a request came from is never a reason to reach a private address. Two things are: settings.json's `fetchPrivate` (`settings.FetchPrivate`, none by default, no Settings page, this computer's own and never brought by a sync or restore), a list of names, `name:port`s, IPs and CIDR ranges the user lets magpie download from (`allowFor`; a settings.json that can't be read lets nothing extra in), and for a vendor's URL the `host:port` of that provider's own endpoints, which the user named themselves (a provider on the LAN that serves its images from itself). A vendor's image that can't be downloaded is answered by its URL, never dropped; its video is a 502 naming why. A URL left for the vendor to fetch (a chat model's message) is the vendor's to fetch.## Verificationdiff --git a/internal/gateway/fakeip_test.go b/internal/gateway/fakeip_test.gonew file mode 100644index 000000000..b6b38e787--- /dev/null+++ b/internal/gateway/fakeip_test.go@@ -0,0 +1,252 @@+package gateway++import (+ "context"+ "errors"+ "net"+ "net/http"+ "net/http/httptest"+ "net/netip"+ "strings"+ "sync"+ "testing"++ "github.com/yetone/magpie/internal/settings"+)++// #2593: with Clash's TUN fake-ip (Clash Verge Rev on Windows, TomXPRIME),+// Surge's enhanced mode (h4rk8s) or Loon on, the system resolver answers+// every name in 198.18.0.0/15, and every web_fetch, caller's image and+// vendor's image was refused as "a private address". These tests stand a+// fake resolver in for the fake-IP one, a local server for the TUN or the+// proxy; nothing reaches the network.++// answers points lookupIP at several addresses per name.+func answers(t *testing.T, names map[string][]string) {+ t.Helper()+ was := lookupIP+ lookupIP = func(_ context.Context, host string) ([]netip.Addr, error) {+ as, ok := names[host]+ if !ok {+ return nil, errors.New("no such host in the test: " + host)+ }+ var out []netip.Addr+ for _, a := range as {+ out = append(out, netip.MustParseAddr(a))+ }+ return out, nil+ }+ t.Cleanup(func() { lookupIP = was })+}++// useProxy sets magpie's proxy ("direct" for none) for the test.+func useProxy(t *testing.T, v string) {+ t.Helper()+ was := settings.Load()+ st := was+ st.Proxy = v+ if err := settings.Save(st); err != nil {+ t.Fatal(err)+ }+ t.Cleanup(func() { settings.Save(was) })+}++// tun stands a local server in for the TUN: a direct request's dial to the+// pinned address goes there, once the guard has said the address may be+// dialled. It records the addresses dialled, and fails the test when the+// same dial guard would let in another address in the fake range, or a+// private one.+func tun(t *testing.T, srv *httptest.Server) func() []string {+ t.Helper()+ var mu sync.Mutex+ var dialled []string+ was := dialPinned+ dialPinned = func(ctx context.Context, d *net.Dialer, network, addr string) (net.Conn, error) {+ mu.Lock()+ dialled = append(dialled, addr)+ mu.Unlock()+ if err := d.Control(network, addr, nil); err != nil {+ return nil, err+ }+ for _, other := range []string{"198.18.0.8:80", "198.19.1.1:443", "127.0.0.1:80", "192.168.1.10:80"} {+ if other != addr && d.Control(network, other, nil) == nil {+ t.Errorf("dialling %s, the guard would let %s in too", addr, other)+ }+ }+ return (&net.Dialer{}).DialContext(ctx, network, srv.Listener.Addr().String())+ }+ t.Cleanup(func() { dialPinned = was })+ return func() []string { mu.Lock(); defer mu.Unlock(); return append([]string(nil), dialled...) }+}++// recorder is a proxy on this computer that records what it is asked: the+// URL of a plain request, the authority of a CONNECT.+func fakeProxy(t *testing.T, body string) (*httptest.Server, func() []string) {+ t.Helper()+ var mu sync.Mutex+ var asked []string+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ mu.Lock()+ if r.Method == http.MethodConnect {+ asked = append(asked, "CONNECT "+r.Host)+ } else {+ asked = append(asked, r.URL.String()+" Host="+r.Host)+ }+ mu.Unlock()+ if r.Method == http.MethodConnect {+ w.WriteHeader(http.StatusOK) // and forwards nowhere: TLS fails after+ return+ }+ switch r.URL.Path {+ case "/to-private":+ http.Redirect(w, r, "http://inside.test/", http.StatusFound)+ default:+ w.Header().Set("Content-Type", "text/html; charset=utf-8")+ w.Write([]byte(body))+ }+ }))+ t.Cleanup(srv.Close)+ return srv, func() []string { mu.Lock(); defer mu.Unlock(); return append([]string(nil), asked...) }+}++func guardedGet(u string) error {+ req, _ := http.NewRequest("GET", u, nil)+ resp, err := publicClient.Do(req)+ if err == nil {+ resp.Body.Close()+ }+ return err+}++// Through a proxy on this computer (Clash's 127.0.0.1:7890, Surge's+// 6152), a name the fake-IP resolver answered is handed to the proxy by+// its name: a real proxy can't route the 198.18 address (h4rk8s: CONNECT+// by the name went through, by the fake address didn't). web_fetch reads+// the page; https:// is a CONNECT by name.+func TestFakeIPNameGoesToALocalProxyByName(t *testing.T) {+ fresh(t)+ proxy, asked := fakeProxy(t, "<html><head><title>Example Domain</title></head><body><p>fetched by name</p></body></html>")+ useProxy(t, proxy.URL)+ answers(t, map[string][]string{+ "example.com": {"198.18.0.7"},+ "loon.test": {"198.18.1.57", "fd27:712::c612:139"}, // Loon's AAAA for its own A+ "inside.test": {"198.18.0.9", "192.168.1.10"},+ })+ h := lanGuard(New().Handler())+ got, isErr := fetchText_(t, h, fetchCall("http://example.com/", ""))+ if isErr || !strings.Contains(got, "fetched by name") {+ t.Fatalf("web_fetch of a fake-IP name through a local proxy: %v %q", isErr, got)+ }+ if a := asked(); len(a) != 1 || a[0] != "http://example.com:80/ Host=example.com:80" {+ t.Fatalf("the proxy was asked %q, want the name", a)+ }+ if err := guardedGet("http://loon.test/x.png"); err != nil {+ t.Fatalf("Loon's answer: %v", err)+ }+ if a := asked(); len(a) != 2 || !strings.HasPrefix(a[1], "http://loon.test:80/x.png") {+ t.Fatalf("Loon: the proxy was asked %q", a)+ }+ // https://: the CONNECT names the host, never 198.18+ if err := guardedGet("https://example.com/x.png"); err == nil {+ t.Fatal("a CONNECT that forwards nowhere answered")+ }+ if a := asked(); len(a) != 3 || a[2] != "CONNECT example.com:443" {+ t.Fatalf("https: the proxy was asked %q", a)+ }+ // a request's own Host can't name the proxy another target+ req, _ := http.NewRequest("GET", "http://example.com/", nil)+ req.Host = "169.254.169.254"+ if resp, err := publicClient.Do(req); err == nil {+ resp.Body.Close()+ }+ if a := asked(); len(a) != 4 || a[3] != "http://example.com:80/ Host=example.com:80" {+ t.Fatalf("a foreign Host: the proxy was asked %q", a)+ }+ // a redirect is judged again: one to a name with a private answer+ // beside its fake one is refused before the proxy is asked for it+ if err := guardedGet("http://example.com/to-private"); !errors.Is(err, errPrivateAddress) {+ t.Fatalf("a redirect to a mixed answer: %v", err)+ }+ if a := asked(); len(a) != 5 {+ t.Fatalf("a redirect to a mixed answer reached the proxy: %q", a)+ }+}++// Pure TUN, no proxy set (TomXPRIME: Windows, Clash Verge Rev, no+// HTTPS_PROXY): the fake address is pinned and dialled, and the TUN+// carries it. The dial guard lets that one address in, no other in the+// range.+func TestFakeIPNameDialledThroughTheTUN(t *testing.T) {+ fresh(t)+ useProxy(t, "direct")+ page := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {+ w.Header().Set("Content-Type", "text/plain")+ w.Write([]byte("through the tun " + r.Host))+ }))+ t.Cleanup(page.Close)+ dialled := tun(t, page)+ answers(t, map[string][]string{"example.com": {"198.18.0.7"}, "real.test": {"93.184.216.34", "fd27:712::5db8:d822"}})+ h := lanGuard(New().Handler())+ got, isErr := fetchText_(t, h, fetchCall("http://example.com/", ""))+ if isErr || !strings.Contains(got, "through the tun example.com") {+ t.Fatalf("web_fetch of a fake-IP name with no proxy: %v %q", isErr, got)+ }+ if d := dialled(); len(d) != 1 || d[0] != "198.18.0.7:80" {+ t.Fatalf("dialled %q, want the pinned fake address", d)+ }+ // a real answer is pinned as ever; Loon's AAAA beside it isn't dialled+ if err := guardedGet("http://real.test/"); err != nil {+ t.Fatalf("a real answer: %v", err)+ }+ if d := dialled(); len(d) != 2 || d[1] != "93.184.216.34:80" {+ t.Fatalf("dialled %q", d)+ }+}++// What stays refused, with a proxy on this computer and without: a+// private answer, a fake answer beside a private (or any real) one, a+// fake address written as the URL, and a fake answer through a proxy+// elsewhere. Neither the proxy nor the TUN is asked.+func TestFakeIPRefusesTheRest(t *testing.T) {+ fresh(t)+ page := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("reached")) }))+ t.Cleanup(page.Close)+ dialled := tun(t, page)+ answers(t, map[string][]string{+ "example.com": {"198.18.0.7"},+ "lan.test": {"192.168.1.10"},+ "loop.test": {"127.0.0.1"},+ "meta.test": {"169.254.169.254"},+ "cgnat.test": {"100.64.0.1"},+ "ten.test": {"10.0.0.1"},+ "mixed.test": {"198.18.0.9", "192.168.1.10"},+ "mixed6.test": {"198.18.0.9", "::1"},+ "mixedpub.test": {"198.18.0.9", "93.184.216.34"},+ "loonlan.test": {"fd27:712::a00:1", "10.0.0.1"},+ "loonbare.test": {"fd27:712::c612:139"},+ })+ refused := []string{+ "http://lan.test/", "http://loop.test/", "http://meta.test/latest/meta-data/", "http://cgnat.test/",+ "http://ten.test/", "http://mixed.test/", "http://mixed6.test/", "http://mixedpub.test/",+ "http://loonlan.test/", "http://loonbare.test/", "http://198.18.0.7/", "https://198.18.0.7/",+ "http://[::ffff:198.18.0.7]/",+ }+ proxy, asked := fakeProxy(t, "reached")+ for _, p := range []string{"direct", proxy.URL, "http://localhost:" + proxy.URL[strings.LastIndex(proxy.URL, ":")+1:]} {+ useProxy(t, p)+ for _, u := range refused {+ if err := guardedGet(u); !errors.Is(err, errPrivateAddress) {+ t.Errorf("proxy %s, %s: %v, want the private-address refusal", p, u, err)+ }+ }+ }+ // a proxy elsewhere can't be handed the name for a fake answer+ // (203.0.113.9 is never dialled: the request is refused first)+ useProxy(t, "http://203.0.113.9:3128")+ if err := guardedGet("http://example.com/"); !errors.Is(err, errPrivateAddress) {+ t.Errorf("a fake answer through a proxy elsewhere: %v", err)+ }+ if a, d := asked(), dialled(); len(a) != 0 || len(d) != 0 {+ t.Fatalf("the proxy was asked %q, the TUN dialled %q", a, d)+ }+}diff --git a/internal/gateway/image_fetch_guard.go b/internal/gateway/image_fetch_guard.goindex b91472f34..5584f552d 100644--- a/internal/gateway/image_fetch_guard.go+++ b/internal/gateway/image_fetch_guard.go@@ -13,6 +13,7 @@ import ("syscall""time"+ "github.com/yetone/magpie/internal/netguard""github.com/yetone/magpie/internal/netproxy""github.com/yetone/magpie/internal/provider""github.com/yetone/magpie/internal/settings"@@ -51,6 +52,20 @@ import (// it can't resolve it to another (an https:// CONNECT, SOCKS, or a// plain http:// request whose Host is then the address too);// - each redirect goes through the same, at most imageRedirects of them.+//+// One more shape is let through, a fake-IP resolver's (Clash's TUN with+// fake-ip, Clash Verge, Surge's enhanced mode, Loon; #2593): a name whose+// every answer is in 198.18.0.0/15 (netguard.FakeIP). Pure TUN, no proxy:+// the fake address is pinned and dialled, and the TUN carries it to the+// host the name stands for. Through a proxy on this computer (loopback or+// localhost), the proxy is given the name, never the fake address, which+// it can't route; through a proxy elsewhere it is refused. An address+// literal in the range, or an answer with any other address beside the+// fake ones, is judged as ever. What the name stands for is then the+// fake-IP resolver's to decide, where magpie can't see: a name that+// really points at the user's network is reached if that resolver's+// rules send it there. The same holds for a provider's icon+// (internal/provider/icon.go), which shares netguard.// imageRedirects is how many redirects a caller's image URL is followed// through.@@ -144,56 +159,9 @@ var lookupIP = func(ctx context.Context, host string) ([]netip.Addr, error) {return net.DefaultResolver.LookupNetIP(ctx, "ip", host)}-// specialNets are ranges no caller's image is fetched from beyond those-// net/netip names (loopback, private, link-local, multicast, unspecified).-var specialNets = func() []netip.Prefix {- var out []netip.Prefix- for _, s := range []string{- "0.0.0.0/8", // "this" network- "100.64.0.0/10", // CGNAT, Tailscale- "192.0.0.0/24", // IETF protocol assignments- "192.0.2.0/24", // documentation- "198.18.0.0/15", // benchmarking- "198.51.100.0/24", // documentation- "203.0.113.0/24", // documentation- "240.0.0.0/4", // reserved, broadcast- "::/96", // IPv4-compatible (deprecated)- "100::/64", // discard- "2001::/32", // Teredo- "2001:db8::/32", // documentation- "fec0::/10", // site-local (deprecated)- "64:ff9b:1::/48", // local-use NAT64- } {- out = append(out, netip.MustParsePrefix(s))- }- return out-}()--// publicAddr says a may be fetched a caller's image from.-func publicAddr(a netip.Addr) bool {- a = a.Unmap() // ::ffff:127.0.0.1 is 127.0.0.1- if !a.IsValid() || a.IsLoopback() || a.IsPrivate() || a.IsUnspecified() ||- a.IsLinkLocalUnicast() || a.IsLinkLocalMulticast() || a.IsInterfaceLocalMulticast() || a.IsMulticast() {- return false- }- for _, n := range specialNets {- if n.Contains(a) {- return false- }- }- if a.Is6() {- b := a.As16()- // an IPv4 address inside an IPv6 one: NAT64 (64:ff9b::/96) and- // 6to4 (2002::/16) reach it- if netip.MustParsePrefix("64:ff9b::/96").Contains(a) {- return publicAddr(netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]}))- }- if b[0] == 0x20 && b[1] == 0x02 {- return publicAddr(netip.AddrFrom4([4]byte{b[2], b[3], b[4], b[5]}))- }- }- return true-}+// publicAddr says a may be fetched a caller's image from: netguard's rule,+// shared with a provider's icon.+func publicAddr(a netip.Addr) bool { return netguard.Public(a) }// guardDialer dials only addresses ok allows, checked on the address the// socket is about to connect to.@@ -211,6 +179,12 @@ func guardDialer(ok func(netip.Addr) bool) *net.Dialer {}}+// dialPinned is how a direct request's pinned address is dialled; a test+// stands a local server in for a TUN behind an address it can't reach.+var dialPinned = func(ctx context.Context, d *net.Dialer, network, addr string) (net.Conn, error) {+ return d.DialContext(ctx, network, addr)+}+// privateDialer dials only public addresses.var privateDialer = guardDialer(publicAddr)@@ -233,11 +207,13 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {return nil, err}var addrs []netip.Addr+ literal := falseif a, err := netip.ParseAddr(strings.TrimSuffix(strings.TrimPrefix(host, "["), "]")); err == nil {- addrs = []netip.Addr{a}+ addrs, literal = []netip.Addr{a}, true} else if addrs, err = lookupIP(req.Context(), host); err != nil {return nil, err}+ addrs = dropLoon(addrs)if len(addrs) == 0 {return nil, fmt.Errorf("%s resolves to no address", host)}@@ -247,11 +223,22 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {}allow := allowFor(pt.also)named := allow.host(host, port)- for _, a := range addrs {- if !named && !allow.addr(a) {+ // a name a fake-IP resolver answered (fakeAnswer) is carried by the+ // TUN when dialled directly, its address pinned, or by a proxy on+ // this computer given the name, never the address (byName)+ fake := !literal && fakeAnswer(addrs)+ if !named && !allAllowed(allow, addrs) {+ if !fake {+ return nil, errPrivateAddress+ }+ if proxy != nil && !onThisComputer(proxy) {+ // a proxy elsewhere has no fake-IP resolver of ours to ask:+ // neither the name nor the address is known to reach the+ // host the name stands forreturn nil, errPrivateAddress}}+ byName := fake && proxy != nil && onThisComputer(proxy)var tc *tls.Configif upstreamTLS != nil {tc = upstreamTLS.Clone()@@ -269,12 +256,23 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {}if proxy == nil {pinned := addrs[0].Unmap()- t.DialContext = guardDialer(func(a netip.Addr) bool {- return allow.addr(a) || named && a.Unmap() == pinned- }).DialContext+ d := guardDialer(func(a netip.Addr) bool {+ return allow.addr(a) || (named || fake) && a.Unmap() == pinned+ })+ t.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {+ return dialPinned(ctx, d, network, addr)+ }}out := req.Clone(req.Context())- out.URL.Host = net.JoinHostPort(addrs[0].Unmap().String(), port)+ if byName {+ // the proxy resolves the name itself: it is the fake-IP resolver+ // (or carried by it), and a fake address pinned here is one it+ // can't route (h4rk8s on #2593: CONNECT by the name went through,+ // by its 198.18 address didn't)+ out.URL.Host = net.JoinHostPort(host, port)+ } else {+ out.URL.Host = net.JoinHostPort(addrs[0].Unmap().String(), port)+ }out.Host = req.Hostif out.Host == "" {out.Host = req.URL.Host@@ -282,13 +280,63 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {// a plain http:// URL through an HTTP proxy is sent to it whole, its// host taken from Host: the proxy would resolve the name again, so it// is given the address instead (a site that serves only by name won't- // answer it; https://, and any URL through SOCKS, keep the name)+ // answer it; https://, and any URL through SOCKS, keep the name). The+ // name given on purpose (byName) is the one checked, never a Host of+ // the caller's own.if proxy != nil && req.URL.Scheme == "http" && (proxy.Scheme == "http" || proxy.Scheme == "https") {out.Host = out.URL.Host}return t.RoundTrip(out)}+// allAllowed says every address may be fetched from (public, or the+// user's fetchPrivate).+func allAllowed(allow fetchAllow, addrs []netip.Addr) bool {+ for _, a := range addrs {+ if !allow.addr(a) {+ return false+ }+ }+ return true+}++// fakeAnswer says every address a name resolved to is a fake-IP+// resolver's (netguard.FakeIP). One real address beside them, private or+// public, is not that resolver's answer alone, and is judged as any is.+func fakeAnswer(addrs []netip.Addr) bool {+ for _, a := range addrs {+ if !netguard.FakeIP(a) {+ return false+ }+ }+ return len(addrs) > 0+}++// dropLoon leaves out the AAAA records Loon synthesizes for an A answer+// in the same reply (netguard.LoonPaired): the A answer is judged, and+// pinned, in their place.+func dropLoon(addrs []netip.Addr) []netip.Addr {+ var out []netip.Addr+ for _, a := range addrs {+ if !netguard.LoonPaired(a, addrs) {+ out = append(out, a)+ }+ }+ return out+}++// onThisComputer says proxy listens on this computer (Clash's+// 127.0.0.1:7890, Surge's 127.0.0.1:6152): a loopback address, or+// localhost.+func onThisComputer(proxy *url.URL) bool {+ h := strings.ToLower(proxy.Hostname())+ if h == "localhost" {+ return true+ }+ a, err := netip.ParseAddr(h)+ return err == nil && a.Unmap().IsLoopback()+}+// publicClient downloads a URL magpie didn't choose: from public// addresses only (and fetchPrivate's), through at most imageRedirects// redirects, each checked the same.diff --git a/internal/netguard/netguard.go b/internal/netguard/netguard.gonew file mode 100644index 000000000..3f75d538d--- /dev/null+++ b/internal/netguard/netguard.go@@ -0,0 +1,97 @@+// Package netguard is the one rule for where a download of a URL magpie+// didn't choose may connect: the gateway's (a caller's image, web_fetch, a+// vendor's image or video, generate_image's download) and a provider's+// icon. It judges addresses only; each caller decides what to do with a+// name's answers (pin them, or hand a proxy the name).+package netguard++import "net/netip"++// specialNets are ranges never fetched from beyond those net/netip names+// (loopback, private, link-local, multicast, unspecified).+var specialNets = func() []netip.Prefix {+ var out []netip.Prefix+ for _, s := range []string{+ "0.0.0.0/8", // "this" network+ "100.64.0.0/10", // CGNAT, Tailscale+ "192.0.0.0/24", // IETF protocol assignments+ "192.0.2.0/24", // documentation+ "198.18.0.0/15", // benchmarking; a fake-IP resolver's (FakeIP)+ "198.51.100.0/24", // documentation+ "203.0.113.0/24", // documentation+ "240.0.0.0/4", // reserved, broadcast+ "::/96", // IPv4-compatible (deprecated)+ "100::/64", // discard+ "2001::/32", // Teredo+ "2001:db8::/32", // documentation+ "fec0::/10", // site-local (deprecated)+ "64:ff9b:1::/48", // local-use NAT64+ } {+ out = append(out, netip.MustParsePrefix(s))+ }+ return out+}()++var (+ nat64 = netip.MustParsePrefix("64:ff9b::/96")+ fakeNet = netip.MustParsePrefix("198.18.0.0/15")+ loonNet = netip.MustParsePrefix("fd27:712::/96")+)++// Public says a is on the open internet: not loopback, RFC 1918, fc00::/7,+// link-local (169.254.169.254 among them), CGNAT, unspecified, multicast,+// documentation, reserved, benchmarking, Teredo, nor an IPv4-mapped,+// NAT64 or 6to4 form of one. A fake-IP resolver's answer is not public+// (FakeIP says when it may be fetched from all the same).+func Public(a netip.Addr) bool {+ a = a.Unmap() // ::ffff:127.0.0.1 is 127.0.0.1+ if !a.IsValid() || a.IsLoopback() || a.IsPrivate() || a.IsUnspecified() ||+ a.IsLinkLocalUnicast() || a.IsLinkLocalMulticast() || a.IsInterfaceLocalMulticast() || a.IsMulticast() {+ return false+ }+ for _, n := range specialNets {+ if n.Contains(a) {+ return false+ }+ }+ if a.Is6() {+ b := a.As16()+ // an IPv4 address inside an IPv6 one: NAT64 (64:ff9b::/96) and+ // 6to4 (2002::/16) reach it+ if nat64.Contains(a) {+ return Public(netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]}))+ }+ if b[0] == 0x20 && b[1] == 0x02 {+ return Public(netip.AddrFrom4([4]byte{b[2], b[3], b[4], b[5]}))+ }+ }+ return true+}++// FakeIP says a is in 198.18.0.0/15, RFC 2544's benchmarking range, where+// Clash's TUN mode with fake-ip, Clash Verge, Surge's enhanced mode and+// Loon answer every name, and carry a connection to it to the real host+// the name stands for (#252, #2593). No machine on the user's network+// sits there, so a connection to a fake-IP answer reaches the internet+// host the name stands for, or nothing; but which host that is, the+// fake-IP resolver decides where magpie can't see.+func FakeIP(a netip.Addr) bool { return fakeNet.Contains(a.Unmap()) }++// LoonPaired says a is the AAAA record Loon synthesizes for an A answer+// (its IPv4 address inside fd27:712::/96) and that A answer is among+// answers. Such an a stands for that IPv4 address, which the caller+// judges on its own; a is not dialled. An unpaired one, or any other+// fc00::/7 address, is private as ever.+func LoonPaired(a netip.Addr, answers []netip.Addr) bool {+ if !loonNet.Contains(a) {+ return false+ }+ b := a.As16()+ v4 := netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]})+ for _, o := range answers {+ if o.Unmap() == v4 {+ return true+ }+ }+ return false+}diff --git a/internal/netguard/netguard_test.go b/internal/netguard/netguard_test.gonew file mode 100644index 000000000..f5e03900a--- /dev/null+++ b/internal/netguard/netguard_test.go@@ -0,0 +1,63 @@+package netguard++import (+ "net/netip"+ "testing"+)++func TestPublic(t *testing.T) {+ for s, want := range map[string]bool{+ "127.0.0.1": false, "10.1.2.3": false, "172.16.0.1": false, "192.168.1.1": false,+ "169.254.169.254": false, "100.64.0.1": false, "0.0.0.0": false, "224.0.0.1": false,+ "192.0.2.1": false, "198.51.100.1": false, "203.0.113.1": false, "240.0.0.1": false,+ "198.18.0.0": false, "198.19.255.255": false, "::ffff:198.18.0.7": false, // fake-IP: FakeIP's, not public+ "::1": false, "fc00::1": false, "fe80::1": false, "2001::1": false, "2001:db8::1": false,+ "64:ff9b::7f00:1": false, "2002:c612:7::": false,+ "8.8.8.8": true, "93.184.216.34": true, "198.17.255.255": true, "198.20.0.0": true,+ "2606:4700::1111": true, "64:ff9b::808:808": true,+ } {+ if got := Public(netip.MustParseAddr(s)); got != want {+ t.Errorf("Public(%s) = %v, want %v", s, got, want)+ }+ }+}++func TestFakeIP(t *testing.T) {+ for s, want := range map[string]bool{+ "198.18.0.0": true, "198.18.0.7": true, "198.19.255.255": true, "::ffff:198.18.0.7": true,+ "198.17.255.255": false, "198.20.0.0": false, "127.0.0.1": false, "10.0.0.1": false,+ "fd27:712::c612:7": false, "2002:c612:7::": false,+ } {+ if got := FakeIP(netip.MustParseAddr(s)); got != want {+ t.Errorf("FakeIP(%s) = %v, want %v", s, got, want)+ }+ }+}++func TestLoonPaired(t *testing.T) {+ addrs := func(ss ...string) []netip.Addr {+ var out []netip.Addr+ for _, s := range ss {+ out = append(out, netip.MustParseAddr(s))+ }+ return out+ }+ for _, c := range []struct {+ name string+ a string+ answers []netip.Addr+ want bool+ }{+ {"paired with a fake A", "fd27:712::c612:139", addrs("198.18.1.57", "fd27:712::c612:139"), true},+ {"paired with a real A", "fd27:712::c600:1061", addrs("fd27:712::c600:1061", "198.0.16.97"), true},+ {"paired, the A IPv4-mapped", "fd27:712::c600:1061", addrs("::ffff:198.0.16.97"), true},+ {"unpaired", "fd27:712::c600:1061", addrs("fd27:712::c600:1061"), false},+ {"another A", "fd27:712::c600:1061", addrs("8.8.8.8"), false},+ {"outside Loon's /96", "fd27:712:1::c600:1061", addrs("198.0.16.97"), false},+ {"an IPv4 address", "198.0.16.97", addrs("198.0.16.97"), false},+ } {+ if got := LoonPaired(netip.MustParseAddr(c.a), c.answers); got != c.want {+ t.Errorf("%s: %v, want %v", c.name, got, c.want)+ }+ }+}diff --git a/internal/provider/icon.go b/internal/provider/icon.goindex 9a88f9717..b2a70ab12 100644--- a/internal/provider/icon.go+++ b/internal/provider/icon.go@@ -19,6 +19,7 @@ import ("time""github.com/yetone/magpie/internal/iconproxy"+ "github.com/yetone/magpie/internal/netguard")// A provider's own picture lives in the icons folder beside providers.json,@@ -132,30 +133,22 @@ func publicHost(ctx context.Context, host string) error {return err}-// Loon synthesizes AAAA records by embedding its fake IPv4 in fd27:712::/96.-// Prefer the matching A record already accepted by our IPv4 guard. Do not-// dial the private IPv6 address or exempt the whole ULA prefix: an unpaired-// record (or any other private answer) still fails the check.-var loonIconPrefix = netip.MustParsePrefix("fd27:712::/96")-+// iconIPs is ips without the AAAA records Loon synthesizes for an A+// answer beside them (netguard.LoonPaired; that A answer is judged in+// their place), or an error when any other is not public.func iconIPs(host string, ips []net.IPAddr) ([]net.IPAddr, error) {+ all := make([]netip.Addr, len(ips))+ for i, ip := range ips {+ all[i], _ = netip.AddrFromSlice(ip.IP)+ }var allowed []net.IPAddr- for _, ip := range ips {+ for i, ip := range ips {if publicIP(ip.IP) {allowed = append(allowed, ip)continue}- if addr, ok := netip.AddrFromSlice(ip.IP); ok && loonIconPrefix.Contains(addr) {- paired := false- for _, other := range ips {- if v4 := other.IP.To4(); v4 != nil && publicIP(v4) && v4.Equal(ip.IP[12:]) {- paired = true- break- }- }- if paired {- continue- }+ if netguard.LoonPaired(all[i].Unmap(), all) {+ continue}return nil, errorf("the icon host %s resolves to %s, which is not public", host, ip.IP)}@@ -208,49 +201,17 @@ func publicDial(ctx context.Context, dial func(context.Context, string, string)return nil, lastErr}-// publicIP is true for an address on the open internet: not loopback,-// private, link-local, unspecified, multicast or otherwise reserved.-//-// 198.18.0.0/15, RFC 2544's benchmarking range, is let through: Clash's TUN-// mode with fake-ip and Surge's enhanced mode answer every name with an-// address there and carry the connection to the real host through the proxy-// (#252). No machine on the user's network sits in that range, so a fetch-// aimed at it reaches the internet host the name stands for, not this-// computer or the LAN.+// publicIP is true for an address an icon may be fetched from: one on the+// open internet (netguard.Public, the gateway's rule), or a fake-IP+// resolver's answer in 198.18.0.0/15 (netguard.FakeIP): Clash's TUN mode+// with fake-ip and Surge's enhanced mode answer every name with an+// address there and carry the connection to the real host through the+// proxy (#252). No machine on the user's network sits in that range, so a+// fetch aimed at it reaches the internet host the name stands for, not+// this computer or the LAN.func publicIP(ip net.IP) bool {- if ip.IsLoopback() || ip.IsPrivate() || ip.IsUnspecified() ||- ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() ||- ip.IsInterfaceLocalMulticast() || ip.IsMulticast() {- return false- }- if v4 := ip.To4(); v4 != nil {- // 100.64/10 carrier NAT, 192.0.0.0/24, 192.0.2/24 and friends:- // never a public web host.- switch {- case v4[0] == 100 && v4[1]&0xc0 == 64:- return false- case v4[0] == 192 && v4[1] == 0 && v4[2] == 0:- return false- case v4[0] == 192 && v4[1] == 0 && v4[2] == 2:- return false- case v4[0] == 198 && v4[1] == 51 && v4[2] == 100:- return false- case v4[0] == 203 && v4[1] == 0 && v4[2] == 113:- return false- }- return true- }- // IPv6: unique local (fc00::/7) and 6to4/teredo tunnels are not public- // web hosts either.- if len(ip) == net.IPv6len {- if ip[0]&0xfe == 0xfc {- return false- }- if ip[0] == 0x20 && ip[1] == 0x02 {- return false // 2002::/16 6to4- }- }- return true+ a, ok := netip.AddrFromSlice(ip)+ return ok && (netguard.Public(a) || netguard.FakeIP(a))}// fetchIcon is the download itself, split out so tests can point it at a