Skip to content

Every agent's model. One place. Codex on DeepSeek, Claude Code on Kimi, from the menu bar.

gateway: a name a fake-IP resolver answers is fetched through its TUN or a local proxy (#2593)

yetone240f3eb4c5fa
diff --git a/docs/subsystems/gateway-routing.md b/docs/subsystems/gateway-routing.md
index cac121363..77c891d0a 100644
--- a/docs/subsystems/gateway-routing.md
+++ b/docs/subsystems/gateway-routing.md
@@ -611,7 +611,9 @@ its URL for an agent with no fetch tool of its own. The URL is the agent's,
so it is downloaded as a caller's image is (`webFetchClient`, which is
`publicClient`): public addresses only, every resolved address checked and
pinned, through at most five redirects each checked the same, wherever
-the request came from. It reads for at most 30 seconds and 5 MiB. An
+the request came from (a name a fake-IP resolver answers in 198.18.0.0/15
+is read through its TUN or a proxy on this computer, as an image is: see
+the guard below, #2593). It reads for at most 30 seconds and 5 MiB. An
HTML page is decoded from its charset and given as Markdown (`htmlText`:
headings, lists, links made absolute, code; scripts, styles and hidden
elements dropped), other text as it is, anything else is a tool error, and
@@ -764,7 +766,7 @@ the row of the entry that id reaches, with its label, under the bare id.
[`bare_ids_test.go`](../../internal/gateway/bare_ids_test.go),
`TestBareModelIDsSetting` in `internal/gui`, and
`internal/gui/tests/bare-ids.test.cjs`.
-- A URL magpie downloads but didn't choose itself is downloaded from public addresses only, wherever the request came from ([`image_fetch_guard.go`](../../internal/gateway/image_fetch_guard.go)). That is an image a caller names (an `/v1/images/edits` or `/v1/images/generations` reference image, for an images API that takes files, Gemini's own API or Code Assist: `fetchPicture` in [`draw.go`](../../internal/gateway/draw.go), `publicClient`), a page `web_fetch` reads (`webFetchClient`), and an image or video a vendor answers by URL, which what the model was asked to draw or do can steer (the inline download for a `b64_json` caller in `draw.go` and `/v1/videos/{id}/content` in [`video.go`](../../internal/gateway/video.go), `vendorClient`; generate_image's own download in [`internal/imagemcp`](../../internal/imagemcp/imagemcp.go), `PublicClient`). Every address the host resolves to must be public (`publicAddr`: not loopback, RFC 1918, fc00::/7, link-local with 169.254.169.254, CGNAT 100.64/10, unspecified, multicast, documentation, reserved, Teredo, nor an IPv4-mapped, NAT64 or 6to4 form of one), and the request goes to the address checked, so a name that resolves elsewhere the second time changes nothing; a direct connection is checked again on the address dialled (`guardDialer`). Through a proxy the proxy is sent that address, never the name (an `https://` CONNECT or SOCKS keeps the name for TLS and Host; a plain `http://` URL through an HTTP proxy is asked for by the address in Host too). Each redirect, up to 5, is checked the same, and a caller's image must sniff as an image. A request from this computer gets nothing more: a reverse proxy or relay in front of magpie on the same host makes every request it carries arrive from loopback, so where a request came from is never a reason to reach a private address. Two things are: settings.json's `fetchPrivate` (`settings.FetchPrivate`, none by default, no Settings page, this computer's own and never brought by a sync or restore), a list of names, `name:port`s, IPs and CIDR ranges the user lets magpie download from (`allowFor`; a settings.json that can't be read lets nothing extra in), and for a vendor's URL the `host:port` of that provider's own endpoints, which the user named themselves (a provider on the LAN that serves its images from itself). A vendor's image that can't be downloaded is answered by its URL, never dropped; its video is a 502 naming why. A URL left for the vendor to fetch (a chat model's message) is the vendor's to fetch.
+- A URL magpie downloads but didn't choose itself is downloaded from public addresses only, wherever the request came from ([`image_fetch_guard.go`](../../internal/gateway/image_fetch_guard.go)). That is an image a caller names (an `/v1/images/edits` or `/v1/images/generations` reference image, for an images API that takes files, Gemini's own API or Code Assist: `fetchPicture` in [`draw.go`](../../internal/gateway/draw.go), `publicClient`), a page `web_fetch` reads (`webFetchClient`), and an image or video a vendor answers by URL, which what the model was asked to draw or do can steer (the inline download for a `b64_json` caller in `draw.go` and `/v1/videos/{id}/content` in [`video.go`](../../internal/gateway/video.go), `vendorClient`; generate_image's own download in [`internal/imagemcp`](../../internal/imagemcp/imagemcp.go), `PublicClient`). Every address the host resolves to must be public (`publicAddr`, which is [`internal/netguard`](../../internal/netguard/netguard.go)'s `Public`, the rule a provider's icon uses too: not loopback, RFC 1918, fc00::/7, link-local with 169.254.169.254, CGNAT 100.64/10, unspecified, multicast, documentation, reserved, benchmarking, Teredo, nor an IPv4-mapped, NAT64 or 6to4 form of one; an AAAA record Loon synthesizes for an A answer in the same reply, `fd27:712::/96` with that IPv4 address inside, is left out and the A answer judged in its place, `dropLoon`), and the request goes to the address checked, so a name that resolves elsewhere the second time changes nothing; a direct connection is checked again on the address dialled (`guardDialer`). Through a proxy the proxy is sent that address, never the name (an `https://` CONNECT or SOCKS keeps the name for TLS and Host; a plain `http://` URL through an HTTP proxy is asked for by the address in Host too). A fake-IP resolver's answer is the one shape let past that (#2593; ggbdpq, with h4rk8s's and TomXPRIME's reports): Clash's TUN with fake-ip, Clash Verge, Surge's enhanced mode and Loon answer every name in 198.18.0.0/15 (`netguard.FakeIP`), so with one on every download was refused as a private address. A name (never an address literal) whose every answer is in that range is fetched: with no proxy (pure TUN) its fake address is pinned and dialled, the dial guard letting that one address in, and the TUN carries it; through a proxy on this computer (a loopback address or `localhost`, `onThisComputer`) the proxy is given the name, never the fake address, which a real proxy can't route (h4rk8s: a CONNECT by the name went through where one by its 198.18 address failed), and a plain `http://` request's Host is the checked name, never the caller's own; through a proxy elsewhere it is refused. An answer with any other address beside the fake ones (private or public) is judged as any, so a fake one beside 192.168.x is refused. The trust this asks for is the fake-IP resolver's: which host the name stands for, and so where the TUN or the proxy connects, is decided where magpie can't see, and a name that really points at the user's network is reached if that resolver's rules send it there. A provider's icon was already fetched this way (`publicIP` in [`internal/provider/icon.go`](../../internal/provider/icon.go), #252), and shares the rule. `TestFakeIPNameGoesToALocalProxyByName`, `TestFakeIPNameDialledThroughTheTUN`, `TestFakeIPRefusesTheRest` in [`fakeip_test.go`](../../internal/gateway/fakeip_test.go). Each redirect, up to 5, is checked the same, and a caller's image must sniff as an image. A request from this computer gets nothing more: a reverse proxy or relay in front of magpie on the same host makes every request it carries arrive from loopback, so where a request came from is never a reason to reach a private address. Two things are: settings.json's `fetchPrivate` (`settings.FetchPrivate`, none by default, no Settings page, this computer's own and never brought by a sync or restore), a list of names, `name:port`s, IPs and CIDR ranges the user lets magpie download from (`allowFor`; a settings.json that can't be read lets nothing extra in), and for a vendor's URL the `host:port` of that provider's own endpoints, which the user named themselves (a provider on the LAN that serves its images from itself). A vendor's image that can't be downloaded is answered by its URL, never dropped; its video is a 502 naming why. A URL left for the vendor to fetch (a chat model's message) is the vendor's to fetch.
## Verification
diff --git a/internal/gateway/fakeip_test.go b/internal/gateway/fakeip_test.go
new file mode 100644
index 000000000..b6b38e787
--- /dev/null
+++ b/internal/gateway/fakeip_test.go
@@ -0,0 +1,252 @@
+package gateway
+
+import (
+ "context"
+ "errors"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "net/netip"
+ "strings"
+ "sync"
+ "testing"
+
+ "github.com/yetone/magpie/internal/settings"
+)
+
+// #2593: with Clash's TUN fake-ip (Clash Verge Rev on Windows, TomXPRIME),
+// Surge's enhanced mode (h4rk8s) or Loon on, the system resolver answers
+// every name in 198.18.0.0/15, and every web_fetch, caller's image and
+// vendor's image was refused as "a private address". These tests stand a
+// fake resolver in for the fake-IP one, a local server for the TUN or the
+// proxy; nothing reaches the network.
+
+// answers points lookupIP at several addresses per name.
+func answers(t *testing.T, names map[string][]string) {
+ t.Helper()
+ was := lookupIP
+ lookupIP = func(_ context.Context, host string) ([]netip.Addr, error) {
+ as, ok := names[host]
+ if !ok {
+ return nil, errors.New("no such host in the test: " + host)
+ }
+ var out []netip.Addr
+ for _, a := range as {
+ out = append(out, netip.MustParseAddr(a))
+ }
+ return out, nil
+ }
+ t.Cleanup(func() { lookupIP = was })
+}
+
+// useProxy sets magpie's proxy ("direct" for none) for the test.
+func useProxy(t *testing.T, v string) {
+ t.Helper()
+ was := settings.Load()
+ st := was
+ st.Proxy = v
+ if err := settings.Save(st); err != nil {
+ t.Fatal(err)
+ }
+ t.Cleanup(func() { settings.Save(was) })
+}
+
+// tun stands a local server in for the TUN: a direct request's dial to the
+// pinned address goes there, once the guard has said the address may be
+// dialled. It records the addresses dialled, and fails the test when the
+// same dial guard would let in another address in the fake range, or a
+// private one.
+func tun(t *testing.T, srv *httptest.Server) func() []string {
+ t.Helper()
+ var mu sync.Mutex
+ var dialled []string
+ was := dialPinned
+ dialPinned = func(ctx context.Context, d *net.Dialer, network, addr string) (net.Conn, error) {
+ mu.Lock()
+ dialled = append(dialled, addr)
+ mu.Unlock()
+ if err := d.Control(network, addr, nil); err != nil {
+ return nil, err
+ }
+ for _, other := range []string{"198.18.0.8:80", "198.19.1.1:443", "127.0.0.1:80", "192.168.1.10:80"} {
+ if other != addr && d.Control(network, other, nil) == nil {
+ t.Errorf("dialling %s, the guard would let %s in too", addr, other)
+ }
+ }
+ return (&net.Dialer{}).DialContext(ctx, network, srv.Listener.Addr().String())
+ }
+ t.Cleanup(func() { dialPinned = was })
+ return func() []string { mu.Lock(); defer mu.Unlock(); return append([]string(nil), dialled...) }
+}
+
+// recorder is a proxy on this computer that records what it is asked: the
+// URL of a plain request, the authority of a CONNECT.
+func fakeProxy(t *testing.T, body string) (*httptest.Server, func() []string) {
+ t.Helper()
+ var mu sync.Mutex
+ var asked []string
+ srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ mu.Lock()
+ if r.Method == http.MethodConnect {
+ asked = append(asked, "CONNECT "+r.Host)
+ } else {
+ asked = append(asked, r.URL.String()+" Host="+r.Host)
+ }
+ mu.Unlock()
+ if r.Method == http.MethodConnect {
+ w.WriteHeader(http.StatusOK) // and forwards nowhere: TLS fails after
+ return
+ }
+ switch r.URL.Path {
+ case "/to-private":
+ http.Redirect(w, r, "http://inside.test/", http.StatusFound)
+ default:
+ w.Header().Set("Content-Type", "text/html; charset=utf-8")
+ w.Write([]byte(body))
+ }
+ }))
+ t.Cleanup(srv.Close)
+ return srv, func() []string { mu.Lock(); defer mu.Unlock(); return append([]string(nil), asked...) }
+}
+
+func guardedGet(u string) error {
+ req, _ := http.NewRequest("GET", u, nil)
+ resp, err := publicClient.Do(req)
+ if err == nil {
+ resp.Body.Close()
+ }
+ return err
+}
+
+// Through a proxy on this computer (Clash's 127.0.0.1:7890, Surge's
+// 6152), a name the fake-IP resolver answered is handed to the proxy by
+// its name: a real proxy can't route the 198.18 address (h4rk8s: CONNECT
+// by the name went through, by the fake address didn't). web_fetch reads
+// the page; https:// is a CONNECT by name.
+func TestFakeIPNameGoesToALocalProxyByName(t *testing.T) {
+ fresh(t)
+ proxy, asked := fakeProxy(t, "<html><head><title>Example Domain</title></head><body><p>fetched by name</p></body></html>")
+ useProxy(t, proxy.URL)
+ answers(t, map[string][]string{
+ "example.com": {"198.18.0.7"},
+ "loon.test": {"198.18.1.57", "fd27:712::c612:139"}, // Loon's AAAA for its own A
+ "inside.test": {"198.18.0.9", "192.168.1.10"},
+ })
+ h := lanGuard(New().Handler())
+ got, isErr := fetchText_(t, h, fetchCall("http://example.com/", ""))
+ if isErr || !strings.Contains(got, "fetched by name") {
+ t.Fatalf("web_fetch of a fake-IP name through a local proxy: %v %q", isErr, got)
+ }
+ if a := asked(); len(a) != 1 || a[0] != "http://example.com:80/ Host=example.com:80" {
+ t.Fatalf("the proxy was asked %q, want the name", a)
+ }
+ if err := guardedGet("http://loon.test/x.png"); err != nil {
+ t.Fatalf("Loon's answer: %v", err)
+ }
+ if a := asked(); len(a) != 2 || !strings.HasPrefix(a[1], "http://loon.test:80/x.png") {
+ t.Fatalf("Loon: the proxy was asked %q", a)
+ }
+ // https://: the CONNECT names the host, never 198.18
+ if err := guardedGet("https://example.com/x.png"); err == nil {
+ t.Fatal("a CONNECT that forwards nowhere answered")
+ }
+ if a := asked(); len(a) != 3 || a[2] != "CONNECT example.com:443" {
+ t.Fatalf("https: the proxy was asked %q", a)
+ }
+ // a request's own Host can't name the proxy another target
+ req, _ := http.NewRequest("GET", "http://example.com/", nil)
+ req.Host = "169.254.169.254"
+ if resp, err := publicClient.Do(req); err == nil {
+ resp.Body.Close()
+ }
+ if a := asked(); len(a) != 4 || a[3] != "http://example.com:80/ Host=example.com:80" {
+ t.Fatalf("a foreign Host: the proxy was asked %q", a)
+ }
+ // a redirect is judged again: one to a name with a private answer
+ // beside its fake one is refused before the proxy is asked for it
+ if err := guardedGet("http://example.com/to-private"); !errors.Is(err, errPrivateAddress) {
+ t.Fatalf("a redirect to a mixed answer: %v", err)
+ }
+ if a := asked(); len(a) != 5 {
+ t.Fatalf("a redirect to a mixed answer reached the proxy: %q", a)
+ }
+}
+
+// Pure TUN, no proxy set (TomXPRIME: Windows, Clash Verge Rev, no
+// HTTPS_PROXY): the fake address is pinned and dialled, and the TUN
+// carries it. The dial guard lets that one address in, no other in the
+// range.
+func TestFakeIPNameDialledThroughTheTUN(t *testing.T) {
+ fresh(t)
+ useProxy(t, "direct")
+ page := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ w.Header().Set("Content-Type", "text/plain")
+ w.Write([]byte("through the tun " + r.Host))
+ }))
+ t.Cleanup(page.Close)
+ dialled := tun(t, page)
+ answers(t, map[string][]string{"example.com": {"198.18.0.7"}, "real.test": {"93.184.216.34", "fd27:712::5db8:d822"}})
+ h := lanGuard(New().Handler())
+ got, isErr := fetchText_(t, h, fetchCall("http://example.com/", ""))
+ if isErr || !strings.Contains(got, "through the tun example.com") {
+ t.Fatalf("web_fetch of a fake-IP name with no proxy: %v %q", isErr, got)
+ }
+ if d := dialled(); len(d) != 1 || d[0] != "198.18.0.7:80" {
+ t.Fatalf("dialled %q, want the pinned fake address", d)
+ }
+ // a real answer is pinned as ever; Loon's AAAA beside it isn't dialled
+ if err := guardedGet("http://real.test/"); err != nil {
+ t.Fatalf("a real answer: %v", err)
+ }
+ if d := dialled(); len(d) != 2 || d[1] != "93.184.216.34:80" {
+ t.Fatalf("dialled %q", d)
+ }
+}
+
+// What stays refused, with a proxy on this computer and without: a
+// private answer, a fake answer beside a private (or any real) one, a
+// fake address written as the URL, and a fake answer through a proxy
+// elsewhere. Neither the proxy nor the TUN is asked.
+func TestFakeIPRefusesTheRest(t *testing.T) {
+ fresh(t)
+ page := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("reached")) }))
+ t.Cleanup(page.Close)
+ dialled := tun(t, page)
+ answers(t, map[string][]string{
+ "example.com": {"198.18.0.7"},
+ "lan.test": {"192.168.1.10"},
+ "loop.test": {"127.0.0.1"},
+ "meta.test": {"169.254.169.254"},
+ "cgnat.test": {"100.64.0.1"},
+ "ten.test": {"10.0.0.1"},
+ "mixed.test": {"198.18.0.9", "192.168.1.10"},
+ "mixed6.test": {"198.18.0.9", "::1"},
+ "mixedpub.test": {"198.18.0.9", "93.184.216.34"},
+ "loonlan.test": {"fd27:712::a00:1", "10.0.0.1"},
+ "loonbare.test": {"fd27:712::c612:139"},
+ })
+ refused := []string{
+ "http://lan.test/", "http://loop.test/", "http://meta.test/latest/meta-data/", "http://cgnat.test/",
+ "http://ten.test/", "http://mixed.test/", "http://mixed6.test/", "http://mixedpub.test/",
+ "http://loonlan.test/", "http://loonbare.test/", "http://198.18.0.7/", "https://198.18.0.7/",
+ "http://[::ffff:198.18.0.7]/",
+ }
+ proxy, asked := fakeProxy(t, "reached")
+ for _, p := range []string{"direct", proxy.URL, "http://localhost:" + proxy.URL[strings.LastIndex(proxy.URL, ":")+1:]} {
+ useProxy(t, p)
+ for _, u := range refused {
+ if err := guardedGet(u); !errors.Is(err, errPrivateAddress) {
+ t.Errorf("proxy %s, %s: %v, want the private-address refusal", p, u, err)
+ }
+ }
+ }
+ // a proxy elsewhere can't be handed the name for a fake answer
+ // (203.0.113.9 is never dialled: the request is refused first)
+ useProxy(t, "http://203.0.113.9:3128")
+ if err := guardedGet("http://example.com/"); !errors.Is(err, errPrivateAddress) {
+ t.Errorf("a fake answer through a proxy elsewhere: %v", err)
+ }
+ if a, d := asked(), dialled(); len(a) != 0 || len(d) != 0 {
+ t.Fatalf("the proxy was asked %q, the TUN dialled %q", a, d)
+ }
+}
diff --git a/internal/gateway/image_fetch_guard.go b/internal/gateway/image_fetch_guard.go
index b91472f34..5584f552d 100644
--- a/internal/gateway/image_fetch_guard.go
+++ b/internal/gateway/image_fetch_guard.go
@@ -13,6 +13,7 @@ import (
"syscall"
"time"
+ "github.com/yetone/magpie/internal/netguard"
"github.com/yetone/magpie/internal/netproxy"
"github.com/yetone/magpie/internal/provider"
"github.com/yetone/magpie/internal/settings"
@@ -51,6 +52,20 @@ import (
// it can't resolve it to another (an https:// CONNECT, SOCKS, or a
// plain http:// request whose Host is then the address too);
// - each redirect goes through the same, at most imageRedirects of them.
+//
+// One more shape is let through, a fake-IP resolver's (Clash's TUN with
+// fake-ip, Clash Verge, Surge's enhanced mode, Loon; #2593): a name whose
+// every answer is in 198.18.0.0/15 (netguard.FakeIP). Pure TUN, no proxy:
+// the fake address is pinned and dialled, and the TUN carries it to the
+// host the name stands for. Through a proxy on this computer (loopback or
+// localhost), the proxy is given the name, never the fake address, which
+// it can't route; through a proxy elsewhere it is refused. An address
+// literal in the range, or an answer with any other address beside the
+// fake ones, is judged as ever. What the name stands for is then the
+// fake-IP resolver's to decide, where magpie can't see: a name that
+// really points at the user's network is reached if that resolver's
+// rules send it there. The same holds for a provider's icon
+// (internal/provider/icon.go), which shares netguard.
// imageRedirects is how many redirects a caller's image URL is followed
// through.
@@ -144,56 +159,9 @@ var lookupIP = func(ctx context.Context, host string) ([]netip.Addr, error) {
return net.DefaultResolver.LookupNetIP(ctx, "ip", host)
}
-// specialNets are ranges no caller's image is fetched from beyond those
-// net/netip names (loopback, private, link-local, multicast, unspecified).
-var specialNets = func() []netip.Prefix {
- var out []netip.Prefix
- for _, s := range []string{
- "0.0.0.0/8", // "this" network
- "100.64.0.0/10", // CGNAT, Tailscale
- "192.0.0.0/24", // IETF protocol assignments
- "192.0.2.0/24", // documentation
- "198.18.0.0/15", // benchmarking
- "198.51.100.0/24", // documentation
- "203.0.113.0/24", // documentation
- "240.0.0.0/4", // reserved, broadcast
- "::/96", // IPv4-compatible (deprecated)
- "100::/64", // discard
- "2001::/32", // Teredo
- "2001:db8::/32", // documentation
- "fec0::/10", // site-local (deprecated)
- "64:ff9b:1::/48", // local-use NAT64
- } {
- out = append(out, netip.MustParsePrefix(s))
- }
- return out
-}()
-
-// publicAddr says a may be fetched a caller's image from.
-func publicAddr(a netip.Addr) bool {
- a = a.Unmap() // ::ffff:127.0.0.1 is 127.0.0.1
- if !a.IsValid() || a.IsLoopback() || a.IsPrivate() || a.IsUnspecified() ||
- a.IsLinkLocalUnicast() || a.IsLinkLocalMulticast() || a.IsInterfaceLocalMulticast() || a.IsMulticast() {
- return false
- }
- for _, n := range specialNets {
- if n.Contains(a) {
- return false
- }
- }
- if a.Is6() {
- b := a.As16()
- // an IPv4 address inside an IPv6 one: NAT64 (64:ff9b::/96) and
- // 6to4 (2002::/16) reach it
- if netip.MustParsePrefix("64:ff9b::/96").Contains(a) {
- return publicAddr(netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]}))
- }
- if b[0] == 0x20 && b[1] == 0x02 {
- return publicAddr(netip.AddrFrom4([4]byte{b[2], b[3], b[4], b[5]}))
- }
- }
- return true
-}
+// publicAddr says a may be fetched a caller's image from: netguard's rule,
+// shared with a provider's icon.
+func publicAddr(a netip.Addr) bool { return netguard.Public(a) }
// guardDialer dials only addresses ok allows, checked on the address the
// socket is about to connect to.
@@ -211,6 +179,12 @@ func guardDialer(ok func(netip.Addr) bool) *net.Dialer {
}
}
+// dialPinned is how a direct request's pinned address is dialled; a test
+// stands a local server in for a TUN behind an address it can't reach.
+var dialPinned = func(ctx context.Context, d *net.Dialer, network, addr string) (net.Conn, error) {
+ return d.DialContext(ctx, network, addr)
+}
+
// privateDialer dials only public addresses.
var privateDialer = guardDialer(publicAddr)
@@ -233,11 +207,13 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return nil, err
}
var addrs []netip.Addr
+ literal := false
if a, err := netip.ParseAddr(strings.TrimSuffix(strings.TrimPrefix(host, "["), "]")); err == nil {
- addrs = []netip.Addr{a}
+ addrs, literal = []netip.Addr{a}, true
} else if addrs, err = lookupIP(req.Context(), host); err != nil {
return nil, err
}
+ addrs = dropLoon(addrs)
if len(addrs) == 0 {
return nil, fmt.Errorf("%s resolves to no address", host)
}
@@ -247,11 +223,22 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {
}
allow := allowFor(pt.also)
named := allow.host(host, port)
- for _, a := range addrs {
- if !named && !allow.addr(a) {
+ // a name a fake-IP resolver answered (fakeAnswer) is carried by the
+ // TUN when dialled directly, its address pinned, or by a proxy on
+ // this computer given the name, never the address (byName)
+ fake := !literal && fakeAnswer(addrs)
+ if !named && !allAllowed(allow, addrs) {
+ if !fake {
+ return nil, errPrivateAddress
+ }
+ if proxy != nil && !onThisComputer(proxy) {
+ // a proxy elsewhere has no fake-IP resolver of ours to ask:
+ // neither the name nor the address is known to reach the
+ // host the name stands for
return nil, errPrivateAddress
}
}
+ byName := fake && proxy != nil && onThisComputer(proxy)
var tc *tls.Config
if upstreamTLS != nil {
tc = upstreamTLS.Clone()
@@ -269,12 +256,23 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {
}
if proxy == nil {
pinned := addrs[0].Unmap()
- t.DialContext = guardDialer(func(a netip.Addr) bool {
- return allow.addr(a) || named && a.Unmap() == pinned
- }).DialContext
+ d := guardDialer(func(a netip.Addr) bool {
+ return allow.addr(a) || (named || fake) && a.Unmap() == pinned
+ })
+ t.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
+ return dialPinned(ctx, d, network, addr)
+ }
}
out := req.Clone(req.Context())
- out.URL.Host = net.JoinHostPort(addrs[0].Unmap().String(), port)
+ if byName {
+ // the proxy resolves the name itself: it is the fake-IP resolver
+ // (or carried by it), and a fake address pinned here is one it
+ // can't route (h4rk8s on #2593: CONNECT by the name went through,
+ // by its 198.18 address didn't)
+ out.URL.Host = net.JoinHostPort(host, port)
+ } else {
+ out.URL.Host = net.JoinHostPort(addrs[0].Unmap().String(), port)
+ }
out.Host = req.Host
if out.Host == "" {
out.Host = req.URL.Host
@@ -282,13 +280,63 @@ func (pt publicTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// a plain http:// URL through an HTTP proxy is sent to it whole, its
// host taken from Host: the proxy would resolve the name again, so it
// is given the address instead (a site that serves only by name won't
- // answer it; https://, and any URL through SOCKS, keep the name)
+ // answer it; https://, and any URL through SOCKS, keep the name). The
+ // name given on purpose (byName) is the one checked, never a Host of
+ // the caller's own.
if proxy != nil && req.URL.Scheme == "http" && (proxy.Scheme == "http" || proxy.Scheme == "https") {
out.Host = out.URL.Host
}
return t.RoundTrip(out)
}
+// allAllowed says every address may be fetched from (public, or the
+// user's fetchPrivate).
+func allAllowed(allow fetchAllow, addrs []netip.Addr) bool {
+ for _, a := range addrs {
+ if !allow.addr(a) {
+ return false
+ }
+ }
+ return true
+}
+
+// fakeAnswer says every address a name resolved to is a fake-IP
+// resolver's (netguard.FakeIP). One real address beside them, private or
+// public, is not that resolver's answer alone, and is judged as any is.
+func fakeAnswer(addrs []netip.Addr) bool {
+ for _, a := range addrs {
+ if !netguard.FakeIP(a) {
+ return false
+ }
+ }
+ return len(addrs) > 0
+}
+
+// dropLoon leaves out the AAAA records Loon synthesizes for an A answer
+// in the same reply (netguard.LoonPaired): the A answer is judged, and
+// pinned, in their place.
+func dropLoon(addrs []netip.Addr) []netip.Addr {
+ var out []netip.Addr
+ for _, a := range addrs {
+ if !netguard.LoonPaired(a, addrs) {
+ out = append(out, a)
+ }
+ }
+ return out
+}
+
+// onThisComputer says proxy listens on this computer (Clash's
+// 127.0.0.1:7890, Surge's 127.0.0.1:6152): a loopback address, or
+// localhost.
+func onThisComputer(proxy *url.URL) bool {
+ h := strings.ToLower(proxy.Hostname())
+ if h == "localhost" {
+ return true
+ }
+ a, err := netip.ParseAddr(h)
+ return err == nil && a.Unmap().IsLoopback()
+}
+
// publicClient downloads a URL magpie didn't choose: from public
// addresses only (and fetchPrivate's), through at most imageRedirects
// redirects, each checked the same.
diff --git a/internal/netguard/netguard.go b/internal/netguard/netguard.go
new file mode 100644
index 000000000..3f75d538d
--- /dev/null
+++ b/internal/netguard/netguard.go
@@ -0,0 +1,97 @@
+// Package netguard is the one rule for where a download of a URL magpie
+// didn't choose may connect: the gateway's (a caller's image, web_fetch, a
+// vendor's image or video, generate_image's download) and a provider's
+// icon. It judges addresses only; each caller decides what to do with a
+// name's answers (pin them, or hand a proxy the name).
+package netguard
+
+import "net/netip"
+
+// specialNets are ranges never fetched from beyond those net/netip names
+// (loopback, private, link-local, multicast, unspecified).
+var specialNets = func() []netip.Prefix {
+ var out []netip.Prefix
+ for _, s := range []string{
+ "0.0.0.0/8", // "this" network
+ "100.64.0.0/10", // CGNAT, Tailscale
+ "192.0.0.0/24", // IETF protocol assignments
+ "192.0.2.0/24", // documentation
+ "198.18.0.0/15", // benchmarking; a fake-IP resolver's (FakeIP)
+ "198.51.100.0/24", // documentation
+ "203.0.113.0/24", // documentation
+ "240.0.0.0/4", // reserved, broadcast
+ "::/96", // IPv4-compatible (deprecated)
+ "100::/64", // discard
+ "2001::/32", // Teredo
+ "2001:db8::/32", // documentation
+ "fec0::/10", // site-local (deprecated)
+ "64:ff9b:1::/48", // local-use NAT64
+ } {
+ out = append(out, netip.MustParsePrefix(s))
+ }
+ return out
+}()
+
+var (
+ nat64 = netip.MustParsePrefix("64:ff9b::/96")
+ fakeNet = netip.MustParsePrefix("198.18.0.0/15")
+ loonNet = netip.MustParsePrefix("fd27:712::/96")
+)
+
+// Public says a is on the open internet: not loopback, RFC 1918, fc00::/7,
+// link-local (169.254.169.254 among them), CGNAT, unspecified, multicast,
+// documentation, reserved, benchmarking, Teredo, nor an IPv4-mapped,
+// NAT64 or 6to4 form of one. A fake-IP resolver's answer is not public
+// (FakeIP says when it may be fetched from all the same).
+func Public(a netip.Addr) bool {
+ a = a.Unmap() // ::ffff:127.0.0.1 is 127.0.0.1
+ if !a.IsValid() || a.IsLoopback() || a.IsPrivate() || a.IsUnspecified() ||
+ a.IsLinkLocalUnicast() || a.IsLinkLocalMulticast() || a.IsInterfaceLocalMulticast() || a.IsMulticast() {
+ return false
+ }
+ for _, n := range specialNets {
+ if n.Contains(a) {
+ return false
+ }
+ }
+ if a.Is6() {
+ b := a.As16()
+ // an IPv4 address inside an IPv6 one: NAT64 (64:ff9b::/96) and
+ // 6to4 (2002::/16) reach it
+ if nat64.Contains(a) {
+ return Public(netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]}))
+ }
+ if b[0] == 0x20 && b[1] == 0x02 {
+ return Public(netip.AddrFrom4([4]byte{b[2], b[3], b[4], b[5]}))
+ }
+ }
+ return true
+}
+
+// FakeIP says a is in 198.18.0.0/15, RFC 2544's benchmarking range, where
+// Clash's TUN mode with fake-ip, Clash Verge, Surge's enhanced mode and
+// Loon answer every name, and carry a connection to it to the real host
+// the name stands for (#252, #2593). No machine on the user's network
+// sits there, so a connection to a fake-IP answer reaches the internet
+// host the name stands for, or nothing; but which host that is, the
+// fake-IP resolver decides where magpie can't see.
+func FakeIP(a netip.Addr) bool { return fakeNet.Contains(a.Unmap()) }
+
+// LoonPaired says a is the AAAA record Loon synthesizes for an A answer
+// (its IPv4 address inside fd27:712::/96) and that A answer is among
+// answers. Such an a stands for that IPv4 address, which the caller
+// judges on its own; a is not dialled. An unpaired one, or any other
+// fc00::/7 address, is private as ever.
+func LoonPaired(a netip.Addr, answers []netip.Addr) bool {
+ if !loonNet.Contains(a) {
+ return false
+ }
+ b := a.As16()
+ v4 := netip.AddrFrom4([4]byte{b[12], b[13], b[14], b[15]})
+ for _, o := range answers {
+ if o.Unmap() == v4 {
+ return true
+ }
+ }
+ return false
+}
diff --git a/internal/netguard/netguard_test.go b/internal/netguard/netguard_test.go
new file mode 100644
index 000000000..f5e03900a
--- /dev/null
+++ b/internal/netguard/netguard_test.go
@@ -0,0 +1,63 @@
+package netguard
+
+import (
+ "net/netip"
+ "testing"
+)
+
+func TestPublic(t *testing.T) {
+ for s, want := range map[string]bool{
+ "127.0.0.1": false, "10.1.2.3": false, "172.16.0.1": false, "192.168.1.1": false,
+ "169.254.169.254": false, "100.64.0.1": false, "0.0.0.0": false, "224.0.0.1": false,
+ "192.0.2.1": false, "198.51.100.1": false, "203.0.113.1": false, "240.0.0.1": false,
+ "198.18.0.0": false, "198.19.255.255": false, "::ffff:198.18.0.7": false, // fake-IP: FakeIP's, not public
+ "::1": false, "fc00::1": false, "fe80::1": false, "2001::1": false, "2001:db8::1": false,
+ "64:ff9b::7f00:1": false, "2002:c612:7::": false,
+ "8.8.8.8": true, "93.184.216.34": true, "198.17.255.255": true, "198.20.0.0": true,
+ "2606:4700::1111": true, "64:ff9b::808:808": true,
+ } {
+ if got := Public(netip.MustParseAddr(s)); got != want {
+ t.Errorf("Public(%s) = %v, want %v", s, got, want)
+ }
+ }
+}
+
+func TestFakeIP(t *testing.T) {
+ for s, want := range map[string]bool{
+ "198.18.0.0": true, "198.18.0.7": true, "198.19.255.255": true, "::ffff:198.18.0.7": true,
+ "198.17.255.255": false, "198.20.0.0": false, "127.0.0.1": false, "10.0.0.1": false,
+ "fd27:712::c612:7": false, "2002:c612:7::": false,
+ } {
+ if got := FakeIP(netip.MustParseAddr(s)); got != want {
+ t.Errorf("FakeIP(%s) = %v, want %v", s, got, want)
+ }
+ }
+}
+
+func TestLoonPaired(t *testing.T) {
+ addrs := func(ss ...string) []netip.Addr {
+ var out []netip.Addr
+ for _, s := range ss {
+ out = append(out, netip.MustParseAddr(s))
+ }
+ return out
+ }
+ for _, c := range []struct {
+ name string
+ a string
+ answers []netip.Addr
+ want bool
+ }{
+ {"paired with a fake A", "fd27:712::c612:139", addrs("198.18.1.57", "fd27:712::c612:139"), true},
+ {"paired with a real A", "fd27:712::c600:1061", addrs("fd27:712::c600:1061", "198.0.16.97"), true},
+ {"paired, the A IPv4-mapped", "fd27:712::c600:1061", addrs("::ffff:198.0.16.97"), true},
+ {"unpaired", "fd27:712::c600:1061", addrs("fd27:712::c600:1061"), false},
+ {"another A", "fd27:712::c600:1061", addrs("8.8.8.8"), false},
+ {"outside Loon's /96", "fd27:712:1::c600:1061", addrs("198.0.16.97"), false},
+ {"an IPv4 address", "198.0.16.97", addrs("198.0.16.97"), false},
+ } {
+ if got := LoonPaired(netip.MustParseAddr(c.a), c.answers); got != c.want {
+ t.Errorf("%s: %v, want %v", c.name, got, c.want)
+ }
+ }
+}
diff --git a/internal/provider/icon.go b/internal/provider/icon.go
index 9a88f9717..b2a70ab12 100644
--- a/internal/provider/icon.go
+++ b/internal/provider/icon.go
@@ -19,6 +19,7 @@ import (
"time"
"github.com/yetone/magpie/internal/iconproxy"
+ "github.com/yetone/magpie/internal/netguard"
)
// A provider's own picture lives in the icons folder beside providers.json,
@@ -132,30 +133,22 @@ func publicHost(ctx context.Context, host string) error {
return err
}
-// Loon synthesizes AAAA records by embedding its fake IPv4 in fd27:712::/96.
-// Prefer the matching A record already accepted by our IPv4 guard. Do not
-// dial the private IPv6 address or exempt the whole ULA prefix: an unpaired
-// record (or any other private answer) still fails the check.
-var loonIconPrefix = netip.MustParsePrefix("fd27:712::/96")
-
+// iconIPs is ips without the AAAA records Loon synthesizes for an A
+// answer beside them (netguard.LoonPaired; that A answer is judged in
+// their place), or an error when any other is not public.
func iconIPs(host string, ips []net.IPAddr) ([]net.IPAddr, error) {
+ all := make([]netip.Addr, len(ips))
+ for i, ip := range ips {
+ all[i], _ = netip.AddrFromSlice(ip.IP)
+ }
var allowed []net.IPAddr
- for _, ip := range ips {
+ for i, ip := range ips {
if publicIP(ip.IP) {
allowed = append(allowed, ip)
continue
}
- if addr, ok := netip.AddrFromSlice(ip.IP); ok && loonIconPrefix.Contains(addr) {
- paired := false
- for _, other := range ips {
- if v4 := other.IP.To4(); v4 != nil && publicIP(v4) && v4.Equal(ip.IP[12:]) {
- paired = true
- break
- }
- }
- if paired {
- continue
- }
+ if netguard.LoonPaired(all[i].Unmap(), all) {
+ continue
}
return nil, errorf("the icon host %s resolves to %s, which is not public", host, ip.IP)
}
@@ -208,49 +201,17 @@ func publicDial(ctx context.Context, dial func(context.Context, string, string)
return nil, lastErr
}
-// publicIP is true for an address on the open internet: not loopback,
-// private, link-local, unspecified, multicast or otherwise reserved.
-//
-// 198.18.0.0/15, RFC 2544's benchmarking range, is let through: Clash's TUN
-// mode with fake-ip and Surge's enhanced mode answer every name with an
-// address there and carry the connection to the real host through the proxy
-// (#252). No machine on the user's network sits in that range, so a fetch
-// aimed at it reaches the internet host the name stands for, not this
-// computer or the LAN.
+// publicIP is true for an address an icon may be fetched from: one on the
+// open internet (netguard.Public, the gateway's rule), or a fake-IP
+// resolver's answer in 198.18.0.0/15 (netguard.FakeIP): Clash's TUN mode
+// with fake-ip and Surge's enhanced mode answer every name with an
+// address there and carry the connection to the real host through the
+// proxy (#252). No machine on the user's network sits in that range, so a
+// fetch aimed at it reaches the internet host the name stands for, not
+// this computer or the LAN.
func publicIP(ip net.IP) bool {
- if ip.IsLoopback() || ip.IsPrivate() || ip.IsUnspecified() ||
- ip.IsLinkLocalUnicast() || ip.IsLinkLocalMulticast() ||
- ip.IsInterfaceLocalMulticast() || ip.IsMulticast() {
- return false
- }
- if v4 := ip.To4(); v4 != nil {
- // 100.64/10 carrier NAT, 192.0.0.0/24, 192.0.2/24 and friends:
- // never a public web host.
- switch {
- case v4[0] == 100 && v4[1]&0xc0 == 64:
- return false
- case v4[0] == 192 && v4[1] == 0 && v4[2] == 0:
- return false
- case v4[0] == 192 && v4[1] == 0 && v4[2] == 2:
- return false
- case v4[0] == 198 && v4[1] == 51 && v4[2] == 100:
- return false
- case v4[0] == 203 && v4[1] == 0 && v4[2] == 113:
- return false
- }
- return true
- }
- // IPv6: unique local (fc00::/7) and 6to4/teredo tunnels are not public
- // web hosts either.
- if len(ip) == net.IPv6len {
- if ip[0]&0xfe == 0xfc {
- return false
- }
- if ip[0] == 0x20 && ip[1] == 0x02 {
- return false // 2002::/16 6to4
- }
- }
- return true
+ a, ok := netip.AddrFromSlice(ip)
+ return ok && (netguard.Public(a) || netguard.FakeIP(a))
}
// fetchIcon is the download itself, split out so tests can point it at a