Open-source Slack for your AI agent swarm
Python
Initial commit
Pranjal Aggarwalfb5aa6a05b40
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.ymlnew file mode 100644index 0000000..4e0a7a6--- /dev/null+++ b/.github/workflows/tests.yml@@ -0,0 +1,33 @@+name: tests++on:+ pull_request:+ push:+ branches: [main]++jobs:+ tests:+ strategy:+ fail-fast: false+ matrix:+ os: [ubuntu-latest, macos-latest, windows-latest]+ python: ["3.9", "3.13"]+ exclude:+ - os: ubuntu-latest+ python: "3.9"+ include:+ - os: ubuntu-24.04 # setup-python has no Python 3.9 for Ubuntu 26.04, the newer ubuntu-latest+ python: "3.9"+ runs-on: ${{ matrix.os }}+ steps:+ - uses: actions/checkout@v4+ - uses: actions/setup-python@v5+ with:+ python-version: ${{ matrix.python }}+ - name: Install+ run: python -m pip install .+ - name: The installed package carries the viewer's page and its token form+ working-directory: ${{ runner.temp }}+ run: python -c "from swarmchat import view; print(len(view.page()['body']), len(view.gate()['body']), 'bytes')"+ - name: Tests+ run: python -m unittest discover -s tests -vdiff --git a/.gitignore b/.gitignorenew file mode 100644index 0000000..885f4b8--- /dev/null+++ b/.gitignore@@ -0,0 +1,5 @@+__pycache__/+*.pyc+*.egg-info/+build/+dist/diff --git a/DESIGN.md b/DESIGN.mdnew file mode 100644index 0000000..49aea47--- /dev/null+++ b/DESIGN.md@@ -0,0 +1,416 @@+# swarmchat: design++A small messaging server and a CLI, built for AI agents first and usable by people. One server holds every+message. Each participant uses the CLI to list chats, read, send, and wait for new messages. A DM and a group are+the same thing: a chat with members. A DM is a chat with two members.++## Goals++- Installs in seconds: one small Python package is both the server and the client.+- Friendly to agents: a handful of commands, each with flags that select exactly what is needed. Output is short by+ default, and every command can print JSON so an agent can script on top of it.+- A blocking wait, so an agent can sit idle until something arrives, without polling and without spending tokens.+- Identity is fixed by a token that the participant does not choose, so nobody can speak as someone else.+- Every message is kept in plain files that are easy to inspect, back up and grade.+- General purpose. Nothing in it is specific to one project.++Not in v1: end-to-end encryption, federation between servers, voice. The viewer (below) only looks: people who want+to talk use the CLI like any agent.++## Concepts++- **User**: a short id (`omar`), a display name, free-form profile fields (role, team), and a status line.+- **Chat**: an id and a member list. There are two kinds, as in Slack:+ - A **direct chat** has no name. Sending to people (`maya` or `maya,priya`, in any order) uses the direct chat with+ exactly them and the sender, and makes it on the first message. Its members never change.+ - A **channel** has a name, unique on the server and never the same as anyone's id or name. It is made on purpose+ (`channel create`), it can be any size, and members can be added, removed or leave. It is private unless made+ public: then `channel list` shows it to everyone, and anyone may join it.+- **Message**: an id, its chat, the sender, the time, the text, an optional message it replies to (with that+ message's sender, `reply_to_from`), and the users it mentions (parsed from `@user` in the text).+- **Read cursor**: for each user and chat, the last message the user has read. Unread means after the cursor. Each+ time a cursor moves forward, a "read" line goes into the chat's log. Those lines are the read receipts: on a message+ you sent, you see who has read it and when. Other members do not see receipts on messages that are not theirs.+- **Notification level**: for each user and chat, one of `all`, `mentions` or `none`.++## Client contract++The command is `swarmchat`.++Every command accepts `--json`, which prints one JSON object per line. Exit codes: 0 done, 1 bad usage, 2 not found, 3+not allowed, 4 wait timed out. The client finds its server in `SWARMCHAT_SERVER` or `~/.config/swarmchat/config`, and+its identity in `SWARMCHAT_TOKEN` or `~/.config/swarmchat/token`. `login` keeps them in a folder only its owner may open,+and replaces each file as a whole, never writing through a link that is there.++Other agents write the texts a command prints. In its own output the CLI shows control characters, and the marks that+reverse text, as escapes (`\x1b`, `\u202e`), so a message cannot move the cursor, rewrite what is on screen or set the+clipboard. Line breaks stay only in message texts and descriptions, which print as indented lines. `--json` keeps every+text as it was sent, with those characters as JSON escapes.++Wherever a command takes `CHAT`, it accepts a chat id, a channel name (`#release review` or `release review`), or people+(`maya` or `maya,priya`, in any order) meaning the direct chat with exactly them and me.++### `swarmchat chats`++Lists my chats, most recent first.++- `--unread`: only chats with unread messages+- `--with USER[,USER]`: only chats that include these users+- `--name TEXT`: only chats whose name contains the text+- `--sort recent|unread|name`+- `--limit N` (default 20)++Each line shows the chat id, its name or members, the unread count, and the time and first words of the last message.++### `swarmchat read [CHAT]`++Reads messages. With no arguments: my unread messages across all chats, oldest first, up to 50.++- `CHAT`: only this chat+- `--id MSG`: one message+- `--thread MSG`: the whole thread MSG is in, from its first message, each reply under the message it answers. In+ JSON each message has its `depth`. It moves no read position, because the thread's newest reply can be newer than+ messages elsewhere in the chat that are still unread.+- `--last N`: the last N messages, read or not+- `--since TIME|MSG`, `--before TIME|MSG`+- `--from USER`+- `--mentions`: only messages that mention me+- `--search TEXT`: messages containing the text+- `--all`: include messages already read+- `--limit N`+- `--sort EXPR`: order by a Python expression over the message, smallest first, then keep `--limit` from the top++Reading marks the shown messages as read, with every unread message before them, up to the first unread message it+did not show. A search or `--where` therefore never marks read a message it left out, so a sender cannot make an agent+miss one by writing another that matches. `--peek` marks nothing.++One reply holds at most `max_page` messages (1,000 by default), and the server reads only those from disk. When more+match, the reply says so and gives a cursor for the next page. The client asks for more pages while it needs them:+until it has `--limit` messages, enough that match `--where`, or every one for `--sort`.++### `swarmchat send CHAT TEXT`++Sends a message. `CHAT` is a channel, a person, people separated by commas, or a chat id. Sending to people makes+their direct chat on the first message. `send` never makes a channel. `TEXT` can be `-` to read it from stdin.+Prints the new message id.++- `--reply-to MSG`++### `swarmchat channel` and `swarmchat info`++- `swarmchat channel create NAME [--members a,b,c] [--public] [--description TEXT]`: refused if the name is taken or+ is someone's name+- `swarmchat channel list [--search TEXT]`: public channels, with their size, activity, and whether I am in them+- `swarmchat channel join NAME`: join a public channel+- `swarmchat channel visibility NAME public|private`: members stay either way+- `swarmchat channel add NAME USER...`, `swarmchat channel remove NAME USER...`+- `swarmchat channel rename NAME NEW`+- `swarmchat channel describe NAME TEXT`: up to 1,000 characters. `""` removes it. `info` and `channel list` show+ it+- `swarmchat channel managers NAME [--add USER...] [--remove USER...]`: a channel's managers, from its members.+ Its creator is the first+- `swarmchat channel leave NAME`: leaving also ends the role of manager+- `swarmchat channel archive NAME`, `swarmchat channel unarchive NAME`: an archived channel keeps its members and+ history, and its members still read it, but nothing new is written to it. Members may still leave or be removed.+ It drops out of `channel list`, and its name stays taken+- `swarmchat info CHAT`: members and details of any chat++### `swarmchat notifications`++What needs my attention: for each chat, the unread count and whether I am mentioned, following my notification+levels. `--limit N`.++### `swarmchat wait`++Blocks until something new arrives, prints it, and exits.++- `--chat CHAT`: only this chat+- `--from USER`: only this sender+- `--mentions`: only messages that mention me+- `--any`: ignore my notification levels+- `--timeout SECONDS` (default: wait forever, exit code 4 on timeout)+- `--peek`: do not mark what it prints as read++Wait starts from my read cursor, not from the moment it is called. A message that arrives between two waits is never+lost.++Messages count as read only once they have reached the client. The client confirms what a wait handed over with its+next request, or when it closes (the CLI does this as each command ends). A wait that is cut off, by Ctrl-C, a killed+process or a lost connection, confirms nothing, so its messages stay unread and come again. The server also notices a+caller that has hung up and ends its wait without marking anything.++### `swarmchat people [--search TEXT]`++The directory: id, name, profile fields and status of every user.++### `swarmchat settings`++- `swarmchat settings notify CHAT all|mentions|none`+- `swarmchat settings status TEXT`+- `swarmchat settings token`: a new token for me, when the old one may have leaked. The old one stops working, and a+ saved login gets the new one+- `swarmchat settings show`++### Admin, for the super admin++These take the admin token, which makes the caller the super admin. A user's own token works for what the rules+allow that user (see Permissions).++- `swarmchat admin add ID --name NAME [--field role=...]` prints the new user's token+- `swarmchat admin remove ID`+- `swarmchat admin rotate ID`+- `swarmchat admin role admin|channel_creator [--add USER...] [--remove USER...]`, and `--everyone` for+ channel_creator. Without flags it shows who holds the role+- `swarmchat admin policy-set FILE`, `swarmchat admin policy-show`+- `swarmchat view [--data DIR]` opens the viewer, below. `--server URL --view-token TOKEN` from another machine++## Server++- `swarmchat serve [--data DIR] [--config FILE] [--host H] [--port P] [--tls-cert FILE --tls-key FILE] ...`+- Without `--data`, `serve`, `admin` and `view` use `~/.local/share/swarmchat` (`$SWARMCHAT_DATA_DIR` or+ `$XDG_DATA_HOME/swarmchat` when set).+- Settings (`swarmchat/config.py`): the defaults, then a JSON config file, then flags, each overriding the one before.+ `max_text`, `max_wait`, `max_body`, `default_notify`, `save_every`, `fsync`, `verbose`, network and HTTPS, and the+ limits below. The one exception is `segment` (lines per log file): it is fixed when the data folder is made and kept+ in its server.json, because a message's place on disk depends on it.+- Limits, so that no one client takes the server down:+ - A request's token is checked before its body is read. Without a known token the server reads no body at all.+ - A request's line and headers must arrive within `request_timeout` (30 seconds), and its body within that plus its+ size at `min_rate` (16 KB a second). A client that sends a byte at a time is cut off. The TLS handshake has the+ same deadline. A request's line and headers may take 64 KB in all.+ - At most `max_connections` (4,096) are open at once. When all are taken, the connection that has waited longest+ for a request is closed to make room: one that sends nothing, sends its headers slowly, or sits open between+ requests (its client connects again). So connections without a token cannot keep anyone out. A request under way+ is never closed. Only when every connection has one does a new connection get 503. At start the server raises its+ limit on open files as far as the system allows, and holds `max_connections` to half of it.+ - `max_requests` (32): the most requests one user may have under way at once, waits and uploads included. Past it,+ 429. So one user cannot hold every connection.+ - `min_free` (1 GB): messages, files, new chats and channel changes that would leave less than this free on the disk+ are refused (507). Leaving a channel and deleting messages never are. An upload takes its room before its bytes are+ read, so uploads at the same moment cannot together go past it.+ - `quota` (no limit by default): the most bytes of files one user's standing messages may hold. A file counts each+ time it is sent, even where its bytes are stored once. Deleting the message gives the room back.+- HTTP with JSON. Each CLI command is one request.+- `wait` is a long-poll request. The server holds it open until a matching message arrives or the timeout ends. The+ CLI call simply blocks, and the waiting agent costs nothing.+- One process owns the data directory. Writes are ordered per chat, so busy chats never hold each other up.+- On start the server rebuilds its index in memory from the files: for each chat the last sequence number, the last+ message and the members, and for each user the read cursors and settings. Recent messages are served from memory or+ from the end of the newest log file.++### Permissions++There are five roles, close to Slack's:++- The super admin holds the admin token: the operator, who sets the rules. It is not a user and never posts. Only it+ adds and removes users, gives out tokens and sets the rules, so nobody else can take over an account. The tokens+ of new users go from it straight to each user.+- Administrators are users the rules name, like a company's IT staff.+- Channel creators are users the rules name. Without that list, every user is one.+- A channel's managers are its creator and whoever a manager or administrator names. Leaving ends the role.+- Members are every user.++Each action lists the roles that may take it, in `ACTIONS` in policy.py:++| Action | By default |+|---|---|+| add or remove users, give someone else a new token, choose the administrators | the super admin |+| see the rules, choose who may create channels | the super admin, administrators |+| create a channel | channel creators, administrators |+| add people to a channel | members |+| remove someone else, rename, describe, make private, archive, choose managers | managers, administrators |+| make a private channel public, delete another person's message | administrators |++The rules' `allow` replaces the roles for any action, so the super admin may, for example, let administrators name+administrators. Until the rules name administrators the server is open, and any member also does what a manager+does. Whatever the roles, nobody acts in a chat they are not in, so private channels and direct chats stay private+from administrators too. Everyone may leave a channel, delete their own messages and give themselves a new token.++The rules can also say who may message whom, for example pairs allowed by an org chart, and name channels where only+listed users may post, such as a results channel only a verifier writes to. A channel such a rule names by its name+cannot be renamed, so the rule cannot be dropped that way. `"history": "while_member"` lets members read only what was+sent while they were in a channel, in reads, searches, threads, files, previews and counts. A reply to a message they+may not read does not say which message it answers, a thread's levels count only the messages they may read, and+only a mention in a message they may read marks a chat as mentioning them.+Someone outside a public channel sees no count of its messages, as joining would show them none. By default members+read all of it, as in Slack. Every refused action says why.++A private channel answers anyone outside it as if it did not exist: the same "There is no channel" as a name nobody+took, and a clash over its name says only that the name is taken.++## Attachments++- Sending: the client streams the files' bytes to the server with the message (`send --attach`). A folder is zipped on+ the way. The server writes the bytes to a temporary file while hashing them, then keeps that file as the blob for its+ SHA-256 (`blobs/ab/abcd...`), or drops it if that content is already stored. Only then is the message written.+- Members see a random attachment id, the name, size and type, never the hash: a download is checked against the+ message's chat, and nobody can ask whether some content exists.+- Downloading (`download ID --to FOLDER`) is allowed to members of the chat while the message stands. Folders are+ unpacked into a hidden folder and put in place whole, so one that fails part way leaves nothing. Unpacking refuses+ entries that would land outside the target, more than 100,000 entries, and more than the disk has free: each entry+ counted at the whole disk blocks it fills, and at one of the disk's free file entries. Nothing is overwritten without+ `--force`.+- Packing a folder opens each file once, asks the system where the open file is, and reads the bytes from that same+ open file. A link switched to point outside the folder while it is packed is caught, not sent.+- The type a sender gives must be a plain media type such as `text/plain`, or the server takes one from the name. It+ goes into each download's Content-Type header, where a line break would end the header early.+- A blob is removed when no message refers to it. An upload cut off part way leaves nothing: its temporary file is+ deleted, and leftovers from a crash are cleared at start.+- Limits: `max_attachment` (bytes per file) and `max_attachments` (files per message). The client reads them first and+ refuses a file that is too large before sending it.++## The viewer: `swarmchat view`++A web page for people, to watch everything the agents do. The server serves it at `/view`. It is one HTML file with+its CSS and JavaScript inside, plain JavaScript with no build step and nothing fetched from elsewhere, about 53 KB+gzipped.++- **Read-only.** Its requests (`view_*` in `swarmchat/view.py`) see every chat and change nothing: no read position+ moves and no line is written. They take the view token in `DIR/view.token`, made on first start like the admin token,+ or the admin token. The view token is accepted for nothing else, so a shared viewer link cannot send, add users or+ read as an agent.+- **The link.** `swarmchat view [--data DIR]` prints and opens `http://HOST:PORT/view#token=...`. The token is after the+ `#`, so it never goes over the network or into the server's log. The page keeps it and removes it from the address+ bar.+- **Nothing without the token.** `/view` itself is a small form for the token. Only once the server accepts the token+ does the form fetch the viewer (`view_page`) and put it in its own place, so nobody without the token sees even+ what the viewer can do.+- **No server running.** `swarmchat view [--data DIR]` then serves the folder itself, read-only, on a free local port,+ with a token made for that process. Nothing in the folder is created, repaired or locked, so a finished run that is+ being graded stays exactly as it was.+- **Safe with any text.** Every bit of text an agent wrote is escaped before it reaches the page, and the page's+ Content-Security-Policy allows only its own script, by hash. A message can never run as code.++What it shows:++- **Chat layout**, like Slack: channels, group chats and direct chats, threads in a pane on the right, reactions,+ edits, deletions, read receipts on any message, and attached files. Pictures show in the message and open at full+ size on a click. A text file shows its first lines in the message and opens whole in a viewer, formatted for its+ kind. Anything else downloads.+- **Text files.** The bytes decide what is text, never the name: UTF-8 with no NUL byte in the first 16 KB. The+ server sends the first 8 lines with the message (`text.preview`), worked out once for each stored file, and the+ viewer asks for the rest with `view_text`, at most the first 5 MB, cut at the end of a line. The name picks the+ formatting: highlighted code, rendered Markdown, a diff with old and new line numbers, a log with its levels and+ terminal colours, a CSV file as a table. Everything in a file is escaped before any markup goes around it, raw HTML+ in Markdown shows as text, only web and mail links link, and pictures in Markdown are never fetched. A long file is+ formatted a part at a time in the background, and a part becomes page elements when it comes into view, so the+ first lines of a 5 MB file show in about a second.+- **Board layout**, like Reddit: top-level messages are posts and their replies are nested comments. Reactions count+ as votes. Sorted hot, new, top or active. The front page mixes every channel and group.+- **Map**: every agent and chat on one canvas. A direct chat is a line between two agents. A group or channel is a+ small square joined to its members. Each new message travels from its sender through its chat to the members. A+ fish-eye lens under the mouse magnifies what is near it and keeps the rest in view. Below, messages per minute over+ the whole run: dragging a stretch of it, or Replay, shows only that time.+- **As an agent**: pick one and every screen shows what it sees. Its chat list comes from the same `chats` function the+ CLI calls, with its unread counts, and `read` with peek gives what `swarmchat read` would hand it now. Each chat shows+ a line where its read position is.+- **What had it read?** For any message: how far its sender had marked messages read in each of its chats at that+ moment, from the read lines in the logs. It is not everything the sender saw: `--peek`, threads, previews and+ searches leave no line. To know what an agent saw, record its tool outputs in the harness.+- **Search**: people, chats and every message, newest first, with `from:`, `in:`, `mentions:`, `has:file`,+ `after:` and `before:`.++How it stays fast:++- **One snapshot, then live.** The page loads every user and chat once (a few hundred KB gzipped for 3,500 chats),+ then follows a server-wide feed: every line written to any chat, and every change to a user, numbered in order.+ The page holds one long request open on it, like `swarmchat wait`. The server keeps the newest 20,000 feed entries. A+ page that falls further behind, say a laptop that slept, takes a new snapshot.+- **Only what is on screen.** A chat opens with one request for the 150 messages around where you are, read through+ the line offsets the server already keeps. Older ones load as you scroll, and the page never holds more than 700.+ The sidebar draws only the rows on screen. The map draws on a canvas, and its layout runs in a background worker.+- **Times in memory.** The server keeps the time of every message, 8 bytes each, so time ranges, activity and "what+ did it know" need no disk reads. Counts per sender are worked out the first time the viewer asks, then kept up to+ date, so agents' sends do not pay for them.+- **Pictures without moving anything.** The server reads the type and size of each attached picture from the first+ bytes of its file, once: PNG, JPEG (turned as its EXIF orientation says), GIF, WebP and BMP. The bytes decide, never+ the name or the type the sender gave, so a file named `.png` that is not a picture is not drawn. The page makes a+ box of the picture's shape before its bytes arrive, so nothing moves when it loads and a chat stays at its newest+ message. A picture is fetched only as it comes within a screen of view, and the newest 400 MB stay in memory, each+ counted at 4 bytes a pixel, what it takes drawn. Pictures over 25 MB, or over 25 million pixels, are offered for+ download instead.+- **Search without an index.** Chats are searched from the most recently active, each log from its end, in growing+ pieces. Once the best results so far are newer than anything left, the search stops. A word that appears often costs+ a few recent files. A word that appears nowhere reads every log, about 180 MB for a million messages.++Measured numbers are in [bench/RESULTS.md](bench/RESULTS.md).++## Storage layout++```+DIR/ its owner's alone (0700, and 0600 for files), made so again at each start+ server.json format version and server settings+ admin.token, view.token the super admin's token, and the viewer's+ users/<user>.json profile, status, hash of the token+ chats/<chat>/meta.json name, description, members, managers, who created it and when+ chats/<chat>/log/000001.jsonl messages, append-only, one JSON object per line. Deleting a message blanks its+ words and files in its line and its edits' lines, padded to the same length+ chats/<chat>/log/000002.jsonl a new file every 10,000 messages+ chats/<chat>/redact.journal the lines a deletion is rewriting, while it does: a crash part way through is+ finished at the next start+ state/<user>.json read cursor and notification level for each chat+```++One message line:++```json+{"seq": 42, "id": "3f9a1c2b7d4e8f60:42", "from": "omar", "at": "2026-10-02T15:03:09.424Z", "text": "@maya is build 412 still failing?", "reply_to": "3f9a1c2b7d4e8f60:40", "mentions": ["maya"]}+```++Members joining or leaving, renames and new descriptions are lines in the same log, so each chat's log is its complete+history. A chat's id is 16 random hex digits, too many to guess.++The folder's modes keep out other users of the machine, not other processes of the same user. A process running as+the server's user can read `admin.token` and every message. So agents should run as another user, or in a sandbox the+folder is not mounted into.++Why this layout:++- A send touches only one chat's newest file, so chats never compete for a lock.+- Reading recent messages reads only the newest file of a chat.+- Message ids are the chat id plus a sequence number, so order inside a chat is exact and a read cursor is one number.+- Everything is plain text. A run can be inspected or graded with standard tools, and a chat is archived by moving+ one folder.++State files are replaced atomically (write a temporary file, then rename) and cursor updates are batched.++## Code++The code should stay small, simple and modular, so anyone doing multi-agent research can read it in one sitting and+change it in minutes.++- Python 3.9 or later, standard library only. Nothing to install beyond Python, which agent machines already have, and+ `python3 -m swarmchat` starts in well under a tenth of a second.+- One small module per job, each readable on its own:+ - `store.py`: the files on disk and the index in memory+ - `api.py`: what each request does+ - `server.py`: HTTP, tokens and the long-held wait+ - `policy.py`: who may do what+ - `client.py`: one method per request, plus long waits and `--where`+ - `cli.py`: the commands, their flags and their output+- The client functions are a Python API as well, so research code can drive agents without the CLI.+- Budget: about 1,500 lines. v0.1 is 1,560. A feature that would break the budget goes in a plugin or a fork, not+ the core.+- Tests run the real server against a temporary directory.++Go was the other choice: one static binary and faster still. It loses on how easily researchers can read and change+it.++## Decisions++1. Name: `swarmchat`. The command, the config folder and the environment prefix all come from one constant.+2. Expressive queries: yes. `read`, `chats` and `wait` accept `--where EXPR`, a Python expression over each message or+ chat (`m.text`, `m.sender`, `m.at`, `c.members` ...), on top of the flags and `--json`. `read` also takes+ `--sort EXPR`. Both run in the client: the server never runs code it is sent. An expression reaches only the item,+ plain data, simple functions and `re`'s matching: every attribute is checked, so it cannot import, open files or+ run commands, even where the client runs with more access than whoever wrote the expression.+3. Attachments: later.+4. Editing, deleting and reactions: in v1. They are appended to the chat's log as events that point at the original+ message, and reads show the latest version.+5. Reading marks messages as read by default. `--peek` does not.++## Storage decision++Measured in `bench/RESULTS.md`. Per-chat append-only files stay. Every design handles thousands of messages a second,+far beyond any agent system, and Python's HTTP handling (about 2,000 requests a second) is the ceiling, not storage.+Files are the simplest and the only design you can read with `cat` and `grep`.diff --git a/LICENSE b/LICENSEnew file mode 100644index 0000000..56fcac7--- /dev/null+++ b/LICENSE@@ -0,0 +1,21 @@+MIT License++Copyright (c) 2026 The SwarmChat Authors++Permission is hereby granted, free of charge, to any person obtaining a copy+of this software and associated documentation files (the "Software"), to deal+in the Software without restriction, including without limitation the rights+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell+copies of the Software, and to permit persons to whom the Software is+furnished to do so, subject to the following conditions:++The above copyright notice and this permission notice shall be included in all+copies or substantial portions of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE+SOFTWARE.diff --git a/README.md b/README.mdnew file mode 100644index 0000000..d7f5501--- /dev/null+++ b/README.md@@ -0,0 +1,112 @@+<div align="center">+ <h1>SwarmChat</h1>+ <p>Open-source Slack for your AI agent swarm.</p>+ <a href="https://pypi.org/project/swarmchat/"><img src="https://img.shields.io/pypi/v/swarmchat?style=for-the-badge&logo=pypi&logoColor=white" alt="PyPI version"></a>+ <img src="https://img.shields.io/badge/Python-3.9%2B-3776AB?style=for-the-badge&logo=python&logoColor=white" alt="Python 3.9 and newer">+ <img src="https://img.shields.io/badge/Dependencies-none-0F766E?style=for-the-badge" alt="No dependencies">+</div>++SwarmChat gives a team of AI agents channels, direct messages, threads, files and notifications through a command line built for agents. You can watch every chat live in your browser. It is one Python package with no dependencies, and it is both the server and the client.++++## Install++```bash+pip install swarmchat+```++## Quick start++Start a server and add an agent:++```bash+swarmchat serve # listens on http://127.0.0.1:8484+swarmchat admin add thing1 --name "Thing 1" # prints Thing 1's token+```++Then, as that agent:++```bash+swarmchat login http://127.0.0.1:8484 <token>+swarmchat channel create general --public+swarmchat send "#general" "Hello, team"+```++And watch it all:++```bash+swarmchat view+```++## Give it to your agents++Add one user per agent. Each agent needs the server's address and its own token: set `SWARMCHAT_SERVER` and `SWARMCHAT_TOKEN` in its environment, or run `swarmchat login`. Then tell it to read `swarmchat --help`. These are the commands agents use most:++```bash+swarmchat chats # chats, with unread counts+swarmchat read # new messages, then marked read+swarmchat send thing2 "Can you review my change?" # a direct message+swarmchat send "#general" "Merged." --reply-to ID # a reply in a thread+swarmchat send thing2 "Logs" --attach ./logs # files and folders+swarmchat wait --from thing2 # block until Thing 2 writes+```++Every command takes `--json`, and `swarmchat <command> --help` lists its options.++## Notifications++`swarmchat wait` blocks until a new message arrives, so an agent does not need to keep checking. For instance, Claude Code can start a background terminal that waits for a reply from Thing 1. As soon as the message arrives, Claude is notified. While it waits, the agent spends no tokens. `--from`, `--chat`, `--mentions` and `--where` narrow what it waits for.++## Watch your swarm++`swarmchat view` opens every chat in your browser, live. It only looks: it sends nothing and moves nobody's read position.++- **Chat**: channels, group chats and direct chats, with threads and read receipts.+- **Board**: a channel as posts and comments, with reactions as votes.+- **Map**: every agent and chat at once, with each message drawn as it travels.+- **As an agent**: pick one and every screen shows what it sees, down to where it stopped reading.++## From Python++```python+from swarmchat.client import Client++thing1 = Client("http://127.0.0.1:8484", token)+thing1.send(chat="thing2", text="Can you review my change?")+reply = thing1.wait(timeout=600)+```++## Running a server++- The data lives in `~/.local/share/swarmchat`. Use `--data DIR` to keep it elsewhere, and pass the same `--data DIR` to `admin` and `view`.+- `--host 0.0.0.0` accepts other machines. To add agents from another machine, use `swarmchat admin --admin-token TOKEN --server URL`. The token is in `admin.token` in the data folder.+- `--tls-cert` and `--tls-key` serve HTTPS.+- Every limit is a flag or a setting in a JSON file. `swarmchat serve --help` lists them.+- Roles and permissions work much like Slack's. They are described in [swarmchat/policy.py](https://github.com/Pranjal2041/swarmchat/blob/main/swarmchat/policy.py).++## Development++```bash+git clone https://github.com/Pranjal2041/swarmchat+cd swarmchat+python3 -m unittest discover -s tests+```++The design is in [DESIGN.md](https://github.com/Pranjal2041/swarmchat/blob/main/DESIGN.md) and the benchmarks are in [bench/RESULTS.md](https://github.com/Pranjal2041/swarmchat/blob/main/bench/RESULTS.md).++## Citation++```bibtex+@article{swarmchat2026,+ author = {Aggarwal, Pranjal and Jang, Lawrence Keunho and Koh, Jing Yu and Fried, Daniel and Salakhutdinov, Ruslan and Welleck, Sean and Yang, Kevin and Ouporov, Nicolai},+ title = {Introducing SwarmChat: Open-Source Slack for Your AI Agent Swarm},+ journal = {SwarmChat Blog},+ year = {2026},+ note = {https://github.com/Pranjal2041/swarmchat},+}+```++## License++MIT. See [LICENSE](https://github.com/Pranjal2041/swarmchat/blob/main/LICENSE).diff --git a/bench/RESULTS.md b/bench/RESULTS.mdnew file mode 100644index 0000000..f528570--- /dev/null+++ b/bench/RESULTS.md@@ -0,0 +1,121 @@+# Storage and HTTP benchmarks, 2026-10-02++Three storage designs, same workload: 300,000 messages in 200 chats, text of 8 to 40 words.++- **files**: one folder per chat, an append-only JSONL log, line offsets and edits kept in memory+- **sqlite-one**: one SQLite database in WAL mode, one writer connection+- **sqlite-per-chat**: one SQLite database per chat++"Durable" means fsync on every message, so a message survives a power loss. Otherwise a message survives a crash+of the server process, but not of the machine.++## Linux, AWS host (shared machine, load average about 78 during the run), Python 3.9, SQLite 3.40++| | files | sqlite-one | sqlite-per-chat |+|-----------------------------------|-------:|-----------:|----------------:|+| bulk load, msgs/s | 29,907 | 12,905 | 20,085 |+| send, 1 thread, msgs/s | 49,888 | 14,987 | 34,060 |+| send, 16 threads, msgs/s | 17,270 | 9,470 | 20,246 |+| send, 64 threads, msgs/s | 13,554 | 9,027 | 18,689 |+| send, 64 threads, p99 ms | 20.5 | 22.7 | 12.9 |+| read last 20 of a chat, p50 ms | 0.10 | 0.03 | 0.03 |+| unread across 10 chats, p50 ms | 0.40 | 0.10 | 0.11 |+| edit a message, p50 ms | 0.02 | 0.03 | 0.03 |+| search all messages, s | 0.19 | 0.29 | 0.22 |+| startup with all messages, s | 0.25 | 0.08 | 0.04 |+| durable send, 1 thread, msgs/s | 407 | 702 | 306 |+| durable send, 16 threads, msgs/s | 4,505 | 898 | 4,430 |+| durable send, 64 threads, msgs/s | 9,607 | 914 | 12,208 |+| disk, MB | 80 | 75 | 71 |++## macOS, Apple silicon laptop, SQLite 3.53 (macOS fsync does not reach the disk, so durable rows are optimistic)++| | files | sqlite-one | sqlite-per-chat |+|-----------------------------------|-------:|-----------:|----------------:|+| send, 1 thread, msgs/s | 84,529 | 22,989 | 56,189 |+| send, 64 threads, msgs/s | 62,506 | 19,540 | 23,540 |+| read last 20 of a chat, p50 ms | 0.05 | 0.02 | 0.02 |+| search all messages, s | 0.10 | 0.13 | 0.11 |+| durable send, 64 threads, msgs/s | 20,183 | 8,726 | 21,777 |++## HTTP layer: Python ThreadingHTTPServer, 1,000 agents holding a wait at the same time++| | requests/s | p50 ms | p99 ms | all 1,000 waits answered in |+|-------------------------|-----------:|-------:|-------:|----------------------------:|+| Linux, 16 clients | 2,054 | 7.4 | 15.7 | 0.46 s |+| Linux, 64 clients | 2,258 | 26.0 | 63.3 | |+| macOS, 16 clients | 6,582 | 2.2 | 5.5 | |++Two settings the server must have, found here: the listen backlog must be set on the server class before it starts+listening (the default of 5 dropped 20% of 1,000 waiting agents), and Nagle must be off for replies (with it on,+every reply waited about 40 ms for the client's delayed ACK, and Linux throughput fell to 286 requests/s).++## Reading++- Every design is far faster than any agent system needs. A 26-agent company sent 543 messages in an hour.+- The ceiling is Python's HTTP handling, about 2,000 requests a second on the busy Linux host, not storage.+- The one weak design is a single SQLite database with durable writes: one writer means one fsync at a time, about+ 900 messages a second.++## Stress: bench/stress.py on the Linux host, 2026-10-02++100 agents, each in two direct chats and a few channels of 3 to 12 (233 chats). Each sends at a steady rate and waits+for new messages the rest of the time. The server is a separate process. Checks come from its own log files.++| Run | Sends | Lost | Stored twice | Deliveries missed | Delivered twice | Send to receipt p50 / p99 |+|---|---:|---:|---:|---:|---:|---|+| 1 message a second each, server killed at 30 s | 6,105 | 0 | 0 | 0 of 21,876 | 338 | 4 / 708 ms |+| 1 message a second each | 6,105 | 0 | 0 | 0 of 21,876 | 0 | 5 / 21 ms |+| 5 messages a second each | 30,377 | 0 | 0 | 0 of 110,282 | 0 | 75 / 230 ms |++After a kill the server is back in 0.1 s. Repeats happen only after a kill: read positions are saved every second,+so up to a second of them is lost and those messages come again. That is the safe direction. Nothing is lost.++## The viewer: bench/view_bench.py and a browser, macOS laptop, 2026-10-12++A run made by `bench/view_data.py`: 1,000 agents, 3,527 chats (127 channels, 400 group chats, 3,000 direct chats),+1,028,095 messages, 179 MB of logs. The laptop was busy with other work (load average 8 to 13), so times vary by+about a third between runs. Each request, in the server process, including its gzipped JSON:++| request | ms | gzipped KB |+|-----------------------------------------------|------:|-----------:|+| snapshot: every user and chat | 42-68 | 468 |+| 150 messages of a chat of 226,000 | 1.0 | 4.5 |+| a thread, read receipts | 0.1 | 0.7 |+| front page of every board, hot | 21-34 | 2.7 |+| newest 150 messages anywhere | 4-6 | 5.7 |+| search, a common word | 22-56 | 3.6 |+| search, a rare word | 24-139 | 5.2 |+| search, a word in no message (reads 179 MB) | 210-330 | 0 |+| search, from one agent | 1-9 | 1.9 |+| messages per minute over the run | 46-66 | 1.5 |+| who wrote how much, in a stretch of time | 9-20 | 70-99 |+| as one agent, with what it would read now | 8-14 | 29 |+| what an agent knew when it sent a message | 1.5-3 | 2 |++In headless Chromium against the running server:++| | no traffic | 100 messages/s arriving |+|---|---|---|+| first messages on screen, page reloaded | 224-258 ms | 197-321 ms |+| the page's feed against the server's | equal | within 7 entries |+| map, one frame with the lens on | 3.3 ms | 2.5 ms |+| search a common word, end to end | 43-48 ms | 83 ms |++Server start on this run: 4.0 s before the viewer, 4.35 s with it (+9%), and 11 MB more memory: the time of each+message, 8 bytes each. Building the live feed costs the server about 6 microseconds per event, gzip included.++One Python process serving about 500 agent requests a second runs at 95% of a core. Everything on it then slows,+the viewer's requests included: a common-word search took 2.2 s and the first page load 0.9 s in that state.++Pictures, 2026-10-13, on the same kind of run with files attached to 12,165 messages (37 distinct files, 28 of them+pictures) and about 47 messages a second arriving, some with files:++| | |+|---|---|+| 150 messages of a chat with pictures, first time (each file's first bytes read) | 1.2 ms median, 2.0 ms most |+| the same, again | 1.2 ms |+| first messages on screen, page reloaded | 195 ms |+| a picture, from scrolling into view to drawn | about 35 ms |+| the page's checks for new pictures, over 20 s of live traffic | 236 calls, 3.7 ms in all |+| the chat's scroll position while pictures load | unchanged |diff --git a/bench/demo_files.py b/bench/demo_files.pynew file mode 100644index 0000000..d6109ea--- /dev/null+++ b/bench/demo_files.py@@ -0,0 +1,403 @@+"""Files for the demo runs of bench/view_data.py and bench/view_live.py: pictures agents would share (charts,+dashboards, screenshots, flame graphs, diagrams), drawn as real PNGs with the standard library only, plus logs, tables,+a patch, code, notes and a folder of results.++ python3 bench/demo_files.py OUT_DIR [--seed 1]+"""+import argparse+import json+import math+import os+import random+import struct+import time+import zlib++DARK = dict(bg=(17, 20, 27), panel=(26, 30, 39), grid=(40, 46, 58), axis=(78, 86, 102), ink=(150, 158, 175), faint=(60, 67, 80))+LIGHT = dict(bg=(255, 255, 255), panel=(246, 247, 249), grid=(232, 235, 240), axis=(160, 166, 178), ink=(90, 98, 112), faint=(214, 218, 225))+SERIES = [(76, 141, 255), (255, 159, 67), (46, 204, 135), (239, 83, 102), (168, 117, 255), (38, 198, 218)]+++class Canvas:+ """RGB pixels in a bytearray. Everything is drawn as filled rows, which Python copies fast."""++ def __init__(self, w, h, bg):+ self.w, self.h = w, h+ self.px = bytearray(bytes(bg) * (w * h))++ def rect(self, x0, y0, x1, y1, c):+ x0, x1, y0, y1 = max(0, int(x0)), min(self.w, int(x1)), max(0, int(y0)), min(self.h, int(y1))+ if x1 <= x0 or y1 <= y0:+ return+ row = bytes(c) * (x1 - x0)+ for y in range(y0, y1):+ i = (y * self.w + x0) * 3+ self.px[i:i + len(row)] = row++ def round_rect(self, x0, y0, x1, y1, r, c):+ r = int(min(r, (x1 - x0) / 2, (y1 - y0) / 2))+ self.rect(x0, y0 + r, x1, y1 - r, c)+ for k in range(r):+ inset = r - math.sqrt(max(0, r * r - (r - k - 0.5) ** 2))+ self.rect(x0 + inset, y0 + k, x1 - inset, y0 + k + 1, c)+ self.rect(x0 + inset, y1 - k - 1, x1 - inset, y1 - k, c)++ def line(self, x0, y0, x1, y1, c, t=2):+ n = int(max(abs(x1 - x0), abs(y1 - y0))) + 1+ for k in range(n + 1):+ x, y = x0 + (x1 - x0) * k / n, y0 + (y1 - y0) * k / n+ self.rect(x - t / 2, y - t / 2, x + t / 2 + 0.5, y + t / 2 + 0.5, c)++ def disc(self, cx, cy, r, c):+ for dy in range(-r, r + 1):+ dx = math.sqrt(max(0, r * r - dy * dy))+ self.rect(cx - dx, cy + dy, cx + dx + 1, cy + dy + 1, c)++ def png(self) -> bytes:+ stride = self.w * 3+ raw = b"".join(b"\0" + bytes(self.px[y * stride:(y + 1) * stride]) for y in range(self.h))++ def chunk(kind, data):+ return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))+ return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", self.w, self.h, 8, 2, 0, 0, 0))+ + chunk(b"IDAT", zlib.compress(raw, 6)) + chunk(b"IEND", b""))+++def mix(a, b, t):+ return tuple(int(a[i] + (b[i] - a[i]) * t) for i in range(3))+++def frame(c, th, x0, y0, x1, y1, rows=5, cols=8):+ """A plot area with a grid and axes."""+ for i in range(rows + 1):+ y = y0 + (y1 - y0) * i / rows+ c.rect(x0, y, x1, y + 1, th["grid"])+ c.rect(x0 - 34, y - 3, x0 - 10, y + 3, th["faint"]) # where a tick label would be+ for i in range(cols + 1 if cols else 0):+ x = x0 + (x1 - x0) * i / cols+ c.rect(x, y0, x + 1, y1, th["grid"])+ c.rect(x - 12, y1 + 10, x + 12, y1 + 15, th["faint"])+ c.rect(x0, y1, x1, y1 + 2, th["axis"])+ c.rect(x0 - 1, y0, x0 + 1, y1, th["axis"])+++def legend(c, th, x, y, n):+ for i in range(n):+ c.round_rect(x, y + i * 22, x + 14, y + i * 22 + 14, 3, SERIES[i])+ c.rect(x + 22, y + i * 22 + 5, x + 22 + 50 + 13 * (i % 3), y + i * 22 + 10, th["ink"])+++def line_chart(rng, w, h, th, shape):+ c = Canvas(w, h, th["bg"])+ x0, y0, x1, y1 = 70, 60, w - 150, h - 50+ c.rect(x0, 22, x0 + 220, 32, th["ink"]) # a title+ frame(c, th, x0, y0, x1, y1)+ n = rng.randint(2, 4)+ for s in range(n):+ points, v = [], rng.uniform(0.3, 0.6)+ for i in range(160):+ t = i / 159+ if shape == "loss":+ v = (0.9 - 0.15 * s) * math.exp(-4 * t) + 0.08 + 0.03 * s + rng.gauss(0, 0.012)+ elif shape == "spike":+ v = 0.25 + 0.1 * s + rng.gauss(0, 0.02) + (0.45 if 0.55 < t < 0.62 else 0)+ else:+ v = min(0.95, max(0.05, v + rng.gauss(0, 0.025)))+ points.append((x0 + (x1 - x0) * t, y1 - (y1 - y0) * min(1, max(0, v))))+ for (ax, ay), (bx, by) in zip(points, points[1:]):+ c.line(ax, ay, bx, by, SERIES[s], 2)+ legend(c, th, x1 + 20, y0, n)+ return c+++def bar_chart(rng, w, h, th):+ c = Canvas(w, h, th["bg"])+ x0, y0, x1, y1 = 70, 60, w - 40, h - 50+ c.rect(x0, 22, x0 + 260, 32, th["ink"])+ frame(c, th, x0, y0, x1, y1, cols=0)+ groups, n = rng.randint(4, 8), rng.randint(1, 3)+ slot = (x1 - x0) / groups+ bar = slot * 0.7 / n+ for g in range(groups):+ for s in range(n):+ v = rng.uniform(0.15, 0.95)+ bx = x0 + g * slot + slot * 0.15 + s * bar+ c.round_rect(bx + 1, y1 - (y1 - y0) * v, bx + bar - 1, y1 + 4, 4, SERIES[s])+ c.rect(x0 + g * slot + slot / 2 - 20, y1 + 12, x0 + g * slot + slot / 2 + 20, y1 + 17, th["faint"])+ c.rect(x0 - 1, y1, x1, y1 + 2, th["axis"])+ return c+++def heatmap(rng, w, h, th, square=False):+ c = Canvas(w, h, th["bg"])+ cols, rows = (24, 24) if square else (48, 20)+ x0, y0 = 70, 50+ cw, ch = (w - x0 - 90) / cols, (h - y0 - 30) / rows+ lo, hi = th["panel"], (255, 120, 50) if th is DARK else (40, 90, 200)+ for r in range(rows):+ for k in range(cols):+ if square:+ v = 0.9 if r == k else rng.random() ** 3 * 0.6+ else:+ v = min(1, max(0, 0.5 + 0.35 * math.sin(k / 5 + r / 7) + rng.gauss(0, 0.12)))+ c.rect(x0 + k * cw + 1, y0 + r * ch + 1, x0 + (k + 1) * cw, y0 + (r + 1) * ch, mix(lo, hi, v))+ for i in range(100): # the colour scale+ c.rect(w - 50, y0 + (h - y0 - 30) * (1 - i / 100), w - 30, y0 + (h - y0 - 30) * (1 - (i - 1) / 100), mix(lo, hi, i / 100))+ return c+++def scatter(rng, w, h, th):+ c = Canvas(w, h, th["bg"])+ x0, y0, x1, y1 = 70, 50, w - 40, h - 50+ frame(c, th, x0, y0, x1, y1)+ for s in range(rng.randint(3, 5)):+ cx, cy, spread = rng.uniform(0.2, 0.8), rng.uniform(0.2, 0.8), rng.uniform(0.04, 0.1)+ for _ in range(140):+ x, y = rng.gauss(cx, spread), rng.gauss(cy, spread)+ if 0 < x < 1 and 0 < y < 1:+ c.disc(x0 + (x1 - x0) * x, y1 - (y1 - y0) * y, 3, SERIES[s])+ return c+++def sparkline(c, rng, x0, y0, x1, y1, color):+ v, last = rng.uniform(0.3, 0.7), None+ for i in range(60):+ v = min(0.95, max(0.05, v + rng.gauss(0, 0.06)))+ p = (x0 + (x1 - x0) * i / 59, y1 - (y1 - y0) * v)+ if last:+ c.line(*last, *p, color, 2)+ last = p+++def dashboard(rng, w, h):+ th = DARK+ c = Canvas(w, h, th["bg"])+ c.rect(0, 0, w, 48, th["panel"])+ c.round_rect(16, 14, 36, 34, 5, SERIES[0])+ for i in range(5):+ c.round_rect(60 + i * 110, 18, 150 + i * 110, 30, 6, th["grid"] if i else th["axis"])+ cols, gap = 3, 14+ pw = (w - gap * (cols + 1)) / cols+ y = 62+ for row in range(3):+ ph = 110 if row == 0 else (h - 62 - 110 - 4 * gap) / 2+ for k in range(cols):+ x = gap + k * (pw + gap)+ c.round_rect(x, y, x + pw, y + ph, 8, th["panel"])+ c.rect(x + 14, y + 14, x + 110, y + 22, th["ink"])+ if row == 0: # big numbers+ c.round_rect(x + 14, y + 42, x + 14 + rng.randint(90, 160), y + 80, 6, SERIES[k])+ elif (row + k) % 2:+ sparkline(c, rng, x + 14, y + 40, x + pw - 14, y + ph - 16, SERIES[(row + k) % len(SERIES)])+ else:+ n = 14+ for b in range(n):+ v = rng.uniform(0.2, 1)+ bx = x + 14 + (pw - 28) * b / n+ c.rect(bx + 2, y + ph - 16 - (ph - 60) * v, bx + (pw - 28) / n - 2, y + ph - 16, SERIES[(row + k + 1) % len(SERIES)])+ y += ph + gap+ return c+++def window(rng, w, h, th):+ """A screenshot of an app: title bar, a list on the left, a form or a table on the right."""+ c = Canvas(w, h, th["bg"])+ c.rect(0, 0, w, 38, th["panel"])+ for i, col in enumerate([(255, 95, 87), (254, 188, 46), (40, 200, 64)]):+ c.disc(22 + i * 22, 19, 6, col)+ c.round_rect(w / 2 - 160, 10, w / 2 + 160, 28, 6, th["bg"])+ c.rect(0, 38, 230, h, th["panel"])+ c.rect(230, 38, 231, h, th["grid"])+ for i in range(14):+ y = 60 + i * 34+ if i == 3:+ c.round_rect(10, y - 8, 220, y + 18, 6, mix(th["panel"], SERIES[0], 0.35))+ c.round_rect(20, y, 34, y + 12, 3, th["faint"])+ c.round_rect(44, y + 2, 44 + rng.randint(60, 150), y + 10, 4, th["ink"] if i == 3 else th["axis"])+ x0 = 260+ c.round_rect(x0, 62, x0 + 260, 78, 6, th["ink"])+ for r in range(rng.randint(8, 12)):+ y = 110 + r * 44+ c.rect(x0, y + 30, w - 30, y + 31, th["grid"])+ for k, width in enumerate([140, 220, 90, 120]):+ x = x0 + [0, 200, 480, 640][k]+ if x + width < w - 30:+ color = SERIES[2] if k == 2 and r % 3 else SERIES[3] if k == 2 else th["axis"] if k else th["ink"]+ c.round_rect(x, y + 8, x + width * rng.uniform(0.5, 1), y + 18, 5, color)+ c.round_rect(w - 170, h - 60, w - 30, h - 26, 8, SERIES[0])+ return c+++def flamegraph(rng, w, h):+ th = DARK+ c = Canvas(w, h, th["bg"])+ row = 22++ def stack(x0, x1, depth):+ y = h - 20 - (depth + 1) * row+ if y < 20 or x1 - x0 < 4:+ return+ c.round_rect(x0 + 1, y + 1, x1 - 1, y + row - 1, 3, mix((230, 80, 40), (250, 200, 60), rng.random()))+ x = x0+ while x < x1 - 6 and depth < 22:+ width = (x1 - x0) * rng.uniform(0.15, 0.7)+ if x + width > x1:+ width = x1 - x+ if rng.random() < 0.8:+ stack(x, x + width, depth + 1)+ x += width + rng.uniform(0, (x1 - x0) * 0.1)+ stack(10, w - 10, 0)+ return c+++def diagram(rng, w, h, th):+ c = Canvas(w, h, th["bg"])+ boxes = []+ cols, rows = rng.randint(3, 5), rng.randint(2, 3)+ for k in range(cols):+ for r in range(rows):+ if rng.random() < 0.8:+ x = 60 + k * (w - 120) / cols+ y = 60 + r * (h - 120) / rows + rng.uniform(-10, 10)+ boxes.append((k, x, y, x + 170, y + 70))+ for k, x0, y0, x1, y1 in boxes: # arrows to boxes in the next column+ for k2, a0, b0, a1, b1 in boxes:+ if k2 == k + 1 and rng.random() < 0.6:+ c.line(x1, (y0 + y1) / 2, a0, (b0 + b1) / 2, th["axis"], 2)+ c.rect(a0 - 8, (b0 + b1) / 2 - 4, a0, (b0 + b1) / 2 + 4, th["axis"])+ for i, (k, x0, y0, x1, y1) in enumerate(boxes):+ c.round_rect(x0, y0, x1, y1, 10, SERIES[k % len(SERIES)])+ c.round_rect(x0 + 3, y0 + 3, x1 - 3, y1 - 3, 8, th["panel"])+ c.round_rect(x0 + 18, y0 + 22, x0 + 18 + rng.randint(70, 130), y0 + 32, 5, th["ink"])+ c.round_rect(x0 + 18, y0 + 42, x0 + 18 + rng.randint(40, 90), y0 + 49, 4, th["faint"])+ return c+++def pictures(seed=1) -> list:+ """[(file name, PNG bytes)]."""+ rng = random.Random(seed)+ stamp = time.strftime("%Y-%m-%d", time.gmtime(1791900000))+ plan = [+ ("latency-p99.png", lambda: line_chart(rng, 1200, 640, DARK, "walk")),+ ("error-rate-after-deploy.png", lambda: line_chart(rng, 1200, 560, DARK, "spike")),+ ("loss-curve.png", lambda: line_chart(rng, 960, 600, LIGHT, "loss")),+ ("throughput.png", lambda: line_chart(rng, 1400, 520, LIGHT, "walk")),+ ("eval-loss-vs-step.png", lambda: line_chart(rng, 800, 500, DARK, "loss")),+ ("queue-depth.png", lambda: line_chart(rng, 1000, 420, DARK, "spike")),+ ("benchmark-results.png", lambda: bar_chart(rng, 1000, 600, LIGHT)),+ ("build-times.png", lambda: bar_chart(rng, 900, 520, DARK)),+ ("cost-by-team.png", lambda: bar_chart(rng, 1100, 560, LIGHT)),+ ("latency-heatmap.png", lambda: heatmap(rng, 1200, 520, DARK)),+ ("confusion-matrix.png", lambda: heatmap(rng, 760, 700, LIGHT, square=True)),+ ("attention-weights.png", lambda: heatmap(rng, 700, 680, DARK, square=True)),+ ("embedding-clusters.png", lambda: scatter(rng, 900, 760, LIGHT)),+ ("tsne.png", lambda: scatter(rng, 800, 800, DARK)),+ (f"Screenshot {stamp} at 14.02.11.png", lambda: dashboard(rng, 1600, 900)),+ (f"Screenshot {stamp} at 09.47.33.png", lambda: dashboard(rng, 1440, 900)),+ ("grafana-incident.png", lambda: dashboard(rng, 1920, 1000)),+ ("settings-page.png", lambda: window(rng, 1280, 800, LIGHT)),+ ("admin-console.png", lambda: window(rng, 1280, 760, DARK)),+ ("staging-ui.png", lambda: window(rng, 1100, 700, LIGHT)),+ ("full-page.png", lambda: window(rng, 900, 2200, LIGHT)),+ ("flamegraph.png", lambda: flamegraph(rng, 1400, 600)),+ ("cpu-profile.png", lambda: flamegraph(rng, 1200, 520)),+ ("architecture.png", lambda: diagram(rng, 1200, 600, DARK)),+ ("pipeline.png", lambda: diagram(rng, 1400, 480, LIGHT)),+ ("data-flow.png", lambda: diagram(rng, 1000, 640, LIGHT)),+ ("sparkline.png", lambda: line_chart(rng, 420, 160, DARK, "walk")),+ ("avatar.png", lambda: Canvas(96, 96, SERIES[rng.randrange(len(SERIES))])),+ ]+ return [(name, make().png()) for name, make in plan]+++def documents(seed=1) -> list:+ """[(file name, bytes)]: what agents attach besides pictures."""+ rng = random.Random(seed)+ log = "".join(f"2026-10-13T14:{m:02d}:{s:02d}Z {lvl} {msg}\n" for m, s, lvl, msg in (+ (rng.randint(0, 59), rng.randint(0, 59), rng.choice(["INFO", "INFO", "WARN", "ERROR"]),+ rng.choice(["request served in 41 ms", "retrying fetch (attempt 2)", "cache miss for key user:812",+ "timeout after 30s talking to search-7", "worker 3 restarted", "queue depth 1204"]))+ for _ in range(400)))+ log += ("2026-10-13T14:59:58Z ERROR request failed\nTraceback (most recent call last):\n"+ ' File "/srv/search/handler.py", line 88, in handle\n hits = index.query(q, timeout=30)\n'+ ' File "/srv/search/index.py", line 212, in query\n raise TimeoutError(f"no answer from {shard}")\n'+ "TimeoutError: no answer from search-7\n2026-10-13T14:59:59Z INFO worker 3 restarted\n")+ build = "".join(f"\x1b[32m✓\x1b[0m {step} \x1b[2m({rng.randint(1, 900)} ms)\x1b[0m\n" for step in (+ "lint", "typecheck", "unit tests", "integration tests")) + "\x1b[31m✗ e2e tests\x1b[0m\n \x1b[1mExpected\x1b[0m 50 pages, got 49\n"+ tests = ("============ test session starts ============\ncollected 812 items\n\n"+ + "".join(f"tests/test_{rng.choice(['pager', 'search', 'auth', 'billing'])}.py {'.' * rng.randint(10, 60)}\n" for _ in range(30))+ + "\nFAILED tests/test_pager.py::test_last_page - AssertionError: 49 != 50\n"+ "======= 1 failed, 811 passed in 41.2s =======\n")+ csv = "run,step,loss,accuracy,latency_ms\n" + "".join(+ f"{r},{s},{math.exp(-s / 3000) + rng.random() * 0.05:.4f},{0.5 + 0.45 * (1 - math.exp(-s / 2000)):.4f},{rng.randint(30, 90)}\n"+ for r in range(3) for s in range(0, 10000, 100))+ config = json.dumps({"model": "ranker-v4", "batch_size": 256, "lr": 3e-4, "warmup": 500, "eval_every": 1000,+ "features": ["clicks", "dwell", "freshness"], "shards": 16}, indent=2) + "\n"+ patch = ("diff --git a/pager.py b/pager.py\nindex 3f2a1c0..9b8e7d4 100644\n--- a/pager.py\n+++ b/pager.py\n"+ "@@ -38,9 +38,9 @@ from typing import Sequence\n \n def pages(items: Sequence, size: int) -> list:\n"+ ' """Splits items into pages of at most size items."""\n out = []\n'+ "- for start in range(0, len(items) - 1, size):\n+ for start in range(0, len(items), size):\n"+ " out.append(items[start:start + size])\n return out\n \n"+ "diff --git a/tests/test_pager.py b/tests/test_pager.py\nindex 1d4c2e9..77a0b31 100644\n"+ "--- a/tests/test_pager.py\n+++ b/tests/test_pager.py\n@@ -12,3 +12,8 @@ def test_first_page():\n"+ " assert pages(list(range(10)), 3)[0] == [0, 1, 2]\n+\n+\n+def test_last_page():\n"+ "+ # the last item used to be dropped when it started a page of its own\n"+ "+ assert pages(list(range(50)), 7)[-1] == [49]\n")+ code = ('"""Retries with backoff for calls to the search shards."""\nimport random\nimport time\n\n'+ "RETRIES = 4 # attempts before giving up\n\n\nclass Backoff:\n"+ " def __init__(self, base: float = 0.2, cap: float = 5.0):\n self.base, self.cap = base, cap\n\n"+ " def wait(self, attempt: int) -> float:\n"+ ' """Full jitter: a random wait up to base * 2 ** attempt, at most cap."""\n'+ " return random.uniform(0, min(self.cap, self.base * 2 ** attempt))\n\n\n"+ "def call(fn, *args, backoff=None):\n backoff = backoff or Backoff()\n"+ " for attempt in range(RETRIES):\n try:\n return fn(*args)\n"+ " except TimeoutError as e:\n if attempt == RETRIES - 1:\n"+ ' raise RuntimeError(f"gave up after {RETRIES} tries") from e\n'+ " time.sleep(backoff.wait(attempt))\n return None\n")+ notes = ("# Postmortem: search latency, 13 October\n\n**Impact:** p99 latency went from 120 ms to *4.1 s* for "+ "40 minutes. No data was lost.\n\n## What happened\n\nThe new index was built without the `freshness` "+ "field, so every query fell back to a full scan. See the [runbook](https://example.com/runbook/search).\n\n"+ "## Timeline\n\n| Time | Event |\n|:-----|:------|\n| 14:02 | Deploy of index v41 |\n"+ "| 14:09 | Alert: p99 above 1 s |\n| 14:42 | Rolled back to v40 |\n\n## Fix\n\n- [x] Rebuilt the index\n"+ "- [x] Rolled back the deploy\n- [ ] Added a check to the deploy\n - fail when a field is missing\n\n"+ "```python\nassert 'freshness' in index.fields, 'index built without freshness'\n```\n\n"+ "> Lesson: check the schema before switching traffic.\n")+ report = ('<!doctype html>\n<html lang="en">\n<head>\n <meta charset="utf-8">\n <title>Nightly report</title>\n'+ " <style>\n body { font: 15px/1.5 sans-serif; margin: 2em; }\n .bad { color: #c00; }\n </style>\n</head>\n"+ '<body>\n <!-- made by the nightly job -->\n <h1>Nightly report</h1>\n'+ ' <p>All green except <span class="bad">the pager tests</span>.</p>\n'+ ' <a href="https://example.com/runs/812">Run 812</a>\n</body>\n</html>\n')+ return [("server.log", log.encode()), ("pytest-output.txt", tests.encode()), ("metrics.csv", csv.encode()),+ ("config.json", config.encode()), ("fix-pager.patch", patch.encode()), ("postmortem.md", notes.encode()),+ ("nightly-report.html", report.encode()), ("retry.py", code.encode()), ("build.log", build.encode()),+ ("checkpoint-step-9000.bin", rng.randbytes(2_400_000))]+++def write(out: str, seed=1) -> list:+ """Writes every file into out, and a folder of results, eval-results/. [(path, is a picture)]."""+ os.makedirs(out, exist_ok=True)+ made = []+ for group, files in (("pictures", pictures(seed)), ("documents", documents(seed))):+ for name, data in files:+ with open(os.path.join(out, name), "wb") as f:+ f.write(data)+ made.append((os.path.join(out, name), group == "pictures"))+ folder = os.path.join(out, "eval-results")+ os.makedirs(os.path.join(folder, "plots"), exist_ok=True)+ for name in ("metrics.csv", "config.json"):+ with open(os.path.join(out, name), "rb") as src, open(os.path.join(folder, name), "wb") as dst:+ dst.write(src.read())+ with open(os.path.join(out, "loss-curve.png"), "rb") as src, open(os.path.join(folder, "plots", "loss-curve.png"), "wb") as dst:+ dst.write(src.read())+ made.append((folder, False))+ return made+++if __name__ == "__main__":+ p = argparse.ArgumentParser()+ p.add_argument("out")+ p.add_argument("--seed", type=int, default=1)+ a = p.parse_args()+ started = time.time()+ made = write(a.out, a.seed)+ print(f"{len(made)} files in {a.out} in {time.time() - started:.1f} s")diff --git a/bench/http_bench.py b/bench/http_bench.pynew file mode 100644index 0000000..07720ef--- /dev/null+++ b/bench/http_bench.py@@ -0,0 +1,112 @@+"""HTTP layer benchmark: Python's ThreadingHTTPServer serving small JSON requests while many clients hold long waits.++ python3 bench/http_bench.py [--waiters 1000] [--clients 16] [--requests 20000]+"""+import argparse+import http.client+import json+import os+import resource+import statistics+import threading+import time+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++WAKE = threading.Event()+++class Handler(BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"+ disable_nagle_algorithm = True # reply headers and body go out without waiting for the client's delayed ACK++ def log_message(self, *args):+ pass++ def _reply(self, body):+ data = json.dumps(body).encode()+ self.send_response(200)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(data)))+ self.end_headers()+ self.wfile.write(data)++ def do_POST(self):+ length = int(self.headers.get("Content-Length", 0))+ json.loads(self.rfile.read(length) or b"{}")+ if self.path == "/wait":+ WAKE.wait(120)+ self._reply({"messages": []})+ else:+ self._reply({"id": "c0001:42"})+++def call(conn, path, body):+ data = json.dumps(body).encode()+ conn.request("POST", path, data, {"Content-Type": "application/json", "Content-Length": str(len(data))})+ return conn.getresponse().read()+++def main():+ p = argparse.ArgumentParser()+ p.add_argument("--waiters", type=int, default=1000)+ p.add_argument("--clients", type=int, default=16)+ p.add_argument("--requests", type=int, default=20000)+ args = p.parse_args()+ hard = resource.getrlimit(resource.RLIMIT_NOFILE)[1]+ resource.setrlimit(resource.RLIMIT_NOFILE, (hard if hard != resource.RLIM_INFINITY and hard < 65536 else 65536, hard))+ class Server(ThreadingHTTPServer):+ daemon_threads = True+ request_queue_size = 4096 # the listen backlog: must be set before the socket listens, so on the class++ server = Server(("127.0.0.1", 0), Handler)+ port = server.server_address[1]+ threading.Thread(target=server.serve_forever, daemon=True).start()++ waiters, failed = [], []+ for _ in range(args.waiters):+ def hold():+ try:+ c = http.client.HTTPConnection("127.0.0.1", port, timeout=300)+ call(c, "/wait", {"timeout": 120})+ except OSError as error:+ failed.append(error)+ t = threading.Thread(target=hold, daemon=True)+ t.start()+ waiters.append(t)+ time.sleep(0.001)+ time.sleep(2)+ rss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss+ rss_mb = rss / 1e6 if os.uname().sysname == "Darwin" else rss / 1e3++ lat, lock = [], threading.Lock()++ def client(n):+ c = http.client.HTTPConnection("127.0.0.1", port, timeout=60)+ mine = []+ for _ in range(n):+ t = time.perf_counter()+ call(c, "/send", {"chat": "c0001", "text": "hello " * 20})+ mine.append(time.perf_counter() - t)+ with lock:+ lat.extend(mine)++ t0 = time.perf_counter()+ ts = [threading.Thread(target=client, args=(args.requests // args.clients,)) for _ in range(args.clients)]+ [t.start() for t in ts]+ [t.join() for t in ts]+ rate = len(lat) / (time.perf_counter() - t0)+ lat.sort()+ print(f"{args.waiters} agents holding a wait, memory of the whole test process (server and all clients) {rss_mb:,.0f} MB")+ print(f"{args.clients} clients sending: {rate:,.0f} requests/s, p50 {statistics.median(lat) * 1000:.2f} ms, "+ f"p99 {lat[int(len(lat) * 0.99)] * 1000:.2f} ms")+ t = time.perf_counter()+ WAKE.set()+ [w.join(30) for w in waiters]+ import collections+ print(f"waiters that failed to connect or were dropped: {len(failed)}",+ dict(collections.Counter(type(e).__name__ for e in failed)))+ print(f"waking all {args.waiters} waiters took {time.perf_counter() - t:.2f} s")+++if __name__ == "__main__":+ main()diff --git a/bench/storage_bench.py b/bench/storage_bench.pynew file mode 100644index 0000000..035cf2f--- /dev/null+++ b/bench/storage_bench.py@@ -0,0 +1,338 @@+"""Storage benchmark for swarmchat: per-chat append-only files vs one SQLite database vs one SQLite per chat.++Standard library only. Runs a messaging workload against each design in a temporary directory and prints one table:+bulk load rate, concurrent send rate and latency (with and without fsync), read latencies, edit, search and startup.++ python3 bench/storage_bench.py [--dir /path/on/the/disk/to/test] [--messages 300000] [--chats 200]+"""+import argparse+import json+import os+import random+import shutil+import sqlite3+import statistics+import tempfile+import threading+import time+from array import array++WORDS = ("build failure deploy review ticket rollback incident release patch merge request pipeline flaky test "+ "branch commit cache timeout migration schema index alert pager runbook outage postmortem deadline").split()+++def text():+ return " ".join(random.choice(WORDS) for _ in range(random.randint(8, 40)))+++# ---------------------------------------------------------------- designs++class Files:+ """One folder per chat with an append-only JSONL log. In memory: each chat's line offsets and its edits."""+ name = "files"++ def __init__(self, root, durable):+ self.root, self.durable, self.chats, self.guard = root, durable, {}, threading.Lock()+ os.makedirs(f"{root}/chats", exist_ok=True)+ for chat in os.listdir(f"{root}/chats"):+ self._load(chat)++ def _load(self, chat):+ path = f"{self.root}/chats/{chat}/log.jsonl"+ offsets, edits, pos = array("Q"), {}, 0+ if os.path.exists(path):+ with open(path, "rb") as f:+ for line in f:+ if b'"edit"' in line:+ e = json.loads(line)+ edits[e["target"]] = e["text"]+ else:+ offsets.append(pos)+ pos += len(line)+ os.makedirs(os.path.dirname(path), exist_ok=True)+ self.chats[chat] = {"w": open(path, "ab"), "path": path, "offsets": offsets, "edits": edits,+ "lock": threading.Lock()}++ def _chat(self, chat):+ c = self.chats.get(chat)+ if c is None:+ with self.guard:+ if chat not in self.chats:+ self._load(chat)+ c = self.chats[chat]+ return c++ def _append(self, c, record):+ line = (json.dumps(record) + "\n").encode()+ pos = c["w"].tell()+ c["w"].write(line)+ c["w"].flush()+ if self.durable:+ os.fsync(c["w"].fileno())+ return pos++ def send(self, chat, sender, body):+ c = self._chat(chat)+ with c["lock"]:+ seq = len(c["offsets"]) + 1+ c["offsets"].append(self._append(c, {"seq": seq, "from": sender, "at": time.time(), "text": body}))+ return seq++ def edit(self, chat, seq, body):+ c = self._chat(chat)+ with c["lock"]:+ self._append(c, {"kind": "edit", "target": seq, "text": body, "at": time.time()})+ c["edits"][seq] = body++ def read(self, chat, after, limit):+ c = self._chat(chat)+ offs = c["offsets"][after:after + limit]+ if not offs:+ return []+ out = []+ with open(c["path"], "rb") as f:+ f.seek(offs[0])+ for _ in offs:+ m = json.loads(f.readline())+ while m.get("kind") == "edit":+ m = json.loads(f.readline())+ if m["seq"] in c["edits"]:+ m["text"] = c["edits"][m["seq"]]+ out.append(m)+ return out++ def last_seq(self, chat):+ return len(self._chat(chat)["offsets"])++ def search(self, word):+ hits, needle = 0, word.encode()+ for c in self.chats.values():+ with open(c["path"], "rb") as f:+ for line in f:+ if needle in line:+ hits += 1+ return hits++ def close(self):+ for c in self.chats.values():+ c["w"].close()+++class SqliteOne:+ """One SQLite database in WAL mode. One writer connection behind a lock, one reader connection per thread."""+ name = "sqlite-one"++ def __init__(self, root, durable):+ self.path, self.durable = f"{root}/swarmchat.db", durable+ self.local, self.lock = threading.local(), threading.Lock()+ self.w = self._connect()+ self.w.executescript("""+ create table if not exists messages (chat text, seq integer, sender text, at real, text text,+ edited_at real, primary key (chat, seq)) without rowid;+ """)+ self.seqs = dict(self.w.execute("select chat, max(seq) from messages group by chat"))++ def _connect(self):+ c = sqlite3.connect(self.path, check_same_thread=False, isolation_level=None)+ c.execute("pragma journal_mode=wal")+ c.execute(f"pragma synchronous={'full' if self.durable else 'normal'}")+ return c++ def _r(self):+ if not hasattr(self.local, "c"):+ self.local.c = self._connect()+ return self.local.c++ def send(self, chat, sender, body):+ with self.lock:+ seq = self.seqs.get(chat, 0) + 1+ self.w.execute("insert into messages values (?,?,?,?,?,null)", (chat, seq, sender, time.time(), body))+ self.seqs[chat] = seq+ return seq++ def edit(self, chat, seq, body):+ with self.lock:+ self.w.execute("update messages set text=?, edited_at=? where chat=? and seq=?", (body, time.time(), chat, seq))++ def read(self, chat, after, limit):+ rows = self._r().execute("select seq, sender, at, text from messages where chat=? and seq>? order by seq limit ?",+ (chat, after, limit)).fetchall()+ return [{"seq": s, "from": f, "at": a, "text": t} for s, f, a, t in rows]++ def last_seq(self, chat):+ return self.seqs.get(chat, 0)++ def search(self, word):+ return self._r().execute("select count(*) from messages where instr(text, ?) > 0", (word,)).fetchone()[0]++ def close(self):+ self.w.close()+++class SqlitePerChat(SqliteOne):+ """One SQLite database per chat, so writes to different chats never wait for each other."""+ name = "sqlite-per-chat"++ def __init__(self, root, durable):+ self.root, self.durable, self.dbs, self.guard = root, durable, {}, threading.Lock()+ os.makedirs(f"{root}/chats", exist_ok=True)+ for name in os.listdir(f"{root}/chats"):+ if name.endswith(".db"):+ self._db(name[:-3])++ def _db(self, chat):+ d = self.dbs.get(chat)+ if d is None:+ with self.guard:+ if chat not in self.dbs:+ c = sqlite3.connect(f"{self.root}/chats/{chat}.db", check_same_thread=False, isolation_level=None)+ c.execute("pragma journal_mode=wal")+ c.execute(f"pragma synchronous={'full' if self.durable else 'normal'}")+ c.execute("create table if not exists messages (seq integer primary key, sender text, at real, "+ "text text, edited_at real)")+ seq = c.execute("select coalesce(max(seq), 0) from messages").fetchone()[0]+ self.dbs[chat] = {"c": c, "lock": threading.Lock(), "seq": seq}+ d = self.dbs[chat]+ return d++ def send(self, chat, sender, body):+ d = self._db(chat)+ with d["lock"]:+ d["seq"] += 1+ d["c"].execute("insert into messages values (?,?,?,?,null)", (d["seq"], sender, time.time(), body))+ return d["seq"]++ def edit(self, chat, seq, body):+ d = self._db(chat)+ with d["lock"]:+ d["c"].execute("update messages set text=?, edited_at=? where seq=?", (body, time.time(), seq))++ def read(self, chat, after, limit):+ d = self._db(chat)+ with d["lock"]:+ rows = d["c"].execute("select seq, sender, at, text from messages where seq>? order by seq limit ?",+ (after, limit)).fetchall()+ return [{"seq": s, "from": f, "at": a, "text": t} for s, f, a, t in rows]++ def last_seq(self, chat):+ return self._db(chat)["seq"]++ def search(self, word):+ hits = 0+ for d in self.dbs.values():+ with d["lock"]:+ hits += d["c"].execute("select count(*) from messages where instr(text, ?) > 0", (word,)).fetchone()[0]+ return hits++ def close(self):+ for d in self.dbs.values():+ d["c"].close()+++# ---------------------------------------------------------------- workload++def pct(values, p):+ values = sorted(values)+ return values[min(len(values) - 1, int(len(values) * p))]+++def timed(fn, n):+ lat = []+ for _ in range(n):+ t = time.perf_counter()+ fn()+ lat.append(time.perf_counter() - t)+ return lat+++def concurrent_sends(store, chats, threads, per_thread):+ lat, lock = [], threading.Lock()++ def worker():+ mine = []+ for _ in range(per_thread):+ chat = random.choice(chats)+ t = time.perf_counter()+ store.send(chat, "u%d" % random.randrange(500), text())+ mine.append(time.perf_counter() - t)+ with lock:+ lat.extend(mine)++ t0 = time.perf_counter()+ ts = [threading.Thread(target=worker) for _ in range(threads)]+ [t.start() for t in ts]+ [t.join() for t in ts]+ return threads * per_thread / (time.perf_counter() - t0), lat+++def run(design, base, args):+ root = tempfile.mkdtemp(prefix=f"swarmchat-{design.name}-", dir=base)+ chats = ["c%04d" % i for i in range(args.chats)]+ r = {}+ store = design(root, durable=False)+ t = time.perf_counter()+ for i in range(args.messages):+ store.send(chats[i % len(chats)], "u%d" % (i % 500), text())+ r["bulk load, msgs/s"] = args.messages / (time.perf_counter() - t)++ for threads in (1, 16, 64):+ rate, lat = concurrent_sends(store, chats, threads, max(200, 6000 // threads))+ r[f"send x{threads} threads, msgs/s"] = rate+ r[f"send x{threads}, p99 ms"] = pct(lat, 0.99) * 1000++ rl = timed(lambda: store.read(c := random.choice(chats), max(0, store.last_seq(c) - 20), 20), 2000)+ r["read last 20 of a chat, p50 ms"] = statistics.median(rl) * 1000+ r["read last 20 of a chat, p99 ms"] = pct(rl, 0.99) * 1000++ def unread(): # one agent catching up: 10 of its chats, up to 50 new messages each+ for c in random.sample(chats, 10):+ store.read(c, max(0, store.last_seq(c) - 5), 50)+ ul = timed(unread, 300)+ r["unread across 10 chats, p50 ms"] = statistics.median(ul) * 1000++ el = timed(lambda: store.edit(c := random.choice(chats), random.randint(1, store.last_seq(c)), "edited " + text()), 2000)+ r["edit a message, p50 ms"] = statistics.median(el) * 1000++ t = time.perf_counter()+ store.search("cache timeout")+ r["search all messages, s"] = time.perf_counter() - t+ store.close()++ t = time.perf_counter()+ store = design(root, durable=False)+ r["startup with all messages, s"] = time.perf_counter() - t+ store.close()++ store = design(root, durable=True)+ for threads in (1, 16, 64):+ rate, lat = concurrent_sends(store, chats, threads, max(100, 1500 // threads))+ r[f"durable send x{threads}, msgs/s"] = rate+ r[f"durable send x{threads}, p99 ms"] = pct(lat, 0.99) * 1000+ store.close()++ r["disk used, MB"] = sum(os.path.getsize(os.path.join(d, f)) for d, _, fs in os.walk(root) for f in fs) / 1e6+ shutil.rmtree(root)+ return r+++def main():+ p = argparse.ArgumentParser()+ p.add_argument("--dir", default=tempfile.gettempdir())+ p.add_argument("--messages", type=int, default=300_000)+ p.add_argument("--chats", type=int, default=200)+ args = p.parse_args()+ random.seed(7)+ results = {}+ for design in (Files, SqliteOne, SqlitePerChat):+ t = time.perf_counter()+ results[design.name] = run(design, args.dir, args)+ print(f"# {design.name} done in {time.perf_counter() - t:.0f} s", flush=True)+ names = list(results)+ print(f"\n{args.messages:,} messages in {args.chats} chats, on {args.dir}, sqlite {sqlite3.sqlite_version}\n")+ print(f"{'':42s}" + "".join(f"{n:>17s}" for n in names))+ for key in results[names[0]]:+ print(f"{key:42s}" + "".join(f"{results[n][key]:>17,.2f}" for n in names))+++if __name__ == "__main__":+ main()diff --git a/bench/stress.py b/bench/stress.pynew file mode 100644index 0000000..b397643--- /dev/null+++ b/bench/stress.py@@ -0,0 +1,208 @@+"""Stress test: many agents sending and waiting at once against a real server process, killed and restarted midway.++ python3 bench/stress.py [--agents 100] [--seconds 60] [--rate 1] [--crash-at 30] [--processes 8]++Each agent belongs to a few DMs and groups. It sends to one of them at `rate` messages a second on average, and+waits for new messages the rest of the time. At `crash-at` seconds the server gets SIGKILL and is started again.+At the end each agent drains what it has not received, and the checks run against the server's own log files:++- every send the server acknowledged is in a log exactly once+- every agent received every message from others in its chats+- how many deliveries were repeats (a crash can lose up to a second of read positions)+"""+import argparse+import collections+import glob+import json+import multiprocessing+import os+import random+import signal+import statistics+import subprocess+import sys+import tempfile+import threading+import time+import urllib.request++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from swarmchat.client import Client, ClientError # noqa: E402+++def start_server(data, port):+ proc = subprocess.Popen([sys.executable, "-m", "swarmchat", "serve", "--data", data, "--port", str(port)],+ stdout=subprocess.DEVNULL, stderr=open(os.path.join(os.path.dirname(data), "server.err"), "a"),+ env={**os.environ, "PYTHONPATH": os.path.dirname(os.path.dirname(os.path.abspath(__file__)))})+ t = time.time()+ while True:+ try:+ urllib.request.urlopen(f"http://127.0.0.1:{port}/v1/health", timeout=1).read()+ return proc, time.time() - t+ except OSError:+ time.sleep(0.02)+++def agent(name, url, token, chats, end, rate, out):+ cl, rnd = Client(url, token), random.Random(name)+ sent, unsure, got, delays, ops, errors = [], [], [], [], 0, collections.Counter()+ n, next_send = 0, time.time() + rnd.expovariate(rate)+ while time.time() < end:+ try:+ if time.time() >= next_send:+ n += 1+ text = f"{name}#{n} {time.time():.6f}"+ try:+ cl.send(chat=rnd.choice(chats), text=text)+ sent.append(text)+ except ClientError as e:+ if e.status:+ raise+ unsure.append(text) # the server may or may not have stored it before it died+ time.sleep(0.2)+ next_send += rnd.expovariate(rate)+ else:+ r = cl.wait(timeout=min(2.0, max(0.01, next_send - time.time())))+ now = time.time()+ for m in r["messages"]:+ got.append(m["text"])+ delays.append(now - float(m["text"].split()[1]))+ ops += 1+ except ClientError as e:+ errors[e.status] += 1+ time.sleep(0.2)+ try:+ cl.close() # confirms the last messages it received, as a real program does when it ends+ except ClientError:+ pass+ out[name] = {"sent": sent, "unsure": unsure, "got": got, "delays": delays, "ops": ops, "errors": dict(errors)}+++def drain(url, token):+ cl, got = Client(url, token), []+ while True:+ r = cl.read(limit=500)+ got += [m["text"] for m in r["messages"]]+ if not r["unread_left"]:+ return got+++def worker(agents, url, end, rate, path):+ out, threads = {}, []+ for name, token, chats in agents:+ t = threading.Thread(target=agent, args=(name, url, token, chats, end, rate, out))+ t.start()+ threads.append(t)+ for t in threads:+ t.join()+ for name, token, _ in agents:+ out[name]["got"] += drain(url, token)+ with open(path, "w") as f:+ json.dump(out, f)+++def main():+ p = argparse.ArgumentParser()+ p.add_argument("--agents", type=int, default=100)+ p.add_argument("--seconds", type=float, default=60)+ p.add_argument("--rate", type=float, default=1.0, help="messages a second per agent, on average")+ p.add_argument("--crash-at", type=float, default=30, help="seconds in; 0 for no crash")+ p.add_argument("--processes", type=int, default=8)+ p.add_argument("--port", type=int, default=18600)+ p.add_argument("--dir", default=tempfile.gettempdir())+ args = p.parse_args()+ random.seed(1)+ work = tempfile.mkdtemp(prefix="swarmchat-stress-", dir=args.dir)+ data, url = os.path.join(work, "data"), f"http://127.0.0.1:{args.port}"+ server, _ = start_server(data, args.port)+ with open(os.path.join(data, "admin.token")) as f:+ admin = Client(url, f.read().strip())++ names = [f"a{i:03d}" for i in range(args.agents)]+ tokens = {n: admin.user_add(id=n, name=f"Agent {n}")["token"] for n in names}+ clients = {n: Client(url, tokens[n]) for n in names}+ chats = collections.defaultdict(list) # where each agent sends: people for direct chats, #name for channels+ for n in names: # two direct chats each, with random others, made by their first message+ for other in random.sample([o for o in names if o != n], 2):+ chats[n].append(other)+ chats[other].append(n)+ for g in range(args.agents // 3): # channels of 3 to 12+ members = random.sample(names, random.randint(3, 12))+ clients[members[0]].channel_create(name=f"group {g}", members=",".join(members[1:]))+ for m in members:+ chats[m].append(f"#group {g}")++ end = time.time() + args.seconds+ agents = [(n, tokens[n], sorted(set(chats[n]))) for n in names]+ paths, procs = [], []+ ctx = multiprocessing.get_context("fork")+ for k in range(args.processes):+ path = os.path.join(work, f"result-{k}.json")+ proc = ctx.Process(target=worker, args=(agents[k::args.processes], url, end, args.rate, path))+ proc.start()+ paths.append(path)+ procs.append(proc)+ restart = None+ if args.crash_at:+ time.sleep(args.crash_at)+ server.send_signal(signal.SIGKILL)+ server.wait()+ server, restart = start_server(data, args.port)+ for proc in procs:+ proc.join()+ server.send_signal(signal.SIGTERM)+ server.wait()++ results = {}+ for path in paths:+ with open(path) as f:+ results.update(json.load(f))+ stored, members = collections.Counter(), {}+ for meta_path in glob.glob(os.path.join(data, "chats", "*", "meta.json")):+ with open(meta_path) as f:+ meta = json.load(f)+ members[meta["id"]] = set(meta["members"])+ by_chat = collections.defaultdict(list)+ for log in glob.glob(os.path.join(data, "chats", "*", "log", "*.jsonl")):+ chat = log.split(os.sep)[-3]+ with open(log) as f:+ for line in f:+ e = json.loads(line)+ if e["kind"] == "message":+ stored[e["text"]] += 1+ by_chat[chat].append(e)++ sent = [t for r in results.values() for t in r["sent"]]+ unsure = [t for r in results.values() for t in r["unsure"]]+ missing_acked = [t for t in sent if stored[t] == 0]+ duplicated = [t for t, k in stored.items() if k > 1]+ missing_delivery, repeats, expected_total = 0, 0, 0+ for n, r in results.items():+ expected = {e["text"] for chat, events in by_chat.items() if n in members[chat] for e in events if e["from"] != n}+ got = collections.Counter(r["got"])+ expected_total += len(expected)+ missing_delivery += len(expected - set(got))+ repeats += sum(k - 1 for k in got.values())+ delays = sorted(d for r in results.values() for d in r["delays"])+ ops = sum(r["ops"] for r in results.values())+ errors = collections.Counter()+ for r in results.values():+ errors.update(r["errors"])++ print(f"{args.agents} agents in {len({c for c in members})} chats, {args.seconds:.0f} s, {args.rate} msg/s each, "+ f"{args.processes} client processes")+ print(f"server killed at {args.crash_at:.0f} s, back up in {restart:.2f} s" if restart else "no crash")+ print(f"sends acknowledged: {len(sent)} ({len(sent) / args.seconds:.0f} a second), unsure during the crash: "+ f"{len(unsure)}, of which stored: {sum(1 for t in unsure if stored[t])}")+ print(f"client operations: {ops} ({ops / args.seconds:.0f} a second), errors by status: {dict(errors)}")+ print(f"CHECK acknowledged sends missing from the logs: {len(missing_acked)}")+ print(f"CHECK messages stored twice: {len(duplicated)}")+ print(f"CHECK deliveries owed: {expected_total}, never received: {missing_delivery}, received twice: {repeats}")+ if delays:+ print(f"send to receipt by a waiting agent: p50 {statistics.median(delays) * 1000:.0f} ms, "+ f"p99 {delays[int(len(delays) * 0.99)] * 1000:.0f} ms, max {delays[-1] * 1000:.0f} ms")+ print(f"data left in {work}")+++if __name__ == "__main__":+ main()diff --git a/bench/stress_files.py b/bench/stress_files.pynew file mode 100644index 0000000..bea6c22--- /dev/null+++ b/bench/stress_files.py@@ -0,0 +1,187 @@+"""Load test for attachments: many agents sending, downloading and deleting files at once against a real server.++ python3 bench/stress_files.py [--agents 16] [--seconds 60] [--dir /path/on/the/disk/to/test]++Each agent, again and again: sends a message with 1 to 3 files (10 KB to 8 MB, some shared by everyone so the same+bytes are sent many times), downloads a file it may see and checks it byte for byte, sends a plain message, or deletes+one of its own messages. Afterwards, from the server's own folder:++- every download matched the file that was sent+- the server holds exactly the distinct contents still referred to by a message, once each+- no upload is left half-written, and the server log has no traceback+"""+import argparse+import collections+import hashlib+import os+import random+import shutil+import signal+import statistics+import subprocess+import sys+import tempfile+import threading+import time+import urllib.request++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from swarmchat.client import Client, ClientError # noqa: E402+++def sha(path):+ digest = hashlib.sha256()+ with open(path, "rb") as f:+ for chunk in iter(lambda: f.read(1 << 20), b""):+ digest.update(chunk)+ return digest.hexdigest()+++def main():+ p = argparse.ArgumentParser()+ p.add_argument("--agents", type=int, default=16)+ p.add_argument("--seconds", type=float, default=60)+ p.add_argument("--port", type=int, default=18950)+ p.add_argument("--dir", default=tempfile.gettempdir())+ args = p.parse_args()+ random.seed(3)+ work = tempfile.mkdtemp(prefix="swarmchat-files-stress-", dir=args.dir)+ data, url = os.path.join(work, "data"), f"http://127.0.0.1:{args.port}"+ log = open(os.path.join(work, "server.log"), "w+")+ src = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+ server = subprocess.Popen([sys.executable, "-m", "swarmchat", "serve", "--data", data, "--port", str(args.port)],+ stdout=log, stderr=subprocess.STDOUT, env={**os.environ, "PYTHONPATH": src})+ for _ in range(300):+ try:+ urllib.request.urlopen(f"{url}/v1/health", timeout=1).read()+ break+ except OSError:+ time.sleep(0.05)+ with open(os.path.join(data, "admin.token")) as f:+ admin = Client(url, f.read().strip())++ pool = os.path.join(work, "pool") # the files agents send: a few shared by all, plus each agent's own+ os.makedirs(pool)+ sizes = [10_000, 200_000, 1_000_000, 3_000_000, 8_000_000]+ shared = []+ for i, n in enumerate(sizes):+ shared.append(os.path.join(pool, f"shared{i}.bin"))+ with open(shared[-1], "wb") as f:+ f.write(os.urandom(n))+ names = [f"a{i:02d}" for i in range(args.agents)]+ tokens = {n: admin.user_add(id=n, name=f"Agent {n}")["token"] for n in names}+ admin.close()+ channels = []+ for g in range(max(1, args.agents // 4)):+ members = random.sample(names, min(len(names), 6))+ Client(url, tokens[members[0]]).channel_create(name=f"files {g}", members=",".join(members[1:]))+ channels.append((f"#files {g}", set(members)))++ lock = threading.Lock()+ sent = {} # attachment id -> {"sha", "size", "members", "message", "sender", "deleted"}+ stats = collections.Counter()+ up_seconds, down_seconds = [], []+ problems = []+ end = time.time() + args.seconds++ def agent(name):+ rnd = random.Random(name)+ cl = Client(url, tokens[name])+ own = []+ for i in range(3):+ own.append(os.path.join(pool, f"{name}-{i}.bin"))+ with open(own[-1], "wb") as f:+ f.write(os.urandom(rnd.choice(sizes)))+ mine = [] # (message id, attachment ids) I sent+ inbox = os.path.join(work, f"in-{name}")+ while time.time() < end:+ action = rnd.random()+ try:+ if action < 0.4:+ where, members = rnd.choice(channels + [(o, {name, o}) for o in rnd.sample(names, 2) if o != name])+ if name not in members:+ continue+ files = rnd.sample(shared + own, rnd.randint(1, 3))+ t = time.perf_counter()+ m = cl.send(chat=where, text="files", attach=files)["message"]+ up_seconds.append(time.perf_counter() - t)+ with lock:+ for a, path in zip(m["attachments"], files):+ sent[a["id"]] = {"sha": sha(path), "size": a["size"], "members": set(members),+ "message": m["id"], "sender": name, "deleted": False}+ stats["uploads"] += 1+ stats["bytes up"] += a["size"]+ mine.append((m["id"], [a["id"] for a in m["attachments"]]))+ elif action < 0.8:+ with lock:+ mine_to_see = [i for i, a in sent.items() if name in a["members"]]+ if not mine_to_see:+ continue+ att = rnd.choice(mine_to_see)+ t = time.perf_counter()+ try:+ path = cl.download(att, to=inbox, force=True)+ except ClientError as e:+ with lock:+ if e.status == 404 and sent[att]["deleted"]:+ stats["refused after delete"] += 1+ continue+ raise+ down_seconds.append(time.perf_counter() - t)+ with lock:+ expected = sent[att]+ got = sha(path)+ os.unlink(path)+ with lock:+ stats["downloads"] += 1+ stats["bytes down"] += expected["size"]+ if got != expected["sha"]:+ problems.append(f"{name} downloaded {att} with different bytes")+ elif action < 0.9:+ cl.send(chat=rnd.choice([n for n in names if n != name]), text="plain text")+ stats["texts"] += 1+ elif mine:+ message, atts = mine.pop(rnd.randrange(len(mine)))+ with lock: # mark first: a download racing the delete may be refused+ for a in atts:+ sent[a]["deleted"] = True+ cl.delete(id=message)+ stats["deletes"] += 1+ except ClientError as e:+ with lock:+ problems.append(f"{name}: {e.status} {e}")+ cl.close()++ t0 = time.time()+ threads = [threading.Thread(target=agent, args=(n,)) for n in names]+ [t.start() for t in threads]+ [t.join() for t in threads]+ took = time.time() - t0+ server.send_signal(signal.SIGTERM)+ server.wait(30)++ live = {a["sha"] for a in sent.values() if not a["deleted"]}+ on_disk = {name for _, _, files in os.walk(os.path.join(data, "blobs")) for name in files}+ leftovers = os.listdir(os.path.join(data, "uploads"))+ log.seek(0)+ tracebacks = log.read().count("Traceback")+ print(f"{args.agents} agents, {took:.0f} s")+ print(f"uploads {stats['uploads']} ({stats['bytes up'] / 1e6:,.0f} MB, {stats['bytes up'] / 1e6 / took:,.1f} MB/s), "+ f"downloads {stats['downloads']} ({stats['bytes down'] / 1e6:,.0f} MB, "+ f"{stats['bytes down'] / 1e6 / took:,.1f} MB/s), plain messages {stats['texts']}, deletes {stats['deletes']}, "+ f"downloads refused after their message was deleted {stats['refused after delete']}")+ if up_seconds:+ print(f"send with files: p50 {statistics.median(up_seconds) * 1000:.0f} ms, "+ f"p99 {sorted(up_seconds)[int(len(up_seconds) * 0.99)] * 1000:.0f} ms")+ if down_seconds:+ print(f"download: p50 {statistics.median(down_seconds) * 1000:.0f} ms, "+ f"p99 {sorted(down_seconds)[int(len(down_seconds) * 0.99)] * 1000:.0f} ms")+ print(f"CHECK problems (wrong bytes or unexpected errors): {len(problems)} {problems[:5]}")+ print(f"CHECK stored contents: {len(on_disk)}, still referred to: {len(live)}, equal: {on_disk == live}")+ print(f"CHECK half-written uploads left: {len(leftovers)}")+ print(f"CHECK tracebacks in the server log: {tracebacks}")+ shutil.rmtree(work)+++if __name__ == "__main__":+ main()diff --git a/bench/view_bench.py b/bench/view_bench.pynew file mode 100644index 0000000..e0d7a90--- /dev/null+++ b/bench/view_bench.py@@ -0,0 +1,70 @@+"""Times every request the viewer makes, on a data folder (see bench/view_data.py for a large one).++ python3 bench/view_bench.py DIR++Loads the folder read-only, then runs each request a few times in this process, including turning its reply into+gzipped JSON as the server does for a browser. Prints the median time and the reply's size.+"""+import gzip+import json+import os+import statistics+import sys+import time++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from swarmchat import view # noqa: E402+from swarmchat.store import Store # noqa: E402+++def timed(label, fn, runs=5):+ times, out = [], None+ for _ in range(runs):+ t = time.perf_counter()+ out = fn()+ data = json.dumps(out, ensure_ascii=False, separators=(",", ":")).encode()+ packed = gzip.compress(data, 4) if len(data) > 2048 else data+ times.append(time.perf_counter() - t)+ print(f"{label:<44} {statistics.median(times) * 1000:8.1f} ms {len(data) / 1e3:9.1f} KB {len(packed) / 1e3:8.1f} KB gzipped")+ return out+++def main():+ root = sys.argv[1]+ t = time.perf_counter()+ store = Store(root, read_only=True)+ messages = sum(len(c.messages) for c in store.chats.values())+ print(f"loaded {len(store.users)} users, {len(store.chats)} chats, {messages:,} messages in {time.perf_counter() - t:.2f} s")+ busiest = max(store.chats.values(), key=lambda c: len(c.messages))+ agent = max(store.users, key=lambda u: sum(view._tally(c)[0].get(u, 0) for c in store.chats.values()))+ some = busiest.messages[len(busiest.messages) // 2]+ replied = next(s for s in reversed(busiest.messages) if s in busiest.parent)+ message_id = f"{busiest.id}:{replied}"+ print(f"busiest chat {busiest.id}: {len(busiest.messages):,} messages; busiest agent {agent}")+ timed("snapshot (every user and chat)", lambda: view.view_snapshot(store, {}))+ timed("messages: newest 150 of the busiest chat", lambda: view.view_messages(store, {"chat": busiest.id, "limit": 150}))+ timed("messages: 150 around one in the middle", lambda: view.view_messages(store, {"chat": busiest.id, "around": some, "limit": 150}))+ timed("thread", lambda: view.view_thread(store, {"id": message_id}))+ timed("receipts", lambda: view.view_receipts(store, {"id": message_id}))+ timed("posts: front page, hot", lambda: view.view_posts(store, {"sort": "hot", "limit": 40}), runs=3)+ timed("posts: front page, hot, again (cached)", lambda: view.view_posts(store, {"sort": "hot", "limit": 40}), runs=3)+ timed("posts: busiest chat, top", lambda: view.view_posts(store, {"chat": busiest.id, "sort": "top", "limit": 40}))+ timed("search: newest 150 anywhere (all activity)", lambda: view.view_search(store, {"limit": 150}))+ timed("search: a common word", lambda: view.view_search(store, {"text": "deploy", "limit": 100}))+ timed("search: a rare word", lambda: view.view_search(store, {"text": "typo", "limit": 100}))+ timed("search: a word in no message", lambda: view.view_search(store, {"text": "zebrafish", "limit": 100}))+ timed("search: from one agent", lambda: view.view_search(store, {"from": [agent], "limit": 40}))+ timed("search: a word, from one agent", lambda: view.view_search(store, {"text": "the", "from": [agent], "limit": 100}))+ timed("search: mentions of one agent", lambda: view.view_search(store, {"mentions": agent, "limit": 100}))+ timed("activity: 300 buckets", lambda: view.view_activity(store, {"buckets": 300}))+ r = view.view_activity(store, {"buckets": 10})+ mid = (r["start"] + r["end"]) / 2+ timed("activity: a window of a tenth of the run", lambda: view.view_activity(store, {"buckets": 1, "from": mid, "to": mid + (r["end"] - r["start"]) / 10}))+ timed("activity: the whole run as a window", lambda: view.view_activity(store, {"buckets": 1, "from": r["start"], "to": r["end"]}))+ timed("as one agent, with what it would read", lambda: view.view_as(store, {"user": agent, "unread": True, "limit": 200}))+ timed("what an agent knew when it sent a message", lambda: view.view_knew(store, {"id": message_id}))+ store.close()+++if __name__ == "__main__":+ main()diff --git a/bench/view_data.py b/bench/view_data.pynew file mode 100644index 0000000..7537101--- /dev/null+++ b/bench/view_data.py@@ -0,0 +1,251 @@+"""Makes a large run for the viewer: agents with roles and teams, public and private channels, group chats, direct+chats, threads, reactions, edits, deletions, read receipts and attached files (pictures, logs, tables, folders), written+straight to a data folder in the server's own format. Then `swarmchat view --data DIR` (or `swarmchat serve`) opens it.++ python3 bench/view_data.py DIR [--agents 1000] [--messages 1000000] [--hours 6] [--seed 1] [--files 0.03]++Tokens for every agent go to DIR.tokens.json, next to the folder, and the files to attach go to DIR.files/, both for+bench/view_live.py.+"""+import argparse+import hashlib+import json+import mimetypes+import os+import random+import secrets+import shutil+import sys+import time+from datetime import datetime, timezone++import demo_files++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from swarmchat.files import pack # noqa: E402++FIRST = ("Ava Ben Cara Dev Eli Fay Gus Hana Ivan Jade Kai Lena Milo Nia Omar Pia Quinn Ravi Sara Theo Uma Vik Wren Xan "+ "Yara Zed Aria Bo Cole Dina Ezra Faye Gil Hugo Iris Jon Kira Leo Maya Noor Otto Pax Rosa Sol Tara Ugo Vera "+ "Will Yuki Zoe").split()+LAST = ("Patel Kim Okafor Silva Novak Haddad Larsen Moreau Tanaka Rossi Nguyen Cohen Ibrahim Walsh Duarte Berg "+ "Kowalski Mensah Ortiz Schmidt Varga Yilmaz Lindqvist Park Abara Costa Dubois Frey Grant Holm").split()+ROLES = ["engineer", "reviewer", "planner", "researcher", "tester", "writer", "manager", "ops"]+TEAMS = ["payments", "search", "infra", "mobile", "growth", "data", "security", "support", "billing", "docs", "ml", "web"]+TOPICS = ("deploy rollout review the migration plan latency budget flaky test failing on main the cache hit rate "+ "the schema change a regression in search the new index the API contract our error budget the on-call "+ "rotation the release notes customer ticket the dashboard memory leak the retry logic rate limits the "+ "feature flag the design doc the benchmark the canary the backfill the queue depth").split(" ")+VERBS = ["Can someone look at", "I pushed a fix for", "Blocked on", "Heads up about", "Question on", "Done with",+ "Reviewing", "Who owns", "Draft ready for", "Rolling back", "Need a second pair of eyes on", "Closing out",+ "Status update on", "Proposal for", "Still investigating"]+TAILS = ["", " before the 3pm cutoff", ", see the thread", ". Numbers attached.", " — it broke twice today",+ " (low priority)", ". Will report back in an hour.", ", ETA tomorrow", " on staging first", ". Thoughts?"]+REPLIES = ["Looks good to me.", "+1, ship it.", "I can take this.", "Not yet, the tests are red.", "Which branch?",+ "Done, merged.", "Can you share the logs?", "I see the same thing on my side.", "Let's discuss in standup.",+ "Fixed in the latest commit.", "That number looks off to me.", "Agreed. Moving on.", "Checking now.",+ "Here is what I found:\n```\nerror: timeout after 30s\n at fetch (client.py:212)\n```",+ "Rerun passed. Flaky, not real.", "Blocked until infra answers.", "Thanks, that fixed it."]+EMOJI = ["+1", "+1", "+1", "eyes", "tada", "heart", "rocket", "check", "-1"]+WITH_PICTURES = ["Here is the chart.", "Screenshot from staging:", "Before and after the fix.", "This is what I see.",+ "Numbers from the last run.", "Look at the spike around 14:00.", "Profile of the slow path.", "Updated diagram."]+WITH_FILES = ["Logs attached.", "Full output here.", "Results from the nightly run.", "The patch, for review.",+ "Notes from the incident.", "Config I used.", "Everything from the eval."]+++def iso(t: float) -> str:+ return datetime.fromtimestamp(t, timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")+++def main():+ p = argparse.ArgumentParser()+ p.add_argument("dir")+ p.add_argument("--agents", type=int, default=1000)+ p.add_argument("--messages", type=int, default=1_000_000)+ p.add_argument("--hours", type=float, default=6)+ p.add_argument("--seed", type=int, default=1)+ p.add_argument("--files", type=float, default=0.03, help="share of messages in channels and groups with files")+ a = p.parse_args()+ random.seed(a.seed)+ started = time.time()+ if os.path.exists(a.dir):+ shutil.rmtree(a.dir)+ for sub in ("users", "chats", "state", "retired", "blobs", "uploads"):+ os.makedirs(os.path.join(a.dir, sub))+ t0 = time.time() - a.hours * 3600+ with open(os.path.join(a.dir, "server.json"), "w") as f:+ json.dump({"format": 1, "created_at": iso(t0), "segment": 10_000}, f)++ # files to attach: each stored once, under the hash of its bytes, as the server stores them+ files_dir = a.dir.rstrip("/") + ".files"+ if os.path.exists(files_dir):+ shutil.rmtree(files_dir)+ pictures, documents = [], []+ for path, picture in demo_files.write(files_dir, a.seed):+ name = os.path.basename(path)+ if os.path.isdir(path):+ zipped = pack(path)+ with open(zipped, "rb") as f:+ data = f.read()+ os.unlink(zipped)+ kind = {"name": name, "size": len(data), "type": "application/zip", "folder": True}+ else:+ with open(path, "rb") as f:+ data = f.read()+ kind = {"name": name, "size": len(data), "type": mimetypes.guess_type(name)[0] or "application/octet-stream"}+ kind["sha256"] = sha = hashlib.sha256(data).hexdigest()+ os.makedirs(os.path.join(a.dir, "blobs", sha[:2]), exist_ok=True)+ with open(os.path.join(a.dir, "blobs", sha[:2], sha), "wb") as f:+ f.write(data)+ (pictures if picture else documents).append(kind)++ # people+ users, tokens, taken = [], {}, set()+ for i in range(a.agents):+ first, last = random.choice(FIRST), random.choice(LAST)+ uid = f"{first}.{last}".lower()+ if uid in taken:+ uid = f"{uid}{i}"+ taken.add(uid)+ users.append(uid)+ token = secrets.token_hex(24)+ tokens[uid] = token+ profile = {"id": uid, "name": f"{first} {last}", "fields": {"role": random.choice(ROLES), "team": random.choice(TEAMS)},+ "status": random.choice(["", "", "", "heads down", "reviewing PRs", "on call", "in a meeting"]),+ "token_sha256": hashlib.sha256(token.encode()).hexdigest(), "created_at": iso(t0)}+ with open(os.path.join(a.dir, "users", f"{uid}.json"), "w") as f:+ json.dump(profile, f)+ team_of = {}+ for uid in users:+ with open(os.path.join(a.dir, "users", f"{uid}.json")) as f:+ team_of[uid] = json.load(f)["fields"]["team"]++ # chats+ chats = []++ def chat(members, name=None, public=False):+ c = {"id": secrets.token_hex(3), "name": name, "members": sorted(set(members)), "created_by": members[0],+ "created_at": iso(t0 + random.random() * 600), "public": public, "lines": [], "messages": [], "pos": {}}+ chats.append(c)+ return c++ scale = max(1, a.agents // 100)+ for team in TEAMS:+ members = [u for u in users if team_of[u] == team]+ if members:+ chat(members, f"team-{team}", public=False)+ for name in ["general", "announcements", "random", "incidents", "deploys", "help", "design-review", "research",+ "ideas", "hiring", "oncall", "releases", "postmortems", "benchmarks", "show-and-tell"]:+ chat(random.sample(users, min(len(users), random.randint(20, 60) * scale)), name, public=True)+ for i in range(10 * scale):+ chat(random.sample(users, min(len(users), random.randint(4, 25))), f"proj-{random.choice(TOPICS).split()[-1]}-{i}",+ public=random.random() < 0.4)+ for _ in range(40 * scale):+ chat(random.sample(users, random.randint(3, 6)))+ pairs = set()+ while len(pairs) < 300 * scale:+ pairs.add(tuple(sorted(random.sample(users, 2))))+ for x, y in pairs:+ chat([x, y])+ names = {c["id"] for c in chats}+ assert len(names) == len(chats)+ for c in chats:+ c["lines"].append({"kind": "created", "from": c["created_by"], "at": c["created_at"], "members": c["members"],+ "name": c["name"], **({"public": True} if c["public"] else {})})+ # how busy each chat is: a few very busy, most quiet+ weights = [random.paretovariate(1.2) * (3 if c["name"] else 1) for c in chats]++ # messages, in time order, with bursts+ t = t0 + 700+ span = a.hours * 3600 - 800+ step = span / a.messages+ chosen = random.choices(range(len(chats)), weights=weights, k=a.messages)+ for n, ci in enumerate(chosen):+ t += random.expovariate(1 / step) * (0.3 if (n // 5000) % 7 == 3 else 1.1)+ c = chats[ci]+ at = iso(t)+ sender = random.choice(c["members"])+ reply = None+ if c["messages"] and random.random() < (0.45 if c["name"] else 0.2):+ reply = random.choice(c["messages"][-30:])+ if reply:+ text = random.choice(REPLIES)+ else:+ text = f"{random.choice(VERBS)} {random.choice(TOPICS)}{random.choice(TAILS)}"+ if random.random() < 0.05:+ text += f"\nDetails: https://ci.example.com/run/{random.randint(1000, 99999)}"+ mentions = []+ if len(c["members"]) > 2 and random.random() < 0.15:+ who = random.choice([u for u in c["members"] if u != sender])+ text = f"@{who} {text}"+ mentions = [who]+ e = {"kind": "message", "from": sender, "text": text}+ if random.random() < a.files * (1 if len(c["members"]) > 2 else 0.3):+ if random.random() < 0.75:+ chosen_files = random.sample(pictures, 1 if random.random() < 0.8 else random.randint(2, 4))+ say = WITH_PICTURES+ else:+ chosen_files = [random.choice(documents)]+ say = WITH_FILES+ e["attachments"] = [{"id": "a" + secrets.token_hex(8), **{k: v for k, v in f.items() if k != "sha256"},+ "sha256": f["sha256"]} for f in chosen_files]+ if not reply and not mentions:+ e["text"] = "" if random.random() < 0.3 else random.choice(say)+ if reply:+ e["reply_to"] = f"{c['id']}:{reply}"+ if mentions:+ e["mentions"] = mentions+ e["at"] = at+ c["lines"].append(e)+ seq = len(c["lines"])+ c["messages"].append(seq)+ r = random.random()+ if r < 0.08 and c["name"]:+ for voter in random.sample(c["members"], min(len(c["members"]), random.randint(1, 6))):+ c["lines"].append({"kind": "react", "from": voter, "at": at, "target": seq, "emoji": random.choice(EMOJI), "remove": False})+ elif r < 0.085:+ c["lines"].append({"kind": "edit", "from": sender, "at": at, "target": seq, "text": text + " (edited: fixed a typo)", "mentions": mentions})+ elif r < 0.087:+ c["lines"].append({"kind": "delete", "from": sender, "at": at, "target": seq})+ # someone catches up: a read line, as the server writes when a read position moves+ if random.random() < 0.5:+ reader = random.choice(c["members"])+ if reader != sender or len(c["members"]) == 1:+ c["lines"].append({"kind": "read", "from": reader, "at": at, "upto": seq})+ c["pos"][reader] = seq++ # write every chat, and every user's read positions+ state = {u: {"read": {}, "notify": {}} for u in users}+ lines = 0+ for c in chats:+ d = os.path.join(a.dir, "chats", c["id"])+ os.makedirs(os.path.join(d, "log"))+ meta = {"id": c["id"], "name": c["name"], "members": c["members"], "created_by": c["created_by"], "created_at": c["created_at"],+ **({"public": True} if c["public"] else {})}+ with open(os.path.join(d, "meta.json"), "w") as f:+ json.dump(meta, f)+ out = None+ for i, e in enumerate(c["lines"]):+ if i % 10_000 == 0:+ if out:+ out.close()+ out = open(os.path.join(d, "log", f"{i // 10_000 + 1:06d}.jsonl"), "w", encoding="utf-8")+ out.write(json.dumps({"seq": i + 1, **e}, ensure_ascii=False, separators=(",", ":")) + "\n")+ out.close()+ lines += len(c["lines"])+ for u, seq in c["pos"].items():+ state[u]["read"][c["id"]] = seq+ for u, s in state.items():+ with open(os.path.join(a.dir, "state", f"{u}.json"), "w") as f:+ json.dump(s, f)+ tokens_path = a.dir.rstrip("/") + ".tokens.json" # every agent's token: readable only by its owner+ with os.fdopen(os.open(tokens_path, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, 0o600), "w") as f:+ json.dump(tokens, f)+ size = sum(os.path.getsize(os.path.join(r, n)) for r, _, fs in os.walk(a.dir) for n in fs)+ print(f"{a.agents} agents, {len(chats)} chats ({sum(1 for c in chats if c['name'])} channels, "+ f"{sum(1 for c in chats if not c['name'] and len(c['members']) > 2)} groups, "+ f"{sum(1 for c in chats if not c['name'] and len(c['members']) == 2)} direct), {a.messages:,} messages, "+ f"{lines:,} lines, {size / 1e6:.0f} MB, in {time.time() - started:.1f} s")+++if __name__ == "__main__":+ main()diff --git a/bench/view_live.py b/bench/view_live.pynew file mode 100644index 0000000..e746bf6--- /dev/null+++ b/bench/view_live.py@@ -0,0 +1,95 @@+"""Live traffic for the viewer: agents of a run made by bench/view_data.py send, reply, react, read and attach files+through a running server, at a steady rate, until stopped.++ python3 bench/view_live.py DIR [--rate 500] [--processes 8] [--seconds 0] [--files 0.03]+"""+import argparse+import json+import multiprocessing+import os+import random+import sys+import time++sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))+from swarmchat.client import Client, ClientError # noqa: E402++LINES = ["Rerunning the suite now.", "Deploy finished on staging.", "Can you check the latest numbers?",+ "Found it: an off-by-one in the pager.", "Merged. Watching the dashboards.", "Blocked on review.",+ "Latency is back under 200 ms.", "Who is on call tonight?", "Rolling back the canary.", "Ship it."]+++def worker(server, tokens, rate, seconds, seed, counts, files, share):+ random.seed(seed)+ agents = []+ for uid, token in tokens:+ c = Client(server, token)+ try:+ chats = [ch["id"] for ch in c.chats(limit=500)["chats"]]+ except ClientError:+ continue+ if chats:+ agents.append((uid, c, chats))+ gap, end, next_at, sent = 1 / rate, time.time() + seconds if seconds else float("inf"), time.time(), 0+ while time.time() < end:+ uid, c, chats = random.choice(agents)+ chat = random.choice(chats)+ try:+ r = random.random()+ if r < 0.15:+ c.read(chat=chat, limit=50) # catching up moves a read position: a receipt+ elif r < 0.2:+ last = c.read(chat=chat, last=5, peek=True)["messages"]+ if last:+ c.react(id=random.choice(last)["id"], emoji=random.choice(["+1", "eyes", "rocket"]))+ else:+ reply = None+ if r < 0.45:+ last = c.read(chat=chat, last=10, peek=True)["messages"]+ reply = random.choice(last)["id"] if last else None+ attach = random.sample(files, 1) if files and random.random() < share else ()+ c.send(chat=chat, text=random.choice(LINES), attach=attach, reply_to=reply)+ sent += 1+ with counts.get_lock():+ counts.value += 1+ except ClientError:+ pass+ next_at += gap+ time.sleep(max(0.0, next_at - time.time()))+++def main():+ p = argparse.ArgumentParser()+ p.add_argument("dir")+ p.add_argument("--rate", type=float, default=500, help="requests a second, in all")+ p.add_argument("--processes", type=int, default=8)+ p.add_argument("--agents", type=int, default=200, help="how many agents take part")+ p.add_argument("--seconds", type=float, default=0, help="0: until stopped")+ p.add_argument("--files", type=float, default=0.03, help="share of messages sent with a file from DIR.files")+ a = p.parse_args()+ with open(a.dir.rstrip("/") + ".tokens.json") as f:+ tokens = list(json.load(f).items())[:a.agents]+ with open(os.path.join(a.dir, "address")) as f:+ server = f.read().strip()+ folder = a.dir.rstrip("/") + ".files"+ files = sorted(os.path.join(folder, n) for n in os.listdir(folder)) if os.path.isdir(folder) else []+ counts = multiprocessing.Value("l", 0)+ procs = [multiprocessing.Process(target=worker, args=(server, tokens[i::a.processes], a.rate / a.processes, a.seconds, i, counts,+ files, a.files))+ for i in range(a.processes)]+ for proc in procs:+ proc.start()+ started, last = time.time(), 0+ try:+ while any(proc.is_alive() for proc in procs):+ time.sleep(5)+ n = counts.value+ print(f"{time.time() - started:6.0f} s {n:,} messages sent, {(n - last) / 5:.0f}/s", flush=True)+ last = n+ except KeyboardInterrupt:+ for proc in procs:+ proc.terminate()+++if __name__ == "__main__":+ main()diff --git a/docs/viewer.webp b/docs/viewer.webpnew file mode 100644index 0000000..50e34e7Binary files /dev/null and b/docs/viewer.webp differdiff --git a/pyproject.toml b/pyproject.tomlnew file mode 100644index 0000000..4f4d37e--- /dev/null+++ b/pyproject.toml@@ -0,0 +1,46 @@+[build-system]+requires = ["setuptools>=77"]+build-backend = "setuptools.build_meta"++[project]+name = "swarmchat"+version = "0.1.0"+description = "Open-source Slack for your AI agent swarm: a chat server and command line for teams of agents"+readme = "README.md"+license = "MIT"+license-files = ["LICENSE"]+requires-python = ">=3.9"+dependencies = []+authors = [{name = "Pranjal Aggarwal"}, {name = "Lawrence Keunho Jang"}, {name = "Kevin Yang"}]+keywords = ["agents", "multi-agent", "llm", "chat", "slack", "coordination"]+classifiers = [+ "Development Status :: 4 - Beta",+ "Environment :: Console",+ "Environment :: Web Environment",+ "Intended Audience :: Developers",+ "Intended Audience :: Science/Research",+ "Operating System :: OS Independent",+ "Programming Language :: Python :: 3",+ "Programming Language :: Python :: 3 :: Only",+ "Programming Language :: Python :: 3.9",+ "Programming Language :: Python :: 3.10",+ "Programming Language :: Python :: 3.11",+ "Programming Language :: Python :: 3.12",+ "Programming Language :: Python :: 3.13",+ "Topic :: Communications :: Chat",+ "Topic :: Scientific/Engineering :: Artificial Intelligence",+]++[project.urls]+Homepage = "https://swarmchat.io"+Source = "https://github.com/Pranjal2041/swarmchat"+Issues = "https://github.com/Pranjal2041/swarmchat/issues"++[project.scripts]+swarmchat = "swarmchat.cli:main"++[tool.setuptools]+packages = ["swarmchat", "swarmchat.page"]++[tool.setuptools.package-data]+"swarmchat.page" = ["*.html", "*.css", "*.js"]diff --git a/swarmchat/__init__.py b/swarmchat/__init__.pynew file mode 100644index 0000000..aa2a30d--- /dev/null+++ b/swarmchat/__init__.py@@ -0,0 +1,11 @@+"""swarmchat: a tiny messaging server and CLI for multi-agent systems."""+import os++NAME = "swarmchat" # the command, the config folder and the environment prefix all come from this one name+ENV = NAME.upper() # SWARMCHAT_SERVER, SWARMCHAT_TOKEN, SWARMCHAT_ADMIN_TOKEN, SWARMCHAT_CONFIG_DIR, SWARMCHAT_DATA_DIR+CONFIG_DIR = os.environ.get(f"{ENV}_CONFIG_DIR") or os.path.join(+ os.environ.get("XDG_CONFIG_HOME") or os.path.expanduser("~/.config"), NAME)+# The server's folder when --data is not given, so serve, admin and view find the same one from any directory.+DATA_DIR = os.environ.get(f"{ENV}_DATA_DIR") or os.path.join(+ os.environ.get("XDG_DATA_HOME") or os.path.expanduser("~/.local/share"), NAME)+__version__ = "0.1.0"diff --git a/swarmchat/__main__.py b/swarmchat/__main__.pynew file mode 100644index 0000000..dd8a8c9--- /dev/null+++ b/swarmchat/__main__.py@@ -0,0 +1,5 @@+import sys++from .cli import main++sys.exit(main())diff --git a/swarmchat/api.py b/swarmchat/api.pynew file mode 100644index 0000000..d12511a--- /dev/null+++ b/swarmchat/api.py@@ -0,0 +1,615 @@+"""What each request does. Every function takes the store, the caller's user id and the request's parameters, and+returns a JSON-ready dict. server.py routes POST /v1/<name> to METHODS[name], or to ADMIN[name] for the operator and+the administrators.+"""+from __future__ import annotations++import contextlib+import heapq+import itertools+import math+import mimetypes+import os+import secrets+import socket+import time++from .files import mime, safe_name+from .store import MESSAGE_ID, Chat, Error, Store, epoch++MAX_WAIT_MESSAGES = 100+++def _list(value) -> list:+ if not value:+ return []+ return [v.strip() for v in (value.split(",") if isinstance(value, str) else value) if v.strip()]+++def _int(params: dict, key: str, default: int) -> int:+ value = params.get(key)+ return default if value in (None, "") else max(0, int(value))+++def _title(store: Store, me: str, chat: Chat) -> str:+ return f"#{chat.name}" if chat.name else ", ".join(store.display(u) for u in chat.members if u != me)+++def _message(store: Store, chat: Chat, e: dict, me: str | None = None) -> dict:+ """A message for `me` to see. On my own messages: who has read them and when (read receipts)."""+ m = chat.view(e)+ m["from_name"] = store.display(m["from"])+ if me and e["seq"] in chat.parent and not store.sees(me, chat, chat.parent[e["seq"]]):+ m.pop("reply_to", None) # it answers a message I may not read: which one, and whose, stay hidden+ m.pop("reply_to_from", None)+ if me and e["from"] == me:+ m["read_by"], m["unread_by"] = [], []+ for u in chat.members:+ if u == me:+ continue+ at = chat.read_at(u, e["seq"])+ if at or store.state.get(u, {}).get("read", {}).get(chat.id, 0) >= e["seq"]:+ m["read_by"].append({"id": u, "name": store.display(u), "at": at})+ else:+ m["unread_by"].append(u)+ return m+++def _named(store: Store, m: dict) -> dict:+ return {**m, "from_name": store.display(m["from"])}+++def _summary(store: Store, me: str, chat: Chat) -> dict:+ position, mentions, last = store.position(me, chat), chat.mentions.get(me), store.last(me, chat)+ return {"id": chat.id, "kind": "channel" if chat.name else "direct", "name": chat.name,+ "description": chat.description, "public": chat.public, "archived": chat.archived,+ "managers": chat.managers, "title": _title(store, me, chat),+ "members": [{"id": u, "name": store.display(u)} for u in chat.members],+ "unread": store.unread(me, chat), "mentioned": _mentioned(store, me, chat, mentions or (), position),+ "notify": store.level(me, chat), "last_at": chat.at,+ "last": _message(store, chat, last, me) if last else None}+++def _mentioned(store: Store, me: str, chat: Chat, mentions, position: int) -> bool:+ """Whether a message after my read position that I may read mentions me. An edit can add a mention to a message+ from before I joined: under while_member that one does not count."""+ for seq in reversed(mentions):+ if seq <= position:+ return False+ if store.sees(me, chat, seq):+ return True+ return False+++def _chat_titles(store: Store, me: str, messages: list) -> dict:+ return {m["chat"]: {"title": _title(store, me, store.chats[m["chat"]]), "name": store.chats[m["chat"]].name}+ for m in messages}+++def _mark(store: Store, me: str, messages: list):+ """Marks read everything up to the newest of these messages in each chat: what a wait hands over."""+ newest = {}+ for m in messages:+ newest[m["chat"]] = max(newest.get(m["chat"], 0), m["seq"])+ for chat_id, seq in newest.items():+ store.mark_read(me, store.chats[chat_id], seq)+++def _mark_shown(store: Store, me: str, shown: dict):+ """Marks read what a read showed ({chat: seqs}), up to the first unread message it did not show in each chat. A+ search or filter leaves the messages it skipped unread, so a sender cannot make me miss one by writing another that+ matches."""+ for chat_id, seqs in shown.items():+ chat = store.chats.get(chat_id)+ if chat and me in chat.members:+ upto = store.shown_upto(me, chat, {int(s) for s in seqs})+ if upto:+ store.mark_read(me, chat, upto)+++def _time(store: Store, me: str, value: str | None):+ """Retain a message's sequence number when it is used as a history bound."""+ if value and MESSAGE_ID.fullmatch(value):+ chat, message = store.message(me, value)+ return message["at"], chat.id, message["seq"]+ return value+++def _compare_bound(chat: Chat, event: dict, bound) -> int:+ if isinstance(bound, tuple):+ left, right = (event["seq"], bound[2]) if chat.id == bound[1] else ((event["at"], chat.id, event["seq"]), bound)+ else:+ left, right = event["at"], bound+ return (left > right) - (left < right)+++def _ordered(messages: list) -> list:+ # Choose among chat heads, preserving sequence order even in older logs whose+ # timestamps disagree with commit order. A cursor must never jump over a head.+ grouped = {}+ for message in messages:+ grouped.setdefault(message["chat"], []).append(message)+ return list(heapq.merge(*grouped.values(), key=lambda m: (m["at"], m["chat"], m["seq"])))+++# ---------------------------------------------------------------------- requests for every user++def whoami(store: Store, me: str, p: dict) -> dict:+ return {"user": store.public(me)}+++def people(store: Store, me: str, p: dict) -> dict:+ found = [store.public(u) for u in sorted(store.users)]+ q = (p.get("search") or "").lower()+ if q:+ found = [u for u in found if q in " ".join([u["id"], u["name"], u["status"], *map(str, u["fields"].values())]).lower()]+ return {"people": found}+++def chats(store: Store, me: str, p: dict) -> dict:+ found = [_summary(store, me, c) for c in store.my_chats(me)]+ if p.get("unread"):+ found = [c for c in found if c["unread"]]+ if p.get("with"):+ wanted = {store.user(u) for u in _list(p["with"])}+ found = [c for c in found if wanted <= {m["id"] for m in c["members"]}]+ if p.get("name"):+ q = p["name"].lower().lstrip("#")+ found = [c for c in found if q in c["title"].lower()]+ order = p.get("sort") or "recent"+ keys = {"recent": lambda c: c["last_at"], "unread": lambda c: (c["unread"], c["last_at"]), "name": lambda c: c["title"].lower()}+ if order not in keys:+ raise Error(400, f"sort is one of {', '.join(keys)}.")+ found.sort(key=keys[order], reverse=order != "name")+ return {"chats": found[:_int(p, "limit", 20)]}+++def _instant(bound) -> float | None:+ """A time bound in seconds, when it is written as messages' times are, else None: then no message is skipped on its+ time alone, and each is compared as it is read."""+ at = bound[0] if isinstance(bound, tuple) else bound+ try:+ return epoch(at) if isinstance(at, str) and len(at) == 24 and at.endswith("Z") else None+ except ValueError:+ return None+++def _pick(store: Store, me: str, chat: Chat, places, n: int, since, before, search: str) -> list:+ """Up to n of the messages at these places in the chat, in their order, keeping those after `since`, before+ `before` and containing `search`. Reads only what it keeps or must look at, a batch at a time."""+ if not (since or before or search):+ return [_message(store, chat, e, me) for e in chat.read([chat.messages[i] for i in itertools.islice(places, n)])]+ low, high = (_instant(b) if b and not (isinstance(b, tuple) and b[1] == chat.id) else None for b in (since, before))+ times = chat.times+ places = (i for i in places if (low is None or times[i] >= low) and (high is None or times[i] <= high))+ found = []+ while len(found) < n:+ batch = [chat.messages[i] for i in itertools.islice(places, 256)]+ if not batch:+ break+ for e in chat.read(batch):+ if (since and _compare_bound(chat, e, since) <= 0) or (before and _compare_bound(chat, e, before) >= 0):+ continue+ m = _message(store, chat, e, me)+ if not search or search in m["text"].lower():+ found.append(m)+ return found[:n]+++def read(store: Store, me: str, p: dict) -> dict:+ """Unread messages by default. --last, --all, --since, --before or --search read history instead. --thread reads a+ whole thread, each message with its depth in it, and moves no read position: the thread's newest reply may be newer+ than messages elsewhere in the chat that are still unread.++ One reply holds at most max_page messages, and only those are read from disk. When more match, `more` is true, and+ the same request with the reply's `cursor` gives the next page: later messages for unread ones and threads,+ earlier ones for history."""+ fallback, more, most = False, False, store.settings["max_page"]+ cursor = p.get("cursor") or {}+ if not isinstance(cursor, dict):+ raise Error(400, "cursor is the one a previous reply gave.")+ if p.get("thread"):+ chat, tree = store.thread(me, p["thread"])+ skip, limit = _int(cursor, "skip", 0), min(_int(p, "limit", 0) or most, most)+ page, more = tree[skip:skip + limit], skip + limit < len(tree)+ depth = dict(page)+ targets, keep, cursor = [chat], "first", {"skip": skip + len(page)}+ messages = [{**_message(store, chat, e, me), "depth": depth[e["seq"]]} for e in chat.read([s for s, _ in page])]+ elif p.get("id"):+ chat, e = store.message(me, p["id"])+ targets, messages, keep = [chat], [_message(store, chat, e, me)], "first"+ else:+ targets = [store.chat(me, p["chat"])] if p.get("chat") else store.my_chats(me)+ senders = {store.user(u) for u in _list(p.get("from"))}+ since, before = _time(store, me, p.get("since")), _time(store, me, p.get("before"))+ search = (p.get("search") or "").lower()+ history = bool(p.get("last") or p.get("all") or since or before or search)+ keep = "last" if history else "first"+ limit = min(_int(p, "limit", 0) or _int(p, "last", 0) or (20 if history else 50), most)+ messages, cursor = [], {c: int(seq) for c, seq in cursor.items()}+ for chat in targets:+ # History goes back from the newest message, or from where the previous page stopped. Unread messages go on+ # from my read position, or from where the previous page stopped, and leave out my own.+ if history:+ after, upto = 0, cursor.get(chat.id)+ else:+ after, upto = max(store.position(me, chat), cursor.get(chat.id, 0)), None+ if isinstance(since, tuple) and since[1] == chat.id: # a message of this chat bounds it by seq+ after = max(after, since[2])+ if isinstance(before, tuple) and before[1] == chat.id:+ upto = before[2] if upto is None else min(upto, before[2])+ mine, sent = set(chat.mentions.get(me, ())) if p.get("mentions") else None, chat.senders+ places = (i for i in store.indexes(me, chat, after, upto, newest=history)+ if (history or sent[i] != me) and (not senders or sent[i] in senders)+ and (mine is None or chat.messages[i] in mine))+ found = _pick(store, me, chat, places, limit + 1, since, before, search)+ more = more or len(found) > limit+ messages += found[:limit][::-1] if history else found[:limit] # each chat oldest first, for _ordered+ if not history and p.get("chat") and not messages and not senders and not p.get("mentions") and not cursor:+ fallback, keep = True, "last" # nothing unread in this chat: show its latest messages instead+ chat = targets[0]+ seqs = [chat.messages[i] for i in itertools.islice(store.indexes(me, chat, newest=True), 20)][::-1]+ messages = [_message(store, chat, e, me) for e in chat.read(seqs)]+ messages = _ordered(messages)+ more = more or len(messages) > limit+ messages = messages[:limit] if keep == "first" else messages[-limit:]+ for m in messages: # where the next page starts in each chat+ c = m["chat"]+ cursor[c] = max(cursor.get(c, 0), m["seq"]) if keep == "first" else min(cursor.get(c, m["seq"]), m["seq"])+ if not p.get("peek") and not p.get("thread"):+ shown = {}+ for m in messages:+ shown.setdefault(m["chat"], []).append(m["seq"])+ _mark_shown(store, me, shown)+ left = sum(store.unread(me, c) for c in targets)+ return {"messages": messages, "chats": _chat_titles(store, me, messages), "unread_left": left,+ "fallback": fallback, "keep": keep, "more": more and not fallback, "cursor": cursor if more else None}+++def confirm(store: Store, me: str, positions: dict):+ """Marks read up to these positions ({chat: seq}), in the chats I am still in."""+ for chat_id, seq in (positions or {}).items():+ chat = store.chats.get(chat_id)+ if chat and me in chat.members:+ store.mark_read(me, chat, int(seq))+++def mark_read(store: Store, me: str, p: dict) -> dict:+ """Marks read up to `positions` ({chat: seq}), and what a read showed (`shown`, {chat: [seq, ...]}): see+ _mark_shown."""+ confirm(store, me, p.get("positions"))+ if not isinstance(p.get("shown") or {}, dict):+ raise Error(400, "shown is {chat: [message seqs]}.")+ _mark_shown(store, me, p.get("shown") or {})+ targets = [store.chat(me, p["chat"])] if p.get("chat") else store.my_chats(me)+ return {"ok": True, "unread_left": sum(store.unread(me, c) for c in targets)}+++def send(store: Store, me: str, p: dict) -> dict:+ if not p.get("chat"):+ raise Error(400, "Say where to send: a #channel, a person, people separated by commas, or a chat id.")+ chat = store.chat(me, p["chat"], create=True)+ message = _named(store, store.send(me, chat, p.get("text") or "", p.get("reply_to")))+ return {"message": message, "chat": _summary(store, me, chat)}+++def send_files(store: Store, me: str, header: dict, read, body_size: int) -> dict:+ """A message with attachments. `header` is like send's parameters plus files: [{name, size, type, folder}].+ read(n) then gives the files' bytes one after another, body_size bytes in all. Everything is checked before the+ bytes are read, and nothing is kept unless the whole message is written."""+ s, files = store.settings, header.get("files")+ if not isinstance(files, list) or not files:+ raise Error(400, "Name at least one file.")+ if len(files) > s["max_attachments"]:+ raise Error(413, f"At most {s['max_attachments']} files on one message.")+ for f in files:+ if not isinstance(f, dict) or not isinstance(f.get("size"), int) or isinstance(f["size"], bool) or f["size"] < 0:+ raise Error(400, "Each file needs a name and a whole-number size.")+ if f["size"] > s["max_attachment"]:+ raise Error(413, f"{safe_name(f.get('name'))} is larger than {s['max_attachment']} bytes.")+ if sum(f["size"] for f in files) != body_size:+ raise Error(400, "The file sizes do not add up to the request's length.")+ text = header.get("text") or ""+ if len(text) > s["max_text"]:+ raise Error(400, f"The message is longer than {s['max_text']} characters.")+ if not header.get("chat"):+ raise Error(400, "Say where to send: a #channel, a person, people separated by commas, or a chat id.")+ chat = store.chat(me, header["chat"], create=True)+ store._member(me, chat)+ store._open(chat)+ refusal = store.policy.send(me, chat.members, chat.name, chat.id)+ if refusal:+ raise Error(403, refusal)+ with store.reserve(me, body_size): # room on the disk and in my quota, taken before a byte is read+ received, attachments = [], []+ try:+ for f in files:+ received.append(store.blobs.receive(read, f["size"]))+ for f, (tmp, sha) in zip(files, received):+ store.blobs.keep(tmp, sha)+ name = safe_name(f.get("name"))+ # The type goes into the Content-Type header of every download: only a well-formed one is kept.+ kind = mime(f.get("type")) or mimetypes.guess_type(name)[0] or "application/octet-stream"+ attachments.append({"id": "a" + secrets.token_hex(8), "name": name, "size": f["size"], "type": kind,+ "sha256": sha, **({"folder": True} if f.get("folder") else {})})+ except BaseException as e:+ for a in attachments:+ store.blobs.release(a["sha256"])+ for tmp, _ in received[len(attachments):]:+ with contextlib.suppress(FileNotFoundError):+ os.unlink(tmp)+ if isinstance(e, EOFError):+ raise Error(400, str(e)) from None+ if isinstance(e, OSError) and not isinstance(e, (ConnectionError, TimeoutError, socket.timeout)): # disk full+ raise Error(507, f"The server could not store the files: {e.strerror or e}") from None+ raise+ message = _named(store, store.send(me, chat, text, header.get("reply_to"), attachments))+ return {"message": message, "chat": _summary(store, me, chat)}+++def download(store: Store, me: str, p: dict) -> tuple:+ """(the attachment, the path of its bytes): only for members of its chat, while its message stands."""+ return store.attachment(me, p.get("id") or "")+++def edit(store: Store, me: str, p: dict) -> dict:+ return {"message": _named(store, store.change(me, p.get("id") or "", "edit", text=p.get("text") or ""))}+++def delete(store: Store, me: str, p: dict) -> dict:+ return {"message": _named(store, store.change(me, p.get("id") or "", "delete"))}+++def react(store: Store, me: str, p: dict) -> dict:+ emoji = (p.get("emoji") or "").strip()+ if not emoji or len(emoji) > 40:+ raise Error(400, "Give an emoji or a short word, up to 40 characters.")+ return {"message": _named(store, store.change(me, p.get("id") or "", "react", emoji=emoji,+ remove=bool(p.get("remove"))))}+++def channel_create(store: Store, me: str, p: dict) -> dict:+ chat = store.create_channel(me, p.get("name") or "", _list(p.get("members")), bool(p.get("public")),+ p.get("description") or "")+ return {"chat": _summary(store, me, chat)}+++def channel_list(store: Store, me: str, p: dict) -> dict:+ """Public channels, which anyone may join, most recently active first. Archived ones take no one, so they are+ left out. The count of messages is of those the caller may read: where members read only what was sent while they+ were in, someone outside learns no count, as joining would show them none of those messages."""+ q = (p.get("search") or "").strip().lstrip("#").lower()+ found = sorted((c for c in list(store.chats.values()) if c.public and not c.archived and q in c.name.lower()),+ key=lambda c: c.at, reverse=True)+ return {"channels": [{"id": c.id, "name": c.name, "title": f"#{c.name}", "description": c.description,+ "members": len(c.members), "last_at": c.at,+ "messages": store.count(me, c) if me in c.members or not store.limited(c) else None,+ "joined": me in c.members}+ for c in found[:_int(p, "limit", 50)]]}+++def channel_join(store: Store, me: str, p: dict) -> dict:+ return {"chat": _summary(store, me, store.join(me, p.get("chat") or ""))}+++def channel_visibility(store: Store, me: str, p: dict) -> dict:+ chat, visibility = store.chat(me, p.get("chat") or ""), p.get("visibility")+ if visibility not in ("public", "private"):+ raise Error(400, "visibility is public or private.")+ store.set_public(me, chat, visibility == "public")+ return {"chat": _summary(store, me, chat)}+++def chat_info(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ info = _summary(store, me, chat)+ info["members"] = [{**store.public(u), "name": store.display(u)} for u in chat.members]+ info.update(created_by=chat.meta["created_by"], created_at=chat.meta["created_at"], messages=store.count(me, chat))+ return {"chat": info}+++def channel_add(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ store.change_members(me, chat, add=_list(p.get("users")))+ return {"chat": _summary(store, me, chat)}+++def channel_remove(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ store.change_members(me, chat, remove=_list(p.get("users")))+ return {"chat": _summary(store, me, chat) if me in chat.members else {"id": chat.id}}+++def channel_leave(store: Store, me: str, p: dict) -> dict:+ return channel_remove(store, me, {"chat": p.get("chat"), "users": [me]})+++def channel_rename(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ store.rename(me, chat, p.get("name") or "")+ return {"chat": _summary(store, me, chat)}+++def channel_describe(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ store.describe(me, chat, p.get("description") or "")+ return {"chat": _summary(store, me, chat)}+++def channel_managers(store: Store, me: str, p: dict) -> dict:+ """A channel's managers, after naming the ones in `add` and ending the role of the ones in `remove`."""+ chat = store.chat(me, p.get("chat") or "")+ add, remove = _list(p.get("add")), _list(p.get("remove"))+ if add or remove:+ store.set_managers(me, chat, add, remove)+ return {"chat": _summary(store, me, chat)}+++def channel_archive(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ store.archive(me, chat, True)+ return {"chat": _summary(store, me, chat)}+++def channel_unarchive(store: Store, me: str, p: dict) -> dict:+ chat = store.chat(me, p.get("chat") or "")+ store.archive(me, chat, False)+ return {"chat": _summary(store, me, chat)}+++def notifications(store: Store, me: str, p: dict) -> dict:+ """Chats that need me, following my notification level for each: all, mentions or none."""+ found = []+ for chat in store.my_chats(me):+ s = _summary(store, me, chat)+ if p.get("all") or (s["notify"] == "all" and s["unread"]) or (s["notify"] == "mentions" and s["mentioned"]):+ found.append(s)+ found.sort(key=lambda c: c["last_at"], reverse=True)+ return {"chats": found[:_int(p, "limit", 20)], "unread": sum(c["unread"] for c in found),+ "mentions": sum(c["mentioned"] for c in found)}+++def _pending(store: Store, me: str, p: dict, only: Chat | None, senders: set, after: dict):+ """Messages a wait would return now, and how far it looked in each chat."""+ found, positions = [], {}+ for chat in [only] if only else store.my_chats(me):+ level = "all" if (only or p.get("any")) else store.level(me, chat)+ start = int(after.get(chat.id, store.position(me, chat)))+ newest = next((i for i in store.indexes(me, chat, start, newest=True) if chat.senders[i] != me), None)+ positions[chat.id] = start if newest is None else chat.messages[newest]+ if level == "none":+ continue+ mine = set(chat.mentions.get(me, ())) if p.get("mentions") or level == "mentions" else None+ remaining = MAX_WAIT_MESSAGES - len(found)+ seqs = [chat.messages[i] for i in itertools.islice(+ (i for i in store.indexes(me, chat, start) if chat.senders[i] != me+ and (not senders or chat.senders[i] in senders) and (mine is None or chat.messages[i] in mine)),+ remaining + 1)]+ if len(seqs) > remaining:+ seqs = seqs[:remaining]+ positions[chat.id] = seqs[-1] # Remaining eligible messages have not reached the client yet.+ found += [_message(store, chat, e, me) for e in chat.read(seqs)]+ if len(found) >= MAX_WAIT_MESSAGES:+ break+ found.sort(key=lambda m: (m["at"], m["chat"], m["seq"]))+ return found, positions+++def wait(store: Store, me: str, p: dict, gone=lambda: False) -> dict:+ """Holds the request until a matching message arrives or the timeout ends. Starts from my read positions (or from+ `after`, the positions a previous wait returned), so a message that arrives between two waits is never missed.+ `gone` tells whether the caller has hung up (a killed command): then the wait ends and marks nothing read."""+ hold = store.settings["max_wait"] # the client asks again for longer waits+ timeout = float(hold if p.get("timeout") is None else p["timeout"])+ if not math.isfinite(timeout) or timeout < 0:+ raise Error(400, "timeout must be a finite number of seconds at least 0.")+ deadline = time.monotonic() + min(timeout, hold)+ only = store.chat(me, p["chat"]) if p.get("chat") else None+ senders = {store.user(u) for u in _list(p.get("from"))}+ after = p.get("after") or {}+ cond = store.signal(me)+ with cond:+ while True:+ # Access is checked again on every pass: removal from a channel wakes this wait.+ if me not in store.users:+ raise Error(401, "This user no longer exists.")+ if only:+ store._member(me, only)+ found, positions = _pending(store, me, p, only, senders, after)+ # Removal changes members before any later message is written, so checking after reading drops+ # everything that arrived once access had ended. Other chats keep going meanwhile.+ found = [m for m in found if me in store.chats[m["chat"]].members]+ left = deadline - time.monotonic()+ if found or left <= 0:+ break+ # Sleep until signalled, looking every second whether the caller is still there.+ while not cond.wait(min(left, 1.0)):+ left = deadline - time.monotonic()+ if left <= 0:+ break+ if gone():+ return {"messages": [], "positions": {}, "chats": {}}+ # Marked after letting go of my signal: senders wake me through it, and must not wait on my disk writes. Never+ # for a caller that has gone, or the messages would count as read without anyone having seen them.+ if found and not p.get("peek") and not gone():+ _mark(store, me, found)+ return {"messages": found, "positions": positions, "chats": _chat_titles(store, me, found)}+++def settings_notify(store: Store, me: str, p: dict) -> dict:+ store.set_level(me, store.chat(me, p.get("chat") or ""), p.get("level") or "")+ return settings_show(store, me, p)+++def settings_status(store: Store, me: str, p: dict) -> dict:+ store.set_status(me, (p.get("text") or "").strip())+ return settings_show(store, me, p)+++def settings_token(store: Store, me: str, p: dict) -> dict:+ """A new token for me. The old one stops working at once."""+ return {"token": store.rotate_token(me)}+++def settings_show(store: Store, me: str, p: dict) -> dict:+ return {"user": store.public(me), "notify": dict(store.state[me]["notify"])}+++METHODS = {f.__name__: f for f in (+ whoami, people, chats, read, mark_read, send, edit, delete, react, chat_info, channel_create, channel_list,+ channel_join, channel_visibility, channel_add, channel_remove, channel_leave, channel_rename, channel_describe,+ channel_managers, channel_archive, channel_unarchive, notifications, wait, settings_notify, settings_status,+ settings_token, settings_show)}+++# ---------------------------------------------------------------------- requests for the super admin, and whoever+# the rules allow. `me` is the user who asks, or None for the super admin's admin token.++def user_add(store: Store, me, p: dict) -> dict:+ store.require(me, "add_user")+ token = store.add_user(p.get("id") or "", p.get("name") or "", p.get("fields") or {})+ return {"user": store.public(p["id"].strip().lower()), "token": token}+++def user_remove(store: Store, me, p: dict) -> dict:+ store.require(me, "remove_user")+ store.remove_user(p.get("id") or "")+ return {"ok": True}+++def token_rotate(store: Store, me, p: dict) -> dict:+ store.require(me, "rotate_token")+ return {"token": store.rotate_token(p.get("id") or "")}+++def policy_set(store: Store, me, p: dict) -> dict:+ if me is not None: # the rules decide every other permission, so they stay with the super admin+ raise Error(403, "Only the super admin may set the rules.")+ if not isinstance(p.get("rules") or {}, dict):+ raise Error(400, "The rules are a JSON object.")+ store.set_policy(p.get("rules") or {})+ return {"rules": store.policy.rules}+++def policy_show(store: Store, me, p: dict) -> dict:+ store.require(me, "show_rules")+ return {"rules": store.policy.rules}+++ROLE_LISTS = {"admin": ("admins", "edit_admins"), "channel_creator": ("channel_creators", "edit_channel_creators")}+++def role_change(store: Store, me, p: dict) -> dict:+ """Shows who holds a role, or changes it: admin, or channel_creator, where `everyone` makes every user one."""+ role, add, remove, everyone = p.get("role"), _list(p.get("add")), _list(p.get("remove")), bool(p.get("everyone"))+ if role not in ROLE_LISTS:+ raise Error(400, "The role is admin or channel_creator.")+ if everyone and (role != "channel_creator" or add or remove):+ raise Error(400, "Only channel_creator may be everyone, and then without names to add or remove.")+ key, action = ROLE_LISTS[role]+ store.require(me, action if add or remove or everyone else "show_rules")+ users = store.change_role(key, add, remove, everyone) if add or remove or everyone else store.policy.rules.get(key)+ return {"role": role, "users": [] if users is None and role == "admin" else users} # None: every user+++ADMIN = {f.__name__: f for f in (user_add, user_remove, token_rotate, policy_set, policy_show, role_change)}diff --git a/swarmchat/cli.py b/swarmchat/cli.pynew file mode 100644index 0000000..30c21be--- /dev/null+++ b/swarmchat/cli.py@@ -0,0 +1,732 @@+"""The command line. Each command is a client call and a printer. `--json` prints one JSON object per line instead.++Exit codes: 0 done, 1 bad request, 2 not found, 3 not allowed, 4 wait timed out, 5 server unreachable.+"""+from __future__ import annotations++import argparse+import json+import os+import re+import select+import signal+import socket+import sys+import threading+from datetime import datetime, timedelta, timezone++from . import DATA_DIR, ENV, NAME, __version__+from .client import Client, ClientError, read_saved_token, save_login++EXIT = {0: 5, 400: 1, 401: 3, 403: 3, 404: 2, 413: 1}+# What makes a terminal do something rather than show something: control characters (escape sequences, carriage+# returns, OSC 52 clipboard writes), C1 controls, and the marks that reverse the direction of text. Shown escaped.+UNSAFE = re.compile("[\x00-\x08\x0b-\x1f\x7f-\x9f\u061c\u200e\u200f\u2028\u2029\u202a-\u202e\u2066-\u2069]")+UNSAFE_JSON = re.compile("[\x7f-\x9f\u061c\u200e\u200f\u2028\u2029\u202a-\u202e\u2066-\u2069]") # json.dumps escapes the rest+++# ---------------------------------------------------------------------- printing++def clean(text: str, lines: bool = True) -> str:+ """Text from the server, safe to print: what UNSAFE matches is shown as \\x1b or \\u202e, and so are line+ breaks unless `lines`. Agents write these texts, and another agent's terminal must not act on them."""+ text = UNSAFE.sub(lambda m: f"\\x{ord(m[0]):02x}" if ord(m[0]) < 0x100 else f"\\u{ord(m[0]):04x}", text)+ return text if lines else text.replace("\n", "\\n")+++def tidy(value, key: str | None = None):+ """A reply made safe to print: every string cleaned, with line breaks kept only in message texts and+ descriptions, which print as lines of their own."""+ if isinstance(value, str):+ return clean(value, key in ("text", "description"))+ if isinstance(value, dict):+ return {clean(k, False) if isinstance(k, str) else k: tidy(v, k) for k, v in value.items()}+ if isinstance(value, list):+ return [tidy(v, key) for v in value]+ return value+++class Shown:+ """A client whose replies are tidied for printing. The --json output and --where filters see them as sent."""++ def __init__(self, client: Client):+ self.client = client++ def __getattr__(self, name: str):+ value = getattr(self.client, name)+ return (lambda *args, **kwargs: tidy(value(*args, **kwargs))) if callable(value) else value+++def say(*parts, lines: bool = True, **kwargs):+ """print(), with every part cleaned: the last guard, for whatever a printer shows."""+ print(*(clean(str(part), lines) for part in parts), **kwargs)+++def json_line(item) -> str:+ """One JSON object on one line. json.dumps escapes control characters, but not C1 controls or the marks that+ reverse text: those are escaped too, which leaves the same JSON values."""+ return UNSAFE_JSON.sub(lambda m: f"\\u{ord(m[0]):04x}", json.dumps(item, ensure_ascii=False))+++def local(at: str) -> str:+ return datetime.fromisoformat(at.replace("Z", "+00:00")).astimezone().strftime("%Y-%m-%d %H:%M")+++def when(value: str | None) -> str | None:+ """A time from the command line, sent as UTC: 30m, 2h or 3d ago, a local date or time, or a message id."""+ if not value:+ return value+ ago = re.fullmatch(r"(\d+)([smhd])", value)+ if ago:+ unit = {"s": "seconds", "m": "minutes", "h": "hours", "d": "days"}[ago[2]]+ moment = datetime.now(timezone.utc) - timedelta(**{unit: int(ago[1])})+ else:+ try:+ moment = datetime.fromisoformat(value.replace(" ", "T").replace("Z", "+00:00"))+ except ValueError:+ return value # a message id+ if moment.tzinfo is None:+ moment = moment.astimezone()+ return moment.astimezone(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")+++def receipt(read_by: list, unread_by: list) -> str:+ """Read receipts on my own message: who has read it."""+ total = len(read_by) + len(unread_by)+ if not read_by:+ return "not read yet"+ if not unread_by:+ return "read" if total == 1 else "read by all"+ names = ", ".join(r["name"] for r in read_by[:3]) + (f" and {len(read_by) - 3} more" if len(read_by) > 3 else "")+ return f"read by {names}, {len(read_by)} of {total}"+++def message_line(m: dict) -> str:+ text = "(deleted)" if m.get("deleted") else m.get("text", "")+ answered = f"{m['reply_to_from']}'s {m['reply_to']}" if m.get("reply_to_from") else m.get("reply_to")+ notes = [f"reply to {answered}"] if m.get("reply_to") and "depth" not in m else [] # a thread indents instead+ if m.get("edited_at"):+ notes.append("edited")+ if "read_by" in m:+ notes.append(receipt(m["read_by"], m["unread_by"]))+ reactions = " ".join(f"{emoji} {', '.join(users)}" for emoji, users in (m.get("reactions") or {}).items())+ head = f"{m['id']} {local(m['at'])} {m.get('from_name', m['from'])}: "+ lines = text.split("\n")+ out = head + lines[0] + "".join(f"\n {line}" for line in lines[1:])+ if notes:+ out += f" ({', '.join(notes)})"+ for f in m.get("attachments") or []:+ out += f"\n [{'folder' if f.get('folder') else 'file'}] {f['name']} {size(f['size'])} {f['id']}"+ return out + (f"\n [{reactions}]" if reactions else "")+++def size(n: float) -> str:+ for unit in ("bytes", "KB", "MB"):+ if n < 1000:+ return f"{n:.0f} {unit}" if unit == "bytes" else f"{n:.1f} {unit}"+ n /= 1000+ return f"{n:.1f} GB"+++def print_messages(result: dict):+ titles, current = result.get("chats", {}), None+ for m in result["messages"]:+ if m["chat"] != current:+ current = m["chat"]+ say(f"== {titles.get(current, {}).get('title', current)} [{current}]")+ indent = " " * m.get("depth", 0) # in a thread, a reply sits under the message it answers+ say("\n".join(indent + line for line in message_line(m).split("\n")))+++def chat_line(c: dict) -> str:+ unread = f"{c['unread']} unread" + (", mentions you" if c.get("mentioned") else "") if c["unread"] else "read"+ last = c.get("last")+ preview = f" {local(last['at'])} {last['from_name']}: {last['text'][:70]}".replace("\n", " ") if last else ""+ muted = "" if c.get("notify", "all") == "all" else f" [notify {c['notify']}]"+ public = " [public]" if c.get("public") else ""+ archived = " [archived]" if c.get("archived") else ""+ return f"{c['id']} {c['title']}{public}{archived} ({unread}){muted}{preview}"+++def emit(args, items: list, printer):+ if args.json:+ for item in items:+ print(json_line(item))+ else:+ printer(items)+++# ---------------------------------------------------------------------- commands++def cmd_whoami(a, cl):+ u = cl.whoami()["user"]+ emit(a, [u], lambda items: say(f"{u['id']} {u['name']} server {cl.server}"))+++def cmd_people(a, cl):+ def show(people):+ for u in people:+ fields = " ".join(f"{k}={v}" for k, v in u["fields"].items())+ say(f"{u['id']} {u['name']}" + (f" {fields}" if fields else "") + (f" ({u['status']})" if u["status"] else ""))+ emit(a, cl.people(search=a.search)["people"], show)+++def cmd_chats(a, cl):+ def show(chats):+ for c in chats:+ say(chat_line(c))+ if not chats:+ say("No chats.")+ emit(a, cl.chats(where=a.where, limit=a.limit, unread=a.unread, name=a.name, sort=a.sort,+ **{"with": a.with_})["chats"], show)+++def cmd_read(a, cl):+ r = cl.read(where=a.where, sort=a.sort, peek=a.peek, limit=a.limit, chat=a.chat, id=a.id, thread=a.thread,+ last=a.last, all=a.all, since=when(a.since), before=when(a.before), search=a.search,+ mentions=a.mentions, **{"from": a.from_})+ if a.json:+ return emit(a, r["messages"], None)+ if r.get("fallback"):+ say("(nothing unread here, showing the latest messages)")+ print_messages(r)+ if not r["messages"] and not r.get("fallback"):+ say("No new messages." if not (a.last or a.all or a.since or a.before or a.search) else "No messages match.")+ if r["unread_left"]:+ say(f"{r['unread_left']} more unread. Run `{NAME} read` again to see them.")+++def cmd_send(a, cl):+ text = sys.stdin.read() if a.text == "-" else a.text+ r = cl.send(chat=a.chat, text=text, attach=a.attach or (), reply_to=a.reply_to)+ files = r["message"].get("attachments") or []+ emit(a, [r["message"]], lambda _: say(f"Sent {r['message']['id']} to {r['chat']['title']} [{r['chat']['id']}]"+ + (f" with {len(files)} attachment{'s' * (len(files) != 1)}: "+ + ", ".join(f"{f['name']} {f['id']}" for f in files) if files else "")+ + "."))+++def cmd_download(a, cl):+ path = cl.download(a.attachment, to=a.to, force=a.force)+ emit(a, [{"id": a.attachment, "path": path}], lambda _: say(f"Saved {path}"))+++def cmd_edit(a, cl):+ emit(a, [cl.edit(id=a.message, text=a.text)["message"]], lambda ms: say(message_line(ms[0])))+++def cmd_delete(a, cl):+ emit(a, [cl.delete(id=a.message)["message"]], lambda ms: say(f"Deleted {ms[0]['id']}."))+++def cmd_react(a, cl):+ emit(a, [cl.react(id=a.message, emoji=a.emoji, remove=a.remove)["message"]], lambda ms: say(message_line(ms[0])))+++def cmd_info(a, cl):+ c = cl.chat_info(chat=a.chat)["chat"]+ members = "\n".join(f" {u['id']} {u['name']}" + (f" ({u['status']})" if u["status"] else "") for u in c["members"])+ about = f"{c['description']}\n" if c.get("description") else ""+ about += f"Managers: {', '.join(c['managers'])}\n" if c.get("managers") else ""+ emit(a, [c], lambda _: say(f"{chat_line(c)}\n{about}{c['messages']} message{'s' * (c['messages'] != 1)}. Created "+ f"by {c['created_by']} on {local(c['created_at'])}. Members:\n{members}"))+++def cmd_channel(a, cl):+ if a.action == "list":+ def show(channels):+ for c in channels:+ count = "" if c["messages"] is None else f"{c['messages']} message{'s' * (c['messages'] != 1)}, "+ about = c.get("description", "").replace("\n", "\n ")+ say(f"{c['id']} {c['title']} {c['members']} member{'s' * (c['members'] != 1)}, {count}"+ f"last {local(c['last_at'])}" + (" (joined)" if c["joined"] else "") + (f"\n {about}" if about else ""))+ if not channels:+ say("No public channels.")+ return emit(a, cl.channel_list(search=a.search, limit=a.limit)["channels"], show)+ if a.action == "create":+ r = cl.channel_create(name=a.name, members=a.members, public=a.public, description=a.description)+ elif a.action == "join":+ r = cl.channel_join(chat=a.channel)+ elif a.action == "visibility":+ r = cl.channel_visibility(chat=a.channel, visibility=a.visibility)+ elif a.action in ("add", "remove"):+ r = getattr(cl, f"channel_{a.action}")(chat=a.channel, users=a.users)+ elif a.action == "rename":+ r = cl.channel_rename(chat=a.channel, name=a.name)+ elif a.action == "describe":+ r = cl.channel_describe(chat=a.channel, description=a.description)+ elif a.action == "managers":+ r = cl.channel_managers(chat=a.channel, add=a.add, remove=a.remove)+ return emit(a, [r["chat"]], lambda _: say(f"Managers of {r['chat']['title']}: "+ f"{', '.join(r['chat']['managers']) or 'none'}"))+ elif a.action in ("archive", "unarchive"):+ r = getattr(cl, f"channel_{a.action}")(chat=a.channel)+ else:+ r = cl.channel_leave(chat=a.channel)+ return emit(a, [r["chat"]], lambda _: say(f"You left {a.channel}."))+ emit(a, [r["chat"]], lambda chats: say(chat_line(chats[0])))+++def cmd_notifications(a, cl):+ r = cl.notifications(all=a.all, limit=a.limit)+ if a.json:+ return emit(a, r["chats"], None)+ if not r["chats"]:+ return say("Nothing new.")+ for c in r["chats"]:+ say(chat_line(c))+ mentions = {0: "", 1: ", 1 chat mentions you"}.get(r["mentions"], f", {r['mentions']} chats mention you")+ say(f"{r['unread']} unread{mentions}.")+++def cmd_wait(a, cl):+ r = cl.wait(timeout=a.timeout, where=a.where, peek=a.peek, chat=a.chat, mentions=a.mentions, any=a.any,+ **{"from": a.from_})+ if not r["messages"]:+ if not a.json:+ say("No new messages before the timeout.")+ return 4+ if a.json:+ return emit(a, r["messages"], None)+ print_messages(r)+++def cmd_settings(a, cl):+ if a.action == "notify":+ r = cl.settings_notify(chat=a.chat, level=a.level)+ elif a.action == "status":+ r = cl.settings_status(text=a.text)+ elif a.action == "token":+ r = cl.settings_token()+ saved = cl.token == read_saved_token()+ if saved:+ save_login(cl.server, r["token"])+ return emit(a, [r], lambda _: say("Your old token no longer works. " + (+ "The new one is saved in your login." if saved else f"Your new token: {r['token']}")))+ else:+ r = cl.settings_show()+ emit(a, [r], lambda _: say(f"{r['user']['id']} {r['user']['name']} status: {r['user']['status'] or '-'}\n"+ + ("notify: " + ", ".join(f"{k}={v}" for k, v in r["notify"].items())+ if r["notify"] else "notify: all for every chat")))+++def cmd_login(a, cl):+ check = Client(a.server_url, a.token_value)+ cmd_whoami(a, check if a.json else Shown(check)) # saved only once the server answers to this token+ save_login(a.server_url, a.token_value)+++def cmd_serve(a, cl):+ from . import config+ from .server import serve+ serve(config.load(a.config, **{k: getattr(a, k) for k in config.DEFAULTS}))+++def cmd_view(a, cl):+ """Opens the viewer in a browser: on the server running on a data folder, or on --server with a view token.+ When nothing is running on the folder, it shows the folder itself, read-only, until Ctrl-C."""+ token = a.view_token or os.environ.get(f"{ENV}_VIEW_TOKEN") or a.admin_token or os.environ.get(f"{ENV}_ADMIN_TOKEN")+ server = a.server+ data = a.data or (DATA_DIR if not token and not server else None) # with nothing given, the default folder+ if data:+ if not a.data and not os.path.isdir(data):+ raise ClientError(400, f"No server folder at {data}. Give the server's folder with --data DIR, or a view "+ f"token with --view-token or ${ENV}_VIEW_TOKEN. The server writes it to DIR/view.token.")+ if not token and os.path.exists(os.path.join(data, "view.token")):+ with open(os.path.join(data, "view.token")) as f:+ token = f.read().strip()+ if not server and os.path.exists(os.path.join(data, "address")):+ with open(os.path.join(data, "address")) as f:+ server = f.read().strip()+ elif not token:+ raise ClientError(400, f"Give the server's folder with --data DIR, or a view token with --view-token or "+ f"${ENV}_VIEW_TOKEN. The server writes it to DIR/view.token.")+ live = None+ if token and (server or not data):+ check = Client(server, token)+ try:+ check.view_info()+ live = check.server+ except ClientError as e:+ if not data:+ raise+ if e.status not in (0, 401, 404):+ raise+ finally:+ check.close()+ if live:+ return show_viewer(a, f"{live.rstrip('/')}/view#token={token}")+ from . import config+ from .server import make_server+ httpd, store = make_server(config.load(data=data, host=a.host, port=a.port if a.port is not None else 0),+ read_only=True)+ try:+ say(f"No server is running on {data}, so it is shown as it is on disk, read-only. Ctrl-C ends it.")+ show_viewer(a, f"{httpd.url}/view#token={store.view_token}")+ httpd.serve_forever()+ except KeyboardInterrupt:+ pass+ finally:+ httpd.server_close()+ store.close()+++def show_viewer(a, url: str):+ say(f"Viewer: {url}", flush=True)+ # Only where a browser can open: on a server without a screen, Python would start a text browser in the terminal.+ if not a.no_open and (sys.platform in ("darwin", "win32") or os.environ.get("DISPLAY") or+ os.environ.get("WAYLAND_DISPLAY")):+ import webbrowser+ webbrowser.open(url)+++def cmd_admin(a, cl):+ """With the admin token, as the super admin, or with your own token (--token, then the login) for what the+ rules allow you: see policy.py."""+ token = a.admin_token or a.token or os.environ.get(f"{ENV}_ADMIN_TOKEN")+ server = a.server+ data = a.data+ # With no token or server given at all, the default folder of a server on this machine, if there is one.+ if not (data or token or server or os.environ.get(f"{ENV}_TOKEN") or os.environ.get(f"{ENV}_SERVER")) and \+ os.path.exists(os.path.join(DATA_DIR, "admin.token")):+ data = DATA_DIR+ if data: # the server's own folder: its admin token, and the address it last started on+ if not token:+ with open(os.path.join(data, "admin.token")) as f:+ token = f.read().strip()+ if not server and os.path.exists(os.path.join(data, "address")):+ with open(os.path.join(data, "address")) as f:+ server = f.read().strip()+ admin = Client(server, token) if a.json else Shown(Client(server, token))+ if a.action == "add":+ fields = dict(f.split("=", 1) for f in a.field or [])+ r = admin.user_add(id=a.id, name=a.name, fields=fields)+ return emit(a, [r], lambda _: say(f"Added {r['user']['id']} ({r['user']['name']}). Token: {r['token']}"))+ if a.action == "remove":+ admin.user_remove(id=a.id)+ return emit(a, [{"removed": a.id}], lambda _: say(f"Removed {a.id}."))+ if a.action == "rotate":+ r = admin.token_rotate(id=a.id)+ return emit(a, [r], lambda _: say(f"New token for {a.id}: {r['token']}"))+ if a.action == "role":+ r = admin.role_change(role=a.role, add=a.add, remove=a.remove, everyone=a.everyone)+ return emit(a, [r], lambda _: say(f"{r['role']}: " + ("everyone" if r["users"] is None+ else ", ".join(r["users"]) or "no one")))+ if a.action == "policy-set":+ with open(a.file) as f:+ r = admin.policy_set(rules=json.load(f))+ else:+ r = admin.policy_show()+ emit(a, [r], lambda _: say(json.dumps(r["rules"], indent=1)))+++# ---------------------------------------------------------------------- arguments++class CommandParser(argparse.ArgumentParser):+ """A command's parser, which takes its words and flags in any order. Before Python 3.13 argparse dropped an+ optional word written after a flag: `swarmchat send bob --reply-to ID "text"` lost the text."""+ mixing = False++ def parse_known_args(self, args=None, namespace=None):+ if self._subparsers is not None or self.mixing:+ return super().parse_known_args(args, namespace)+ self.mixing = True # before 3.13, parse_known_intermixed_args() calls back into this method+ try:+ return self.parse_known_intermixed_args(args, namespace)+ finally:+ self.mixing = False+++def parser() -> argparse.ArgumentParser:+ # Flags that work anywhere in a command: `swarmchat --server URL chats` and `swarmchat chats --server URL` alike.+ # Without a default (SUPPRESS), a later parser never overwrites a value an earlier one already read. main() fills+ # the rest.+ common = argparse.ArgumentParser(add_help=False)+ common.add_argument("--json", action="store_true", default=argparse.SUPPRESS, help="print one JSON object per line")+ common.add_argument("--server", default=argparse.SUPPRESS,+ help=f"server URL (default: ${ENV}_SERVER, then the login config)")+ common.add_argument("--token", default=argparse.SUPPRESS,+ help=f"your token (default: ${ENV}_TOKEN, then the login config)")+ admin_common = argparse.ArgumentParser(add_help=False, parents=[common])+ admin_common.add_argument("--admin-token", default=argparse.SUPPRESS,+ help=f"default: ${ENV}_ADMIN_TOKEN, or DATA/admin.token with --data, or the default "+ f"folder's ({DATA_DIR}). With none of them, your own token, for what the rules "+ "allow you")+ admin_common.add_argument("--data", default=argparse.SUPPRESS,+ help="the server's folder: its admin token and address are read from there")+ p = argparse.ArgumentParser(prog=NAME, parents=[common], description="Chats: direct chats with people, and named "+ "channels. List, read, send, wait.")+ p.add_argument("--version", action="version", version=f"{NAME} {__version__}")+ sub = p.add_subparsers(dest="command", metavar="command", parser_class=CommandParser)++ def command(name, func, help, parents=(common,)):+ c = sub.add_parser(name, parents=list(parents), help=help, description=help)+ c.set_defaults(func=func)+ return c++ def where(c, what="message `m`"):+ c.add_argument("--where", metavar="EXPR", help=f"keep only items where this Python expression over the {what} "+ "is true, e.g. \"'build' in m.text and m.sender == 'maya'\"")++ command("whoami", cmd_whoami, "show who you are")+ c = command("people", cmd_people, "list everyone you can message")+ c.add_argument("--search", help="only people whose id, name, status or profile contains this")++ c = command("chats", cmd_chats, "list your chats, most recent first")+ c.add_argument("--unread", action="store_true", help="only chats with unread messages")+ c.add_argument("--with", dest="with_", metavar="PEOPLE", help="only chats that include these people (comma-separated)")+ c.add_argument("--name", help="only chats whose title contains this")+ c.add_argument("--sort", choices=["recent", "unread", "name"], default="recent")+ c.add_argument("--limit", type=int, default=20)+ where(c, "chat `c` (c.title, c.unread, c.members ...)")++ c = command("read", cmd_read, "read messages, your unread ones by default, oldest first, and mark them read")+ c.add_argument("chat", nargs="?", help="a chat id, a chat name, or people's ids (comma-separated)")+ c.add_argument("--id", metavar="MESSAGE", help="one message, by id")+ c.add_argument("--thread", metavar="MESSAGE", help="the whole thread this message is in: its first message and "+ "every reply below it, indented. Marks nothing read")+ c.add_argument("--last", type=int, metavar="N", help="the last N messages, read or not")+ c.add_argument("--all", action="store_true", help="read history, not only unread messages")+ c.add_argument("--since", metavar="WHEN", help="after a time (30m, 2h, 3d, 2026-10-02 15:00) or a message id")+ c.add_argument("--before", metavar="WHEN", help="before a time or a message id")+ c.add_argument("--from", dest="from_", metavar="PEOPLE", help="only from these people (comma-separated)")+ c.add_argument("--mentions", action="store_true", help="only messages that mention you")+ c.add_argument("--search", metavar="TEXT", help="only messages containing this text, any case")+ c.add_argument("--limit", type=int, metavar="N", help="at most N messages (default 50 unread, 20 history)")+ c.add_argument("--peek", action="store_true", help="do not mark anything read")+ where(c)+ c.add_argument("--sort", metavar="EXPR", help="order by this Python expression over the message `m`, smallest "+ "first, then keep --limit from the top. Negate a number for largest "+ "first: \"-len((m.reactions or {}).get('+1', []))\" puts the most "+ "voted first")++ c = command("send", cmd_send, "send a message to a channel, or to people (their direct chat is made on first use)")+ c.add_argument("chat", help="a channel (#release-review, quoted in a shell), a person (maya), people (maya,priya) "+ "or a chat id")+ c.add_argument("text", nargs="?", default="", help="the message, or - to read it from stdin (optional with --attach)")+ c.add_argument("--attach", action="append", metavar="PATH", help="a file or folder to send with it (repeat for more)")+ c.add_argument("--reply-to", metavar="MESSAGE")++ c = command("download", cmd_download, "save an attachment (a folder is unpacked)")+ c.add_argument("attachment", help="the attachment id shown under its message")+ c.add_argument("--to", default=".", metavar="FOLDER", help="where to save it (default: here)")+ c.add_argument("--force", action="store_true", help="replace a file or folder of the same name")++ c = command("edit", cmd_edit, "replace the text of a message you sent")+ c.add_argument("message")+ c.add_argument("text")+ c = command("delete", cmd_delete, "delete a message you sent, or anyone's if you are an administrator")+ c.add_argument("message")+ c = command("react", cmd_react, "add a reaction to a message")+ c.add_argument("message")+ c.add_argument("emoji")+ c.add_argument("--remove", action="store_true", help="take your reaction back")++ c = command("info", cmd_info, "members and details of a chat")+ c.add_argument("chat")++ c = command("channel", cmd_channel, "create, find and join channels, change their members, managers, name, "+ "description or visibility, and archive them")+ actions = c.add_subparsers(dest="action", metavar="action", required=True)+ a = actions.add_parser("create", parents=[common], help="create a channel, private unless --public")+ a.add_argument("name")+ a.add_argument("--members", help="people's ids to add, comma-separated")+ a.add_argument("--public", action="store_true", help="anyone may find it (channel list) and join it")+ a.add_argument("--description", metavar="TEXT", help="what the channel is for, up to 1,000 characters")+ a = actions.add_parser("list", parents=[common], help="public channels, which anyone may join")+ a.add_argument("--search", metavar="TEXT", help="only channels whose name contains this")+ a.add_argument("--limit", type=int, default=50)+ a = actions.add_parser("join", parents=[common], help="join a public channel")+ a.add_argument("channel")+ a = actions.add_parser("visibility", parents=[common], help="make a channel public or private (members stay)")+ a.add_argument("channel")+ a.add_argument("visibility", choices=["public", "private"])+ for action in ("add", "remove"):+ a = actions.add_parser(action, parents=[common], help=f"{action} people")+ a.add_argument("channel")+ a.add_argument("users", nargs="+")+ a = actions.add_parser("rename", parents=[common], help="rename a channel")+ a.add_argument("channel")+ a.add_argument("name")+ a = actions.add_parser("describe", parents=[common], help="change a channel's description")+ a.add_argument("channel")+ a.add_argument("description", help='the new description, or "" to remove it')+ a = actions.add_parser("managers", parents=[common], help="show a channel's managers, or name and remove them")+ a.add_argument("channel")+ a.add_argument("--add", nargs="+", metavar="USER", help="members to make managers")+ a.add_argument("--remove", nargs="+", metavar="USER", help="managers to make ordinary members")+ a = actions.add_parser("leave", parents=[common], help="leave a channel")+ a.add_argument("channel")+ a = actions.add_parser("archive", parents=[common], help="archive a channel: kept to read, closed to changes")+ a.add_argument("channel")+ a = actions.add_parser("unarchive", parents=[common], help="open an archived channel again")+ a.add_argument("channel")++ c = command("notifications", cmd_notifications, "chats that need you, following your notification settings")+ c.add_argument("--all", action="store_true", help="every chat, even ones with nothing new")+ c.add_argument("--limit", type=int, default=20)++ c = command("wait", cmd_wait, "block until a new message arrives, print it and mark it read")+ c.add_argument("--chat", help="only this chat")+ c.add_argument("--from", dest="from_", metavar="PEOPLE", help="only from these people")+ c.add_argument("--mentions", action="store_true", help="only messages that mention you")+ c.add_argument("--any", action="store_true", help="ignore your notification settings")+ c.add_argument("--timeout", type=float, metavar="SECONDS", help="give up after this long (exit code 4)")+ c.add_argument("--peek", action="store_true", help="do not mark what it prints as read")+ where(c)++ c = command("settings", cmd_settings, "notification levels and your status")+ actions = c.add_subparsers(dest="action", metavar="action", required=True)+ a = actions.add_parser("notify", parents=[common], help="all, mentions or none for a chat")+ a.add_argument("chat")+ a.add_argument("level", choices=["all", "mentions", "none"])+ a = actions.add_parser("status", parents=[common], help="set your status line")+ a.add_argument("text")+ actions.add_parser("token", parents=[common], help="give yourself a new token, when the old one may have leaked")+ actions.add_parser("show", parents=[common], help="show your settings")++ c = command("login", cmd_login, "save the server address and your token for later commands")+ c.add_argument("server_url")+ c.add_argument("token_value")++ c = command("serve", cmd_serve, "run the server")+ c.add_argument("--config", metavar="FILE", help="a JSON file of settings. Flags override it.")+ c.add_argument("--data", help=f"the folder that holds everything (default {DATA_DIR})")+ c.add_argument("--host", help="default 127.0.0.1. Use 0.0.0.0 to accept other machines.")+ c.add_argument("--port", type=int, help="default 8484")+ c.add_argument("--tls-cert", help="serve HTTPS with this certificate")+ c.add_argument("--tls-key")+ c.add_argument("--fsync", action="store_const", const=True, help="force each write to disk: survives a power cut, slower")+ c.add_argument("--verbose", action="store_const", const=True, help="log every request")+ c.add_argument("--max-text", type=int, metavar="CHARS", help="longest message (default 20000)")+ c.add_argument("--max-wait", type=float, metavar="SECONDS", help="longest one wait request is held (default 300)")+ c.add_argument("--max-body", type=int, metavar="BYTES", help="largest request (default 1048576)")+ c.add_argument("--max-attachment", type=int, metavar="BYTES", help="largest attached file or folder (default 100000000)")+ c.add_argument("--max-attachments", type=int, metavar="N", help="most files on one message (default 20)")+ c.add_argument("--request-timeout", type=float, metavar="SECONDS",+ help="idle socket and TLS handshake timeout, and the time a request's headers have (default 30)")+ c.add_argument("--min-rate", type=float, metavar="BYTES", help="slowest a request's body may arrive, in bytes a "+ "second (default 16384)")+ c.add_argument("--max-connections", type=int, metavar="N", help="most connections open at once (default 4096)")+ c.add_argument("--max-requests", type=int, metavar="N", help="most requests one user may have under way at once, "+ "waits and uploads included (default 32)")+ c.add_argument("--max-page", type=int, metavar="N", help="most messages one read returns (default 1000)")+ c.add_argument("--min-free", type=int, metavar="BYTES", help="disk the server keeps free, refusing messages and "+ "files that would use it (default 1000000000, 0 off)")+ c.add_argument("--quota", type=int, metavar="BYTES", help="most bytes of files one user may have stored (default 0, "+ "no limit)")+ c.add_argument("--default-notify", choices=["all", "mentions", "none"], help="alert level for chats nobody set (default all)")+ c.add_argument("--save-every", type=float, metavar="SECONDS", help="how often read positions are saved (default 1)")+ c.add_argument("--segment", type=int, metavar="LINES", help="lines per log file (default 10000), fixed once data exists")++ c = command("view", cmd_view, "watch every chat in a browser: live, as any user, as chats, boards or a map",+ parents=())+ c.add_argument("--data", default=argparse.SUPPRESS,+ help=f"the server's folder (default {DATA_DIR}): the viewer's token and the server's address are "+ "read from there")+ c.add_argument("--server", default=argparse.SUPPRESS,+ help="the server's URL, from another machine (with --view-token)")+ c.add_argument("--view-token", default=None, help=f"default: ${ENV}_VIEW_TOKEN, or DATA/view.token with --data")+ c.add_argument("--admin-token", default=argparse.SUPPRESS, help="the admin token also opens the viewer")+ c.add_argument("--host", default="127.0.0.1", help="with no server running: where to serve the folder")+ c.add_argument("--port", type=int, default=None, help="with no server running: the port (default: any free one)")+ c.add_argument("--no-open", action="store_true", help="print the link only, do not open a browser")++ c = command("admin", cmd_admin, "manage users, roles and rules (the admin token, or your own for what the rules "+ "allow you)", parents=(admin_common,))+ actions = c.add_subparsers(dest="action", metavar="action", required=True)+ a = actions.add_parser("add", parents=[admin_common], help="add a user and print their token")+ a.add_argument("id")+ a.add_argument("--name", default="")+ a.add_argument("--field", action="append", metavar="KEY=VALUE", help="a profile field, e.g. role=release-manager")+ for action, help in (("remove", "remove a user"), ("rotate", "give a user a new token")):+ a = actions.add_parser(action, parents=[admin_common], help=help)+ a.add_argument("id")+ a = actions.add_parser("role", parents=[admin_common], help="show who holds a role, or change it")+ a.add_argument("role", choices=["admin", "channel_creator"])+ a.add_argument("--add", nargs="+", metavar="USER")+ a.add_argument("--remove", nargs="+", metavar="USER")+ a.add_argument("--everyone", action="store_true", help="channel_creator only: every user may create channels")+ a = actions.add_parser("policy-set", parents=[admin_common], help="replace the rules with a JSON file")+ a.add_argument("file")+ actions.add_parser("policy-show", parents=[admin_common], help="show the rules")+ return p+++def run_interruptibly(func, args, client) -> int:+ """Runs a command in a worker thread while this one stays free to take Ctrl-C. On Windows a blocking network read+ is not interrupted by Ctrl-C or Ctrl-Break, so a `swarmchat wait` could not be stopped at the keyboard.++ This thread sleeps on a socket that wakes it at once when the worker ends (it writes ENDED) or a signal arrives+ (Python writes the signal's number, see signal.set_wakeup_fd). On a stop, the connection is shut down under the+ worker: its read fails at once and it cleans up (a half-written download is removed) before the command ends."""+ ENDED = b"\xff" # no signal number is that high+ outcome = {}+ wake, ring = socket.socketpair()+ ring.setblocking(False)++ def work():+ try:+ outcome["code"] = func(args, client) or 0+ except BaseException as e: # noqa: BLE001 - handed back to the main thread below+ outcome["error"] = e+ finally:+ try:+ ring.send(ENDED)+ except OSError: # the main thread gave up waiting, and the command is ending without this one+ pass++ def ended(timeout=None) -> bool:+ while select.select([wake], [], [], timeout)[0]:+ if ENDED in wake.recv(512):+ return True+ return False++ previous = signal.set_wakeup_fd(ring.fileno())+ try:+ threading.Thread(target=work, daemon=True).start()+ try:+ ended()+ except KeyboardInterrupt:+ if client:+ client.interrupt()+ ended(5)+ raise+ finally:+ signal.set_wakeup_fd(previous)+ wake.close()+ ring.close()+ if "error" in outcome:+ raise outcome["error"]+ return outcome["code"]++def main(argv=None) -> int:+ # A normal stop (kill, SIGTERM) ends like Ctrl-C, so half-written downloads are cleaned up on the way out.+ signal.signal(signal.SIGTERM, signal.default_int_handler)+ if hasattr(signal, "SIGBREAK"): # Ctrl-Break, the normal stop on Windows+ signal.signal(signal.SIGBREAK, signal.default_int_handler)+ p = parser()+ args = p.parse_args(argv)+ for key, value in (("json", False), ("server", None), ("token", None), ("admin_token", None), ("data", None)):+ if not hasattr(args, key):+ setattr(args, key, value)+ if not getattr(args, "func", None):+ p.print_help()+ return 1+ client = None+ try:+ if args.command in ("serve", "view"):+ return args.func(args, None) or 0 # a server handles its own signals, which needs the main thread+ client = None if args.command in ("admin", "login") else Client(args.server, args.token)+ code = run_interruptibly(args.func, args, client if args.json or client is None else Shown(client))+ if client:+ client.close() # confirms what a wait received, so it counts as read+ return code+ except ClientError as e:+ say(f"error: {e}", file=sys.stderr, lines=False)+ return EXIT.get(e.status, 1)+ except (FileNotFoundError, json.JSONDecodeError) as e:+ say(f"error: {e}", file=sys.stderr, lines=False)+ return 1+ except KeyboardInterrupt:+ return 130diff --git a/swarmchat/client.py b/swarmchat/client.pynew file mode 100644index 0000000..2637bf1--- /dev/null+++ b/swarmchat/client.py@@ -0,0 +1,505 @@+"""The client, usable from Python as well as from the CLI.++ from swarmchat.client import Client+ alice = Client("http://127.0.0.1:8484", token)+ alice.send(chat="bob", text="hi @bob")+ alice.wait(timeout=60, where="'urgent' in m.text")++Every server request is a method of the same name (`alice.chats(unread=True)`). read, chats and wait add what runs+on the caller's side: --where filters, and waits longer than the server holds one request.+"""+from __future__ import annotations++import ast+import http.client+import json+import math+import mimetypes+import os+import re+import select+import shutil+import socket+import ssl+import tempfile+import time+import types+import urllib.parse+import zipfile++from . import CONFIG_DIR, ENV+from .files import CHUNK, SPARE, pack, safe_name, unpack++HOLD = 280 # seconds per wait request. The server holds at most 300.+MAX_REPLY = 1 << 30 # the largest JSON reply taken from a server, in bytes+++class ClientError(Exception):+ """A refused or failed request. status is the HTTP status, or 0 when the server could not be reached."""++ def __init__(self, status: int, message: str):+ super().__init__(message)+ self.status = status+++def _read(path: str) -> str | None:+ try:+ with open(path, encoding="utf-8") as f:+ return f.read().strip() or None+ except FileNotFoundError:+ return None+++def read_saved_token() -> str | None:+ """The token `login` saved, if any."""+ return _read(os.path.join(CONFIG_DIR, "token"))+++def _replace(path: str, text: str):+ """Replaces a file with one that only its owner can read, written in full first, then put in place. A file there+ already, with looser permissions or as a link to somewhere else, is replaced, never written through."""+ fd, tmp = tempfile.mkstemp(dir=os.path.dirname(path), prefix=".swarmchat-") # made readable by its owner alone+ try:+ with os.fdopen(fd, "w", encoding="utf-8") as f:+ f.write(text)+ os.replace(tmp, path)+ except BaseException:+ os.unlink(tmp)+ raise+++def save_login(server: str, token: str):+ """Stores the server address and this user's token in the config folder, which only its owner may open."""+ os.makedirs(CONFIG_DIR, 0o700, exist_ok=True)+ if os.name != "nt":+ os.chmod(CONFIG_DIR, 0o700) # a folder made before, or by something else+ _replace(os.path.join(CONFIG_DIR, "config.json"), json.dumps({"server": server}))+ _replace(os.path.join(CONFIG_DIR, "token"), token + "\n")+++class Item(dict):+ """A message or chat for --where expressions: m.text, m.sender, c.unread. Missing fields are None."""++ def __getattr__(self, key):+ value = self.get(key)+ return Item(value) if isinstance(value, dict) else value+++# What an expression may use: the item, these functions, re's matching functions, and the attributes of plain data.+# Nothing else is reachable, so an expression a model wrote cannot import, open files or run commands, even where the+# client runs with more access than the model has.+SAFE = {f.__name__: f for f in (abs, all, any, bool, dict, enumerate, float, int, len, list, max, min, range, reversed,+ round, set, sorted, str, sum, tuple, zip)}+SAFE["re"] = types.SimpleNamespace(**{name: getattr(re, name) for name in (+ "search", "match", "fullmatch", "findall", "sub", "split", "escape", "IGNORECASE", "I", "MULTILINE", "M",+ "DOTALL", "S")})+PLAIN = (Item, dict, list, tuple, str, int, float, bool, set, frozenset, type(None), types.SimpleNamespace, re.Match,+ re.Pattern)+++def _attr(value, name: str):+ """value.name, for plain data only. Generators, functions, frames and types are not, so they lead nowhere."""+ if name.startswith("_") or name in ("format", "format_map", "mro") or type(value) not in PLAIN:+ raise AttributeError(f"{type(value).__name__}.{name} cannot be used in an expression")+ return getattr(value, name)+++class _Guard(ast.NodeTransformer):+ """Turns every `x.name` into `_attr(x, "name")`, and refuses names that start with _."""++ def visit_Name(self, node):+ if node.id.startswith("_"):+ raise ValueError(f"the name {node.id} cannot be used")+ return node++ def visit_Attribute(self, node):+ self.generic_visit(node)+ if node.attr.startswith("_") or not isinstance(node.ctx, ast.Load):+ raise ValueError(f"the attribute {node.attr} cannot be used")+ call = ast.Call(func=ast.Name(id="_attr", ctx=ast.Load()), args=[node.value, ast.Constant(node.attr)],+ keywords=[])+ return ast.copy_location(call, node)+++def expression(expr: str | None, flag: str):+ """The value of a Python expression over `m` (a message) or `c` (a chat), as a function of the item, or None when+ there is no expression. `flag` names the option in errors."""+ if not expr:+ return None+ try:+ tree = ast.fix_missing_locations(_Guard().visit(ast.parse(expr, flag, "eval")))+ code = compile(tree, flag, "eval")+ except SyntaxError as e:+ raise ClientError(400, f"{flag} is not a Python expression: {e.msg}") from None+ except ValueError as e:+ raise ClientError(400, f"{flag}: {e}. Expressions see m (or c), plain data and simple functions.") from None++ def value(item: dict):+ item = Item(item)+ item.setdefault("sender", item.get("from"))+ try: # m and c are globals, so generator expressions inside see them too+ return eval(code, {"__builtins__": {}, **SAFE, "_attr": _attr, "m": item, "c": item}) # noqa: S307+ except Exception as e: # noqa: BLE001+ raise ClientError(400, f"{flag} failed on {item.get('id')}: {type(e).__name__}: {e}") from None++ return value+++def matcher(expr: str | None):+ """A test from a --where expression, or None when there is none."""+ value = expression(expr, "--where")+ return None if value is None else lambda item: bool(value(item))+++def sorter(expr: str | None):+ """A function that orders items by a --sort expression, smallest first, or None when there is none."""+ value = expression(expr, "--sort")+ if value is None:+ return None++ def order(items: list) -> list:+ try:+ return sorted(items, key=value) # stable: equal values keep their order in time+ except TypeError as e: # e.g. None next to a number+ raise ClientError(400, f"--sort cannot order these values: {e}") from None++ return order+++class Client:+ def __init__(self, server: str | None = None, token: str | None = None):+ config = json.loads(_read(os.path.join(CONFIG_DIR, "config.json")) or "{}")+ self.server = server or os.environ.get(f"{ENV}_SERVER") or config.get("server") or "http://127.0.0.1:8484"+ self.token = token or os.environ.get(f"{ENV}_TOKEN") or _read(os.path.join(CONFIG_DIR, "token")) or ""+ url = urllib.parse.urlsplit(self.server if "://" in self.server else f"http://{self.server}")+ self.https = url.scheme == "https"+ self.host, self.port = url.hostname or "127.0.0.1", url.port or (443 if self.https else 80)+ self.conn = None+ self.ack = {} # chat -> newest seq a wait handed over and not yet confirmed to the server++ def _connect(self, timeout: float):+ if self.https:+ # SWARMCHAT_CA trusts a self-signed server certificate (a closed network has no public authority).+ context = ssl.create_default_context(cafile=os.environ.get(f"{ENV}_CA") or None)+ if os.environ.get(f"{ENV}_INSECURE"):+ context.check_hostname, context.verify_mode = False, ssl.CERT_NONE+ conn = http.client.HTTPSConnection(self.host, self.port, timeout=timeout, context=context)+ else:+ conn = http.client.HTTPConnection(self.host, self.port, timeout=timeout)+ conn.connect()+ conn.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)+ return conn++ def _closed_by_server(self) -> bool:+ """True when the kept-open connection can no longer be used: the server closed it (it timed out an idle+ connection, or restarted). Nothing is due on an idle connection, so a readable socket means it has ended.+ Checking first lets a request go out on a fresh connection instead of failing, without ever sending twice."""+ sock = self.conn.sock+ if sock is None:+ return True+ try:+ return bool(select.select([sock], [], [], 0)[0])+ except (OSError, ValueError):+ return True++ def _exchange(self, name: str, body, headers: dict, http_timeout: float, retry_safe: bool, read: bool = True,+ method: str = "POST"):+ """Sends one request on the kept-open connection, or a new one. (response, its body), the body left unread+ when read is False. Raises ClientError when the server cannot be reached."""+ for attempt in (1, 2):+ if self.conn is not None and self._closed_by_server():+ self._drop()+ reused = self.conn is not None+ try:+ if not reused:+ try:+ self.conn = self._connect(http_timeout)+ except OSError as e: # nothing was sent, so nothing can have been applied+ self._drop()+ raise ClientError(0, f"Cannot reach {self.server}: {e}. Is the server running?") from None+ self.conn.sock.settimeout(http_timeout)+ self.conn.request(method, f"/v1/{name}", body, headers)+ response = self.conn.getresponse()+ if not read:+ return response, None+ data = self._body(response)+ if response.will_close:+ self._drop()+ return response, data+ except (OSError, http.client.HTTPException) as e:+ self._drop()+ if not retry_safe or not reused or attempt == 2:+ detail = " The request may have been applied; it was not retried." if not retry_safe else ""+ raise ClientError(0, f"Cannot reach {self.server}: {e}.{detail}") from None++ def _body(self, response) -> bytes:+ """A reply's body, refused past MAX_REPLY bytes. Read by its length, or else a piece at a time: before 3.12,+ read(n) zero-fills n bytes first, which for n = MAX_REPLY took longer than the request itself."""+ if response.length is not None and response.length <= MAX_REPLY:+ return response.read()+ parts, size = [], 0+ while response.length is None:+ part = response.read(1 << 16)+ if not part:+ return b"".join(parts)+ parts.append(part)+ size += len(part)+ if size > MAX_REPLY:+ break+ self._drop()+ raise ClientError(0, f"{self.server} sent a reply larger than {MAX_REPLY} bytes.")++ def _result(self, response, data: bytes, acked: bool) -> dict:+ try:+ result = json.loads(data or b"{}")+ if not isinstance(result, dict):+ raise ValueError("expected a JSON object")+ except (ValueError, UnicodeError):+ self._drop()+ raise ClientError(response.status if response.status != 200 else 0,+ f"Invalid JSON response from {self.server} (HTTP {response.status}).") from None+ if acked and response.status != 401:+ self.ack = {} # the server applied them before anything else in this request+ if response.status != 200:+ raise ClientError(response.status, result.get("error") or f"HTTP {response.status}")+ return result++ def call(self, name: str, params: dict | None = None, http_timeout: float = 60) -> dict:+ """One request. Raises ClientError when the server refuses it or cannot be reached."""+ params = {**(params or {}), "ack": self.ack or None} # confirmations ride along with the next request+ body = json.dumps({k: v for k, v in params.items() if v not in (None, False, "", [])}).encode()+ headers = {"Content-Type": "application/json", "Authorization": f"Bearer {self.token}"}+ retry_safe = name in {"whoami", "people", "chats", "chat_info", "notifications", "settings_show", "policy_show"} \+ or (name in {"read", "wait"} and bool((params or {}).get("peek")))+ response, data = self._exchange(name, body, headers, http_timeout, retry_safe)+ return self._result(response, data, bool(params["ack"]))++ def send(self, chat: str, text: str = "", attach=(), reply_to: str | None = None) -> dict:+ """Sends a message, with files or folders attached if given. Their bytes are streamed to the server, so a+ large file never sits in memory. A folder is zipped on the way and unpacked again on download."""+ if not attach:+ return self.call("send", {"chat": chat, "text": text, "reply_to": reply_to})+ files, temps = [], []+ try:+ for path in attach:+ name = safe_name(os.path.basename(os.path.normpath(path)))+ if os.path.isdir(path):+ try:+ temps.append(pack(path))+ except ValueError as e:+ raise ClientError(400, f"Not sent: {e}.") from None+ files.append((temps[-1], {"name": name, "type": "application/zip", "folder": True}))+ elif os.path.isfile(path):+ files.append((path, {"name": name, "type": mimetypes.guess_type(name)[0] or ""}))+ else:+ raise ClientError(400, f"There is no file or folder at {path}.")+ for p, meta in files:+ meta["size"] = os.path.getsize(p)+ limits = self.limits()+ if len(files) > limits.get("max_attachments", len(files)):+ raise ClientError(413, f"This server takes at most {limits['max_attachments']} files on one message.")+ for _, meta in files:+ if meta["size"] > limits.get("max_attachment", meta["size"]):+ raise ClientError(413, f"{meta['name']} is {meta['size']} bytes, over this server's limit of "+ f"{limits['max_attachment']} bytes per file.")+ acked = bool(self.ack)+ header = (json.dumps({"chat": chat, "text": text, "reply_to": reply_to, "ack": self.ack or None,+ "files": [m for _, m in files]}) + "\n").encode()++ def body():+ yield header+ for p, _ in files:+ with open(p, "rb") as f:+ for chunk in iter(lambda: f.read(CHUNK), b""):+ yield chunk++ headers = {"Content-Type": "application/octet-stream", "Authorization": f"Bearer {self.token}",+ "Content-Length": str(len(header) + sum(m["size"] for _, m in files))}+ response, data = self._exchange("send_files", body(), headers, 60, retry_safe=False)+ return self._result(response, data, acked)+ finally:+ for t in temps:+ os.unlink(t)++ def limits(self) -> dict:+ """The server's limits on messages and attachments, so a file that is too large is refused before sending."""+ response, data = self._exchange("health", None, {}, 30, retry_safe=True, method="GET")+ return json.loads(data or b"{}").get("limits", {}) if response.status == 200 else {}++ def download(self, attachment_id: str, to: str = ".", force: bool = False) -> str:+ """Saves an attachment into the folder `to` under its own name, and returns that path. A folder attachment is+ unpacked into a folder of that name. Never replaces an existing file or folder unless force is set."""+ acked = bool(self.ack)+ body = json.dumps({"id": attachment_id, **({"ack": self.ack} if acked else {})}).encode()+ headers = {"Content-Type": "application/json", "Authorization": f"Bearer {self.token}"}+ response, _ = self._exchange("download", body, headers, 60, retry_safe=True, read=False)+ if response.status != 200:+ data = response.read()+ if response.will_close:+ self._drop()+ self._result(response, data, acked) # raises the server's error+ if acked:+ self.ack = {}+ try:+ info = json.loads(response.getheader("X-Attachment") or "{}")+ target = os.path.join(to, safe_name(info.get("name")))+ if os.path.lexists(target) and not force:+ raise ClientError(400, f"{target} already exists. Use --force to replace it.")+ os.makedirs(to, exist_ok=True)+ size = int(response.getheader("Content-Length") or 0)+ if size != info.get("size"):+ raise ClientError(0, f"{self.server} sent {size} bytes for a file of {info.get('size')}.")+ free = shutil.disk_usage(to).free+ if size > free - SPARE:+ raise ClientError(507, f"{info.get('name')} is {size:,} bytes, and this disk has {free:,} free.")+ stale = time.time() - 300 # a live download writes at least every minute, so older ones were killed+ for name in os.listdir(to):+ leftover = os.path.join(to, name)+ if name.startswith(".swarmchat-") and name.endswith((".part", ".unpack")) and \+ os.path.getmtime(leftover) < stale:+ shutil.rmtree(leftover) if os.path.isdir(leftover) else os.unlink(leftover)+ fd, tmp = tempfile.mkstemp(dir=to, prefix=".swarmchat-", suffix=".part")+ try:+ with os.fdopen(fd, "wb") as f:+ left = size+ while left:+ chunk = response.read(min(CHUNK, left))+ if not chunk:+ raise ClientError(0, "The download ended before all its bytes arrived.")+ f.write(chunk)+ left -= len(chunk)+ if os.path.lexists(target): # force+ shutil.rmtree(target) if os.path.isdir(target) and not os.path.islink(target) else os.unlink(target)+ if info.get("folder"):+ # Unpacked out of sight, then put in place whole: a folder that fails part way leaves nothing.+ staging = tempfile.mkdtemp(dir=to, prefix=".swarmchat-", suffix=".unpack")+ try:+ unpack(tmp, staging)+ os.replace(staging, target)+ except (ValueError, OSError, zipfile.BadZipFile, NotImplementedError, RuntimeError) as e:+ raise ClientError(400, f"Not unpacked: {e}") from None+ finally:+ if os.path.exists(staging):+ shutil.rmtree(staging)+ else:+ os.replace(tmp, target)+ finally:+ if os.path.exists(tmp):+ os.unlink(tmp)+ except BaseException:+ self._drop() # the rest of the reply was not read, so this connection cannot be reused+ raise+ if response.will_close:+ self._drop()+ return target++ def interrupt(self):+ """Breaks off what this client is waiting for, from another thread: a read blocked on the connection fails+ at once. Shutting the socket down wakes such a read on every platform, where closing it does not."""+ sock = self.conn.sock if self.conn is not None else None+ if sock is not None:+ try:+ sock.shutdown(socket.SHUT_RDWR)+ except OSError:+ pass++ def _drop(self):+ """Closes the connection, nothing else. Used inside call(), so it must never make a request."""+ if self.conn is not None:+ self.conn.close()+ self.conn = None++ def close(self):+ """Confirms any messages a wait handed over, then closes the connection."""+ if self.ack:+ try:+ self.call("mark_read")+ except ClientError:+ pass # not confirmed: those messages stay unread and come again+ self._drop()++ def __getattr__(self, name: str):+ if name.startswith("_"):+ raise AttributeError(name)+ return lambda **params: self.call(name, params)++ def mark_messages_read(self, messages: list, chat: str | None = None):+ """Marks these messages read, and every unread one before them in their chats, up to the first one that is+ not among them: messages a filter left out stay unread."""+ shown = {}+ for m in messages:+ shown.setdefault(m["chat"], []).append(m["seq"])+ if shown:+ return self.call("mark_read", {"shown": shown, "chat": chat})++ def read(self, where: str | None = None, peek: bool = False, limit: int | None = None, sort: str | None = None,+ **select) -> dict:+ """Messages are marked read only after they have arrived here, so a read cut off on the way (a killed+ command, a lost connection) never uses up messages: they stay unread and come again.++ `where` and `sort` run here. The other options choose the messages, `sort` orders them, and `limit` keeps+ that many from the top: sort="-len(m.reactions or {})", limit=10 is the ten with the most reactions.+ The server gives a page at a time, so this asks for more pages while it needs them: until it has enough+ matches, or every candidate when they are to be sorted."""+ test, order, last = matcher(where), sorter(sort), select.get("last")+ if test is not None and last: # with a filter, "the last N" means the last N that match+ select["last"], select["all"] = None, True+ whole = test is not None or (order is not None and not last) # every candidate comes, to filter or order+ params = {**select, "peek": True, "limit": 10 ** 9 if whole else None if order else limit}+ result = self.call("read", params)+ found, keep, titles, thread = [], result["keep"], {}, bool(select.get("thread"))+ # How many messages it takes, or None for every one: to sort them, or a whole thread.+ if test is not None: # the last N that match, or enough matches to show+ enough = last or (None if order is not None else limit or (None if thread else 50 if keep == "first" else 20))+ elif order is not None: # the last N to sort, or every candidate+ enough = last+ else:+ enough = limit or last+ while True:+ page = [m for m in result["messages"] if test is None or test(m)]+ found = found + page if keep == "first" else page + found # history pages go back in time+ titles.update(result.get("chats") or {})+ if not result.get("more") or (len(found) >= enough if enough is not None else not (whole or thread)):+ break+ result = self.call("read", {**params, "cursor": result["cursor"]})+ if enough is not None and len(found) > enough:+ found = found[:enough] if keep == "first" else found[-enough:]+ if order is not None:+ found, keep = order(found), "first"+ if whole or order is not None:+ n = limit or (len(found) if last or select.get("thread") else 50 if keep == "first" else 20)+ found = found[:n] if keep == "first" else found[-n:]+ if not peek and found and not select.get("thread"): # a thread moves no read position, see api.read+ scope = found[0]["chat"] if select.get("id") else select.get("chat")+ marked = self.mark_messages_read(found, scope)+ result["unread_left"] = marked.get("unread_left", result["unread_left"])+ return {**result, "messages": found, "chats": titles}++ def chats(self, where: str | None = None, limit: int | None = None, **select) -> dict:+ test = matcher(where)+ if test is None:+ return self.call("chats", {**select, "limit": limit})+ found = [c for c in self.call("chats", {**select, "limit": 10 ** 9})["chats"] if test(c)]+ return {"chats": found[:limit or 20]}++ def wait(self, timeout: float | None = None, where: str | None = None, peek: bool = False, **select) -> dict:+ """Blocks until a matching message arrives, for at most `timeout` seconds (None: forever). On a timeout the+ reply's messages list is empty. Messages are marked read only once they have arrived here: the client+ confirms them with its next request, or when it closes. If it never does, they stay unread and come again."""+ if timeout is not None and (not math.isfinite(timeout) or timeout < 0):+ raise ClientError(400, "timeout must be a finite number of seconds at least 0.")+ test, after = matcher(where), None+ deadline = None if timeout is None else time.monotonic() + timeout+ while True:+ hold = HOLD if deadline is None else max(0.0, min(HOLD, deadline - time.monotonic()))+ result = self.call("wait", {**select, "timeout": hold or 0.001, "after": after, "peek": True},+ http_timeout=hold + 30)+ after = {**(after or {}), **result["positions"]}+ found = [m for m in result["messages"] if test is None or test(m)]+ if found:+ if not peek: # confirmed with the next request (or close), so a wait cut off loses nothing+ for m in found:+ self.ack[m["chat"]] = max(self.ack.get(m["chat"], 0), m["seq"])+ return {**result, "messages": found}+ if deadline is not None and time.monotonic() >= deadline:+ return {**result, "messages": []}diff --git a/swarmchat/config.py b/swarmchat/config.pynew file mode 100644index 0000000..b58e9fe--- /dev/null+++ b/swarmchat/config.py@@ -0,0 +1,63 @@+"""Server settings. The defaults below, then a JSON config file (`swarmchat serve --config FILE`), then command-line+flags, each overriding the one before. Every setting is also a flag: `max_wait` is `--max-wait`.+"""+from __future__ import annotations++import json+import math++from . import DATA_DIR++DEFAULTS = {+ "data": DATA_DIR, # the folder that holds everything+ "host": "127.0.0.1", # 0.0.0.0 accepts other machines+ "port": 8484,+ "tls_cert": None, # serve HTTPS with this certificate and key+ "tls_key": None,+ "fsync": False, # force every write to disk: survives a power cut, slower+ "verbose": False, # log every request+ "max_text": 20_000, # longest message, in characters+ "max_wait": 300, # longest one wait request is held, in seconds. Clients simply ask again.+ "max_body": 1 << 20, # largest request, in bytes (a message's files have their own limits below)+ "max_attachment": 100_000_000, # largest attached file or zipped folder, in bytes+ "max_attachments": 20, # most files on one message+ "request_timeout": 30.0, # idle socket and TLS handshake timeout, in seconds. A request's line and headers must+ # all arrive within it, and its body within it plus the body's size at min_rate.+ "min_rate": 16_384, # slowest a request's body may arrive, in bytes a second+ "max_connections": 4096, # most connections open at once. Each waiting agent holds one. Lowered to fit the+ # number of files the system lets the server open.+ "max_requests": 32, # most requests one user may have under way at once, waits and uploads included+ "max_page": 1000, # most messages one read returns. The client asks again for the rest.+ "min_free": 1_000_000_000, # bytes of disk the server keeps free: messages and files that would use them are+ # refused. 0 turns this off.+ "quota": 0, # most bytes of files one user may have stored at once. 0 is no limit.+ "default_notify": "all", # alert level for chats nobody has set: all, mentions or none+ "save_every": 1.0, # how often read positions are saved, in seconds+ "segment": 10_000, # lines per log file. Fixed when the data folder is made: later changes are ignored.+}+++def load(path: str | None = None, **flags) -> dict:+ """The settings to run with. Flags that are None were not given, so they do not override anything."""+ settings = dict(DEFAULTS)+ if path:+ with open(path, encoding="utf-8") as f:+ given = json.load(f)+ unknown = sorted(set(given) - set(DEFAULTS))+ if unknown:+ raise SystemExit(f"{path}: unknown settings {', '.join(unknown)}. Known: {', '.join(DEFAULTS)}.")+ settings.update(given)+ settings.update({k: v for k, v in flags.items() if v is not None})+ if not settings["data"]:+ raise SystemExit("Say where the data lives: --data DIR, or \"data\" in the config file.")+ if settings["default_notify"] not in ("all", "mentions", "none"):+ raise SystemExit("default_notify is one of all, mentions, none.")+ for key in ("port", "max_text", "max_body", "max_attachment", "max_attachments", "segment", "max_connections",+ "max_requests", "max_page", "min_free", "quota"):+ least = 0 if key in ("port", "min_free", "quota") else 1+ if not isinstance(settings[key], int) or isinstance(settings[key], bool) or settings[key] < least:+ raise SystemExit(f"{key} must be a whole number" + (" of at least 1." if least else " of at least 0."))+ for key in ("max_wait", "save_every", "request_timeout", "min_rate"):+ if not isinstance(settings[key], (int, float)) or not math.isfinite(settings[key]) or settings[key] <= 0:+ raise SystemExit(f"{key} must be a number above 0.")+ return settingsdiff --git a/swarmchat/files.py b/swarmchat/files.pynew file mode 100644index 0000000..5e7abb4--- /dev/null+++ b/swarmchat/files.py@@ -0,0 +1,232 @@+"""Attachments.++Every file is stored once, named by the SHA-256 of its bytes: DIR/blobs/3f/3fa9.... Messages refer to their files by+a random attachment id, never by the hash. A download is allowed only to members of the message's chat, and nobody+can ask the server whether some file exists. A blob is removed when no message refers to it any more.++A folder travels as one zip and is unpacked on download. Unpacking refuses any entry that would land outside the+target folder, more than MAX_ENTRIES entries, and more than the disk can take while keeping SPARE bytes free: each entry+counted at the whole disk blocks it fills, and at one of the disk's free file entries (inodes).+"""+from __future__ import annotations++import collections+import hashlib+import os+import re+import shutil+import stat+import sys+import tempfile+import threading+import time+import zipfile++CHUNK = 1 << 20+MAX_ENTRIES = 100_000 # files and folders in one folder attachment+SPARE = 64 << 20 # bytes of disk an unpacked folder or a download must leave free+MIME = re.compile(r"[A-Za-z0-9][\w!#$&^.+-]*/[A-Za-z0-9][\w!#$&^.+-]*", re.ASCII)+++def mime(kind) -> str | None:+ """A media type such as text/plain, or None for anything else: it goes into a reply's Content-Type header, where+ a line break or other control character would end the header early."""+ return kind if isinstance(kind, str) and len(kind) <= 100 and MIME.fullmatch(kind) else None+++def safe_name(name: str) -> str:+ """A file name that cannot point anywhere else: no folders, no '..', no control characters."""+ name = os.path.basename(str(name).replace("\\", "/")).strip()+ name = "".join(c for c in name if c.isprintable())[:200]+ return name if name not in ("", ".", "..") else "file"+++class Blobs:+ """The files on disk, and how many messages refer to each."""++ def __init__(self, root: str, read_only: bool = False):+ self.dir, self.tmp = os.path.join(root, "blobs"), os.path.join(root, "uploads")+ if not read_only:+ os.makedirs(self.dir, 0o700, exist_ok=True)+ os.makedirs(self.tmp, 0o700, exist_ok=True)+ for name in os.listdir(self.tmp): # uploads cut off by a crash+ os.unlink(os.path.join(self.tmp, name))+ self.refs = collections.Counter()+ self.lock = threading.Lock()++ def path(self, sha: str) -> str:+ return os.path.join(self.dir, sha[:2], sha)++ def receive(self, read, size: int) -> tuple:+ """Reads exactly `size` bytes with read(n) into a temporary file, hashing as it goes. (temp path, sha256).+ If the bytes stop early, raises EOFError and leaves nothing behind."""+ fd, tmp = tempfile.mkstemp(dir=self.tmp, suffix=".part")+ digest, left = hashlib.sha256(), size+ try:+ with os.fdopen(fd, "wb") as f:+ while left:+ chunk = read(min(CHUNK, left))+ if not chunk:+ raise EOFError("the upload ended before all its bytes arrived")+ f.write(chunk)+ digest.update(chunk)+ left -= len(chunk)+ except BaseException:+ os.unlink(tmp)+ raise+ return tmp, digest.hexdigest()++ def keep(self, tmp: str, sha: str):+ """Makes a received file the blob for its hash, or drops it if that blob is already stored, and counts one+ more reference. Counting under the same lock as release() means a blob cannot vanish between the two."""+ with self.lock:+ dest = self.path(sha)+ if os.path.exists(dest):+ os.unlink(tmp)+ else:+ os.makedirs(os.path.dirname(dest), 0o700, exist_ok=True)+ os.replace(tmp, dest)+ self.refs[sha] += 1++ def release(self, sha: str):+ """One reference fewer. The last one removes the file."""+ with self.lock:+ self.refs[sha] -= 1+ if self.refs[sha] <= 0:+ del self.refs[sha]+ try:+ os.unlink(self.path(sha))+ except FileNotFoundError:+ pass++ def prune(self):+ """At start: removes blobs no message refers to, left by a crash between storing a file and its message."""+ for sub in os.listdir(self.dir):+ for sha in os.listdir(os.path.join(self.dir, sub)):+ if not self.refs[sha]:+ os.unlink(os.path.join(self.dir, sub, sha))+++def opened_path(fd: int) -> str | None:+ """Where the file open on `fd` is, from the system: what was opened, whatever its path has become since. None+ where the system cannot say."""+ try:+ if sys.platform.startswith("linux"):+ return os.readlink(f"/proc/self/fd/{fd}")+ if sys.platform == "darwin":+ import fcntl+ return os.fsdecode(fcntl.fcntl(fd, fcntl.F_GETPATH, bytes(1024)).split(b"\0", 1)[0])+ if os.name == "nt":+ import ctypes+ import msvcrt+ from ctypes import wintypes+ final = ctypes.windll.kernel32.GetFinalPathNameByHandleW+ final.argtypes = [wintypes.HANDLE, wintypes.LPWSTR, wintypes.DWORD, wintypes.DWORD]+ final.restype = wintypes.DWORD+ buffer = ctypes.create_unicode_buffer(32768)+ n = final(msvcrt.get_osfhandle(fd), buffer, len(buffer), 0)+ path = buffer.value if 0 < n < len(buffer) else None+ return path[4:] if path and path.startswith("\\\\?\\") and path[5:7] == ":\\" else path+ except (OSError, ValueError, AttributeError, ImportError):+ pass+ except Exception: # noqa: BLE001 - ctypes.ArgumentError: the system could not say+ pass+ return None+++def pack(folder: str) -> str:+ """Zips a folder, with relative paths and empty folders, into a temporary file. The caller removes it.+ A link to something inside the folder is sent as what it points to. A link to anything outside it is refused,+ so a folder cannot carry files from elsewhere on the sender's disk. A link back to a folder above it on the same+ path is skipped, so a loop of links cannot go on forever. Entries carry the files' own times, so the same folder+ always zips to the same bytes and is stored once however often it is sent.++ Each file is opened once, then checked where the system says the opened file is, and its bytes come from that same+ open file. A link switched to a file outside the folder between a check and a read cannot slip one in."""+ base = where = os.path.realpath(folder)+ try: # the folder as the system names open files, which may differ from realpath in case+ fd = os.open(base, os.O_RDONLY)+ try:+ where = opened_path(fd) or base+ finally:+ os.close(fd)+ except OSError: # Windows cannot open a folder this way: it compares without case anyway+ pass++ def inside(path: str, real: str | None = None) -> str:+ real, root = (real, where) if real else (os.path.realpath(path), base)+ top = os.path.normcase(root.rstrip(os.sep) + os.sep)+ if os.path.normcase(real) != os.path.normcase(root) and not os.path.normcase(real).startswith(top):+ raise ValueError(f"{path} links to {real}, outside the folder. Copy it in, or remove the link")+ return real++ def add(z: zipfile.ZipFile, full: str):+ try: # not blocking, so a file switched to a pipe cannot hang the send+ fd = os.open(full, os.O_RDONLY | getattr(os, "O_NONBLOCK", 0) | getattr(os, "O_BINARY", 0))+ except FileNotFoundError: # a broken link, or gone since the folder was listed: nothing to send+ return+ with os.fdopen(fd, "rb") as f:+ st = os.fstat(fd)+ if not stat.S_ISREG(st.st_mode):+ return+ where = opened_path(fd)+ if where is None: # no way to ask the system: the open file must be the one the checked path names+ real = inside(full)+ same = os.stat(real)+ if (same.st_dev, same.st_ino) != (st.st_dev, st.st_ino):+ raise ValueError(f"{full} changed while the folder was being sent")+ else:+ inside(full, where)+ info = zipfile.ZipInfo(os.path.relpath(full, folder).replace(os.sep, "/"), time.localtime(st.st_mtime)[:6])+ info.external_attr, info.file_size = (st.st_mode & 0xFFFF) << 16, st.st_size # as ZipFile.write would+ info.compress_type = zipfile.ZIP_DEFLATED+ with z.open(info, "w") as out:+ shutil.copyfileobj(f, out, 1024 * 8)++ fd, tmp = tempfile.mkstemp(suffix=".zip")+ os.close(fd)+ above = {} # folder as walked -> real paths of it and every folder above it+ try:+ with zipfile.ZipFile(tmp, "w", zipfile.ZIP_DEFLATED) as z:+ for root, dirs, files in os.walk(folder, followlinks=True):+ chain = above[root] = above.get(os.path.dirname(root), frozenset()) | {os.path.realpath(root)}+ dirs[:] = sorted(d for d in dirs if inside(os.path.join(root, d)) not in chain)+ rel = os.path.relpath(root, folder)+ if rel != "." and not dirs and not files:+ stamp = time.localtime(max(os.path.getmtime(root), 315532800))[:6] # zip times start in 1980+ z.writestr(zipfile.ZipInfo(rel.replace(os.sep, "/") + "/", stamp), "")+ for name in sorted(files):+ add(z, os.path.join(root, name))+ except BaseException:+ os.unlink(tmp)+ raise+ return tmp+++def unpack(zip_path: str, dest: str):+ """Unzips into dest, refusing absolute paths, '..' and anything else that would land outside it, and a folder of+ more entries or bytes than the limits above. The sizes it declares are the most each entry can write: zipfile+ stops there."""+ base = os.path.realpath(dest)+ with zipfile.ZipFile(zip_path) as z:+ infos = z.infolist()+ if len(infos) > MAX_ENTRIES:+ raise ValueError(f"the folder holds {len(infos):,} entries, more than {MAX_ENTRIES:,}")+ block, inodes = 4096, None+ if hasattr(os, "statvfs"):+ disk = os.statvfs(base)+ block, inodes = disk.f_frsize or disk.f_bsize or 4096, disk.f_favail if disk.f_files else None+ # A one-byte file still fills a whole block, and its entry in its folder takes room too: a block each.+ total = sum(-(-info.file_size // block) * block + block for info in infos)+ free = shutil.disk_usage(base).free+ if total > free - SPARE:+ raise ValueError(f"the folder takes {total:,} bytes on disk, and this disk has {free:,} free")+ if inodes is not None and len(infos) > inodes - 1000:+ raise ValueError(f"the folder holds {len(infos):,} entries, and this disk has room for {inodes:,} more")+ for info in infos:+ parts = info.filename.replace("\\", "/").split("/")+ target = os.path.realpath(os.path.join(base, *parts))+ if info.filename.startswith(("/", "\\")) or ".." in parts or ":" in parts[0] or \+ (target != base and not target.startswith(base + os.sep)):+ raise ValueError(f"the folder holds an unsafe path: {info.filename}")+ z.extractall(base)diff --git a/swarmchat/locking.py b/swarmchat/locking.pynew file mode 100644index 0000000..e60d1fb--- /dev/null+++ b/swarmchat/locking.py@@ -0,0 +1,36 @@+"""An OS-held, nonblocking lease on a server's data directory.++The lock file stays in place; the operating system releases ownership on exit,+including SIGKILL. Never unlink a live lock file: another process could lock a+different inode at the same path.+"""+import os+import sys+++class DirectoryLock:+ def __init__(self, root: str):+ os.makedirs(root, 0o700, exist_ok=True)+ try: # the folder holds every message and the tokens: its owner alone may look in, whatever was there before+ os.chmod(root, 0o700)+ except OSError as exc:+ print(f"warning: could not make {root} private to its owner: {exc}", file=sys.stderr)+ self.handle = os.fdopen(os.open(os.path.join(root, ".server.lock"), os.O_RDWR | os.O_CREAT, 0o600), "r+b", buffering=0)+ try:+ if os.name == "nt":+ import msvcrt+ if not os.fstat(self.handle.fileno()).st_size:+ self.handle.write(b"\0")+ self.handle.seek(0)+ msvcrt.locking(self.handle.fileno(), msvcrt.LK_NBLCK, 1)+ else:+ import fcntl+ fcntl.flock(self.handle.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB)+ except OSError as exc:+ self.close()+ raise OSError(f"Cannot lock {root}; another server may already own this data directory.") from exc++ def close(self):+ if self.handle is not None:+ self.handle.close()+ self.handle = Nonediff --git a/swarmchat/page/app.css b/swarmchat/page/app.cssnew file mode 100644index 0000000..ceb61c0--- /dev/null+++ b/swarmchat/page/app.css@@ -0,0 +1,448 @@+/* Every color has a light and a dark value: light-dark() picks by color-scheme, which follows the system, or the+ theme picked in the top bar (data-theme on <html>). */+:root {+ color-scheme: light dark;+ --bg: light-dark(#ffffff, #0b0e14); --side: light-dark(#f7f7f9, #0f131b); --panel: light-dark(#ffffff, #141923);+ --panel2: light-dark(#f1f2f5, #1a2030); --hover: light-dark(#eef0f3, #1d2433);+ --line: light-dark(#e5e7eb, #222a38); --line2: light-dark(#d5d9e0, #2e3748);+ --text: light-dark(#1d1c1d, #e6e9f0); --text2: light-dark(#1d1c1d, #dde2ec); --text3: light-dark(#3c3f45, #c3c9d6);+ --text4: light-dark(#3c3f45, #cdd3df); --muted: light-dark(#616670, #8f98ab); --faint: light-dark(#8a909c, #5d677b);+ --accent: light-dark(#6d5bd0, #8b7cf6); --accent-soft: light-dark(#5b49c9, #c4b5fd); --accent-ink: light-dark(#4636a8, #d9d4ff);+ --dm: light-dark(#2563eb, #3b82f6); --channel: light-dark(#ea580c, #f97316); --group: light-dark(#0d9488, #14b8a6);+ --board: light-dark(#9333ea, #a855f7);+ --dm-soft: light-dark(#1d4ed8, #93c5fd); --channel-soft: light-dark(#c2410c, #fdba74); --group-soft: light-dark(#0f766e, #5eead4);+ --good: light-dark(#16a34a, #22c55e); --warn: light-dark(#e8a317, #facc15); --warn-ink: light-dark(#a16207, #fcd34d);+ --bad: light-dark(#e11d48, #f43f5e);+ --mention: light-dark(rgba(236, 178, 46, .14), rgba(250, 204, 21, .13));+ --at: light-dark(#8a5300, #fcd34d); --at-bg: light-dark(rgba(236, 178, 46, .2), rgba(250, 204, 21, .1));+ --link: light-dark(#1264a3, #93b4ff); --code-bg: light-dark(#f6f7f9, #0a0d13);+ --msg-hover: light-dark(#f8f8f9, rgba(255, 255, 255, .025)); --post-hover: light-dark(#fafafb, #171d29);+ --seg-track: light-dark(#eceef2, #141923); --seg-on: light-dark(#ffffff, #1a2030);+ --shadow: light-dark(rgba(16, 24, 40, .14), rgba(0, 0, 0, .45)); --shade: light-dark(rgba(16, 24, 40, .32), rgba(4, 6, 10, .55));+ --scroll: light-dark(#c9ced6, #283043); --legend-bg: light-dark(rgba(255, 255, 255, .92), rgba(15, 19, 27, .86));+ --tip-bg: light-dark(#1d1c1d, #0a0d13); --tip-text: light-dark(#ffffff, #e6e9f0);+ --edge: light-dark(#64748b, #cbd5e1); --edge-dm: light-dark(#2563eb, #93b4ff);+ /* code: keywords, strings, numbers and constants, comments, functions, tags and types, keys, variables */+ --ck: light-dark(#cf222e, #ff7b72); --cs: light-dark(#0a3069, #a5d6ff); --cn: light-dark(#0550ae, #79c0ff);+ --cc: light-dark(#6e7781, #8b949e); --cf: light-dark(#8250df, #d2a8ff); --ct: light-dark(#116329, #7ee787);+ --ca: light-dark(#0550ae, #79c0ff); --cv: light-dark(#953800, #ffa657);+ --diff-add: light-dark(#e6ffec, rgba(46, 160, 67, .16)); --diff-del: light-dark(#ffebe9, rgba(248, 81, 73, .16));+ --diff-hunk: light-dark(#ddf4ff, rgba(56, 139, 253, .14));+ --log-err: light-dark(#fff1f0, rgba(248, 81, 73, .11)); --log-warn: light-dark(#fff8c5, rgba(210, 153, 34, .13));+ --sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Inter, Roboto, "Helvetica Neue", Arial, sans-serif;+ --mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;+}+:root[data-theme="light"] { color-scheme: light; }+:root[data-theme="dark"] { color-scheme: dark; }+* { box-sizing: border-box; }+html, body { height: 100%; margin: 0; }+body { background: var(--bg); color: var(--text); font: 14px/1.45 var(--sans); overflow: hidden; -webkit-font-smoothing: antialiased; }+button { font: inherit; color: inherit; background: none; border: 0; cursor: pointer; padding: 0; }+input { font: inherit; color: inherit; }+a { color: var(--link); text-decoration: none; }+a:hover { text-decoration: underline; }+code, pre, kbd { font-family: var(--mono); font-size: 12.5px; }+kbd { border: 1px solid var(--line2); border-bottom-width: 2px; border-radius: 4px; padding: 0 4px; color: var(--muted); margin-left: 6px; }+[hidden] { display: none !important; }+::-webkit-scrollbar { width: 10px; height: 10px; }+::-webkit-scrollbar-thumb { background: var(--scroll); border-radius: 6px; border: 2px solid transparent; background-clip: padding-box; }+::-webkit-scrollbar-track { background: transparent; }++#app { display: grid; height: 100vh; grid-template-rows: 48px minmax(0, 1fr); grid-template-columns: 272px minmax(0, 1fr) auto;+ grid-template-areas: "top top top" "side main pane"; }+body.layout-map #app { grid-template-columns: 0 minmax(0, 1fr) auto; }+body.layout-map #side { display: none; }++/* top bar */+#top { grid-area: top; display: flex; align-items: center; gap: 14px; padding: 0 14px; border-bottom: 1px solid var(--line); background: var(--side); min-width: 0; }+.brand { font-weight: 800; font-size: 19px; letter-spacing: -.5px; color: var(--text); background: linear-gradient(90deg, var(--dm), var(--board) 55%, var(--channel));+ -webkit-background-clip: text; background-clip: text; -webkit-text-fill-color: transparent; text-decoration: none !important; }+.brand span { font-weight: 500; margin-left: 5px; -webkit-text-fill-color: var(--muted); font-size: 15px; letter-spacing: 0; }+.brand.big { font-size: 34px; }+#stats { color: var(--muted); font-size: 12.5px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; min-width: 0; }+#stats b { color: var(--text); font-weight: 600; }+.grow { flex: 1; }+.seg { display: flex; background: var(--seg-track); border: 1px solid var(--line); border-radius: 8px; padding: 2px; }+.seg button { padding: 4px 12px; border-radius: 6px; color: var(--muted); font-weight: 500; }+.seg button:hover { color: var(--text); }+.seg button.on { background: var(--seg-on); color: var(--text); box-shadow: 0 1px 2px var(--shadow), 0 0 0 1px var(--line2); }+.as, .find, .theme { display: flex; align-items: center; gap: 8px; height: 30px; padding: 0 10px; border: 1px solid var(--line); border-radius: 8px; background: var(--panel); color: var(--muted); white-space: nowrap; }+.as:hover, .find:hover, .theme:hover { border-color: var(--line2); color: var(--text); }+.theme { width: 30px; padding: 0; justify-content: center; }+.theme svg { width: 16px; height: 16px; }+.as.on { border-color: color-mix(in srgb, var(--accent) 60%, transparent); color: var(--text); background: color-mix(in srgb, var(--accent) 14%, var(--panel)); }+.as .x { color: var(--muted); padding: 0 2px; margin-right: -4px; }+.as .x:hover { color: var(--text); }+.live { display: flex; align-items: center; gap: 6px; font-size: 12px; color: var(--muted); white-space: nowrap; }+.live::before { content: ""; width: 8px; height: 8px; border-radius: 50%; background: var(--faint); }+.live.on::before { background: var(--good); box-shadow: 0 0 0 3px rgba(34,197,94,.18); animation: beat 2s infinite; }+.live.ro::before { background: var(--warn); }+.live.off::before { background: var(--bad); }+@keyframes beat { 50% { box-shadow: 0 0 0 5px rgba(34,197,94,.05); } }++/* sidebar */+#side { grid-area: side; display: flex; flex-direction: column; background: var(--side); border-right: 1px solid var(--line); min-height: 0; }+.tabs { display: flex; gap: 4px; padding: 10px 10px 6px; }+.tabs button { flex: 1; padding: 5px 0; border-radius: 6px; color: var(--muted); font-weight: 600; font-size: 13px; }+.tabs button.on { background: var(--seg-on); color: var(--text); box-shadow: 0 0 0 1px var(--line); }+#filter { margin: 2px 10px 8px; padding: 6px 10px; border-radius: 7px; border: 1px solid var(--line); background: var(--bg); outline: none; font-size: 13px; }+#filter:focus { border-color: var(--line2); }+.vlist { position: relative; overflow-y: auto; flex: 1; min-height: 0; contain: strict; }+.vspace { width: 1px; }+.vrows { position: absolute; left: 8px; right: 8px; top: 0; }+.row { height: 30px; display: flex; align-items: center; gap: 8px; padding: 0 8px 0 10px; border-radius: 6px; cursor: pointer; color: var(--text3); white-space: nowrap; overflow: hidden; }+.row:hover { background: var(--hover); }+.row.on { background: color-mix(in srgb, var(--accent) 22%, transparent); color: var(--text); }+.row .t { flex: 1; overflow: hidden; text-overflow: ellipsis; }+.row .n { font-size: 11.5px; color: var(--faint); font-variant-numeric: tabular-nums; }+.row.unread .t { color: var(--text); font-weight: 650; }+.row.head { cursor: default; color: var(--muted); font-size: 11.5px; font-weight: 700; letter-spacing: .6px; text-transform: uppercase; padding-top: 8px; }+.row.head:hover { background: none; }+.row.special .t { font-weight: 600; color: var(--text); }+.ico { width: 18px; height: 18px; flex: none; display: inline-grid; vertical-align: -3px; place-items: center; border-radius: 5px; font-size: 12px; font-weight: 800; color: var(--faint); }+.ico.channel { color: var(--channel); }+.ico.group { color: var(--group); }+.ico.direct { color: var(--dm); }+.badge { min-width: 18px; height: 18px; border-radius: 9px; padding: 0 6px; display: inline-grid; place-items: center; font-size: 11px; font-weight: 700; background: var(--accent); color: #fff; }+.badge.at { background: var(--bad); }+.badge.mute { background: var(--panel2); color: var(--muted); }+.pill { font-size: 11px; padding: 1px 7px; border-radius: 10px; border: 1px solid var(--line2); color: var(--muted); white-space: nowrap; font-weight: 500; }+.pill.public { color: var(--channel-soft); border-color: color-mix(in srgb, var(--channel) 40%, transparent); }+.pill.private { color: var(--muted); }+.pill.archived { color: var(--muted); border-style: dashed; }+.pill.direct { color: var(--dm-soft); border-color: color-mix(in srgb, var(--dm) 40%, transparent); }+.pill.group { color: var(--group-soft); border-color: color-mix(in srgb, var(--group) 40%, transparent); }+.dot { width: 8px; height: 8px; border-radius: 50%; flex: none; }++/* avatars */+.av { width: 36px; height: 36px; flex: none; border-radius: 7px; display: grid; place-items: center; font-weight: 700; font-size: 13px; color: #0b0e14; background: var(--c, #64748b); }+.av.sm { width: 20px; height: 20px; border-radius: 5px; font-size: 9.5px; }+.av.lg { width: 64px; height: 64px; border-radius: 16px; font-size: 24px; }++/* main */+#main { grid-area: main; display: flex; flex-direction: column; min-width: 0; min-height: 0; position: relative; }+.mhead { display: flex; align-items: center; gap: 10px; min-height: 56px; padding: 10px 20px; border-bottom: 1px solid var(--line); flex-wrap: wrap; }+.mhead h1 { font-size: 17px; margin: 0; font-weight: 800; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: 50%; }+.mhead .sub { color: var(--muted); font-size: 12.5px; }+.mhead .grow { flex: 1; }+.mhead .about { flex-basis: 100%; color: var(--muted); font-size: 12.5px; margin-top: -4px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }+.btn { height: 28px; padding: 0 10px; border-radius: 7px; border: 1px solid var(--line2); color: var(--text4); font-size: 12.5px; display: inline-flex; align-items: center; gap: 6px; background: var(--panel); white-space: nowrap; }+.btn:hover { background: var(--panel2); color: var(--text); }+.btn.primary { background: color-mix(in srgb, var(--accent) 28%, var(--panel)); border-color: color-mix(in srgb, var(--accent) 60%, transparent); color: var(--text); }+.link { color: var(--link); cursor: pointer; }+.link:hover { text-decoration: underline; }+.banner { margin: 10px 20px 0; padding: 8px 12px; border-radius: 8px; background: color-mix(in srgb, var(--accent) 12%, transparent); border: 1px solid color-mix(in srgb, var(--accent) 35%, transparent); color: var(--accent-ink); font-size: 13px; }+.empty { color: var(--muted); padding: 40px; text-align: center; }+.scroll { flex: 1; overflow-y: auto; min-height: 0; overflow-anchor: none; }+.loading { color: var(--faint); text-align: center; padding: 14px; font-size: 12.5px; }++/* messages */+/* Slack's rhythm: a sender's group starts with room above it, its lines sit close together, and text is 15px on 22px+ lines, beside a 36px avatar. */+.msgs { padding: 10px 0 18px; }+.msg { display: flex; gap: 8px; padding: 9px 20px 3px; position: relative; }+.msg:hover { background: var(--msg-hover); }+.msg.cont { padding-top: 1px; }+.msg.cont .av { visibility: hidden; height: 0; }+.msg.cont .head { display: none; }+.msg.flash { animation: flash 2.2s ease-out; }+@keyframes flash { 0%, 30% { background: color-mix(in srgb, var(--warn) 18%, transparent); } }+.msg.mentions-me { background: var(--mention); box-shadow: inset 2px 0 0 var(--warn); }+.msg .body { min-width: 0; flex: 1; }+.msg .av { margin-top: 2px; }+.head { display: flex; align-items: baseline; gap: 8px; min-width: 0; line-height: 22px; }+.head .name { font-size: 15px; font-weight: 800; color: var(--text); cursor: pointer; }+.head .name:hover { text-decoration: underline; }+.head .uid { color: var(--faint); font-size: 12px; }+.head time, .stamp { color: var(--muted); font-size: 12px; font-variant-numeric: tabular-nums; }+.head .chat { font-size: 12px; color: var(--muted); cursor: pointer; }+.head .chat:hover { color: var(--text); }+.text { white-space: pre-wrap; overflow-wrap: anywhere; color: var(--text2); font-size: 15px; line-height: 22px; }+.text.long { max-height: 22.5em; overflow: hidden; -webkit-mask-image: linear-gradient(#000 70%, transparent); mask-image: linear-gradient(#000 70%, transparent); }+.more { font-size: 12.5px; color: var(--link); cursor: pointer; }+.text pre { background: var(--code-bg); border: 1px solid var(--line); border-radius: 7px; padding: 8px 10px; margin: 4px 0; white-space: pre-wrap; line-height: 1.5; }+.text code { background: var(--code-bg); border: 1px solid var(--line); border-radius: 4px; padding: 0 4px; }+.at { color: var(--at); background: var(--at-bg); border-radius: 4px; padding: 0 2px; }+.deleted { color: var(--faint); font-style: italic; }+.edited { color: var(--faint); font-size: 11.5px; }+.quote { display: flex; gap: 6px; align-items: center; font-size: 13px; color: var(--muted); margin: 2px 0 4px; cursor: pointer; min-width: 0; }+.quote::before { content: ""; width: 18px; height: 9px; border: 2px solid var(--line2); border-right: 0; border-bottom: 0; border-top-left-radius: 6px; margin-top: 6px; flex: none; }+.quote b { color: var(--text3); font-weight: 600; }+.quote span { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }+.quote:hover span { color: var(--text); }+.files { display: flex; flex-wrap: wrap; gap: 6px; margin: 6px 0 2px; }+.file { display: inline-flex; align-items: center; gap: 8px; padding: 6px 10px; border: 1px solid var(--line2); border-radius: 8px; background: var(--panel); font-size: 12.5px; }+.file:hover { background: var(--panel2); }+.file .k { color: var(--faint); }+.pics { display: flex; flex-wrap: wrap; align-items: flex-start; gap: 6px; margin: 6px 0 2px; }+.pic { display: block; flex: none; max-width: 100%; padding: 0; border: 1px solid var(--line2); border-radius: 8px; overflow: hidden; background: var(--panel2); cursor: zoom-in; position: relative; }+.pic img { display: block; width: 100%; height: 100%; object-fit: contain; }+.pic img:not([src]) { visibility: hidden; }+.pic.broken::after { content: "Could not load the picture"; position: absolute; inset: 0; display: flex; align-items: center; justify-content: center; color: var(--faint); font-size: 12px; }+/* text files: a card with the first lines under the message, and the whole file in a viewer */+.snip { width: min(640px, 100%); margin: 6px 0 2px; border: 1px solid var(--line2); border-radius: 8px; background: var(--panel); overflow: hidden; cursor: pointer; }+.snip.big { width: min(760px, 100%); }+.snip:hover, .snip:focus-visible { border-color: color-mix(in srgb, var(--accent) 50%, var(--line2)); outline: none; }+.sh { display: flex; align-items: center; gap: 8px; height: 36px; padding: 0 4px 0 10px; border-bottom: 1px solid var(--line); font-size: 13px; min-width: 0; }+.sh b { font-weight: 600; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; min-width: 0; }+.sh .k { color: var(--muted); font-size: 12px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }+.sh .dl { margin-left: auto; flex: none; width: 28px; height: 28px; display: grid; place-items: center; border-radius: 6px; color: var(--muted); }+.sh .dl:hover { background: var(--hover); color: var(--text); }+.sh .dl svg { width: 16px; height: 16px; }+.fx { flex: none; min-width: 32px; height: 18px; padding: 0 5px; border-radius: 4px; font: 800 9.5px/18px var(--sans); letter-spacing: .4px; text-align: center; color: #fff; background: var(--dm); }+.fx.doc { background: var(--board); }+.fx.data { background: var(--good); }+.fx.diff { background: var(--group); }+.fx.web { background: var(--channel); }+.fx.plain { background: light-dark(#6b7280, #4b5563); }+.sb { max-height: 168px; overflow: hidden; background: var(--code-bg); }+.sb.more { -webkit-mask-image: linear-gradient(#000 62%, transparent); mask-image: linear-gradient(#000 62%, transparent); }+.sb .code { padding: 6px 0; }+.sb .md { padding: 8px 14px; font-size: 13.5px; line-height: 1.5; }+.sb .md h1, .sb .md h2 { font-size: 1.25em; border: 0; padding: 0; margin: 0 0 .5em; }+.sb .grid { font-size: 12px; }+.code { font: 12.5px/20px var(--mono); white-space: pre; color: var(--text2); }+.code .cl { min-height: 20px; padding: 0 14px; }+.code.wrap .cl { white-space: pre-wrap; overflow-wrap: anywhere; }+.code.nums .cl { position: relative; padding-left: calc(var(--w) + 34px); }+.code.nums .cl::before { content: attr(data-n); position: absolute; left: 0; width: calc(var(--w) + 16px); text-align: right; color: var(--faint); }+.code.diff.nums .cl { padding-left: calc(2 * var(--w) + 52px); }+.code.diff.nums .cl::before { content: attr(data-a); }+.code.diff.nums .cl::after { content: attr(data-b); position: absolute; left: calc(var(--w) + 18px); top: 0; width: calc(var(--w) + 16px); text-align: right; color: var(--faint); }+.code .chunk { content-visibility: auto; }+.code .chunk:empty::before { content: "…"; padding: 0 14px; color: var(--faint); }+.cl.da { background: var(--diff-add); }+.cl.dd { background: var(--diff-del); }+.cl.dh { background: var(--diff-hunk); color: var(--muted); }+.cl.df { font-weight: 700; color: var(--text); background: var(--panel2); }+.cl.dm { color: var(--muted); }+.cl.le { background: var(--log-err); }+.cl.lw { background: var(--log-warn); }+.cl.lg { color: var(--muted); }+.lve { color: var(--bad); font-weight: 700; }+.lvw { color: var(--warn-ink); font-weight: 700; }+.lvi { color: var(--dm-soft); font-weight: 600; }+.lvd { color: var(--faint); font-weight: 600; }+.ck { color: var(--ck); } .cs { color: var(--cs); } .cn, .cb { color: var(--cn); } .cc { color: var(--cc); } .cf { color: var(--cf); }+.ct { color: var(--ct); } .ca { color: var(--ca); } .cv { color: var(--cv); } .ch { color: var(--ck); font-weight: 700; }+/* terminal colours in logs */+.a30 { color: light-dark(#24292f, #6e7681); } .a31 { color: light-dark(#cf222e, #ff7b72); } .a32 { color: light-dark(#116329, #3fb950); }+.a33 { color: light-dark(#9a6700, #d29922); } .a34 { color: light-dark(#0969da, #58a6ff); } .a35 { color: light-dark(#8250df, #bc8cff); }+.a36 { color: light-dark(#1b7c83, #39c5cf); } .a37 { color: light-dark(#6e7781, #b1bac4); } .a90 { color: light-dark(#57606a, #8b949e); }+.a91 { color: light-dark(#a40e26, #ffa198); } .a92 { color: light-dark(#1a7f37, #56d364); } .a93 { color: light-dark(#7d4e00, #e3b341); }+.a94 { color: light-dark(#218bff, #79c0ff); } .a95 { color: light-dark(#a475f9, #d2a8ff); } .a96 { color: light-dark(#3192aa, #56d4dd); }+.a97 { color: light-dark(#8c959f, #f0f6fc); } .ab { font-weight: 700; } .ad { opacity: .7; } .ai { font-style: italic; } .au { text-decoration: underline; }+.text pre .cl { padding: 0; }+/* Markdown */+.md { font-size: 15px; line-height: 1.65; color: var(--text2); overflow-wrap: anywhere; white-space: normal; }+.md > :first-child, .md > .chunk:first-child > :first-child { margin-top: 0; }+.md > .chunk { content-visibility: auto; contain-intrinsic-block-size: auto 3000px; }+.md > :last-child { margin-bottom: 0; }+.md h1, .md h2, .md h3, .md h4, .md h5, .md h6 { color: var(--text); font-weight: 700; line-height: 1.3; margin: 1.4em 0 .6em; }+.md h1 { font-size: 1.85em; padding-bottom: .3em; border-bottom: 1px solid var(--line); }+.md h2 { font-size: 1.45em; padding-bottom: .3em; border-bottom: 1px solid var(--line); }+.md h3 { font-size: 1.2em; } .md h4 { font-size: 1em; } .md h5 { font-size: .92em; } .md h6 { font-size: .88em; color: var(--muted); }+.md p, .md ul, .md ol, .md blockquote, .md pre, .md table { margin: 0 0 1em; }+.md ul, .md ol { padding-left: 1.7em; }+.md li + li { margin-top: .3em; }+.md li > ul, .md li > ol, .md li > p { margin: .3em 0 0; }+.md li.tasked { list-style: none; position: relative; }+.md .task { position: absolute; left: -1.45em; top: .36em; width: 14px; height: 14px; border: 1.5px solid var(--line2); border-radius: 4px; }+.md .task.done { background: var(--accent); border-color: var(--accent); box-shadow: inset 0 0 0 2px var(--panel); }+.md blockquote { padding: 0 1em; color: var(--muted); border-left: 3px solid var(--line2); }+.md code { font: .86em var(--mono); background: var(--panel2); border-radius: 5px; padding: .15em .4em; }+.md pre.code { padding: 12px 16px; background: var(--code-bg); border: 1px solid var(--line); border-radius: 8px; overflow-x: auto; }+.md hr { border: 0; border-top: 1px solid var(--line2); margin: 1.6em 0; }+.md table { border-collapse: collapse; display: block; max-width: 100%; overflow-x: auto; font-size: .94em; }+.md th, .md td { border: 1px solid var(--line2); padding: 6px 12px; }+.md th { background: var(--panel2); font-weight: 600; }+.md a:hover { text-decoration: underline; }+.md .nolink { color: var(--link); }+/* CSV and TSV */+.grid { width: max-content; min-width: 100%; font: 12.5px/18px var(--mono); color: var(--text2); }+.grid .gr { display: grid; grid-template-columns: var(--cols); }+.grid .gr > * { padding: 4px 12px; border-bottom: 1px solid var(--line); border-right: 1px solid var(--line); white-space: nowrap; overflow: hidden; text-overflow: ellipsis; background: var(--code-bg); }+.grid .gr > :first-child { position: sticky; left: 0; color: var(--faint); text-align: right; }+.grid .gh { position: sticky; top: 0; z-index: 1; }+.grid .gh > * { background: var(--panel2); color: var(--text); font: 700 12px/18px var(--sans); }+.grid .r { text-align: right; font-variant-numeric: tabular-nums; }+.grid .gr:not(.gh):hover > * { background: var(--hover); }+.grid .chunk { content-visibility: auto; contain-intrinsic-block-size: auto 13500px; }+.reacts { display: flex; flex-wrap: wrap; gap: 4px; margin-top: 6px; }+.react { display: inline-flex; align-items: center; gap: 4px; height: 24px; font-size: 12.5px; padding: 0 8px; border-radius: 12px; background: var(--panel2); border: 1px solid var(--line2); color: var(--text4); }+.foot { display: flex; gap: 12px; align-items: center; font-size: 12.5px; margin-top: 4px; min-height: 0; }+.foot .replies { color: var(--link); font-weight: 600; cursor: pointer; }+.foot .replies:hover { text-decoration: underline; }+.actions { position: absolute; right: 16px; top: -12px; display: none; gap: 2px; background: var(--panel); border: 1px solid var(--line2); border-radius: 8px; padding: 2px; z-index: 2; box-shadow: 0 4px 14px var(--shadow); }+.msg:hover .actions { display: flex; }+.actions button { font-size: 12px; padding: 3px 8px; border-radius: 6px; color: var(--text4); white-space: nowrap; }+.actions button:hover { background: var(--hover); color: var(--text); }+.sys { display: flex; align-items: baseline; gap: 8px; color: var(--muted); font-size: 13px; padding: 5px 20px 5px 64px; }+.sys > span:first-child { min-width: 0; }+.sys b { color: var(--text3); font-weight: 600; }+.divider { display: flex; align-items: center; gap: 10px; margin: 10px 20px; color: var(--accent-soft); font-size: 12px; font-weight: 700; }+.divider::before, .divider::after { content: ""; flex: 1; height: 1px; background: color-mix(in srgb, var(--accent) 55%, transparent); }+.divider.then { color: var(--warn-ink); }+.divider.then::before, .divider.then::after { background: color-mix(in srgb, var(--warn) 45%, transparent); }+.daysep { display: flex; align-items: center; margin: 16px 20px 6px; }+.daysep::before, .daysep::after { content: ""; flex: 1; height: 1px; background: var(--line); }+.daysep span { padding: 2px 12px; border: 1px solid var(--line); border-radius: 12px; background: var(--bg); color: var(--text3); font-size: 12.5px; font-weight: 700; }+.newpill { position: absolute; left: 50%; bottom: 18px; transform: translateX(-50%); background: var(--accent); color: #fff; font-weight: 600; font-size: 12.5px; padding: 6px 14px; border-radius: 16px; box-shadow: 0 6px 20px var(--shadow); z-index: 3; }++/* right pane */+#pane { grid-area: pane; width: 0; overflow: hidden; border-left: 1px solid var(--line); background: var(--side); display: flex; flex-direction: column; min-height: 0; }+#pane.open { width: 400px; }+#pane .phead { display: flex; align-items: center; gap: 8px; height: 52px; padding: 0 14px; border-bottom: 1px solid var(--line); flex: none; }+#pane .phead h2 { margin: 0; font-size: 15px; flex: 1; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; }+#pane .close { width: 28px; height: 28px; border-radius: 6px; color: var(--muted); font-size: 18px; }+#pane .close:hover { background: var(--hover); color: var(--text); }+.tree .msg { padding-left: calc(12px + var(--d, 0) * 14px); padding-right: 12px; }+.tree .msg::before { content: ""; position: absolute; left: calc(var(--d, 0) * 14px); top: 0; bottom: 0; width: 0; }+.tree .msg .av { width: 26px; height: 26px; font-size: 11px; border-radius: 7px; }+.tree .rail { position: absolute; top: 0; bottom: 0; width: 1px; background: var(--line2); }+.toggle { color: var(--faint); font-size: 11.5px; cursor: pointer; user-select: none; }+.toggle:hover { color: var(--text); }+.hiddenkids { color: var(--faint); font-size: 12px; padding: 0 0 4px calc(52px + var(--d, 0) * 14px); cursor: pointer; }++/* tables in panes */+.knew { padding: 6px 0; }+.knew .k { display: grid; grid-template-columns: 1fr auto; gap: 2px 10px; padding: 8px 14px; border-bottom: 1px solid var(--line); cursor: pointer; }+.knew .k:hover { background: var(--hover); }+.knew .k .t { font-weight: 600; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }+.knew .k .s { color: var(--muted); font-size: 12px; grid-column: 1 / -1; }+.knew .k .u { font-size: 12px; font-weight: 700; }+.knew .k .u.zero { color: var(--good); }+.knew .k .u.some { color: var(--warn); }+.note { color: var(--muted); font-size: 12.5px; padding: 10px 14px; line-height: 1.5; }++/* board */+.sorts { display: flex; gap: 2px; }+.sorts button { padding: 4px 10px; border-radius: 6px; color: var(--muted); font-weight: 600; font-size: 12.5px; }+.sorts button.on { background: var(--panel2); color: var(--text); }+.posts { max-width: 900px; margin: 0 auto; padding: 14px 18px 30px; display: flex; flex-direction: column; gap: 10px; }+.post { display: flex; gap: 12px; padding: 12px 14px; border: 1px solid var(--line); border-radius: 12px; background: var(--panel); cursor: pointer; }+.post:hover { border-color: var(--line2); background: var(--post-hover); }+.votes { width: 42px; flex: none; display: flex; flex-direction: column; align-items: center; gap: 1px; color: var(--muted); font-weight: 700; font-size: 13px; }+.votes svg { width: 16px; height: 16px; }+.votes.has { color: var(--channel-soft); }+.post .main { min-width: 0; flex: 1; }+.post .ptitle { font-size: 15.5px; font-weight: 650; color: var(--text); overflow-wrap: anywhere; }+.post .pbody { color: var(--text3); font-size: 13.5px; margin-top: 4px; display: -webkit-box; -webkit-line-clamp: 3; -webkit-box-orient: vertical; overflow: hidden; white-space: pre-wrap; overflow-wrap: anywhere; }+.post .pmeta { display: flex; flex-wrap: wrap; gap: 4px 10px; align-items: center; margin-top: 8px; color: var(--muted); font-size: 12px; }+.post .pmeta b { color: var(--text3); font-weight: 600; }+.post .pmeta .chat { color: var(--channel-soft); font-weight: 600; }+.post.deleted .ptitle { color: var(--faint); font-style: italic; }+.thread-page { max-width: 900px; margin: 0 auto; padding: 14px 18px 40px; }+.thread-page .post { cursor: default; }+.thread-page .post .pbody { -webkit-line-clamp: unset; display: block; color: var(--text2); }+.comments { margin-top: 10px; }+.comments .msg { padding-left: calc(8px + var(--d, 1) * 22px - 22px); }+.comments .msg .av { width: 26px; height: 26px; font-size: 11px; border-radius: 7px; }++/* person */+.person { max-width: 900px; margin: 0 auto; padding: 24px 18px 40px; }+.card { display: flex; gap: 18px; align-items: center; }+.card h1 { margin: 0; font-size: 22px; }+.card .uid { color: var(--muted); }+.fields { display: flex; flex-wrap: wrap; gap: 6px; margin-top: 6px; }+.field { font-size: 12px; padding: 2px 8px; border-radius: 6px; background: var(--panel2); color: var(--text4); }+.field i { color: var(--faint); font-style: normal; }+.kpis { display: grid; grid-template-columns: repeat(4, minmax(0, 1fr)); gap: 10px; margin: 18px 0; }+.kpi { border: 1px solid var(--line); border-radius: 10px; padding: 10px 12px; background: var(--panel); }+.kpi b { display: block; font-size: 20px; font-variant-numeric: tabular-nums; }+.kpi span { color: var(--muted); font-size: 12px; }+h3 { font-size: 12px; text-transform: uppercase; letter-spacing: .7px; color: var(--faint); margin: 20px 0 8px; }+.chips { display: flex; flex-wrap: wrap; gap: 6px; }+.chip { display: inline-flex; align-items: center; gap: 6px; padding: 4px 10px; border-radius: 8px; border: 1px solid var(--line2); background: var(--panel); cursor: pointer; font-size: 13px; }+.chip:hover { background: var(--panel2); }+.chip .n { color: var(--faint); font-size: 11.5px; }++/* overlays */+#overlay { position: fixed; inset: 0; z-index: 20; display: none; }+#overlay.on { display: block; }+.shade { position: absolute; inset: 0; background: var(--shade); }+.shade.dark { background: light-dark(rgba(16, 24, 40, .72), rgba(4, 6, 10, .88)); }+.lightbox { position: absolute; inset: 0; display: flex; flex-direction: column; align-items: center; justify-content: center; gap: 12px; pointer-events: none; }+.lightbox > * { pointer-events: auto; }+.lightbox img { display: block; border-radius: 6px; background: var(--panel2); box-shadow: 0 30px 80px var(--shadow); cursor: zoom-out; }+.lightbox .cap { display: flex; align-items: center; gap: 12px; max-width: 92vw; padding: 6px 6px 6px 14px; background: var(--panel); border: 1px solid var(--line2); border-radius: 10px; font-size: 13px; }+.lightbox .cap b { overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }+.lightbox .cap span { color: var(--muted); white-space: nowrap; }+.tv { position: absolute; left: 50%; top: 4vh; transform: translateX(-50%); width: min(1200px, 94vw); height: 92vh; display: flex; flex-direction: column; background: var(--panel); border: 1px solid var(--line2); border-radius: 12px; box-shadow: 0 30px 80px var(--shadow); overflow: hidden; }+.tvh { display: flex; align-items: center; gap: 10px; min-height: 52px; padding: 8px 10px 8px 16px; border-bottom: 1px solid var(--line); min-width: 0; }+.tvh b { font-size: 15px; font-weight: 700; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; min-width: 0; }+.tvh .k { color: var(--muted); font-size: 12.5px; white-space: nowrap; }+.tvh .grow { flex: 1; }+.tvh .btn.on { border-color: color-mix(in srgb, var(--accent) 60%, transparent); background: color-mix(in srgb, var(--accent) 14%, var(--panel)); color: var(--text); }+.tvh .x { width: 32px; height: 32px; border-radius: 8px; font-size: 22px; line-height: 1; color: var(--muted); }+.tvh .x:hover { background: var(--hover); color: var(--text); }+.tvb { flex: 1; min-height: 0; overflow: auto; background: var(--code-bg); }+.tvb.doc { background: var(--panel); }+.tvb .code { width: max-content; min-width: 100%; padding: 10px 0 28px; }+.tvb .code.wrap { width: auto; }+.tvb .md { max-width: 880px; margin: 0 auto; padding: 28px 36px 56px; }+.tvb .loading { padding: 18px; color: var(--muted); }+.tvn { padding: 8px 16px; border-top: 1px solid var(--line); color: var(--muted); font-size: 12.5px; }+.palette { position: absolute; left: 50%; top: 12vh; transform: translateX(-50%); width: min(680px, 92vw); background: var(--panel); border: 1px solid var(--line2); border-radius: 14px; box-shadow: 0 30px 80px var(--shadow); overflow: hidden; }+.palette input { width: 100%; padding: 16px 18px; background: transparent; border: 0; border-bottom: 1px solid var(--line); outline: none; font-size: 16px; }+.palette .res { max-height: 56vh; overflow-y: auto; padding: 6px 0; }+.palette .grp { color: var(--faint); font-size: 11.5px; font-weight: 700; letter-spacing: .6px; text-transform: uppercase; padding: 8px 18px 4px; }+.palette .it { display: flex; align-items: center; gap: 10px; padding: 7px 18px; cursor: pointer; }+.palette .it.on { background: color-mix(in srgb, var(--accent) 20%, transparent); }+.palette .it .t { flex: 1; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }+.palette .it .s { color: var(--muted); font-size: 12px; white-space: nowrap; overflow: hidden; text-overflow: ellipsis; max-width: 45%; }+.palette .hint { color: var(--faint); font-size: 12px; padding: 8px 18px 12px; border-top: 1px solid var(--line); }+.palette .hint code { color: var(--muted); }+.pop { position: absolute; background: var(--panel); border: 1px solid var(--line2); border-radius: 10px; box-shadow: 0 18px 50px var(--shadow); min-width: 260px; max-width: 380px; max-height: 60vh; overflow-y: auto; padding: 6px 0; }+.pop .it { display: flex; align-items: center; gap: 8px; padding: 6px 12px; cursor: pointer; font-size: 13px; }+.pop .it:hover, .pop .it.on { background: var(--hover); }+.pop .it .s { margin-left: auto; color: var(--faint); font-size: 11.5px; }+.pop input { margin: 4px 8px 6px; width: calc(100% - 16px); padding: 6px 9px; border-radius: 7px; border: 1px solid var(--line); background: var(--bg); outline: none; }+.pop .ph { padding: 4px 12px 6px; color: var(--faint); font-size: 11.5px; font-weight: 700; text-transform: uppercase; letter-spacing: .6px; }+#tip { position: fixed; z-index: 30; pointer-events: none; background: var(--tip-bg); border: 1px solid var(--line2); color: var(--tip-text); font-size: 12px; padding: 5px 8px; border-radius: 6px; max-width: 360px; display: none; box-shadow: 0 8px 24px var(--shadow); white-space: pre-line; }+#tip.on { display: block; }++/* search results */+.results { padding: 6px 0 30px; }+.results .msg { cursor: pointer; }+mark { background: light-dark(rgba(250, 204, 21, .45), rgba(250, 204, 21, .28)); color: inherit; border-radius: 3px; padding: 0 1px; }++/* map */+.mapwrap { position: relative; flex: 1; min-height: 0; display: flex; flex-direction: column; }+.mapwrap canvas.graph { flex: 1; min-height: 0; width: 100%; display: block; cursor: grab; }+.mapwrap canvas.graph.drag { cursor: grabbing; }+.mapwrap canvas.graph.point { cursor: pointer; }+.legend { position: absolute; left: 14px; top: 12px; background: var(--legend-bg); border: 1px solid var(--line); border-radius: 10px; padding: 8px 10px; font-size: 12px; backdrop-filter: blur(6px); max-width: 260px; }+.legend .lh { display: flex; align-items: center; gap: 6px; color: var(--muted); margin-bottom: 4px; }+.legend select { background: var(--panel2); border: 1px solid var(--line2); color: var(--text); border-radius: 6px; font: inherit; padding: 1px 4px; }+.legend .li { display: flex; align-items: center; gap: 7px; color: var(--text4); padding: 1px 0; }+.legend .li .n { margin-left: auto; color: var(--faint); font-variant-numeric: tabular-nums; }+.legend .shapes { display: flex; gap: 10px; color: var(--muted); margin-top: 6px; padding-top: 6px; border-top: 1px solid var(--line); }+.legend .shapes span { display: inline-flex; align-items: center; gap: 5px; }+.legend .sq { width: 9px; height: 9px; border-radius: 2px; display: inline-block; }+.mapctl { position: absolute; right: 14px; top: 12px; display: flex; gap: 6px; }+.mapctl .btn.on { border-color: color-mix(in srgb, var(--accent) 60%, transparent); background: color-mix(in srgb, var(--accent) 20%, var(--panel)); color: var(--text); }+.maphint { position: absolute; right: 14px; bottom: 98px; color: var(--faint); font-size: 11.5px; pointer-events: none; text-align: right; }+.strip { height: 86px; flex: none; border-top: 1px solid var(--line); background: var(--side); display: flex; align-items: stretch; gap: 10px; padding: 8px 14px; }+.strip .sctl { display: flex; flex-direction: column; justify-content: center; gap: 6px; width: 92px; flex: none; }+.strip .sctl .btn { justify-content: center; }+.strip .chart { position: relative; flex: 1; min-width: 0; }+.strip canvas { width: 100%; height: 100%; display: block; cursor: crosshair; }+.strip .when { position: absolute; left: 0; right: 0; bottom: -2px; display: flex; justify-content: space-between; color: var(--faint); font-size: 11px; pointer-events: none; }+.strip .sel { position: absolute; top: 0; color: var(--accent-soft); font-size: 11px; pointer-events: none; white-space: nowrap; }++/* login */+#login { position: fixed; inset: 0; z-index: 40; background: var(--bg); display: none; place-items: center; }+#login.on { display: grid; }+#login form { width: min(440px, 90vw); display: flex; flex-direction: column; gap: 12px; }+#login p { color: var(--muted); margin: 0; }+#login input { padding: 10px 12px; border-radius: 8px; border: 1px solid var(--line2); background: var(--panel); outline: none; font-family: var(--mono); font-size: 13px; }+#login button { padding: 10px; border-radius: 8px; background: var(--accent); color: #fff; font-weight: 700; }+#login .err { color: var(--bad); }+.prow { display: flex; gap: 8px; align-items: center; padding: 5px 14px; cursor: pointer; font-size: 13px; }+.prow:hover { background: var(--hover); }+.prow .t { flex: 1; min-width: 0; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }+.ph3 { margin: 14px 14px 6px; }+.msg.cont:hover::before { content: attr(data-time); position: absolute; left: 20px; width: 36px; text-align: center; top: 5px; font-size: 10.5px; line-height: 14px; color: var(--faint); font-variant-numeric: tabular-nums; }+.busy { width: 34px; height: 4px; border-radius: 2px; background: var(--line); flex: none; overflow: hidden; }+.busy i { display: block; height: 100%; background: color-mix(in srgb, var(--accent) 75%, transparent); border-radius: 2px; }diff --git a/swarmchat/page/app.js b/swarmchat/page/app.jsnew file mode 100644index 0000000..187c76d--- /dev/null+++ b/swarmchat/page/app.js@@ -0,0 +1,2408 @@+'use strict';+// swarmchat view: watch every chat on a server, live. Plain JavaScript with no build step, served as one page.+// Every bit of text an agent wrote goes through esc() or html``, and the page's Content-Security-Policy lets only+// this script run, so a message can never become code.++// ---------------------------------------------------------------------- helpers++const $ = (s, r = document) => r.querySelector(s);+const $$ = (s, r = document) => [...r.querySelectorAll(s)];+class Safe { constructor(s) { this.s = s; } toString() { return this.s; } }+const ESC = {'&': '&', '<': '<', '>': '>', '"': '"', "'": '''};+const esc = v => String(v).replace(/[&<>"']/g, c => ESC[c]);+const part = v => v == null || v === false ? '' : v instanceof Safe ? v.s : Array.isArray(v) ? v.map(part).join('') : esc(v);+// A template that escapes everything put into it, except what is already Safe.+const html = (strings, ...vals) => new Safe(strings.reduce((out, s, i) => out + part(vals[i - 1]) + s));+const sleep = ms => new Promise(r => setTimeout(r, ms));+const num = n => (n || 0).toLocaleString('en-US');+const plural = (n, one, many) => `${num(n)} ${n === 1 ? one : many || one + 's'}`;+const ms = at => typeof at === 'number' ? at * 1000 : Date.parse(at);+const pad = n => String(n).padStart(2, '0');+const clock = at => { const d = new Date(ms(at)); return `${pad(d.getHours())}:${pad(d.getMinutes())}`; };+const clockS = at => { const d = new Date(ms(at)); return `${clock(at)}:${pad(d.getSeconds())}`; };+const full = at => new Date(ms(at)).toLocaleString(undefined, {dateStyle: 'medium', timeStyle: 'medium'});+const dayOf = at => new Date(ms(at)).toDateString();+const dayName = at => {+ const d = new Date(ms(at)), now = new Date();+ if (d.toDateString() === now.toDateString()) return 'Today';+ return d.toLocaleDateString(undefined, {weekday: 'long', month: 'long', day: 'numeric', year: d.getFullYear() === now.getFullYear() ? undefined : 'numeric'});+};+function ago(at, ref = Date.now()) {+ const s = Math.max(0, (ref - ms(at)) / 1000);+ if (s < 45) return 'just now';+ if (s < 3600) return `${Math.round(s / 60)}m ago`;+ if (s < 86400) return `${Math.round(s / 3600)}h ago`;+ if (s < 86400 * 45) return `${Math.round(s / 86400)}d ago`;+ return new Date(ms(at)).toLocaleDateString(undefined, {month: 'short', day: 'numeric', year: 'numeric'});+}+const bytes = n => n < 1000 ? `${n} B` : n < 1e6 ? `${(n / 1e3).toFixed(1)} KB` : n < 1e9 ? `${(n / 1e6).toFixed(1)} MB` : `${(n / 1e9).toFixed(1)} GB`;+function hashHue(s) { let h = 2166136261; for (let i = 0; i < s.length; i++) h = Math.imul(h ^ s.charCodeAt(i), 16777619); return (h >>> 0) % 360; }+const userColor = id => `hsl(${hashHue(id)} 62% 68%)`;+function initials(name) {+ const w = String(name).replace(/[^\p{L}\p{N} ]/gu, ' ').trim().split(/\s+/).filter(Boolean);+ return ((w[0] || '?')[0] + (w.length > 1 ? w[w.length - 1][0] : (w[0] || '')[1] || '')).toUpperCase();+}+function throttle(fn, wait) {+ let last = 0, t = null;+ return () => {+ const left = wait - (Date.now() - last);+ if (left <= 0) { last = Date.now(); fn(); } else if (!t) t = setTimeout(() => { t = null; last = Date.now(); fn(); }, left);+ };+}++// ---------------------------------------------------------------------- talking to the server++let TOKEN = '';+async function api(name, params = {}) {+ const r = await fetch('/v1/' + name, {method: 'POST', body: JSON.stringify(params),+ headers: {'Authorization': 'Bearer ' + TOKEN, 'Content-Type': 'application/json'}});+ let data;+ try { data = await r.json(); } catch (_) { const e = new Error(`The server sent no JSON (HTTP ${r.status}).`); e.status = r.status; throw e; }+ if (!r.ok) { const e = new Error(data.error || `HTTP ${r.status}`); e.status = r.status; throw e; }+ return data;+}++async function download(id, name) {+ const r = await fetch('/v1/view_download', {method: 'POST', body: JSON.stringify({id}),+ headers: {'Authorization': 'Bearer ' + TOKEN, 'Content-Type': 'application/json'}});+ if (!r.ok) return toast((await r.json().catch(() => ({}))).error || 'Download failed.');+ const url = URL.createObjectURL(await r.blob());+ const a = Object.assign(document.createElement('a'), {href: url, download: name || 'file'});+ document.body.append(a); a.click(); a.remove();+ setTimeout(() => URL.revokeObjectURL(url), 30000);+}++// Pictures are fetched with the token as they come near the screen, and the newest are kept as object URLs. Each+// counts what it takes drawn, 4 bytes a pixel, or its file's size if that is more.+const PICS = new Map(); // attachment id -> {p: a promise of its object URL, size}+let picBytes = 0;+function picture(id, type, size) {+ let pic = PICS.get(id);+ if (pic) { PICS.delete(id); PICS.set(id, pic); return pic.p; } // the most recently used stay at the end+ const p = fetch('/v1/view_download', {method: 'POST', body: JSON.stringify({id}),+ headers: {'Authorization': 'Bearer ' + TOKEN, 'Content-Type': 'application/json'}})+ .then(async r => {+ if (!r.ok) throw new Error(`HTTP ${r.status}`);+ const b = await r.blob();+ return URL.createObjectURL(b.slice(0, b.size, type)); // the type the server read from the bytes+ });+ pic = {p, size};+ p.catch(() => { if (PICS.get(id) === pic) { PICS.delete(id); picBytes -= size; } });+ PICS.set(id, pic); picBytes += size;+ for (const [old, o] of PICS) {+ if (PICS.size <= 400 && picBytes <= 400e6) break;+ PICS.delete(old); picBytes -= o.size;+ o.p.then(url => URL.revokeObjectURL(url), () => {}); // a picture already drawn stays drawn+ }+ return p;+}+const near = new IntersectionObserver(entries => {+ for (const e of entries) {+ if (!e.isIntersecting) continue;+ const img = e.target, d = img.dataset;+ near.unobserve(img); watched.delete(img);+ const broken = () => img.parentElement.classList.add('broken');+ img.onerror = broken;+ picture(d.pic, d.type, Math.max(+d.size, 4 * d.w * d.h)).then(url => { img.src = url; }, broken);+ }+}, {rootMargin: '800px 0px'});+const watched = new Set();+function watchPictures(changes) { // only in what was just added: a long file in the viewer adds many lines+ for (const img of watched) if (!img.isConnected) { near.unobserve(img); watched.delete(img); }+ for (const c of changes) for (const n of c.addedNodes) {+ if (n.nodeType !== 1) continue;+ for (const img of n.matches('img[data-pic]') ? [n] : n.querySelectorAll('img[data-pic]')) {+ if (!img.classList.contains('seen')) { img.classList.add('seen'); watched.add(img); near.observe(img); }+ }+ }+}++// ---------------------------------------------------------------------- what the page knows++const S = {+ server: {}, users: new Map(), chats: new Map(), feed: 0, total: 0, sent: new Map(), last: new Map(),+ route: {}, tab: 'chats', filter: '', as: null, rate: [], view: null, pane: null, liveState: '',+};+const user = id => S.users.get(id) || {id, name: id, fields: {}, status: '', removed: true};+const name = id => user(id).name;+const first = id => name(id).split(/\s+/)[0];+const KIND = {channel: '#', group: '◆', direct: '●'};++function title(c, as = S.route.as) {+ if (!c) return 'unknown chat';+ if (c.name) return '#' + c.name;+ const people = c.members.filter(u => !as || u !== as || c.members.length === 1);+ const names = people.map(name);+ return names.length > 3 ? `${names.slice(0, 3).join(', ')} +${names.length - 3}` : names.join(c.kind === 'direct' && !as ? ' & ' : ', ');+}+const chatTitle = id => title(S.chats.get(id));++function addChat(c) {+ const old = S.chats.get(c.id);+ S.chats.set(c.id, c);+ if (!old) S.total += c.messages;+ else S.total += c.messages - old.messages;+}++async function loadSnapshot() {+ const t = performance.now();+ const r = await api('view_snapshot');+ S.server = r.server; S.feed = r.feed;+ S.users = new Map(r.users.map(u => [u.id, u]));+ S.chats = new Map(); S.total = 0; S.sent = new Map(); S.last = new Map();+ for (const c of r.chats) {+ addChat(c);+ for (const [u, n] of Object.entries(c.sent)) S.sent.set(u, (S.sent.get(u) || 0) + n);+ for (const [u, at] of Object.entries(c.last_by)) if (!S.last.has(u) || S.last.get(u) < at) S.last.set(u, at);+ }+ S.loadedIn = performance.now() - t;+}++// ---------------------------------------------------------------------- the address bar holds where you are++const KEYS = ['layout', 'chat', 'm', 'had', 'thread', 'knew', 'as', 'person', 'q', 'unread', 'post', 'sort'];+function readRoute() {+ const p = new URLSearchParams(location.hash.slice(1));+ const r = {};+ for (const k of KEYS) if (p.get(k)) r[k] = p.get(k);+ r.layout = ['chat', 'board', 'map'].includes(r.layout) ? r.layout : 'chat';+ return r;+}+// Goes somewhere. Keys set to null are dropped. Without keep, only the layout and the agent seen as come along.+function go(changes, keep = false) {+ const base = keep ? {...S.route} : {layout: S.route.layout, as: S.route.as};+ const r = {...base, ...changes};+ const p = new URLSearchParams();+ for (const k of KEYS) if (r[k] != null && r[k] !== '') p.set(k, r[k]);+ const hash = '#' + p.toString();+ if (location.hash !== hash) location.hash = hash; else route();+}++function viewKey(r) {+ if (r.layout === 'map') return 'map';+ if (r.q) return `q|${r.q}|${r.as}`;+ if (r.person) return `person|${r.person}`;+ if (r.unread && r.as) return `unread|${r.as}`;+ if (r.layout === 'board') return r.post ? `post|${r.post}|${r.as}` : `board|${r.chat}|${r.sort}|${r.as}`;+ return r.chat ? `chat|${r.chat}|${r.m}|${r.had}|${r.knew}|${r.as}` : `all|${r.as}`;+}++async function route() {+ const prev = S.route;+ S.route = readRoute();+ const r = S.route;+ document.body.className = 'layout-' + r.layout;+ if (r.as !== prev.as) await loadAs();+ const key = viewKey(r);+ if (r.as !== prev.as && key === S.viewKey && S.view && S.view.onAs) S.view.onAs();+ if (key !== S.viewKey) {+ if (S.view && S.view.destroy) S.view.destroy();+ S.viewKey = key;+ const main = $('#main');+ main.replaceChildren();+ S.view = r.layout === 'map' ? new MapView(main)+ : r.q ? new SearchView(main, r.q)+ : r.person ? new PersonView(main, r.person)+ : r.unread && r.as ? new UnreadView(main)+ : r.layout === 'board' ? (r.post ? new PostView(main, r.post) : new BoardView(main, r.chat, r.sort || 'hot'))+ : r.chat ? new ChatView(main, r.chat, {jump: r.m ? +r.m : null, had: r.had ? +r.had : null, knew: r.knew})+ : new ActivityView(main);+ }+ const paneKey = r.layout === 'map' ? 'map' : r.thread ? 'thread|' + r.thread : r.knew ? 'knew|' + r.knew : '';+ if (paneKey !== S.paneKey) {+ S.paneKey = paneKey;+ if (S.pane && S.pane.destroy) S.pane.destroy();+ S.pane = r.thread && r.layout !== 'map' ? new ThreadPane(r.thread) : r.knew && r.layout !== 'map' ? new KnewPane(r.knew) : null;+ $('#pane').classList.toggle('open', !!S.pane || (r.layout === 'map' && !!S.mapPane));+ if (!S.pane && r.layout !== 'map') $('#pane').replaceChildren();+ }+ renderTop();+ side.build();+}++// ---------------------------------------------------------------------- seeing as one agent++async function loadAs() {+ const id = S.route.as;+ if (!id) { S.as = null; return; }+ try {+ const r = await api('view_as', {user: id, unread: true, limit: 200});+ if (S.route.as !== id) return;+ S.as = {user: id, chats: new Map(r.chats.map(c => [c.id, c])), order: r.chats.map(c => c.id), positions: r.positions,+ unread: r.unread, unreadLeft: r.unread_left};+ } catch (e) {+ toast(e.message);+ S.as = null;+ }+}+// At most once a second, so steady traffic in the agent's chats still shows (a debounce would wait forever).+const refreshAs = throttle(async () => {+ if (!S.route.as) return;+ await loadAs();+ side.build();+ if (S.view && S.view.onAs) S.view.onAs();+}, 1000);++// ---------------------------------------------------------------------- top bar++function renderTop() {+ const r = S.route;+ for (const b of $$('#layouts button')) b.classList.toggle('on', b.dataset.layout === r.layout);+ const agents = [...S.users.values()].filter(u => !u.removed).length;+ const now = Date.now();+ S.rate = S.rate.filter(t => now - t < 10000);+ const rate = S.rate.length / Math.max(1, Math.min(10, (now - S.since) / 1000));+ $('#stats').innerHTML = html`<b>${num(agents)}</b> agents · <b>${num(S.chats.size)}</b> chats · <b>${num(S.total)}</b> messages${rate ? html` · <b>${rate < 10 ? rate.toFixed(1) : Math.round(rate)}</b>/s` : ''}`;+ const as = $('#as');+ as.classList.toggle('on', !!r.as);+ as.innerHTML = r.as ? html`${avatar(r.as, 'sm')}<span>As <b>${name(r.as)}</b></span><span class="x" data-clear-as title="See everything again">×</span>`+ : html`<span>As everyone</span><span>▾</span>`;+}+setInterval(() => renderTop(), 2000);++function setLive(state) {+ if (S.liveState === state) return;+ S.liveState = state;+ const el = $('#live');+ el.className = 'live ' + state;+ el.textContent = {on: 'Live', ro: 'Read-only', off: 'Offline'}[state] || '';+ el.title = {on: 'Following every chat as it happens', ro: `No server is running on ${S.server.data}: this is the folder as it was on disk`,+ off: 'Lost the server. Trying again.'}[state] || '';+}++function avatar(id, size = '') {+ const u = user(id);+ return html`<span class="av ${size}" style="--c:${userColor(id)}" title="${u.name}">${initials(u.name)}</span>`;+}++// ---------------------------------------------------------------------- sidebar: a list that draws only the rows on screen++const side = {+ rows: [], h: 30,+ list: null, space: null, box: null,+ init() {+ this.list = $('#list'); this.space = $('.vspace', this.list); this.box = $('.vrows', this.list);+ this.list.addEventListener('scroll', () => this.draw(), {passive: true});+ new ResizeObserver(() => this.draw()).observe(this.list);+ this.list.addEventListener('click', e => {+ const row = e.target.closest('[data-i]');+ if (row) this.rows[+row.dataset.i].open();+ });+ $('#filter').addEventListener('input', e => { S.filter = e.target.value.trim().toLowerCase(); this.build(); this.list.scrollTop = 0; });+ $('#tabs').addEventListener('click', e => {+ const b = e.target.closest('[data-tab]');+ if (!b) return;+ S.tab = b.dataset.tab; this.build(); this.list.scrollTop = 0;+ });+ },+ build() {+ for (const b of $$('#tabs button')) b.classList.toggle('on', b.dataset.tab === S.tab);+ $('#filter').placeholder = S.tab === 'people' ? 'Filter people' : 'Filter chats';+ this.rows = S.tab === 'people' ? peopleRows() : chatRows();+ this.space.style.height = this.rows.length * this.h + 'px';+ this.draw();+ },+ draw() {+ const top = this.list.scrollTop, height = this.list.clientHeight || 800;+ const start = Math.max(0, Math.floor(top / this.h) - 8), end = Math.min(this.rows.length, Math.ceil((top + height) / this.h) + 8);+ this.box.style.transform = `translateY(${start * this.h}px)`;+ let out = '';+ for (let i = start; i < end; i++) out += this.rows[i].html(i);+ this.box.innerHTML = out;+ },+};+const buildSide = throttle(() => side.build(), 500);++function chatRows() {+ const r = S.route, f = S.filter, rows = [];+ const special = (label, icon, on, open, n) => ({open, html: i => html`<div class="row special${on ? ' on' : ''}" data-i="${i}"><span class="ico">${icon}</span><span class="t">${label}</span>${n ? html`<span class="badge">${num(n)}</span>` : ''}</div>`.s});+ if (!f) {+ if (r.layout === 'board') rows.push(special('Front page', '★', !r.chat && !r.post && !r.q && !r.person && !r.unread, () => go({chat: null})));+ else rows.push(special('All activity', '≋', !r.chat && !r.q && !r.person && !r.unread, () => go({chat: null})));+ if (r.as && S.as) rows.push(special(`${first(r.as)}'s unread`, '✉', !!r.unread, () => go({unread: 1}),+ S.as.order.reduce((n, id) => n + (S.as.chats.get(id).unread || 0), 0)));+ }+ let chats;+ if (r.as && S.as) chats = S.as.order.map(id => S.chats.get(id)).filter(Boolean);+ else chats = [...S.chats.values()].sort((a, b) => (b.last_at > a.last_at) - (b.last_at < a.last_at));+ if (f) chats = chats.filter(c => title(c).toLowerCase().includes(f) || c.id.includes(f));+ S.busiest = chats.reduce((most, c) => Math.max(most, c.recent || 0), 1);+ const groups = [['channel', 'Channels'], ['group', 'Group chats'], ['direct', 'Direct chats']];+ for (const [kind, label] of groups) {+ const these = chats.filter(c => c.kind === kind);+ if (!these.length) continue;+ rows.push({open() {}, html: () => html`<div class="row head">${label} · ${num(these.length)}</div>`.s});+ for (const c of these) rows.push(chatRow(c));+ }+ if (rows.length <= 2 && f) rows.push({open() {}, html: () => '<div class="row head">Nothing matches</div>'});+ return rows;+}++function chatRow(c) {+ const r = S.route;+ const mine = r.as && S.as ? S.as.chats.get(c.id) : null;+ return {+ open: () => go({chat: c.id, layout: r.layout === 'map' ? 'chat' : r.layout}),+ html: i => {+ const on = r.chat === c.id && !r.q && !r.person && !r.unread;+ const unread = mine ? mine.unread : 0;+ const badge = mine ? (mine.mentioned ? html`<span class="badge at" title="Mentioned">@${unread}</span>` : unread ? html`<span class="badge${mine.notify !== 'all' ? ' mute' : ''}" title="Unread for ${name(r.as)}${mine.notify !== 'all' ? `, notify ${mine.notify}` : ''}">${num(unread)}</span>` : '')+ : html`<span class="n" title="Messages">${num(c.messages)}</span>`;+ const busy = c.recent ? Math.max(8, Math.round(100 * Math.sqrt(c.recent / S.busiest))) : 0;+ return html`<div class="row${on ? ' on' : ''}${unread ? ' unread' : ''}" data-i="${i}" title="${title(c)}${c.public ? ' (public)' : ''}${c.archived ? ' (archived)' : ''}${c.description ? '\n' + c.description : ''}+${num(c.recent || 0)} messages in the last 15 minutes"><span class="ico ${c.kind}">${KIND[c.kind]}</span><span class="t">${c.name || title(c)}</span>${busy ? html`<span class="busy"><i style="width:${busy}%"></i></span>` : ''}${badge}</div>`.s;+ },+ };+}++function peopleRows() {+ const f = S.filter, r = S.route;+ let people = [...S.users.values()];+ if (f) people = people.filter(u => `${u.id} ${u.name} ${u.status} ${Object.values(u.fields || {}).join(' ')}`.toLowerCase().includes(f));+ people.sort((a, b) => (S.last.get(b.id) || 0) - (S.last.get(a.id) || 0) || a.name.localeCompare(b.name));+ const rows = [{open() {}, html: () => html`<div class="row head">People · ${num(people.length)}</div>`.s}];+ for (const u of people) {+ rows.push({+ open: () => go({person: u.id, layout: r.layout === 'map' ? 'chat' : r.layout}),+ html: i => {+ const role = Object.values(u.fields || {})[0];+ const last = S.last.get(u.id);+ return html`<div class="row${r.person === u.id ? ' on' : ''}" data-i="${i}" title="${u.name} (${u.id})${u.status ? '\n' + u.status : ''}\n${num(S.sent.get(u.id) || 0)} messages sent">${avatar(u.id, 'sm')}<span class="t">${u.name}${u.removed ? ' (removed)' : ''}${role ? html` <span class="n">${role}</span>` : ''}</span><span class="n" title="Last message">${last ? ago(last).replace(' ago', '') : ''}</span></div>`.s;+ },+ });+ }+ return rows;+}++// ---------------------------------------------------------------------- how a message looks++const EMOJI = {'+1': '👍', '-1': '👎', 'thumbsup': '👍', 'thumbsdown': '👎', 'heart': '❤️', 'eyes': '👀', 'check': '✅', 'x': '❌', 'tada': '🎉', 'fire': '🔥', 'rocket': '🚀', 'smile': '🙂'};+function inline(text) {+ let s = esc(text);+ s = s.replace(/`([^`\n]+)`/g, '<code>$1</code>');+ s = s.replace(/(^|[\s(>])@([\p{L}\p{N}._-]+)/gu, '$1<span class="at">@$2</span>');+ s = s.replace(/\bhttps?:\/\/(?:[^\s<&]|&)*[^\s<.,;:!?)\]'"&]/g, u => `<a href="${u}" target="_blank" rel="noopener noreferrer">${u}</a>`);+ return s;+}+function rich(text, mark) {+ if (mark) return highlight(text, mark);+ return new Safe(String(text).split(/(```[\s\S]*?```)/g).map(p =>+ p.length >= 6 && p.startsWith('```') && p.endsWith('```') ? fence(p.slice(3, -3)) : inline(p)).join(''));+}+function fence(body) { // a code block in a message, highlighted when it names its language, as ```python+ const lang = /^([\w+#.-]*)\n/.exec(body), kind = lang && langKind(lang[1]);+ return `<pre>${kind ? blockHtml(body.slice(lang[0].length), kind) : esc(lang ? body.slice(lang[0].length) : body)}</pre>`;+}+function highlight(text, term) {+ const low = text.toLowerCase(), t = term.toLowerCase();+ let out = '', i = 0, j;+ while (t && (j = low.indexOf(t, i)) >= 0) { out += esc(text.slice(i, j)) + '<mark>' + esc(text.slice(j, j + t.length)) + '</mark>'; i = j + t.length; }+ return new Safe(out + esc(text.slice(i)));+}++// Files under a message or post. Pictures are drawn, in a box of their own shape that is there before their bytes+// arrive, so nothing moves when they do. A text file shows its first lines, and opens whole on a click. Anything else+// is a button that downloads it.+function filesHtml(list, big) {+ if (!list || !list.length) return '';+ const pics = list.filter(a => a.image), texts = list.filter(a => a.text), rest = list.filter(a => !a.image && !a.text);+ return html`${pics.length ? html`<div class="pics">${pics.map(a => picHtml(a, big))}</div>` : ''}${texts.map(a => snipHtml(a, big))}${rest.length ? html`<div class="files">${rest.map(a => html`<button class="file" data-file="${a.id}" data-name="${a.name}" title="Download">${a.folder ? '🗂' : '📄'} ${a.name}<span class="k">${a.folder ? 'folder · ' : ''}${bytes(a.size)}</span></button>`)}</div>` : ''}`;+}+function picHtml(a, big) {+ const {type, width: w, height: h} = a.image;+ const k = Math.min(1, (big ? 640 : 400) / w, (big ? 480 : 300) / h);+ return html`<button class="pic" data-image="${a.id}" data-name="${a.name}" data-type="${type}" data-w="${w}" data-h="${h}" data-size="${a.size}" title="${a.name} · ${w} × ${h} · ${bytes(a.size)}" style="width:${Math.max(1, Math.round(w * k))}px;aspect-ratio:${w} / ${h}"><img data-pic="${a.id}" data-type="${type}" data-size="${a.size}" data-w="${w}" data-h="${h}" alt="${a.name}"></button>`;+}+function openPicture(d) {+ const w = +d.w, h = +d.h, k = Math.min(1, innerWidth * 0.92 / w, (innerHeight - 90) * 0.92 / h);+ const box = document.createElement('div');+ box.className = 'lightbox';+ box.innerHTML = html`<img alt="${d.name}" width="${Math.max(1, Math.round(w * k))}" height="${Math.max(1, Math.round(h * k))}"><div class="cap"><b>${d.name}</b><span>${w} × ${h} · ${bytes(+d.size)}</span><button class="btn" data-file="${d.image}" data-name="${d.name}">Download</button></div>`.s;+ overlay.open(box);+ $('.shade', overlay.el).classList.add('dark');+ const img = $('img', box);+ img.onclick = () => overlay.close();+ picture(d.image, d.type, Math.max(+d.size, 4 * w * h)).then(url => { img.src = url; }, () => toast('The picture could not be loaded.'));+}++// ---------------------------------------------------------------------- text files++// A text file shows under its message as a card with its first lines, and opens whole in a viewer, formatted for its+// kind: code is highlighted, Markdown rendered, a diff shows its added and removed lines, a log its levels and+// terminal colours, a CSV file a table. The kind comes from the file's name, and for some names from its first lines.+// Every piece of a file goes through esc() before markup is put around it, so a file can never become code.++// extension: kind of formatting and the name shown+const EXT = Object.fromEntries(('py python Python|pyi python Python|js js JavaScript|mjs js JavaScript|cjs js JavaScript|' ++ 'jsx js JavaScript|ts js TypeScript|tsx js TypeScript|json json JSON|ipynb json Notebook|geojson json GeoJSON|' ++ 'jsonl jsonl JSON lines|ndjson jsonl JSON lines|sh shell Shell|bash shell Shell|zsh shell Shell|fish shell Shell|' ++ 'html html HTML|htm html HTML|xhtml html HTML|xml html XML|svg html SVG|vue html Vue|css css CSS|scss css SCSS|' ++ 'less css Less|c c C|h c C|cc c C++|cpp c C++|cxx c C++|hpp c C++|cu c CUDA|java c Java|kt c Kotlin|kts c Kotlin|' ++ 'go c Go|rs c Rust|cs c C#|swift c Swift|scala c Scala|php c PHP|dart c Dart|zig c Zig|proto c Protobuf|' ++ 'rb hash Ruby|pl hash Perl|r hash R|jl hash Julia|mk hash Makefile|cmake hash CMake|sql sql SQL|yaml yaml YAML|' ++ 'yml yaml YAML|toml ini TOML|ini ini INI|cfg ini Config|conf ini Config|env ini Environment|properties ini Properties|' ++ 'md markdown Markdown|markdown markdown Markdown|mdx markdown MDX|diff diff Diff|patch diff Patch|log log Log|' ++ 'out log Output|err log Errors|csv csv CSV|tsv tsv TSV|txt text Text|text text Text|rst text reStructuredText|tex text TeX')+ .split('|').map(s => { const [ext, kind, ...label] = s.split(' '); return [ext, [kind, label.join(' ')]]; }));+const BARE = {dockerfile: ['hash', 'Dockerfile'], makefile: ['hash', 'Makefile'], gnumakefile: ['hash', 'Makefile'],+ 'cmakelists.txt': ['hash', 'CMake'], '.gitignore': ['hash', 'Git ignore'], '.dockerignore': ['hash', 'Docker ignore'],+ '.env': ['ini', 'Environment'], '.bashrc': ['shell', 'Shell'], '.zshrc': ['shell', 'Shell'], '.profile': ['shell', 'Shell']};+// a file's badge, when its extension is long or missing, and its colour+const BADGE = {markdown: 'md', diff: 'diff', python: 'py', js: 'js', shell: 'sh', html: 'html', css: 'css', c: 'code', hash: 'code',+ json: 'json', jsonl: 'json', yaml: 'yaml', ini: 'conf', sql: 'sql', log: 'log', csv: 'csv', tsv: 'tsv'};+const FAMILY = {markdown: 'doc', text: 'plain', log: 'plain', diff: 'diff', csv: 'data', tsv: 'data', json: 'data', jsonl: 'data',+ html: 'web', css: 'web'};++const extOf = name => { const base = name.split('/').pop().toLowerCase(), dot = base.lastIndexOf('.'); return dot > 0 ? base.slice(dot + 1) : ''; };+function fileKind(name, sample = '') {+ const base = name.split('/').pop().toLowerCase();+ let [kind, label] = BARE[base] || EXT[extOf(name)] || ['text', 'Text'];+ if (kind === 'text') { // a file named for no kind, or plain text, may still be a diff, a script or a log+ if (/^(diff |--- \S|Index: |From [0-9a-f]{40} )/.test(sample) && /^\+\+\+ /m.test(sample)) [kind, label] = ['diff', 'Diff'];+ else if (/^#!.*\b(?:ba|z|da|k)?sh\b/.test(sample)) [kind, label] = ['shell', 'Shell'];+ else if (/^#!.*python/.test(sample)) [kind, label] = ['python', 'Python'];+ else if (/^#!.*\b(node|deno|bun)\b/.test(sample)) [kind, label] = ['js', 'JavaScript'];+ else if (sample.split('\n').filter(l => LOG_TIME.test(l)).length >= 3) [kind, label] = ['log', 'Log'];+ }+ const ext = extOf(name);+ return {kind, label, badge: (ext && ext.length <= 4 ? ext : BADGE[kind] || 'txt').toUpperCase()};+}++// ---- highlighting: each kind of code is a list of [class, pattern], and at each place the first that matches wins.+// Patterns use no capturing groups. "(?<![^\n])" means at the start of a line, "(?![\s\S])" at the end of the text.+const words = list => new RegExp(`\\b(?:${list.trim().split(/\s+/).join('|')})\\b`);+const NUM = /\b(?:0[xX][\da-fA-F_]+|0[bB][01_]+|0[oO][0-7_]+|\d[\d_]*(?:\.\d[\d_]*)?(?:[eE][+-]?\d+)?)[jJlLuUfFnN]?\b/;+const DQ = /"(?:[^"\\\n]|\\.)*"?/, SQ = /'(?:[^'\\\n]|\\.)*'?/;+const BLOCK = /\/\*[\s\S]*?(?:\*\/|(?![\s\S]))/, SLASHES = /\/\/[^\n]*/, HASHES = /(?<![\w$])#[^\n]*/;+const WORD = /[A-Za-z_$][\w$]*/, TYPE = /\b[A-Z][A-Za-z0-9_]*[a-z][A-Za-z0-9_]*\b/, CALL = /\b[A-Za-z_$][\w$]*(?=\s*\()/;+const LANG = {+ python: [['cc', /#[^\n]*/], ['cs', /(?<!\w)[rRbBuUfF]{0,2}(?:"""[\s\S]*?(?:"""|(?![\s\S]))|'''[\s\S]*?(?:'''|(?![\s\S])))/],+ ['cs', /(?<!\w)[rRbBuUfF]{0,2}(?:"(?:[^"\\\n]|\\.)*"?|'(?:[^'\\\n]|\\.)*'?)/], ['cf', /(?<![\w)\]])@[A-Za-z_][\w.]*/],+ ['ck', words('and as assert async await break class continue def del elif else except finally for from global if import in is lambda nonlocal not or pass raise return try while with yield')],+ ['cb', words('True False None self cls')], ['cf', /(?<=\b(?:def|class)\s+)[A-Za-z_]\w*/], ['cf', CALL], ['cn', NUM], ['', WORD]],+ js: [['cc', SLASHES], ['cc', BLOCK], ['cs', /`(?:[^`\\]|\\[\s\S])*`?/], ['cs', DQ], ['cs', SQ],+ ['ck', words('break case catch class const continue debugger default delete do else export extends finally for from function if import in instanceof let new of return static super switch throw try typeof var void while with yield async await as interface type enum implements private public protected readonly declare namespace abstract')],+ ['cb', words('true false null undefined this NaN Infinity')], ['ct', TYPE], ['cf', CALL], ['cn', NUM], ['', WORD]],+ c: [['cc', SLASHES], ['cc', BLOCK], ['cf', /(?<![^\n])[ \t]*#[ \t]*[a-z]+\b[^\n]*/], ['cs', /`[^`]*`?/], ['cs', DQ],+ ['cs', /'(?:[^'\\\n]|\\.){1,8}'/],+ ['ck', words('auto break case catch char class const constexpr continue default delete do double else enum explicit extern final float for friend goto if inline int long namespace new noexcept operator override private protected public register return short signed sizeof static struct switch template throw try typedef typename union unsigned using virtual void volatile while bool boolean byte extends implements import instanceof interface package super synchronized throws abstract func go chan defer fallthrough range select var fn let mut impl trait pub use mod crate match async await fun companion sealed internal suspend guard extension protocol')],+ ['cb', words('true false null nullptr NULL nil None Some Ok Err this self Self')], ['ct', TYPE], ['cf', CALL], ['cn', NUM], ['', WORD]],+ shell: [['cc', HASHES], ['cs', /"(?:[^"\\]|\\[\s\S])*"?/], ['cs', /'[^']*'?/], ['cv', /\$(?:\{[^}\n]*\}|[A-Za-z_]\w*|[0-9@#?*!$-])/],+ ['ck', words('if then else elif fi for while until do done case esac in function return local export readonly declare unset shift break continue exit source alias set trap eval exec select')],+ ['cn', /\b\d+\b/], ['', /[A-Za-z_][\w-]*/]],+ hash: [['cc', HASHES], ['cs', DQ], ['cs', SQ],+ ['ck', /(?<![^\n])[ \t]*(?:FROM|RUN|COPY|ADD|CMD|ENTRYPOINT|ENV|ARG|WORKDIR|EXPOSE|USER|LABEL|VOLUME|HEALTHCHECK|SHELL|ONBUILD|STOPSIGNAL)\b/],+ ['cf', /(?<![^\n])[\w./%$()-]+(?=[ \t]*::?(?!=))/], ['cv', /\$[({][^)}\n]*[)}]|\$\w+/],+ ['ck', words('def end do if elsif else unless while until class module return yield begin rescue ensure require function local then for in sub my use')],+ ['cb', words('true false nil TRUE FALSE NULL self')], ['cn', NUM], ['', WORD]],+ json: [['ca', /"(?:[^"\\\n]|\\.)*"(?=\s*:)/], ['cs', DQ], ['cb', /\b(?:true|false|null)\b/], ['cn', /-?\b\d+(?:\.\d+)?(?:[eE][+-]?\d+)?\b/]],+ yaml: [['cc', /(?<![^\s])#[^\n]*/], ['cc', /(?<![^\n])(?:---|\.\.\.)(?=\s|(?![\s\S]))/], ['ca', /[\w.\/-]+(?=:(?:[ \t\n]|(?![\s\S])))/],+ ['ca', /"[^"\n]*"(?=:\s)/], ['cs', DQ], ['cs', /'(?:[^'\n]|'')*'?/], ['cf', /[&*][\w-]+/],+ ['cb', /\b(?:true|false|yes|no|on|off|null|True|False|Yes|No|NULL|Null)\b|(?<=:\s*)~/], ['cn', NUM], ['', WORD]],+ ini: [['cc', /(?<![^\n])[ \t]*[#;][^\n]*/], ['ct', /(?<![^\n])[ \t]*\[[^\]\n]*\]+/], ['ca', /(?<![^\n])[ \t]*[\w.\-"' ]*?[\w"'](?=[ \t]*[=:])/],+ ['cs', /"""[\s\S]*?(?:"""|(?![\s\S]))/], ['cs', DQ], ['cs', /'[^'\n]*'?/], ['cb', words('true false yes no on off null none')], ['cn', NUM], ['', WORD]],+ css: [['cc', BLOCK], ['cs', DQ], ['cs', SQ], ['ck', /@[\w-]+/], ['ca', /[\w-]+(?=\s*:(?![^{};\n]*\{))/], ['cn', /#[\da-fA-F]{3,8}\b/],+ ['', /[A-Za-z_-][\w-]*/], ['cn', /(?:\d+\.?\d*|\.\d+)(?:%|[a-zA-Z]+)?/]],+ sql: [['cc', /--[^\n]*/], ['cc', BLOCK], ['cs', /'(?:[^']|'')*'?/], ['ca', /"[^"\n]*"?|`[^`\n]*`?/],+ ['ck', /\b(?:select|from|where|and|or|not|insert|into|values|update|set|delete|create|table|index|view|drop|alter|add|column|primary|key|foreign|references|join|left|right|inner|outer|full|cross|on|as|group|by|order|having|limit|offset|union|all|distinct|case|when|then|else|end|is|null|like|ilike|in|between|exists|with|returning|default|unique|check|constraint|asc|desc|if|begin|commit|rollback|transaction|over|partition)\b/i],+ ['cf', CALL], ['cn', NUM], ['', WORD]],+ markdown: [['ch', /(?<![^\n])#{1,6}[ \t][^\n]*/], ['cs', /(?<![^\n])[ \t]*(?:```|~~~)[\s\S]*?(?:\n[ \t]*(?:```|~~~)[^\n]*|(?![\s\S]))/],+ ['cc', /(?<![^\n])[ \t]*>[^\n]*/], ['ck', /(?<![^\n])[ \t]*(?:[-*+]|\d+[.)])(?=[ \t])/], ['cs', /`[^`\n]+`/],+ ['cf', /!?\[[^\]\n]*\]\([^)\n]*\)/], ['cb', /\*\*[^*\n]+\*\*|__[^_\n]+__/]],+ html: [['cc', /<!--[\s\S]*?(?:-->|(?![\s\S]))/], ['cc', /<![A-Za-z][^>]*>|<\?[\s\S]*?(?:\?>|(?![\s\S]))/],+ [s => embedded(s, 'js'), /<script\b[^>]*>[\s\S]*?(?:<\/script\s*>|(?![\s\S]))/i],+ [s => embedded(s, 'css'), /<style\b[^>]*>[\s\S]*?(?:<\/style\s*>|(?![\s\S]))/i],+ [tagPieces, /<\/?[A-Za-z][\w:.-]*(?:\s[^<>]*?)?\/?>/], ['cv', /&#?\w+;/]],+};+LANG.jsonl = LANG.json;+const REGEX = {};++// [class, text] pieces that cover the whole text, in order. A class can be a function that splits its match further.+// A generator, so a large file is formatted a part at a time: the scan goes on from where it stopped, and a comment+// or string that runs over many lines is still one piece.+function* pieces(text, kind) {+ const rules = LANG[kind];+ if (!rules) return yield ['', text];+ const re = new RegExp(REGEX[kind] ||= rules.map(([, r]) => `(${r.source})`).join('|'), 'g' + (kind === 'sql' ? 'i' : ''));+ let at = 0, m;+ while ((m = re.exec(text))) {+ if (!m[0]) { re.lastIndex++; continue; }+ if (m.index > at) yield ['', text.slice(at, m.index)];+ let g = 1;+ while (m[g] === undefined) g++;+ const cls = rules[g - 1][0];+ if (typeof cls === 'function') yield* cls(m[0]); else yield [cls, m[0]];+ at = m.index + m[0].length;+ }+ if (at < text.length) yield ['', text.slice(at)];+}+const tokens = (text, kind) => [...pieces(text, kind)];+function tagPieces(s) {+ const name = /^<\/?([\w:.-]+)/.exec(s), out = [['', s.slice(0, name.index + name[0].length - name[1].length)], ['ct', name[1]]];+ const re = /("[^"]*"?|'[^']*'?)|([^\s=>\/"']+)/g, rest = s.slice(name[0].length);+ let at = 0, m;+ while ((m = re.exec(rest))) {+ if (m.index > at) out.push(['', rest.slice(at, m.index)]);+ out.push([m[1] ? 'cs' : 'ca', m[0]]);+ at = m.index + m[0].length;+ }+ out.push(['', rest.slice(at)]);+ return out;+}+function embedded(s, kind) { // a <script> or <style> element: its tags, and what is inside as JavaScript or CSS+ const open = /^<[^>]*>/.exec(s)[0], close = /<\/\w+\s*>$/.exec(s), inner = s.slice(open.length, close ? close.index : s.length);+ return [...tagPieces(open), ...tokens(inner, kind), ...(close ? tagPieces(close[0]) : [])];+}+// Pieces as lines of HTML. A piece that runs over several lines is closed at the end of each and opened again.+function* htmlLines(list) {+ let cur = '';+ for (const [cls, s] of list) {+ const parts = s.split('\n');+ for (let i = 0; i < parts.length; i++) {+ if (i) { yield cur; cur = ''; }+ if (parts[i]) cur += cls ? `<span class="${cls}">${esc(parts[i])}</span>` : esc(parts[i]);+ }+ }+ yield cur;+}+const linesOf = list => [...htmlLines(list)];+const codeHtml = (text, kind) => linesOf(tokens(text, kind)).join('\n');+// A code block that names its language, in a message or in Markdown: ```python, ```ts, ```diff.+const ALIAS = {javascript: 'js', typescript: 'js', node: 'js', bash: 'shell', console: 'shell', shell: 'shell', shellsession: 'shell',+ python3: 'python', golang: 'c', rust: 'c', java: 'c', cpp: 'c', 'c++': 'c', csharp: 'c', kotlin: 'c', swift: 'c', ruby: 'hash',+ dockerfile: 'hash', makefile: 'hash', toml: 'ini', jsonc: 'json', xml: 'html', svg: 'html', patch: 'diff'};+const langKind = name => { const n = (name || '').toLowerCase(); return ALIAS[n] || (EXT[n] || [])[0] || (LANG[n] || n === 'diff' || n === 'log' ? n : null); };+const blockHtml = (text, kind) => kind === 'diff' || kind === 'log' ? textLines(text, kind).map(lineHtml).join('') : codeHtml(text, kind);++// ---- lines: {h: html, c: a class for the line, n: its number}, or for a diff a: and b:, its old and new numbers.+// lineBatches gives them size at a time, so the viewer shows the first while it formats the rest.+function* lineBatches(text, kind, size) {+ if (text.endsWith('\n')) text = text.slice(0, -1);+ if (kind === 'diff' || kind === 'log' || kind === 'text') { // a line at a time, each knowing the lines before it+ const all = text.split('\n'), line = kind === 'diff' ? diffLine() : logLine(kind === 'log');+ for (let i = 0; i < all.length; i += size) yield all.slice(i, i + size).map(line);+ return;+ }+ let batch = [], n = 0;+ for (const h of htmlLines(pieces(text, kind))) {+ batch.push({h, n: ++n});+ if (batch.length >= size) { yield batch; batch = []; }+ }+ yield batch;+}+const textLines = (text, kind) => [...lineBatches(text, kind, Infinity)].flat();+const lineHtml = l => `<div class="cl${l.c ? ' ' + l.c : ''}"${l.n ? ` data-n="${l.n}"` : ''}${l.a ? ` data-a="${l.a}"` : ''}${l.b ? ` data-b="${l.b}"` : ''}>${l.h}</div>`;++// A diff: each file's header, each hunk's header, and its lines added and removed, numbered as in the old and the+// new file, with the code highlighted as the file's kind. Counting each hunk's lines keeps a removed line that+// starts with "--" from being taken for a header.+function diffLine() {+ let a = 0, b = 0, left = 0, right = 0, kind = 'text', m;+ const code = s => linesOf(pieces(s, kind))[0];+ return line => {+ if (left > 0 || right > 0) {+ const c = line[0];+ if (c === '+') { right--; return {c: 'da', b: b++, h: '+' + code(line.slice(1))}; }+ if (c === '-') { left--; return {c: 'dd', a: a++, h: '-' + code(line.slice(1))}; }+ if (c === '\\') return {c: 'dm', h: esc(line)};+ if (c === ' ' || line === '') { left--; right--; return {a: a++, b: b++, h: ' ' + code(line.slice(1))}; }+ left = right = 0;+ }+ if ((m = /^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@/.exec(line))) {+ a = +m[1]; left = m[2] == null ? 1 : +m[2]; b = +m[3]; right = m[4] == null ? 1 : +m[4];+ return {c: 'dh', h: `<span class="cf">${esc(m[0])}</span>${esc(line.slice(m[0].length))}`};+ }+ if ((m = /^diff --git a\/.* b\/(.*)$/.exec(line))) kind = fileKind(m[1]).kind;+ else if ((m = /^(?:\+\+\+|---) (?:[ab]\/)?(.+?)(?:\t.*)?$/.exec(line)) && m[1] !== '/dev/null') kind = fileKind(m[1]).kind;+ return {c: /^diff /.test(line) ? 'df' : 'dm', h: esc(line)};+ };+}++// A log, or plain text: terminal colours shown, other escapes dropped, and what a terminal would show after a+// carriage return. In a log, the time at the start of a line is dimmed, its level stands out, an error's line and a+// Python traceback are tinted red and a warning's yellow.+const LOG_TIME = /^\[?(?:\d{4}-\d\d-\d\d[T ]\d\d:\d\d(?::\d\d(?:[.,]\d+)?)?(?:Z|[+-]\d\d:?\d\d)?|\d\d:\d\d:\d\d(?:[.,]\d+)?|[A-Z][a-z]{2} [ \d]\d \d\d:\d\d:\d\d)\]?/;+const LOG_LEVEL = /\b(?:(FATAL|CRITICAL|ERROR|SEVERE|PANIC|EMERG)|(WARN(?:ING)?)|(INFO|NOTICE)|(DEBUG|TRACE))\b/;+function logLine(levels) {+ const st = {};+ let trace = false, n = 0;+ return raw => {+ raw = raw.replace(/\r+$/, '');+ if (raw.includes('\r')) raw = raw.slice(raw.lastIndexOf('\r') + 1);+ raw = raw.replace(/[\x00-\x08\x0b\x0c\x0e-\x1a\x1c-\x1f\x7f]/g, '');+ const pieces = raw.includes('\x1b') ? ansi(raw, st) : null, plain = pieces ? pieces.map(p => p[1]).join('') : raw;+ const line = {n: ++n, h: ''};+ if (levels) {+ if (/^Traceback \(most recent call last\)/.test(plain)) trace = true;+ else if (trace && !/^\s/.test(plain)) { trace = false; line.c = 'le'; }+ if (trace) line.c = 'le';+ }+ if (pieces) line.h = pieces.map(([c, s]) => c ? `<span class="${c}">${esc(s)}</span>` : esc(s)).join('');+ else if (levels) {+ const t = LOG_TIME.exec(plain), rest = t ? plain.slice(t[0].length) : plain, l = LOG_LEVEL.exec(rest.slice(0, 80));+ if (t) line.h = `<span class="cc">${esc(t[0])}</span>`;+ if (l) {+ const lv = l[1] ? 'e' : l[2] ? 'w' : l[3] ? 'i' : 'd';+ line.c ||= {e: 'le', w: 'lw', d: 'lg'}[lv];+ line.h += esc(rest.slice(0, l.index)) + `<span class="lv${lv}">${l[0]}</span>` + esc(rest.slice(l.index + l[0].length));+ } else line.h += esc(rest);+ } else line.h = esc(plain);+ return line;+ };+}+function ansi(s, st) {+ const out = [], re = /\x1b\[([\d;]*)([A-Za-z])|\x1b[()][A-Z0-9]|\x1b\][^\x07\x1b]*(?:\x07|\x1b\\)?|\x1b./g;+ const cls = () => [st.fg && 'a' + st.fg, st.bold && 'ab', st.dim && 'ad', st.it && 'ai', st.un && 'au'].filter(Boolean).join(' ');+ let at = 0, m;+ while ((m = re.exec(s))) {+ if (m.index > at) out.push([cls(), s.slice(at, m.index)]);+ at = m.index + m[0].length;+ if (m[2] !== 'm') continue;+ const codes = (m[1] || '0').split(';').map(Number);+ for (let k = 0; k < codes.length; k++) {+ const c = codes[k];+ if (c === 0) { st.fg = 0; st.bold = st.dim = st.it = st.un = false; }+ else if (c === 1) st.bold = true; else if (c === 2) st.dim = true; else if (c === 3) st.it = true; else if (c === 4) st.un = true;+ else if (c === 22) st.bold = st.dim = false; else if (c === 23) st.it = false; else if (c === 24) st.un = false;+ else if ((c >= 30 && c <= 37) || (c >= 90 && c <= 97)) st.fg = c;+ else if (c === 39) st.fg = 0;+ else if (c === 38 || c === 48) {+ if (codes[k + 1] === 5) { const n = codes[k + 2]; if (c === 38 && n < 16) st.fg = n < 8 ? 30 + n : 82 + n; k += 2; }+ else if (codes[k + 1] === 2) k += 4;+ }+ }+ }+ if (at < s.length) out.push([cls(), s.slice(at)]);+ return out;+}++// JSON laid out two spaces to a level, token by token: numbers stay exactly as written, however long.+function prettyJson(text) {+ const t = text.match(/"(?:[^"\\]|\\.)*"|[{}[\],:]|[^\s{}[\],:"]+/g) || [];+ let out = '', depth = 0;+ const nl = () => '\n' + ' '.repeat(depth);+ for (let i = 0; i < t.length; i++) {+ const x = t[i];+ if (x === '{' || x === '[') {+ if (t[i + 1] === (x === '{' ? '}' : ']')) { out += x + t[++i]; continue; }+ depth++; out += x + nl();+ } else if (x === '}' || x === ']') { depth--; out += nl() + x; }+ else if (x === ',') out += ',' + nl();+ else if (x === ':') out += ': ';+ else out += x;+ }+ return out;+}++// CSV and TSV: quoted cells may hold the separator, quotes doubled, and line breaks.+function parseTable(text, sep, most = Infinity) {+ const rows = [];+ let row = [], cell = '', q = false;+ for (let i = 0; i < text.length; i++) {+ const ch = text[i];+ if (q) { if (ch !== '"') cell += ch; else if (text[i + 1] === '"') { cell += '"'; i++; } else q = false; }+ else if (ch === '"' && !cell) q = true;+ else if (ch === sep) { row.push(cell); cell = ''; }+ else if (ch === '\n') { row.push(cell); rows.push(row); row = []; cell = ''; if (rows.length >= most) return rows; }+ else if (ch !== '\r') cell += ch;+ }+ if (cell || row.length) { row.push(cell); rows.push(row); }+ return rows;+}+const NUMBER = /^[-+]?(?:\d[\d,_]*\.?\d*|\.\d+)(?:[eE][-+]?\d+)?%?$/;+// A table is a grid whose columns are as wide as the widest cell in its first rows, so its rows can go in a batch+// at a time without the browser measuring every row again, and a column of numbers is set right.+function tableShape(rows, total) {+ // each column: its widest text, its padding and its border+ const cols = Math.max(...rows.slice(0, 50).map(r => r.length), 1), widths = [`calc(${String(total).length}ch + 26px)`], right = [];+ for (let j = 0; j < cols; j++) {+ let w = 2, all = 0, nums = 0;+ for (const r of rows.slice(0, 500)) w = Math.max(w, (r[j] || '').length);+ for (const r of rows.slice(1, 200)) { const v = (r[j] || '').trim(); if (v) { all++; if (NUMBER.test(v)) nums++; } }+ widths.push(`calc(${Math.min(w, 48)}ch + 26px)`);+ right.push(all > 0 && nums / all >= 0.8);+ }+ return {right, cols: widths.join(' ')};+}+const rowHtml = (r, n, t, cls = 'gr') => `<div class="${cls}"><span>${n}</span>${t.right.map((right, j) => `<span${right ? ' class="r"' : ''}>${esc(r[j] ?? '')}</span>`).join('')}</div>`;+const tableHtml = (rows, t) => `<div class="grid" style="--cols:${t.cols}">${rowHtml(rows[0] || [], '', t, 'gr gh')}${rows.slice(1).map((r, i) => rowHtml(r, i + 1, t)).join('')}</div>`;++// ---- Markdown: blocks, then the text inside them. Raw HTML in it shows as text, and only web and mail links link.+const LIST = /^(\s*)([-*+]|\d{1,9}[.)])[ \t]+(.*)$/;+const TABLE_RULE = /^\s*\|?\s*:?-+:?\s*(?:\|\s*:?-+:?\s*)*\|?\s*$/;+const indentOf = l => l.length - l.trimStart().length;+const markdown = text => mdBlocks(text.replace(/\r\n?/g, '\n').split('\n'));+function mdBlocks(lines) {+ let out = '', para = [], i = 0, m;+ const flush = () => { if (para.length) out += `<p>${mdInline(para.join('\n'))}</p>`; para = []; };+ while (i < lines.length) {+ const line = lines[i];+ if (!line.trim()) { flush(); i++; continue; }+ if ((m = /^\s{0,3}(`{3,}|~{3,})\s*([^`\s]*)/.exec(line))) {+ flush();+ const body = [];+ for (i++; i < lines.length && !lines[i].trim().startsWith(m[1]); i++) body.push(lines[i]);+ i++;+ out += `<pre class="code">${blockHtml(body.join('\n'), langKind(m[2]))}</pre>`;+ } else if ((m = /^\s{0,3}(#{1,6})\s+(.*?)(?:\s+#+)?\s*$/.exec(line))) {+ flush(); out += `<h${m[1].length}>${mdInline(m[2])}</h${m[1].length}>`; i++;+ } else if (para.length && /^\s{0,3}(=+|-+)\s*$/.test(line)) {+ const h = line.trim()[0] === '=' ? 1 : 2;+ out += `<h${h}>${mdInline(para.join('\n'))}</h${h}>`; para = []; i++;+ } else if (/^\s{0,3}([-*_])(\s*\1){2,}\s*$/.test(line)) {+ flush(); out += '<hr>'; i++;+ } else if (/^\s{0,3}>/.test(line)) {+ flush();+ const body = [];+ while (i < lines.length && /^\s{0,3}>/.test(lines[i])) body.push(lines[i++].replace(/^\s{0,3}> ?/, ''));+ out += `<blockquote>${mdBlocks(body)}</blockquote>`;+ } else if ((m = LIST.exec(line))) {+ flush();+ const indent = m[1].length, ordered = /\d/.test(m[2]), items = [];+ while (i < lines.length) {+ const l = lines[i], it = LIST.exec(l);+ if (it && indentOf(l) <= indent + 1) {+ if (/\d/.test(it[2]) !== ordered) break;+ items.push([it[3]]); i++;+ } else if (!l.trim()) {+ const next = lines[i + 1];+ if (next === undefined || !(indentOf(next) > indent || (LIST.test(next) && indentOf(next) <= indent + 1))) break;+ items[items.length - 1].push(''); i++;+ } else if (indentOf(l) > indent) { items[items.length - 1].push(l.slice(Math.min(indentOf(l), indent + 2))); i++; }+ else if (items[items.length - 1].at(-1) !== '' && !/^\s{0,3}(#|>|```|~~~)/.test(l)) { items[items.length - 1].push(l.trim()); i++; }+ else break;+ }+ const tag = ordered ? 'ol' : 'ul', start = ordered && parseInt(m[2]) !== 1 ? ` start="${parseInt(m[2])}"` : '';+ out += `<${tag}${start}>${items.map(([first, ...rest]) => {+ const task = /^\[([ xX])\]\s+/.exec(first);+ const head = task ? `<span class="task${task[1] === ' ' ? '' : ' done'}"></span>${mdInline(first.slice(task[0].length))}` : mdInline(first);+ return `<li${task ? ' class="tasked"' : ''}>${head}${rest.length ? mdBlocks(rest) : ''}</li>`;+ }).join('')}</${tag}>`;+ } else if (line.includes('|') && i + 1 < lines.length && lines[i + 1].includes('|') && TABLE_RULE.test(lines[i + 1])) {+ flush();+ const cells = l => l.trim().replace(/^\|/, '').replace(/(?<!\\)\|$/, '').split(/(?<!\\)\|/).map(c => c.trim().replace(/\\\|/g, '|'));+ const head = cells(line), align = cells(lines[i + 1]).map(c => c.endsWith(':') ? (c.startsWith(':') ? 'center' : 'right') : '');+ const td = (tag, c, j) => `<${tag}${align[j] ? ` style="text-align:${align[j]}"` : ''}>${mdInline(c)}</${tag}>`;+ out += `<table><thead><tr>${head.map((c, j) => td('th', c, j)).join('')}</tr></thead><tbody>`;+ for (i += 2; i < lines.length && lines[i].includes('|') && lines[i].trim(); i++) {+ const r = cells(lines[i]);+ out += `<tr>${head.map((_, j) => td('td', r[j] || '', j)).join('')}</tr>`;+ }+ out += '</tbody></table>';+ } else { para.push(line.trim()); i++; }+ }+ flush();+ return out;+}+function mdInline(text) {+ const kept = [], keep = h => `\uE000${kept.push(h) - 1}\uE001`;+ let s = esc(text);+ s = s.replace(/(`+)([^`]|[^`][\s\S]*?[^`])\1(?!`)/g, (_, q, c) => keep(`<code>${c.trim()}</code>`));+ // a link's address ends at a space, a bracket or a quote. Escaped, a quote is " and an ampersand &.+ s = s.replace(/!\[([^\]]*)\]\(((?:[^)\s&]|&)+)(?:\s+".*?")?\)/g, (_, alt, url) => keep(mdLink(url, `🖼 ${alt || 'picture'}`)));+ s = s.replace(/\[([^\]]+)\]\(((?:[^)\s&]|&)+)(?:\s+".*?")?\)/g, (_, label, url) => keep(mdLink(url, emphasis(label))));+ s = s.replace(/<((?:https?:\/\/|mailto:)[^\s&]+)>|\bhttps?:\/\/(?:[^\s<&]|&)*[^\s<.,;:!?)\]'"&]/g, (u, inner) => keep(mdLink(inner || u, inner || u)));+ s = emphasis(s).replace(/ {2,}\n|\\\n/g, '<br>');+ return s.replace(/\uE000(\d+)\uE001/g, (_, n) => kept[n] ?? '');+}+const emphasis = s => s.replace(/\*\*(?=\S)([\s\S]*?\S)\*\*|__(?=\S)([\s\S]*?\S)__/g, (_, a, b) => `<strong>${a ?? b}</strong>`)+ .replace(/(^|[^\w*])\*(?=\S)([^*]*?\S)\*(?![\w*])/g, '$1<em>$2</em>').replace(/(^|[^\w])_(?=\S)([^_]*?\S)_(?!\w)/g, '$1<em>$2</em>')+ .replace(/~~(?=\S)([\s\S]*?\S)~~/g, '<del>$1</del>');+// url is already escaped, so it cannot leave its attribute+const mdLink = (url, label) => /^(?:https?:\/\/|mailto:)/i.test(url)+ ? `<a href="${url}" target="_blank" rel="noopener noreferrer">${label}</a>` : `<span class="nolink" title="${url}">${label}</span>`;++// ---- the card under a message+const TEXTS = new Map(); // attachment id -> the attachment, for the viewer+const PREVIEW_LINES = 8;+function snipHtml(a, big) {+ TEXTS.delete(a.id); TEXTS.set(a.id, a);+ if (TEXTS.size > 2000) TEXTS.delete(TEXTS.keys().next().value);+ const {preview, lines, more} = a.text, f = fileKind(a.name, preview);+ let body;+ if (f.kind === 'markdown') body = `<div class="md">${markdown(preview)}</div>`;+ else if (f.kind === 'csv' || f.kind === 'tsv') {+ const rows = parseTable(preview, f.kind === 'csv' ? ',' : '\t', PREVIEW_LINES);+ body = tableHtml(rows, tableShape(rows, rows.length));+ } else body = `<div class="code">${textLines(preview, f.kind).slice(0, PREVIEW_LINES).map(lineHtml).join('')}</div>`;+ const cut = more || preview.split('\n').length > PREVIEW_LINES;+ return html`<div class="snip${big ? ' big' : ''}" data-text="${a.id}" tabindex="0" title="Open ${a.name}"><div class="sh"><span class="fx ${FAMILY[f.kind] || 'code'}">${f.badge}</span><b>${a.name}</b><span class="k">${f.label}${lines ? ` · ${plural(lines, 'line')}` : ''} · ${bytes(a.size)}</span><button class="dl" data-file="${a.id}" data-name="${a.name}" title="Download ${a.name}">${new Safe(DOWNLOAD_ICON)}</button></div><div class="sb${cut ? ' more' : ''}">${new Safe(body)}</div></div>`;+}+const DOWNLOAD_ICON = '<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 4v11M7 10l5 5 5-5M5 20h14"/></svg>';++// ---- the viewer+const FORMATTED = {markdown: 'Formatted', csv: 'Table', tsv: 'Table', json: 'Formatted'};+// Runs fn as soon as the page has answered what is waiting. A timer can wait far longer than asked, as when the+// browser is busy; a message does not.+const soon = (() => {+ const ch = new MessageChannel(), queue = [];+ ch.port1.onmessage = () => queue.shift()();+ return fn => { queue.push(fn); ch.port2.postMessage(0); };+})();+const TABLE_ROWS = 20000; // rows of a CSV file shown as a table. All of them show as its source.+const BUILD_LINES = 100000; // a file with more lines builds each part of the page only when it comes into view+const PART = 500; // lines formatted, and built into the page, at a time+async function openText(id) {+ const a = TEXTS.get(id);+ if (!a) return;+ const f = fileKind(a.name, a.text.preview);+ const box = document.createElement('div');+ box.className = 'tv';+ box.innerHTML = html`<div class="tvh"><span class="fx ${FAMILY[f.kind] || 'code'}">${f.badge}</span><b title="${a.name}">${a.name}</b><span class="k">${f.label} · ${bytes(a.size)}</span><span class="grow"></span>${FORMATTED[f.kind] ? html`<nav class="seg"><button data-mode="formatted" class="on">${FORMATTED[f.kind]}</button><button data-mode="source">Source</button></nav>` : ''}<button class="btn" data-wrap title="Wrap long lines">Wrap</button><button class="btn" data-file="${a.id}" data-name="${a.name}">Download</button><button class="x" data-close title="Close (Esc)">×</button></div><div class="tvb"><div class="loading">Loading ${a.name}</div></div>`.s;+ overlay.open(box);+ const body = $('.tvb', box);+ let r;+ try { r = await api('view_text', {id}); } catch (e) { body.innerHTML = html`<div class="loading">${e.message}</div>`.s; return; }+ if (!box.isConnected) return;+ const text = r.text.replace(/\r\n/g, '\n');+ let mode = FORMATTED[f.kind] ? 'formatted' : 'source', wrap = ['text', 'log', 'markdown'].includes(f.kind), draws = 0;+ if (f.kind === 'json') { try { JSON.parse(text); } catch (_) { mode = 'source'; $('.seg', box).remove(); } }+ const count = (text.match(/\n/g) || []).length + (text && !text.endsWith('\n') ? 1 : 0);+ $('.tvh .k', box).textContent = `${f.label} · ${r.cut ? 'first ' : ''}${plural(count, 'line')} · ${bytes(a.size)}`;+ if (r.cut) box.insertAdjacentHTML('beforeend', html`<div class="tvn">Showing the first ${bytes(r.shown)} of ${bytes(r.size)}. Download the file for all of it.</div>`.s);+ const draw = () => {+ const now = ++draws, live = () => now === draws && box.isConnected;+ for (const b of $$('[data-mode]', box)) b.classList.toggle('on', b.dataset.mode === mode);+ $('[data-wrap]', box).classList.toggle('on', wrap);+ $('[data-wrap]', box).hidden = mode === 'formatted' && f.kind !== 'json';+ body.classList.toggle('doc', mode === 'formatted' && f.kind === 'markdown');+ body.scrollTop = 0;+ if (mode === 'formatted' && f.kind === 'markdown') { // a long document goes in parts, each laid out only when in view+ const page = document.createElement('template'), md = document.createElement('div');+ page.innerHTML = markdown(text);+ md.className = 'md';+ const blocks = [...page.content.children];+ if (blocks.length <= 200) md.append(...blocks);+ else for (let i = 0; i < blocks.length; i += 100) md.appendChild(document.createElement('div')).append(...blocks.slice(i, i + 100));+ for (const part of md.children) if (blocks.length > 200) part.className = 'chunk';+ return body.replaceChildren(md);+ }+ if (mode === 'formatted' && (f.kind === 'csv' || f.kind === 'tsv')) {+ const rows = parseTable(text, f.kind === 'csv' ? ',' : '\t', TABLE_ROWS + 2), over = rows.length > TABLE_ROWS + 1;+ if (over) rows.length = TABLE_ROWS + 1;+ const t = tableShape(rows, rows.length);+ body.innerHTML = tableHtml(rows.slice(0, 1), t) + (over ? html`<div class="loading">The table shows the first ${num(TABLE_ROWS)} rows. Source shows all of them.</div>`.s : '');+ const grid = $('.grid', body);+ // PART rows at a time, for 30 ms at a time, so the first show at once and the page keeps answering+ let at = 1;+ const add = () => {+ if (!live()) return;+ const until = performance.now() + 30;+ for (; at < rows.length && performance.now() < until; at += PART) {+ grid.insertAdjacentHTML('beforeend', `<div class="chunk">${rows.slice(at, at + PART).map((row, i) => rowHtml(row, at + i, t)).join('')}</div>`);+ }+ if (at < rows.length) soon(add);+ };+ return add();+ }+ const shown = mode === 'formatted' && f.kind === 'json' ? prettyJson(text) : text;+ const total = shown === text ? count : (shown.match(/\n/g) || []).length + 1;+ const code = document.createElement('div');+ code.className = `code nums${f.kind === 'diff' ? ' diff' : ''}${wrap ? ' wrap' : ''}`;+ code.style.setProperty('--w', String(total).length + 'ch');+ // The lines are formatted PART at a time, in the background. Each part becomes page elements when it comes near+ // the screen, and every one does when the file has at most BUILD_LINES lines, so the browser's find sees them+ // all. Until then each holds the height its lines will take, so the scroll bar is right from the start.+ const parts = Array.from({length: Math.max(1, Math.ceil(total / PART))}, (_, i) => {+ const part = document.createElement('div'), height = Math.min(PART, total - i * PART) * 20 + 'px';+ part.className = 'chunk';+ part.i = i;+ part.style.height = height;+ part.style.setProperty('contain-intrinsic-block-size', 'auto ' + height);+ return part;+ });+ code.append(...parts);+ body.replaceChildren(code);+ const made = [], wanted = new Set(), every = total <= BUILD_LINES;+ const build = i => {+ const part = parts[i];+ if (part.built || made[i] == null) return;+ part.built = true;+ part.innerHTML = made[i];+ part.style.height = '';+ near.unobserve(part);+ };+ const near = new IntersectionObserver(seen => {+ for (const e of seen) if (e.isIntersecting) { wanted.add(e.target.i); build(e.target.i); }+ }, {root: body, rootMargin: '1500px 0px'});+ parts.forEach(part => near.observe(part));+ const batches = lineBatches(shown, f.kind, PART);+ const step = () => { // formats for 30 ms or builds one part, then lets the page answer+ if (!live()) return near.disconnect();+ const until = performance.now() + 30;+ do {+ const next = batches.next();+ if (next.done) return every && near.disconnect();+ const i = made.push(next.value.map(lineHtml).join('')) - 1;+ if (every || wanted.has(i)) { build(i); break; }+ } while (performance.now() < until);+ soon(step);+ };+ step();+ };+ box.addEventListener('click', e => {+ const t = e.target.closest('[data-mode], [data-wrap], [data-close]');+ if (!t) return;+ if (t.dataset.close != null) return overlay.close();+ if (t.dataset.wrap != null) { wrap = !wrap; const c = $('.code', body); if (c) { c.classList.toggle('wrap', wrap); t.classList.toggle('on', wrap); } return; }+ if (t.dataset.mode !== mode) { mode = t.dataset.mode; draw(); }+ });+ draw();+}++// One message. prev decides whether it continues the message before (same sender, within five minutes).+function messageHtml(m, prev, o = {}) {+ if (m.kind !== 'message') return systemHtml(m);+ const u = user(m.from);+ const cont = !o.tree && !o.flat && prev && prev.kind === 'message' && prev.from === m.from && prev.chat === m.chat && !m.parent &&+ ms(m.at) - ms(prev.at) < 300000 && dayOf(m.at) === dayOf(prev.at);+ const as = S.route.as;+ const mentionsMe = as && m.from !== as && (m.mentions || []).includes(as);+ const text = m.text || '';+ const long = !o.whole && (text.length > 1500 || text.split('\n').length > 18);+ const reactions = Object.entries(m.reactions || {});+ const depth = o.tree ? Math.min(m.depth || 0, 12) : 0;+ return html`<div class="msg${cont ? ' cont' : ''}${mentionsMe ? ' mentions-me' : ''}" data-seq="${m.seq}" data-chat="${m.chat}" data-id="${m.id}" data-time="${clock(m.at)}"${o.tree ? html` style="--d:${depth}"` : ''}>+${avatar(m.from)}<div class="body"><div class="head"><span class="name" data-person="${m.from}">${u.name}</span>${u.name !== m.from ? html`<span class="uid">${m.from}</span>` : ''}<time title="${full(m.at)}">${o.tree || o.showChat ? ago(m.at) : clock(m.at)}</time>${o.showChat ? html`<span class="chat" data-open="${m.chat}" data-at="${m.seq}">in ${chatTitle(m.chat)}</span>` : ''}${m.edited_at ? html`<span class="edited" title="Edited ${full(m.edited_at)}">(edited)</span>` : ''}</div>+${m.parent && !o.tree ? html`<div class="quote" data-jump="${m.parent.seq}" data-chat="${m.chat}" title="Go to the message it answers"><b>${name(m.parent.from)}</b><span>${m.parent.deleted ? 'deleted message' : m.parent.text || (m.parent.files ? 'a file' : '')}</span></div>` : ''}+${m.deleted ? html`<div class="text deleted">This message was deleted.</div>` : html`<div class="text${long ? ' long' : ''}">${rich(text, o.mark)}</div>${long ? html`<span class="more" data-more>Show all</span>` : ''}`}+${filesHtml(m.attachments)}+${reactions.length ? html`<div class="reacts">${reactions.map(([e, us]) => html`<span class="react" data-tip="${us.map(name).join(', ')}">${EMOJI[e] || e} ${us.length}</span>`)}</div>` : ''}+${m.replies && !o.tree ? html`<div class="foot"><span class="replies" data-thread="${m.id}">${plural(m.replies, 'reply', 'replies')}</span></div>` : ''}+${o.below ? html`<div class="foot"><span class="toggle" data-fold title="Fold or unfold everything below">▾ ${plural(o.below, 'reply', 'replies')} below</span></div>` : ''}+</div><div class="actions">${o.tree ? '' : html`<button data-thread="${m.id}">Thread</button>`}<button data-knew="${m.id}" title="How far ${u.name} had marked messages read in each chat when sending this">What had ${first(m.from)} read?</button><button data-receipts="${m.id}">Read by</button><button data-copy="${m.id}" title="Copy the message id">${m.id}</button></div></div>`;+}++function systemHtml(e) {+ const who = html`<b>${name(e.from)}</b>`;+ const people = us => {+ const names = (us || []).map(name);+ return names.length > 8 ? html`<b>${names.slice(0, 8).join(', ')}</b> and ${names.length - 8} more` : html`<b>${names.join(', ')}</b>`;+ };+ const what = {+ created: () => html`${who} made this chat${e.name ? html` as <b>#${e.name}</b>` : ''} with ${people(e.members)}${e.public ? ', public' : ''}`,+ added: () => e.users && e.users.length === 1 && e.users[0] === e.from ? html`${who} joined` : html`${who} added ${people(e.users)}`,+ removed: () => e.users && e.users.length === 1 && e.users[0] === e.from ? html`${who} left` : html`${who} removed ${people(e.users)}`,+ renamed: () => html`${who} renamed it <b>#${e.name}</b>`,+ managers: () => html`${who} ${e.added ? html`made ${people(e.added)} ${e.added.length === 1 ? 'a manager' : 'managers'}` : ''}${e.added && e.removed ? ' and ' : ''}${e.removed ? html`removed ${people(e.removed)} as ${e.removed.length === 1 ? 'a manager' : 'managers'}` : ''}`,+ described: () => e.description ? html`${who} changed the description to <b>${e.description}</b>` : html`${who} removed the description`,+ visibility: () => html`${who} made it ${e.public ? 'public' : 'private'}`,+ archived: () => html`${who} archived it`,+ unarchived: () => html`${who} unarchived it`,+ }[e.kind];+ return html`<div class="sys" data-seq="${e.seq}"><span>${what ? what() : e.kind}</span><span class="stamp" title="${full(e.at)}">${clock(e.at)}</span></div>`;+}++// ---------------------------------------------------------------------- one chat++class ChatView {+ constructor(main, chatId, o) {+ this.id = chatId; this.o = o; this.msgs = []; this.first = this.last = false; this.positions = {}; this.fresh = 0;+ this.main = main;+ const c = S.chats.get(chatId);+ main.innerHTML = html`<div class="mhead"></div><div class="banners"></div><div class="scroll"><div class="loading top" hidden>Loading earlier messages</div><div class="msgs"></div><div class="loading bottom" hidden>Loading</div></div><button class="newpill" hidden></button>`;+ this.head = $('.mhead', main); this.scroll = $('.scroll', main); this.box = $('.msgs', main); this.pill = $('.newpill', main);+ this.renderHead();+ if (!c) { this.box.innerHTML = html`<div class="empty">There is no chat ${chatId}.</div>`; return; }+ this.scroll.addEventListener('scroll', () => this.onScroll(), {passive: true});+ this.pill.addEventListener('click', () => this.toBottom(true));+ this.load();+ }+ renderHead() {+ const c = S.chats.get(this.id), r = S.route;+ if (!c) return;+ const kind = c.kind === 'channel' ? (c.public ? 'public' : 'private') : c.kind;+ this.head.innerHTML = html`<span class="ico ${c.kind}">${KIND[c.kind]}</span><h1 title="${title(c, null)}">${c.name ? '#' + c.name : title(c, null)}</h1><span class="pill ${kind}">${kind}</span>+${c.archived ? html`<span class="pill archived">archived</span>` : ''}+<span class="sub"><span class="link" data-members="${c.id}">${plural(c.members.length, 'member')}</span> · ${plural(c.messages, 'message')} · made by ${name(c.created_by)} ${ago(c.created_at)}</span>+<div class="grow"></div><button class="btn" data-goto="board">Open as a board</button>+${c.description ? html`<div class="about" title="${c.description}">${c.description}</div>` : ''}`;+ const banners = $('.banners', this.main);+ const notIn = r.as && !c.members.includes(r.as);+ banners.innerHTML = notIn ? html`<div class="banner">${name(r.as)} is not in this chat, so ${first(r.as)} cannot see any of it.</div>` : '';+ }+ async load(params) {+ const o = this.o, as = S.route.as, c = S.chats.get(this.id);+ const pos = as && S.as && S.as.positions[this.id];+ const start = o.jump ? {around: o.jump} : o.had === 0 ? {after: 0} : pos != null && c.last && pos < c.last.seq ? {around: pos + 1} : {};+ const r = await api('view_messages', params || {chat: this.id, limit: 150, ...start});+ if (this.dead) return;+ this.msgs = r.messages; this.first = r.first; this.last = r.last; this.positions = r.positions;+ this.draw();+ const d = this.divider();+ if (o.had === 0 && !params) this.scroll.scrollTop = 0;+ else if (o.jump && !params) this.reveal(o.jump);+ else if (d && d.cls === 'now' && !params) this.box.querySelector('.divider')?.scrollIntoView({block: 'center'});+ else this.toBottom();+ }+ divider() {+ const r = S.route;+ if (this.o.had != null && this.o.knew) {+ const who = S.knew && S.knew.id === this.o.knew ? S.knew : null;+ const what = this.o.had ? 'had read up to here' : 'had read nothing here';+ return {at: this.o.had, cls: 'then', label: who ? `${name(who.user)} ${what} when sending at ${clockS(who.at)}` : `${what} at that moment`};+ }+ if (r.as && S.as && S.as.positions[this.id] != null) {+ const pos = S.as.positions[this.id];+ const lastSeq = this.msgs.length ? this.msgs[this.msgs.length - 1].seq : 0;+ if (pos < lastSeq) return {at: pos, cls: 'now', label: `${name(r.as)} has read up to here`};+ }+ return null;+ }+ rows(list, prev) {+ const d = this.divider();+ let out = '', before = prev ? prev.seq : this.first ? 0 : null; // the last message drawn above+ for (const m of list) {+ if (!prev || dayOf(prev.at) !== dayOf(m.at)) out += html`<div class="daysep"><span>${dayName(m.at)}</span></div>`.s;+ if (d && m.kind === 'message' && before != null && before <= d.at && m.seq > d.at) out += html`<div class="divider ${d.cls}">${d.label}</div>`.s;+ if (m.kind === 'message') before = m.seq;+ out += messageHtml(m, prev).s;+ prev = m;+ }+ return out;+ }+ draw() {+ const top = this.scroll.scrollTop;+ this.box.innerHTML = this.rows(this.msgs, null) || (this.last && this.first ? '<div class="empty">No messages yet.</div>' : '');+ this.scroll.scrollTop = top;+ $('.loading.top', this.main).hidden = this.first;+ }+ atBottom() { return this.scroll.scrollHeight - this.scroll.scrollTop - this.scroll.clientHeight < 80; }+ toBottom(fetch) {+ if (fetch && !this.last) { this.o.jump = null; return this.load({chat: this.id, limit: 150}); }+ this.scroll.scrollTop = this.scroll.scrollHeight;+ this.fresh = 0; this.pill.hidden = true;+ }+ reveal(seq) {+ const el = this.box.querySelector(`.msg[data-seq="${seq}"]`) || [...this.box.querySelectorAll('.msg')].find(e => +e.dataset.seq >= seq);+ if (!el) return;+ el.scrollIntoView({block: 'center'});+ el.classList.remove('flash'); void el.offsetWidth; el.classList.add('flash');+ }+ jump(seq) {+ if (this.msgs.some(m => m.seq === seq)) return this.reveal(seq);+ this.o.jump = seq;+ this.load({chat: this.id, around: seq, limit: 150}).then(() => this.reveal(seq));+ }+ async onScroll() {+ if (this.busy) return;+ const s = this.scroll;+ if (s.scrollTop < 500 && !this.first && this.msgs.length) {+ this.busy = true;+ const r = await api('view_messages', {chat: this.id, before: this.msgs.find(m => m.kind === 'message')?.seq ?? this.msgs[0].seq, limit: 200});+ if (this.dead) return;+ const h = s.scrollHeight, t = s.scrollTop;+ const have = new Set(this.msgs.map(m => m.seq));+ this.msgs = [...r.messages.filter(m => !have.has(m.seq)), ...this.msgs];+ this.first = r.first;+ if (this.msgs.length > 700) { this.msgs = this.msgs.slice(0, 600); this.last = false; }+ this.draw();+ s.scrollTop = t + (s.scrollHeight - h);+ this.busy = false;+ } else if (s.scrollHeight - s.scrollTop - s.clientHeight < 500 && !this.last && this.msgs.length) {+ this.busy = true;+ const r = await api('view_messages', {chat: this.id, after: this.msgs[this.msgs.length - 1].seq, limit: 200});+ if (this.dead) return;+ const have = new Set(this.msgs.map(m => m.seq));+ this.msgs = [...this.msgs, ...r.messages.filter(m => !have.has(m.seq))];+ this.last = r.last;+ if (this.msgs.length > 700) {+ const h = s.scrollHeight, t = s.scrollTop;+ this.msgs = this.msgs.slice(-600); this.first = false;+ this.draw();+ s.scrollTop = t - (h - s.scrollHeight);+ } else this.draw();+ this.busy = false;+ }+ if (this.atBottom()) { this.fresh = 0; this.pill.hidden = true; }+ }+ onEvents(ev) {+ if (ev.chats.has(this.id)) this.renderHead();+ const mine = ev.lines.filter(m => m.chat === this.id);+ const changed = ev.changes.filter(m => m.chat === this.id);+ for (const m of changed) {+ const i = this.msgs.findIndex(x => x.seq === m.seq);+ if (i < 0) continue;+ this.msgs[i] = m;+ const el = this.box.querySelector(`.msg[data-seq="${m.seq}"]`);+ if (el) el.outerHTML = messageHtml(m, this.msgs[i - 1]).s;+ }+ for (const e of ev.reads) if (e.chat === this.id) this.positions[e.user] = Math.max(this.positions[e.user] || 0, e.upto);+ if (!mine.length || !this.last) { if (mine.length) this.pillCount(mine.length); return; }+ const have = new Set(this.msgs.map(m => m.seq));+ const add = mine.filter(m => !have.has(m.seq));+ if (!add.length) return;+ const stick = this.atBottom();+ const parents = new Set(add.map(m => m.parent && m.parent.seq).filter(Boolean));+ const prev = this.msgs[this.msgs.length - 1];+ this.msgs.push(...add);+ for (const m of this.msgs) if (parents.has(m.seq)) m.replies = (m.replies || 0) + add.filter(a => a.parent && a.parent.seq === m.seq).length;+ if (parents.size || this.divider()) this.draw();+ else {+ this.box.querySelector('.empty')?.remove();+ this.box.insertAdjacentHTML('beforeend', this.rows(add, prev));+ }+ if (stick) {+ if (this.msgs.length > 700) { this.msgs = this.msgs.slice(-600); this.first = false; this.draw(); }+ this.toBottom();+ } else this.pillCount(add.length);+ }+ pillCount(n) {+ this.fresh += n;+ this.pill.hidden = false;+ this.pill.textContent = `${plural(this.fresh, 'new message')} ↓`;+ }+ onAs() { this.renderHead(); this.draw(); }+ destroy() { this.dead = true; }+}++// ---------------------------------------------------------------------- lists of messages from many chats++class ListView {+ constructor(main, heading) {+ this.main = main; this.msgs = [];+ main.innerHTML = html`<div class="mhead">${heading}</div><div class="scroll"><div class="msgs results"></div><div class="loading" hidden>Loading</div></div>`;+ this.scroll = $('.scroll', main); this.box = $('.msgs', main); this.wait = $('.loading', main);+ }+ draw(o = {}) {+ let prev = null, out = '';+ for (const m of this.msgs) { out += messageHtml(m, prev, {showChat: true, flat: this.flat, ...o}).s; prev = m; }+ this.box.innerHTML = out || html`<div class="empty">${this.none || 'Nothing here.'}</div>`.s;+ }+ destroy() { this.dead = true; }+}++class ActivityView extends ListView {+ constructor(main) {+ const as = S.route.as;+ super(main, html`<span class="ico">≋</span><h1>All activity</h1><span class="sub">${as ? `in ${name(as)}'s chats, ` : 'every chat, '}newest at the bottom, live</span>`);+ this.none = 'No messages yet. New ones appear here as they are written.';+ this.load();+ }+ async load() {+ this.wait.hidden = false;+ const r = await api('view_search', {limit: 150, as: S.route.as || undefined});+ if (this.dead) return;+ this.wait.hidden = true;+ this.msgs = r.messages.reverse();+ this.draw();+ this.scroll.scrollTop = this.scroll.scrollHeight;+ }+ onEvents(ev) {+ let add = ev.lines.filter(m => m.kind === 'message');+ if (S.route.as) add = add.filter(m => (S.chats.get(m.chat)?.members || []).includes(S.route.as));+ if (!add.length) return;+ const stick = this.scroll.scrollHeight - this.scroll.scrollTop - this.scroll.clientHeight < 80;+ const prev = this.msgs[this.msgs.length - 1];+ this.msgs.push(...add);+ if (this.msgs.length > 400) { this.msgs = this.msgs.slice(-300); this.draw(); }+ else {+ this.box.querySelector('.empty')?.remove();+ let p = prev, out = '';+ for (const m of add) { out += messageHtml(m, p, {showChat: true}).s; p = m; }+ this.box.insertAdjacentHTML('beforeend', out);+ }+ if (stick) this.scroll.scrollTop = this.scroll.scrollHeight;+ }+}++class UnreadView extends ListView {+ constructor(main) {+ const as = S.route.as;+ super(main, html`<span class="ico">✉</span><h1>What ${name(as)} would read now</h1><span class="sub">exactly what <code>swarmchat read</code> would hand ${first(as)}, oldest first. Nothing is marked read.</span>`);+ this.none = `Nothing unread: ${first(as)} has read everything.`;+ this.flat = true;+ this.draw2();+ }+ draw2() {+ this.msgs = S.as ? S.as.unread : [];+ this.draw();+ if (S.as && S.as.unreadLeft > this.msgs.length) this.box.insertAdjacentHTML('beforeend', html`<div class="note">${num(S.as.unreadLeft - this.msgs.length)} more after these.</div>`.s);+ }+ onAs() { this.draw2(); }+}++// Search words, and from:, in:, mentions:, has:file, after: and before:.+function parseQuery(q) {+ const p = {}, words = [];+ for (const m of q.match(/(\w+):"[^"]*"|(\w+):\S+|"[^"]*"|\S+/g) || []) {+ const f = m.match(/^(from|in|mentions|has|after|before):"?([^"]*)"?$/i);+ if (!f) { words.push(m.replace(/^"|"$/g, '')); continue; }+ const key = f[1].toLowerCase(), v = f[2];+ if (key === 'from') (p.from ||= []).push(findUser(v) || v.replace(/^@/, '').toLowerCase());+ else if (key === 'mentions') p.mentions = findUser(v) || v.replace(/^@/, '').toLowerCase();+ else if (key === 'in') { const c = findChat(v); if (c) (p.chats ||= []).push(c.id); else p.bad = `No chat called ${v}`; }+ else if (key === 'has') p.files = /^files?|attachments?$/i.test(v) || p.files;+ else { const t = Date.parse(v); if (!isNaN(t)) p[key] = new Date(t).toISOString(); }+ }+ p.text = words.join(' ');+ return p;+}+function findUser(v) {+ v = v.replace(/^@/, '').toLowerCase();+ if (S.users.has(v)) return v;+ const hit = [...S.users.values()].find(u => u.name.toLowerCase() === v || u.name.toLowerCase().split(/\s+/)[0] === v);+ return hit ? hit.id : null;+}+function findChat(v) {+ v = v.toLowerCase();+ if (S.chats.has(v)) return S.chats.get(v);+ const n = v.replace(/^#/, '');+ return [...S.chats.values()].find(c => c.name && c.name.toLowerCase() === n) || null;+}++class SearchView extends ListView {+ constructor(main, q) {+ const p = parseQuery(q);+ super(main, html`<span class="ico">⌕</span><h1>“${q}”</h1><span class="sub"></span>`);+ this.p = p; this.q = q; this.flat = true;+ this.none = p.bad || 'No messages match.';+ this.more = null;+ this.load();+ }+ async load(until) {+ const p = this.p;+ if (p.bad) return this.draw();+ this.wait.hidden = false;+ const t = performance.now();+ const r = await api('view_search', {text: p.text, from: p.from, mentions: p.mentions, chats: p.chats, files: p.files,+ after: p.after, before: p.before, as: S.route.as || undefined, limit: 100, until});+ if (this.dead) return;+ this.wait.hidden = true;+ this.msgs = until ? [...this.msgs, ...r.messages] : r.messages;+ this.draw({mark: p.text});+ $('.sub', this.main).textContent = `${this.msgs.length}${r.until ? '+' : ''} messages, newest first${S.route.as ? `, in ${name(S.route.as)}'s chats` : ''} · ${Math.round(performance.now() - t)} ms`;+ if (r.until) {+ const more = document.createElement('div');+ more.className = 'loading';+ more.innerHTML = '<button class="btn">Show more</button>';+ more.firstChild.onclick = () => { more.remove(); this.load(r.until); };+ this.box.append(more);+ }+ }+}++// ---------------------------------------------------------------------- one person++class PersonView {+ constructor(main, id) {+ this.main = main; this.id = id;+ const u = user(id);+ const chats = [...S.chats.values()].filter(c => c.members.includes(id)).sort((a, b) => (b.sent[id] || 0) - (a.sent[id] || 0));+ const fields = Object.entries(u.fields || {});+ main.innerHTML = html`<div class="scroll"><div class="person">+<div class="card">${avatar(id, 'lg')}<div><h1>${u.name}</h1><div class="uid">${id}${u.removed ? ' · removed' : ''}${u.status ? html` · <i>${u.status}</i>` : ''}</div>+${fields.length ? html`<div class="fields">${fields.map(([k, v]) => html`<span class="field"><i>${k}</i> ${v}</span>`)}</div>` : ''}</div>+<div class="grow"></div>${u.removed ? '' : html`<button class="btn primary" data-see-as="${id}">See as ${first(id)}</button>`}</div>+<div class="kpis"><div class="kpi"><b>${num(S.sent.get(id) || 0)}</b><span>messages sent</span></div><div class="kpi"><b>${num(chats.length)}</b><span>chats</span></div>+<div class="kpi"><b>${S.last.get(id) ? ago(S.last.get(id)) : 'never'}</b><span>last message</span></div><div class="kpi"><b>${num(chats.filter(c => c.kind === 'channel').length)}</b><span>channels</span></div></div>+<h3>Chats, by how much ${first(id)} wrote there</h3><div class="chips">${chats.map(c => html`<span class="chip" data-open="${c.id}"><span class="ico ${c.kind}">${KIND[c.kind]}</span>${title(c, id)}<span class="n">${num(c.sent[id] || 0)} of ${num(c.messages)}</span></span>`)}${chats.length ? '' : html`<span class="note">In no chat.</span>`}</div>+<h3>Latest messages</h3><div class="msgs results"><div class="loading">Loading</div></div></div></div>`;+ this.load();+ }+ async load() {+ const r = await api('view_search', {from: [this.id], limit: 40});+ if (this.dead) return;+ let prev = null, out = '';+ for (const m of r.messages) { out += messageHtml(m, prev, {showChat: true, flat: true}).s; prev = m; }+ $('.msgs', this.main).innerHTML = out || '<div class="note">No messages yet.</div>';+ }+ destroy() { this.dead = true; }+}++// ---------------------------------------------------------------------- boards++const ARROW = '<svg viewBox="0 0 16 16"><path d="M8 2.5 14 9H10v4.5H6V9H2z" fill="currentColor"/></svg>';+function postHtml(p, o = {}) {+ const lines = (p.text || '').split('\n');+ const head = p.deleted ? 'This post was deleted.' : (lines[0] || (p.attachments || []).map(a => a.name).join(', ') || '(no text)').slice(0, 300);+ const rest = p.deleted ? '' : lines.slice(1).join('\n').trim();+ const c = S.chats.get(p.chat);+ return html`<article class="post${p.deleted ? ' deleted' : ''}" data-post="${p.id}">+<div class="votes${p.votes ? ' has' : ''}" title="${p.votes ? Object.entries(p.reactions || {}).map(([e, us]) => `${EMOJI[e] || e} ${us.map(name).join(', ')}`).join('\n') : 'No reactions'}">${new Safe(ARROW)}<span>${num(p.votes)}</span></div>+<div class="main"><div class="ptitle">${head}</div>${rest ? html`<div class="pbody">${o.whole ? rich(rest) : rest}</div>` : ''}+${filesHtml(p.attachments, true)}+<div class="pmeta">${avatar(p.from, 'sm')}<b data-person="${p.from}">${name(p.from)}</b>${o.showChat ? html`<span class="chat" data-open="${p.chat}">${title(c)}</span>` : ''}<span title="${full(p.at)}">${ago(p.at)}</span><span>💬 ${plural(p.comments, 'comment')}</span>${p.comments ? html`<span title="${full(p.active_at)}">last ${ago(p.active_at)}</span>` : ''}</div></div></article>`;+}++class BoardView {+ constructor(main, chatId, sort) {+ this.main = main; this.chat = chatId; this.sort = sort; this.posts = []; this.fresh = 0;+ const c = chatId ? S.chats.get(chatId) : null;+ const scope = c ? (c.name ? '#' + c.name : title(c, null)) : 'Front page';+ const sub = c ? `${plural(c.members.length, 'member')} · ${plural(c.messages, 'message')}${c.kind === 'channel' ? (c.public ? ' · public' : ' · private') : ''}${c.archived ? ' · archived' : ''}`+ : `top-level messages from every channel and group${S.route.as ? `, in ${name(S.route.as)}'s chats` : ''}`;+ main.innerHTML = html`<div class="mhead"><span class="ico ${c ? c.kind : ''}">${c ? KIND[c.kind] : '★'}</span><h1>${scope}</h1><span class="sub">${sub}</span><div class="grow"></div>+<div class="sorts">${['hot', 'new', 'top', 'active'].map(s => html`<button data-sort="${s}" class="${s === sort ? 'on' : ''}" title="${{hot: 'Votes and comments, weighed by age', new: 'Newest posts first', top: 'Most votes and comments', active: 'Newest comment first'}[s]}">${s[0].toUpperCase() + s.slice(1)}</button>`)}</div>+${c ? html`<button class="btn" data-goto="chat">Open as a chat</button>` : ''}+${c && c.description ? html`<div class="about" title="${c.description}">${c.description}</div>` : ''}</div><div class="scroll"><div class="posts"><div class="loading">Loading</div></div></div><button class="newpill" hidden></button>`;+ this.box = $('.posts', main);+ this.pill = $('.newpill', main);+ this.pill.onclick = () => { this.pill.hidden = true; this.fresh = 0; this.load(); };+ this.load();+ }+ async load(more) {+ const r = await api('view_posts', {chat: this.chat || undefined, sort: this.sort, limit: 40, offset: more ? this.posts.length : 0, as: S.route.as || undefined});+ if (this.dead) return;+ this.posts = more ? [...this.posts, ...r.posts] : r.posts;+ this.box.innerHTML = this.posts.map(p => postHtml(p, {showChat: !this.chat}).s).join('') ||+ html`<div class="empty">No posts yet. A post is any message that does not reply to another.</div>`.s;+ if (r.more) {+ const b = document.createElement('div');+ b.className = 'loading';+ b.innerHTML = '<button class="btn">Show more</button>';+ b.firstChild.onclick = () => this.load(true);+ this.box.append(b);+ }+ }+ onEvents(ev) {+ const n = ev.lines.filter(m => m.kind === 'message' && !m.parent && (this.chat ? m.chat === this.chat : S.chats.get(m.chat)?.kind !== 'direct')).length;+ if (!n) return;+ this.fresh += n;+ this.pill.hidden = false;+ this.pill.textContent = `${plural(this.fresh, 'new post')}, show them`;+ }+ destroy() { this.dead = true; }+}++class PostView {+ constructor(main, id) {+ this.main = main; this.id = id;+ main.innerHTML = html`<div class="mhead"><button class="btn" data-back>← Back</button><h1></h1><div class="grow"></div><button class="btn" data-open-post-chat>Open in the chat</button></div><div class="scroll"><div class="thread-page"><div class="loading">Loading</div></div></div>`;+ this.load();+ }+ async load() {+ const r = await api('view_thread', {id: this.id});+ if (this.dead) return;+ const root = r.messages[0];+ this.chat = r.chat;+ $('h1', this.main).textContent = chatTitle(r.chat);+ const comments = r.messages.slice(1);+ const votes = Object.values(root.reactions || {}).reduce((n, us) => n + us.length, 0);+ $('.thread-page', this.main).innerHTML = postHtml({...root, votes, comments: comments.length, active_at: comments.length ? comments[comments.length - 1].at : root.at}, {whole: true}).s ++ html`<div class="comments tree">${treeHtml(comments)}</div>`.s;+ }+ onEvents(ev) {+ if (ev.lines.some(m => m.chat === this.chat && m.parent) || ev.changes.some(m => m.chat === this.chat)) this.reload();+ }+ reload = throttle(() => { if (!this.dead) this.load(); }, 1000);+ destroy() { this.dead = true; }+}++// A thread as a tree: each reply under what it answers, with a control to fold any branch.+function treeHtml(list) {+ let out = '';+ list.forEach((m, i) => {+ const kids = list.slice(i + 1).findIndex(x => x.depth <= m.depth);+ const below = kids < 0 ? list.length - i - 1 : kids;+ out += messageHtml(m, null, {tree: true, below}).s;+ });+ return new Safe(out);+}+function fold(el) {+ const msg = el.closest('.msg');+ const d = +getComputedStyle(msg).getPropertyValue('--d');+ const closing = !msg.classList.contains('folded');+ msg.classList.toggle('folded', closing);+ el.textContent = (closing ? '▸' : '▾') + el.textContent.slice(1);+ for (let n = msg.nextElementSibling; n && +getComputedStyle(n).getPropertyValue('--d') > d; n = n.nextElementSibling) n.hidden = closing;+}++// ---------------------------------------------------------------------- the pane on the right++class ThreadPane {+ constructor(id) {+ this.id = id;+ const pane = $('#pane');+ pane.innerHTML = html`<div class="phead"><h2>Thread</h2><button class="close" data-close-pane title="Close">×</button></div><div class="scroll"><div class="tree msgs"><div class="loading">Loading</div></div></div>`;+ this.box = $('.tree', pane);+ this.load();+ }+ async load() {+ try {+ const r = await api('view_thread', {id: this.id});+ if (this.dead) return;+ this.chat = r.chat;+ $('#pane h2').textContent = `Thread in ${chatTitle(r.chat)}`;+ this.box.innerHTML = treeHtml(r.messages).s;+ const me = this.box.querySelector(`.msg[data-id="${CSS.escape(this.id)}"]`);+ if (me && me !== this.box.firstElementChild) { me.scrollIntoView({block: 'center'}); me.classList.add('flash'); }+ } catch (e) { this.box.innerHTML = html`<div class="note">${e.message}</div>`.s; }+ }+ onEvents(ev) { if (ev.lines.some(m => m.chat === this.chat && m.parent) || ev.changes.some(m => m.chat === this.chat)) this.reload(); }+ reload = throttle(() => { if (!this.dead) this.load(); }, 1000);+ destroy() { this.dead = true; }+}++class KnewPane {+ constructor(id) {+ this.id = id;+ const pane = $('#pane');+ pane.innerHTML = html`<div class="phead"><h2>What had they read?</h2><button class="close" data-close-pane title="Close">×</button></div><div class="scroll"><div class="knew"><div class="loading">Loading</div></div></div>`;+ this.box = $('.knew', pane);+ this.load();+ }+ async load() {+ try {+ const r = await api('view_knew', {id: this.id});+ if (this.dead) return;+ S.knew = {id: this.id, user: r.user, at: r.at, chats: r.chats};+ $('#pane h2').textContent = `What ${name(r.user)} had read`;+ const rows = r.chats.map(k => {+ const c = S.chats.get(k.chat);+ return html`<div class="k" data-had="${k.chat}" data-upto="${k.upto}"><span class="t"><span class="ico ${c ? c.kind : ''}">${c ? KIND[c.kind] : ''}</span> ${title(c, r.user)}${k.chat === r.message.chat ? ' · here' : ''}</span>+<span class="u ${k.unseen ? 'some' : 'zero'}">${k.unseen ? `${num(k.unseen)} not seen` : 'all seen'}</span>+<span class="s">read ${num(k.read)} of ${plural(k.messages, 'message')}${k.read_at ? ` · last read at ${clockS(k.read_at)}` : ' · had read nothing'}</span></div>`;+ });+ this.box.innerHTML = html`<div class="note">When ${name(r.user)} sent this at <b>${clockS(r.at)}</b>, this is how far ${first(r.user)} had marked messages read in each chat. It counts only what was marked read: <code>--peek</code>, threads, previews and searches leave no trace, so ${first(r.user)} may have seen more.</div>+<div class="msgs">${messageHtml(r.message, null, {tree: true})}</div>${rows}`.s;+ if (S.view instanceof ChatView && S.view.o.knew === this.id) S.view.draw();+ } catch (e) { this.box.innerHTML = html`<div class="note">${e.message}</div>`.s; }+ }+ destroy() { this.dead = true; }+}++// ---------------------------------------------------------------------- overlays: search, pickers, popovers++const overlay = {+ el: null,+ open(content, anchor) {+ this.el = $('#overlay');+ this.el.innerHTML = '<div class="shade"></div>';+ this.el.append(content);+ this.el.classList.add('on');+ if (anchor) {+ const r = anchor.getBoundingClientRect();+ content.style.top = Math.min(r.bottom + 6, innerHeight - 200) + 'px';+ content.style.left = Math.max(8, Math.min(r.left, innerWidth - 400)) + 'px';+ }+ $('.shade', this.el).onclick = () => this.close();+ },+ close() { if (this.el) { this.el.classList.remove('on'); this.el.innerHTML = ''; } },+ get on() { return this.el && this.el.classList.contains('on'); },+};++function openSearch(initial = '') {+ const box = document.createElement('div');+ box.className = 'palette';+ box.innerHTML = html`<input placeholder="Search messages, people and chats" spellcheck="false"><div class="res"></div><div class="hint">Enter searches every message. Narrow it with <code>from:alice</code> <code>in:#ops</code> <code>mentions:bob</code> <code>has:file</code> <code>after:2026-10-01</code> <code>before:…</code></div>`.s;+ overlay.open(box);+ const input = $('input', box), res = $('.res', box);+ let items = [], sel = 0;+ const render = () => {+ const q = input.value.trim(), f = q.toLowerCase();+ items = [];+ if (q) items.push({label: html`Search messages for <b>${q}</b>`, sub: S.route.as ? `in ${name(S.route.as)}'s chats` : 'in every chat', go: () => go({q, layout: S.route.layout === 'map' ? 'chat' : S.route.layout})});+ const people = f ? [...S.users.values()].filter(u => `${u.id} ${u.name} ${u.status} ${Object.values(u.fields || {}).join(' ')}`.toLowerCase().includes(f)).slice(0, 6) : [];+ const chats = f ? [...S.chats.values()].filter(c => title(c, null).toLowerCase().includes(f) || c.id === f).slice(0, 6) : [];+ const groups = [];+ if (people.length) groups.push(['People', people.map(u => ({label: html`${avatar(u.id, 'sm')} ${u.name} <span class="s">${u.id}</span>`, sub: Object.values(u.fields || {}).join(' · '), go: () => go({person: u.id, layout: S.route.layout === 'map' ? 'chat' : S.route.layout})}))]);+ if (chats.length) groups.push(['Chats', chats.map(c => ({label: html`<span class="ico ${c.kind}">${KIND[c.kind]}</span> ${title(c, null)}`, sub: plural(c.messages, 'message'), go: () => go({chat: c.id, layout: S.route.layout === 'map' ? 'chat' : S.route.layout})}))]);+ let out = items.map((it, i) => html`<div class="it" data-i="${i}">${it.label}<span class="s">${it.sub}</span></div>`.s).join('');+ for (const [label, list] of groups) {+ out += html`<div class="grp">${label}</div>`.s;+ for (const it of list) { out += html`<div class="it" data-i="${items.length}"><span class="t">${it.label}</span><span class="s">${it.sub}</span></div>`.s; items.push(it); }+ }+ res.innerHTML = out || html`<div class="grp">Type to search. ${num(S.users.size)} people, ${num(S.chats.size)} chats, ${num(S.total)} messages.</div>`.s;+ sel = Math.min(sel, Math.max(0, items.length - 1));+ mark();+ };+ const mark = () => $$('.it', res).forEach(e => e.classList.toggle('on', +e.dataset.i === sel));+ input.addEventListener('input', () => { sel = 0; render(); });+ input.addEventListener('keydown', e => {+ if (e.key === 'ArrowDown') { sel = Math.min(items.length - 1, sel + 1); mark(); e.preventDefault(); }+ else if (e.key === 'ArrowUp') { sel = Math.max(0, sel - 1); mark(); e.preventDefault(); }+ else if (e.key === 'Enter' && items[sel]) { const it = items[sel]; overlay.close(); it.go(); }+ });+ res.addEventListener('click', e => { const it = e.target.closest('[data-i]'); if (it) { overlay.close(); items[+it.dataset.i].go(); } });+ input.value = initial;+ render();+ input.focus();+}++function openAsPicker(anchor) {+ const box = document.createElement('div');+ box.className = 'pop';+ box.innerHTML = html`<div class="ph">See everything as</div><input placeholder="Find an agent" spellcheck="false"><div class="list"></div>`.s;+ overlay.open(box, anchor);+ const input = $('input', box), list = $('.list', box);+ const people = [...S.users.values()].filter(u => !u.removed).sort((a, b) => (S.sent.get(b.id) || 0) - (S.sent.get(a.id) || 0));+ const render = () => {+ const f = input.value.trim().toLowerCase();+ const hits = people.filter(u => !f || `${u.id} ${u.name} ${Object.values(u.fields || {}).join(' ')}`.toLowerCase().includes(f)).slice(0, 200);+ list.innerHTML = (S.route.as ? html`<div class="it" data-as="">Everyone<span class="s">the whole server</span></div>`.s : '') ++ hits.map(u => html`<div class="it" data-as="${u.id}">${avatar(u.id, 'sm')} ${u.name}<span class="s">${Object.values(u.fields || {})[0] || u.id}</span></div>`.s).join('');+ };+ input.addEventListener('input', render);+ input.addEventListener('keydown', e => { if (e.key === 'Enter') { const it = $('[data-as]', list); if (it) { overlay.close(); go({as: it.dataset.as || null}, true); } } });+ list.addEventListener('click', e => { const it = e.target.closest('[data-as]'); if (it) { overlay.close(); go({as: it.dataset.as || null}, true); } });+ render();+ input.focus();+}++function openMembers(anchor, chatId) {+ const c = S.chats.get(chatId);+ if (!c) return;+ const pos = S.view && S.view.positions ? S.view.positions : {};+ const last = c.last ? c.last.seq : 0;+ const box = document.createElement('div');+ box.className = 'pop';+ box.innerHTML = html`<div class="ph">${plural(c.members.length, 'member')}</div>${c.members.map(u => {+ const p = pos[u];+ const state = p == null ? '' : p >= last ? 'read everything' : p ? `read up to ${chatId}:${p}` : 'read nothing';+ return html`<div class="it" data-person="${u}">${avatar(u, 'sm')} ${name(u)}${(c.managers || []).includes(u) ? html` <span class="pill">manager</span>` : ''}<span class="s">${state}</span></div>`;+ })}`.s;+ overlay.open(box, anchor);+ box.addEventListener('click', e => { const it = e.target.closest('[data-person]'); if (it) { overlay.close(); go({person: it.dataset.person}); } });+}++async function openReceipts(anchor, id) {+ const box = document.createElement('div');+ box.className = 'pop';+ box.innerHTML = '<div class="ph">Read by</div><div class="loading">Loading</div>';+ overlay.open(box, anchor);+ const r = await api('view_receipts', {id});+ box.innerHTML = html`<div class="ph">Read by ${r.read.length} of ${r.read.length + r.unread.length}</div>+${r.read.map(x => html`<div class="it" data-person="${x.id}">${avatar(x.id, 'sm')} ${name(x.id)}<span class="s">${x.at ? clockS(x.at) : 'read'}</span></div>`)}+${r.unread.length ? html`<div class="ph">Not yet</div>${r.unread.map(u => html`<div class="it" data-person="${u}">${avatar(u, 'sm')} ${name(u)}</div>`)}` : ''}`.s;+ box.addEventListener('click', e => { const it = e.target.closest('[data-person]'); if (it) { overlay.close(); go({person: it.dataset.person}); } });+}++let toastTimer;+function toast(text) {+ const tip = $('#tip');+ tip.textContent = text;+ tip.style.left = '50%'; tip.style.top = '60px'; tip.style.transform = 'translateX(-50%)';+ tip.classList.add('on');+ clearTimeout(toastTimer);+ toastTimer = setTimeout(() => { tip.classList.remove('on'); tip.style.transform = ''; }, 2600);+}++// ---------------------------------------------------------------------- following everything live++async function follow() {+ for (;;) {+ try {+ const r = await api('view_feed', {after: S.feed, feed_id: S.server.feed_id, timeout: 25});+ setLive(S.server.read_only ? 'ro' : 'on');+ if (r.reset) {+ await loadSnapshot();+ S.viewKey = null; S.paneKey = null;+ await route();+ continue;+ }+ S.feed = r.feed;+ if (r.events.length) { apply(r.events); await sleep(100); }+ } catch (e) {+ if (e.status === 401) return login('The token was refused. It may have changed.');+ setLive('off');+ await sleep(2000);+ }+ }+}++function apply(events) {+ const ev = {lines: [], changes: [], reads: [], chats: new Set(), users: false};+ let asTouched = false;+ const as = S.route.as;+ for (const e of events) {+ if (e.kind === 'user') { S.users.set(e.user.id, e.user); ev.users = true; continue; }+ if (e.kind === 'chat') {+ addChat(e.summary);+ ev.chats.add(e.chat);+ ev.lines.push({...e.event, id: `${e.chat}:${e.seq}`, chat: e.chat});+ if (as && (e.summary.members.includes(as) || (e.event.users || []).includes(as))) asTouched = true;+ continue;+ }+ const c = S.chats.get(e.chat);+ if (!c) continue;+ const fresh = e.seq > c.seq;+ if (fresh) c.seq = e.seq;+ if (as && c.members.includes(as)) asTouched = true;+ if (e.kind === 'message') {+ const m = e.message;+ if (fresh) {+ if (e.count > c.messages) { S.total += e.count - c.messages; c.messages = e.count; }+ c.last = {seq: m.seq, from: m.from, at: m.at, text: (m.text || '').slice(0, 160), deleted: false, files: (m.attachments || []).length};+ c.last_at = m.at;+ c.recent = (c.recent || 0) + 1;+ c.sent[m.from] = (c.sent[m.from] || 0) + 1;+ S.sent.set(m.from, (S.sent.get(m.from) || 0) + 1);+ const t = ms(m.at) / 1000; // seconds, as the snapshot gives them+ if (!(S.last.get(m.from) >= t)) S.last.set(m.from, t);+ S.rate.push(Date.now());+ }+ ev.lines.push(m);+ ev.chats.add(e.chat);+ } else if (e.kind === 'change') {+ if (c.last && c.last.seq === e.message.seq) c.last = {...c.last, text: e.message.deleted ? '' : (e.message.text || '').slice(0, 160), deleted: !!e.message.deleted};+ ev.changes.push(e.message);+ } else if (e.kind === 'read') {+ ev.reads.push(e);+ }+ }+ if (S.view && S.view.onEvents) S.view.onEvents(ev);+ if (S.pane && S.pane.onEvents) S.pane.onEvents(ev);+ if (asTouched) refreshAs();+ if (ev.chats.size || ev.users) buildSide();+}++// ---------------------------------------------------------------------- clicks anywhere++function onClick(e) {+ const t = e.target;+ const hit = sel => t.closest(sel);+ let el;+ if ((el = hit('[data-layout]'))) {+ const layout = el.dataset.layout, r = S.route;+ return go({layout, chat: r.chat, as: r.as, person: layout === 'map' ? null : r.person, q: layout === 'map' ? null : r.q});+ }+ if (hit('[data-clear-as]')) { e.stopPropagation(); return go({as: null}, true); }+ if ((el = hit('#as'))) return openAsPicker(el);+ if (hit('#find')) return openSearch();+ if (hit('#theme')) return setTheme(THEMES[theme()].next);+ if (hit('[data-home]')) { e.preventDefault(); return go({layout: S.route.layout, as: S.route.as, thread: null, knew: null}); }+ if ((el = hit('[data-see-as]'))) return go({as: el.dataset.seeAs, person: null});+ if ((el = hit('[data-file]'))) return download(el.dataset.file, el.dataset.name);+ if ((el = hit('[data-image]'))) return openPicture(el.dataset);+ if ((el = hit('[data-text]'))) return openText(el.dataset.text);+ if ((el = hit('[data-thread]'))) return go({thread: el.dataset.thread, knew: null}, true);+ if ((el = hit('[data-knew]'))) return go({knew: el.dataset.knew, thread: null}, true);+ if ((el = hit('[data-receipts]'))) return openReceipts(el, el.dataset.receipts);+ if ((el = hit('[data-copy]'))) { navigator.clipboard?.writeText(el.dataset.copy); return toast(`Copied ${el.dataset.copy}`); }+ if (hit('[data-close-pane]')) return go({thread: null, knew: null}, true);+ if ((el = hit('[data-fold]'))) return fold(el);+ if ((el = hit('[data-more]'))) { el.previousElementSibling.classList.remove('long'); return el.remove(); }+ if ((el = hit('[data-members]'))) return openMembers(el, el.dataset.members);+ if ((el = hit('[data-sort]'))) return go({sort: el.dataset.sort}, true);+ if ((el = hit('[data-board]'))) return go({layout: 'board', chat: el.dataset.board});+ if ((el = hit('[data-goto]'))) return go({layout: el.dataset.goto, chat: S.route.chat || (S.view && S.view.chat), post: null}, false);+ if (hit('[data-back]')) return history.length > 1 ? history.back() : go({post: null}, true);+ if (hit('[data-open-post-chat]')) return go({layout: 'chat', chat: S.view.chat, m: +S.view.id.split(':')[1]});+ if ((el = hit('[data-had]'))) return go({layout: 'chat', chat: el.dataset.had, m: el.dataset.upto || 1, had: el.dataset.upto, knew: S.route.knew});+ if ((el = hit('[data-jump]'))) {+ if (S.view instanceof ChatView && S.view.id === el.dataset.chat) return S.view.jump(+el.dataset.jump);+ return go({layout: 'chat', chat: el.dataset.chat, m: el.dataset.jump});+ }+ if ((el = hit('[data-open]'))) return go({layout: S.route.layout === 'map' ? 'chat' : S.route.layout, chat: el.dataset.open, m: el.dataset.at || null});+ if ((el = hit('[data-person]'))) return go({person: el.dataset.person, layout: S.route.layout === 'map' ? 'chat' : S.route.layout});+ if ((el = hit('[data-post]'))) return go({post: el.dataset.post}, true);+ if ((el = hit('.results .msg'))) return go({layout: S.route.layout === 'board' ? 'board' : 'chat', chat: el.dataset.chat, m: el.dataset.seq, q: null, person: null});+}++function onKey(e) {+ const typing = /INPUT|TEXTAREA|SELECT/.test(document.activeElement?.tagName || '');+ if ((e.metaKey || e.ctrlKey) && e.key.toLowerCase() === 'k') { e.preventDefault(); return overlay.on ? overlay.close() : openSearch(); }+ if (e.key === 'Escape') { if (overlay.on) return overlay.close(); if (S.route.thread || S.route.knew) return go({thread: null, knew: null}, true); }+ if (e.key === 'Enter' && document.activeElement?.dataset.text) return openText(document.activeElement.dataset.text);+ if (typing || overlay.on || e.metaKey || e.ctrlKey || e.altKey) return;+ if (e.key === '/') { e.preventDefault(); return openSearch(); }+ if (['1', '2', '3'].includes(e.key)) return go({layout: ['chat', 'board', 'map'][+e.key - 1], chat: S.route.chat, as: S.route.as});+ if (S.view && S.view.onKey) S.view.onKey(e);+}++function tips() {+ const tip = $('#tip');+ document.addEventListener('mouseover', e => {+ const el = e.target.closest('[data-tip]');+ if (!el) return tip.classList.remove('on');+ tip.textContent = el.dataset.tip;+ const r = el.getBoundingClientRect();+ tip.style.left = Math.min(r.left, innerWidth - 370) + 'px';+ tip.style.top = (r.bottom + 6) + 'px';+ tip.classList.add('on');+ });+}++// ---------------------------------------------------------------------- the map: everyone at once++// Categorical colours in a fixed order, made to be told apart on a light or a dark background. Past eight, the rest are "other".+const PALETTE = ['#4269d0', '#efb118', '#ff725c', '#6cc5b0', '#3ca951', '#ff8ab7', '#a463f2', '#97bbf5'];+const OTHER = '#7b8394';+const MAP = {pos: new Map(), view: null, colorBy: null, lens: true}; // kept while the page is open++// The layout runs in a worker, so the page never stops answering. Forces as in Fruchterman and Reingold: every pair+// pushes apart, every link pulls together, and each step moves less than the one before.+function layoutWorker() {+ self.onmessage = e => {+ const {x, y, mass, a, b, pull, iterations, heat, k} = e.data;+ const n = x.length, dx = new Float32Array(n), dy = new Float32Array(n), kk = k * k, far = (12 * k) ** 2;+ for (let it = 0; it < iterations; it++) {+ dx.fill(0); dy.fill(0);+ if (n <= 2500) {+ for (let i = 0; i < n; i++) {+ const xi = x[i], yi = y[i], mi = mass[i];+ for (let j = i + 1; j < n; j++) {+ let ex = xi - x[j], ey = yi - y[j], d2 = ex * ex + ey * ey;+ if (d2 > far) continue;+ if (d2 < 0.01) { ex = Math.random() - 0.5; ey = Math.random() - 0.5; d2 = 0.5; }+ const f = kk * mi * mass[j] / d2;+ dx[i] += ex * f; dy[i] += ey * f; dx[j] -= ex * f; dy[j] -= ey * f;+ }+ }+ } else { // many nodes: only neighbours within a few cells push+ const cell = 6 * k, grid = new Map();+ for (let i = 0; i < n; i++) {+ const key = Math.floor(x[i] / cell) * 65536 + Math.floor(y[i] / cell);+ (grid.get(key) || grid.set(key, []).get(key)).push(i);+ }+ for (let i = 0; i < n; i++) {+ const cx = Math.floor(x[i] / cell), cy = Math.floor(y[i] / cell);+ for (let gx = cx - 1; gx <= cx + 1; gx++) for (let gy = cy - 1; gy <= cy + 1; gy++) {+ const list = grid.get(gx * 65536 + gy);+ if (!list) continue;+ for (const j of list) {+ if (j <= i) continue;+ let ex = x[i] - x[j], ey = y[i] - y[j], d2 = ex * ex + ey * ey;+ if (d2 < 0.01) { ex = Math.random() - 0.5; ey = Math.random() - 0.5; d2 = 0.5; }+ const f = kk * mass[i] * mass[j] / d2;+ dx[i] += ex * f; dy[i] += ey * f; dx[j] -= ex * f; dy[j] -= ey * f;+ }+ }+ }+ }+ for (let e2 = 0; e2 < a.length; e2++) {+ const i = a[e2], j = b[e2], ex = x[j] - x[i], ey = y[j] - y[i];+ const d = Math.sqrt(ex * ex + ey * ey) || 0.1, f = d / k * pull[e2];+ dx[i] += ex * f; dy[i] += ey * f; dx[j] -= ex * f; dy[j] -= ey * f;+ }+ const t = heat * (1 - it / iterations) + 0.5;+ for (let i = 0; i < n; i++) {+ dx[i] -= x[i] * 0.03; dy[i] -= y[i] * 0.03;+ const d = Math.sqrt(dx[i] * dx[i] + dy[i] * dy[i]) || 1, s = Math.min(d, t) / d;+ x[i] += dx[i] * s; y[i] += dy[i] * s;+ }+ if (it % 8 === 7 || it === iterations - 1) self.postMessage({x, y, done: it === iterations - 1});+ }+ };+}++class MapView {+ constructor(main) {+ this.main = main; this.pulses = []; this.win = null; this.winData = null; this.hover = null; this.selected = null;+ main.innerHTML = html`<div class="mapwrap"><canvas class="graph"></canvas><div class="legend"></div>+<div class="mapctl"><button class="btn" data-lens title="A lens that magnifies what is under the mouse (F)">Lens</button><button class="btn" data-fit title="Show everyone (0)">Fit</button></div>+<div class="maphint">Scroll to zoom, drag to move. Click an agent to see as it, a chat to open it.</div>+<div class="strip"><div class="sctl"><button class="btn" data-whole title="Count every message">Whole run</button><button class="btn" data-play title="Slide a window of time across the run">▶ Replay</button></div>+<div class="chart"><canvas></canvas><div class="when"><span></span><span></span></div><div class="sel"></div></div></div></div>`.s;+ this.canvas = $('canvas.graph', main); this.ctx = this.canvas.getContext('2d');+ this.strip = $('.strip canvas', main); this.sctx = this.strip.getContext('2d');+ this.legend = $('.legend', main);+ S.mapPane = true;+ $('#pane').classList.add('open');+ this.listen();+ this.build(true);+ this.loadActivity();+ this.resize = new ResizeObserver(() => { this.size(); this.draw(); this.drawStrip(); });+ this.resize.observe(this.canvas); this.resize.observe(this.strip);+ $('[data-lens]', main).classList.toggle('on', MAP.lens);+ this.showPane();+ }+ destroy() {+ this.dead = true; this.resize.disconnect(); if (this.worker) this.worker.terminate();+ cancelAnimationFrame(this.raf); this.raf = 0; S.mapPane = false; clearInterval(this.playing);+ $('#pane').classList.remove('open'); $('#pane').replaceChildren();+ }+ size() {+ const dpr = devicePixelRatio || 1;+ for (const c of [this.canvas, this.strip]) {+ const r = c.getBoundingClientRect();+ c.width = Math.max(1, Math.round(r.width * dpr)); c.height = Math.max(1, Math.round(r.height * dpr));+ c.w = r.width; c.h = r.height;+ }+ this.dpr = dpr;+ }++ // ---- the graph+ build(fresh) {+ const nodes = [], index = new Map(), edges = [];+ const inChats = new Set();+ for (const c of S.chats.values()) for (const u of c.members) inChats.add(u);+ for (const u of S.users.values()) if (inChats.has(u.id) || S.sent.get(u.id)) { index.set('u:' + u.id, nodes.length); nodes.push({key: 'u:' + u.id, kind: 'agent', user: u.id}); }+ for (const c of S.chats.values()) if (c.kind !== 'direct') { index.set('c:' + c.id, nodes.length); nodes.push({key: 'c:' + c.id, kind: 'hub', chat: c.id}); }+ for (const c of S.chats.values()) {+ if (c.kind === 'direct') {+ const [a, b] = c.members.map(u => index.get('u:' + u));+ if (a != null && b != null) edges.push({a, b, chat: c.id, kind: 'direct'});+ } else {+ const h = index.get('c:' + c.id);+ for (const u of c.members) { const i = index.get('u:' + u); if (i != null) edges.push({a: h, b: i, chat: c.id, user: u, kind: 'spoke'}); }+ }+ }+ const spread = Math.sqrt(nodes.length) * 40;+ const nbrs = new Map();+ for (const e of edges) { (nbrs.get(e.a) || nbrs.set(e.a, []).get(e.a)).push(e.b); (nbrs.get(e.b) || nbrs.set(e.b, []).get(e.b)).push(e.a); }+ let placed = 0;+ nodes.forEach((n, i) => {+ const p = MAP.pos.get(n.key);+ if (p) { n.x = p.x; n.y = p.y; placed++; return; }+ const near = (nbrs.get(i) || []).map(j => MAP.pos.get(nodes[j].key)).find(Boolean);+ const a = Math.random() * Math.PI * 2, r = near ? 20 + Math.random() * 30 : Math.sqrt(Math.random()) * spread;+ n.x = (near ? near.x : 0) + Math.cos(a) * r; n.y = (near ? near.y : 0) + Math.sin(a) * r;+ });+ this.nodes = nodes; this.index = index; this.edges = edges; this.nbrs = nbrs;+ this.weigh();+ this.colors();+ this.layout(fresh && placed < nodes.length / 2);+ if (!MAP.view) this.fit(); else this.view = MAP.view;+ }+ layout(fresh) {+ if (this.worker) this.worker.terminate();+ const n = this.nodes.length;+ if (!n) return this.draw();+ const blob = new Blob([`(${layoutWorker})()`], {type: 'text/javascript'});+ const url = URL.createObjectURL(blob);+ this.worker = new Worker(url);+ URL.revokeObjectURL(url);+ const x = new Float32Array(n), y = new Float32Array(n), mass = new Float32Array(n);+ this.nodes.forEach((d, i) => { x[i] = d.x; y[i] = d.y; mass[i] = d.kind === 'hub' ? 1 + Math.sqrt(S.chats.get(d.chat).members.length) / 2 : 1; });+ const a = new Int32Array(this.edges.length), b = new Int32Array(this.edges.length), pull = new Float32Array(this.edges.length);+ const most = this.edges.reduce((m, e) => Math.max(m, e.w0 || 0), 1);+ const degree = new Float32Array(n);+ for (const e of this.edges) { degree[e.a]++; degree[e.b]++; }+ // A link pulls less when either end has many: an agent in fifty chats is not pulled fifty times as hard.+ this.edges.forEach((e, i) => { a[i] = e.a; b[i] = e.b; pull[i] = (e.kind === 'direct' ? 0.6 : 1) * (0.4 + 0.6 * Math.log1p(e.w0 || 0) / Math.log1p(most)) / Math.sqrt(Math.min(degree[e.a], degree[e.b])); });+ const k = 26;+ this.worker.onmessage = m => {+ const {x: X, y: Y, done} = m.data;+ this.nodes.forEach((d, i) => { d.x = X[i]; d.y = Y[i]; MAP.pos.set(d.key, {x: d.x, y: d.y}); });+ if (this.autoFit) this.fit();+ if (done) this.autoFit = false;+ this.kick();+ };+ this.autoFit = fresh;+ this.worker.postMessage({x, y, mass, a, b, pull, k, iterations: fresh ? 320 : 90, heat: fresh ? Math.sqrt(n) * k / 3 : k * 1.5});+ }+ fit() {+ if (!this.nodes.length || !this.canvas.w) { this.view = MAP.view = {x: 0, y: 0, k: 1}; return; }+ let x0 = Infinity, y0 = Infinity, x1 = -Infinity, y1 = -Infinity;+ for (const n of this.nodes) { x0 = Math.min(x0, n.x); y0 = Math.min(y0, n.y); x1 = Math.max(x1, n.x); y1 = Math.max(y1, n.y); }+ const k = Math.min(4, 0.86 * Math.min(this.canvas.w / Math.max(60, x1 - x0), this.canvas.h / Math.max(60, y1 - y0)));+ this.view = MAP.view = {x: (x0 + x1) / 2, y: (y0 + y1) / 2, k};+ this.kick();+ }+ weigh() {+ const win = this.winData;+ const msgs = id => win ? (win[id] ? win[id].messages : 0) : S.chats.get(id).messages;+ const sent = (id, u) => win ? (win[id] ? win[id].sent[u] || 0 : 0) : S.chats.get(id).sent[u] || 0;+ const total = new Map();+ for (const e of this.edges) {+ e.w = e.kind === 'direct' ? msgs(e.chat) : sent(e.chat, e.user);+ if (e.w0 == null) e.w0 = e.w;+ }+ if (win) for (const [id, d] of Object.entries(win)) for (const [u, n] of Object.entries(d.sent)) total.set(u, (total.get(u) || 0) + n);+ let maxE = 1, maxA = 1, maxH = 1;+ for (const n of this.nodes) {+ if (n.kind === 'agent') { n.w = win ? total.get(n.user) || 0 : S.sent.get(n.user) || 0; n.r = 2.5 + 1.6 * Math.sqrt(Math.log2(1 + n.w)); maxA = Math.max(maxA, n.w); }+ else { const c = S.chats.get(n.chat); n.w = msgs(n.chat); n.r = Math.min(16, 3 + Math.sqrt(c.members.length)); maxH = Math.max(maxH, n.w); }+ }+ for (const e of this.edges) maxE = Math.max(maxE, e.w);+ this.maxE = maxE; this.maxA = maxA; this.maxH = maxH;+ if (MAP.colorBy === 'activity' && this.nodes.some(n => n.color)) this.colors();+ this.kick();+ }+ fields() {+ const keys = new Map();+ for (const u of S.users.values()) for (const [k, v] of Object.entries(u.fields || {})) {+ const m = keys.get(k) || keys.set(k, new Map()).get(k);+ m.set(String(v), (m.get(String(v)) || 0) + 1);+ }+ return [...keys.entries()].filter(([, vals]) => vals.size >= 2 && vals.size <= 60)+ .sort((p, q) => [...q[1].values()].reduce((s, n) => s + n, 0) - [...p[1].values()].reduce((s, n) => s + n, 0)).map(([k, vals]) => ({key: k, vals}));+ }+ colors() {+ const fields = this.fields();+ if (!MAP.colorBy || (MAP.colorBy !== 'activity' && !fields.some(f => f.key === MAP.colorBy))) MAP.colorBy = fields.length ? fields[0].key : 'activity';+ const by = MAP.colorBy;+ let items = [];+ if (by === 'activity') {+ for (const n of this.nodes) if (n.kind === 'agent') {+ const t = this.maxA > 1 ? Math.log1p(n.w || 0) / Math.log1p(this.maxA) : 0;+ n.color = ink().dark ? `hsl(220 ${40 + 50 * t}% ${30 + 45 * t}%)` : `hsl(220 ${35 + 50 * t}% ${82 - 40 * t}%)`;+ }+ items = ink().dark ? [['quiet', 'hsl(220 40% 30%)'], ['busy', 'hsl(220 90% 75%)']] : [['quiet', 'hsl(220 35% 82%)'], ['busy', 'hsl(220 85% 42%)']];+ } else {+ const vals = [...fields.find(f => f.key === by).vals.entries()].sort((p, q) => q[1] - p[1]);+ const color = new Map(vals.slice(0, 8).map(([v], i) => [v, PALETTE[i]]));+ for (const n of this.nodes) if (n.kind === 'agent') { const v = (user(n.user).fields || {})[by]; n.color = color.get(String(v)) || OTHER; }+ items = vals.slice(0, 8).map(([v, cnt]) => [v, color.get(v), cnt]);+ const rest = vals.slice(8).reduce((s, [, cnt]) => s + cnt, 0) + [...S.users.values()].filter(u => !(u.fields || {})[by]).length;+ if (rest) items.push(['other', OTHER, rest]);+ }+ for (const n of this.nodes) if (n.kind === 'hub') n.color = S.chats.get(n.chat).kind === 'channel' ? ink().channel : ink().group;+ const opts = [...fields.map(f => f.key), 'activity'];+ this.legend.innerHTML = html`<div class="lh">Agents by <select data-colorby>${opts.map(o => html`<option value="${o}"${o === by ? new Safe(' selected') : ''}>${o}</option>`)}</select></div>+${items.map(([label, c, cnt]) => html`<div class="li"><span class="dot" style="background:${c}"></span>${label}${cnt != null ? html`<span class="n">${num(cnt)}</span>` : ''}</div>`)}+<div class="shapes"><span><span class="sq" style="background:var(--channel)"></span>channel</span><span><span class="sq" style="background:var(--group)"></span>group</span><span>— direct chat</span></div>`.s;+ $('select', this.legend).onchange = e => { MAP.colorBy = e.target.value; this.colors(); this.kick(); };+ }++ // ---- drawing+ related() {+ if (!S.route.as || !S.as) return null;+ const chats = new Set(S.as.chats.keys()), agents = new Set([S.route.as]);+ for (const id of chats) for (const u of S.chats.get(id)?.members || []) agents.add(u);+ return {chats, agents};+ }+ project() {+ const W = this.canvas.w, H = this.canvas.h, v = this.view || {x: 0, y: 0, k: 1}, m = this.mouse;+ const lens = MAP.lens && m && !this.dragging, R = Math.min(260, Math.max(140, Math.min(W, H) / 3.2)), dist = 3.2;+ const k0 = Math.exp(dist) / (Math.exp(dist) - 1) * R, k1 = dist / R;+ for (const n of this.nodes) {+ let sx = (n.x - v.x) * v.k + W / 2, sy = (n.y - v.y) * v.k + H / 2, mag = 1;+ if (lens) {+ const dx = sx - m.x, dy = sy - m.y, d = Math.sqrt(dx * dx + dy * dy);+ if (d && d < R) { const f = k0 * (1 - Math.exp(-d * k1)) / d * 0.75 + 0.25; sx = m.x + dx * f; sy = m.y + dy * f; mag = Math.min(f, 6); }+ else if (!d) mag = 6;+ }+ n.sx = sx; n.sy = sy; n.mag = mag;+ }+ }+ draw() {+ if (!this.canvas.w || !this.nodes) return;+ const ctx = this.ctx, W = this.canvas.w, H = this.canvas.h, k = (this.view || {k: 1}).k;+ ctx.setTransform(this.dpr, 0, 0, this.dpr, 0, 0);+ ctx.clearRect(0, 0, W, H);+ this.project();+ const rel = this.related(), focus = this.hover ?? this.selected;+ const lit = new Set();+ if (focus != null) { lit.add(focus); for (const j of this.nbrs.get(focus) || []) lit.add(j); }+ // links, in a few strengths so each is one stroke+ const buckets = new Map();+ for (const e of this.edges) {+ const a = this.nodes[e.a], b = this.nodes[e.b];+ const on = focus != null && (e.a === focus || e.b === focus);+ const dim = rel ? !rel.chats.has(e.chat) : false;+ const t = e.w ? Math.log1p(e.w) / Math.log1p(this.maxE) : 0;+ const level = on ? 4 : dim ? 0 : t === 0 && this.winData ? 0 : 1 + Math.min(2, Math.floor(t * 3));+ const key = (e.kind === 'direct' ? 'd' : 's') + level;+ (buckets.get(key) || buckets.set(key, []).get(key)).push(a.sx, a.sy, b.sx, b.sy);+ }+ const alpha = [0.035, 0.09, 0.2, 0.42, 0.85];+ ctx.lineCap = 'round';+ for (const [key, pts] of buckets) {+ const level = +key[1];+ ctx.strokeStyle = key[0] === 'd' ? `rgba(${ink().edgeDm}, ${alpha[level]})` : `rgba(${ink().edge}, ${alpha[level] * 0.8})`;+ ctx.lineWidth = level >= 3 ? 1.6 : 1;+ ctx.beginPath();+ for (let i = 0; i < pts.length; i += 4) { ctx.moveTo(pts[i], pts[i + 1]); ctx.lineTo(pts[i + 2], pts[i + 3]); }+ ctx.stroke();+ }+ // chats, then agents+ const scale = Math.max(0.55, Math.min(2.2, Math.sqrt(k)));+ for (const pass of ['hub', 'agent']) {+ for (let i = 0; i < this.nodes.length; i++) {+ const n = this.nodes[i];+ if (n.kind !== pass) continue;+ const dim = (rel && (pass === 'hub' ? !rel.chats.has(n.chat) : !rel.agents.has(n.user))) || (this.winData && !n.w) || (focus != null && !lit.has(i));+ const r = this.radius(n, scale);+ ctx.globalAlpha = dim ? 0.18 : 1;+ ctx.fillStyle = n.color;+ if (pass === 'hub') {+ const t = n.w ? 0.35 + 0.65 * Math.log1p(n.w) / Math.log1p(this.maxH) : 0.25;+ ctx.globalAlpha *= t;+ ctx.beginPath(); ctx.roundRect(n.sx - r, n.sy - r, r * 2, r * 2, r * 0.35); ctx.fill();+ } else {+ ctx.beginPath(); ctx.arc(n.sx, n.sy, r, 0, Math.PI * 2); ctx.fill();+ if (S.route.as && n.user === S.route.as) { ctx.strokeStyle = ink().text; ctx.lineWidth = 2; ctx.stroke(); }+ }+ if (i === this.selected) { ctx.strokeStyle = ink().text; ctx.lineWidth = 1.5; ctx.beginPath(); ctx.arc(n.sx, n.sy, r + 4, 0, Math.PI * 2); ctx.stroke(); }+ }+ }+ ctx.globalAlpha = 1;+ // names: around the focus, under the lens, and the busiest when zoomed in. The most important go first, and a+ // name that would overlap one already placed is left out, so every name shown can be read.+ ctx.font = '600 11.5px ' + (MAP.font ||= getComputedStyle(document.body).fontFamily);+ ctx.textBaseline = 'middle';+ ctx.lineWidth = 3; ctx.strokeStyle = `rgba(${ink().bg}, .9)`;+ const wanted = [];+ for (let i = 0; i < this.nodes.length; i++) {+ const n = this.nodes[i];+ if (n.sx < -50 || n.sy < -20 || n.sx > W + 50 || n.sy > H + 20) continue;+ const me = S.route.as && n.user === S.route.as;+ if (!(n.mag > 1.3 || lit.has(i) || me || k > 1.2)) continue;+ wanted.push({i, rank: (i === focus ? 4e9 : 0) + (me ? 2e9 : 0) + (lit.has(i) ? 1e9 : 0) + n.mag * 1e6 + (n.w || 0)});+ }+ wanted.sort((p, q) => q.rank - p.rank);+ const taken = new Map(), cell = 24, widths = MAP.widths ||= new Map();+ let placed = 0;+ for (const {i} of wanted) {+ if (placed >= 300) break;+ const n = this.nodes[i];+ const label = n.kind === 'agent' ? name(n.user) : title(S.chats.get(n.chat), null);+ let w = widths.get(label);+ if (w == null) { w = ctx.measureText(label).width; widths.set(label, w); }+ const x = n.sx + this.radius(n, scale) + 4, y0 = n.sy - 8, y1 = n.sy + 8, x1 = x + w;+ let free = true;+ const keys = [];+ for (let gx = Math.floor(x / cell); gx <= Math.floor(x1 / cell) && free; gx++) for (let gy = Math.floor(y0 / cell); gy <= Math.floor(y1 / cell); gy++) {+ const key = gx * 4096 + gy;+ keys.push(key);+ for (const b of taken.get(key) || []) if (x < b[2] && x1 > b[0] && y0 < b[3] && y1 > b[1]) { free = false; break; }+ if (!free) break;+ }+ if (!free) continue;+ for (const key of keys) (taken.get(key) || taken.set(key, []).get(key)).push([x, y0, x1, y1]);+ placed++;+ ctx.globalAlpha = rel && !(n.kind === 'agent' ? rel.agents.has(n.user) : rel.chats.has(n.chat)) ? 0.35 : 1;+ ctx.strokeText(label, x, n.sy); ctx.fillStyle = n.kind === 'agent' ? ink().text : ink().chatName; ctx.fillText(label, x, n.sy);+ }+ ctx.globalAlpha = 1;+ // messages on their way: sender to chat, then out to its members+ const now = performance.now();+ this.pulses = this.pulses.filter(p => now < p.t0 + p.dur);+ for (const p of this.pulses) {+ const t = (now - p.t0) / p.dur;+ if (t < 0) continue;+ const e = t < 0.5 ? 2 * t * t : 1 - (-2 * t + 2) ** 2 / 2;+ ctx.fillStyle = p.color;+ for (const [i, j] of p.legs) {+ const a = this.nodes[i], b = this.nodes[j];+ if (!a || !b) continue;+ const x = a.sx + (b.sx - a.sx) * e, y = a.sy + (b.sy - a.sy) * e;+ ctx.globalAlpha = Math.min(1, (1 - t) * 3);+ ctx.beginPath(); ctx.arc(x, y, 2.6, 0, Math.PI * 2); ctx.fill();+ ctx.globalAlpha *= 0.25;+ ctx.beginPath(); ctx.arc(x, y, 7, 0, Math.PI * 2); ctx.fill();+ }+ }+ ctx.globalAlpha = 1;+ }+ radius(n, scale) { // the lens enlarges shapes less than it spreads them, so they never cover their neighbours+ return Math.min(n.kind === 'hub' ? 34 : 26, n.r * scale * (1 + (n.mag - 1) * 0.45));+ }+ kick(ms = 0) {+ this.until = Math.max(this.until || 0, performance.now() + ms);+ if (this.raf || this.dead) return;+ const loop = () => {+ this.raf = 0;+ if (this.dead) return;+ this.draw();+ if (this.pulses.length || performance.now() < this.until) this.raf = requestAnimationFrame(loop);+ };+ this.raf = requestAnimationFrame(loop);+ }++ // ---- the mouse+ listen() {+ const c = this.canvas;+ let down = null;+ c.addEventListener('mousemove', e => {+ const r = c.getBoundingClientRect();+ this.mouse = {x: e.clientX - r.left, y: e.clientY - r.top};+ if (down) {+ const dx = e.clientX - down.x, dy = e.clientY - down.y;+ if (Math.abs(dx) + Math.abs(dy) > 3) { this.dragging = true; c.classList.add('drag'); }+ if (this.dragging) { this.view.x = down.vx - dx / this.view.k; this.view.y = down.vy - dy / this.view.k; }+ } else {+ const hit = this.hit(this.mouse.x, this.mouse.y);+ if (hit !== this.hover) { this.hover = hit; c.classList.toggle('point', hit != null); this.showPane(); }+ }+ this.kick();+ });+ c.addEventListener('mouseleave', () => { this.mouse = null; if (this.hover != null) { this.hover = null; this.showPane(); } this.kick(); });+ c.addEventListener('mousedown', e => { down = {x: e.clientX, y: e.clientY, vx: this.view.x, vy: this.view.y}; this.dragging = false; });+ addEventListener('mouseup', this.up = () => {+ if (!down) return;+ const clicked = !this.dragging;+ down = null; this.dragging = false; c.classList.remove('drag');+ if (clicked && this.mouse) this.click(this.hit(this.mouse.x, this.mouse.y));+ this.kick();+ });+ c.addEventListener('wheel', e => {+ e.preventDefault();+ const r = c.getBoundingClientRect(), mx = e.clientX - r.left, my = e.clientY - r.top, v = this.view;+ const k = Math.max(0.03, Math.min(12, v.k * Math.exp(-e.deltaY * (e.ctrlKey ? 0.01 : 0.0015))));+ v.x += (mx - c.w / 2) * (1 / v.k - 1 / k); v.y += (my - c.h / 2) * (1 / v.k - 1 / k); v.k = k;+ this.kick();+ }, {passive: false});+ this.main.addEventListener('click', e => {+ if (e.target.closest('[data-lens]')) { MAP.lens = !MAP.lens; e.target.closest('[data-lens]').classList.toggle('on', MAP.lens); this.kick(); }+ if (e.target.closest('[data-fit]')) this.fit();+ if (e.target.closest('[data-whole]')) this.setWindow(null);+ if (e.target.closest('[data-play]')) this.play();+ });+ this.stripEvents();+ }+ hit(x, y) {+ let best = null, bd = Infinity;+ const scale = Math.max(0.55, Math.min(2.2, Math.sqrt(this.view.k)));+ for (let i = 0; i < this.nodes.length; i++) {+ const n = this.nodes[i], d = Math.hypot(n.sx - x, n.sy - y), r = this.radius(n, scale) + 5;+ if (d < r && d < bd) { bd = d; best = i; }+ }+ return best;+ }+ click(i) {+ if (i == null) { this.selected = null; this.showPane(); return; }+ const n = this.nodes[i];+ this.selected = i;+ if (n.kind === 'agent') go({as: n.user === S.route.as ? null : n.user}, true);+ else go({layout: 'chat', chat: n.chat});+ this.showPane();+ }+ onKey(e) {+ if (e.key === 'f' || e.key === 'F') { MAP.lens = !MAP.lens; $('[data-lens]', this.main).classList.toggle('on', MAP.lens); this.kick(); }+ if (e.key === '0') this.fit();+ }+ onAs() { this.kick(); this.showPane(); }+ onTheme() { this.colors(); this.kick(); this.drawStrip(); }++ // ---- live+ onEvents(ev) {+ let grew = false;+ for (const m of ev.lines) {+ if (m.kind !== 'message') { grew = true; continue; }+ const c = S.chats.get(m.chat), from = this.index.get('u:' + m.from);+ if (!c || from == null) { grew = true; continue; }+ const color = this.nodes[from].color || ink().text, now = performance.now();+ if (c.kind === 'direct') {+ const to = this.index.get('u:' + c.members.find(u => u !== m.from));+ if (to != null) this.pulses.push({legs: [[from, to]], t0: now, dur: 900, color});+ } else {+ const hub = this.index.get('c:' + c.id);+ if (hub == null) { grew = true; continue; }+ this.pulses.push({legs: [[from, hub]], t0: now, dur: 450, color});+ const out = c.members.filter(u => u !== m.from).slice(0, 30).map(u => this.index.get('u:' + u)).filter(i => i != null);+ if (out.length) this.pulses.push({legs: out.map(o => [hub, o]), t0: now + 450, dur: 650, color});+ }+ this.bump(ms(m.at) / 1000);+ }+ if (this.pulses.length > 1500) this.pulses = this.pulses.slice(-1500);+ if (grew) this.regrow();+ if (!this.winData) this.reweigh();+ this.kick();+ }+ regrow = throttle(() => { if (!this.dead) this.build(false); }, 3000);+ reweigh = throttle(() => { if (!this.dead) { this.weigh(); if (this.selected == null && this.hover == null) this.showPane(); } }, 1000);++ // ---- the pane: what is under the mouse, or the busiest when nothing is+ showPane() {+ const pane = $('#pane');+ pane.classList.add('open');+ const i = this.hover ?? this.selected;+ const n = i != null ? this.nodes[i] : null;+ const win = this.win ? ` from ${clock(this.win[0])} to ${clock(this.win[1])}` : '';+ if (!n) {+ const agents = this.nodes.filter(d => d.kind === 'agent').sort((p, q) => q.w - p.w).slice(0, 12);+ const chats = [...S.chats.values()].map(c => ({c, w: this.winData ? (this.winData[c.id]?.messages || 0) : c.messages})).sort((p, q) => q.w - p.w).slice(0, 12);+ pane.innerHTML = html`<div class="phead"><h2>Busiest${win}</h2></div><div class="scroll"><div class="note">Hover over anything for more. Click an agent to see everything as it sees it.</div>+<h3 class="ph3">Agents</h3>${agents.map(d => html`<div class="prow" data-person="${d.user}">${avatar(d.user, 'sm')}<span class="t">${name(d.user)}</span><span class="stamp">${num(d.w)}</span></div>`)}+<h3 class="ph3">Chats</h3>${chats.map(({c, w}) => html`<div class="prow" data-open="${c.id}"><span class="ico ${c.kind}">${KIND[c.kind]}</span><span class="t">${title(c, null)}</span><span class="stamp">${num(w)}</span></div>`)}</div>`.s;+ return;+ }+ if (n.kind === 'agent') {+ const u = user(n.user);+ const chats = this.edges.filter(e => e.a === i || e.b === i).map(e => ({c: S.chats.get(e.chat), w: e.w})).filter(x => x.c).sort((p, q) => q.w - p.w);+ pane.innerHTML = html`<div class="phead">${avatar(n.user, 'sm')}<h2>${u.name}</h2></div><div class="scroll"><div class="note">${u.id}${u.status ? html` · <i>${u.status}</i>` : ''}<br>${Object.entries(u.fields || {}).map(([k, v]) => html`<span class="field"><i>${k}</i> ${v}</span> `)}</div>+<div class="note"><b>${num(n.w)}</b> messages${win || ' in all'} · in ${plural(chats.length, 'chat')}</div>+<div class="note"><button class="btn primary" data-see-as="${n.user}">See as ${first(n.user)}</button> <button class="btn" data-person="${n.user}">Profile</button></div>+<h3 class="ph3">Where ${first(n.user)} writes</h3>${chats.slice(0, 30).map(({c, w}) => html`<div class="prow" data-open="${c.id}"><span class="ico ${c.kind}">${KIND[c.kind]}</span><span class="t">${title(c, n.user)}</span><span class="stamp">${num(w)}</span></div>`)}</div>`.s;+ } else {+ const c = S.chats.get(n.chat);+ const people = c.members.map(u => ({u, w: this.winData ? (this.winData[c.id]?.sent[u] || 0) : c.sent[u] || 0})).sort((p, q) => q.w - p.w);+ pane.innerHTML = html`<div class="phead"><span class="ico ${c.kind}">${KIND[c.kind]}</span><h2>${title(c, null)}</h2></div><div class="scroll">+<div class="note">${c.kind}${c.public ? ', public' : ''} · ${plural(c.members.length, 'member')} · <b>${num(n.w)}</b> messages${win}</div>+${c.last ? html`<div class="note"><b>${name(c.last.from)}</b> ${ago(c.last.at)}: ${c.last.deleted ? 'deleted message' : c.last.text || (c.last.files ? plural(c.last.files, 'file') : '')}</div>` : ''}+<div class="note"><button class="btn primary" data-open="${c.id}">Open the chat</button> <button class="btn" data-board="${c.id}">As a board</button></div>+<h3 class="ph3">Who writes here</h3>${people.slice(0, 40).map(({u, w}) => html`<div class="prow" data-person="${u}">${avatar(u, 'sm')}<span class="t">${name(u)}</span><span class="stamp">${num(w)}</span></div>`)}</div>`.s;+ }+ }++ // ---- the strip of messages over time+ async loadActivity() {+ const buckets = Math.max(60, Math.min(600, Math.round((this.strip.getBoundingClientRect().width || 900) / 4)));+ const r = await api('view_activity', {buckets, as: S.route.as || undefined});+ if (this.dead) return;+ this.act = r;+ const when = $$('.when span', this.main);+ when[0].textContent = r.start ? full(r.start) : '';+ when[1].textContent = r.end ? full(r.end) : '';+ this.drawStrip();+ }+ bump(t) {+ const a = this.act;+ if (!a || !a.start) return;+ const i = Math.floor((t - a.start) / a.width);+ if (i >= a.counts.length) { this.reloadActivity(); return; }+ if (i >= 0) a.counts[i]++;+ this.drawStripSoon();+ }+ reloadActivity = throttle(() => { if (!this.dead) this.loadActivity(); }, 10000);+ drawStripSoon = throttle(() => this.drawStrip(), 250);+ drawStrip() {+ const a = this.act, ctx = this.sctx, W = this.strip.w, H = this.strip.h;+ if (!W || !ctx) return;+ ctx.setTransform(this.dpr, 0, 0, this.dpr, 0, 0);+ ctx.clearRect(0, 0, W, H);+ if (!a || !a.counts.length) { ctx.fillStyle = ink().faint; ctx.font = '12px sans-serif'; ctx.fillText('No messages yet.', 8, H / 2); return; }+ const top = Math.max(...a.counts, 1), bw = W / a.counts.length, h = H - 16;+ for (let i = 0; i < a.counts.length; i++) {+ const t0 = a.start + i * a.width;+ const inWin = !this.win || (t0 + a.width >= this.win[0] && t0 <= this.win[1]);+ ctx.fillStyle = inWin ? ink().accent : ink().line2;+ const bh = a.counts[i] ? Math.max(1.5, h * a.counts[i] / top) : 0;+ ctx.fillRect(i * bw + 0.5, h - bh, Math.max(1, bw - 1), bh);+ }+ if (this.win) {+ const x0 = (this.win[0] - a.start) / (a.end - a.start) * W, x1 = (this.win[1] - a.start) / (a.end - a.start) * W;+ ctx.strokeStyle = ink().accentSoft; ctx.lineWidth = 1;+ ctx.strokeRect(x0 + 0.5, 0.5, Math.max(2, x1 - x0) - 1, h);+ const sel = $('.sel', this.main);+ sel.style.left = Math.min(W - 160, Math.max(0, x0)) + 'px';+ sel.textContent = `${clockS(this.win[0])} to ${clockS(this.win[1])}`;+ } else $('.sel', this.main).textContent = '';+ }+ stripEvents() {+ const c = this.strip;+ let start = null;+ const timeAt = x => { const a = this.act; return a.start + Math.max(0, Math.min(1, x / c.w)) * (a.end - a.start); };+ c.addEventListener('mousedown', e => { if (!this.act || !this.act.start) return; this.stop(); start = timeAt(e.offsetX); });+ c.addEventListener('mousemove', e => {+ if (start == null) return;+ const t = timeAt(e.offsetX);+ this.win = [Math.min(start, t), Math.max(start, t)];+ this.drawStrip();+ });+ addEventListener('mouseup', this.stripUp = e => {+ if (start == null) return;+ const t = this.win ? this.win : null;+ start = null;+ if (!t || t[1] - t[0] < (this.act.end - this.act.start) / 300) this.setWindow(null); else this.setWindow(t);+ });+ }+ setWindow(w) {+ this.win = w;+ this.drawStrip();+ if (!w) { this.winData = null; this.stop(); this.weigh(); this.showPane(); return; }+ this.fetchWindow();+ }+ fetchWindow = throttle(async () => {+ if (!this.win) return;+ const [from, to] = this.win;+ const r = await api('view_activity', {buckets: 1, from, to, as: S.route.as || undefined});+ if (this.dead || !this.win) return;+ this.winData = r.window;+ this.weigh();+ if (this.selected == null && this.hover == null) this.showPane();+ }, 120);+ play() {+ const a = this.act;+ if (!a || !a.start) return;+ if (this.playing) return this.stop();+ const span = a.end - a.start, width = this.win ? this.win[1] - this.win[0] : span / 12;+ let t = this.win && this.win[1] < a.end ? this.win[0] : a.start;+ $('[data-play]', this.main).textContent = '❚❚ Pause';+ this.playing = setInterval(() => {+ t += span / 200;+ if (t + width > a.end) { this.win = [a.end - width, a.end]; this.fetchWindow(); this.drawStrip(); return this.stop(); }+ this.win = [t, t + width];+ this.drawStrip();+ this.fetchWindow();+ }, 100);+ }+ stop() {+ clearInterval(this.playing); this.playing = null;+ const b = $('[data-play]', this.main);+ if (b) b.textContent = '▶ Replay';+ }+}++// ---------------------------------------------------------------------- theme++// The page follows the system's light or dark setting, unless one is picked with the button in the top bar. Every+// color lives in the stylesheet, and the canvases read theirs from it through ink().+const THEME_KEY = 'swarmchat.view.theme';+const ICON = body => `<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">${body}</svg>`;+const THEMES = {+ system: {next: 'light', title: 'Theme: same as the system. Click for light.', icon: ICON('<rect x="3" y="4" width="18" height="12" rx="2"/><path d="M8 20h8M12 16v4"/>')},+ light: {next: 'dark', title: 'Theme: light. Click for dark.', icon: ICON('<circle cx="12" cy="12" r="4"/><path d="M12 2v2M12 20v2M4.9 4.9l1.4 1.4M17.7 17.7l1.4 1.4M2 12h2M20 12h2M4.9 19.1l1.4-1.4M17.7 6.3l1.4-1.4"/>')},+ dark: {next: 'system', title: 'Theme: dark. Click to follow the system.', icon: ICON('<path d="M21 12.8A9 9 0 1 1 11.2 3a7 7 0 0 0 9.8 9.8z"/>')},+};+const systemDark = matchMedia('(prefers-color-scheme: dark)');+let INK = null;++function theme() { const t = localStorage.getItem(THEME_KEY); return THEMES[t] ? t : 'system'; }+function setTheme(t) {+ if (t === 'system') localStorage.removeItem(THEME_KEY); else localStorage.setItem(THEME_KEY, t);+ showTheme();+}+function showTheme() {+ const t = theme();+ if (t === 'system') delete document.documentElement.dataset.theme; else document.documentElement.dataset.theme = t;+ INK = null;+ const b = $('#theme');+ b.innerHTML = THEMES[t].icon;+ b.title = THEMES[t].title;+ if (S.view && S.view.onTheme) S.view.onTheme();+}+systemDark.addEventListener('change', () => { if (theme() === 'system') showTheme(); });++// The theme's colors as a canvas needs them, resolved to rgb(). Some are kept as bare "r, g, b" to add an alpha to.+function ink() {+ if (INK) return INK;+ const probe = document.body.appendChild(document.createElement('i'));+ const read = name => { probe.style.color = `var(${name})`; return getComputedStyle(probe).color; };+ const bare = name => read(name).match(/[\d.]+/g).slice(0, 3).join(', ');+ const t = theme();+ INK = {dark: t === 'dark' || (t === 'system' && systemDark.matches), text: read('--text'), faint: read('--faint'), line2: read('--line2'),+ accent: read('--accent'), accentSoft: read('--accent-soft'), channel: read('--channel'), group: read('--group'),+ chatName: read('--channel-soft'), bg: bare('--bg'), edge: bare('--edge'), edgeDm: bare('--edge-dm')};+ probe.remove();+ return INK;+}++// ---------------------------------------------------------------------- start++function login(error) {+ const box = $('#login');+ box.classList.add('on');+ $('.err', box).textContent = error || '';+ const form = $('form', box);+ form.onsubmit = async e => {+ e.preventDefault();+ TOKEN = form.token.value.trim();+ try {+ await loadSnapshot();+ localStorage.setItem('swarmchat.view.token', TOKEN);+ box.classList.remove('on');+ start();+ } catch (err) { $('.err', box).textContent = err.status === 401 ? 'That token was refused.' : err.message; }+ };+ form.token.focus();+}++let started = false;+async function start() {+ if (started) return;+ started = true;+ document.title = `swarmchat view · ${S.server.data ? S.server.data.split(/[\\/]/).filter(Boolean).pop() : ''}`;+ setLive(S.server.read_only ? 'ro' : 'on');+ S.since = Date.now();+ side.init();+ tips();+ document.addEventListener('click', onClick);+ new MutationObserver(watchPictures).observe(document.body, {childList: true, subtree: true});+ document.addEventListener('keydown', onKey);+ addEventListener('hashchange', route);+ await route();+ follow();+}++(async function boot() {+ showTheme();+ const found = location.hash.match(/(^#|&)token=([0-9a-f]{16,})/);+ if (found) {+ localStorage.setItem('swarmchat.view.token', found[2]);+ const rest = location.hash.replace(/(^#|&)token=[0-9a-f]+/, '$1').replace(/^#&/, '#');+ history.replaceState(null, '', location.pathname + (rest.length > 1 ? rest : ''));+ }+ TOKEN = localStorage.getItem('swarmchat.view.token') || '';+ if (!TOKEN) return login();+ try {+ await loadSnapshot();+ } catch (e) {+ return login(e.status === 401 ? 'The saved token was refused. The server may have a new one.' : e.message);+ }+ start();+})();diff --git a/swarmchat/page/gate.html b/swarmchat/page/gate.htmlnew file mode 100644index 0000000..d695bb3--- /dev/null+++ b/swarmchat/page/gate.html@@ -0,0 +1,30 @@+<!doctype html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1">+<meta name="referrer" content="no-referrer">+<title>swarmchat</title>+<link rel="icon" href="data:,">+<style>+:root { color-scheme: light dark; }+:root[data-theme="light"] { color-scheme: light; }+:root[data-theme="dark"] { color-scheme: dark; }+body { margin: 0; min-height: 100vh; display: grid; place-items: center; background: light-dark(#ffffff, #0f1115);+ color: light-dark(#1d1c1d, #e6e6e6); font: 15px/1.5 system-ui, -apple-system, sans-serif; }+form { display: grid; gap: 10px; width: min(360px, 90vw); }+input, button { font: inherit; padding: 8px 10px; border-radius: 8px; border: 1px solid light-dark(#d5d9e0, #2c313a);+ background: light-dark(#f7f7f9, #181b21); color: inherit; }+button { cursor: pointer; }+.err { color: light-dark(#dc2626, #f87171); min-height: 1.5em; margin: 0; }+</style>+</head>+<body>+<form>+ <input name="token" placeholder="Token" autocomplete="off" spellcheck="false" aria-label="Token">+ <button>Open</button>+ <p class="err"></p>+</form>+<!--JS-->+</body>+</html>diff --git a/swarmchat/page/gate.js b/swarmchat/page/gate.jsnew file mode 100644index 0000000..f63d8fe--- /dev/null+++ b/swarmchat/page/gate.js@@ -0,0 +1,33 @@+// What /view serves: a form for the view token. Only once the server accepts the token does it fetch the viewer and+// put it in place of this page, so the viewer itself is never shown to anyone without the token.+(function () {+ const KEY = 'swarmchat.view.token';+ const theme = localStorage.getItem('swarmchat.view.theme'); // the theme picked in the viewer, if any+ if (theme === 'light' || theme === 'dark') document.documentElement.dataset.theme = theme;+ const form = document.querySelector('form'), err = document.querySelector('.err');++ async function open(token) {+ const r = await fetch('/v1/view_page', {method: 'POST', body: '{}',+ headers: {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json'}});+ if (!r.ok) throw new Error(r.status === 401 ? 'That token was refused.' : `The server answered ${r.status}.`);+ const p = await r.json();+ localStorage.setItem(KEY, token); // where the viewer looks for it+ const doc = new DOMParser().parseFromString(p.html, 'text/html');+ document.replaceChild(document.adoptNode(doc.documentElement), document.documentElement);+ const script = document.createElement('script'); // runs: the page's policy allows this exact script+ script.textContent = p.js;+ document.body.appendChild(script);+ }++ form.onsubmit = e => {+ e.preventDefault();+ open(form.token.value.trim()).catch(x => { err.textContent = x.message; });+ };+ const linked = () => (location.hash.match(/(^#|&)token=([0-9a-f]{16,})/) || [])[2];+ addEventListener('hashchange', function retry() { // a link with another token, pasted into this tab+ if (linked()) open(linked()).then(() => removeEventListener('hashchange', retry), x => { err.textContent = x.message; });+ });+ const token = linked() || localStorage.getItem(KEY);+ if (token) open(token).catch(x => { err.textContent = x.message; });+ form.token.focus();+})();diff --git a/swarmchat/page/index.html b/swarmchat/page/index.htmlnew file mode 100644index 0000000..bba2d78--- /dev/null+++ b/swarmchat/page/index.html@@ -0,0 +1,52 @@+<!doctype html>+<html lang="en">+<head>+<meta charset="utf-8">+<meta name="viewport" content="width=device-width, initial-scale=1">+<meta name="referrer" content="no-referrer">+<title>swarmchat view</title>+<link rel="icon" href="data:,">+<!--CSS-->+</head>+<body class="layout-chat">+<div id="app">+ <header id="top">+ <a class="brand" href="#" data-home>swarmchat<span>view</span></a>+ <div id="stats"></div>+ <nav class="seg" id="layouts">+ <button data-layout="chat" title="Chats, like Slack (1)">Chat</button>+ <button data-layout="board" title="Posts and comments, like Reddit (2)">Board</button>+ <button data-layout="map" title="Everyone at once (3)">Map</button>+ </nav>+ <div class="grow"></div>+ <button id="as" class="as" title="See everything as one agent sees it"></button>+ <button id="find" class="find" title="Search people, chats and messages">Search<kbd>⌘K</kbd></button>+ <button id="theme" class="theme"></button>+ <span id="live" class="live" title=""></span>+ </header>+ <aside id="side">+ <div class="tabs" id="tabs">+ <button data-tab="chats">Chats</button>+ <button data-tab="people">People</button>+ </div>+ <input id="filter" placeholder="Filter" autocomplete="off" spellcheck="false">+ <div id="list" class="vlist"><div class="vspace"></div><div class="vrows"></div></div>+ </aside>+ <main id="main"></main>+ <section id="pane"></section>+</div>+<div id="overlay"></div>+<div id="tip"></div>+<div id="login">+ <form>+ <div class="brand big">swarmchat<span>view</span></div>+ <p>This page needs the view token. <code>swarmchat view --data DIR</code> opens it with the token already in the+ link.</p>+ <input name="token" placeholder="View token" autocomplete="off" spellcheck="false">+ <button>Open</button>+ <p class="err"></p>+ </form>+</div>+<!--JS-->+</body>+</html>diff --git a/swarmchat/policy.py b/swarmchat/policy.pynew file mode 100644index 0000000..1b5f479--- /dev/null+++ b/swarmchat/policy.py@@ -0,0 +1,165 @@+"""Who may do what: the roles, the actions each may take, and who may message whom and post where.++The roles:+- the super admin: whoever holds the server's admin token, the operator who sets the rules. It is not a user and+ never posts, and only it sets the rules.+- administrators: users the rules name.+- channel creators: users the rules name. Without that list, every user is one.+- a channel's managers: its creator, and anyone a manager or administrator names. They hold the role in that channel+ only.+- members: every user.++Rules come from DIR/policy.json, set with `swarmchat admin policy-set FILE`:++ {"admins": ["dana"], "channel_creators": ["maya", "priya"],+ "contacts": {"omar": ["maya", "priya"], "maya": ["ken"]}, "symmetric": true,+ "posters": {"#records": ["verifier"]},+ "allow": {"edit_admins": ["superadmin", "admin"]}}++Each action in ACTIONS lists the roles that may take it. "allow" replaces that list for any action, so the super+admin decides, for example, that administrators may name administrators, as above. Until the rules name+administrators the server is open: any member also does what a manager does, as in OPEN. Whatever the roles, nobody+acts in a chat they are not in, so private channels and direct chats stay private from administrators too.+`swarmchat admin role` changes admins and channel_creators for whoever "edit_admins" or "edit_channel_creators"+allows.++With contacts set, a user may start or write in a direct chat only if every other member is one of their contacts,+and may create a channel with people or add people to one only if those people are their contacts. Anyone may join a+public channel: making it public was the check. Users missing from the map may contact no one. "symmetric" (the+default) makes each contact work both ways.++Posters names channels, by name or id, where only the listed users may post or edit. Other members still read and+react. Anyone in any other channel may post in it: being added was the check. A channel named here by its name cannot+be renamed, so the rule cannot be dropped that way.++History says what a channel's members may read. "all" (the default, as in Slack) is everything, from before they+joined too. "while_member" is only what was sent while they were in it: not before they joined, and not while they+were out of it. It holds for reads, searches, threads, files, previews and counts. Direct chats keep their members,+so they are not affected.++For other rules, edit this class: each check returns None to allow, or the reason it refuses.+"""+from __future__ import annotations++SUPER, ADMIN, CREATOR, MANAGER, MEMBER = "superadmin", "admin", "channel_creator", "manager", "member"+ROLES = {SUPER: "the super admin", ADMIN: "an administrator", CREATOR: "someone chosen to create channels",+ MANAGER: "a channel manager", MEMBER: "a member"}++# action: (what it is, the roles that may take it). The last nine are taken in a chat, by its members.+ACTIONS = {+ "add_user": ("add users", [SUPER]),+ "remove_user": ("remove users", [SUPER]),+ "rotate_token": ("give someone else a new token", [SUPER]),+ "show_rules": ("see the rules", [SUPER, ADMIN]),+ "edit_admins": ("choose the administrators", [SUPER]),+ "edit_channel_creators": ("choose who may create channels", [SUPER, ADMIN]),+ "create_channel": ("create a channel", [CREATOR, ADMIN]),+ "add_members": ("add people to a channel", [MEMBER]),+ "remove_members": ("remove someone else from a channel", [MANAGER, ADMIN]),+ "rename_channel": ("rename a channel", [MANAGER, ADMIN]),+ "describe_channel": ("change a channel's description", [MANAGER, ADMIN]),+ "make_private": ("make a channel private", [MANAGER, ADMIN]),+ "make_public": ("make a private channel public", [ADMIN]), # its whole history opens to everyone, as in Slack+ "archive_channel": ("archive or unarchive a channel", [MANAGER, ADMIN]),+ "set_managers": ("choose a channel's managers", [MANAGER, ADMIN]),+ "delete_message": ("delete another person's message", [ADMIN]),+}+OPEN = {action: [MEMBER] for action in ("remove_members", "rename_channel", "describe_channel", "make_private",+ "make_public", "archive_channel", "set_managers")}+++def _users(rules: dict, key: str) -> set | None:+ users = rules.get(key)+ if users is None:+ return None+ if isinstance(users, str) or not isinstance(users, list) or not all(isinstance(u, str) for u in users):+ raise ValueError(f"{key} is a list of user ids.")+ return {u.strip().lower() for u in users}+++class Policy:+ def __init__(self, rules: dict | None = None):+ self.rules = rules or {}+ contacts = self.rules.get("contacts")+ self.contacts = None+ if contacts is not None:+ self.contacts = {user: set(others) for user, others in contacts.items()}+ if self.rules.get("symmetric", True):+ for user, others in list(self.contacts.items()):+ for other in others:+ self.contacts.setdefault(other, set()).add(user)+ posters = self.rules.get("posters")+ self.posters = None+ if posters is not None:+ self.posters = {chat.strip().lstrip("#").strip().lower(): set(users) for chat, users in posters.items()}+ self.admins, self.creators = _users(self.rules, "admins"), _users(self.rules, "channel_creators")+ self.history = self.rules.get("history", "all")+ if self.history not in ("all", "while_member"):+ raise ValueError('history is "all" or "while_member".')+ allow = self.rules.get("allow") or {}+ if not isinstance(allow, dict):+ raise ValueError("allow maps actions to lists of roles.")+ for action, roles in allow.items():+ if action not in ACTIONS:+ raise ValueError(f"There is no action '{action}'. The actions are {', '.join(ACTIONS)}.")+ if isinstance(roles, str) or not isinstance(roles, list) or not set(roles) <= set(ROLES):+ raise ValueError(f"allow.{action} is a list of roles: {', '.join(ROLES)}.")+ self.allow = {action: roles for action, (_, roles) in ACTIONS.items()} # in the order written, for refusals+ if self.admins is None:+ self.allow.update(OPEN)+ self.allow.update((action, list(dict.fromkeys(roles))) for action, roles in allow.items())++ def admin(self, user: str) -> bool:+ """Whether `user` is an administrator."""+ return self.admins is not None and user in self.admins++ def roles(self, user: str | None, managers=None) -> set:+ """The roles of `user`, or of the super admin for None. `managers` are the managers of the chat acted in."""+ if user is None:+ return {SUPER}+ roles = {MEMBER}+ if self.admin(user):+ roles.add(ADMIN)+ if self.creators is None or user in self.creators:+ roles.add(CREATOR)+ if managers is not None and user in managers:+ roles.add(MANAGER)+ return roles++ def check(self, action: str, roles: set) -> str | None:+ allowed = self.allow[action]+ if roles.intersection(allowed):+ return None+ what = ACTIONS[action][0]+ if not allowed:+ return f"No one may {what}."+ return f"Only {' or '.join(ROLES[r] for r in allowed)} may {what}."++ def _reach(self, user: str, others) -> str | None:+ if self.contacts is None:+ return None+ blocked = sorted(o for o in others if o != user and o not in self.contacts.get(user, ()))+ return f"{user} may not contact {', '.join(blocked)}." if blocked else None++ def send(self, user: str, members, channel: str | None, chat_id: str | None = None) -> str | None:+ """Posting or editing. `channel` is the channel's name, or None in a direct chat."""+ if not channel:+ return self._reach(user, members)+ if self.posters is None:+ return None+ allowed = self.posters.get(channel.lower(), self.posters.get(chat_id or ""))+ if allowed is None or user in allowed:+ return None+ return f"Only {', '.join(sorted(allowed))} may post in #{channel}."++ def react(self, user: str, members, channel: str | None) -> str | None:+ return None if channel else self._reach(user, members)++ def create(self, user: str, members) -> str | None:+ return self._reach(user, members)++ def add(self, user: str, members, new) -> str | None:+ return self._reach(user, new)++ def join(self, user: str, members) -> str | None:+ return Nonediff --git a/swarmchat/server.py b/swarmchat/server.pynew file mode 100644index 0000000..4552631--- /dev/null+++ b/swarmchat/server.py@@ -0,0 +1,548 @@+"""HTTP transport: POST /v1/<request> with a JSON body and `Authorization: Bearer <token>`. GET /v1/health answers+without a token. GET /view is a form for the view token, and the viewer itself (view_page) takes the token too.+Each request runs in its own thread, so a held wait never blocks anyone else.++A request's token is checked before its body is read, so only a known token can make the server read one. Each request+has a deadline: its line and headers must arrive within request_timeout, and its body within that plus its size at+min_rate. Its headers may take at most HEADERS bytes. At most max_connections are open at once: when they are all+taken, the one that has waited longest for a request is closed to make room, so connections that send nothing, or+send their headers slowly, cannot keep anyone out. A request under way is never closed, and one user may have at most+max_requests under way.+"""+from __future__ import annotations++import contextlib+import gzip+import io+import json+import os+import re+import select+import signal+import socket+import socketserver+import ssl+import sys+import threading+import time+import traceback+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer++from . import DATA_DIR, NAME, __version__, api, view+from .files import CHUNK, mime+from .store import Error, Store++HEADER_LIMIT = 64 * 1024 # the JSON line that starts an upload+HEADERS = 64 * 1024 # a request's line and headers, in all. Python's parser alone allows 100 lines of 64 KB.+READ_ONLY = f"This server only shows a data folder to the viewer. Start `{NAME} serve` to use it."+VIEW = {*view.METHODS, "view_feed", "view_download"} # the requests the view token may make+CONTROL = re.compile(r"[\x00-\x1f\x7f]")+++def ready(sock, timeout: float, read: bool = True, write: bool = False) -> bool:+ """Whether a socket can be read (or written) within `timeout` seconds. poll() where there is one: select() fails+ on descriptors above 1023, which a server holding a thousand waits has."""+ if hasattr(select, "poll"):+ poller = select.poll()+ poller.register(sock, (select.POLLIN if read else 0) | (select.POLLOUT if write else 0))+ return bool(poller.poll(timeout * 1000))+ return any(select.select([sock] if read else [], [sock] if write else [], [], timeout))+++def open_files() -> int | None:+ """Raises this process's limit on open files as far as the system allows, up to 65,536, and returns it (None+ where there is no such limit). Each connection is an open file, and so is each chat's log while it is written."""+ try:+ import resource+ except ImportError: # Windows+ return None+ soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)+ for want in (65536, 32768, 16384, 10240, 4096, hard):+ if want == resource.RLIM_INFINITY or want <= soft:+ break+ if hard == resource.RLIM_INFINITY or want <= hard:+ try:+ resource.setrlimit(resource.RLIMIT_NOFILE, (want, hard))+ return want+ except (ValueError, OSError): # above what this system allows one process: try less+ continue+ return soft+++class Server(ThreadingHTTPServer):+ daemon_threads = True+ # Lets a restarted server take its port back at once. Not on Windows, where the same option lets a second server+ # take a port another one is already listening on.+ allow_reuse_address = os.name != "nt"+ request_queue_size = 4096 # the listen backlog. The default of 5 drops agents that connect at the same moment.++ def __init__(self, address, handler, max_connections: int):+ self.max_connections, self.refused_at = max_connections, 0.0+ self.slots = threading.BoundedSemaphore(max_connections)+ self.idle, self.idle_lock = {}, threading.Lock() # connection -> since when it has waited for a request+ body = json.dumps({"error": f"The server is busy: {max_connections} connections are open. Try again "+ "shortly."}).encode()+ self.busy = (f"HTTP/1.1 503 Service Unavailable\r\nContent-Type: application/json\r\nContent-Length: "+ f"{len(body)}\r\nConnection: close\r\n\r\n").encode() + body+ super().__init__(address, handler)++ def server_bind(self):+ # HTTPServer would also look up the host's full name in DNS, which nothing here uses. Where reverse DNS is slow,+ # that lookup alone held up every start by about 30 seconds.+ socketserver.TCPServer.server_bind(self)+ self.server_name, self.server_port = self.server_address[:2]++ def process_request(self, request, client_address):+ if not self.slots.acquire(blocking=False) and not (self.evict() and self.slots.acquire(timeout=1)):+ return self.refuse(request)+ try:+ super().process_request(request, client_address)+ except BaseException: # no thread was started, so none will give the slot back+ self.slots.release()+ raise++ def waiting(self, connection):+ """The connection waits for a request: it may be closed to make room."""+ with self.idle_lock:+ self.idle[connection] = time.monotonic()++ def working(self, connection):+ """The connection has a request under way, or is gone: it is not closed to make room."""+ with self.idle_lock:+ self.idle.pop(connection, None)++ def evict(self) -> bool:+ """Closes the connection that has waited longest for a request: one that sends nothing, or its headers too+ slowly, or that a client kept open between requests (that client simply connects again). Its thread then ends+ and gives back its slot."""+ with self.idle_lock:+ if not self.idle:+ return False+ connection = min(self.idle, key=self.idle.get)+ del self.idle[connection]+ with contextlib.suppress(OSError):+ socket.socket.shutdown(connection, socket.SHUT_RDWR) # wakes its thread, as a closed connection+ return True++ def refuse(self, request):+ """Every connection has a request under way: a short answer where it can go out at once, and the connection+ closed."""+ if not isinstance(request, ssl.SSLSocket): # TLS would need a handshake first+ with contextlib.suppress(OSError):+ request.setblocking(False)+ request.send(self.busy)+ self.shutdown_request(request)+ if time.monotonic() - self.refused_at > 60:+ self.refused_at = time.monotonic()+ sys.stderr.write(f"Refusing connections: {self.max_connections} are open (max_connections).\n")++ def process_request_thread(self, request, client_address):+ try:+ self.waiting(request)+ # Handshaking in accept() lets one silent peer block every new connection.+ if isinstance(request, ssl.SSLSocket) and not self.handshake(request):+ self.shutdown_request(request)+ return+ super().process_request_thread(request, client_address)+ finally:+ self.working(request)+ self.slots.release()++ def handshake(self, request) -> bool:+ """The TLS handshake, done within request_timeout in all, so a peer that sends it a byte at a time is cut off+ rather than holding a thread."""+ timeout = self.RequestHandlerClass.timeout+ deadline = time.monotonic() + timeout+ try:+ request.setblocking(False)+ while True:+ try:+ request.do_handshake()+ return True+ except ssl.SSLWantReadError:+ wanted = (True, False)+ except ssl.SSLWantWriteError:+ wanted = (False, True)+ left = deadline - time.monotonic()+ if left <= 0 or not ready(request, left, *wanted):+ return False+ except OSError:+ return False+ finally:+ with contextlib.suppress(OSError):+ request.settimeout(timeout)++ def handle_error(self, request, client_address):+ # A client that hangs up (a killed command, a closed laptop) is normal: no traceback for that.+ if isinstance(sys.exc_info()[1], (ConnectionError, socket.timeout, ssl.SSLError)):+ if self.RequestHandlerClass.verbose:+ sys.stderr.write(f"{client_address[0]} hung up: {sys.exc_info()[1]}\n")+ return+ super().handle_error(request, client_address)+++class Reader(socket.SocketIO):+ """A connection's reads, each given only the time left before its handler's deadline. The socket's own timeout+ is for silence: a client that sends a byte every few seconds would never meet it."""++ def __init__(self, sock, handler):+ super().__init__(sock, "rb")+ self.handler = handler++ def readinto(self, b):+ n = self._read(b)+ if self.handler.heading and n:+ self.handler.heard += n+ if self.handler.heard > HEADERS + len(b): # a read may run on into the body, up to its buffer's size+ raise ConnectionAbortedError("The request's headers are too large.")+ return n++ def _read(self, b):+ deadline = self.handler.deadline+ if deadline is None:+ return super().readinto(b)+ left = deadline - time.monotonic()+ if left <= 0:+ raise socket.timeout("The request did not arrive in time.")+ if left >= 0.9 * self.handler.timeout: # near enough to the socket's own timeout, which then stands+ return super().readinto(b)+ self._sock.settimeout(left)+ try:+ return super().readinto(b)+ finally:+ self._sock.settimeout(self.handler.timeout)+++class Handler(BaseHTTPRequestHandler):+ protocol_version = "HTTP/1.1"+ disable_nagle_algorithm = True # without this, each reply on Linux waits ~40 ms for the client's delayed ACK+ server_version = f"{NAME}/{__version__}"+ store: Store = None+ verbose = False+ unread = False # whether the reply went out before the request's body was read+ deadline = None # when what the client is sending must have arrived, or None while nothing is due from it+ heading, heard = False, 0 # whether the request's line and headers are being read, and their bytes so far+ LINGER_BYTES, LINGER_SECONDS = 8 << 20, 2.0++ def setup(self):+ super().setup()+ self.rfile.close() # replaced by one that keeps to the request's deadline+ self.rfile = io.BufferedReader(Reader(self.connection, self))++ def handle_one_request(self):+ self.deadline = time.monotonic() + self.timeout # the request line and headers, all within request_timeout+ self.heading, self.heard = True, 0+ self.server.waiting(self.connection)+ super().handle_one_request()++ def parse_request(self):+ parsed = super().parse_request()+ self.heading = False # the headers are in: from here a request is under way, never closed to make room+ self.server.working(self.connection)+ return parsed++ def send_header(self, keyword, value):+ # A line break in a value would end the header there, and the rest would be read as more headers or a body.+ if CONTROL.search(str(value)):+ raise ValueError(f"The {keyword} header holds a control character.")+ super().send_header(keyword, value)++ def finish(self):+ self.server.working(self.connection)+ super().finish()+ if self.unread:+ self.linger()++ def linger(self):+ """Stops writing, then reads and drops what the client is still sending, before the server closes. Closing with+ bytes unread makes the system reset the connection, and a client still sending a refused body would then lose+ the reply already written to it. Bounded, so a huge upload is cut off rather than read to its end."""+ sock, left = self.connection, self.LINGER_BYTES+ deadline = time.monotonic() + self.LINGER_SECONDS+ try:+ sock.shutdown(socket.SHUT_WR)+ while left > 0 and time.monotonic() < deadline:+ sock.settimeout(max(0.01, deadline - time.monotonic()))+ chunk = sock.recv(min(CHUNK, left))+ if not chunk:+ break+ left -= len(chunk)+ except OSError: # the client hung up, or the time ran out+ pass++ def client_gone(self) -> bool:+ """Whether the caller has hung up. Nothing is due from a client while it waits for a reply, so a readable+ connection means it was closed. Peeks at the raw socket, which works under TLS too."""+ try:+ if not ready(self.connection, 0):+ return False+ return socket.socket.recv(self.connection, 1, socket.MSG_PEEK) == b""+ except (OSError, ValueError):+ return True++ def log_message(self, fmt, *args):+ if self.verbose:+ sys.stderr.write(f"{self.address_string()} {fmt % args}\n")++ def reply(self, status: int, body: dict):+ data = json.dumps(body, ensure_ascii=False, separators=(",", ":")).encode()+ packed = len(data) > 2048 and "gzip" in (self.headers.get("Accept-Encoding") or "")+ if packed: # browsers ask for it. The CLI does not, so agents' replies stay as they were.+ data = gzip.compress(data, 4)+ if self.unread: # a body left unread must not be taken for the next request+ self.close_connection = True+ self.send_response(status)+ self.send_header("Content-Type", "application/json")+ if packed:+ self.send_header("Content-Encoding", "gzip")+ self.send_header("Content-Length", str(len(data)))+ if self.close_connection:+ self.send_header("Connection", "close")+ self.end_headers()+ self.wfile.write(data)++ def do_GET(self):+ path = self.path.split("?", 1)[0]+ if path == "/v1/health":+ limits = {k: self.store.settings[k] for k in ("max_text", "max_attachment", "max_attachments")}+ self.reply(200, {"ok": True, "name": NAME, "version": __version__, "limits": limits})+ elif path in ("/view", "/view/"):+ try:+ self.send_page(view.gate())+ except OSError as e: # an install without the page's files+ self.reply(500, {"error": f"The viewer's page is missing from this install: {e}"})+ elif path == "/":+ self.send_response(302)+ self.send_header("Location", "/view")+ self.send_header("Content-Length", "0")+ self.end_headers()+ else:+ self.reply(404, {"error": "Not found."})++ def send_page(self, page: dict):+ if self.headers.get("If-None-Match") == page["etag"]:+ self.send_response(304)+ self.send_header("ETag", page["etag"])+ self.send_header("Content-Length", "0")+ self.end_headers()+ return+ packed = "gzip" in (self.headers.get("Accept-Encoding") or "")+ body = page["gzip"] if packed else page["body"]+ self.send_response(200)+ self.send_header("Content-Type", "text/html; charset=utf-8")+ if packed:+ self.send_header("Content-Encoding", "gzip")+ self.send_header("Content-Length", str(len(body)))+ self.send_header("ETag", page["etag"])+ self.send_header("Cache-Control", "no-cache") # asked again each time, answered 304 while unchanged+ self.send_header("Content-Security-Policy", page["csp"])+ self.send_header("Referrer-Policy", "no-referrer")+ self.send_header("X-Content-Type-Options", "nosniff")+ self.end_headers()+ self.wfile.write(body)++ def do_POST(self):+ name = self.path[len("/v1/"):] if self.path.startswith("/v1/") else ""+ self.unread = True+ try:+ lengths = self.headers.get_all("Content-Length", [])+ length_text = lengths[0].strip() if lengths else "0"+ if (self.headers.get("Transfer-Encoding") is not None or len(lengths) > 1+ or not length_text.isascii() or not length_text.isdecimal()):+ self.close_connection = True+ raise Error(400, "Use one nonnegative Content-Length and no Transfer-Encoding.")+ length = int(length_text)+ auth = self.headers.get("Authorization") or ""+ me = self.caller(name, auth[len("Bearer "):].strip() if auth.startswith("Bearer ") else "")+ with self.store.request(me) if me else contextlib.nullcontext(): # one user's requests under way, counted+ self.answer(name, me, length)+ except (ConnectionError, socket.timeout): # the caller hung up, or did not send the request in time+ self.close_connection = True+ except Error as e:+ self.reply(e.status, {"error": str(e)})+ except (ValueError, TypeError, KeyError) as e:+ self.reply(400, {"error": f"Bad request: {e}"})+ except Exception: # noqa: BLE001 - one bad request must not take the server down+ traceback.print_exc()+ self.reply(500, {"error": "Server error. See the server's log."})++ def answer(self, name: str, me: str | None, length: int):+ """Reads the request's body, within its deadline, and replies."""+ self.deadline = time.monotonic() + self.timeout + length / self.store.settings["min_rate"]+ if name == "send_files":+ self.close_connection = True # one upload per connection: an unread body is never parsed as a request+ result = self.send_files(me, length)+ self.unread, self.deadline = False, None+ return self.reply(200, result)+ if length > self.store.settings["max_body"]:+ raise Error(413, f"The request is larger than {self.store.settings['max_body']} bytes.")+ body = self.rfile.read(length)+ self.unread, self.deadline = False, None+ if len(body) != length:+ self.close_connection = True+ raise Error(400, "The request body ended before Content-Length bytes arrived.")+ params = json.loads(body or b"{}")+ if not isinstance(params, dict):+ raise Error(400, "The request body must be a JSON object.")+ if name in VIEW:+ if name == "view_download":+ return self.send_file(*view.download(self.store, params))+ if name == "view_feed":+ result = view.view_feed(self.store, params, gone=self.client_gone)+ else:+ result = view.METHODS[name](self.store, params)+ elif name in api.ADMIN: # me is None for the super admin's token+ result = api.ADMIN[name](self.store, me, params)+ else:+ if params.get("ack"): # messages a previous wait handed over, now confirmed received+ api.confirm(self.store, me, params.pop("ack"))+ if name == "download":+ return self.send_file(*api.download(self.store, me, params))+ if name == "wait":+ result = api.wait(self.store, me, params, gone=self.client_gone)+ else:+ result = api.METHODS[name](self.store, me, params)+ self.reply(200, result)++ def caller(self, name: str, token: str) -> str | None:+ """Who makes this request, from its token, checked before its body is read. The viewer's requests take the view+ token or the admin token, the admin requests the admin token (None) or a user's, and the rest a user's."""+ if name in VIEW:+ if not self.store.can_view(token):+ raise Error(401, "This request needs the view token or the admin token.")+ return None+ if self.store.read_only:+ raise Error(403, READ_ONLY)+ if name in api.ADMIN and self.store.is_admin(token):+ return None # the super admin. Otherwise a user, whom the rules may allow.+ if name not in api.ADMIN and name not in api.METHODS and name not in ("download", "send_files"):+ raise Error(404, f"There is no request named '{name}'.")+ me = self.store.me(token)+ if not me:+ raise Error(401, "This request needs the admin token, or the token of a user the rules allow."+ if name in api.ADMIN else f"Unknown token. Set {NAME.upper()}_TOKEN or run `{NAME} login`.")+ return me++ def send_files(self, me: str, length: int) -> dict:+ """A message with files: one line of JSON (like send's parameters, plus the files' names and sizes), then+ the files' bytes back to back. Streamed to disk, never held in memory."""+ settings = self.store.settings+ if length > settings["max_attachments"] * settings["max_attachment"] + HEADER_LIMIT:+ raise Error(413, "The request is larger than the attachment limits allow.")+ consumed = 0++ def read(n: int) -> bytes:+ nonlocal consumed+ chunk = self.rfile.read(n)+ consumed += len(chunk)+ return chunk++ try:+ line = self.rfile.readline(HEADER_LIMIT + 1)+ consumed += len(line)+ if len(line) > HEADER_LIMIT or not line.endswith(b"\n"):+ raise Error(400, "Start with one line of JSON describing the message and its files.")+ try:+ header = json.loads(line)+ except ValueError:+ raise Error(400, "The first line must be JSON.") from None+ if not isinstance(header, dict):+ raise Error(400, "The first line must be a JSON object.")+ if header.get("ack"):+ api.confirm(self.store, me, header.pop("ack"))+ return api.send_files(self.store, me, header, read, length - len(line))+ except Error:+ # Refused before all the bytes were read: read the rest, within the limits checked above, so the client+ # can finish sending and get this answer instead of a reset connection.+ left = length - consumed+ while left > 0:+ chunk = self.rfile.read(min(CHUNK, left))+ if not chunk:+ break+ left -= len(chunk)+ raise++ def send_file(self, attachment: dict, path: str):+ """An attachment's bytes as the reply, streamed. Its name and type travel in the X-Attachment header."""+ with open(path, "rb") as f:+ self.send_response(200)+ self.send_header("Content-Type", mime(attachment.get("type")) or "application/octet-stream")+ self.send_header("Content-Length", str(attachment["size"]))+ self.send_header("X-Attachment", json.dumps({k: v for k, v in attachment.items() if k != "sha256"}))+ self.end_headers()+ while True:+ chunk = f.read(CHUNK)+ if not chunk:+ break+ self.wfile.write(chunk)+++def make_server(settings: dict, read_only: bool = False):+ """(the HTTP server, its store), ready for serve_forever(). settings come from config.load(). Tests use this.+ read_only serves a folder to the viewer only, writing nothing to it (`swarmchat view --data DIR` with no server)."""+ store = Store(settings["data"], settings, read_only)+ handler = type("BoundHandler", (Handler,), {"store": store, "verbose": settings["verbose"],+ "timeout": settings["request_timeout"]})+ files = open_files()+ connections = settings["max_connections"] if files is None else min(settings["max_connections"], files // 2)+ if connections < settings["max_connections"]: # the other half is for logs, uploads and the rest+ print(f"max_connections is {connections}: this system lets the server open {files} files.", file=sys.stderr)+ httpd = None+ try:+ httpd = Server((settings["host"], settings["port"]), handler, connections)+ if settings["tls_cert"]:+ context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)+ context.load_cert_chain(settings["tls_cert"], settings["tls_key"])+ httpd.socket = context.wrap_socket(httpd.socket, server_side=True, do_handshake_on_connect=False)+ except BaseException:+ if httpd:+ httpd.server_close()+ store.close()+ raise+ # Where this server listens, so `swarmchat admin --data DIR` reaches it without being told the address.+ host = {"0.0.0.0": "127.0.0.1", "::": "::1", "": "127.0.0.1"}.get(settings["host"], settings["host"])+ host = f"[{host}]" if ":" in host else host+ httpd.url = f"{'https' if settings['tls_cert'] else 'http'}://{host}:{httpd.server_address[1]}"+ if not read_only:+ with open(store.file("address"), "w", encoding="utf-8") as f:+ f.write(httpd.url + "\n")+ return httpd, store+++def save_read_positions(store: Store, stopped: threading.Event):+ while not stopped.wait(store.settings["save_every"]):+ try:+ store.flush()+ except OSError as exc:+ print(f"Could not save read positions; will retry: {exc}", file=sys.stderr, flush=True)+++def serve(settings: dict):+ httpd, store = make_server(settings)+ stopped = threading.Event()+ saver = threading.Thread(target=save_read_positions, args=(store, stopped), daemon=True)+ saver.start()++ def stop(signum, frame): # a normal stop (SIGTERM) saves read positions, like Ctrl-C+ raise KeyboardInterrupt++ signal.signal(signal.SIGTERM, stop)+ if hasattr(signal, "SIGBREAK"): # Windows has no SIGTERM between processes: Ctrl-Break is its normal stop+ signal.signal(signal.SIGBREAK, stop)+ signal.signal(signal.SIGINT, signal.default_int_handler) # a server started in the background ignores Ctrl-C+ shown = ", ".join(f"{k}={v}" for k, v in store.settings.items() if k not in ("data", "host", "port", "tls_cert", "tls_key"))+ print(f"{NAME} {__version__}: {len(store.users)} users, {len(store.chats)} chats from {settings['data']}, serving "+ f"{'https' if settings['tls_cert'] else 'http'}://{settings['host']}:{httpd.server_address[1]}", flush=True)+ print(f"Settings: {shown}", flush=True)+ print(f"The admin token is in {os.path.join(settings['data'], 'admin.token')}", flush=True)+ print(f"To watch everything: {NAME} view" + ("" if settings["data"] == DATA_DIR else f" --data {settings['data']}"),+ flush=True)+ try:+ httpd.serve_forever()+ except KeyboardInterrupt:+ pass+ finally:+ stopped.set()+ saver.join()+ httpd.server_close()+ store.close()diff --git a/swarmchat/store.py b/swarmchat/store.pynew file mode 100644index 0000000..acfb48e--- /dev/null+++ b/swarmchat/store.py@@ -0,0 +1,1249 @@+"""Storage: plain files on disk, and the index in memory that requests are answered from.++ DIR/ readable only by its owner, like everything in it+ server.json format version+ admin.token the super admin's token, readable only by its owner+ view.token a token that may only look: it opens `swarmchat view`, readable only by its owner+ address where the server last started listening, for `swarmchat admin --data DIR`+ policy.json optional rules, see policy.py+ users/<user>.json profile, status and the hash of the user's token+ retired/<user>.json reserved ids of removed accounts (no token or profile)+ blobs/<ab>/<sha256> attachment contents, each stored once (see files.py)+ uploads/ uploads in progress+ chats/<chat>/meta.json name, description, members, managers, public or not, archived or not, who+ created it and when+ chats/<chat>/log/000001.jsonl every event in the chat, one JSON object per line, append-only. Deleting a+ message blanks its words in its line, and in its edits' lines+ chats/<chat>/redact.journal the lines a deletion is rewriting, while it does: a crash part way through is+ finished at the next start+ state/<user>.json read position and notification level for each chat++Each line of a chat log has a sequence number (seq), one higher than the line before. The kinds of line:+- "message"+- "edit", "delete" and "react", which point at a message's seq and are applied on top of it when it is read+- "created", "added", "removed", "renamed", "described", "managers", "visibility", "archived" and "unarchived",+ which keep the chat's history+- "read", written when someone's read position moves forward: the read receipts+A message's id is "<chat>:<seq>". A new log file starts every `segment` lines (a setting fixed when the folder is+made, kept in server.json), so a seq alone says which file and line hold an event.+"""+from __future__ import annotations++import bisect+import collections+import difflib+import hashlib+import contextlib+import itertools+import json+import os+import re+import secrets+import shutil+import threading+import time+from array import array+from datetime import datetime, timezone++from .config import DEFAULTS+from .files import Blobs+from .locking import DirectoryLock+from .policy import Policy++FORMAT = 1+LEVELS = ("all", "mentions", "none")+FULL = "The server's disk is nearly full, so it takes no new messages or files for now. Ask its operator to free space."+USER_ID = re.compile(r"[a-z0-9][a-z0-9._-]{0,39}")+MESSAGE_ID = re.compile(r"([0-9a-f]{6,32}):(\d+)") # chat ids were 6 hex digits, now 16+++def now() -> str:+ return datetime.now(timezone.utc).isoformat(timespec="milliseconds").replace("+00:00", "Z")+++_MINUTES = {} # "2026-10-02T15:03" -> that minute in seconds since 1970+++def epoch(at: str) -> float:+ """Seconds since 1970 from a time as now() writes it, 2026-10-02T15:03:09.424Z. It runs for every message at+ start, so that form is taken apart by hand, with each minute worked out once: a third of the time of parsing it."""+ if len(at) == 24 and at[-1] == "Z" and at[16] == ":":+ minute = _MINUTES.get(at[:16])+ if minute is None:+ minute = _MINUTES[at[:16]] = datetime.fromisoformat(at[:16] + ":00+00:00").timestamp()+ return minute + float(at[17:23])+ moment = datetime.fromisoformat(at.replace("Z", "+00:00"))+ return (moment if moment.tzinfo else moment.replace(tzinfo=timezone.utc)).timestamp()+++def digest(token: str) -> str:+ return hashlib.sha256(token.encode()).hexdigest()+++def new_token(size: int = 24) -> str:+ """A secret of `size` random bytes, in hex: it never starts with '-', so a command line never takes it for a flag."""+ return secrets.token_hex(size)+++def read_json(path: str, default=None):+ try:+ with open(path, encoding="utf-8") as f:+ return json.load(f)+ except FileNotFoundError:+ return default+++def write_json(path: str, data, mode: int | None = None, durable: bool = False):+ """Replaces a small file atomically: a reader sees the old version or the new one, never half of one. durable+ forces it to disk before returning, so it survives a power cut too."""+ encoded = json.dumps(data, ensure_ascii=False, indent=1).encode("utf-8")+ tmp = f"{path}.{threading.get_ident()}.tmp"+ try:+ with os.fdopen(os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, mode or 0o600), "wb") as f:+ f.write(encoded)+ if durable:+ f.flush()+ os.fsync(f.fileno())+ if mode is not None:+ os.chmod(tmp, mode)+ os.replace(tmp, path)+ if durable and os.name != "nt": # the new name, too, must reach the disk+ fd = os.open(os.path.dirname(path) or ".", os.O_RDONLY)+ try:+ os.fsync(fd)+ finally:+ os.close(fd)+ finally:+ if os.path.exists(tmp):+ os.remove(tmp)+++class Error(Exception):+ """A request the store refuses. status is the HTTP status: 400 bad request, 403 not allowed, 404 not found."""++ def __init__(self, status: int, message: str):+ super().__init__(message)+ self.status = status+++class Chat:+ """One chat: its metadata, its log, and what is kept in memory to answer requests without reading the log."""++ def __init__(self, root: str, meta: dict, segment: int):+ self.meta, self.segment = meta, segment+ self.dir = os.path.join(root, "chats", meta["id"])+ self.lock = threading.Lock() # held while appending+ self.seq = 0 # seq of the last event+ self.at = meta["created_at"] # time of the last message or membership change+ self.offsets = [] # one array per log file: the byte offset of each line+ self.messages = array("I") # seqs of message events, increasing+ self.senders = [] # the sender of each entry in self.messages+ self.times = array("d") # the time of each entry in self.messages, in seconds since 1970. Indexed last, so+ # its length is how many messages a reader without the lock can count on.+ self.system = array("I") # seqs of the lines that made the chat or changed its members, name, description+ # or visibility+ self.mentions = {} # user -> array of seqs of messages that mention them+ self.changes = {} # message seq -> {"text", "edited_at", "deleted", "reactions"}+ self.edits = {} # message seq -> seqs of the edits to it+ self.spans = {} # user -> [[seq of the line that let them in, seq of the line that took them out, or None]]+ self.reads = {} # user -> (array of the seqs they read up to, the time of each), increasing+ self.files = {} # attachment id -> (message seq, the attachment as stored), for messages not deleted+ self.parent = {} # seq of a reply -> seq of the message it answers+ self.replies = {} # seq of a message -> seqs of the replies to it, in order+ self.last = None # the newest message event, as stored+ self.writer, self.writer_file = None, 0+ self.failed = False # A failed rollback requires reopening/recovering this chat.+ self.listener = None # called with (chat, event) after each new line: the server-wide feed+ self.cache = {} # what the viewer works out from the indexes above, kept with the chat (see view.py)++ id = property(lambda self: self.meta["id"])+ name = property(lambda self: self.meta.get("name"))+ description = property(lambda self: self.meta.get("description") or "") # what a channel is for, or ""+ public = property(lambda self: bool(self.meta.get("public"))) # a channel anyone may find and join+ archived = property(lambda self: bool(self.meta.get("archived"))) # a channel kept to read, where nothing changes+ members = property(lambda self: self.meta["members"])++ @property+ def managers(self) -> list:+ """A channel's managers: its creator, and whoever a manager or administrator names. Leaving ends the role."""+ if not self.name:+ return []+ return [u for u in self.meta.get("managers", [self.meta["created_by"]]) if u in self.members]++ def path(self, file_no: int) -> str:+ return os.path.join(self.dir, "log", f"{file_no:06d}.jsonl")++ def load(self, read_only: bool = False):+ if not read_only:+ os.makedirs(os.path.join(self.dir, "log"), 0o700, exist_ok=True)+ journal = os.path.join(self.dir, "redact.journal")+ writes = read_json(journal)+ if writes is not None: # a deletion's rewrite that a crash cut off: its lines may be half old, half new+ if read_only:+ raise SystemExit(f"Chat {self.id} has a deletion that a crash cut off. Start the server on this folder "+ f"once to finish it.")+ self._overwrite(writes, True)+ os.remove(journal)+ file_no = 1+ while os.path.exists(self.path(file_no)):+ offsets, pos = array("Q"), 0+ with open(self.path(file_no), "rb") as f:+ for line in f:+ if not line.endswith(b"\n"): # half a line, left by a crash in the middle of a write+ break+ offsets.append(pos)+ pos += len(line)+ event = json.loads(line)+ if event["seq"] != self.seq + 1:+ raise Error(500, f"Corrupt log {self.path(file_no)}: expected sequence {self.seq + 1}.")+ self.index(event)+ if os.path.getsize(self.path(file_no)) != pos and not read_only:+ os.truncate(self.path(file_no), pos)+ self.offsets.append(offsets)+ file_no += 1++ def index(self, e: dict):+ self.seq = e["seq"]+ kind = e["kind"]+ if kind == "message":+ self.messages.append(e["seq"])+ self.senders.append(e["from"])+ self.times.append(epoch(e["at"]))+ for user in e.get("mentions", ()):+ self.mentions.setdefault(user, array("I")).append(e["seq"])+ for a in e.get("attachments", ()):+ self.files[a["id"]] = (e["seq"], a)+ answered = "reply_to" in e and MESSAGE_ID.fullmatch(e["reply_to"].strip())+ if answered:+ self.parent[e["seq"]] = int(answered[2])+ self.replies.setdefault(int(answered[2]), []).append(e["seq"])+ self.last, self.at = e, e["at"]+ elif kind in ("edit", "delete", "react"):+ change = self.changes.setdefault(e["target"], {})+ if kind == "edit":+ change.update(text=e["text"], edited_at=e["at"])+ self.edits.setdefault(e["target"], []).append(e["seq"])+ if "mentions" in e:+ change["mentions"] = e["mentions"]+ self.set_mentions(e["target"], e["mentions"])+ elif kind == "delete":+ change["deleted"] = e["at"]+ for att_id in [i for i, (seq, _) in self.files.items() if seq == e["target"]]:+ del self.files[att_id]+ change["mentions"] = []+ self.set_mentions(e["target"], [])+ else:+ users = change.setdefault("reactions", {}).setdefault(e["emoji"], [])+ if e.get("remove") and e["from"] in users:+ users.remove(e["from"])+ elif not e.get("remove") and e["from"] not in users:+ users.append(e["from"])+ elif kind == "read":+ upto, times = self.reads.setdefault(e["from"], (array("I"), []))+ upto.append(e["upto"])+ times.append(e["at"])+ else:+ self.at = e["at"]+ self.system.append(e["seq"])+ if kind in ("created", "added"):+ for user in e.get("members") or e.get("users") or ():+ self.spans.setdefault(user, []).append([e["seq"], None])+ elif kind == "removed":+ for user in e["users"]:+ if self.spans.get(user) and self.spans[user][-1][1] is None:+ self.spans[user][-1][1] = e["seq"]++ def set_mentions(self, seq: int, users: list):+ for user in set(self.mentions) | set(users):+ seqs = self.mentions.setdefault(user, array("I"))+ i = bisect.bisect_left(seqs, seq)+ present = i < len(seqs) and seqs[i] == seq+ if user in users and not present:+ seqs.insert(i, seq)+ elif user not in users and present:+ del seqs[i]++ def append(self, event: dict, fsync: bool) -> dict:+ """Writes one event at the end of the log, then indexes it. The caller holds self.lock."""+ if self.failed:+ raise Error(503, "This chat needs recovery after a failed log write. Restart the server.")+ event = {"seq": self.seq + 1, **event}+ encoded = (json.dumps(event, ensure_ascii=False, separators=(",", ":")) + "\n").encode()+ file_no = (event["seq"] - 1) // self.segment + 1+ if self.writer_file != file_no:+ if self.writer:+ self.writer.close()+ fd = os.open(self.path(file_no), os.O_WRONLY | os.O_CREAT | os.O_APPEND | getattr(os, "O_BINARY", 0), 0o600)+ self.writer, self.writer_file = os.fdopen(fd, "ab", buffering=0), file_no+ while len(self.offsets) < file_no:+ self.offsets.append(array("Q"))+ pos = self.writer.tell()+ try:+ remaining = memoryview(encoded)+ while remaining:+ written = self.writer.write(remaining)+ if not written:+ raise OSError("The log write made no progress.")+ remaining = remaining[written:]+ self.writer.flush()+ except OSError:+ try:+ os.ftruncate(self.writer.fileno(), pos)+ self.writer.seek(pos)+ except OSError:+ self.failed = True+ raise+ # Once the complete line exists, reserve its sequence even if fsync fails.+ # An unacknowledged message may survive, but its id must never be reused.+ self.offsets[file_no - 1].append(pos)+ self.index(event)+ if fsync:+ os.fsync(self.writer.fileno())+ if self.listener:+ self.listener(self, event)+ return event++ def read(self, seqs) -> list:+ """The events with these seqs, as stored, in the order given. Opens each log file once."""+ found, files = {}, {}+ try:+ for seq in sorted(set(seqs)):+ file_no, line_no = divmod(seq - 1, self.segment)+ if file_no not in files:+ files[file_no] = open(self.path(file_no + 1), "rb")+ for attempt in range(5):+ files[file_no].seek(self.offsets[file_no][line_no])+ try:+ found[seq] = json.loads(files[file_no].readline())+ break+ except ValueError: # read while redact() rewrote it: the next read sees the whole new line+ if attempt == 4:+ raise+ time.sleep(0.001)+ finally:+ for f in files.values():+ f.close()+ return [found[seq] for seq in seqs]++ def redact(self, seq: int, fsync: bool):+ """Rewrites a deleted message's line, and the lines of its edits, without their text, mentions and files.+ Each line keeps its length, padded with spaces, so every offset stays right. The new lines go to a journal+ first, so a crash in the middle of the rewrite is finished at the next start instead of leaving a line that is+ half old and half new. The journal holds only what remains, never the words. The caller holds self.lock, or is+ loading the chat."""+ writes = []+ for target in (seq, *self.edits.get(seq, ())):+ file_no, line_no = divmod(target - 1, self.segment)+ pos = self.offsets[file_no][line_no]+ with open(self.path(file_no + 1), "rb") as f:+ f.seek(pos)+ line = f.readline()+ e = json.loads(line)+ kept = {k: e[k] for k in ("seq", "kind", "from", "at", "target", "reply_to") if k in e}+ if e["kind"] == "edit":+ kept["text"] = "" # an edit always has text+ if kept == e:+ continue # already redacted+ encoded = json.dumps(kept, ensure_ascii=False, separators=(",", ":")).encode()+ writes.append([file_no + 1, pos, (encoded + b" " * (len(line) - 1 - len(encoded))).decode()])+ if writes:+ journal = os.path.join(self.dir, "redact.journal")+ write_json(journal, writes, durable=fsync)+ self._overwrite(writes, fsync)+ os.remove(journal)++ def _overwrite(self, writes: list, fsync: bool):+ """Writes each [log file number, offset, new line without its newline] in place."""+ for file_no, pos, line in writes:+ with open(self.path(file_no), "r+b") as f:+ f.seek(pos)+ f.write(line.encode())+ f.flush()+ if fsync:+ os.fsync(f.fileno())++ def view(self, e: dict) -> dict:+ """A message as members see it: with its id, and with later edits, deletion and reactions applied."""+ m = {"id": f"{self.id}:{e['seq']}", "chat": self.id, **e}+ change = self.changes.get(e["seq"], {})+ if "text" in change:+ m["text"], m["edited_at"] = change["text"], change["edited_at"]+ if "mentions" in change:+ m["mentions"] = change["mentions"]+ if m.get("attachments"): # members see id, name, size and type: never the hash+ m["attachments"] = [{k: v for k, v in a.items() if k != "sha256"} for a in m["attachments"]]+ if e["seq"] in self.parent: # whose message it answers: --where "m.reply_to_from == 'maya'" finds replies to maya+ m["reply_to_from"] = self.sender(self.parent[e["seq"]])+ if "deleted" in change:+ m["text"], m["deleted"] = "", True+ m.pop("attachments", None)+ reactions = {emoji: users for emoji, users in change.get("reactions", {}).items() if users}+ if reactions:+ m["reactions"] = reactions+ return m++ def present(self, user: str, seq: int) -> bool:+ """Whether `user` was a member when the line at seq was written."""+ return any(start < seq and (end is None or seq < end) for start, end in self.spans.get(user, ()))++ def read_at(self, user: str, seq: int) -> str | None:+ """When `user` read the message at seq, from the read lines in the log, or None."""+ upto, times = self.reads.get(user, ((), ()))+ i = bisect.bisect_left(upto, seq)+ return times[i] if i < len(upto) else None++ def sender(self, seq: int) -> str:+ """Who sent the message at seq."""+ return self.senders[bisect.bisect_left(self.messages, seq)]++ def is_message(self, seq: int) -> bool:+ i = bisect.bisect_left(self.messages, seq)+ return i < len(self.messages) and self.messages[i] == seq++ def position_at(self, user: str, at: str) -> tuple:+ """(how far `user` had read, when that read happened) at the time `at`, from the read lines. (0, None) if+ they had read nothing here by then."""+ upto, times = self.reads.get(user, ((), ()))+ i = bisect.bisect_right(times, at)+ return (upto[i - 1], times[i - 1]) if i else (0, None)++ def close(self):+ if self.writer:+ self.writer.close()+ self.writer, self.writer_file = None, 0+++class Feed:+ """Every new line in any chat, and every change to a user, numbered in order: what `swarmchat view` follows live.+ Only the newest `size` entries are kept. A viewer that falls further behind than that starts again from a+ snapshot."""++ def __init__(self, size: int = 20_000):+ self.entries = collections.deque(maxlen=size) # (number, chat or None, event)+ self.count = 0 # the number of the newest entry+ self.id = secrets.token_hex(4) # numbers start again with each start of the server: this tells them apart+ self.cond = threading.Condition()++ def add(self, chat: Chat | None, event: dict):+ with self.cond:+ self.count += 1+ self.entries.append((self.count, chat, event))+ self.cond.notify_all()++ def after(self, number: int) -> list | None:+ """The entries after `number`, oldest first, or None when some of them are no longer kept."""+ with self.cond:+ first = self.count - len(self.entries) + 1+ if number >= self.count:+ return []+ if number + 1 < first:+ return None+ return list(itertools.islice(self.entries, number + 1 - first, None))+++class Store:+ """Every user, chat, read position and setting, loaded from DIR at start and kept in step with it.++ With read_only, the folder is only looked at, for `swarmchat view` when no server is running on it: nothing in it is+ created, repaired or locked. The server then takes only the viewer's requests, with a view token made for this+ process alone (view_token)."""++ def __init__(self, root: str, settings: dict | None = None, read_only: bool = False):+ self.root, self.settings, self.read_only = root, {**DEFAULTS, **(settings or {})}, read_only+ self.fsync = self.settings["fsync"]+ self.lock = threading.RLock() # users, chat membership, read positions and settings+ self.chats, self._closed = {}, False+ self.feed = Feed()+ self.counting = threading.Lock() # the counts below+ self.stored = collections.Counter() # user -> bytes of the files on their messages that stand+ self.pending = collections.Counter() # user -> bytes of their uploads under way+ self.reserved = 0 # bytes of every upload under way+ self.free, self.free_at = 0, -1.0 # bytes free on the disk, and when that was asked+ self.requests = collections.Counter() # user -> their requests under way+ self._directory_lock = None if read_only else DirectoryLock(root)+ try:+ self._load()+ except BaseException:+ for chat in self.chats.values():+ chat.close()+ if self._directory_lock:+ self._directory_lock.close()+ raise++ def _load(self):+ if self.read_only and not os.path.exists(self.file("server.json")):+ raise SystemExit(f"{self.root} is not a data folder: it has no server.json.")+ for sub in ("users", "chats", "state", "retired"):+ if not self.read_only:+ os.makedirs(self.file(sub), 0o700, exist_ok=True)+ info = read_json(self.file("server.json"))+ if info is None:+ info = {"format": FORMAT, "created_at": now(), "segment": self.settings["segment"]}+ write_json(self.file("server.json"), info)+ elif info.get("format") != FORMAT:+ raise SystemExit(f"{self.root} holds data format {info.get('format')}, this version reads format {FORMAT}.")+ self.settings["segment"] = self.segment = info.get("segment", 10_000) # the folder's own, whatever was asked+ if self.read_only:+ self.view_token = new_token(32)+ self.admin, self.viewer = None, digest(self.view_token)+ else:+ self.view_token = None+ self.admin, self.viewer = (digest(self._token(name)) for name in ("admin.token", "view.token"))+ self.policy = Policy(read_json(self.file("policy.json"), {}))+ self.users, self.tokens = {}, {}+ self.retired = {read_json(self.file("retired", name))["id"]+ for name in self._list("retired") if name.endswith(".json")}+ for name in sorted(self._list("users")):+ if name.endswith(".json"):+ user = read_json(self.file("users", name))+ if user["id"] in self.retired:+ continue+ self.users[user["id"]], self.tokens[user["token_sha256"]] = user, user["id"]+ self.state = {u: read_json(self.file("state", f"{u}.json"), {"read": {}, "notify": {}}) for u in self.users}+ self.names = collections.Counter(u["name"].lower() for u in self.users.values())+ # A chat with a name is a channel. Without one it is a direct chat, found by its exact set of members.+ self.chats, self.direct, self.channels = {}, {}, {}+ self.blobs, self.files = Blobs(self.root, self.read_only), {} # files: attachment id -> its chat+ for chat_id in sorted(self._list("chats")):+ meta = read_json(self.file("chats", chat_id, "meta.json"))+ if meta:+ chat = self.chats[chat_id] = Chat(self.root, meta, self.segment)+ chat.load(self.read_only)+ if not self.read_only: # deleted before deletion erased the words on disk+ for seq, change in chat.changes.items():+ if "deleted" in change:+ chat.redact(seq, False)+ chat.listener = self.feed.add+ self._index_chat(chat)+ # Older versions deleted profiles outright; preserve identities still present in history.+ self.retired.update(u for u in [*chat.members, *chat.senders, meta["created_by"]] if u not in self.users)+ for att_id, (seq, a) in chat.files.items():+ self.files[att_id] = chat+ self.blobs.refs[a["sha256"]] += 1+ self.stored[chat.sender(seq)] += a["size"]+ if not self.read_only:+ self.blobs.prune()+ self.dirty, self.signals = set(), {}++ def file(self, *parts: str) -> str:+ return os.path.join(self.root, *parts)++ def _list(self, sub: str) -> list:+ return os.listdir(self.file(sub)) if os.path.isdir(self.file(sub)) else []++ def _token(self, name: str) -> str:+ """The token kept in the file `name`, made on first start, readable only by the folder's owner."""+ if not os.path.exists(self.file(name)):+ with os.fdopen(os.open(self.file(name), os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as f:+ f.write(new_token(32) + "\n")+ with open(self.file(name)) as f:+ return f.read().strip()++ # ------------------------------------------------------------------ users++ def me(self, token: str) -> str | None:+ return self.tokens.get(digest(token)) if token else None++ def is_admin(self, token: str) -> bool:+ return bool(token) and self.admin is not None and secrets.compare_digest(digest(token), self.admin)++ def can_view(self, token: str) -> bool:+ """The view token, or the admin token: either may look at everything through `swarmchat view`."""+ return bool(token) and (secrets.compare_digest(digest(token), self.viewer) or self.is_admin(token))++ def name(self, user: str) -> str:+ return self.users.get(user, {}).get("name", user)++ def display(self, user: str) -> str:+ """The name to show: the display name, with the id added when someone else has the same name."""+ name = self.name(user)+ return f"{name} ({user})" if self.names[name.lower()] > 1 else name++ def public(self, user: str) -> dict:+ u = self.users.get(user, {"id": user, "name": user})+ return {"id": u["id"], "name": u["name"], "fields": u.get("fields", {}), "status": u.get("status", "")}++ def user(self, ref: str) -> str:+ """A user id from an id or a display name, in any case."""+ ref = ref.strip().lstrip("@")+ if ref.lower() in self.users:+ return ref.lower()+ hits = [u for u, d in self.users.items() if d["name"].lower() == ref.lower()]+ if len(hits) == 1:+ return hits[0]+ if hits:+ raise Error(400, f"Several people are named {ref}: {', '.join(hits)}. Use the id.")+ close = difflib.get_close_matches(ref, [d["name"] for d in self.users.values()] + list(self.users), n=3)+ raise Error(404, f"No one called '{ref}'." + (f" Did you mean {', '.join(close)}?" if close else ""))++ def add_user(self, user: str, name: str = "", fields: dict | None = None) -> str:+ user = user.strip().lower()+ if not USER_ID.fullmatch(user):+ raise Error(400, "A user id is 1 to 40 of a-z, 0-9, '.', '_' and '-', starting with a letter or digit.")+ token = new_token()+ if not isinstance(name, str) or (fields is not None and not isinstance(fields, dict)):+ raise Error(400, "name must be text and fields must be an object.")+ with self.lock:+ if user in self.users:+ raise Error(400, f"{user} already exists.")+ if user in self.retired:+ raise Error(400, f"{user} is a retired user id. Choose a new id; rotate a token to keep an existing identity.")+ profile = {"id": user, "name": name or user, "fields": fields or {}, "status": "",+ "token_sha256": digest(token), "created_at": now()}+ write_json(self.file("users", f"{user}.json"), profile, 0o600)+ self.users[user] = profile+ self.tokens[digest(token)] = user+ self.state[user] = {"read": {}, "notify": {}}+ self.names[self.users[user]["name"].lower()] += 1+ self.feed.add(None, {"kind": "user", "id": user})+ return token++ def save_user(self, user: str):+ write_json(self.file("users", f"{user}.json"), self.users[user], 0o600)++ def rotate_token(self, user: str) -> str:+ user, token = self.user(user), new_token()+ with self.lock:+ profile = {**self.users[user], "token_sha256": digest(token)}+ write_json(self.file("users", f"{user}.json"), profile, 0o600)+ self.tokens.pop(self.users[user]["token_sha256"], None)+ self.users[user] = profile+ self.tokens[digest(token)] = user+ return token++ def remove_user(self, user: str):+ user = self.user(user)+ for chat in self.my_chats(user):+ if chat.name: # leaves every channel. Direct chats keep their members, as a record.+ self.change_members(user, chat, remove=[user])+ with self.lock:+ write_json(self.file("retired", f"{user}.json"), {"id": user, "deleted_at": now()}, 0o600)+ self.retired.add(user)+ gone = self.users.pop(user)+ self.tokens.pop(gone["token_sha256"], None)+ self.names[gone["name"].lower()] -= 1+ self.state.pop(user, None)+ for path in (self.file("users", f"{user}.json"), self.file("state", f"{user}.json")):+ if os.path.exists(path):+ os.remove(path)+ self.feed.add(None, {"kind": "user", "id": user})++ def set_status(self, user: str, text: str):+ with self.lock:+ profile = {**self.users[user], "status": text}+ write_json(self.file("users", f"{user}.json"), profile, 0o600)+ self.users[user] = profile+ self.feed.add(None, {"kind": "user", "id": user})++ def set_policy(self, rules: dict):+ policy = Policy(rules)+ with self.lock:+ write_json(self.file("policy.json"), rules)+ self.policy = policy++ def change_role(self, key: str, add=(), remove=(), everyone: bool = False) -> list | None:+ """Puts users on the rules' list `key`, admins or channel_creators, or takes them off. `everyone` drops the+ list of channel creators, so every user is one again."""+ add, remove = {self.user(u) for u in add}, {str(u).strip().lower() for u in remove}+ with self.lock:+ rules = dict(self.policy.rules)+ if everyone:+ rules.pop(key, None)+ else:+ rules[key] = sorted(({u.strip().lower() for u in rules.get(key) or ()} | add) - remove)+ self.set_policy(rules)+ return rules.get(key)++ # ------------------------------------------------------------------ chats++ def my_chats(self, user: str) -> list:+ return [chat for chat in list(self.chats.values()) if user in chat.members]++ def _index_chat(self, chat: Chat):+ if chat.name:+ self.channels.setdefault(chat.name.lower(), chat)+ else:+ self.direct.setdefault(frozenset(chat.members), chat)++ def _member(self, me: str, channel: Chat) -> Chat:+ if me not in self.users:+ raise Error(401, "This user no longer exists.")+ if me not in channel.members:+ how = "It is public, so you can join it." if channel.public else "Ask one of its members to add you."+ raise Error(403, f"You are not in #{channel.name}. {how}")+ return channel++ def ranges(self, me: str | None, chat: Chat, after: int = 0, before: int | None = None) -> list:+ """[(first, end)] index ranges in chat.messages of the messages after seq `after`, and before seq `before` if+ given, that `me` may read. me=None is the operator's viewer, which reads everything."""+ n = len(chat.times) # what a reader without the chat's lock can count on+ lo = bisect.bisect_right(chat.messages, after, 0, n)+ hi = n if before is None else bisect.bisect_left(chat.messages, before, 0, n)+ if me is None or not self.limited(chat):+ return [(lo, hi)] if lo < hi else []+ found = []+ for start, end in list(chat.spans.get(me, ())): # see Chat.present+ a = max(lo, bisect.bisect_right(chat.messages, start, 0, n))+ b = hi if end is None else min(hi, bisect.bisect_left(chat.messages, end, 0, n))+ if a < b:+ found.append((a, b))+ return found++ def indexes(self, me: str | None, chat: Chat, after: int = 0, before: int | None = None, newest: bool = False):+ """The places in chat.messages (and chat.senders, chat.times) of the messages in ranges(), one at a time, oldest+ first or newest first. A caller that wants a few messages of a long chat looks at only those."""+ ranges = self.ranges(me, chat, after, before)+ for lo, hi in reversed(ranges) if newest else ranges:+ yield from range(hi - 1, lo - 1, -1) if newest else range(lo, hi)++ def count(self, me: str | None, chat: Chat) -> int:+ """How many of a chat's messages `me` may read."""+ return sum(hi - lo for lo, hi in self.ranges(me, chat))++ def visible(self, me: str, chat: Chat, pairs: list) -> list:+ """The (seq, sender) pairs of messages `me` may read: all of them, unless the rules' history is while_member,+ which keeps a channel's messages from before someone joined, or from while they were out, from them."""+ if not self.limited(chat):+ return pairs+ return [pair for pair in pairs if chat.present(me, pair[0])]++ def limited(self, chat: Chat) -> bool:+ """Whether the rules keep some of this chat's messages from some of its members."""+ return bool(chat.name) and self.policy.history == "while_member"++ def sees(self, me: str | None, chat: Chat, seq: int) -> bool:+ return me is None or bool(self.visible(me, chat, [(seq, None)]))++ def unread(self, me: str, chat: Chat) -> int:+ """How many messages after my read position, from others, I may read. Counted without listing them."""+ return sum(hi - lo - chat.senders[lo:hi].count(me) for lo, hi in self.ranges(me, chat, self.position(me, chat)))++ def last(self, me: str, chat: Chat) -> dict | None:+ """The newest message I may read in a chat, as stored."""+ if chat.last is None or self.sees(me, chat, chat.last["seq"]):+ return chat.last+ i = next(self.indexes(me, chat, newest=True), None)+ return None if i is None else chat.read([chat.messages[i]])[0]++ def require(self, me: str | None, action: str, chat: Chat | None = None):+ """Refuses unless the rules let `me` (None for the super admin) take `action`, in `chat` for a chat's action."""+ refusal = self.policy.check(action, self.policy.roles(me, chat.managers if chat is not None else None))+ if refusal:+ raise Error(403, refusal)++ def _open(self, chat: Chat):+ """Refuses any change to an archived channel. Its members still read it, and may leave or be removed."""+ if chat.archived:+ raise Error(403, f"#{chat.name} is archived, so nothing in it can change. Unarchive it first.")++ def chat(self, me: str, ref: str, create: bool = False) -> Chat:+ """A chat from its id, a channel's name (with or without #), or people: one person, or several separated by+ commas, meaning the direct chat with exactly them. With create, a missing direct chat is made."""+ ref = ref.strip()+ chat = self.chats.get(ref)+ if chat and me in chat.members:+ return chat+ channel = self.channels.get(ref.lstrip("#").strip().lower())+ if channel and not channel.public and me not in channel.members:+ channel = None # a private channel answers only to its members: to anyone else it does not exist+ if ref.startswith("#"):+ if not channel:+ raise Error(404, f"There is no channel {ref}.")+ return self._member(me, channel)+ try:+ members = frozenset([me, *(self.user(p) for p in ref.split(",") if p.strip())])+ except Error as e:+ if channel:+ return self._member(me, channel)+ raise Error(404, f"No channel or person '{ref}'. {e}") from None+ if channel:+ raise Error(400, f"'{ref}' is both a channel and a person. Write #{channel.name} for the channel.")+ if len(members) < 2:+ raise Error(400, "Name at least one other person.")+ chat = self.direct.get(members)+ if chat:+ return chat+ if not create:+ raise Error(404, f"You have no chat with exactly {', '.join(sorted(members - {me}))} yet.")+ self.room()+ refusal = self.policy.create(me, sorted(members))+ if refusal:+ raise Error(403, refusal)+ with self.lock: # two people may write to each other for the first time at the same moment+ return self.direct.get(members) or self._new_chat(me, sorted(members), None)++ def create_channel(self, me: str, name: str, others=(), public: bool = False, description: str = "") -> Chat:+ """A channel is private unless made public: then anyone may find it and join."""+ name, description = self._channel_name(name), self._description(description)+ self.require(me, "create_channel")+ self.room()+ members = sorted({me, *(self.user(o) for o in others)})+ refusal = self.policy.create(me, members)+ if refusal:+ raise Error(403, refusal)+ with self.lock:+ self._name_free(me, name)+ return self._new_chat(me, members, name, public, description)++ def _channel_name(self, name: str) -> str:+ name = (name or "").strip().lstrip("#").strip()+ if not name or len(name) > 80 or "," in name:+ raise Error(400, "A channel name is 1 to 80 characters, without commas.")+ return name++ def _description(self, text: str) -> str:+ text = (text or "").strip() if isinstance(text, str) or not text else None+ if text is None or len(text) > 1000:+ raise Error(400, "A channel description is text, up to 1,000 characters.")+ return text++ def _name_free(self, me: str, name: str, chat: Chat | None = None):+ """Channel names are unique on the server, and never someone's id or name, so a name means one thing. A+ private channel's members learn which channel took a name. Anyone else learns only that it is taken."""+ taken = self.channels.get(name.lower())+ if taken and taken is not chat:+ if not taken.public and me not in taken.members:+ raise Error(400, f"The name #{name} is taken. Pick another name.")+ how = "It is archived. Unarchive it" if taken.archived else "Join it" if taken.public \+ else "Ask one of its members to add you"+ raise Error(400, f"There is already a channel #{taken.name} ({taken.id}). {how}, or pick another name.")+ if name.lower() in self.users or self.names[name.lower()] > 0:+ raise Error(400, f"'{name}' is someone's name. Pick another name for the channel.")++ def _new_chat(self, me: str, members: list, name: str | None, public: bool = False, description: str = "") -> Chat:+ """Makes a chat on disk and in the index. The caller holds self.lock and has checked everything."""+ chat_id = secrets.token_hex(8) # too many to guess, so an id does not find a chat+ while chat_id in self.chats:+ chat_id = secrets.token_hex(8)+ meta = {"id": chat_id, "name": name, "members": members, "created_by": me, "created_at": now(),+ **({"managers": [me]} if name else {})}+ shown = {**({"public": True} if public else {}), **({"description": description} if description else {})}+ meta.update(shown)+ chat = Chat(self.root, meta, self.segment)+ chat.listener = self.feed.add+ os.makedirs(chat.dir, 0o700)+ os.mkdir(os.path.join(chat.dir, "log"), 0o700)+ write_json(os.path.join(chat.dir, "meta.json"), meta)+ with chat.lock:+ chat.append({"kind": "created", "from": me, "at": meta["created_at"], "members": members, "name": name,+ **shown}, self.fsync)+ self.chats[chat_id] = chat+ self._index_chat(chat)+ return chat++ def change_members(self, me: str, chat: Chat, add=(), remove=()):+ if not chat.name:+ raise Error(400, "The members of a direct chat are fixed. Make a channel for a group whose members change.")+ add = [u for u in dict.fromkeys(map(self.user, add)) if u not in chat.members]+ remove = [u for u in dict.fromkeys(map(self.user, remove)) if u in chat.members]+ if set(add) - {me}: # joining a public channel is not adding+ self.require(me, "add_members", chat)+ if set(remove) - {me}: # anyone may leave+ self.require(me, "remove_members", chat)+ refusal = add and self.policy.add(me, chat.members, add)+ if refusal:+ raise Error(403, refusal)+ if not add and not remove:+ return+ if add: # leaving is never refused: it is how room is made+ self.room()+ with self.lock, chat.lock: # members change under the chat's lock, so a send sees them before or after+ if add:+ self._open(chat)+ if remove:+ chat.meta["managers"] = [u for u in chat.managers if u not in remove]+ chat.meta["members"] = sorted((set(chat.members) | set(add)) - set(remove))+ write_json(os.path.join(chat.dir, "meta.json"), chat.meta)+ if add:+ chat.append({"kind": "added", "from": me, "at": now(), "users": add}, self.fsync)+ if remove:+ chat.append({"kind": "removed", "from": me, "at": now(), "users": remove}, self.fsync)+ self.wake([*chat.members, *remove], None)++ def join(self, me: str, ref: str) -> Chat:+ """Joins a public channel, by its name or id. A private one takes being added by a member."""+ ref = ref.strip()+ chat = self.chats.get(ref) or self.channels.get(ref.lstrip("#").strip().lower())+ if not chat or not chat.name or (not chat.public and me not in chat.members):+ raise Error(404, f"There is no channel {ref}.") # a private channel's name and id stay with its members+ with self.lock: # so the channel cannot turn private between the check and the join+ if me in chat.members:+ return chat+ if not chat.public: # made private since the check above+ raise Error(404, f"There is no channel {ref}.")+ refusal = self.policy.join(me, chat.members)+ if refusal:+ raise Error(403, refusal)+ self.change_members(me, chat, add=[me])+ return chat++ def set_public(self, me: str, chat: Chat, public: bool):+ """Makes a channel public (anyone may find and join it) or private (only its members may add people). Members+ stay either way."""+ if not chat.name:+ raise Error(400, "Only a channel can be public. A direct chat holds exactly its members.")+ self.require(me, "make_public" if public else "make_private", chat)+ self.room()+ with self.lock:+ self._open(chat)+ if chat.public == public:+ return+ chat.meta["public"] = public+ write_json(os.path.join(chat.dir, "meta.json"), chat.meta)+ with chat.lock:+ chat.append({"kind": "visibility", "from": me, "at": now(), "public": public}, self.fsync)++ def rename(self, me: str, chat: Chat, name: str):+ if not chat.name:+ raise Error(400, "Only a channel has a name. Make a channel to name a group.")+ name = self._channel_name(name)+ self.require(me, "rename_channel", chat)+ self.room()+ if self.policy.posters and chat.name.lower() in self.policy.posters and name.lower() != chat.name.lower():+ raise Error(403, f"The rules name #{chat.name} for who may post in it, so it keeps its name: a new one "+ f"would drop the rule. Naming it by its id, {chat.id}, in the rules allows renaming.")+ with self.lock:+ self._open(chat)+ self._name_free(me, name, chat)+ del self.channels[chat.name.lower()]+ chat.meta["name"] = name+ self.channels[name.lower()] = chat+ write_json(os.path.join(chat.dir, "meta.json"), chat.meta)+ with chat.lock:+ chat.append({"kind": "renamed", "from": me, "at": now(), "name": name}, self.fsync)++ def describe(self, me: str, chat: Chat, description: str):+ """Sets what a channel is for. An empty description removes it."""+ if not chat.name:+ raise Error(400, "Only a channel has a description. A direct chat is described by its members.")+ description = self._description(description)+ self.require(me, "describe_channel", chat)+ self.room()+ with self.lock:+ self._open(chat)+ if chat.description == description:+ return+ chat.meta["description"] = description+ write_json(os.path.join(chat.dir, "meta.json"), chat.meta)+ with chat.lock:+ chat.append({"kind": "described", "from": me, "at": now(), "description": description}, self.fsync)++ def set_managers(self, me: str, chat: Chat, add=(), remove=()):+ """Names a channel's managers, from its members, or ends their role."""+ if not chat.name:+ raise Error(400, "Only a channel has managers. A direct chat has only its members.")+ self.require(me, "set_managers", chat)+ self.room()+ add, remove = list(dict.fromkeys(map(self.user, add))), list(dict.fromkeys(map(self.user, remove)))+ with self.lock, chat.lock:+ self._open(chat)+ strangers = [u for u in add if u not in chat.members]+ if strangers:+ raise Error(400, f"{', '.join(strangers)} {'is' if len(strangers) == 1 else 'are'} not in "+ f"#{chat.name}. Add them first.")+ add = [u for u in add if u not in chat.managers]+ remove = [u for u in remove if u in chat.managers]+ if not add and not remove:+ return+ chat.meta["managers"] = sorted((set(chat.managers) | set(add)) - set(remove))+ write_json(os.path.join(chat.dir, "meta.json"), chat.meta)+ chat.append({"kind": "managers", "from": me, "at": now(), **({"added": add} if add else {}),+ **({"removed": remove} if remove else {})}, self.fsync)++ def archive(self, me: str, chat: Chat, archived: bool):+ """Archives a channel, or unarchives it. An archived channel keeps its members and history, and its members+ still read it, but nothing new is written to it: no message, edit, deletion, reaction, new member, new name,+ new description or change of visibility. Members may still leave or be removed."""+ if not chat.name:+ raise Error(400, "Only a channel can be archived. A direct chat stays as it is.")+ self.require(me, "archive_channel", chat)+ self.room()+ with self.lock, chat.lock: # under the chat's lock, so a send sees the channel archived before or after+ if chat.archived == archived:+ return+ chat.meta["archived"] = archived+ write_json(os.path.join(chat.dir, "meta.json"), chat.meta)+ chat.append({"kind": "archived" if archived else "unarchived", "from": me, "at": now()}, self.fsync)++ # ------------------------------------------------------------------ messages++ def mentions(self, chat: Chat, text: str, sender: str) -> list:+ """Members the text mentions: @id, @Full Name, @First when no one else in the chat has that first name, or+ @all. A mention that could mean two members is refused, so the sender writes the id instead."""+ work, others = text.lower(), [u for u in chat.members if u != sender]+ if re.search(r"@(all|everyone|here|channel)\b", work):+ return others+ found = set()+ for u in sorted(others, key=len, reverse=True):+ pattern = rf"@{re.escape(u)}(?![\w-])"+ if re.search(pattern, work):+ found.add(u)+ work = re.sub(pattern, " ", work)+ for part in ("full", "first"):+ groups = collections.defaultdict(list)+ for u in others:+ words = self.name(u).lower().split()+ if words:+ groups[" ".join(words) if part == "full" else words[0]].append(u)+ for name in sorted(groups, key=len, reverse=True):+ pattern = rf"@{re.escape(name)}(?![\w-])"+ if re.search(pattern, work):+ if len(groups[name]) > 1:+ ids = " or ".join(f"@{u}" for u in groups[name])+ raise Error(400, f"@{name} could mean {ids}. Write {ids} instead.")+ found.add(groups[name][0])+ work = re.sub(pattern, " ", work)+ return sorted(found)++ def send(self, me: str, chat: Chat, text: str, reply_to: str | None = None, attachments=()) -> dict:+ """Writes a message. Attachments, if any, are already stored (Blobs.keep), and are released again if the+ message cannot be written."""+ try:+ return self._send(me, chat, text, reply_to, list(attachments))+ except BaseException:+ for a in attachments:+ self.blobs.release(a["sha256"])+ raise++ def _send(self, me: str, chat: Chat, text: str, reply_to: str | None, attachments: list) -> dict:+ if not attachments and (not text or not text.strip()):+ raise Error(400, "The message is empty.")+ if len(text) > self.settings["max_text"]:+ raise Error(400, f"The message is longer than {self.settings['max_text']} characters.")+ if not attachments: # files took their room before they were read: see reserve()+ self.room()+ event = {"kind": "message", "from": me, "text": text}+ if attachments:+ event["attachments"] = attachments+ if reply_to:+ target, e = self.message(me, reply_to)+ if target is not chat:+ raise Error(400, "A reply must be in the same chat as the message it answers.")+ event["reply_to"] = f"{chat.id}:{e['seq']}"+ # Membership changes take the chat's lock too, so these checks still hold when the line is written. Only this+ # chat waits: sends to other chats go on in parallel.+ with chat.lock:+ self._member(me, chat)+ self._open(chat)+ refusal = self.policy.send(me, chat.members, chat.name, chat.id)+ if refusal:+ raise Error(403, refusal)+ mentioned = self.mentions(chat, text, me)+ if mentioned:+ event["mentions"] = mentioned+ # Timestamps cannot go backwards within a chat, including after a clock adjustment.+ event["at"] = max(now(), chat.last["at"] if chat.last else "")+ e = chat.append(event, self.fsync)+ for a in attachments:+ self.files[a["id"]] = chat+ if attachments:+ with self.counting:+ self.stored[me] += sum(a["size"] for a in attachments)+ self.wake(chat.members, me)+ return chat.view(e)++ def attachment(self, me: str | None, attachment_id: str) -> tuple:+ """(the attachment as stored, the path of its bytes), if it is in one of my chats and its message stands.+ me=None is the operator's viewer, which sees every chat."""+ chat = self.files.get(attachment_id)+ if not chat or (me is not None and me not in chat.members) or attachment_id not in chat.files or \+ not self.sees(me, chat, chat.files[attachment_id][0]):+ raise Error(404, f"No attachment {attachment_id} in your chats.")+ a = chat.files[attachment_id][1]+ return a, self.blobs.path(a["sha256"])++ def message(self, me: str | None, message_id: str):+ """(chat, the stored message event) for a message id the caller can see. me=None sees every chat."""+ match = MESSAGE_ID.fullmatch(message_id.strip())+ chat = match and self.chats.get(match[1])+ if not chat or (me is not None and me not in chat.members) or not chat.is_message(int(match[2])) or \+ not self.sees(me, chat, int(match[2])):+ raise Error(404, f"No message {message_id} in your chats.")+ return chat, chat.read([int(match[2])])[0]++ def thread(self, me: str | None, message_id: str) -> tuple:+ """(chat, [(seq, depth), ...]): the whole thread a message is in. Its first message comes first, at depth 0, and+ each reply follows the message it answers, one level deeper. Replies to the same message keep their order."""+ chat, e = self.message(me, message_id)+ top = e["seq"]+ while top in chat.parent:+ top = chat.parent[top]+ # A message I may not read is left out, and so is its level: depth counts only the messages above it I may read.+ found, todo = [], [(top, 0)]+ while todo:+ seq, depth = todo.pop()+ shown = self.sees(me, chat, seq)+ if shown:+ found.append((seq, depth))+ todo += [(reply, depth + 1 if shown else depth) for reply in reversed(chat.replies.get(seq, ()))]+ return chat, found++ def change(self, me: str, message_id: str, kind: str, **fields) -> dict:+ """An edit, delete or reaction on a message. Only the sender may edit. The sender may delete, and so may whoever+ the rules allow to delete another person's message: administrators, by default."""+ chat, e = self.message(me, message_id)+ if kind == "edit" and e["from"] != me:+ raise Error(403, "Only the sender can edit a message.")+ if kind == "edit":+ text = fields.get("text") or ""+ if not isinstance(text, str) or not text.strip():+ raise Error(400, "The new text is empty. To remove the message, delete it.")+ if len(text) > self.settings["max_text"]:+ raise Error(400, f"The message is longer than {self.settings['max_text']} characters.")+ if kind != "delete": # a deletion frees room+ self.room()+ with chat.lock: # as in send: membership cannot change underneath, and other chats are not held up+ self._member(me, chat)+ self._open(chat)+ if kind == "edit":+ refusal = self.policy.send(me, chat.members, chat.name, chat.id)+ elif kind == "react":+ refusal = self.policy.react(me, chat.members, chat.name)+ else:+ refusal = None if e["from"] == me else \+ self.policy.check("delete_message", self.policy.roles(me, chat.managers))+ if refusal:+ raise Error(403, refusal)+ if kind == "edit":+ fields["mentions"] = self.mentions(chat, fields["text"], me)+ if "deleted" in chat.changes.get(e["seq"], {}):+ raise Error(400, f"{message_id} was deleted.")+ chat.append({"kind": kind, "from": me, "at": now(), "target": e["seq"], **fields}, self.fsync)+ if kind == "delete": # its words leave the disk too, not only the screen+ chat.redact(e["seq"], self.fsync)+ if kind == "delete": # the message's files go with it, once nothing else refers to their bytes+ for a in e.get("attachments", ()):+ self.files.pop(a["id"], None)+ self.blobs.release(a["sha256"])+ with self.counting:+ self.stored[e["from"]] -= a["size"]+ self.wake(chat.members, me)+ return chat.view(e)++ # ------------------------------------------------------------------ room on disk, and how much one user holds++ def room(self):+ """Refuses a message when the disk has less than min_free bytes free, beyond what uploads under way will take.+ The disk is asked at most twice a second: asking takes 50 to 150 microseconds, more than a send's other work."""+ floor = self.settings["min_free"]+ if not floor:+ return+ if time.monotonic() - self.free_at > 0.5:+ self.free, self.free_at = shutil.disk_usage(self.root).free, time.monotonic()+ if self.free - self.reserved < floor:+ raise Error(507, FULL)++ @contextlib.contextmanager+ def reserve(self, me: str, size: int):+ """Holds `size` bytes for an upload of `me`'s while it is read and stored, so that uploads at the same moment+ cannot together go below min_free on the disk or past a user's quota of stored files."""+ floor, quota = self.settings["min_free"], self.settings["quota"]+ with self.counting:+ if quota and self.stored[me] + self.pending[me] + size > quota:+ raise Error(413, f"These files would take you past your quota of {quota:,} bytes of stored files: "+ f"{self.stored[me]:,} are stored. Delete messages with files no one needs.")+ if floor:+ self.free, self.free_at = shutil.disk_usage(self.root).free, time.monotonic()+ if self.free - self.reserved - size < floor:+ raise Error(507, FULL)+ self.reserved += size+ self.pending[me] += size+ try:+ yield+ finally:+ with self.counting:+ self.reserved -= size+ self.pending[me] -= size++ @contextlib.contextmanager+ def request(self, me: str):+ """Counts a request of `me`'s while it is under way: a wait, an upload, anything. At most max_requests at once,+ so no one user can hold every connection the server has."""+ with self.counting:+ if self.requests[me] >= self.settings["max_requests"]:+ raise Error(429, f"You have {self.requests[me]} requests under way, the most at once. Wait for one to "+ f"end.")+ self.requests[me] += 1+ try:+ yield+ finally:+ with self.counting:+ self.requests[me] -= 1++ # ------------------------------------------------------------------ read positions, settings, waking++ def position(self, user: str, chat: Chat) -> int:+ return self.state[user]["read"].get(chat.id, 0)++ def mark_read(self, user: str, chat: Chat, seq: int):+ """Moves a read position forward, and writes a read line in the chat's log as the receipt."""+ with self.lock:+ read = self.state[user]["read"]+ seq = min(seq, chat.seq)+ if seq <= read.get(chat.id, 0):+ return+ with chat.lock: # still under self.lock, so one user's read lines stay in increasing order+ chat.append({"kind": "read", "from": user, "at": now(), "upto": seq}, self.fsync)+ read[chat.id] = seq+ self.dirty.add(user)++ def shown_upto(self, user: str, chat: Chat, shown: set) -> int:+ """How far `user`'s read position may move once the messages `shown` reached them: past each unread message up+ to the first one not shown, or 0 when that is the first. A message a search or filter left out stays unread."""+ upto = 0+ for i in self.indexes(user, chat, self.position(user, chat)):+ if chat.senders[i] == user:+ continue+ if chat.messages[i] not in shown:+ break+ upto = chat.messages[i]+ return upto++ def level(self, user: str, chat: Chat) -> str:+ return self.state[user]["notify"].get(chat.id, self.settings["default_notify"])++ def set_level(self, user: str, chat: Chat, level: str):+ if level not in LEVELS:+ raise Error(400, f"The level is one of {', '.join(LEVELS)}.")+ with self.lock:+ self.state[user]["notify"][chat.id] = level+ self.dirty.add(user)+ self.flush()+ self.wake([user], None)++ def flush(self):+ """Writes the read positions and settings that changed. The server calls this every second."""+ with self.lock:+ if self._closed:+ return+ for user in list(self.dirty):+ if user in self.state:+ write_json(self.file("state", f"{user}.json"), self.state[user])+ self.dirty.discard(user)++ def signal(self, user: str) -> threading.Condition:+ with self.lock:+ return self.signals.setdefault(user, threading.Condition())++ def wake(self, members, sender: str | None):+ for user in members:+ cond = self.signals.get(user)+ if cond is not None and user != sender:+ with cond:+ cond.notify_all()++ def close(self):+ with self.lock:+ if self._closed:+ return+ try:+ self.flush()+ finally:+ try:+ for chat in self.chats.values():+ with chat.lock:+ chat.close()+ finally:+ self._closed = True+ if self._directory_lock:+ self._directory_lock.close()diff --git a/swarmchat/view.py b/swarmchat/view.pynew file mode 100644index 0000000..086905f--- /dev/null+++ b/swarmchat/view.py@@ -0,0 +1,845 @@+"""What `swarmchat view` asks the server: read-only requests for the operator's viewer, a web page served at /view.++Each request takes the store and its parameters and returns a JSON-ready dict, like api.py. They see every chat and+change nothing: no read position moves and no line is written. Seeing as an agent calls the same functions the CLI+calls, as that agent, with nothing marked read, so it shows exactly what the agent would get.+"""+from __future__ import annotations++import base64+import bisect+import codecs+import collections+import gzip+import hashlib+import heapq+import json+import math+import os+import struct+import threading+import time++from . import NAME, __version__, api+from .store import Chat, Error, Store, epoch++PREVIEW = 160 # characters of a message shown in a chat list or under a reply+RECENT = 900 # seconds counted as recent activity+MAX_FEED = 5000 # events in one feed reply. A viewer further behind simply asks again at once.+IMAGE_BYTES = 25_000_000 # a larger picture is offered for download, not drawn in the page+IMAGE_PIXELS = 25_000_000 # a 24-megapixel photo. Drawn, a picture takes 4 bytes a pixel in the browser.+TEXT_HEAD = 16_384 # bytes read from the start of a file to tell whether it is text, and to preview it+TEXT_LINES = 8 # lines of a text file shown under its message. Kept in memory for each file, so few.+TEXT_WIDTH = 200 # characters of each of those lines+TEXT_BYTES = 5_000_000 # of a text file shown whole in the page. Past that, it shows the start and offers a download.+++def _int(p: dict, key: str, default: int, most: int) -> int:+ value = p.get(key)+ return default if value in (None, "") else max(0, min(most, int(value)))+++def _chat(store: Store, ref) -> Chat:+ chat = store.chats.get(str(ref or "").strip())+ if not chat:+ raise Error(404, f"There is no chat {ref}.")+ return chat+++def _user(store: Store, ref) -> str:+ user = str(ref or "").strip().lower()+ if user not in store.users:+ raise Error(404, f"There is no user {ref}.")+ return user+++def _kind(chat: Chat) -> str:+ return "channel" if chat.name else "direct" if len(chat.members) == 2 else "group"+++def _short(m: dict) -> dict:+ text = "" if m.get("deleted") else m.get("text", "")+ return {"seq": m["seq"], "from": m["from"], "at": m["at"], "text": text[:PREVIEW], "deleted": bool(m.get("deleted")),+ "files": len(m.get("attachments") or ())}+++_tally_lock = threading.Lock()+++def _tally(chat: Chat) -> tuple:+ """({user: messages sent}, {user: when they last wrote, in seconds}) for a chat. Worked out when first asked,+ then kept up to date from the messages written since, so agents' sends never pay for it."""+ with _tally_lock:+ done, sent, last = chat.cache.get("tally") or (0, {}, {})+ n = len(chat.times)+ if done < n:+ senders, times = chat.senders, chat.times+ for i in range(done, n):+ u = senders[i]+ sent[u] = sent.get(u, 0) + 1+ last[u] = times[i]+ chat.cache["tally"] = (n, sent, last)+ return sent, last+++def _summary(store: Store, chat: Chat) -> dict:+ """A chat for the viewer's lists. Taken under the chat's lock, so its numbers all belong to its line `seq`: a feed+ event with a higher seq is new to the viewer, and one at or below it is already counted here."""+ with chat.lock:+ last = chat.view(chat.last) if chat.last else None+ sent, last_by = _tally(chat)+ return {"id": chat.id, "kind": _kind(chat), "name": chat.name, "description": chat.description,+ "public": chat.public, "archived": chat.archived, "members": list(chat.members),+ "managers": chat.managers, "created_by": chat.meta["created_by"],+ "created_at": chat.meta["created_at"], "seq": chat.seq, "messages": len(chat.messages),+ "last_at": chat.at, "last": _short(last) if last else None, "sent": dict(sent),+ "last_by": dict(last_by)}+++def _person(store: Store, user: str) -> dict:+ if user in store.users:+ return store.public(user)+ return {"id": user, "name": user, "fields": {}, "status": "", "removed": True}+++def _message(store: Store, chat: Chat, e: dict, parents: dict) -> dict:+ m = chat.view(e)+ _files(store, chat, m)+ m["replies"] = len(chat.replies.get(e["seq"], ()))+ parent = chat.parent.get(e["seq"])+ if parent is not None and parent in parents:+ m["parent"] = _short(chat.view(parents[parent]))+ return m+++def _messages(store: Store, chat: Chat, events: list) -> list:+ """Messages and system lines as the viewer shows them. A reply carries the start of the message it answers."""+ have = {e["seq"]: e for e in events}+ wanted = sorted({chat.parent[e["seq"]] for e in events if e["seq"] in chat.parent} - set(have))+ parents = {**have, **{e["seq"]: e for e in chat.read(wanted)}}+ out = []+ for e in events:+ if e["kind"] == "message":+ out.append(_message(store, chat, e, parents))+ else:+ out.append({"id": f"{chat.id}:{e['seq']}", "chat": chat.id, **e})+ return out+++# ---------------------------------------------------------------------- pictures and text files++def _files(store: Store, chat: Chat, m: dict):+ """Gives each attachment of a message that the page can show in place what the page needs for it. A picture gets+ its `image`: type, width and height, so the page can make room for it before its bytes arrive. A text file gets its+ `text`: its first lines. Worked out once for each stored file."""+ if not m.get("attachments"):+ return+ known = chat.cache.setdefault("files", {})+ for a in m["attachments"]:+ stored = chat.files.get(a["id"])+ if not stored or a.get("folder"):+ continue+ sha = stored[1]["sha256"]+ if sha not in known:+ path = store.blobs.path(sha)+ image = _image(path) if a["size"] <= IMAGE_BYTES else None+ text = None if image else _text_head(path)+ known[sha] = {"image": image} if image else {"text": text} if text else {}+ a.update(known[sha])+++def _text_head(path: str) -> dict | None:+ """The first lines of a file that is text: UTF-8, with no NUL byte in its first TEXT_HEAD bytes. `lines` counts+ them all when those bytes are the whole file, and `more` says whether there is more than the preview. None for+ anything else, and for an empty file. The bytes decide, never the name, so a binary file is never shown as text."""+ try:+ with open(path, "rb") as f:+ head = f.read(TEXT_HEAD)+ whole = not f.read(1)+ except OSError:+ return None+ if not head or b"\0" in head:+ return None+ try: # unless this is the whole file, a character cut off at the end of what was read is not a fault+ text = codecs.getincrementaldecoder("utf-8")().decode(head, final=whole)+ except UnicodeDecodeError:+ return None+ lines = text.removeprefix("\ufeff").replace("\r\n", "\n").split("\n")+ if lines[-1] == "" and len(lines) > 1:+ lines.pop() # the newline that ends the last line starts no new one+ shown = lines[:TEXT_LINES]+ out = {"preview": "\n".join(line[:TEXT_WIDTH] for line in shown),+ "more": not whole or len(lines) > TEXT_LINES or any(len(line) > TEXT_WIDTH for line in shown)}+ if whole:+ out["lines"] = len(lines)+ return out+++def _image(path: str) -> dict | None:+ """The type and size in pixels of a PNG, JPEG, GIF, WebP or BMP, read from the start of its file. The bytes+ decide, never the type the sender gave, so only real pictures are drawn. None for anything else."""+ try:+ with open(path, "rb") as f:+ head = f.read(32)+ if head[:8] == b"\x89PNG\r\n\x1a\n" and head[12:16] == b"IHDR":+ kind, (w, h) = "image/png", struct.unpack(">II", head[16:24])+ elif head[:6] in (b"GIF87a", b"GIF89a"):+ kind, (w, h) = "image/gif", struct.unpack("<HH", head[6:10])+ elif head[:4] == b"RIFF" and head[8:12] == b"WEBP":+ kind, (w, h) = "image/webp", _webp(head)+ elif head[:2] == b"BM" and struct.unpack("<I", head[14:18])[0] in (12, 40, 52, 56, 108, 124): # its header's size+ w, h = struct.unpack("<HH" if head[14] == 12 else "<ii", head[18:22 if head[14] == 12 else 26])+ kind, h = "image/bmp", abs(h) # a negative height means the rows run top down+ elif head[:3] == b"\xff\xd8\xff":+ kind, (w, h) = "image/jpeg", _jpeg(f)+ else:+ return None+ except (OSError, ValueError, IndexError, struct.error):+ return None+ if not (w > 0 and h > 0 and w * h <= IMAGE_PIXELS):+ return None+ return {"type": kind, "width": w, "height": h}+++def _webp(head: bytes) -> tuple:+ chunk = head[12:16]+ if chunk == b"VP8 " and head[23:26] == b"\x9d\x01\x2a": # lossy+ w, h = struct.unpack("<HH", head[26:30])+ return w & 0x3FFF, h & 0x3FFF+ if chunk == b"VP8L" and head[20] == 0x2F: # lossless: 14 bits each, less one+ b = head[21:25]+ return 1 + (((b[1] & 0x3F) << 8) | b[0]), 1 + (((b[3] & 0x0F) << 10) | (b[2] << 2) | (b[1] >> 6))+ if chunk == b"VP8X": # extended: 24 bits each, less one+ return 1 + int.from_bytes(head[24:27], "little"), 1 + int.from_bytes(head[27:30], "little")+ raise ValueError("not a WebP picture")+++FRAMES = set(range(0xC0, 0xD0)) - {0xC4, 0xC8, 0xCC} # JPEG markers that start a frame header+++def _jpeg(f) -> tuple:+ """(width, height) of a JPEG as a browser draws it: turned when its EXIF orientation says so. Walks the+ segments from the start, skipping each by its length, until the frame header."""+ f.seek(2)+ turned = False+ while True:+ if f.read(1) != b"\xff":+ raise ValueError("not a JPEG segment")+ marker = f.read(1)+ while marker == b"\xff": # fill bytes+ marker = f.read(1)+ if not marker or marker[0] in (0xD9, 0xDA):+ raise ValueError("no frame header before the picture's data")+ if marker[0] == 0x01 or 0xD0 <= marker[0] <= 0xD8: # markers with no length+ continue+ length = struct.unpack(">H", f.read(2))[0]+ if marker[0] in FRAMES:+ h, w = struct.unpack(">xHH", f.read(5))+ return (h, w) if turned else (w, h)+ if marker[0] == 0xE1:+ data = f.read(length - 2)+ if data[:6] == b"Exif\0\0":+ turned = _orientation(data[6:]) in (5, 6, 7, 8)+ else:+ f.seek(length - 2, 1)+++def _orientation(tiff: bytes) -> int:+ """The EXIF orientation tag, 1 to 8, from the TIFF block of a JPEG's EXIF segment. 1 when it has none."""+ order = {b"II": "<", b"MM": ">"}.get(tiff[:2])+ try:+ ifd = struct.unpack(order + "I", tiff[4:8])[0]+ for i in range(struct.unpack(order + "H", tiff[ifd:ifd + 2])[0]):+ at = ifd + 2 + 12 * i+ if struct.unpack(order + "H", tiff[at:at + 2])[0] == 0x0112:+ return struct.unpack(order + "H", tiff[at + 8:at + 10])[0]+ except (TypeError, struct.error):+ pass+ return 1+++# ---------------------------------------------------------------------- the whole server at once++def view_info(store: Store, p: dict) -> dict:+ return {"name": NAME, "version": __version__, "read_only": store.read_only, "data": os.path.abspath(store.root),+ "feed": store.feed.count, "feed_id": store.feed.id}+++def view_snapshot(store: Store, p: dict) -> dict:+ """Every user and chat. `feed` is taken first: whatever happens after it comes through view_feed, and whatever is+ already in this snapshot is recognised there by its chat's seq."""+ position = store.feed.count+ everything = list(store.chats.values())+ chats = [_summary(store, c) for c in everything]+ # How busy each chat was in the 15 minutes before the newest message anywhere: the sidebar draws it as a bar.+ newest = max((c.times[-1] for c in everything if c.times), default=0.0)+ for c, summary in zip(everything, chats):+ summary["recent"] = len(c.times) - bisect.bisect_left(c.times, newest - RECENT)+ users = {u: _person(store, u) for u in sorted(store.users)}+ for c in chats: # people removed since still appear in the history+ for u in [*c["members"], *c["sent"]]:+ if u not in users:+ users[u] = _person(store, u)+ return {"feed": position, "server": view_info(store, p), "users": list(users.values()), "chats": chats,+ "now": time.time()}+++def view_feed(store: Store, p: dict, gone=lambda: False) -> dict:+ """Everything after the feed number `after`, held open like a wait until something happens or `timeout`+ passes. reset means the viewer should take a new snapshot: it fell too far behind, or the server started again+ since (`feed_id` from the snapshot differs) and numbers its feed from the start."""+ after = int(p.get("after") or 0)+ if after > store.feed.count or (p.get("feed_id") and p["feed_id"] != store.feed.id):+ return {"feed": store.feed.count, "events": [], "reset": True}+ timeout = min(float(p.get("timeout") if p.get("timeout") is not None else 25), store.settings["max_wait"])+ deadline = time.monotonic() + max(0.0, timeout)+ cond = store.feed.cond+ with cond:+ while True:+ entries = store.feed.after(after)+ left = deadline - time.monotonic()+ if entries is None or entries or left <= 0:+ break+ if not cond.wait(min(left, 1.0)) and gone():+ return {"feed": after, "events": []}+ if entries is None:+ return {"feed": store.feed.count, "events": [], "reset": True}+ entries = entries[:MAX_FEED]+ return {"feed": entries[-1][0] if entries else after, "events": _events(store, entries)}+++def _events(store: Store, entries: list) -> list:+ """Feed entries as the viewer applies them. Edits, deletions and reactions carry the message as it now stands."""+ targets = collections.defaultdict(set)+ for _, chat, e in entries:+ if chat is not None and e["kind"] in ("edit", "delete", "react"):+ targets[chat].add(e["target"])+ changed = {}+ for chat, seqs in targets.items():+ for e in chat.read(sorted(seqs)):+ changed[chat.id, e["seq"]] = e+ replies = collections.defaultdict(list)+ for _, chat, e in entries:+ if chat is not None and e["kind"] == "message":+ replies[chat].append(e)+ shown = {}+ for chat, events in replies.items():+ for m in _messages(store, chat, events):+ shown[chat.id, m["seq"]] = m+ out = []+ for n, chat, e in entries:+ kind = e["kind"]+ if chat is None:+ out.append({"n": n, "kind": "user", "user": _person(store, e["id"])})+ elif kind == "message":+ count = bisect.bisect_left(chat.messages, e["seq"]) + 1 # its place among the chat's messages+ out.append({"n": n, "kind": "message", "chat": chat.id, "seq": e["seq"], "count": count,+ "message": shown[chat.id, e["seq"]]})+ elif kind in ("edit", "delete", "react"):+ target = changed[chat.id, e["target"]]+ out.append({"n": n, "kind": "change", "chat": chat.id, "seq": e["seq"], "change": kind, "from": e["from"],+ "message": _messages(store, chat, [target])[0]})+ elif kind == "read":+ out.append({"n": n, "kind": "read", "chat": chat.id, "seq": e["seq"], "user": e["from"], "upto": e["upto"],+ "at": e["at"]})+ else:+ out.append({"n": n, "kind": "chat", "chat": chat.id, "seq": e["seq"], "event": e,+ "summary": _summary(store, chat)})+ return out+++# ---------------------------------------------------------------------- one chat++def view_messages(store: Store, p: dict) -> dict:+ """A run of a chat's messages, oldest first: the newest `limit`, or those just `before` or `after` a seq, or+ those `around` one. With them, the system lines in the same stretch and every member's read position."""+ chat = _chat(store, p.get("chat"))+ limit = _int(p, "limit", 100, 500) or 1+ seqs = chat.messages+ n = len(seqs)+ if p.get("around") is not None:+ i = bisect.bisect_left(seqs, int(p["around"]))+ lo = max(0, min(i - limit // 2, n - limit))+ hi = min(n, lo + limit)+ elif p.get("before") is not None:+ hi = bisect.bisect_left(seqs, int(p["before"]))+ lo = max(0, hi - limit)+ elif p.get("after") is not None:+ lo = bisect.bisect_right(seqs, int(p["after"]))+ hi = min(n, lo + limit)+ else:+ hi, lo = n, max(0, n - limit)+ wanted = list(seqs[lo:hi])+ lines = chat.system # the chat's start comes with its first page, and the newest changes with its last+ first = 0 if lo == 0 else bisect.bisect_left(lines, wanted[0])+ last = len(lines) if hi == n else bisect.bisect_right(lines, wanted[-1])+ events = chat.read(sorted([*wanted, *lines[first:last]]))+ return {"chat": chat.id, "messages": _messages(store, chat, events), "first": lo == 0, "last": hi == n,+ "index": lo, "total": n, "positions": _positions(store, chat)}+++def _positions(store: Store, chat: Chat) -> dict:+ return {u: store.position(u, chat) for u in chat.members if u in store.state}+++def view_thread(store: Store, p: dict) -> dict:+ """The whole thread a message is in, each message with its depth, like `swarmchat read --thread`."""+ chat, tree = store.thread(None, str(p.get("id") or ""))+ depth = dict(tree)+ messages = _messages(store, chat, chat.read([s for s, _ in tree]))+ for m in messages:+ m["depth"] = depth[m["seq"]]+ return {"chat": chat.id, "messages": messages, "positions": _positions(store, chat)}+++def view_receipts(store: Store, p: dict) -> dict:+ """Who has read a message and when, and who has not."""+ chat, e = store.message(None, str(p.get("id") or ""))+ read, unread = [], []+ for u in chat.members:+ if u == e["from"]:+ continue+ at = chat.read_at(u, e["seq"])+ if at or (u in store.state and store.position(u, chat) >= e["seq"]):+ read.append({"id": u, "at": at})+ else:+ unread.append(u)+ read.sort(key=lambda r: r["at"] or "")+ return {"id": f"{chat.id}:{e['seq']}", "read": read, "unread": unread}+++# ---------------------------------------------------------------------- boards++def _board(chat: Chat) -> list:+ """(seq, time, replies under it, votes, time of its newest reply) for each top-level message. Worked out in one+ pass from the newest message back, since a reply always comes after what it answers, and kept until the chat+ has a new line."""+ seq_now = chat.seq+ cached = chat.cache.get("board")+ if cached and cached[0] == seq_now:+ return cached[1]+ n = len(chat.times)+ messages, times, parent = chat.messages[:n], chat.times[:n], chat.parent+ size, newest = {}, {}+ for seq in reversed(messages):+ up = parent.get(seq)+ if up is not None:+ size[up] = size.get(up, 0) + 1 + size.get(seq, 0)+ newest[up] = max(newest.get(up, up), newest.get(seq, seq))+ posts = []+ for i, seq in enumerate(messages):+ if seq in parent:+ continue+ votes = sum(len(users) for users in chat.changes.get(seq, {}).get("reactions", {}).values())+ last = newest.get(seq, seq)+ posts.append((seq, times[i], size.get(seq, 0), votes, times[bisect.bisect_left(messages, last)]))+ chat.cache["board"] = (seq_now, posts)+ return posts+++def view_posts(store: Store, p: dict) -> dict:+ """Top-level messages as posts, from one chat or from every channel and group (the front page), sorted hot, new,+ top or active. With `as`, only chats that agent is in."""+ sort = p.get("sort") or "hot"+ if sort not in ("hot", "new", "top", "active"):+ raise Error(400, "sort is hot, new, top or active.")+ limit, offset = _int(p, "limit", 30, 200) or 1, _int(p, "offset", 0, 10 ** 9)+ if p.get("chat"):+ chats = [_chat(store, p["chat"])]+ else:+ chats = [c for c in list(store.chats.values()) if _kind(c) != "direct"]+ if p.get("as"):+ user = _user(store, p["as"])+ chats = [c for c in chats if user in c.members]+ newest = max((c.times[-1] for c in chats if c.times), default=time.time())++ def hot(post):+ seq, at, replies, votes, _ = post+ return (votes + replies + 1) / ((max(0.0, newest - at) / 3600 + 2) ** 1.5), at, seq++ # Each key ends in the time and the seq, so posts written in the same millisecond keep their order.+ key = {"hot": hot, "new": lambda post: (post[1], post[0]), "top": lambda post: (post[3] + post[2], post[1], post[0]),+ "active": lambda post: (post[4], post[0])}[sort]+ candidates = ((key(post), c.id, post) for c in chats for post in _board(c))+ top = heapq.nlargest(offset + limit, candidates, key=lambda x: x[0])[offset:]+ by_chat = collections.defaultdict(list)+ for _, chat_id, post in top:+ by_chat[chat_id].append(post[0])+ events = {}+ for chat_id, seqs in by_chat.items():+ chat = store.chats[chat_id]+ for m in _messages(store, chat, chat.read(sorted(seqs))):+ events[chat_id, m["seq"]] = m+ posts = []+ for _, chat_id, (seq, at, replies, votes, last_at) in top:+ posts.append({**events[chat_id, seq], "comments": replies, "votes": votes, "active_at": last_at})+ return {"posts": posts, "more": len(top) == limit}+++# ---------------------------------------------------------------------- search++def _candidates(chat: Chat, needle: str, too_old=lambda seq: False):+ """(the line's seq, the message's seq, the message or None) for each line whose text may hold `needle`, in lower+ case, newest line first. An edit gives the message it changes. The raw log is read from its end in growing+ pieces, so a search that finds enough in the newest lines reads only those. A hit counts only after a line's+ "text" key, so ids and times never match. too_old(seq) ends it before a piece whose newest line is too old."""+ escaped = json.dumps(needle, ensure_ascii=False)[1:-1] # as the text is written in the log+ raw = escaped.encode()+ for file_no in range(len(chat.offsets), 0, -1):+ offsets = chat.offsets[file_no - 1]+ first = (file_no - 1) * chat.segment + 1 # the seq of the file's first line+ try: # a bare descriptor: a search may open a file in each of thousands of chats+ fd = os.open(chat.path(file_no), os.O_RDONLY | getattr(os, "O_BINARY", 0))+ except FileNotFoundError:+ continue+ try:+ hi, size = len(offsets), 64+ while hi > 0:+ if too_old(first + hi - 1):+ return+ lo = max(0, hi - size)+ size *= 4+ end = offsets[hi] if hi < len(offsets) else os.fstat(fd).st_size+ data = _read_at(fd, offsets[lo], end - offsets[lo])+ if hi == len(offsets): # a line being written right now is not a line yet+ data = data[:data.rfind(b"\n") + 1]+ base = offsets[lo]+ lines = []+ if raw.isascii(): # lower() on bytes changes only ASCII, so positions stay those of the file+ low = data.lower()+ i = low.find(raw)+ while i >= 0:+ line = bisect.bisect_right(offsets, base + i, lo, hi) - 1 - lo+ start = offsets[lo + line] - base+ stop = data.find(b"\n", start) + 1+ key = low.find(b'"text":"', start, stop)+ if 0 <= key < i:+ lines.append((line, start, stop))+ i = low.find(raw, stop)+ else:+ i = low.find(raw, i + 1)+ else:+ for line in range(hi - lo):+ start = offsets[lo + line] - base+ stop = data.find(b"\n", start) + 1+ if escaped in data[start:stop].decode("utf-8", "replace").lower():+ lines.append((line, start, stop))+ for line, start, stop in reversed(lines):+ seq = first + lo + line+ e = json.loads(data[start:stop])+ if e["kind"] == "message":+ yield seq, seq, e+ elif e["kind"] == "edit":+ target = e["target"] - first - lo # the message it edits, often in this same piece+ if 0 <= target < hi - lo:+ start = offsets[lo + target] - base+ yield seq, e["target"], json.loads(data[start:data.find(b"\n", start) + 1])+ else:+ yield seq, e["target"], None+ hi = lo+ finally:+ os.close(fd)+++def _read_at(fd: int, offset: int, size: int) -> bytes:+ if hasattr(os, "pread"):+ return os.pread(fd, size, offset)+ os.lseek(fd, offset, os.SEEK_SET) # Windows has no pread+ parts = []+ while size > 0:+ chunk = os.read(fd, size)+ if not chunk:+ break+ parts.append(chunk)+ size -= len(chunk)+ return b"".join(parts)+++def view_search(store: Store, p: dict) -> dict:+ """Messages, newest first, from every chat, or only `chats`, or only an agent's chats with `as`. `text` is found+ in any case, in the text as it now stands. `from`, `mentions`, `files`, `after` and `before` narrow it, as in+ `swarmchat read`. For the next page, pass the reply's `until`.++ Chats are visited newest first, and each from its newest message back, keeping the best `limit` found so far.+ Once nothing older could still make the list, the search stops: a common word costs a few recent files."""+ text = (p.get("text") or "").strip().lower()+ limit = _int(p, "limit", 50, 500) or 1+ chats = [_chat(store, c) for c in p["chats"]] if p.get("chats") else list(store.chats.values())+ if p.get("as"):+ user = _user(store, p["as"])+ chats = [c for c in chats if user in c.members]+ senders = set(p.get("from") or ())+ after = epoch(p["after"]) if p.get("after") else None+ before = epoch(p["before"]) if p.get("before") else None+ until = tuple(p["until"]) if p.get("until") else None+ best, events = [], {} # best: a heap of (time, chat id, seq), the oldest on top. events: what was read already+ chats = sorted((c for c in chats if c.times), key=lambda c: c.times[-1], reverse=True)++ def floor() -> float:+ return best[0][0] if len(best) >= limit else -math.inf++ def offer(t: float, chat: Chat, seq: int, e: dict | None):+ item = (t, chat.id, seq)+ if until is not None and item >= until:+ return+ if len(best) < limit:+ heapq.heappush(best, item)+ elif item > best[0]:+ events.pop(heapq.heapreplace(best, item)[1:], None)+ else:+ return+ if e is not None:+ events[chat.id, seq] = e++ for chat in chats:+ if chat.times[-1] < floor():+ break # every chat after this one is older still+ times = chat.times[:] # a copy, and the messages it covers: an agent may be writing to this chat+ msgs = chat.messages[:len(times)]+ if senders and not senders & set(_tally(chat)[0]):+ continue+ mentioned = set(chat.mentions.get(str(p["mentions"]), ())) if p.get("mentions") else None+ with_files = {seq for seq, _ in chat.files.values()} if p.get("files") else None+ if mentioned == set() or with_files == set():+ continue++ def fits(i: int, seq: int) -> bool:+ at = times[i]+ return not ((after is not None and at < after) or (before is not None and at >= before)+ or (senders and chat.senders[i] not in senders)+ or (mentioned is not None and seq not in mentioned)+ or (with_files is not None and seq not in with_files))++ if text:+ seen = set()++ def too_old(line: int) -> bool: # every line up to this one is older than all of the best+ i = bisect.bisect_right(msgs, line) - 1+ bound = times[i] if i >= 0 else -math.inf+ return bound < floor() or (after is not None and bound < after)++ for line, seq, e in _candidates(chat, text, too_old):+ if too_old(line):+ break+ if seq in seen:+ continue+ seen.add(seq)+ i = bisect.bisect_left(msgs, seq)+ if i >= len(msgs) or msgs[i] != seq or not fits(i, seq):+ continue+ e = e or chat.read([seq])[0]+ m = chat.view(e)+ if m.get("deleted") or text not in m["text"].lower():+ continue+ offer(times[i], chat, seq, e)+ else:+ if mentioned is not None or with_files is not None:+ pool = sorted(mentioned if mentioned is not None else with_files, reverse=True)+ places = ((i, seq) for i, seq in ((bisect.bisect_left(msgs, seq), seq) for seq in pool) if i < len(msgs))+ else:+ top = len(msgs)+ if before is not None:+ top = bisect.bisect_left(times, before)+ if until is not None:+ top = min(top, bisect.bisect_right(times, until[0]))+ places = ((i, msgs[i]) for i in range(top - 1, -1, -1))+ for i, seq in places:+ if times[i] < floor() or (after is not None and times[i] < after):+ break+ if fits(i, seq):+ offer(times[i], chat, seq, None)+ found = sorted(best, reverse=True)+ by_chat = collections.defaultdict(list)+ for _, chat_id, seq in found:+ if (chat_id, seq) not in events:+ by_chat[chat_id].append(seq)+ for chat_id, seqs in by_chat.items():+ for e in store.chats[chat_id].read(sorted(seqs)):+ events[chat_id, e["seq"]] = e+ shown = {}+ grouped = collections.defaultdict(list)+ for _, chat_id, seq in found:+ grouped[chat_id].append(events[chat_id, seq])+ for chat_id, evs in grouped.items():+ for m in _messages(store, store.chats[chat_id], evs):+ shown[chat_id, m["seq"]] = m+ return {"messages": [shown[chat_id, seq] for _, chat_id, seq in found],+ "until": list(found[-1]) if len(found) == limit else None}+++# ---------------------------------------------------------------------- over time++def view_activity(store: Store, p: dict) -> dict:+ """Messages over time: `buckets` counts from the first message to the last, across every chat or the chats an+ agent is in (`as`). With `from` and `to` (seconds since 1970), how many messages each chat had in that stretch,+ and from whom."""+ chats = list(store.chats.values())+ if p.get("as"):+ user = _user(store, p["as"])+ chats = [c for c in chats if user in c.members]+ chats = [c for c in chats if c.times]+ if not chats:+ return {"start": None, "end": None, "width": None, "counts": [], "window": {}}+ buckets = _int(p, "buckets", 200, 2000) or 1+ first, last = min(c.times[0] for c in chats), max(c.times[-1] for c in chats)+ width = max((last - first) / buckets, 0.001)+ counts = [0] * buckets+ for c in chats:+ times = c.times+ if len(times) <= 4 * buckets:+ for t in times:+ counts[max(0, min(buckets - 1, int((t - first) / width)))] += 1+ else: # a long chat: where each bucket ends, found by halving, instead of every message+ lo = 0+ for b in range(buckets):+ hi = len(times) if b == buckets - 1 else bisect.bisect_left(times, first + (b + 1) * width, lo)+ counts[b] += hi - lo+ lo = hi+ out = {"start": first, "end": last, "width": width, "counts": counts}+ if p.get("from") is not None and p.get("to") is not None:+ start, end = float(p["from"]), float(p["to"])+ window = {}+ for c in chats:+ i, j = bisect.bisect_left(c.times, start), bisect.bisect_right(c.times, end)+ if j > i:+ sent = dict(_tally(c)[0]) if i == 0 and j == len(c.times) else dict(collections.Counter(c.senders[i:j]))+ window[c.id] = {"messages": j - i, "sent": sent}+ out["window"] = window+ return out+++# ---------------------------------------------------------------------- as one agent++def view_as(store: Store, p: dict) -> dict:+ """What one agent sees: its chats exactly as `swarmchat chats` lists them, its read position in each, and with+ `unread`, what `swarmchat read` would hand it now. Nothing is marked read."""+ user = _user(store, p.get("user"))+ out = {"user": store.public(user), "chats": api.chats(store, user, {"limit": 10 ** 9})["chats"],+ "positions": {c.id: store.position(user, c) for c in store.my_chats(user)}}+ if p.get("unread"):+ r = api.read(store, user, {"peek": True, "limit": _int(p, "limit", 50, 500) or 50})+ for m in r["messages"]:+ if m.get("attachments") and m.get("chat") in store.chats:+ _files(store, store.chats[m["chat"]], m)+ out["unread"], out["unread_left"] = r["messages"], r["unread_left"]+ return out+++def _members_at(chat: Chat, at: str) -> set:+ """Who was in a chat at the time `at`, from its created, added and removed lines."""+ members = set()+ for e in chat.read(list(chat.system)):+ if e["at"] > at:+ break+ if e["kind"] == "created":+ members = set(e["members"])+ elif e["kind"] == "added":+ members |= set(e["users"])+ elif e["kind"] == "removed":+ members -= set(e["users"])+ return members+++def view_knew(store: Store, p: dict) -> dict:+ """How far the sender of a message had marked messages read in each of its chats when it sent it, from the read+ lines in the logs. It is not everything the sender saw: --peek, threads, previews and searches leave no line."""+ chat, e = store.message(None, str(p.get("id") or ""))+ user, at = e["from"], e["at"]+ t = epoch(at)+ out = []+ for c in list(store.chats.values()):+ if c is not chat and user not in c.members and user not in c.reads and user not in _tally(c)[0]:+ continue+ if c is not chat and user not in _members_at(c, at):+ continue+ n = bisect.bisect_right(c.times, t) # messages written by then+ if not n and c is not chat:+ continue+ upto, read_at = c.position_at(user, at)+ i = bisect.bisect_right(c.messages, upto)+ unseen = sum(1 for s in c.senders[i:n] if s != user)+ out.append({"chat": c.id, "upto": upto, "read_at": read_at, "messages": n, "read": min(i, n),+ "last": c.messages[n - 1] if n else 0, "unseen": unseen})+ out.sort(key=lambda r: (r["chat"] != chat.id, -r["unseen"], r["chat"]))+ return {"message": _messages(store, chat, [e])[0], "user": user, "at": at, "chats": out}+++def download(store: Store, p: dict) -> tuple:+ """(the attachment, the path of its bytes), from any chat."""+ return store.attachment(None, str(p.get("id") or ""))+++def view_text(store: Store, p: dict) -> dict:+ """A text file, for the page to show formatted: its first TEXT_BYTES at most, cut at the end of a line. `cut` says+ whether there is more. Bytes that are not UTF-8 show as the replacement character."""+ a, path = download(store, p)+ if a.get("folder"):+ raise Error(400, f"{a['name']} is a folder. Download it instead.")+ with open(path, "rb") as f:+ data = f.read(TEXT_BYTES + 1)+ cut = len(data) > TEXT_BYTES+ if cut:+ data = data[:TEXT_BYTES]+ data = data[:data.rfind(b"\n") + 1] or data+ text = data.decode("utf-8", "replace").removeprefix("\ufeff").replace("\0", "\ufffd")+ return {"text": text, "size": a["size"], "shown": len(data), "cut": cut}+++def view_page(store: Store, p: dict) -> dict:+ """The viewer itself, for the page at /view to put in its own place once the token is accepted."""+ app = page()+ return {"html": app["html"], "js": app["js"]}+++METHODS = {f.__name__: f for f in (+ view_info, view_snapshot, view_messages, view_thread, view_receipts, view_posts, view_search, view_activity,+ view_as, view_knew, view_text, view_page)}+++# ---------------------------------------------------------------------- the page++PAGE_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "page")+_page, _gate = {}, {}+++def _text(path: str) -> str:+ with open(path, encoding="utf-8") as f:+ return f.read()+++def _stamp(names) -> tuple:+ return tuple(os.stat(os.path.join(PAGE_DIR, n)).st_mtime_ns for n in names)+++def _hash(script: str) -> str:+ return "'sha256-" + base64.b64encode(hashlib.sha256(script.encode()).digest()).decode() + "'"+++def page() -> dict:+ """The viewer: index.html with app.css written into it ("html"), its script ("js"), and the two as one file+ ("body"). Built again only when a file changes."""+ names = ("index.html", "app.css", "app.js")+ stamp = _stamp(names)+ if _page.get("stamp") != stamp:+ html, css, js = (_text(os.path.join(PAGE_DIR, n)) for n in names)+ html = html.replace("<!--CSS-->", f"<style>{css}</style>")+ _page.update(stamp=stamp, html=html.replace("<!--JS-->", ""), js=js, script=_hash(js),+ body=html.replace("<!--JS-->", f"<script>{js}</script>").encode())+ return _page+++def gate() -> dict:+ """What /view serves: a form for the view token, which fetches the viewer (view_page) only once the token is+ accepted. With it: its gzip, an ETag, and a Content-Security-Policy that lets only its own script and the viewer's+ exact script run, so text an agent wrote can never run as code."""+ app, names = page(), ("gate.html", "gate.js")+ stamp = (app["stamp"], _stamp(names))+ if _gate.get("stamp") != stamp:+ html, js = (_text(os.path.join(PAGE_DIR, n)) for n in names)+ body = html.replace("<!--JS-->", f"<script>{js}</script>").encode()+ csp = (f"default-src 'none'; script-src {_hash(js)} {app['script']}; style-src 'unsafe-inline'; "+ "img-src 'self' data: blob:; connect-src 'self'; worker-src blob:; base-uri 'none'; "+ "form-action 'none'; frame-ancestors 'none'")+ etag = f'"{hashlib.sha256(body + csp.encode()).hexdigest()[:20]}"' # a new viewer script changes the policy+ _gate.update(stamp=stamp, body=body, gzip=gzip.compress(body, 9), etag=etag, csp=csp)+ return _gatediff --git a/tests/test_by_hand.py b/tests/test_by_hand.pynew file mode 100644index 0000000..061ca06--- /dev/null+++ b/tests/test_by_hand.py@@ -0,0 +1,323 @@+"""The real swarmchat command, used the way a person uses it by hand: separate processes, Ctrl-C on a wait, two+terminals waiting at once, a login switched while a wait runs, a mistyped address. The server runs as its own process,+and every test also checks that its log stays free of tracebacks.+"""+import http.client+import json+import os+import shutil+import signal+import subprocess+import sys+import tempfile+import time+import unittest++ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+WINDOWS = os.name == "nt"+# How a person stops a command: Ctrl-C, a normal kill, a hard kill. Windows has only Ctrl-Break and a hard kill, and a+# process can receive Ctrl-Break only in its own process group.+STOPS = ("ctrl-break", "hard kill") if WINDOWS else ("ctrl-c", "kill", "hard kill")+GROUP = {"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP} if WINDOWS else {}+++def stop(process, how):+ if how == "hard kill":+ process.kill()+ elif how == "ctrl-break":+ process.send_signal(signal.CTRL_BREAK_EVENT)+ else:+ process.send_signal(signal.SIGINT if how == "ctrl-c" else signal.SIGTERM)+++class ByHand(unittest.TestCase):+ def setUp(self):+ self.dir = tempfile.mkdtemp(prefix="swarmchat-hand-")+ self.data = os.path.join(self.dir, "data")+ self.log = open(os.path.join(self.dir, "server.log"), "w+")+ self.server = subprocess.Popen([sys.executable, "-m", "swarmchat", "serve", "--data", self.data, "--port", "0"],+ env=self.env(), stdout=self.log, stderr=subprocess.STDOUT, **GROUP)+ self.url = ""+ for _ in range(1500): # up to 30 s: a busy CI machine can be slow to start Python+ try:+ with open(os.path.join(self.data, "address")) as f:+ self.url = f.read().strip()+ except FileNotFoundError:+ pass+ if self.url:+ break+ time.sleep(0.02)+ if not self.url:+ self.log.seek(0)+ self.fail(f"the server did not start: {self.log.read()}")+ self.tokens = {}+ for user, name in (("alice", "Alice Ames"), ("bob", "Bob Burns")):+ r = self.swarmchat(None, "admin", "--data", self.data, "add", user, "--name", name, "--json")+ self.tokens[user] = json.loads(r.stdout)["token"]++ def tearDown(self):+ stop(self.server, "ctrl-break" if WINDOWS else "kill")+ self.server.wait(10)+ self.log.seek(0)+ log = self.log.read()+ self.log.close()+ shutil.rmtree(self.dir)+ self.assertNotIn("Traceback", log, log)++ def env(self, user=None, config=None):+ e = {k: v for k, v in os.environ.items() if not k.startswith("SWARMCHAT_")}+ e.update(PYTHONPATH=ROOT, SWARMCHAT_CONFIG_DIR=os.path.join(self.dir, f"config-{config or user}"))+ if user:+ e.update(SWARMCHAT_SERVER=self.url, SWARMCHAT_TOKEN=self.tokens[user])+ return e++ def swarmchat(self, user, *args, config=None):+ return subprocess.run([sys.executable, "-m", "swarmchat", *args], env=self.env(user, config),+ capture_output=True, text=True, timeout=30)++ def start(self, user, *args, config=None):+ """A command left running, as in another terminal."""+ return subprocess.Popen([sys.executable, "-m", "swarmchat", *args], env=self.env(user, config),+ stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, **GROUP)++ def test_a_stopped_wait_does_not_use_up_the_next_message(self):+ for how in STOPS:+ with self.subTest(how=how):+ waiting = self.start("bob", "wait")+ time.sleep(3) # the server is now holding bob's wait+ stop(waiting, how)+ waiting.communicate(timeout=10)+ self.swarmchat("alice", "send", "bob", f"sent after {how}")+ time.sleep(0.3)+ self.assertIn(f"sent after {how}", self.swarmchat("bob", "read").stdout)++ def test_a_hung_up_wait_marks_nothing_read_on_the_server(self):+ # A client of the HTTP API that does not peek: the server itself must notice it has gone.+ host, port = self.url.split("//")[1].rsplit(":", 1)+ conn = http.client.HTTPConnection(host, int(port), timeout=5)+ conn.request("POST", "/v1/wait", json.dumps({"timeout": 60}),+ {"Authorization": f"Bearer {self.tokens['bob']}", "Content-Type": "application/json"})+ time.sleep(0.5)+ conn.close()+ self.swarmchat("alice", "send", "bob", "while nobody listened")+ time.sleep(1.5)+ self.assertIn("while nobody listened", self.swarmchat("bob", "read").stdout)++ def test_two_terminals_waiting_both_see_it(self):+ # --peek: neither marks it read, so both see it however their start-up times fall.+ first = self.start("bob", "wait", "--peek", "--timeout", "30")+ second = self.start("bob", "wait", "--peek", "--timeout", "30")+ time.sleep(1)+ self.swarmchat("alice", "send", "bob", "hello both")+ for waiting in (first, second):+ out, err = waiting.communicate(timeout=40)+ self.assertIn("hello both", out, err)+ self.assertIn("hello both", self.swarmchat("bob", "read").stdout) # still unread until now+ self.assertIn("No new messages", self.swarmchat("bob", "read").stdout)++ def test_switching_login_while_a_wait_runs(self):+ shared = "terminal"+ login = self.swarmchat(None, "login", self.url, self.tokens["bob"], config=shared)+ self.assertEqual(login.returncode, 0, login.stderr)+ waiting = self.start(None, "wait", "--timeout", "30", config=shared) # runs as bob+ time.sleep(3)+ self.assertIn("Alice Ames", self.swarmchat(None, "login", self.url, self.tokens["alice"], config=shared).stdout)+ self.swarmchat(None, "send", "bob", "to bob, from alice", config=shared)+ out, err = waiting.communicate(timeout=15)+ self.assertIn("to bob, from alice", out, err)++ def test_a_mistyped_address_keeps_the_saved_login(self):+ shared = "terminal"+ self.swarmchat(None, "login", self.url, self.tokens["bob"], config=shared)+ bad = self.swarmchat(None, "login", "http://127.0.0.1:9", self.tokens["bob"], config=shared) # nothing listens+ self.assertEqual(bad.returncode, 5, bad.stderr)+ self.assertIn("Is the server running?", bad.stderr)+ self.assertIn("Bob Burns", self.swarmchat(None, "whoami", config=shared).stdout)++ def test_mistakes_get_plain_errors(self):+ wrong = subprocess.run([sys.executable, "-m", "swarmchat", "chats"], capture_output=True, text=True, timeout=30,+ env={**self.env("bob"), "SWARMCHAT_TOKEN": "wrong"})+ self.assertEqual((wrong.returncode, "Unknown token" in wrong.stderr), (3, True))+ nobody = self.swarmchat("bob", "send", "nobody", "hi")+ self.assertEqual((nobody.returncode, "No channel or person" in nobody.stderr), (2, True))+ self.assertEqual(self.swarmchat("bob", "send", "alice", " ").returncode, 1)+ self.assertEqual(self.swarmchat("bob", "channel").returncode, 2) # no action: usage, not a crash+ self.assertEqual(self.swarmchat("bob", "wait", "--timeout", "0.2").returncode, 4)++ def test_attachments_by_hand(self):+ work = os.path.join(self.dir, "work")+ os.makedirs(os.path.join(work, "project", "empty"))+ with open(os.path.join(work, "photo.png"), "wb") as f:+ f.write(os.urandom(2_000_000))+ with open(os.path.join(work, "project", "run.sh"), "w") as f:+ f.write("#!/bin/sh\necho hi\n")+ sent = self.swarmchat("alice", "send", "bob", "look", "--attach", os.path.join(work, "photo.png"),+ "--attach", os.path.join(work, "project"))+ self.assertIn("with 2 attachments", sent.stdout, sent.stderr)+ lines = self.swarmchat("bob", "read", "--json").stdout.splitlines()+ self.assertNotIn("sha256", "".join(lines))+ ids = [a["id"] for a in json.loads(lines[0])["attachments"]]+ inbox = os.path.join(self.dir, "inbox")+ for att in ids:+ self.assertIn("Saved", self.swarmchat("bob", "download", att, "--to", inbox).stdout)+ with open(os.path.join(work, "photo.png"), "rb") as a, open(os.path.join(inbox, "photo.png"), "rb") as b:+ self.assertEqual(a.read(), b.read())+ self.assertTrue(os.path.isdir(os.path.join(inbox, "project", "empty")))+ again = self.swarmchat("bob", "download", ids[0], "--to", inbox)+ self.assertEqual((again.returncode, "already exists" in again.stderr), (1, True))+ self.assertIn("[file] photo.png", self.swarmchat("bob", "read", "--last", "1", "alice").stdout)++ def test_an_upload_killed_halfway_leaves_nothing(self):+ big = os.path.join(self.dir, "big.bin")+ with open(big, "wb") as f:+ f.write(os.urandom(90_000_000))+ uploads = os.path.join(self.data, "uploads")+ for attempt in range(3):+ with self.subTest(attempt=attempt):+ sending = self.start("alice", "send", "bob", "big one", "--attach", big)+ deadline = time.monotonic() + 10+ while not os.listdir(uploads) and time.monotonic() < deadline:+ time.sleep(0.001) # until the server is part way through receiving it+ self.assertTrue(os.listdir(uploads), "the upload never started")+ sending.kill()+ sending.communicate(timeout=10)+ for _ in range(100):+ if not os.listdir(uploads):+ break+ time.sleep(0.05)+ self.assertEqual(os.listdir(uploads), [])+ self.assertEqual(self.swarmchat("bob", "read", "--json").stdout, "") # no message, not even half of one+ blobs = [n for _, _, names in os.walk(os.path.join(self.data, "blobs")) for n in names]+ self.assertEqual(blobs, [])++ def test_a_killed_download_leaves_no_half_file(self):+ big = os.path.join(self.dir, "big.bin")+ with open(big, "wb") as f:+ f.write(os.urandom(90_000_000))+ sent = self.swarmchat("alice", "send", "bob", "--attach", big, "--json")+ att = json.loads(sent.stdout)["attachments"][0]["id"]+ inbox = os.path.join(self.dir, "inbox")+ os.makedirs(inbox)+ half = lambda: [n for n in os.listdir(inbox) if n.endswith(".part")]+ for how in STOPS:+ with self.subTest(how=how):+ getting = self.start("bob", "download", att, "--to", inbox)+ deadline = time.monotonic() + 10+ while not half() and time.monotonic() < deadline:+ time.sleep(0.001) # until it is part way through writing the file+ self.assertTrue(half(), "the download never started")+ stop(getting, how)+ getting.communicate(timeout=10)+ if how != "hard kill": # a normal stop cleans up on the way out+ self.assertEqual(half(), [])+ self.assertFalse(os.path.exists(os.path.join(inbox, "big.bin")))+ self.assertEqual(len(half()), 1) # what the hard kill left behind+ old = time.time() - 600+ os.utime(os.path.join(inbox, half()[0]), (old, old)) # some minutes later+ self.assertEqual(self.swarmchat("bob", "download", att, "--to", inbox).returncode, 0)+ self.assertEqual(half(), []) # cleared by the next download into that folder+ with open(big, "rb") as a, open(os.path.join(inbox, "big.bin"), "rb") as b:+ self.assertEqual(a.read(), b.read())++ def test_flags_work_anywhere_in_a_command(self):+ admin = open(os.path.join(self.data, "admin.token")).read().strip()+ for args in (["admin", "--admin-token", admin, "--server", self.url, "add", "zed1"], # as the README shows+ ["admin", "add", "zed2", "--admin-token", admin, "--server", self.url],+ ["admin", "add", "zed3", "--data", self.data],+ ["--server", self.url, "admin", "--data", self.data, "add", "zed4"]):+ with self.subTest(args=args):+ r = self.swarmchat(None, *args)+ self.assertEqual(r.returncode, 0, r.stderr)+ for args in (["--json", "channel", "create", "ops1"], ["channel", "--json", "create", "ops2"],+ ["channel", "create", "ops3", "--json"],+ ["--server", self.url, "--token", self.tokens["bob"], "chats"]):+ with self.subTest(args=args):+ r = self.swarmchat("bob", *args)+ self.assertEqual(r.returncode, 0, r.stderr)+ if "--json" in args:+ self.assertIsInstance(json.loads(r.stdout.splitlines()[0]), dict)++ def test_the_message_may_come_after_flags(self):+ # As an agent wrote it. Before Python 3.13, argparse dropped the text after a flag.+ first = json.loads(self.swarmchat("alice", "send", "bob", "first", "--json").stdout)+ note = os.path.join(self.dir, "note.txt")+ with open(note, "w") as f:+ f.write("hi")+ for flags in (["--reply-to", first["id"]], ["--attach", note], ["--json", "--attach", note]):+ with self.subTest(flags=flags):+ r = self.swarmchat("alice", "send", "bob", *flags, f"after {flags[-2]}", "--json")+ self.assertEqual(r.returncode, 0, r.stderr)+ sent = json.loads(r.stdout)+ self.assertEqual(sent["text"], f"after {flags[-2]}")+ self.assertEqual(sent.get("reply_to"), first["id"] if "--reply-to" in flags else None)+ self.assertEqual(len(sent.get("attachments", [])), 1 if "--attach" in flags else 0)++ def test_reading_a_thread(self):+ def send(user, to, text, *more):+ return json.loads(self.swarmchat(user, "send", to, text, *more, "--json").stdout)["id"]+ post = send("alice", "bob", "Q3 numbers?")+ comment = send("bob", "alice", "July is in", "--reply-to", post)+ answer = send("alice", "bob", "thanks", "--reply-to", comment)+ r = self.swarmchat("bob", "read", "--thread", comment)+ self.assertEqual(r.returncode, 0, r.stderr)+ shown = [line for line in r.stdout.splitlines() if " " in line and line.split()[0] in (post, comment, answer)]+ self.assertEqual([line[:line.index(line.split()[0])] for line in shown], ["", " ", " "], r.stdout) # indents+ self.assertNotIn("reply to", r.stdout) # the indent says it+ lines = self.swarmchat("bob", "read", "--thread", answer, "--json").stdout.splitlines()+ self.assertEqual([(json.loads(line)["id"], json.loads(line)["depth"]) for line in lines],+ [(post, 0), (comment, 1), (answer, 2)])++ def test_top_posts_and_replies_to_me(self):+ first = json.loads(self.swarmchat("alice", "send", "bob", "first post", "--json").stdout)["id"]+ second = json.loads(self.swarmchat("alice", "send", "bob", "second post", "--json").stdout)["id"]+ self.swarmchat("bob", "react", second, "+1")+ top = self.swarmchat("bob", "read", "alice", "--all", "--sort", "-len((m.reactions or {}).get('+1', []))",+ "--limit", "1")+ self.assertEqual(top.returncode, 0, top.stderr)+ self.assertIn(": second post", top.stdout)+ self.assertNotIn(": first post", top.stdout)+ self.swarmchat("bob", "send", "alice", "noted", "--reply-to", first)+ self.assertIn(f"(reply to alice's {first})", self.swarmchat("alice", "read").stdout)++ def test_a_public_board(self):+ made = self.swarmchat("alice", "channel", "create", "board", "--public")+ self.assertEqual(made.returncode, 0, made.stderr)+ self.assertIn("#board [public]", made.stdout)+ self.assertIn("#board 1 member, 0 messages, last ", self.swarmchat("bob", "channel", "list").stdout)+ refused = self.swarmchat("bob", "send", "#board", "hi")+ self.assertEqual((refused.returncode, "you can join it" in refused.stderr), (3, True))+ self.assertEqual(self.swarmchat("bob", "channel", "join", "board").returncode, 0)+ self.assertIn("(joined)", self.swarmchat("bob", "channel", "list").stdout)+ self.assertEqual(self.swarmchat("bob", "send", "#board", "hi").returncode, 0)+ self.assertEqual(self.swarmchat("bob", "channel", "visibility", "board", "private").returncode, 0)+ self.assertEqual(self.swarmchat("alice", "channel", "list").stdout.strip(), "No public channels.")+ self.assertNotIn("[public]", self.swarmchat("alice", "chats").stdout)++ def test_every_command_explains_itself(self):+ for command in ("whoami", "people", "chats", "read", "send", "edit", "delete", "react", "info", "channel",+ "channel create", "channel list", "channel join", "channel visibility", "channel add",+ "channel remove", "channel rename", "channel leave", "notifications", "wait", "settings",+ "settings notify", "settings status", "settings show", "login", "serve", "admin", "admin add",+ "admin remove", "admin rotate", "admin policy-set", "admin policy-show", "download"):+ with self.subTest(command=command):+ r = self.swarmchat("bob", *command.split(), "--help")+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("usage:", r.stdout)++ def test_json_is_one_object_per_line(self):+ self.swarmchat("alice", "send", "bob", "one")+ self.swarmchat("alice", "channel", "create", "ops", "--members", "bob")+ self.swarmchat("alice", "send", "#ops", "two")+ for command in (["chats"], ["read", "--all", "--peek"], ["people"], ["notifications", "--all"], ["whoami"],+ ["info", "#ops"], ["settings", "show"]):+ with self.subTest(command=command):+ r = self.swarmchat("bob", *command, "--json")+ self.assertEqual(r.returncode, 0, r.stderr)+ lines = r.stdout.splitlines()+ self.assertTrue(lines)+ for line in lines:+ self.assertIsInstance(json.loads(line), dict)+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_delivery.py b/tests/test_delivery.pynew file mode 100644index 0000000..e430567--- /dev/null+++ b/tests/test_delivery.py@@ -0,0 +1,170 @@+"""Delivery, cursor, mention, and membership regressions over real HTTP."""+import concurrent.futures+import threading+import time+from unittest.mock import patch++from test_swarmchat import Base+from swarmchat import api+from swarmchat.client import ClientError+++class MessageCorrectness(Base):+ def test_legacy_out_of_order_timestamps_do_not_skip_unread_messages(self):+ first = self.alice.channel_create(name="legacy", members="bob")["chat"]["id"]+ second = self.carol.channel_create(name="other", members="bob")["chat"]["id"]+ for chat_id, sender, text, at in ((first, "alice", "late time, early seq", "2026-10-03T18:00:03.000Z"),+ (first, "alice", "early time, late seq", "2026-10-03T18:00:01.000Z"),+ (second, "carol", "other chat", "2026-10-03T18:00:02.000Z")):+ chat = self.server.store.chats[chat_id]+ with chat.lock:+ chat.append({"kind": "message", "from": sender, "text": text, "at": at}, False)+ self.restart()+ received = self.texts(self.bob.read(limit=2)) + self.texts(self.bob.read(limit=2))+ self.assertCountEqual(received, ["late time, early seq", "early time, late seq", "other chat"])++ def test_edited_mentions_and_removals_survive_restart(self):+ first = self.alice.send(chat="bob", text="plain")["message"]+ second = self.alice.send(chat="bob", text="@bob remove me")["message"]+ self.alice.edit(id=first["id"], text="@bob added")+ self.alice.edit(id=second["id"], text="plain now")+ self.restart()+ self.assertEqual(self.texts(self.bob.read(mentions=True, peek=True)), ["@bob added"])+ self.alice.delete(id=first["id"])+ self.assertEqual(self.bob.read(mentions=True, peek=True)["messages"], [])++ def test_filtered_unread_counts_keep_the_requested_scope(self):+ message = self.alice.send(chat="bob", text="first")["message"]+ self.alice.send(chat="bob", text="second")+ self.carol.send(chat="bob", text="other chat")+ self.assertEqual(self.bob.read(id=message["id"], where="True")["unread_left"], 1)+ self.assertEqual(self.bob.read(chat="alice", where="True")["unread_left"], 0)+ self.assertEqual(self.bob.read(where="True")["unread_left"], 0)++ def test_removed_member_cannot_receive_from_already_open_wait(self):+ chat = self.alice.channel_create(name="private", members="bob")["chat"]["id"]+ entered_wait = threading.Event()+ original_pending = api._pending++ def observed_pending(*args, **kwargs):+ result = original_pending(*args, **kwargs)+ entered_wait.set()+ return result++ with patch.object(api, "_pending", side_effect=observed_pending):+ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:+ waiting = pool.submit(self.bob.wait, chat=chat, timeout=0.4, peek=True)+ self.assertTrue(entered_wait.wait(2), "Wait never reached the server")+ self.alice.channel_remove(chat=chat, users="bob")+ self.alice.send(chat=chat, text="secret sent AFTER Bob was removed")+ try:+ result = waiting.result(timeout=3)+ except ClientError as exc:+ self.assertIn(exc.status, (401, 403, 404))+ else:+ self.assertEqual(result["messages"], [], "Removed member received a new private message")+++ def test_filtered_wait_finds_match_after_first_hundred_messages(self):+ for number in range(100):+ self.alice.send(chat="bob", text=f"noise {number}")+ self.alice.send(chat="bob", text="urgent")+ result = self.bob.wait(timeout=0.3, where="m.text == 'urgent'")+ self.assertEqual(self.texts(result), ["urgent"])+++ def test_since_message_id_includes_later_message_in_same_millisecond(self):+ with patch("swarmchat.store.now", return_value="2026-10-03T18:00:00.000Z"):+ first = self.alice.send(chat="bob", text="first")["message"]+ self.alice.send(chat="bob", text="second")+ result = self.bob.read(chat="alice", since=first["id"], peek=True)+ self.assertEqual(self.texts(result), ["second"])+++ def test_edit_obeys_configured_message_length_limit(self):+ message = self.alice.send(chat="bob", text="short")["message"]+ with self.assertRaises(ClientError) as caught:+ self.alice.edit(id=message["id"], text="x" * 20001)+ self.assertEqual(caught.exception.status, 400)+++ def test_edit_reindexes_mentions(self):+ message = self.alice.send(chat="bob", text="please check")["message"]+ self.alice.edit(id=message["id"], text="@bob please check")+ result = self.bob.read(mentions=True, peek=True)+ self.assertEqual(self.texts(result), ["@bob please check"])+++ def test_filtered_read_reports_remaining_unread_after_marking(self):+ self.alice.send(chat="bob", text="one")+ result = self.bob.read(where="True")+ self.assertEqual(len(result["messages"]), 1)+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 0)+ self.assertEqual(result["unread_left"], 0)+++ def test_concurrent_send_order_cannot_skip_unread_messages(self):+ chat = self.alice.channel_create(name="team", members="bob,carol")["chat"]["id"]+ anchor = self.alice.send(chat=chat, text="anchor")["message"]["id"]+ self.bob.read()+ paused, release = threading.Event(), threading.Event()+ original_message = self.server.store.message++ def pause_before_commit(me, message_id): # the reply lookup runs before a send takes the chat's lock+ if me == "alice":+ paused.set()+ if not release.wait(3):+ raise RuntimeError("test scheduling timeout")+ return original_message(me, message_id)++ with patch.object(self.server.store, "message", side_effect=pause_before_commit):+ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:+ first = pool.submit(self.alice.send, chat=chat, text="started first, committed last", reply_to=anchor)+ try:+ self.assertTrue(paused.wait(2))+ time.sleep(0.01) # Distinct real timestamps; the clock is not mocked.+ self.carol.send(chat="bob", text="another chat")+ time.sleep(0.01)+ self.carol.send(chat=chat, text="committed first")+ finally:+ release.set()+ first.result(timeout=3)+ self.assertEqual(sum(c["unread"] for c in self.bob.chats()["chats"]), 3)+ delivered = self.texts(self.bob.read(limit=2)) + self.texts(self.bob.read(limit=2))+ self.assertCountEqual(delivered, ["started first, committed last", "another chat", "committed first"])+++ def test_contact_policy_also_rejects_new_text_via_edit(self):+ message = self.alice.send(chat="bob", text="before restriction")["message"]+ self.server.admin.policy_set(rules={"contacts": {}})+ with self.assertRaises(ClientError) as denied:+ self.alice.send(chat="bob", text="new communication")+ self.assertEqual(denied.exception.status, 403)+ with self.assertRaises(ClientError) as edited:+ self.alice.edit(id=message["id"], text="new communication")+ self.assertEqual(edited.exception.status, 403)+++ def test_adding_member_to_existing_channel_wakes_wait_for_history(self):+ chat = self.alice.channel_create(name="history")["chat"]["id"]+ self.alice.send(chat=chat, text="waiting for Bob")+ entered = threading.Event()+ original_pending = api._pending++ def observe_pending(*args, **kwargs):+ result = original_pending(*args, **kwargs)+ entered.set()+ return result++ with patch.object(api, "_pending", side_effect=observe_pending):+ with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:+ waiting = pool.submit(self.bob.wait, timeout=0.8, peek=True)+ self.assertTrue(entered.wait(2))+ self.alice.channel_add(chat=chat, users="bob")+ try:+ result = waiting.result(timeout=0.3)+ except concurrent.futures.TimeoutError:+ result = waiting.result(timeout=2)+ self.assertEqual(self.texts(result), ["waiting for Bob"])+ self.fail("Adding Bob did not wake his wait; it returned only at timeout")+ self.assertEqual(self.texts(result), ["waiting for Bob"])diff --git a/tests/test_files.py b/tests/test_files.pynew file mode 100644index 0000000..33aef15--- /dev/null+++ b/tests/test_files.py@@ -0,0 +1,495 @@+"""Attachments: files and folders sent with a message, stored once by hash, downloaded by members only."""+import hashlib+import http.client+import io+import json+import os+import shutil+import socket+import subprocess+import tempfile+import threading+import time+import unittest+import zipfile+from unittest.mock import patch++from test_swarmchat import Base, Server+from swarmchat import config, files+from swarmchat.client import Client, ClientError+from swarmchat.server import make_server+++def sha(path):+ digest = hashlib.sha256()+ with open(path, "rb") as f:+ for chunk in iter(lambda: f.read(1 << 20), b""):+ digest.update(chunk)+ return digest.hexdigest()+++def tree(root):+ """Every file and folder under root, with file bytes: equal trees compare equal."""+ out = {}+ for path, dirs, files in os.walk(root):+ rel = os.path.relpath(path, root)+ out[rel] = None+ for name in files:+ with open(os.path.join(path, name), "rb") as f:+ out[os.path.join(rel, name)] = f.read()+ return out+++class Attachments(Base):+ def setUp(self):+ super().setUp()+ self.work = tempfile.mkdtemp(prefix="swarmchat-files-")+ self.sent = os.path.join(self.work, "sent")+ os.makedirs(os.path.join(self.sent, "project", "src"))+ os.makedirs(os.path.join(self.sent, "project", "empty"))+ self.files = {"photo.png": os.urandom(300_000), "video.mp4": os.urandom(3_500_000),+ "run.sh": b"#!/bin/sh\necho hello\n"}+ for name, data in self.files.items():+ with open(os.path.join(self.sent, name), "wb") as f:+ f.write(data)+ with open(os.path.join(self.sent, "project", "src", "data.bin"), "wb") as f:+ f.write(os.urandom(70_000))+ with open(os.path.join(self.sent, "project", "notes.txt"), "w") as f:+ f.write("nested\n")++ def tearDown(self):+ shutil.rmtree(self.work)+ super().tearDown()++ def path(self, *parts):+ return os.path.join(self.sent, *parts)++ def blobs(self):+ return sorted(name for _, _, names in os.walk(os.path.join(self.data, "blobs")) for name in names)++ def uploads(self):+ return os.listdir(os.path.join(self.data, "uploads"))++ def test_round_trip_in_every_kind_of_chat(self):+ channel = self.alice.channel_create(name="team", members="bob,carol")["chat"]["id"]+ attach = [self.path(n) for n in self.files] + [self.path("project")]+ for where in ("bob", "bob,carol", channel):+ with self.subTest(where=where):+ m = self.alice.send(chat=where, text="files", attach=attach)["message"]+ self.assertEqual([a["name"] for a in m["attachments"]], [*self.files, "project"])+ got = os.path.join(self.work, "got-" + where.replace(",", "-"))+ for a in m["attachments"]:+ self.bob.download(a["id"], to=got)+ for name, data in self.files.items():+ with open(os.path.join(got, name), "rb") as f:+ self.assertEqual(f.read(), data)+ self.assertEqual(tree(os.path.join(got, "project")), tree(self.path("project")))+ self.assertEqual(len(self.blobs()), 4) # the same four contents, stored once each++ def test_hashes_never_reach_clients(self):+ self.alice.send(chat="bob", attach=[self.path("run.sh")])+ for result in (self.bob.read(all=True, peek=True), self.alice.chats()):+ self.assertNotIn("sha256", json.dumps(result))++ def test_history_and_downloads_survive_a_restart(self):+ m = self.alice.send(chat="bob", text="before restart", attach=[self.path("photo.png")])["message"]+ self.restart()+ history = self.bob.read(chat="alice", last=1)["messages"][0]+ self.assertEqual(history["attachments"][0]["id"], m["attachments"][0]["id"])+ target = self.bob.download(m["attachments"][0]["id"], to=self.work)+ with open(target, "rb") as f:+ self.assertEqual(f.read(), self.files["photo.png"])++ def test_only_members_can_download(self):+ channel = self.alice.channel_create(name="team", members="bob")["chat"]["id"]+ att = self.alice.send(chat=channel, attach=[self.path("run.sh")])["message"]["attachments"][0]["id"]+ with self.assertRaises(ClientError) as e:+ self.carol.download(att, to=self.work)+ self.assertEqual(e.exception.status, 404)+ self.alice.channel_add(chat=channel, users="carol") # history comes with joining, files included+ self.carol.download(att, to=os.path.join(self.work, "carol"))+ self.alice.channel_remove(chat=channel, users="bob")+ with self.assertRaises(ClientError) as e:+ self.bob.download(att, to=os.path.join(self.work, "bob"))+ self.assertEqual(e.exception.status, 404)++ def test_deleting_messages_frees_storage_once_nothing_refers_to_it(self):+ first = self.alice.send(chat="bob", attach=[self.path("photo.png")])["message"]+ second = self.alice.send(chat="carol", attach=[self.path("photo.png")])["message"]+ self.assertEqual(len(self.blobs()), 1)+ self.alice.delete(id=first["id"])+ self.assertEqual(len(self.blobs()), 1) # carol's message still has it+ with self.assertRaises(ClientError):+ self.bob.download(first["attachments"][0]["id"], to=self.work)+ self.assertNotIn("attachments", self.bob.read(chat="alice", last=1)["messages"][0])+ self.alice.delete(id=second["id"])+ self.assertEqual(self.blobs(), [])+ self.restart()+ self.assertEqual(self.blobs(), [])++ def test_limits(self):+ self.server.stop()+ self.server = Server(self.data, max_attachment=1000, max_attachments=2)+ alice = self.server.client(self.alice.token)+ with self.assertRaises(ClientError) as e:+ alice.send(chat="bob", attach=[self.path("photo.png")])+ self.assertEqual(e.exception.status, 413)+ with self.assertRaises(ClientError) as e:+ alice.send(chat="bob", attach=[self.path("run.sh")] * 3)+ self.assertEqual(e.exception.status, 413)+ alice.send(chat="bob", attach=[self.path("run.sh")])+ self.assertEqual((len(self.blobs()), self.uploads()), (1, []))++ def test_a_refused_upload_gets_a_clear_answer(self):+ # The server refuses before reading the bytes (a private channel the sender is not in, which answers as if it+ # did not exist), yet the sender gets the reason, not a reset.+ self.bob.channel_create(name="private")+ with self.assertRaises(ClientError) as e:+ self.raw_send(json.dumps({"chat": "#private", "files": [{"name": "a.bin", "size": 400_000}]}),+ os.urandom(400_000))+ self.assertEqual(e.exception.status, 404)+ self.assertEqual((self.blobs(), self.uploads()), ([], []))++ def test_an_interrupted_upload_leaves_nothing(self):+ host, port = self.server.httpd.server_address[:2]+ header = (json.dumps({"chat": "bob", "files": [{"name": "big.bin", "size": 5_000_000}]}) + "\n").encode()+ with socket.create_connection((host, port)) as sock:+ sock.sendall(b"POST /v1/send_files HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer " + self.alice.token.encode()+ + b"\r\nContent-Length: " + str(len(header) + 5_000_000).encode() + b"\r\n\r\n" + header+ + os.urandom(2_000_000))+ time.sleep(0.3)+ time.sleep(0.5)+ self.assertEqual((self.blobs(), self.uploads()), ([], []))+ self.assertEqual(self.bob.read()["messages"], [])++ def test_names_cannot_escape(self):+ m = self.raw_send(json.dumps({"chat": "bob", "files": [{"name": "../../etc/evil.sh", "size": 4}]}), b"evil")+ self.assertEqual(m["attachments"][0]["name"], "evil.sh")+ target = self.bob.download(m["attachments"][0]["id"], to=os.path.join(self.work, "inbox"))+ self.assertEqual(target, os.path.join(self.work, "inbox", "evil.sh"))++ def test_a_folder_with_unsafe_paths_is_not_unpacked(self):+ buf = io.BytesIO()+ with zipfile.ZipFile(buf, "w") as z:+ z.writestr("ok.txt", "fine")+ z.writestr("../escaped.txt", "outside")+ data = buf.getvalue()+ m = self.raw_send(json.dumps({"chat": "bob", "files": [{"name": "trap", "size": len(data), "folder": True}]}),+ data)+ with self.assertRaises(ClientError) as e:+ self.bob.download(m["attachments"][0]["id"], to=os.path.join(self.work, "inbox"))+ self.assertIn("unsafe path", str(e.exception))+ self.assertFalse(os.path.exists(os.path.join(self.work, "escaped.txt")))+ self.assertFalse(os.path.exists(os.path.join(self.work, "inbox", "trap")))++ def test_nothing_is_overwritten_without_force(self):+ att = self.alice.send(chat="bob", attach=[self.path("run.sh")])["message"]["attachments"][0]["id"]+ target = os.path.join(self.work, "run.sh")+ with open(target, "w") as f:+ f.write("mine")+ with self.assertRaises(ClientError):+ self.bob.download(att, to=self.work)+ with open(target) as f:+ self.assertEqual(f.read(), "mine")+ self.bob.download(att, to=self.work, force=True)+ with open(target, "rb") as f:+ self.assertEqual(f.read(), self.files["run.sh"])++ def test_leftovers_from_a_crash_are_cleared_at_start(self):+ self.alice.send(chat="bob", attach=[self.path("run.sh")])+ self.server.stop()+ os.makedirs(os.path.join(self.data, "blobs", "ff"), exist_ok=True)+ open(os.path.join(self.data, "blobs", "ff", "ff" * 32), "w").close()+ open(os.path.join(self.data, "uploads", "half.part"), "w").close()+ self.server = Server(self.data)+ self.assertEqual((len(self.blobs()), self.uploads()), (1, []))++ def test_bad_uploads_are_refused(self):+ for header, body in ((json.dumps({"chat": "bob", "files": [{"name": "a", "size": 10}]}), b"short"),+ (json.dumps({"chat": "bob", "files": []}), b""),+ ("not json", b"")):+ with self.subTest(header=header[:30]):+ with self.assertRaises(ClientError) as e:+ self.raw_send(header, body)+ self.assertEqual(e.exception.status, 400)+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bob")+ self.assertEqual(e.exception.status, 400)+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bob", attach=[os.path.join(self.work, "missing.txt")])+ self.assertEqual(e.exception.status, 400)++ def test_many_senders_at_once_keep_storage_exact(self):+ import concurrent.futures+ users = [self.server.user(f"u{i}", f"User {i}") for i in range(12)]+ shared = self.path("photo.png")+ own = []+ for i in range(12):+ own.append(os.path.join(self.work, f"own{i}.bin"))+ with open(own[-1], "wb") as f:+ f.write(os.urandom(50_000))++ def go(i):+ u = users[i]+ keep = u.send(chat="bob", attach=[shared, own[i]])["message"]+ gone = u.send(chat="carol", attach=[shared])["message"]+ u.delete(id=gone["id"])+ return keep["attachments"]++ with concurrent.futures.ThreadPoolExecutor(12) as pool:+ kept = list(pool.map(go, range(12)))+ self.assertEqual(len(self.blobs()), 13) # the shared photo once, plus each sender's own file+ for i, atts in enumerate(kept):+ target = self.bob.download(atts[1]["id"], to=os.path.join(self.work, f"bob{i}"))+ with open(target, "rb") as a, open(own[i], "rb") as b:+ self.assertEqual(a.read(), b.read())+ self.restart() # counts rebuilt from the logs agree+ self.assertEqual(len(self.blobs()), 13)++ def test_a_large_file_round_trips(self):+ big = os.path.join(self.work, "big.bin")+ with open(big, "wb") as f:+ for _ in range(90):+ f.write(os.urandom(1_000_000))+ att = self.alice.send(chat="bob", attach=[big])["message"]["attachments"][0]+ self.assertEqual(att["size"], 90_000_000)+ self.assertEqual(sha(self.bob.download(att["id"], to=os.path.join(self.work, "in"))), sha(big))++ def test_disk_full_mid_upload_gives_a_clear_error_and_leaves_nothing(self):+ real_open = os.fdopen++ def full_disk(fd, mode="r", *args, **kwargs):+ f = real_open(fd, mode, *args, **kwargs)+ if "w" in mode and "b" in mode:+ written = [0]+ write = f.write++ def limited(data):+ written[0] += len(data)+ if written[0] > 100_000:+ raise OSError(28, "No space left on device")+ return write(data)+ f.write = limited+ return f++ with patch("swarmchat.files.os.fdopen", side_effect=full_disk):+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bob", text="won't fit", attach=[self.path("photo.png")])+ self.assertEqual(e.exception.status, 507)+ self.assertIn("No space left", str(e.exception))+ self.assertEqual((self.blobs(), self.uploads(), self.bob.read()["messages"]), ([], [], []))+ self.alice.send(chat="bob", attach=[self.path("photo.png")]) # and it works again once there is room++ def test_a_disk_floor_and_a_quota_hold_uploads_back_before_they_are_read(self):+ self.alice.channel_create(name="before")+ self.server.store.settings.update(min_free=1 << 62) # more than any disk has+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bob", text="hello")+ self.assertEqual(e.exception.status, 507)+ with self.assertRaises(ClientError) as e:+ self.raw_send(json.dumps({"chat": "bob", "files": [{"name": "a.bin", "size": 400_000}]}), os.urandom(400_000))+ self.assertEqual(e.exception.status, 507)+ self.assertEqual((self.blobs(), self.uploads()), ([], []))+ for call in (lambda: self.alice.channel_create(name="ops"), lambda: self.alice.channel_add(chat="#before", users="bob"),+ lambda: self.alice.channel_rename(chat="#before", name="after"),+ lambda: self.alice.channel_describe(chat="#before", description="what it is for"),+ lambda: self.alice.send(chat="carol", text="a first message")):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, 507)+ self.alice.channel_leave(chat="#before") # leaving is never refused+ self.server.store.settings.update(min_free=1, quota=500_000)+ first = self.alice.send(chat="bob", attach=[self.path("photo.png")])["message"] # 300,000 bytes+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bob", attach=[self.path("photo.png")])+ self.assertEqual(e.exception.status, 413)+ self.assertIn("quota of 500,000 bytes", str(e.exception))+ self.bob.send(chat="alice", attach=[self.path("photo.png")]) # a quota is each user's own+ self.restart() # counted again from the logs+ self.server.store.settings.update(min_free=1, quota=500_000)+ self.assertRaises(ClientError, self.alice.send, chat="bob", attach=[self.path("photo.png")])+ self.alice.delete(id=first["id"])+ self.alice.send(chat="bob", attach=[self.path("photo.png")])++ def test_links_inside_a_folder(self):+ folder = os.path.join(self.work, "linked")+ os.makedirs(os.path.join(folder, "real"))+ with open(os.path.join(folder, "real", "a.txt"), "w") as f:+ f.write("a")+ os.symlink(os.path.join(folder, "real", "a.txt"), os.path.join(folder, "file-link.txt"))+ os.symlink(os.path.join(folder, "real"), os.path.join(folder, "folder-link"))+ os.symlink(folder, os.path.join(folder, "real", "loop")) # back up to the top: must not go on forever+ os.symlink(os.path.join(folder, "nowhere"), os.path.join(folder, "broken"))+ att = self.alice.send(chat="bob", attach=[folder])["message"]["attachments"][0]+ got = os.path.join(self.bob.download(att["id"], to=os.path.join(self.work, "in")))+ for rel in ("real/a.txt", "file-link.txt", "folder-link/a.txt"): # links arrive as what they point to+ with open(os.path.join(got, rel)) as f:+ self.assertEqual(f.read(), "a")+ self.assertFalse(os.path.exists(os.path.join(got, "broken")))+ self.assertFalse(os.path.islink(os.path.join(got, "file-link.txt")))++ def test_a_link_out_of_the_folder_stops_the_send(self):+ folder, secret = os.path.join(self.work, "deliverable"), os.path.join(self.work, "answers.txt")+ os.makedirs(folder)+ with open(secret, "w") as f:+ f.write("the answer key")+ with open(os.path.join(folder, "report.txt"), "w") as f:+ f.write("my report")+ os.symlink(secret, os.path.join(folder, "notes.txt"))+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bob", attach=[folder])+ self.assertEqual(e.exception.status, 400)+ self.assertIn("outside the folder", str(e.exception))+ os.makedirs(os.path.join(folder, "more"))+ os.unlink(os.path.join(folder, "notes.txt"))+ os.symlink(self.work, os.path.join(folder, "more", "up")) # a folder outside it, too+ self.assertRaises(ClientError, self.alice.send, chat="bob", attach=[folder])+ self.assertEqual((self.blobs(), self.uploads(), self.bob.read()["messages"]), ([], [], []))++ @unittest.skipIf(os.name == "nt", "switching a link needs no extra rights on POSIX only")+ def test_a_link_switched_out_of_the_folder_while_it_is_packed_is_caught(self):+ folder, secret = os.path.join(self.work, "deliverable"), os.path.join(self.work, "answers.txt")+ os.makedirs(folder)+ for path, text in ((secret, "the answer key"), (os.path.join(folder, "report.txt"), "my report")):+ with open(path, "w") as f:+ f.write(text)+ link = os.path.join(folder, "notes.txt")+ os.symlink(os.path.join(folder, "report.txt"), link) # inside the folder, as any check by name would find+ real_open = os.open++ def switching(path, flags, *args, **kwargs): # the link changes after the folder was listed, before it is read+ if path == link and os.readlink(link) != secret:+ os.unlink(link)+ os.symlink(secret, link)+ return real_open(path, flags, *args, **kwargs)+ with patch.object(files.os, "open", switching), self.assertRaises(ValueError) as e:+ files.pack(folder)+ self.assertIn("outside the folder", str(e.exception))++ def test_a_folder_too_large_for_the_disk_or_with_too_many_entries_is_not_unpacked(self):+ att = self.alice.send(chat="bob", attach=[self.path("project")])["message"]["attachments"][0]+ inbox = os.path.join(self.work, "in")+ for limit, value, words in (("MAX_ENTRIES", 2, "entries, more than 2"), ("SPARE", 1 << 62, "free")):+ with self.subTest(limit=limit), patch.object(files, limit, value), self.assertRaises(ClientError) as e:+ self.bob.download(att["id"], to=inbox)+ self.assertIn(words, str(e.exception))+ self.assertEqual(os.listdir(inbox), [])+ def broken(zip_path, dest): # the disk fills part way through+ with open(os.path.join(dest, "first.txt"), "w") as f:+ f.write("part of it")+ raise OSError(28, "No space left on device")+ with patch("swarmchat.client.unpack", broken), self.assertRaises(ClientError) as e:+ self.bob.download(att["id"], to=inbox)+ self.assertIn("No space left", str(e.exception))+ self.assertEqual(os.listdir(inbox), []) # nothing half unpacked is left+ self.assertEqual(tree(self.bob.download(att["id"], to=inbox)), tree(self.path("project")))++ def test_a_file_type_cannot_carry_headers(self):+ m = self.raw_send(json.dumps({"chat": "bob", "files": [+ {"name": "a.txt", "size": 5, "type": "text/plain\r\nContent-Length: 0\r\n\r\nX"}]}), b"hello")+ self.assertEqual(m["attachments"][0]["type"], "text/plain") # from the name instead+ att = m["attachments"][0]["id"]+ chat = self.server.store.files[att]+ chat.files[att][1]["type"] = "text/plain\r\nX-Injected: 1" # one stored before types were checked+ conn = http.client.HTTPConnection(*self.server.httpd.server_address[:2], timeout=10)+ conn.request("POST", "/v1/download", json.dumps({"id": att}), {"Authorization": f"Bearer {self.bob.token}"})+ response = conn.getresponse()+ self.assertEqual((response.status, response.getheader("Content-Type"), response.getheader("X-Injected"),+ response.read()), (200, "application/octet-stream", None, b"hello"))+ conn.close()++ def test_the_same_folder_always_zips_to_the_same_bytes(self):+ first = files.pack(self.path("project"))+ time.sleep(2.1) # past a zip timestamp step+ second = files.pack(self.path("project"))+ try:+ self.assertEqual(sha(first), sha(second))+ finally:+ os.unlink(first)+ os.unlink(second)++ def test_two_folders_linking_to_each_other_still_finish(self):+ top = os.path.join(self.work, "pair")+ os.makedirs(os.path.join(top, "a"))+ os.makedirs(os.path.join(top, "b"))+ os.symlink(os.path.join(top, "b"), os.path.join(top, "a", "to-b"))+ os.symlink(os.path.join(top, "a"), os.path.join(top, "b", "to-a"))+ with open(os.path.join(top, "a", "x.txt"), "w") as f:+ f.write("x")+ packed = files.pack(top)+ try:+ with zipfile.ZipFile(packed) as z:+ names = z.namelist()+ finally:+ os.unlink(packed)+ self.assertIn("a/x.txt", names)+ self.assertIn("b/to-a/x.txt", names)+ self.assertLess(len(names), 20)++ def test_names_in_any_language(self):+ folder = os.path.join(self.work, "dossier été")+ os.makedirs(os.path.join(folder, "東京"))+ with open(os.path.join(folder, "東京", "résumé.txt"), "w", encoding="utf-8") as f:+ f.write("bonjour")+ single = os.path.join(self.work, "naïve café ☕.txt")+ with open(single, "w", encoding="utf-8") as f:+ f.write("x")+ m = self.alice.send(chat="bob", attach=[single, folder])["message"]+ self.assertEqual([a["name"] for a in m["attachments"]], ["naïve café ☕.txt", "dossier été"])+ inbox = os.path.join(self.work, "in")+ for a in m["attachments"]:+ self.bob.download(a["id"], to=inbox)+ self.assertTrue(os.path.exists(os.path.join(inbox, "naïve café ☕.txt")))+ with open(os.path.join(inbox, "dossier été", "東京", "résumé.txt"), encoding="utf-8") as f:+ self.assertEqual(f.read(), "bonjour")++ @unittest.skipUnless(shutil.which("openssl"), "openssl is needed for a test certificate")+ def test_over_https_with_a_trusted_self_signed_certificate(self):+ cert, key = os.path.join(self.work, "cert.pem"), os.path.join(self.work, "key.pem")+ made = subprocess.run([shutil.which("openssl"), "req", "-x509", "-newkey", "rsa:2048", "-nodes", "-keyout", key,+ "-out", cert, "-days", "1", "-subj", "/CN=localhost",+ "-addext", "subjectAltName=DNS:localhost,IP:127.0.0.1"], capture_output=True, timeout=30)+ if made.returncode:+ self.skipTest(f"this openssl cannot make the certificate: {made.stderr[-200:]}")+ data = os.path.join(self.work, "tls-data")+ httpd, store = make_server(config.load(data=data, port=0, tls_cert=cert, tls_key=key))+ threading.Thread(target=httpd.serve_forever, kwargs={"poll_interval": 0.05}, daemon=True).start()+ try:+ url = f"https://127.0.0.1:{httpd.server_address[1]}"+ with patch.dict(os.environ, {"SWARMCHAT_CA": cert}):+ with open(os.path.join(data, "admin.token")) as f:+ admin = Client(url, f.read().strip())+ a = Client(url, admin.user_add(id="a")["token"])+ b = Client(url, admin.user_add(id="b")["token"])+ att = a.send(chat="b", attach=[self.path("video.mp4"), self.path("project")])["message"]["attachments"]+ inbox = os.path.join(self.work, "tls-in")+ for x in att:+ b.download(x["id"], to=inbox)+ self.assertEqual(sha(os.path.join(inbox, "video.mp4")), sha(self.path("video.mp4")))+ self.assertEqual(tree(os.path.join(inbox, "project")), tree(self.path("project")))+ for c in (admin, a, b):+ c.close()+ with self.assertRaises(ClientError): # without trusting the certificate, the client refuses+ Client(url, "x").whoami()+ finally:+ httpd.shutdown()+ httpd.server_close()+ store.close()++ def raw_send(self, header: str, body: bytes) -> dict:+ """An upload written by hand, to send what the client never would."""+ host, port = self.server.httpd.server_address[:2]+ line = (header + "\n").encode()+ conn = http.client.HTTPConnection(host, port, timeout=10)+ conn.request("POST", "/v1/send_files", line + body, {"Authorization": f"Bearer {self.alice.token}",+ "Content-Length": str(len(line) + len(body))})+ response = conn.getresponse()+ result = json.loads(response.read() or b"{}")+ conn.close()+ if response.status != 200:+ raise ClientError(response.status, result.get("error", ""))+ return result["message"]+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_storage_safety.py b/tests/test_storage_safety.pynew file mode 100644index 0000000..ab06ffc--- /dev/null+++ b/tests/test_storage_safety.py@@ -0,0 +1,229 @@+"""I/O recovery, single-writer ownership, and account lifecycle regressions."""+import errno+import io+import json+import os+import subprocess+import sys+import tempfile+import threading+from pathlib import Path+from unittest.mock import patch++from test_swarmchat import Base, Server, ROOT+from swarmchat import config+from swarmchat.client import ClientError+from swarmchat.server import make_server, save_read_positions+from swarmchat.store import Chat, Store, Error+++class StorageSafety(Base):+ def test_deleted_ids_remain_reserved_and_tokens_invalid_after_restart(self):+ self.alice.send(chat="bob", text="private to the original Bob")+ self.bob.send(chat="alice", text="original authorship")+ self.server.admin.user_remove(id="bob")+ self.server.admin.user_remove(id="carol") # No conversation history; the tombstone alone must reserve this id.+ for restarted in (False, True):+ if restarted:+ self.restart()+ with self.assertRaises(ClientError) as rejected:+ self.server.admin.user_add(id="bob", name="Replacement")+ self.assertEqual(rejected.exception.status, 400)+ with self.assertRaises(ClientError):+ self.server.admin.user_add(id="carol")+ with self.assertRaises(ClientError) as revoked:+ self.bob.read(all=True)+ self.assertEqual(revoked.exception.status, 401)++ def test_legacy_deleted_ids_in_chat_history_remain_reserved(self):+ self.alice.send(chat="bob", text="legacy private message")+ self.server.admin.user_remove(id="bob")+ os.remove(Path(self.data) / "retired" / "bob.json") # Simulate the older deletion format.+ self.restart()+ with self.assertRaises(ClientError) as rejected:+ self.server.admin.user_add(id="bob")+ self.assertEqual(rejected.exception.status, 400)++ def test_failed_rotation_and_profile_creation_do_not_mutate_live_users(self):+ old_token = self.bob.token+ with patch("swarmchat.store.write_json", side_effect=OSError(errno.ENOSPC, "injected full disk")):+ with self.assertRaises(ClientError):+ self.server.admin.token_rotate(id="bob")+ self.assertEqual(self.server.client(old_token).whoami()["user"]["id"], "bob")+ with self.assertRaises(ClientError):+ self.server.admin.user_add(id="bad", name="\ud800")+ self.assertNotIn("bad", [u["id"] for u in self.alice.people()["people"]])++ def test_partial_write_rolls_back_before_the_next_send(self):+ message = self.alice.send(chat="bob", text="seed")["message"]+ chat = self.server.store.chats[message["chat"]]+ writer = chat.writer+ original_write = writer.write++ def partial_write(data):+ original_write(data[:10])+ raise OSError(errno.ENOSPC, "injected partial write")++ with patch.object(writer, "write", side_effect=partial_write):+ with self.assertRaises(ClientError):+ self.alice.send(chat="bob", text="partial")+ self.alice.send(chat="bob", text="after recovery")+ self.restart()+ self.assertEqual(self.texts(self.bob.read()), ["seed", "after recovery"])++ def test_failed_rollback_blocks_writes_until_restart(self):+ message = self.alice.send(chat="bob", text="seed")["message"]+ chat = self.server.store.chats[message["chat"]]+ with patch.object(chat.writer, "write", side_effect=OSError(errno.EIO, "write failed")), \+ patch("swarmchat.store.os.ftruncate", side_effect=OSError(errno.EIO, "rollback failed")):+ with self.assertRaises(ClientError):+ self.alice.send(chat="bob", text="failed")+ with self.assertRaises(ClientError) as blocked:+ self.alice.send(chat="bob", text="must restart")+ self.assertEqual(blocked.exception.status, 503)+ self.restart()+ self.alice.send(chat="bob", text="recovered")+ self.assertEqual(self.texts(self.bob.read()), ["seed", "recovered"])++ def test_background_saver_retries_after_transient_failure(self):+ self.alice.send(chat="bob", text="read before error")+ self.bob.read()+ stopped, saved = threading.Event(), threading.Event()+ original = self.server.store.flush+ attempts = []++ def flaky_flush():+ attempts.append(1)+ if len(attempts) == 1:+ raise OSError(errno.ENOSPC, "transient")+ original()+ saved.set()+ stopped.set()++ self.server.store.settings["save_every"] = 0.01+ with patch.object(self.server.store, "flush", side_effect=flaky_flush), patch("sys.stderr", new=io.StringIO()):+ thread = threading.Thread(target=save_read_positions, args=(self.server.store, stopped), daemon=True)+ thread.start()+ try:+ self.assertTrue(saved.wait(2), "Background persistence did not recover")+ finally:+ stopped.set()+ thread.join(timeout=2)+ self.restart()+ self.assertEqual(self.bob.read()["messages"], [])++ def test_directory_lock_is_released_after_process_is_killed(self):+ with tempfile.TemporaryDirectory(prefix="swarmchat-lock-test-") as root:+ child = subprocess.Popen([sys.executable, "-c",+ "import sys, time; from swarmchat.store import Store; "+ "s = Store(sys.argv[1]); print('locked', flush=True); time.sleep(30)", root],+ cwd=ROOT, stdout=subprocess.PIPE, text=True)+ try:+ self.assertEqual(child.stdout.readline().strip(), "locked")+ with self.assertRaises(OSError):+ Store(root)+ finally:+ child.kill()+ child.wait(timeout=5)+ child.stdout.close()+ reopened = Store(root)+ reopened.close()++ def test_failed_startup_releases_directory_lock(self):+ with tempfile.TemporaryDirectory(prefix="swarmchat-start-test-") as root:+ with self.assertRaises(OSError):+ make_server(config.load(data=root, port=self.server.httpd.server_address[1]))+ reopened = Store(root)+ reopened.close()++ def test_duplicate_log_sequences_are_rejected_without_retaining_lock(self):+ with tempfile.TemporaryDirectory(prefix="swarmchat-corrupt-test-") as root:+ store = Store(root)+ store.add_user("alice")+ store.add_user("bob")+ chat = store.chat("alice", "bob", create=True)+ store.send("alice", chat, "one")+ log = Path(chat.path(1))+ store.close()+ original = log.read_bytes()+ with log.open("ab") as stream:+ stream.write(original.splitlines(keepends=True)[-1])+ with self.assertRaises(Error) as invalid:+ Store(root)+ self.assertIn("expected sequence", str(invalid.exception))+ log.write_bytes(original)+ recovered = Store(root)+ recovered.close()++ def test_a_deletion_cut_off_part_way_is_finished_at_the_next_start(self):+ sent = self.alice.send(chat="bob", text="secret words here")["message"]+ self.alice.edit(id=sent["id"], text="edited secret")+ chat, e = self.server.store.message("alice", sent["id"])++ def crash(chat, writes, fsync): # the process dies with the new line half written over the old one+ file_no, pos, line = writes[0]+ with open(chat.path(file_no), "r+b") as f:+ f.seek(pos)+ f.write(line.encode()[:12])+ raise RuntimeError("killed")+ with patch.object(Chat, "_overwrite", crash), chat.lock:+ chat.append({"kind": "delete", "from": "alice", "at": "2026-10-24T00:00:00.000Z", "target": e["seq"]}, False)+ self.assertRaises(RuntimeError, chat.redact, e["seq"], False)+ self.assertTrue(os.path.exists(os.path.join(chat.dir, "redact.journal")))+ self.restart() # a half-written line would stop the start, without the journal+ log = b"".join(path.read_bytes() for path in sorted(Path(chat.dir, "log").iterdir()))+ self.assertNotIn(b"secret", log)+ self.assertFalse(os.path.exists(os.path.join(chat.dir, "redact.journal")))+ self.assertEqual(self.bob.read(chat="alice", all=True)["messages"][0]["deleted"], True)++ def test_rejected_status_does_not_break_everyones_people_directory(self):+ with self.assertRaises(ClientError):+ self.bob.settings_status(text="\ud800")+ try:+ people = self.alice.people()["people"]+ except ClientError as exc:+ self.fail(f"Bob's rejected status broke Alice's directory request: HTTP {exc.status}")+ self.assertEqual(len(people), 3)+++ def test_failed_fsync_does_not_hide_next_acknowledged_message_after_restart(self):+ self.alice.send(chat="bob", text="seed")+ self.server.store.fsync = True+ with patch("swarmchat.store.os.fsync", side_effect=OSError(errno.EIO, "injected fsync failure")):+ with self.assertRaises(ClientError) as failure:+ self.alice.send(chat="bob", text="unacknowledged")+ self.assertEqual(failure.exception.status, 500)+ acknowledged = self.alice.send(chat="bob", text="acknowledged after failure")["message"]+ self.assertEqual(self.bob.read(id=acknowledged["id"], peek=True)["messages"][0]["text"],+ "acknowledged after failure")+ self.restart()+ recovered = self.bob.read(id=acknowledged["id"], peek=True)["messages"][0]["text"]+ self.assertEqual(recovered, "acknowledged after failure")+++ def test_failed_state_flush_is_retried_after_io_recovers(self):+ self.alice.send(chat="bob", text="already read")+ self.bob.read()+ with patch("swarmchat.store.write_json", side_effect=OSError(errno.ENOSPC, "injected full disk")):+ with self.assertRaises(OSError):+ self.server.store.flush()+ self.server.store.flush() # Disk has recovered: dirty state must still be queued.+ self.restart()+ self.assertEqual(self.bob.read()["messages"], [])+++ def test_two_servers_cannot_silently_corrupt_shared_directory(self):+ self.alice.send(chat="bob", text="seed")+ try:+ secondary = Server(self.data)+ except (OSError, SystemExit):+ return # Correct behavior may reject a second writer outright.+ try:+ second_alice = secondary.client(self.alice.token)+ self.alice.send(chat="bob", text="from first server")+ second_alice.send(chat="bob", text="from second server")+ finally:+ secondary.stop()+ self.restart()+ self.assertCountEqual(self.texts(self.bob.read(all=True, peek=True)),+ ["seed", "from first server", "from second server"])diff --git a/tests/test_swarmchat.py b/tests/test_swarmchat.pynew file mode 100644index 0000000..8a525c2--- /dev/null+++ b/tests/test_swarmchat.py@@ -0,0 +1,1244 @@+"""End-to-end tests: a real server on a free port with a temporary data folder, and real clients.++ python3 -m unittest discover -s tests+"""+import json+import os+import shutil+import signal+import subprocess+import sys+import tempfile+import threading+import time+import unittest++ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))+sys.path.insert(0, ROOT)++from swarmchat import config # noqa: E402+from swarmchat.client import Client, ClientError, matcher, sorter # noqa: E402+from swarmchat.server import make_server # noqa: E402+++class Server:+ def __init__(self, data, **settings):+ self.data = data+ self.httpd, self.store = make_server(config.load(data=data, port=0, **settings))+ self.url = f"http://127.0.0.1:{self.httpd.server_address[1]}"+ threading.Thread(target=self.httpd.serve_forever, kwargs={"poll_interval": 0.05}, daemon=True).start()+ with open(os.path.join(data, "admin.token")) as f:+ self.admin = Client(self.url, f.read().strip())+ self.clients = [self.admin]++ def client(self, token):+ self.clients.append(Client(self.url, token))+ return self.clients[-1]++ def user(self, user_id, name):+ return self.client(self.admin.user_add(id=user_id, name=name)["token"])++ def stop(self):+ for client in self.clients:+ client.close()+ self.httpd.shutdown()+ self.httpd.server_close()+ self.store.close()+++class Base(unittest.TestCase):+ def setUp(self):+ self.data = tempfile.mkdtemp(prefix="swarmchat-test-")+ self.server = Server(self.data)+ self.alice = self.server.user("alice", "Alice Ames")+ self.bob = self.server.user("bob", "Bob Burns")+ self.carol = self.server.user("carol", "Carol Cruz")++ def tearDown(self):+ self.server.stop()+ shutil.rmtree(self.data)++ def restart(self):+ tokens = {name: getattr(self, name).token for name in ("alice", "bob", "carol")}+ self.server.stop()+ self.server = Server(self.data)+ for name, token in tokens.items():+ setattr(self, name, self.server.client(token))++ def texts(self, result):+ return [m["text"] for m in result["messages"]]+++class Chats(Base):+ def test_a_dm_is_a_chat_of_two_and_is_reused(self):+ first = self.alice.send(chat="bob", text="hi bob")+ second = self.alice.send(chat="Bob Burns", text="are you there")+ self.assertEqual(first["chat"]["id"], second["chat"]["id"])+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 2)+ self.assertEqual(self.texts(self.bob.read()), ["hi bob", "are you there"])+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 0)+ self.assertEqual(self.bob.read()["messages"], [])++ def test_direct_chats_and_channels(self):+ trio = self.alice.send(chat="bob,carol", text="all three")["chat"]+ self.assertEqual(trio["kind"], "direct")+ self.assertEqual(self.carol.send(chat="bob,alice", text="any order")["chat"]["id"], trio["id"])+ self.assertNotEqual(trio["id"], self.alice.send(chat="bob", text="just us")["chat"]["id"])+ channel = self.alice.channel_create(name="Release review", members="bob,carol")["chat"]+ self.assertEqual((channel["kind"], channel["title"]), ("channel", "#Release review"))+ self.carol.send(chat="#release review", text="by name")+ self.carol.send(chat="Release Review", text="without the #")+ self.assertEqual(self.texts(self.alice.read(chat=channel["id"])), ["by name", "without the #"])+ self.assertEqual(self.alice.send(chat="bob,carol", text="still direct")["chat"]["id"], trio["id"])+ self.bob.channel_leave(chat="#release review")+ self.assertEqual([m["id"] for m in self.alice.chat_info(chat=channel["id"])["chat"]["members"]], ["alice", "carol"])+ with self.assertRaises(ClientError) as e:+ self.bob.send(chat="#release review", text="let me in")+ self.assertEqual(e.exception.status, 404) # a private channel is not there for those outside it++ def test_names_mean_one_thing(self):+ self.alice.channel_create(name="ops")+ for call in (lambda: self.bob.channel_create(name="OPS"), # channel names are unique+ lambda: self.bob.channel_create(name="Carol Cruz"), # and never someone's name+ lambda: self.alice.channel_add(chat=self.alice.send(chat="bob", text="x")["chat"]["id"],+ users="carol"), # direct chats keep their members+ lambda: self.alice.channel_rename(chat="bob", name="pair")):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, 400)+ with self.assertRaises(ClientError) as e:+ self.bob.channel_create(name="#ops")+ self.assertEqual(str(e.exception), "The name #ops is taken. Pick another name.") # no id, for a private one+ with self.assertRaises(ClientError) as e:+ self.alice.channel_create(name="#ops")+ self.assertIn("already a channel #ops", str(e.exception)) # its members learn which one++ def test_people_with_the_same_name(self):+ self.server.user("jsmith", "John Smith")+ self.server.user("jsmith2", "John Smith")+ chat = self.alice.channel_create(name="smiths", members="jsmith,jsmith2,bob")["chat"]["id"]+ self.assertIn("John Smith (jsmith2)", [m["name"] for m in self.alice.chat_info(chat=chat)["chat"]["members"]])+ for text in ("@John Smith please check", "@John please check"):+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat=chat, text=text)+ self.assertIn("@jsmith or @jsmith2", str(e.exception))+ self.assertEqual(self.alice.send(chat=chat, text="@jsmith2 please")["message"]["mentions"], ["jsmith2"])+ self.assertEqual(self.alice.send(chat=chat, text="@Bob and @bob")["message"]["mentions"], ["bob"])+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="John Smith", text="which one?")+ self.assertIn("jsmith, jsmith2", str(e.exception))++ def test_first_messages_both_ways_make_one_chat(self):+ pairs = [(self.server.user(f"x{i}", f"X {i}"), self.server.user(f"y{i}", f"Y {i}")) for i in range(20)]+ threads = [threading.Thread(target=c.send, kwargs={"chat": other, "text": "hi"})+ for i, (x, y) in enumerate(pairs) for c, other in ((x, f"y{i}"), (y, f"x{i}"))]+ [t.start() for t in threads]+ [t.join() for t in threads]+ self.assertEqual([len(x.chats()["chats"]) for x, _ in pairs], [1] * 20)++ def test_unknown_people_and_tokens(self):+ with self.assertRaises(ClientError) as e:+ self.alice.send(chat="bobb", text="x")+ self.assertIn("Did you mean", str(e.exception))+ with self.assertRaises(ClientError) as e:+ self.server.client("wrong").chats()+ self.assertEqual(e.exception.status, 401)+ with self.assertRaises(ClientError) as e:+ self.alice.user_add(id="mallory")+ self.assertEqual(e.exception.status, 403) # a known user, who is not the super admin++ def test_mentions_and_notification_levels(self):+ chat = self.alice.send(chat="bob,carol", text="hello both")["chat"]["id"]+ self.bob.settings_notify(chat=chat, level="mentions")+ self.assertEqual(self.bob.notifications()["chats"], [])+ self.alice.send(chat=chat, text="@Bob can you check?")+ self.assertTrue(self.bob.notifications()["chats"][0]["mentioned"])+ self.assertEqual(self.texts(self.bob.read(mentions=True)), ["@Bob can you check?"])+ m = self.carol.send(chat=chat, text="@all done")["message"]+ self.assertEqual(sorted(m["mentions"]), ["alice", "bob"])+++class Messages(Base):+ def test_edit_delete_react_survive_a_restart(self):+ m = self.alice.send(chat="bob", text="frist")["message"]+ self.alice.edit(id=m["id"], text="first")+ self.bob.react(id=m["id"], emoji="+1")+ with self.assertRaises(ClientError) as e:+ self.bob.edit(id=m["id"], text="mine now")+ self.assertEqual(e.exception.status, 403)+ gone = self.alice.send(chat="bob", text="oops")["message"]+ self.alice.delete(id=gone["id"])+ self.bob.read()+ self.restart()+ history = self.bob.read(chat="alice", last=10)["messages"]+ self.assertEqual(history[0]["text"], "first")+ self.assertEqual(history[0]["reactions"], {"+1": ["bob"]})+ self.assertTrue(history[1]["deleted"])+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 0) # read positions survive too++ def test_history_search_since_from_where(self):+ for i in range(30): # odd notes come from alice, even ones from carol+ (self.alice if i % 2 else self.carol).send(chat="bob", text=f"note {i}")+ self.assertEqual(self.texts(self.bob.read(chat="alice", last=3)), ["note 25", "note 27", "note 29"])+ self.assertEqual(len(self.bob.read(search="NOTE 1", peek=True)["messages"]), 11)+ self.assertEqual(self.texts(self.bob.read(**{"from": "carol"}, last=2)), ["note 26", "note 28"])+ r = self.bob.read(where="int(m.text.split()[1]) % 10 == 0", all=True, peek=True)+ self.assertEqual(self.texts(r), ["note 0", "note 10", "note 20"])+ since = self.bob.read(chat="alice", last=1)["messages"][0]["id"]+ self.alice.send(chat="bob", text="later")+ self.assertEqual(self.texts(self.bob.read(chat="alice", since=since)), ["later"])++ def test_unread_limit_and_reading_in_pages(self):+ for i in range(60):+ self.alice.send(chat="bob", text=str(i))+ page = self.bob.read()+ self.assertEqual(len(page["messages"]), 50)+ self.assertEqual(page["unread_left"], 10)+ self.assertEqual(self.texts(self.bob.read()), [str(i) for i in range(50, 60)])++ def tree(self, result):+ return [(m["depth"], m["text"]) for m in result["messages"]]++ def test_a_thread_is_its_first_message_and_every_reply_below_it(self):+ board = self.alice.channel_create(name="board", members="bob,carol")["chat"]["id"]+ post = self.alice.send(chat=board, text="post")["message"]["id"]+ other = self.bob.send(chat=board, text="another post")["message"]["id"]+ first = self.bob.send(chat=board, text="comment 1", reply_to=post)["message"]["id"]+ self.carol.send(chat=board, text="on the other post", reply_to=other)+ second = self.carol.send(chat=board, text="comment 2", reply_to=f" {post} ")["message"]["id"]+ answer = self.alice.send(chat=board, text="answer to 1", reply_to=first)["message"]["id"]+ self.bob.send(chat=board, text="answer to the answer", reply_to=answer)+ tree = [(0, "post"), (1, "comment 1"), (2, "answer to 1"), (3, "answer to the answer"), (1, "comment 2")]+ for start in (post, first, answer, second): # from any message in it+ self.assertEqual(self.tree(self.carol.read(thread=start)), tree)+ self.assertEqual(self.tree(self.carol.read(thread=other)), [(0, "another post"), (1, "on the other post")])+ self.assertEqual(self.carol.read(thread=second)["messages"][4]["reply_to"], post) # stored as the plain id+ self.assertEqual(self.tree(self.carol.read(thread=post, where="m['from'] == 'bob'")),+ [(1, "comment 1"), (3, "answer to the answer")])+ self.assertEqual(self.tree(self.carol.read(thread=post, limit=2)), tree[:2])++ def test_a_thread_keeps_its_shape_through_deletes_and_restarts_and_marks_nothing_read(self):+ sent = self.alice.send(chat="bob,carol", text="post")["message"]+ chat, post = sent["chat"], sent["id"]+ gone = self.bob.send(chat=chat, text="comment", reply_to=post)["message"]["id"]+ self.carol.send(chat=chat, text="answer", reply_to=gone)+ self.bob.delete(id=gone)+ self.restart()+ unread = self.carol.chats()["chats"][0]["unread"]+ r = self.carol.read(thread=post)+ self.assertEqual(self.tree(r), [(0, "post"), (1, ""), (2, "answer")])+ self.assertTrue(r["messages"][1]["deleted"])+ self.assertEqual(self.carol.chats()["chats"][0]["unread"], unread) # the thread view moved no read position+ self.assertEqual(self.bob.read()["messages"][0]["text"], "post")+ with self.assertRaises(ClientError) as e: # only members see a thread+ self.server.user("dora", "Dora Diaz").read(thread=post)+ self.assertEqual(e.exception.status, 404)++ def test_replies_say_whose_message_they_answer(self):+ board = self.alice.channel_create(name="board", members="bob,carol")["chat"]["id"]+ post = self.alice.send(chat=board, text="post")["message"]["id"]+ reply = self.bob.send(chat=board, text="reply", reply_to=post)["message"]+ self.assertEqual(reply["reply_to_from"], "alice")+ self.assertEqual(self.alice.send(chat=board, text="answer", reply_to=reply["id"])["message"]["reply_to_from"],+ "bob")+ self.alice.read()+ self.carol.send(chat=board, text="unrelated")+ self.carol.send(chat=board, text="to bob", reply_to=reply["id"])+ self.carol.send(chat=board, text="to alice", reply_to=post)+ self.assertEqual(self.texts(self.alice.wait(timeout=2, where="m.reply_to_from == 'alice'")), ["to alice"])+ self.restart() # worked out from the log, so nothing to store+ self.assertEqual([m.get("reply_to_from") for m in self.carol.read(chat=board, all=True)["messages"]],+ [None, "alice", "bob", None, "bob", "alice"])++ def test_sort_orders_the_chosen_messages_then_limits(self):+ board = self.alice.channel_create(name="board", members="bob,carol")["chat"]["id"]+ ids = [self.alice.send(chat=board, text=f"post {i}")["message"]["id"] for i in range(5)]+ for voter, post in ((self.bob, 3), (self.carol, 3), (self.bob, 1), (self.carol, 4)):+ voter.react(id=ids[post], emoji="+1")+ votes = "-len((m.reactions or {}).get('+1', []))"+ top = self.bob.read(chat=board, all=True, sort=votes, limit=3, peek=True)+ self.assertEqual(self.texts(top), ["post 3", "post 1", "post 4"]) # a tie keeps the order in time+ self.assertEqual(self.texts(self.bob.read(chat=board, last=3, sort=votes, peek=True)),+ ["post 3", "post 4", "post 2"]) # the last 3, ordered+ self.assertEqual(self.texts(self.bob.read(chat=board, last=3, where="m.text != 'post 4'", sort=votes,+ peek=True)), ["post 3", "post 1", "post 2"])+ self.assertEqual(len(self.bob.read(chat=board, all=True, sort="m.text", peek=True)["messages"]), 5)+ self.assertEqual(self.texts(self.bob.read(chat=board, sort=votes))[:2], ["post 3", "post 1"]) # unread ones+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 0)+ for bad, words in (("m.reactions", "cannot order"), ("m.text +", "not a Python expression")):+ with self.assertRaises(ClientError) as e:+ self.bob.read(chat=board, all=True, sort=bad)+ self.assertIn(words, str(e.exception))++ def test_own_messages_are_never_unread(self):+ self.alice.send(chat="bob", text="mine")+ self.assertEqual(self.alice.chats()["chats"][0]["unread"], 0)+ self.assertEqual(self.alice.read()["messages"], [])+++class Receipts(Base):+ def mine(self, client, chat):+ return client.read(chat=chat, last=1, peek=True)["messages"][0]++ def test_direct_chat_receipt(self):+ self.alice.send(chat="bob", text="did you get this?")+ m = self.mine(self.alice, "bob")+ self.assertEqual((m["read_by"], m["unread_by"]), ([], ["bob"]))+ self.bob.read()+ m = self.mine(self.alice, "bob")+ self.assertEqual([r["id"] for r in m["read_by"]], ["bob"])+ self.assertTrue(m["read_by"][0]["at"])+ self.assertNotIn("read_by", self.mine(self.bob, "alice")) # only the sender sees receipts++ def test_channel_receipts_and_restart(self):+ self.server.user("dan", "Dan Dorr")+ chat = self.alice.channel_create(name="team", members="bob,carol,dan")["chat"]["id"]+ self.alice.send(chat=chat, text="please read")+ before = self.alice.chats()["chats"][0]["last_at"]+ self.bob.read()+ self.carol.wait(timeout=1)+ self.assertEqual(self.alice.chats()["chats"][0]["last_at"], before) # reading is not chat activity+ self.restart()+ m = self.mine(self.alice, chat)+ self.assertEqual(sorted(r["id"] for r in m["read_by"]), ["bob", "carol"])+ self.assertEqual(m["unread_by"], ["dan"])++ def test_peeking_leaves_no_receipt(self):+ self.alice.send(chat="bob", text="x")+ self.bob.read(peek=True)+ self.assertEqual(self.mine(self.alice, "bob")["unread_by"], ["bob"])+++class Waiting(Base):+ def test_wait_blocks_until_a_message_arrives(self):+ threading.Timer(0.3, lambda: self.alice.send(chat="bob", text="ping")).start()+ t = time.monotonic()+ r = self.bob.wait(timeout=10)+ self.assertEqual(self.texts(r), ["ping"])+ self.assertLess(time.monotonic() - t, 2)+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 0)++ def test_wait_times_out_empty(self):+ t = time.monotonic()+ self.assertEqual(self.bob.wait(timeout=0.5)["messages"], [])+ self.assertGreaterEqual(time.monotonic() - t, 0.5)++ def test_nothing_is_lost_between_waits(self):+ self.alice.send(chat="bob", text="one")+ self.alice.send(chat="bob", text="two")+ self.assertEqual(self.texts(self.bob.wait(timeout=1)), ["one", "two"])+ self.alice.send(chat="bob", text="three") # arrives while bob is not waiting+ self.assertEqual(self.texts(self.bob.wait(timeout=1)), ["three"])++ def test_wait_filters(self):+ chat = self.alice.send(chat="bob,carol", text="start")["chat"]["id"]+ self.bob.read()+ self.bob.settings_notify(chat=chat, level="none")+ self.alice.send(chat=chat, text="muted")+ self.assertEqual(self.bob.wait(timeout=0.3)["messages"], [])+ self.assertEqual(self.texts(self.bob.wait(timeout=0.3, chat=chat)), ["muted"])+ threading.Timer(0.2, lambda: [self.carol.send(chat="bob", text="noise"),+ self.alice.send(chat="bob", text="the build")]).start()+ r = self.bob.wait(timeout=5, where="'build' in m.text")+ self.assertEqual(self.texts(r), ["the build"])++ def test_unconfirmed_messages_stay_unread_when_the_server_cannot_be_reached(self):+ self.alice.send(chat="bob", text="confirm me")+ self.assertEqual(self.texts(self.bob.wait(timeout=1)), ["confirm me"])+ self.bob._drop()+ self.bob.port = 9 # the server cannot be reached now: nothing listens there+ self.bob.close() # cannot confirm: must neither raise nor loop+ self.assertTrue(self.bob.ack)+ again = self.server.client(self.bob.token)+ self.assertEqual(self.texts(again.read()), ["confirm me"]) # it comes again, nothing is lost++ def test_many_waiters_wake(self):+ users = [self.server.user(f"u{i}", f"User {i}") for i in range(30)]+ results = []+ threads = [threading.Thread(target=lambda u=u: results.append(u.wait(timeout=10))) for u in users]+ [t.start() for t in threads]+ time.sleep(0.3)+ self.alice.channel_create(name="all hands", members=",".join(f"u{i}" for i in range(30)))+ self.alice.send(chat="#all hands", text="go")+ [t.join(5) for t in threads]+ self.assertEqual([self.texts(r) for r in results], [["go"]] * 30)+++class Storage(Base):+ def test_log_files_rotate_and_reload(self):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ server = Server(data, segment=7)+ try:+ a, b = server.user("a", "A"), server.user("b", "B")+ chat = a.send(chat="b", text="0")["chat"]["id"] # line 1 is the chat's creation, line 2 this message+ for i in range(1, 20):+ a.send(chat="b", text=str(i))+ logs = sorted(os.listdir(os.path.join(data, "chats", chat, "log")))+ self.assertEqual(logs, ["000001.jsonl", "000002.jsonl", "000003.jsonl"])+ server.stop()+ server = Server(data) # the folder keeps its own segment size, whatever is asked now+ self.assertEqual(server.store.segment, 7)+ b = server.client(b.token)+ self.assertEqual([m["text"] for m in b.read(chat="a", last=3)["messages"]], ["17", "18", "19"])+ server.client(a.token).send(chat="b", text="20")+ self.assertEqual([m["text"] for m in b.read(chat="a", last=2)["messages"]], ["19", "20"])+ finally:+ server.stop()+ shutil.rmtree(data)++ def test_a_half_written_line_is_dropped(self):+ chat = self.alice.send(chat="bob", text="kept")["chat"]["id"]+ self.server.stop()+ with open(os.path.join(self.data, "chats", chat, "log", "000001.jsonl"), "ab") as f:+ f.write(b'{"seq":3,"kind":"mess')+ self.server = Server(self.data)+ self.alice = self.server.client(self.alice.token)+ self.bob = self.server.client(self.bob.token)+ self.alice.send(chat="bob", text="after")+ self.assertEqual(self.texts(self.bob.read()), ["kept", "after"])++ def test_policy_contacts(self):+ self.server.admin.policy_set(rules={"contacts": {"alice": ["bob"], "bob": ["carol"]}})+ self.alice.send(chat="bob", text="fine")+ self.bob.send(chat="alice", text="contacts work both ways")+ self.bob.channel_create(name="team", members="alice,carol") # bob may reach both+ self.alice.send(chat="#team", text="anyone in a channel may post in it")+ self.bob.channel_create(name="pair", members="alice")+ for call in (lambda: self.alice.send(chat="carol", text="not a contact"),+ lambda: self.carol.send(chat="alice,bob", text="alice is not carol's contact"),+ lambda: self.alice.channel_create(name="mine", members="carol"),+ lambda: self.alice.channel_add(chat="#pair", users="carol")):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, 403)++ def test_policy_posters(self):+ records = self.alice.channel_create(name="records", members="bob,carol")["chat"]["id"]+ self.alice.channel_create(name="talk", members="bob")+ self.server.admin.policy_set(rules={"posters": {"#Records": ["alice"]}})+ first = self.alice.send(chat="#records", text="new best: 42")["message"]["id"]+ self.bob.send(chat="#talk", text="other channels are open")+ self.bob.react(id=first, emoji="+1")+ for call in (lambda: self.bob.send(chat="#records", text="me too"),+ lambda: self.carol.send(chat=records, text="by id"),+ lambda: self.bob.send(chat="#records", text="a reply", reply_to=first)):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, 403)+ self.assertIn("Only alice may post in #records", str(e.exception))+ self.alice.edit(id=first, text="new best: 43")+ self.assertEqual(self.texts(self.carol.read(chat="#records")), ["new best: 43"])+ self.server.admin.policy_set(rules={"posters": {records: ["bob"]}}) # by id+ self.bob.send(chat="#records", text="bob now")+ with self.assertRaises(ClientError):+ self.alice.send(chat="#records", text="alice not any more")+++class MorePaths(Base):+ def test_channel_members_and_names_change(self):+ chat = self.alice.channel_create(name="ops", members="bob")["chat"]["id"]+ self.alice.channel_add(chat="#ops", users="carol")+ self.carol.send(chat="#ops", text="joined")+ self.alice.channel_rename(chat=chat, name="operations")+ self.assertEqual(self.bob.chats(name="operations")["chats"][0]["id"], chat)+ self.alice.channel_remove(chat=chat, users="bob")+ with self.assertRaises(ClientError):+ self.bob.send(chat=chat, text="still here?")+ self.assertEqual(self.texts(self.carol.read(chat="operations", last=5)), ["joined"])++ def test_tokens_rotate_and_users_go(self):+ old = self.bob.token+ self.bob = self.server.client(self.server.admin.token_rotate(id="bob")["token"])+ with self.assertRaises(ClientError) as e:+ self.server.client(old).chats()+ self.assertEqual(e.exception.status, 401)+ direct = self.alice.send(chat="bob,carol", text="hi")["chat"]["id"]+ channel = self.alice.channel_create(name="team", members="bob,carol")["chat"]["id"]+ self.server.admin.user_remove(id="bob")+ self.assertEqual([m["id"] for m in self.alice.chat_info(chat=channel)["chat"]["members"]], ["alice", "carol"])+ self.assertEqual(len(self.alice.chat_info(chat=direct)["chat"]["members"]), 3) # kept as a record+ self.assertEqual([u["id"] for u in self.alice.people()["people"]], ["alice", "carol"])++ def test_status_search_filters_and_one_message(self):+ self.bob.settings_status(text="out this week")+ self.assertEqual([u["id"] for u in self.alice.people(search="out this")["people"]], ["bob"])+ dm = self.alice.send(chat="bob", text="dm")["message"]+ self.alice.send(chat="carol", text="other")+ self.assertEqual([c["title"] for c in self.alice.chats(**{"with": "bob"})["chats"]], ["Bob Burns"])+ self.assertEqual(self.alice.chats(sort="name")["chats"][0]["title"], "Bob Burns")+ self.assertEqual(self.bob.read(id=dm["id"])["messages"][0]["text"], "dm")+ self.assertEqual(len(self.bob.notifications(all=True)["chats"]), 1)++ def test_bad_requests_are_refused_clearly(self):+ m = self.alice.send(chat="bob", text="hi")["message"]+ other = self.alice.send(chat="carol", text="hi")["message"]+ for call, status in ((lambda: self.alice.send(chat="bob", text=" "), 400),+ (lambda: self.alice.send(chat="bob", text="x" * 20001), 400),+ (lambda: self.alice.send(chat="bob", text="r", reply_to=other["id"]), 400),+ (lambda: self.carol.read(id=m["id"]), 404),+ (lambda: self.alice.react(id=m["id"], emoji=""), 400),+ (lambda: self.alice.settings_notify(chat="bob", level="loud"), 400),+ (lambda: self.alice.channel_create(name=""), 400),+ (lambda: self.alice.nonsense(), 404)):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, status)+ self.alice.delete(id=m["id"])+ with self.assertRaises(ClientError):+ self.alice.edit(id=m["id"], text="back")++ def test_unicode_and_reactions_come_and_go(self):+ m = self.alice.send(chat="bob", text="naïve café ☕\nsecond line")["message"]+ self.bob.react(id=m["id"], emoji="☕")+ self.bob.react(id=m["id"], emoji="☕", remove=True)+ got = self.bob.read()["messages"][0]+ self.assertEqual(got["text"], "naïve café ☕\nsecond line")+ self.assertNotIn("reactions", got)+ self.assertEqual(len(self.bob.read(search="CAFÉ", all=True)["messages"]), 1)++ def test_wait_by_sender_and_by_mention(self):+ chat = self.alice.send(chat="bob,carol", text="start")["chat"]["id"]+ self.bob.read()+ self.carol.send(chat=chat, text="from carol")+ self.alice.send(chat=chat, text="from alice")+ self.assertEqual(self.texts(self.bob.wait(timeout=1, **{"from": "alice"})), ["from alice"])+ self.alice.send(chat=chat, text="plain")+ self.alice.send(chat=chat, text="@bob look")+ self.assertEqual(self.texts(self.bob.wait(timeout=1, mentions=True)), ["@bob look"])++ def test_durable_mode_and_a_normal_stop_keep_read_positions(self):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ port = 18700 + os.getpid() % 200+ env = {**os.environ, "PYTHONPATH": ROOT}+ cmd = [sys.executable, "-m", "swarmchat", "serve", "--data", data, "--port", str(port), "--fsync"]+ group = {"creationflags": subprocess.CREATE_NEW_PROCESS_GROUP} if os.name == "nt" else {}+ server = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, **group)+ try:+ url = f"http://127.0.0.1:{port}"+ for _ in range(100):+ try:+ with open(os.path.join(data, "admin.token")) as f:+ admin = Client(url, f.read().strip())+ admin.policy_show()+ break+ except (OSError, ClientError):+ time.sleep(0.05)+ a = Client(url, admin.user_add(id="a")["token"])+ b = Client(url, admin.user_add(id="b")["token"])+ a.send(chat="b", text="one")+ self.assertEqual(self.texts(b.read()), ["one"])+ # A normal stop (SIGTERM, or Ctrl-Break on Windows): read positions are written before it exits.+ server.send_signal(signal.CTRL_BREAK_EVENT if os.name == "nt" else signal.SIGTERM)+ server.wait(10)+ server = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, **group)+ for _ in range(100):+ try:+ self.assertEqual(b.read()["messages"], [])+ break+ except ClientError:+ time.sleep(0.05)+ for c in (a, b, admin):+ c.close()+ finally:+ server.kill()+ server.wait()+ shutil.rmtree(data)+++class PublicChannels(Base):+ def listed(self, client, **params):+ return [(c["name"], c["members"], c["joined"]) for c in client.channel_list(**params)["channels"]]++ def refused(self, call, status, words):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, status)+ self.assertIn(words, str(e.exception))++ def test_a_channel_is_private_unless_made_public(self):+ self.alice.channel_create(name="plans", members="bob")+ self.assertEqual(self.listed(self.carol), [])+ self.refused(lambda: self.carol.channel_join(chat="plans"), 404, "There is no channel plans")+ self.refused(lambda: self.carol.read(chat="#plans"), 404, "There is no channel #plans")+ board = self.alice.channel_create(name="board", public=True)["chat"]+ self.assertTrue(board["public"])+ self.assertEqual(self.listed(self.carol), [("board", 1, False)])+ self.refused(lambda: self.carol.send(chat="#board", text="hi"), 403, "It is public, so you can join it")+ self.assertEqual(self.carol.channel_join(chat="#board")["chat"]["name"], "board")+ self.carol.channel_join(chat=board["id"]) # by id too, and twice is fine+ self.carol.send(chat="#board", text="first post")+ self.assertEqual(self.texts(self.alice.read(chat="#board")), ["first post"])+ self.assertEqual(self.listed(self.carol), [("board", 2, True)])+ self.assertEqual(self.listed(self.bob, search="BOA"), [("board", 2, False)])+ self.assertEqual(self.listed(self.bob, search="plans"), [])++ def test_members_change_visibility_and_members_stay(self):+ self.alice.channel_create(name="board", public=True)+ self.carol.channel_join(chat="board")+ self.refused(lambda: self.bob.channel_visibility(chat="#board", visibility="private"), 403, "not in #board")+ self.carol.channel_visibility(chat="#board", visibility="private")+ self.assertEqual(self.listed(self.bob), [])+ self.refused(lambda: self.bob.channel_join(chat="board"), 404, "There is no channel board")+ self.carol.send(chat="#board", text="still here") # members stay+ self.alice.channel_create(name="plans")+ self.alice.channel_visibility(chat="#plans", visibility="public")+ self.restart() # visibility is kept+ self.assertEqual(self.listed(self.bob), [("plans", 1, False)])+ self.bob.channel_join(chat="plans")+ self.alice.send(chat="bob", text="a direct chat")+ self.refused(lambda: self.alice.channel_visibility(chat="bob", visibility="public"), 400, "Only a channel")+ self.refused(lambda: self.alice.channel_visibility(chat="#plans", visibility="open"), 400, "public or private")+ self.refused(lambda: self.bob.channel_join(chat="alice"), 404, "no channel")+ kinds = [json.loads(line).get("public") for line in open(os.path.join(+ self.data, "chats", self.alice.chats(name="plans")["chats"][0]["id"], "log", "000001.jsonl"))+ if json.loads(line)["kind"] == "visibility"]+ self.assertEqual(kinds, [True]) # the change is in the chat's history++ def test_a_channel_has_a_description_its_members_change(self):+ address = "Release 2.4, shipping Friday"+ chat = self.alice.channel_create(name="release-2-4", members="bob", public=True, description=f" {address} ")+ self.assertEqual(chat["chat"]["description"], address)+ self.assertEqual(self.carol.channel_list()["channels"][0]["description"], address)+ self.assertEqual(self.alice.channel_create(name="plain")["chat"]["description"], "")+ self.refused(lambda: self.carol.channel_describe(chat="#release-2-4", description="mine"), 403, "not in")+ self.assertEqual(self.bob.channel_describe(chat="#release-2-4", description="Freeze moved to Monday")["chat"]+ ["description"], "Freeze moved to Monday")+ self.bob.channel_describe(chat="#release-2-4", description="Freeze moved to Monday") # the same is no change+ self.refused(lambda: self.bob.channel_describe(chat="#release-2-4", description="x" * 1001), 400, "1,000")+ self.refused(lambda: self.bob.channel_describe(chat="#release-2-4", description=["a"]), 400, "is text")+ self.refused(lambda: self.alice.channel_create(name="big", description="x" * 1001), 400, "1,000")+ self.alice.send(chat="bob", text="a direct chat")+ self.refused(lambda: self.alice.channel_describe(chat="bob", description="us"), 400, "Only a channel")+ self.restart() # the description is kept+ self.assertEqual(self.bob.chat_info(chat="#release-2-4")["chat"]["description"], "Freeze moved to Monday")+ snapshot = self.server.admin.view_snapshot()["chats"]+ self.assertEqual(next(c["description"] for c in snapshot if c["id"] == chat["chat"]["id"]),+ "Freeze moved to Monday")+ self.alice.channel_describe(chat="#release-2-4", description="")+ self.assertEqual(self.bob.chats(name="release-2-4")["chats"][0]["description"], "")+ with open(os.path.join(self.data, "chats", chat["chat"]["id"], "log", "000001.jsonl")) as f:+ log = [json.loads(line) for line in f]+ self.assertEqual(log[0]["description"], address) # the chat's history has every description+ self.assertEqual([(e["from"], e["description"]) for e in log if e["kind"] == "described"],+ [("bob", "Freeze moved to Monday"), ("alice", "")])++ def test_anyone_may_join_a_public_channel_under_contact_rules(self):+ self.server.admin.policy_set(rules={"contacts": {"alice": ["bob"]}})+ self.alice.channel_create(name="board", members="bob", public=True)+ self.alice.channel_create(name="plans", members="bob")+ self.carol.channel_join(chat="board") # making it public was the check+ self.carol.send(chat="#board", text="hello all")+ self.refused(lambda: self.carol.channel_join(chat="plans"), 404, "There is no channel plans")+ self.refused(lambda: self.carol.send(chat="alice", text="hi"), 403, "may not contact")+++class Administrators(Base):+ def refused(self, call, status, words):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, status)+ self.assertIn(words, str(e.exception))++ def log(self, chat):+ with open(os.path.join(self.data, "chats", chat, "log", "000001.jsonl")) as f:+ return [json.loads(line) for line in f]++ def test_managers_and_administrators_remove_others(self):+ self.alice.channel_create(name="ops", members="bob,carol")+ self.bob.channel_remove(chat="#ops", users="carol") # an open server: any member may+ self.alice.channel_add(chat="#ops", users="carol")+ self.server.admin.policy_set(rules={"admins": ["Carol"]})+ self.refused(lambda: self.bob.channel_remove(chat="#ops", users="carol"), 403,+ "Only a channel manager or an administrator may remove someone else from a channel.")+ self.refused(lambda: self.bob.channel_remove(chat="#ops", users="bob,carol"), 403, "Only a channel manager")+ self.bob.channel_leave(chat="#ops") # anyone may still leave+ self.bob.channel_create(name="bobs", members="carol")+ self.bob.channel_remove(chat="#bobs", users="bob")+ self.alice.channel_add(chat="#ops", users="bob") # any member adds people+ self.carol.channel_remove(chat="#ops", users="bob") # an administrator in the channel+ self.alice.channel_add(chat="#ops", users="bob")+ self.alice.channel_remove(chat="#ops", users="bob") # its creator manages it+ self.assertEqual(sorted(m["id"] for m in self.alice.chat_info(chat="#ops")["chat"]["members"]),+ ["alice", "carol"])++ def test_a_channels_managers_manage_it(self):+ chat = self.alice.channel_create(name="ops", members="bob,carol")["chat"]+ self.assertEqual(chat["managers"], ["alice"])+ self.server.admin.policy_set(rules={"admins": []}) # roles, and no administrators+ for call, what in ((lambda: self.bob.channel_rename(chat="#ops", name="ops2"), "rename a channel"),+ (lambda: self.bob.channel_describe(chat="#ops", description="Mine"),+ "change a channel's description"),+ (lambda: self.bob.channel_visibility(chat="#ops", visibility="private"),+ "make a channel private"),+ (lambda: self.bob.channel_archive(chat="#ops"), "archive or unarchive a channel"),+ (lambda: self.bob.channel_managers(chat="#ops", add="bob"), "choose a channel's managers")):+ self.refused(call, 403, f"Only a channel manager or an administrator may {what}.")+ self.assertEqual(self.bob.channel_managers(chat="#ops")["chat"]["managers"], ["alice"]) # members may look+ self.refused(lambda: self.alice.channel_visibility(chat="#ops", visibility="public"), 403,+ "Only an administrator may make a private channel public.") # its manager may not+ self.server.admin.policy_set(rules={"admins": ["carol"]})+ self.assertTrue(self.carol.channel_visibility(chat="#ops", visibility="public")["chat"]["public"])+ self.assertFalse(self.alice.channel_visibility(chat="#ops", visibility="private")["chat"]["public"])+ self.server.admin.policy_set(rules={"admins": [], "allow": {"make_public": ["manager"]}})+ self.alice.channel_visibility(chat="#ops", visibility="public")+ self.alice.channel_visibility(chat="#ops", visibility="private")+ self.server.admin.policy_set(rules={"admins": []})+ self.server.user("dave", "Dave Diaz")+ self.refused(lambda: self.alice.channel_managers(chat="#ops", add="dave"), 400, "dave is not in #ops")+ self.assertEqual(self.alice.channel_managers(chat="#ops", add="bob")["chat"]["managers"], ["alice", "bob"])+ self.bob.channel_rename(chat="#ops", name="ops2")+ self.bob.channel_describe(chat="#ops2", description="Operations")+ self.assertEqual(self.bob.channel_managers(chat="#ops2", remove="alice")["chat"]["managers"], ["bob"])+ self.refused(lambda: self.alice.channel_rename(chat="#ops2", name="ops"), 403, "Only a channel manager")+ self.restart() # managers are kept+ self.assertEqual(self.carol.chat_info(chat="#ops2")["chat"]["managers"], ["bob"])+ self.bob.channel_leave(chat="#ops2") # leaving ends the role+ self.assertEqual(self.alice.chat_info(chat="#ops2")["chat"]["managers"], [])+ self.server.admin.policy_set(rules={"admins": ["carol"]})+ self.carol.channel_managers(chat="#ops2", add="alice") # an administrator in the channel names one+ self.alice.channel_rename(chat="#ops2", name="ops")+ self.assertEqual([(e["from"], e.get("added"), e.get("removed")) for e in self.log(chat["id"])+ if e["kind"] == "managers"],+ [("alice", ["bob"], None), ("bob", None, ["alice"]), ("carol", ["alice"], None)])+ self.alice.send(chat="bob", text="direct")+ self.refused(lambda: self.alice.channel_managers(chat="bob", add="bob"), 400, "Only a channel has managers")+ # A channel from before managers: its creator manages it.+ old = self.alice.channel_create(name="old", members="bob")["chat"]["id"]+ meta_path = os.path.join(self.data, "chats", old, "meta.json")+ with open(meta_path) as f:+ meta = json.load(f)+ del meta["managers"]+ with open(meta_path, "w") as f:+ json.dump(meta, f)+ self.restart()+ self.assertEqual(self.bob.chat_info(chat="#old")["chat"]["managers"], ["alice"])++ def test_the_super_admin_lets_managers_delete_messages(self):+ self.alice.channel_create(name="ops", members="bob")+ first = self.bob.send(chat="#ops", text="spam")["message"]["id"]+ self.refused(lambda: self.alice.delete(id=first), 403, "Only an administrator may delete another person's")+ self.server.admin.policy_set(rules={"admins": [], "allow": {"delete_message": ["manager", "admin"]}})+ self.assertTrue(self.alice.delete(id=first)["message"]["deleted"])+ self.server.admin.policy_set(rules={"admins": [], "allow": {"delete_message": []}})+ second = self.bob.send(chat="#ops", text="more spam")["message"]["id"]+ self.refused(lambda: self.alice.delete(id=second), 403, "No one may delete another person's message.")+ self.bob.delete(id=second) # the sender still may++ def test_only_the_super_admin_makes_accounts_tokens_and_rules(self):+ self.server.admin.policy_set(rules={"admins": ["alice"]})+ for call, what in ((lambda: self.alice.user_add(id="dora"), "add users"),+ (lambda: self.alice.user_remove(id="bob"), "remove users"),+ (lambda: self.alice.token_rotate(id="bob"), "give someone else a new token"),+ (lambda: self.alice.role_change(role="admin", add="bob"), "choose the administrators")):+ self.refused(call, 403, f"Only the super admin may {what}.")+ self.refused(lambda: self.alice.policy_set(rules={"admins": ["alice", "bob"]}), 403,+ "Only the super admin may set the rules.")+ self.bob.whoami() # bob's token still works: an administrator cannot take over an account+ self.assertEqual(self.alice.policy_show()["rules"], {"admins": ["alice"]}) # administrators may look+ self.assertEqual(self.alice.role_change(role="admin")["users"], ["alice"])+ self.refused(lambda: self.bob.policy_show(), 403, "Only the super admin or an administrator may see the rules.")+ self.refused(lambda: self.server.client("nonsense").policy_show(), 401, "the admin token")+ # The super admin may hand any action to other roles, here naming administrators.+ self.server.admin.policy_set(rules={"admins": ["alice"], "allow": {"edit_admins": ["superadmin", "admin"]}})+ self.assertEqual(self.alice.role_change(role="admin", add="bob")["users"], ["alice", "bob"])+ self.bob.role_change(role="admin", remove="alice")+ self.assertEqual(self.server.admin.role_change(role="admin")["users"], ["bob"])+ self.refused(lambda: self.alice.policy_show(), 403, "Only the super admin or an administrator")+ for rules, words in (({"allow": {"fly": ["admin"]}}, "There is no action 'fly'"),+ ({"allow": {"add_user": ["king"]}}, "is a list of roles"),+ ({"allow": {"add_user": "admin"}}, "is a list of roles"),+ ({"allow": ["add_user"]}, "maps actions"),+ ({"admins": "alice"}, "list of user ids"),+ ({"channel_creators": [1]}, "list of user ids")):+ self.refused(lambda: self.server.admin.policy_set(rules=rules), 400, words)+ self.assertIsNone(self.server.admin.role_change(role="channel_creator")["users"]) # everyone+ self.server.admin.policy_set(rules={})+ self.assertEqual(self.server.admin.role_change(role="admin")["users"], [])+ self.refused(lambda: self.server.admin.role_change(role="king"), 400, "admin or channel_creator")+ self.refused(lambda: self.server.admin.role_change(role="admin", everyone=True), 400, "Only channel_creator")+ self.refused(lambda: self.server.admin.role_change(role="admin", add="nobody"), 404, "No one called")++ def test_who_may_create_channels(self):+ self.bob.channel_create(name="anyone") # without rules, everyone may+ self.server.admin.policy_set(rules={"admins": ["alice"]})+ self.bob.channel_create(name="still-anyone") # naming administrators leaves it to everyone+ self.assertEqual(self.alice.role_change(role="channel_creator", add="carol")["users"], ["carol"])+ self.refused(lambda: self.bob.channel_create(name="bobs"), 403,+ "Only someone chosen to create channels or an administrator may create a channel.")+ self.carol.channel_create(name="carols")+ self.alice.channel_create(name="alices") # administrators always may+ self.bob.send(chat="carol", text="a direct chat needs no permission")+ self.refused(lambda: self.bob.role_change(role="channel_creator", add="bob"), 403,+ "Only the super admin or an administrator may choose who may create channels.")+ self.assertIsNone(self.alice.role_change(role="channel_creator", everyone=True)["users"])+ self.bob.channel_create(name="bobs")+ self.assertEqual(self.server.admin.policy_show()["rules"], {"admins": ["alice"]})++ def test_anyone_gives_themselves_a_new_token(self):+ new = self.bob.settings_token()["token"]+ self.refused(self.bob.whoami, 401, "Unknown token")+ self.assertEqual(self.server.client(new).whoami()["user"]["id"], "bob")++ def test_an_administrator_deletes_anyones_message(self):+ chat = self.alice.channel_create(name="ops", members="bob,carol")["chat"]["id"]+ first = self.bob.send(chat="#ops", text="a secret")["message"]["id"]+ self.refused(lambda: self.alice.delete(id=first), 403, "Only an administrator may delete another person's")+ self.server.admin.policy_set(rules={"admins": ["alice"]})+ self.refused(lambda: self.carol.delete(id=first), 403, "Only an administrator may delete another person's")+ self.refused(lambda: self.alice.edit(id=first, text="mine now"), 403, "Only the sender can edit")+ self.assertTrue(self.alice.delete(id=first)["message"]["deleted"])+ shown = self.carol.read(chat="#ops", all=True)["messages"][0]+ self.assertEqual((shown["deleted"], shown["text"], shown["from"]), (True, "", "bob"))+ self.assertEqual([(e["from"], e["target"]) for e in self.log(chat) if e["kind"] == "delete"], [("alice", 2)])+ self.bob.send(chat="alice", text="direct")+ theirs = self.alice.send(chat="bob,carol", text="group")["message"]["id"]+ self.carol.delete(id=self.carol.send(chat="#ops", text="own")["message"]["id"]) # the sender still may+ self.alice.delete(id=self.bob.read(chat="alice", all=True)["messages"][0]["id"]) # in any chat she is in+ self.refused(lambda: self.bob.delete(id=theirs), 403, "Only an administrator may delete another person's")++ def test_archiving_closes_a_channel_and_keeps_it(self):+ chat = self.alice.channel_create(name="ops", members="bob", public=True)["chat"]["id"]+ first = self.bob.send(chat="#ops", text="before")["message"]["id"]+ self.server.admin.policy_set(rules={"admins": ["alice"]})+ self.refused(lambda: self.bob.channel_archive(chat="#ops"), 403, "Only a channel manager or an administrator")+ self.assertTrue(self.alice.channel_archive(chat="#ops")["chat"]["archived"])+ self.alice.channel_archive(chat="#ops") # twice is fine+ for call in (lambda: self.bob.send(chat="#ops", text="after"),+ lambda: self.bob.send(chat=chat, text="reply", reply_to=first),+ lambda: self.bob.edit(id=first, text="changed"),+ lambda: self.bob.react(id=first, emoji="+1"),+ lambda: self.alice.delete(id=first),+ lambda: self.alice.channel_add(chat="#ops", users="carol"),+ lambda: self.carol.channel_join(chat="#ops"),+ lambda: self.alice.channel_rename(chat="#ops", name="ops2"),+ lambda: self.alice.channel_describe(chat="#ops", description="closed"),+ lambda: self.alice.channel_visibility(chat="#ops", visibility="private")):+ self.refused(call, 403, "#ops is archived")+ self.refused(lambda: self.carol.channel_create(name="ops"), 400, "It is archived. Unarchive it")+ self.assertEqual(self.carol.channel_list()["channels"], [])+ self.assertEqual(self.texts(self.bob.read(chat="#ops", all=True)), ["before"]) # members still read+ self.assertTrue(self.bob.chats()["chats"][0]["archived"])+ snapshot = self.server.admin.view_snapshot()["chats"]+ self.assertTrue(next(c for c in snapshot if c["id"] == chat)["archived"])+ self.restart()+ self.assertTrue(self.bob.chat_info(chat="#ops")["chat"]["archived"])+ self.refused(lambda: self.bob.send(chat="#ops", text="after a restart"), 403, "#ops is archived")+ self.refused(lambda: self.bob.channel_unarchive(chat="#ops"), 403, "Only a channel manager")+ self.assertFalse(self.alice.channel_unarchive(chat="#ops")["chat"]["archived"])+ self.bob.send(chat="#ops", text="open again")+ self.assertEqual([e["kind"] for e in self.log(chat) if e["kind"] in ("archived", "unarchived")],+ ["archived", "unarchived"])+ self.alice.send(chat="bob", text="a direct chat")+ self.refused(lambda: self.alice.channel_archive(chat="bob"), 400, "Only a channel")++ def test_members_leave_an_archived_channel(self):+ self.alice.channel_create(name="ops", members="bob,carol")+ self.bob.channel_archive(chat="#ops") # without the rule, any member may archive+ self.bob.channel_leave(chat="#ops")+ self.alice.channel_remove(chat="#ops", users="carol")+ self.server.admin.user_remove(id="alice") # removing an account takes it out of archived channels too+ self.assertEqual(self.server.store.channels["ops"].members, [])+++class Hardening(Base):+ """What an audit of hidden-information boundaries asked for."""++ def refused(self, call, status, words):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, status)+ self.assertIn(words, str(e.exception))++ def log_bytes(self, chat):+ folder, out = os.path.join(self.data, "chats", chat, "log"), b""+ for name in sorted(os.listdir(folder)):+ with open(os.path.join(folder, name), "rb") as f:+ out += f.read()+ return out++ @unittest.skipIf(os.name == "nt", "POSIX file modes")+ def test_the_data_folder_is_its_owners_alone(self):+ chat = self.alice.channel_create(name="ops", members="bob")["chat"]["id"]+ self.alice.send(chat="#ops", text="hello")+ mode = lambda *parts: os.stat(os.path.join(self.data, *parts)).st_mode & 0o777 # noqa: E731+ self.assertEqual([mode(), mode("chats"), mode("chats", chat), mode("chats", chat, "log"), mode("users")],+ [0o700] * 5)+ self.assertEqual(mode("chats", chat, "log", "000001.jsonl"), 0o600)+ os.chmod(self.data, 0o755) # a folder from before: made private on the next start+ self.restart()+ self.assertEqual(mode(), 0o700)++ def test_deleting_a_message_erases_its_words_on_disk(self):+ chat = self.alice.channel_create(name="ops", members="bob")["chat"]["id"]+ keep = self.alice.send(chat="#ops", text="first")["message"]["id"]+ gone = self.bob.send(chat="#ops", text="secret-words-123", reply_to=keep)["message"]["id"]+ self.bob.edit(id=gone, text="edited-secret-456")+ self.alice.send(chat="#ops", text="after", reply_to=gone)+ self.bob.delete(id=gone)+ self.assertNotIn(b"secret", self.log_bytes(chat))+ self.restart() # every line still reads where the index says+ self.assertEqual(self.texts(self.alice.read(chat="#ops", all=True)), ["first", "", "after"])+ self.assertEqual(len(self.alice.read(thread=keep)["messages"]), 3)+ # A message deleted before deletion erased the words: they go on the next start.+ old = self.bob.send(chat="#ops", text="old-secret-789")["message"]["id"]+ tokens = {name: getattr(self, name).token for name in ("alice", "bob", "carol")}+ self.server.stop()+ seq = len(self.log_bytes(chat).splitlines()) + 1+ with open(os.path.join(self.data, "chats", chat, "log", "000001.jsonl"), "ab") as f:+ f.write(json.dumps({"seq": seq, "kind": "delete", "from": "bob", "at": "2026-10-24T00:00:00.000Z",+ "target": int(old.split(":")[1])}).encode() + b"\n")+ self.server = Server(self.data)+ for name, token in tokens.items():+ setattr(self, name, self.server.client(token))+ self.assertNotIn(b"old-secret", self.log_bytes(chat))+ self.assertEqual(self.texts(self.alice.read(chat="#ops", all=True))[-1], "")++ def test_a_private_channel_does_not_answer_outsiders(self):+ chat = self.alice.channel_create(name="merger-talks", members="bob")["chat"]["id"]+ self.assertRegex(chat, r"^[0-9a-f]{16}$")+ for call in (lambda: self.carol.channel_join(chat=chat), lambda: self.carol.read(chat=chat),+ lambda: self.carol.send(chat=chat, text="hi"), lambda: self.carol.read(chat="#merger-talks"),+ lambda: self.carol.channel_join(chat="merger-talks")):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, 404)+ self.assertNotIn("merger" if "merger" not in str(call.__code__.co_consts) else chat, str(e.exception))+ self.refused(lambda: self.carol.channel_create(name="Merger-Talks"), 400, "The name #Merger-Talks is taken.")+ self.refused(lambda: self.carol.channel_create(name="Merger-Talks"), 400, "Pick another name.")+ with self.assertRaises(ClientError) as e:+ self.carol.channel_create(name="merger-talks")+ self.assertNotIn(chat, str(e.exception))+ first = self.alice.send(chat=chat, text="numbers")["message"]["id"]+ self.assertEqual(self.bob.read(id=first)["messages"][0]["text"], "numbers") # longer ids work everywhere+ self.bob.send(chat=chat, text="reply", reply_to=first)+ self.refused(lambda: self.carol.read(id=first), 404, "No message")++ def test_filters_reach_only_plain_data(self):+ m = {"id": "x:1", "text": "build 412 failing in CI", "from": "maya", "reactions": {"+1": ["a"]}}+ self.assertTrue(matcher("any(w in m.text for w in ['failing'])")(m)) # m reaches generator expressions too+ self.assertTrue(matcher("re.search(r'build \\d+', m.text) and m.sender == 'maya'")(m))+ self.assertEqual([x["id"] for x in sorter("-len((m.reactions or {}).get('+1', []))")([m])], ["x:1"])+ for expr in ("open('/etc/passwd').read()", "__import__('os')", "m.__class__", "(x for x in []).gi_frame",+ "'{0.__class__}'.format(m)", "getattr(m, 'text')", "type(m)", "re.compile('a')"):+ with self.assertRaises(ClientError, msg=expr):+ matcher(expr)(m)++ def test_a_posting_rule_by_name_keeps_the_channel_name(self):+ chat = self.alice.channel_create(name="records", members="bob")["chat"]["id"]+ self.server.admin.policy_set(rules={"posters": {"#records": ["alice"]}})+ self.refused(lambda: self.bob.send(chat="#records", text="forged"), 403, "Only alice may post")+ self.refused(lambda: self.bob.channel_rename(chat="#records", name="open"), 403, "keeps its name")+ self.alice.channel_rename(chat="#records", name="Records") # the same name, so the rule holds+ self.refused(lambda: self.bob.send(chat="#Records", text="forged"), 403, "Only alice may post")+ self.server.admin.policy_set(rules={"posters": {chat: ["alice"]}}) # by id, it may be renamed+ self.bob.channel_rename(chat="#Records", name="results")+ self.refused(lambda: self.bob.send(chat="#results", text="forged"), 403, "Only alice may post")++ def test_a_search_or_filter_marks_read_nothing_it_left_out(self):+ self.alice.send(chat="bob", text="important instruction")+ self.alice.send(chat="bob", text="needle")+ self.alice.send(chat="bob", text="after")+ self.assertEqual(self.texts(self.bob.read(search="needle")), ["needle"])+ self.assertEqual(self.texts(self.bob.read(where="m.text == 'after'")), ["after"])+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 3) # the first was never shown, so nothing moved+ self.assertEqual(self.texts(self.bob.read(limit=1)), ["important instruction"])+ self.assertEqual(self.texts(self.bob.read(search="after")), ["after"])+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 2) # needle comes first, and was not shown this time+ self.assertEqual(self.texts(self.bob.read()), ["needle", "after"])+ self.assertEqual(self.bob.chats()["chats"][0]["unread"], 0)++ def test_members_may_read_only_what_was_sent_while_they_were_in(self):+ self.refused(lambda: self.server.admin.policy_set(rules={"history": "some"}), 400, "while_member")+ self.server.admin.policy_set(rules={"history": "while_member"})+ chat = self.alice.channel_create(name="ops", members="bob")["chat"]["id"]+ before = self.alice.send(chat="#ops", text="before carol")["message"]["id"]+ self.alice.channel_add(chat="#ops", users="carol")+ reply = self.bob.send(chat="#ops", text="with carol", reply_to=before)["message"]["id"]+ self.alice.channel_remove(chat="#ops", users="carol")+ work = tempfile.mkdtemp(prefix="swarmchat-history-")+ try:+ with open(os.path.join(work, "plan.txt"), "w") as f:+ f.write("plan")+ out = self.alice.send(chat="#ops", text="carol is out", attach=[os.path.join(work, "plan.txt")])["message"]+ self.alice.channel_add(chat="#ops", users="carol")+ summary = self.carol.chats(name="ops")["chats"][0]+ self.assertEqual((summary["last"]["text"], summary["unread"]), ("with carol", 1))+ self.assertEqual(self.carol.chat_info(chat="#ops")["chat"]["messages"], 1)+ self.assertEqual(self.texts(self.carol.read(chat="#ops")), ["with carol"])+ seen = self.carol.read(chat="#ops", all=True)["messages"][0]+ self.assertEqual(("reply_to" in seen, "reply_to_from" in seen), (False, False)) # it answers a hidden one+ self.alice.send(chat="#ops", text="carol is back")+ self.assertEqual(self.texts(self.carol.read(chat="#ops", all=True)), ["with carol", "carol is back"])+ self.assertEqual(self.texts(self.carol.read(search="carol is out")), [])+ self.assertEqual(self.texts(self.carol.read(thread=reply)), ["with carol"]) # without its start+ self.assertEqual(self.carol.read(thread=reply)["messages"][0]["depth"], 0) # nor its level+ for call in (lambda: self.carol.read(id=out["id"]), lambda: self.carol.read(id=before),+ lambda: self.carol.react(id=out["id"], emoji="+1"),+ lambda: self.carol.send(chat="#ops", text="re", reply_to=out["id"]),+ lambda: self.carol.download(out["attachments"][0]["id"], to=os.path.join(work, "in"))):+ self.refused(call, 404, "No ")+ self.alice.send(chat="#ops", text="ping")+ self.assertEqual(self.texts(self.carol.wait(timeout=1)), ["ping"])+ self.assertEqual(len(self.bob.read(chat="#ops", all=True)["messages"]), 5) # bob was in all along+ self.alice.channel_add(chat="#ops", users="carol") # an edit adding a mention to a hidden message+ self.alice.edit(id=before, text="before carol, @carol")+ self.assertFalse(self.carol.chats(name="ops")["chats"][0]["mentioned"])+ self.alice.channel_create(name="town", public=True)+ self.alice.send(chat="#town", text="before anyone joined")+ counts = lambda who: {c["name"]: c["messages"] for c in who.channel_list()["channels"]} # noqa: E731+ self.assertEqual(counts(self.carol), {"town": None}) # joining would show none of them+ self.carol.channel_join(chat="#town")+ self.alice.send(chat="#town", text="after")+ self.assertEqual((counts(self.carol), counts(self.alice)), ({"town": 1}, {"town": 2}))+ self.server.admin.policy_set(rules={}) # by default, members read everything, as in Slack+ self.assertEqual(len(self.carol.read(chat="#ops", all=True)["messages"]), 5)+ finally:+ shutil.rmtree(work)+++class Pages(unittest.TestCase):+ """A read gives at most max_page messages. The client asks for the next pages while it needs them."""++ def setUp(self):+ self.data = tempfile.mkdtemp(prefix="swarmchat-test-")+ self.server = Server(self.data, max_page=5)+ self.a, self.b = self.server.user("a", "A"), self.server.user("b", "B")+ self.ids = [self.a.send(chat="b", text=f"m{i:02}")["message"]["id"] for i in range(12)]++ def tearDown(self):+ self.server.stop()+ shutil.rmtree(self.data)++ def texts(self, result):+ return [m["text"] for m in result["messages"]]++ def test_one_reply_holds_a_page_and_a_cursor_to_the_next(self):+ first = self.b.call("read", {"peek": True, "limit": 100})+ self.assertEqual((self.texts(first), first["more"]), ([f"m{i:02}" for i in range(5)], True))+ second = self.b.call("read", {"peek": True, "limit": 100, "cursor": first["cursor"]})+ self.assertEqual(self.texts(second), [f"m{i:02}" for i in range(5, 10)])+ back = self.b.call("read", {"all": True, "peek": True, "limit": 100})+ self.assertEqual(self.texts(back), [f"m{i:02}" for i in range(7, 12)]) # history goes back from the newest+ back = self.b.call("read", {"all": True, "peek": True, "limit": 100, "cursor": back["cursor"]})+ self.assertEqual(self.texts(back), [f"m{i:02}" for i in range(2, 7)])+ self.assertEqual(self.b.call("read", {"peek": True})["unread_left"], 12)++ def test_the_client_puts_pages_together(self):+ every = [f"m{i:02}" for i in range(12)]+ self.assertEqual(self.texts(self.b.read(peek=True, limit=12)), every)+ history = lambda **select: self.texts(self.b.read(peek=True, **select)) # noqa: E731+ self.assertEqual(history(chat="a", all=True, limit=100), every)+ self.assertEqual(history(last=7), every[5:])+ self.assertEqual(history(all=True, where="m.text.endswith('1')"), ["m01", "m11"])+ self.assertEqual(history(all=True, sort="m.text", limit=3), every[:3])+ self.assertEqual(history(all=True, sort="-int(m.text[1:])", limit=2), ["m11", "m10"])+ self.assertEqual(history(chat="a", since=self.ids[2], limit=100), every[3:])+ self.assertEqual(history(chat="a", before=self.ids[9], search="m0", limit=100), every[:9])+ self.assertEqual(self.texts(self.b.read(peek=True, where="m.text > 'm06'")), every[7:]) # every unread one+ top = self.ids[0]+ for i in range(8):+ top = self.a.send(chat="b", text=f"reply {i}", reply_to=top)["message"]["id"]+ thread = self.b.read(thread=self.ids[0])["messages"]+ self.assertEqual(([m["text"] for m in thread], [m["depth"] for m in thread]),+ (["m00"] + [f"reply {i}" for i in range(8)], list(range(9))))+ r = self.b.read() # no limit asked for: one page, and the rest stay unread+ self.assertEqual((len(r["messages"]), r["unread_left"]), (5, 15))+++class Settings(unittest.TestCase):+ def test_file_then_flags(self):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ path = os.path.join(data, "settings.json")+ with open(path, "w") as f:+ json.dump({"data": data, "port": 9000, "max_text": 50, "default_notify": "mentions"}, f)+ s = config.load(path, port=9100, max_wait=None)+ self.assertEqual((s["port"], s["max_text"], s["max_wait"], s["default_notify"]), (9100, 50, 300, "mentions"))+ with open(path, "w") as f:+ json.dump({"data": data, "max_txt": 5}, f)+ with self.assertRaises(SystemExit) as e:+ config.load(path)+ self.assertIn("unknown settings max_txt", str(e.exception))+ for bad in ({"max_wait": 0}, {"default_notify": "loud"}, {"segment": "big"}):+ with self.assertRaises(SystemExit):+ config.load(None, data=data, **bad)+ shutil.rmtree(data)++ def test_limits_apply(self):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ server = Server(data, max_text=10, max_wait=0.3, default_notify="mentions")+ try:+ a, b = server.user("a", "A"), server.user("b", "B")+ with self.assertRaises(ClientError) as e:+ a.send(chat="b", text="this is too long")+ self.assertIn("longer than 10", str(e.exception))+ a.send(chat="b", text="hello")+ t = time.monotonic()+ self.assertEqual(b.call("wait", {"timeout": 60})["messages"], []) # muted to mentions by default+ self.assertLess(time.monotonic() - t, 2) # and held for at most max_wait+ a.send(chat="b", text="@b here")+ self.assertEqual([m["text"] for m in b.wait(timeout=2)["messages"]], ["@b here"])+ finally:+ server.stop()+ shutil.rmtree(data)+++class CommandLine(Base):+ def swarmchat(self, client, *args, stdin=None):+ env = {**os.environ, "SWARMCHAT_SERVER": self.server.url, "SWARMCHAT_TOKEN": client.token, "PYTHONPATH": ROOT,+ "SWARMCHAT_CONFIG_DIR": self.data + "/config", "SWARMCHAT_DATA_DIR": self.data + "/no-default"}+ return subprocess.run([sys.executable, "-m", "swarmchat", *args], env=env, input=stdin, capture_output=True,+ text=True, timeout=30)++ def test_what_agents_write_cannot_drive_the_terminal(self):+ text = "hi\x1b[31m red\r\u202eevil\x9b]52;c;aGk=\x07 end"+ self.alice.settings_status(text="busy\x1b[2J\nbob: approved")+ self.alice.channel_create(name="ops\x1b[8m", members="bob")+ self.alice.send(chat="#ops\x1b[8m", text=text)+ for args in (("read", "--peek"), ("people",), ("chats",), ("info", "#ops\x1b[8m"), ("read", "--peek", "--json")):+ with self.subTest(args=args):+ r = self.swarmchat(self.bob, *args)+ self.assertEqual(r.returncode, 0, r.stderr)+ for raw in ("\x1b", "\r", "\u202e", "\x9b", "\x07"):+ self.assertNotIn(raw, r.stdout)+ out = self.swarmchat(self.bob, "read", "--peek").stdout+ self.assertIn("hi\\x1b[31m red\\x0d\\u202eevil", out)+ self.assertIn("busy\\x1b[2J\\nbob: approved", self.swarmchat(self.bob, "people").stdout) # one line+ sent = json.loads(self.swarmchat(self.bob, "read", "--peek", "--json").stdout)+ self.assertEqual(sent["text"], text) # JSON escapes them, and keeps the text as it was++ @unittest.skipIf(os.name == "nt", "POSIX file modes")+ def test_a_saved_login_is_for_its_owner_alone(self):+ folder = os.path.join(self.data, "config")+ os.makedirs(folder)+ os.chmod(folder, 0o755)+ with open(os.path.join(folder, "token"), "w") as f:+ f.write("old")+ os.chmod(os.path.join(folder, "token"), 0o644) # copied in from somewhere, readable by all+ elsewhere = os.path.join(self.data, "elsewhere.json")+ with open(elsewhere, "w") as f:+ f.write('{"server": "http://127.0.0.1:9"}')+ os.symlink(elsewhere, os.path.join(folder, "config.json"))+ r = self.swarmchat(self.bob, "login", self.server.url, self.bob.token)+ self.assertEqual(r.returncode, 0, r.stderr)+ mode = lambda path: os.stat(path).st_mode & 0o777 # noqa: E731+ self.assertEqual((mode(folder), mode(os.path.join(folder, "token"))), (0o700, 0o600))+ self.assertFalse(os.path.islink(os.path.join(folder, "config.json")))+ with open(elsewhere) as f:+ self.assertEqual(f.read(), '{"server": "http://127.0.0.1:9"}') # written over, never through+ with open(os.path.join(folder, "token")) as f:+ self.assertEqual(f.read().strip(), self.bob.token)++ def test_admin_finds_the_server_from_its_folder(self):+ env = {**os.environ, "PYTHONPATH": ROOT, "SWARMCHAT_SERVER": "http://127.0.0.1:9",+ "SWARMCHAT_CONFIG_DIR": self.data + "/config"}+ cmd = [sys.executable, "-m", "swarmchat", "admin", "--data", self.data, "add", "dora", "--name", "Dora"]+ r = subprocess.run(cmd, env=env, capture_output=True, text=True, timeout=30)+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("Added dora", r.stdout)++ def test_admin_and_view_find_the_default_folder(self):+ env = {k: v for k, v in os.environ.items() if not k.startswith("SWARMCHAT_")}+ env.update(PYTHONPATH=ROOT, SWARMCHAT_CONFIG_DIR=self.data + "/config", SWARMCHAT_DATA_DIR=self.data)+ run = lambda *args: subprocess.run([sys.executable, "-m", "swarmchat", *args], env=env, # noqa: E731+ capture_output=True, text=True, timeout=30)+ r = run("admin", "add", "dora", "--name", "Dora")+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("Added dora", r.stdout)+ r = run("view", "--no-open")+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn(f"Viewer: {self.server.url.rstrip('/')}/view#token=", r.stdout)+ env["SWARMCHAT_DATA_DIR"] = self.data + "/missing"+ r = run("view", "--no-open")+ self.assertEqual(r.returncode, 1)+ self.assertIn("No server folder at", r.stderr)++ def test_an_administrator_runs_role_and_channel_commands(self):+ self.server.admin.policy_set(rules={"admins": ["alice"]})+ r = self.swarmchat(self.alice, "admin", "add", "dora")+ self.assertEqual(r.returncode, 3, r.stderr)+ self.assertIn("Only the super admin may add users.", r.stderr)+ r = self.swarmchat(self.alice, "admin", "role", "channel_creator", "--add", "carol")+ self.assertEqual((r.returncode, r.stdout), (0, "channel_creator: carol\n"), r.stderr)+ self.assertEqual(self.swarmchat(self.alice, "admin", "role", "admin").stdout, "admin: alice\n")+ r = self.swarmchat(self.alice, "admin", "role", "channel_creator", "--everyone")+ self.assertEqual(r.stdout, "channel_creator: everyone\n", r.stderr)+ self.alice.channel_create(name="ops", members="bob")+ r = self.swarmchat(self.bob, "channel", "archive", "ops")+ self.assertEqual(r.returncode, 3, r.stderr)+ r = self.swarmchat(self.alice, "channel", "archive", "ops")+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("#ops [archived]", r.stdout)+ self.assertIn("[archived]", self.swarmchat(self.bob, "chats").stdout)+ self.assertEqual(self.swarmchat(self.alice, "channel", "unarchive", "ops").returncode, 0)+ r = self.swarmchat(self.alice, "channel", "managers", "ops", "--add", "bob")+ self.assertEqual(r.stdout, "Managers of #ops: alice, bob\n", r.stderr)+ self.assertIn("\nManagers: alice, bob\n", self.swarmchat(self.bob, "info", "#ops").stdout)++ def test_a_new_token_replaces_the_saved_login(self):+ r = self.swarmchat(self.bob, "login", self.server.url, self.bob.token)+ self.assertEqual(r.returncode, 0, r.stderr)+ r = self.swarmchat(self.bob, "settings", "token")+ self.assertEqual(r.stdout, "Your old token no longer works. The new one is saved in your login.\n", r.stderr)+ env = {**os.environ, "PYTHONPATH": ROOT, "SWARMCHAT_CONFIG_DIR": self.data + "/config"}+ env.pop("SWARMCHAT_TOKEN", None)+ env.pop("SWARMCHAT_SERVER", None)+ r = subprocess.run([sys.executable, "-m", "swarmchat", "whoami"], env=env, capture_output=True, text=True,+ timeout=30)+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("bob", r.stdout)+ self.assertEqual(self.swarmchat(self.bob, "whoami").returncode, 3) # the old token, from the environment++ def test_channel_descriptions(self):+ r = self.swarmchat(self.alice, "channel", "create", "release-2-4", "--public", "--description",+ "Release 2.4, shipping Friday")+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("Release 2.4, shipping Friday", self.swarmchat(self.bob, "channel", "list").stdout)+ r = self.swarmchat(self.alice, "channel", "describe", "release-2-4", "Freeze moved to Monday")+ self.assertEqual(r.returncode, 0, r.stderr)+ self.assertIn("\nFreeze moved to Monday\n", self.swarmchat(self.alice, "info", "#release-2-4").stdout)+ self.assertEqual(self.swarmchat(self.alice, "channel", "describe", "release-2-4", "").returncode, 0)+ self.assertNotIn("Freeze", self.swarmchat(self.alice, "info", "#release-2-4").stdout)++ def test_the_viewer_and_the_help_say_little_without_a_token(self):+ import urllib.request+ from swarmchat import view+ with urllib.request.urlopen(self.server.url + "/view") as r:+ page, policy = r.read().decode(), r.headers["Content-Security-Policy"]+ for word in ('id="app"', "What had", "Board", "agent"):+ self.assertNotIn(word, page) # a form for the token, and nothing of the viewer+ self.assertIn(view.page()["script"], policy) # the viewer's own script may run, once fetched+ with self.assertRaises(ClientError) as e:+ self.alice.view_page()+ self.assertEqual(e.exception.status, 401)+ self.assertIn('id="app"', self.server.admin.view_page()["html"])+ self.assertNotIn("agent", self.swarmchat(self.alice, "--help").stdout.lower())++ def test_no_server_is_reported_plainly(self):+ with self.assertRaises(ClientError) as e:+ Client("http://127.0.0.1:9", "x").whoami()+ self.assertIn("Is the server running?", str(e.exception))+ self.assertNotIn("may have been applied", str(e.exception))++ def test_no_token_looks_like_a_flag(self):+ # One in 64 tokens used to start with '-', and then `swarmchat login URL TOKEN` and --token TOKEN refused it.+ tokens = [self.server.admin.user_add(id=f"t{i}")["token"] for i in range(200)]+ tokens += [self.server.admin.token_rotate(id="t0")["token"] for _ in range(50)]+ with open(os.path.join(self.data, "admin.token")) as f:+ tokens.append(f.read().strip())+ self.assertEqual([t for t in tokens if t.startswith("-")], [])+ r = self.swarmchat(self.bob, "login", self.server.url, tokens[-2])+ self.assertEqual(r.returncode, 0, r.stderr)++ def test_send_read_and_exit_codes(self):+ r = self.swarmchat(self.alice, "send", "bob,carol", "-", stdin="line one\nline two")+ self.assertEqual(r.returncode, 0, r.stderr)+ out = self.swarmchat(self.bob, "read").stdout+ self.assertIn("Alice Ames: line one\n line two", out)+ self.assertEqual(self.swarmchat(self.bob, "send", "nobody", "x").returncode, 2)+ self.assertEqual(self.swarmchat(self.bob, "wait", "--timeout", "0.2").returncode, 4)+ lines = self.swarmchat(self.carol, "read", "--json").stdout.splitlines()+ self.assertEqual(json.loads(lines[0])["text"], "line one\nline two")+++if __name__ == "__main__":+ unittest.main()diff --git a/tests/test_transport.py b/tests/test_transport.pynew file mode 100644index 0000000..65e84a4--- /dev/null+++ b/tests/test_transport.py@@ -0,0 +1,316 @@+"""HTTP framing, TLS isolation, finite waits, and ambiguous transport failures."""+import http.client+import json+import math+import shutil+import socket+import ssl+import subprocess+import tempfile+import threading+import time+import unittest+from pathlib import Path+from unittest.mock import patch++from test_swarmchat import Base, Server+from swarmchat import api, config+from swarmchat.client import ClientError+from swarmchat.server import Handler, make_server+++class TransportSafety(Base):+ def test_ambiguous_body_framing_is_rejected_and_connection_closed(self):+ for framing, body in (("Content-Length: 2\r\nContent-Length: 2\r\n", b"{}"),+ ("Transfer-Encoding: chunked\r\n", b"0\r\n\r\n"),+ ("Content-Length: 8\r\n", b"{}")):+ with self.subTest(framing=framing):+ with socket.create_connection(self.server.httpd.server_address, timeout=2) as sock:+ sock.sendall(("POST /v1/whoami HTTP/1.1\r\nHost: localhost\r\nAuthorization: Bearer "+ f"{self.alice.token}\r\n" + framing + "\r\n").encode() + body)+ sock.shutdown(socket.SHUT_WR)+ response = http.client.HTTPResponse(sock)+ response.begin()+ self.assertEqual(response.status, 400)+ self.assertEqual(response.getheader("Connection"), "close")+ response.read()++ def test_nonfinite_time_settings_and_client_waits_are_rejected(self):+ for value in (math.nan, math.inf, -math.inf):+ for setting in ("max_wait", "save_every", "request_timeout"):+ with self.subTest(value=value, setting=setting), self.assertRaises(SystemExit):+ config.load(data=self.data, **{setting: value})+ with self.assertRaises(ClientError):+ self.bob.wait(timeout=value)++ def test_send_is_not_duplicated_when_response_connection_is_lost(self):+ self.alice.whoami() # Reuse a healthy HTTP connection, as normal clients do.+ original_reply = Handler.reply+ dropped = threading.Event()++ def drop_one_send_response(handler, status, body):+ if handler.path == "/v1/send" and status == 200 and not dropped.is_set():+ dropped.set() # The message was already committed by api.send.+ handler.close_connection = True+ handler.connection.shutdown(socket.SHUT_RDWR)+ handler.connection.close()+ return+ return original_reply(handler, status, body)++ with patch.object(Handler, "reply", drop_one_send_response):+ try:+ self.alice.send(chat="bob", text="perform this action once")+ except ClientError:+ pass # An ambiguous transport failure must not replay a mutation.+ self.assertTrue(dropped.is_set())+ self.assertEqual(self.texts(self.bob.read()), ["perform this action once"])+++ def test_nonfinite_wait_timeout_is_rejected_without_busy_loop(self):+ pending_calls = 0+ original_pending = api._pending++ def bounded_pending(*args, **kwargs):+ nonlocal pending_calls+ pending_calls += 1+ if pending_calls > 20:+ # Terminate the defective request in the test; production has no such guard.+ raise api.Error(400, "Test guard stopped a non-terminating wait")+ return original_pending(*args, **kwargs)++ with patch.object(api, "_pending", side_effect=bounded_pending):+ with self.assertRaises(ClientError) as caught:+ self.bob.call("wait", {"timeout": "nan"}, http_timeout=2)+ self.assertEqual(caught.exception.status, 400)+ self.assertLessEqual(pending_calls, 1, "NaN timeout repeatedly polled instead of rejecting or blocking")++ def test_negative_content_length_cannot_bypass_max_body(self):+ self.server.store.settings["max_body"] = 128+ body = json.dumps({"ignored": "x" * 1024}).encode()+ headers = (+ "POST /v1/whoami HTTP/1.1\r\nHost: localhost\r\n"+ f"Authorization: Bearer {self.alice.token}\r\n"+ "Content-Length: -1\r\nConnection: close\r\n\r\n"+ ).encode()+ with socket.create_connection(self.server.httpd.server_address, timeout=2) as sock:+ sock.sendall(headers + body)+ sock.shutdown(socket.SHUT_WR)+ response = http.client.HTTPResponse(sock)+ response.begin()+ status = response.status+ response.read()+ self.assertIn(status, (400, 413), "Server accepted 1 KiB despite a 128-byte request limit")++ def test_oversized_request_does_not_poison_next_client_call(self):+ self.server.store.settings["max_body"] = 256+ with self.assertRaises(ClientError) as caught:+ self.alice.send(chat="bob", text="x" * 1024)+ self.assertEqual(caught.exception.status, 413)+ try:+ result = self.alice.whoami()+ except Exception as exc:+ self.fail(f"A valid request after HTTP 413 failed with {type(exc).__name__}: {exc}")+ self.assertEqual(result["user"]["id"], "alice")++ def test_a_refused_body_larger_than_the_socket_buffers_still_gets_its_answer(self):+ # The server answers before reading the body. Closing with the body unread would reset the connection and the+ # client, still sending, would never see the answer.+ self.server.store.settings["max_body"] = 256+ for _ in range(3):+ with self.assertRaises(ClientError) as caught:+ self.alice.send(chat="bob", text="x" * 3_000_000)+ self.assertEqual(caught.exception.status, 413)+ self.assertEqual(self.alice.whoami()["user"]["id"], "alice")++++class RequestLimits(unittest.TestCase):+ """What a client can hold: without a known token, no body is read. A request that arrives too slowly is cut off,+ and no more than max_connections are open at once."""++ def start(self, **settings):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ self.addCleanup(shutil.rmtree, data)+ server = Server(data, **settings)+ self.addCleanup(server.stop)+ return server++ def connect(self, server, sent: bytes = b"") -> socket.socket:+ sock = socket.create_connection(server.httpd.server_address, timeout=5)+ self.addCleanup(sock.close)+ sock.sendall(sent)+ return sock++ def closed_after(self, sock, trickle: bytes) -> float:+ """Seconds until the server closes the connection, while one byte of `trickle` goes out every 0.1 seconds:+ never silent long enough for the idle timeout."""+ start, stop = time.monotonic(), threading.Event()++ def drip():+ for byte in trickle:+ if stop.wait(0.1):+ return+ try:+ sock.send(bytes([byte]))+ except OSError:+ return+ threading.Thread(target=drip, daemon=True).start()+ try:+ while sock.recv(4096):+ pass+ except OSError: # reset rather than closed: the same for the client+ pass+ stop.set()+ return time.monotonic() - start++ def test_an_unknown_token_is_refused_before_its_body_is_read(self):+ server = self.start()+ for name, status in (("whoami", 401), ("send_files", 401), ("user_add", 401), ("view_info", 401),+ ("no_such_request", 404)):+ with self.subTest(name=name):+ sock = self.connect(server, f"POST /v1/{name} HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer nope\r\n"+ "Content-Length: 100000000\r\n\r\n".encode())+ response = http.client.HTTPResponse(sock)+ response.begin() # answered at once: the server did not wait for 100 MB that never come+ self.assertEqual((response.status, response.getheader("Connection")), (status, "close"))++ def test_headers_that_arrive_too_slowly_are_cut_off(self):+ server = self.start(request_timeout=1)+ sock = self.connect(server, b"POST /v1/whoami HTTP/1.1\r\n")+ self.assertLess(self.closed_after(sock, b"X" * 100), 3)+ self.assertEqual(server.admin.policy_show()["rules"], {}) # everyone else is served as before++ def test_a_body_that_arrives_too_slowly_is_cut_off(self):+ server = self.start(request_timeout=1, min_rate=10_000)+ token = server.user("a", "A").token+ sock = self.connect(server, f"POST /v1/whoami HTTP/1.1\r\nHost: x\r\nAuthorization: Bearer {token}\r\n"+ "Content-Length: 1000\r\n\r\n".encode())+ self.assertLess(self.closed_after(sock, b" " * 1000), 3) # 1 second, and 0.1 for 1,000 bytes at 10,000 a second++ def test_connections_that_send_nothing_cannot_keep_others_out(self):+ server = self.start(max_connections=4, request_timeout=30)+ silent = [self.connect(server, b"POST /v1/whoami HTTP/1.1\r\nHost: x\r\n") for _ in range(50)]+ time.sleep(0.3) # every slot taken, by connections that never finish their headers+ start = time.monotonic()+ self.assertEqual(server.admin.policy_show()["rules"], {}) # the longest waiting one made room+ self.assertLess(time.monotonic() - start, 3)++ def closed(sock):+ sock.settimeout(0.2)+ try:+ return sock.recv(1) == b""+ except socket.timeout:+ return False+ self.assertEqual([closed(sock) for sock in silent], [True] * 47 + [False] * 3) # the oldest made room++ def test_connections_with_requests_under_way_are_never_closed_to_make_room(self):+ server = self.start(max_connections=2)+ a, b = server.user("a", "A"), server.user("b", "B")+ got = []+ holders = [threading.Thread(target=lambda c=c: got.append(c.call("wait", {"timeout": 2}, http_timeout=10)))+ for c in (a, b)]+ for t in holders:+ t.start()+ time.sleep(0.5)+ turned = self.connect(server)+ reply = turned.recv(4096)+ self.assertTrue(reply.startswith(b"HTTP/1.1 503 "), reply)+ self.assertIn(b"2 connections are open", reply)+ for t in holders:+ t.join()+ self.assertEqual(len(got), 2) # both waits ran to their end++ def test_one_user_has_at_most_max_requests_under_way(self):+ server = self.start(max_requests=1)+ a = server.user("a", "A")+ waiting = threading.Thread(target=lambda: a.call("wait", {"timeout": 2}, http_timeout=10))+ waiting.start()+ time.sleep(0.5)+ for name in ("wait", "whoami"):+ with self.subTest(name=name), self.assertRaises(ClientError) as e:+ server.client(a.token).call(name, {"timeout": 1})+ self.assertEqual(e.exception.status, 429)+ self.assertEqual(server.user("b", "B").whoami()["user"]["id"], "b") # others are not held back+ waiting.join()+ self.assertEqual(server.client(a.token).call("wait", {"timeout": 0.01})["messages"], [])++ def test_headers_have_a_size_limit(self):+ server = self.start()+ sock = self.connect(server)+ try:+ sock.sendall(b"GET /v1/health HTTP/1.1\r\nHost: x\r\n" + b"".join(+ f"X-Filler-{i}: {'a' * 4000}\r\n".encode() for i in range(90)) + b"\r\n")+ reply = sock.recv(4096)+ except (ConnectionResetError, BrokenPipeError):+ reply = b""+ self.assertEqual(reply, b"") # 360 KB of headers: closed, never parsed+ self.assertEqual(server.admin.policy_show()["rules"], {})++ def test_no_header_value_can_hold_a_line_break(self):+ handler = Handler.__new__(Handler)+ handler.request_version = "HTTP/1.1"+ for value in ("text/plain\r\nX-Injected: 1", "a\nb", "a\x00b"):+ with self.assertRaises(ValueError):+ handler.send_header("Content-Type", value)+ handler.send_header("Content-Type", "text/plain")+ self.assertEqual(handler._headers_buffer, [b"Content-Type: text/plain\r\n"])+++class TLSRegression(unittest.TestCase):+ @unittest.skipUnless(shutil.which("openssl"), "openssl is needed for a temporary test certificate")+ def test_one_idle_tls_connection_does_not_block_other_clients(self):+ with tempfile.TemporaryDirectory(prefix="swarmchat-review-tls-") as temporary:+ cert, key = Path(temporary) / "cert.pem", Path(temporary) / "key.pem"+ subprocess.run([shutil.which("openssl"), "req", "-x509", "-newkey", "rsa:2048", "-nodes",+ "-keyout", str(key), "-out", str(cert), "-days", "1", "-subj", "/CN=localhost"],+ check=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, timeout=10)+ httpd, store = make_server(config.load(data=str(Path(temporary) / "data"), port=0,+ tls_cert=str(cert), tls_key=str(key)))+ thread = threading.Thread(target=httpd.serve_forever, kwargs={"poll_interval": 0.02}, daemon=True)+ thread.start()+ idle = None+ try:+ def healthy_request():+ client = http.client.HTTPSConnection(*httpd.server_address, timeout=0.4,+ context=ssl._create_unverified_context())+ try:+ client.request("GET", "/v1/health", headers={"Connection": "close"})+ response = client.getresponse()+ response.read()+ return response.status+ finally:+ client.close()++ self.assertEqual(healthy_request(), 200)+ idle = socket.create_connection(httpd.server_address, timeout=1)+ time.sleep(0.05)+ try:+ status = healthy_request()+ except (OSError, http.client.HTTPException) as exc:+ idle.close()+ idle = None+ self.assertEqual(healthy_request(), 200, "Server failed to recover after idle peer closed")+ self.fail(f"A single idle unauthenticated socket blocked HTTPS: {type(exc).__name__}: {exc}")+ self.assertEqual(status, 200)+ finally:+ if idle:+ idle.close()+ httpd.shutdown()+ httpd.server_close()+ thread.join(timeout=2)+ store.close()+++class IdleConnections(unittest.TestCase):+ def test_send_after_the_server_closed_an_idle_connection(self):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ server = Server(data, request_timeout=0.5)+ try:+ a, b = server.user("a", "A"), server.user("b", "B")+ a.send(chat="b", text="before")+ time.sleep(1.2) # the server has closed a's idle kept-open connection by now+ a.send(chat="b", text="after idle") # goes out on a fresh connection, once+ self.assertEqual([m["text"] for m in b.read()["messages"]], ["before", "after idle"])+ finally:+ server.stop()+ shutil.rmtree(data)diff --git a/tests/test_view.py b/tests/test_view.pynew file mode 100644index 0000000..f02650e--- /dev/null+++ b/tests/test_view.py@@ -0,0 +1,602 @@+"""The viewer's requests (`swarmchat view`): they see every chat, follow everything live, and change nothing."""+import gzip+import hashlib+import http.client+import json+import os+import shutil+import struct+import subprocess+import sys+import tempfile+import threading+import time+import unittest+import zlib+from unittest import mock++from test_swarmchat import ROOT, Base, Server+from swarmchat import config, view+from swarmchat.client import Client, ClientError+from swarmchat.server import make_server+++def folder_state(root):+ """Every file under root with its size and modification time, to show that nothing changed."""+ state = {}+ for path, dirs, files in os.walk(root):+ for name in dirs + files:+ full = os.path.join(path, name)+ st = os.stat(full)+ state[os.path.relpath(full, root)] = (st.st_size, st.st_mtime_ns)+ return state+++class ViewBase(Base):+ def setUp(self):+ super().setUp()+ with open(os.path.join(self.data, "view.token")) as f:+ self.view = self.server.client(f.read().strip())++ def ids(self, messages):+ return [m["id"] for m in messages]+++class Access(ViewBase):+ def test_the_view_token_only_looks_and_agents_cannot_look_at_everything(self):+ self.alice.send(chat="bob", text="private to bob")+ self.assertEqual(len(self.view.view_snapshot()["chats"]), 1)+ self.assertEqual(len(self.server.admin.view_snapshot()["chats"]), 1) # the admin token also works+ with self.assertRaises(ClientError) as e:+ self.carol.view_snapshot()+ self.assertEqual(e.exception.status, 401)+ for call in (lambda: self.view.send(chat="bob", text="x"), lambda: self.view.user_add(id="eve"),+ lambda: self.view.chats()):+ with self.assertRaises(ClientError) as e:+ call()+ self.assertEqual(e.exception.status, 401)+ mode = os.stat(os.path.join(self.data, "view.token")).st_mode & 0o777+ if os.name != "nt":+ self.assertEqual(mode, 0o600)++ def test_looking_moves_no_read_position_and_writes_nothing(self):+ sent = self.alice.send(chat="bob", text="hello @bob")["message"]+ self.alice.channel_create(name="team", members="bob,carol")+ reply = self.bob.send(chat="#team", text="hi all", reply_to=None)["message"]+ self.carol.send(chat="#team", text="answer", reply_to=reply["id"])+ self.server.store.flush()+ before = folder_state(self.data)+ v = self.view+ v.view_snapshot()+ v.view_messages(chat=sent["chat"])+ v.view_thread(id=reply["id"])+ v.view_receipts(id=sent["id"])+ v.view_posts()+ v.view_search(text="hello")+ v.view_activity(buckets=10)+ v.view_as(user="bob", unread=True)+ v.view_knew(id=reply["id"])+ v.view_feed(after=0, timeout=0.01)+ self.server.store.flush()+ self.assertEqual(folder_state(self.data), before)+ self.assertEqual(self.bob.chats()["chats"][-1]["unread"], 1) # still unread for bob+++class Snapshot(ViewBase):+ def test_every_user_and_chat_with_counts(self):+ self.alice.send(chat="bob", text="one")+ self.bob.send(chat="alice", text="two")+ self.alice.send(chat="bob,carol", text="three")+ self.alice.channel_create(name="news", members="bob", public=True)+ snap = self.view.view_snapshot()+ self.assertEqual([u["id"] for u in snap["users"]], ["alice", "bob", "carol"])+ kinds = sorted(c["kind"] for c in snap["chats"])+ self.assertEqual(kinds, ["channel", "direct", "group"])+ direct = next(c for c in snap["chats"] if c["kind"] == "direct")+ self.assertEqual((direct["messages"], direct["sent"], direct["last"]["text"]), (2, {"alice": 1, "bob": 1}, "two"))+ self.assertTrue(next(c for c in snap["chats"] if c["kind"] == "channel")["public"])+ self.assertGreater(snap["feed"], 0)++ def test_removed_people_still_have_names(self):+ self.alice.send(chat="carol", text="bye")+ self.server.admin.user_remove(id="carol")+ users = {u["id"]: u for u in self.view.view_snapshot()["users"]}+ self.assertTrue(users["carol"]["removed"])+++class Messages(ViewBase):+ def test_windows_of_a_long_chat(self):+ ids = [self.alice.send(chat="bob", text=f"m{i}")["message"]["seq"] for i in range(30)]+ chat = self.alice.chats()["chats"][0]["id"]+ newest = self.view.view_messages(chat=chat, limit=10)+ self.assertEqual([m["text"] for m in newest["messages"] if m["kind"] == "message"], [f"m{i}" for i in range(20, 30)])+ self.assertTrue(newest["last"])+ self.assertFalse(newest["first"])+ older = self.view.view_messages(chat=chat, limit=10, before=ids[20])+ self.assertEqual([m["text"] for m in older["messages"] if m["kind"] == "message"], [f"m{i}" for i in range(10, 20)])+ first = self.view.view_messages(chat=chat, limit=10, before=ids[10])+ self.assertTrue(first["first"])+ self.assertEqual(first["messages"][0]["kind"], "created") # the chat's start comes with its first page+ around = self.view.view_messages(chat=chat, limit=5, around=ids[15])+ self.assertEqual([m["text"] for m in around["messages"]], ["m13", "m14", "m15", "m16", "m17"])+ after = self.view.view_messages(chat=chat, limit=5, after=ids[27])+ self.assertEqual([m["text"] for m in after["messages"]], ["m28", "m29"])++ def test_replies_carry_what_they_answer_and_read_positions_come_along(self):+ first = self.alice.send(chat="bob,carol", text="question for everyone")["message"]+ filler = [self.alice.send(chat="bob,carol", text=f"filler {i}")["message"] for i in range(5)]+ self.bob.read()+ self.bob.send(chat="alice,carol", text="an answer", reply_to=first["id"])+ r = self.view.view_messages(chat=first["chat"], limit=1)+ reply = r["messages"][-1]+ self.assertEqual(reply["parent"]["text"], "question for everyone")+ self.assertEqual(reply["reply_to_from"], "alice")+ self.assertEqual(r["positions"]["bob"], filler[-1]["seq"])+ self.assertEqual(r["positions"]["carol"], 0)+ whole = self.view.view_messages(chat=first["chat"])+ self.assertEqual(whole["messages"][1]["replies"], 1)++ def test_thread_and_receipts(self):+ root = self.alice.send(chat="bob,carol", text="root")["message"]+ self.bob.read()+ self.carol.read()+ a = self.bob.send(chat="alice,carol", text="a", reply_to=root["id"])["message"]+ self.carol.send(chat="alice,bob", text="b", reply_to=a["id"])+ t = self.view.view_thread(id=a["id"])+ self.assertEqual([(m["text"], m["depth"]) for m in t["messages"]], [("root", 0), ("a", 1), ("b", 2)])+ r = self.view.view_receipts(id=root["id"])+ self.assertEqual(sorted(x["id"] for x in r["read"]), ["bob", "carol"]) # each replied after reading+ self.assertTrue(all(x["at"] for x in r["read"]))+ # carol answered a without reading it: --reply-to needs only its id+ self.assertEqual(self.view.view_receipts(id=a["id"])["unread"], ["alice", "carol"])+ self.assertEqual(r["unread"], [])+++class Boards(ViewBase):+ def test_posts_sort_by_votes_comments_and_time(self):+ self.alice.channel_create(name="board", members="bob,carol", public=True)+ quiet = self.alice.send(chat="#board", text="quiet post")["message"]+ busy = self.bob.send(chat="#board", text="busy post\nwith a body")["message"]+ voted = self.carol.send(chat="#board", text="voted post")["message"]+ for user in (self.alice, self.bob):+ user.react(id=voted["id"], emoji="+1")+ r1 = self.alice.send(chat="#board", text="reply", reply_to=busy["id"])["message"]+ self.carol.send(chat="#board", text="reply to reply", reply_to=r1["id"])+ self.alice.send(chat="bob", text="a direct chat is no board")+ new = self.view.view_posts(sort="new")["posts"]+ self.assertEqual([p["text"].split("\n")[0] for p in new], ["voted post", "busy post", "quiet post"])+ top = self.view.view_posts(sort="top")["posts"]+ self.assertEqual([(p["votes"], p["comments"]) for p in top], [(2, 0), (0, 2), (0, 0)])+ active = self.view.view_posts(sort="active")["posts"]+ self.assertEqual(active[0]["id"], busy["id"])+ self.assertEqual(self.view.view_posts(sort="new", limit=1, offset=1)["posts"][0]["id"], busy["id"])+ self.assertEqual(len(self.view.view_posts(chat=quiet["chat"])["posts"]), 3)+ self.assertEqual(self.view.view_posts(**{"as": "carol"})["posts"][0]["chat"], quiet["chat"])+++class Search(ViewBase):+ def setUp(self):+ super().setUp()+ self.alice.channel_create(name="ops", members="bob,carol")+ self.m = []+ for user, chat, text in ((self.alice, "#ops", "Deploy the API now"), (self.bob, "#ops", "deploying, @carol watch it"),+ (self.carol, "bob", "private: deploy went fine"), (self.bob, "#ops", 'she said "deploy" twice')):+ self.m.append(user.send(chat=chat, text=text)["message"])+ time.sleep(0.005) # times are kept to the millisecond++ def texts(self, **params):+ return [m["text"] for m in self.view.view_search(**params)["messages"]]++ def test_text_anywhere_newest_first_in_any_case(self):+ self.assertEqual(self.texts(text="DEPLOY"), ['she said "deploy" twice', "private: deploy went fine",+ "deploying, @carol watch it", "Deploy the API now"])+ self.assertEqual(self.texts(text='"deploy"'), ['she said "deploy" twice'])+ self.assertEqual(self.texts(text="alice"), []) # in every line's sender, but in no text+ self.assertEqual(self.texts(text="2026"), []) # in every line's time++ def test_filters(self):+ self.assertEqual(self.texts(text="deploy", **{"from": ["bob"]}), ['she said "deploy" twice', "deploying, @carol watch it"])+ self.assertEqual(self.texts(mentions="carol"), ["deploying, @carol watch it"])+ self.assertEqual(self.texts(text="deploy", chats=[self.m[2]["chat"]]), ["private: deploy went fine"])+ self.assertEqual(self.texts(text="deploy", **{"as": "alice"}), ['she said "deploy" twice',+ "deploying, @carol watch it", "Deploy the API now"])+ self.assertEqual(len(self.texts()), 4) # no text: the newest messages everywhere+ self.assertEqual(self.texts(after=self.m[3]["at"]), ['she said "deploy" twice'])+ self.assertEqual(self.texts(before=self.m[1]["at"]), ["Deploy the API now"])++ def test_edits_deletes_and_pages(self):+ self.alice.edit(id=self.m[0]["id"], text="Ship the API now")+ self.bob.delete(id=self.m[1]["id"])+ self.assertEqual(self.texts(text="deploy"), ['she said "deploy" twice', "private: deploy went fine"])+ self.assertEqual(self.texts(text="ship"), ["Ship the API now"])+ page = self.view.view_search(text="deploy", limit=1)+ self.assertEqual([m["text"] for m in page["messages"]], ['she said "deploy" twice'])+ rest = self.view.view_search(text="deploy", limit=10, until=page["until"])+ self.assertEqual([m["text"] for m in rest["messages"]], ["private: deploy went fine"])+ self.assertIsNone(rest["until"])++ def test_words_found_in_long_logs_and_unicode(self):+ data = tempfile.mkdtemp(prefix="swarmchat-test-")+ server = Server(data, segment=5) # a new log file every 5 lines+ try:+ alice = server.user("alice", "Alice")+ server.user("bob", "Bob")+ alice.send(chat="bob", text="Schöne Grüße aus München")+ for i in range(40):+ alice.send(chat="bob", text=f"note {i}" + (" needle" if i % 10 == 3 else ""))+ with open(os.path.join(data, "view.token")) as f:+ self.view = server.client(f.read().strip())+ self.check_words()+ finally:+ server.stop()+ shutil.rmtree(data)++ def check_words(self):+ self.assertEqual(self.texts(text="NEEDLE"), [f"note {i} needle" for i in (33, 23, 13, 3)])+ self.assertEqual(self.texts(text="NEEDLE", limit=2), ["note 33 needle", "note 23 needle"])+ self.assertEqual(self.texts(text="GRÜSSE"), [])+ self.assertEqual(self.texts(text="GRÜẞE".lower()), ["Schöne Grüße aus München"])+ self.assertEqual(self.texts(text="MÜNCHEN"), ["Schöne Grüße aus München"])+++class Live(ViewBase):+ def test_the_feed_carries_everything_in_order(self):+ start = self.view.view_snapshot()["feed"]+ got = []++ def follow(): # as the page does: ask again until the message has come+ position = start+ while not any(e["kind"] == "message" for e in got):+ r = self.view.view_feed(after=position, timeout=10)+ got.extend(r["events"])+ position = r["feed"]+ got.append(position)++ t = threading.Thread(target=follow)+ t.start()+ time.sleep(0.2)+ sent = self.alice.send(chat="bob", text="live one")["message"]+ t.join(5)+ position = got.pop()+ self.assertEqual([e["kind"] for e in got], ["chat", "message"]) # the direct chat is made, then written to+ self.assertEqual(got[0]["event"]["kind"], "created")+ self.assertEqual((got[1]["message"]["text"], got[1]["count"]), ("live one", 1))+ self.bob.read()+ self.alice.react(id=sent["id"], emoji="+1")+ self.alice.edit(id=sent["id"], text="live one, edited")+ self.server.admin.user_add(id="dora", name="Dora")+ self.alice.channel_create(name="new", members="dora")+ r = self.view.view_feed(after=position, timeout=0.01)+ kinds = [e["kind"] for e in r["events"]]+ self.assertEqual(kinds[:4], ["read", "change", "change", "user"])+ self.assertIn("chat", kinds)+ self.assertEqual(r["events"][2]["message"]["text"], "live one, edited")+ self.assertEqual(r["events"][2]["message"]["reactions"], {"+1": ["alice"]})+ self.assertEqual(r["events"][0]["user"], "bob")+ self.assertEqual(self.view.view_feed(after=r["feed"], timeout=0.01)["events"], [])++ def test_a_restarted_server_tells_the_viewer_to_start_again(self):+ snap = self.view.view_snapshot()+ self.alice.send(chat="bob", text="before")+ self.assertFalse(self.view.view_feed(after=snap["feed"], feed_id=snap["server"]["feed_id"], timeout=0.01).get("reset"))+ self.restart()+ with open(os.path.join(self.data, "view.token")) as f:+ view = self.server.client(f.read().strip())+ self.assertTrue(view.view_feed(after=1, feed_id=snap["server"]["feed_id"], timeout=0.01)["reset"])+ self.assertTrue(view.view_feed(after=snap["feed"] + 5, timeout=0.01)["reset"]) # ahead of the new count++ def test_a_viewer_too_far_behind_starts_again(self):+ self.server.store.feed.entries = __import__("collections").deque(maxlen=3)+ for i in range(5):+ self.alice.send(chat="bob", text=str(i))+ self.assertTrue(self.view.view_feed(after=0, timeout=0.01)["reset"])+ latest = self.server.store.feed.count+ self.assertEqual(len(self.view.view_feed(after=latest - 3, timeout=0.01)["events"]), 3)+++class Perspective(ViewBase):+ def test_as_an_agent_and_what_it_knew(self):+ hello = self.alice.send(chat="bob", text="hello bob")["message"]+ self.alice.send(chat="bob,carol", text="group news")+ self.carol.send(chat="bob", text="from carol, unread")+ self.bob.read(chat="alice") # bob reads alice's direct chat only+ answer = self.bob.send(chat="alice", text="hi alice", reply_to=hello["id"])["message"]+ seen = self.view.view_as(user="bob", unread=True)+ unread = {c["title"]: c["unread"] for c in seen["chats"]}+ self.assertEqual(unread, {"Alice Ames": 0, "Alice Ames, Carol Cruz": 1, "Carol Cruz": 1})+ self.assertEqual([m["text"] for m in seen["unread"]], ["group news", "from carol, unread"])+ self.assertEqual(seen["positions"][hello["chat"]], hello["seq"])+ knew = self.view.view_knew(id=answer["id"])+ rows = {r["chat"]: r for r in knew["chats"]}+ self.assertEqual(knew["chats"][0]["chat"], hello["chat"])+ self.assertEqual(rows[hello["chat"]]["unseen"], 0)+ self.assertEqual(sorted(r["unseen"] for r in knew["chats"]), [0, 1, 1])+ self.assertEqual(self.bob.chats(unread=True)["chats"][0]["unread"], 1) # still unread after all that++ def test_what_it_knew_counts_only_chats_it_was_in(self):+ self.alice.channel_create(name="later", members="carol")+ self.alice.send(chat="#later", text="before bob joined")+ first = self.bob.send(chat="alice", text="first")["message"]+ time.sleep(0.01) # times are kept to the millisecond+ self.alice.channel_add(chat="#later", users="bob")+ knew = self.view.view_knew(id=first["id"])+ self.assertEqual([r["chat"] for r in knew["chats"]], [first["chat"]])+++class Activity(ViewBase):+ def test_counts_over_time_and_in_a_window(self):+ for i in range(12):+ (self.alice if i % 3 else self.bob).send(chat="alice" if i % 3 == 0 else "bob", text=str(i))+ time.sleep(0.005)+ r = self.view.view_activity(buckets=4)+ self.assertEqual(sum(r["counts"]), 12)+ whole = self.view.view_activity(buckets=4, **{"from": r["start"], "to": r["end"]})["window"]+ chat = next(iter(whole))+ self.assertEqual(whole[chat], {"messages": 12, "sent": {"alice": 8, "bob": 4}})+ part = self.view.view_activity(buckets=4, **{"from": r["start"] + 0.002, "to": r["end"]})["window"]+ self.assertEqual(part[chat]["messages"], 11)+++def png(w, h) -> bytes:+ """A real PNG of w by h grey pixels."""+ def chunk(kind, data):+ return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))+ raw = (b"\0" + b"\x80" * w) * h+ return (b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 0, 0, 0, 0))+ + chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))+++def jpeg(w, h, orientation=None, frame=b"\xc0") -> bytes:+ """The start of a JPEG: an optional EXIF segment with an orientation, then a frame header."""+ out = b"\xff\xd8"+ out += b"\xff\xe0" + struct.pack(">H", 16) + b"JFIF\0" + b"\1\1\0\0\1\0\1\0\0"+ if orientation:+ tiff = b"II" + struct.pack("<HI", 42, 8) + struct.pack("<H", 1) + struct.pack("<HHIHH", 0x0112, 3, 1, orientation, 0)+ payload = b"Exif\0\0" + tiff + struct.pack("<I", 0)+ out += b"\xff\xe1" + struct.pack(">H", len(payload) + 2) + payload+ out += b"\xff" + frame + struct.pack(">HBHHB", 11, 8, h, w, 1) + b"\1\x11\0"+ return out + b"\xff\xd9"+++class Pictures(ViewBase):+ """The page draws attached pictures, so each one comes with its type and size, read from its own bytes."""++ def test_pictures_are_known_by_their_bytes_whatever_their_name(self):+ tmp = tempfile.mkdtemp(prefix="swarmchat-pictures-")+ self.addCleanup(shutil.rmtree, tmp)+ riff = lambda chunk, body: b"RIFF" + struct.pack("<I", 4 + 8 + len(body)) + b"WEBP" + chunk + struct.pack("<I", len(body)) + body+ files = {+ "chart.png": png(30, 20),+ "anim.gif": b"GIF89a" + struct.pack("<HH", 31, 21) + bytes(20),+ "lossy.webp": riff(b"VP8 ", b"\0\0\0\x9d\x01\x2a" + struct.pack("<HH", 40, 30) + bytes(10)),+ "lossless.webp": riff(b"VP8L", b"\x2f" + struct.pack("<I", (41 - 1) | (31 - 1) << 14) + bytes(10)),+ "extended.webp": riff(b"VP8X", bytes(4) + (1999).to_bytes(3, "little") + (999).to_bytes(3, "little") + bytes(10)),+ "old.bmp": b"BM" + bytes(12) + struct.pack("<Iii", 40, 50, -60) + bytes(30),+ "older.bmp": b"BM" + bytes(12) + struct.pack("<IHH", 12, 8, 9) + bytes(30),+ "BMW notes.txt": b"BMW notes: the M3 needs new tyres, the X5 a service, nothing else.",+ "plain.jpg": jpeg(200, 100, frame=b"\xc2"),+ "turned.jpg": jpeg(200, 100, orientation=6),+ "upright.jpg": jpeg(200, 100, orientation=1),+ "renamed": png(5, 7), # no extension: the bytes still say PNG+ "fake.png": b"just some text, not a picture",+ "cut.jpg": jpeg(200, 100)[:12],+ "huge.png": png(1, 1)[:16] + struct.pack(">II", 20000, 20000) + bytes(20),+ "large.png": png(1, 1)[:16] + struct.pack(">II", 6000, 5000) + bytes(20), # 30 megapixels+ }+ paths = []+ for name, data in files.items():+ paths.append(os.path.join(tmp, name))+ with open(paths[-1], "wb") as f:+ f.write(data)+ os.makedirs(os.path.join(tmp, "shots"))+ with open(os.path.join(tmp, "shots", "a.png"), "wb") as f:+ f.write(png(3, 3))+ start = self.server.store.feed.count+ sent = self.alice.send(chat="bob", text="pictures", attach=[*paths, os.path.join(tmp, "shots")])["message"]++ def shapes(m):+ return {a["name"]: (a["image"]["type"], a["image"]["width"], a["image"]["height"]) if "image" in a else None+ for a in m["attachments"]}+ expected = {+ "chart.png": ("image/png", 30, 20), "anim.gif": ("image/gif", 31, 21), "lossy.webp": ("image/webp", 40, 30),+ "lossless.webp": ("image/webp", 41, 31), "extended.webp": ("image/webp", 2000, 1000),+ "old.bmp": ("image/bmp", 50, 60), "older.bmp": ("image/bmp", 8, 9), "BMW notes.txt": None, "plain.jpg": ("image/jpeg", 200, 100),+ "turned.jpg": ("image/jpeg", 100, 200), # turned a quarter, as a browser draws it+ "upright.jpg": ("image/jpeg", 200, 100), "renamed": ("image/png", 5, 7),+ "fake.png": None, "cut.jpg": None, "huge.png": None, "large.png": None, "shots": None,+ }+ window = self.view.view_messages(chat=sent["chat"])["messages"]+ self.assertEqual(shapes(next(m for m in window if m["kind"] == "message")), expected)+ event = next(e for e in self.view.view_feed(after=start, timeout=0.01)["events"] if e["kind"] == "message")+ self.assertEqual(shapes(event["message"]), expected)+ self.assertEqual(shapes(self.view.view_search(text="pictures")["messages"][0]), expected)+ self.assertEqual(shapes(self.view.view_as(user="bob", unread=True)["unread"][0]), expected)+ self.assertNotIn("image", self.bob.read(chat="alice", peek=True)["messages"][0]["attachments"][0]) # the viewer's only+ self.alice.delete(id=sent["id"])+ self.assertNotIn("attachments", self.view.view_messages(chat=sent["chat"])["messages"][-1])+++class Texts(ViewBase):+ """The page shows a text file's first lines under its message, and the whole of it in a viewer."""++ def setUp(self):+ super().setUp()+ tmp = tempfile.mkdtemp(prefix="swarmchat-texts-")+ self.addCleanup(shutil.rmtree, tmp)+ files = {+ "notes.md": b"# Title\n\nSome *text*.\n",+ "run.log": "".join(f"2026-10-13T14:00:00Z INFO line {i}\n" for i in range(2000)).encode(),+ "windows.txt": b"one\r\ntwo\r\n",+ "marked.py": "\ufeffprint('hi')\n".encode(),+ "wide.txt": b"x" * 1000 + b"\n",+ "cut.txt": ("a" + "\u00e9" * 20000).encode(), # the bytes read end inside a character+ "latin1.txt": "caf\u00e9 cr\u00e8me".encode("latin-1"), # not UTF-8+ "zeros.txt": b"text\0with a NUL",+ "empty.txt": b"",+ "chart.png": png(3, 2),+ }+ paths = []+ for name, data in files.items():+ paths.append(os.path.join(tmp, name))+ with open(paths[-1], "wb") as f:+ f.write(data)+ os.makedirs(os.path.join(tmp, "folder"))+ with open(os.path.join(tmp, "folder", "a.txt"), "w") as f:+ f.write("inside\n")+ self.sent = self.alice.send(chat="bob", text="files", attach=[*paths, os.path.join(tmp, "folder")])["message"]+ self.ids = {a["name"]: a["id"] for a in self.sent["attachments"]}++ def test_text_is_known_by_its_bytes_and_comes_with_its_first_lines(self):+ m = next(m for m in self.view.view_messages(chat=self.sent["chat"])["messages"] if m["kind"] == "message")+ texts = {a["name"]: a.get("text") for a in m["attachments"]}+ self.assertEqual(texts["notes.md"], {"preview": "# Title\n\nSome *text*.", "more": False, "lines": 3})+ self.assertEqual(texts["run.log"]["preview"].split("\n"), [f"2026-10-13T14:00:00Z INFO line {i}" for i in range(8)])+ self.assertTrue(texts["run.log"]["more"])+ self.assertNotIn("lines", texts["run.log"]) # only its start was read+ self.assertEqual(texts["windows.txt"], {"preview": "one\ntwo", "more": False, "lines": 2})+ self.assertEqual(texts["marked.py"], {"preview": "print('hi')", "more": False, "lines": 1})+ self.assertEqual(texts["wide.txt"], {"preview": "x" * 200, "more": True, "lines": 1})+ self.assertTrue(texts["cut.txt"]["preview"].startswith("a\u00e9\u00e9") and texts["cut.txt"]["more"])+ for name in ("latin1.txt", "zeros.txt", "empty.txt", "chart.png", "folder"):+ self.assertIsNone(texts[name], name)+ self.assertIn("image", next(a for a in m["attachments"] if a["name"] == "chart.png"))+ self.assertEqual(self.view.view_search(text="files")["messages"][0]["attachments"][0]["text"], texts["notes.md"])+ self.assertNotIn("text", self.bob.read(chat="alice", peek=True)["messages"][0]["attachments"][0]) # the viewer's only++ def test_the_viewer_gets_the_whole_text_or_its_start_cut_at_a_line(self):+ r = self.view.view_text(id=self.ids["notes.md"])+ self.assertEqual(r, {"text": "# Title\n\nSome *text*.\n", "size": 22, "shown": 22, "cut": False})+ self.assertEqual(self.view.view_text(id=self.ids["marked.py"])["text"], "print('hi')\n")+ self.assertEqual(self.view.view_text(id=self.ids["latin1.txt"])["text"], "caf\ufffd cr\ufffdme")+ self.assertEqual(self.view.view_text(id=self.ids["zeros.txt"])["text"], "text\ufffdwith a NUL")+ with mock.patch.object(view, "TEXT_BYTES", 100):+ r = self.view.view_text(id=self.ids["run.log"])+ self.assertTrue(r["cut"])+ self.assertEqual(r["text"], "".join(f"2026-10-13T14:00:00Z INFO line {i}\n" for i in range(3)))+ self.assertEqual(r["shown"], len(r["text"]))+ with self.assertRaises(ClientError) as e:+ self.view.view_text(id=self.ids["folder"])+ self.assertEqual(e.exception.status, 400)+ with self.assertRaises(ClientError) as e:+ self.view.view_text(id="nope")+ self.assertEqual(e.exception.status, 404)+ with self.assertRaises(ClientError) as e:+ self.bob.view_text(id=self.ids["notes.md"]) # an agent cannot look through the viewer+ self.assertEqual(e.exception.status, 401)+++class Page(ViewBase):+ def get(self, path, headers=None):+ conn = http.client.HTTPConnection("127.0.0.1", self.server.httpd.server_address[1], timeout=10)+ try:+ conn.request("GET", path, headers=headers or {})+ r = conn.getresponse()+ return r, r.read()+ finally:+ conn.close()++ def test_the_page_is_one_file_with_a_strict_policy(self):+ r, body = self.get("/view")+ self.assertEqual(r.status, 200)+ html = body.decode()+ self.assertNotIn("<!--JS-->", html)+ policy = r.getheader("Content-Security-Policy")+ script = html.split("<script>", 1)[1].split("</script>", 1)[0]+ digest = __import__("base64").b64encode(hashlib.sha256(script.encode()).digest()).decode()+ self.assertIn(f"script-src 'sha256-{digest}'", policy)+ self.assertIn("frame-ancestors 'none'", policy)+ packed, data = self.get("/view", {"Accept-Encoding": "gzip"})+ self.assertEqual(packed.getheader("Content-Encoding"), "gzip")+ self.assertEqual(gzip.decompress(data), body)+ again, _ = self.get("/view", {"If-None-Match": r.getheader("ETag")})+ self.assertEqual(again.status, 304)+ home, _ = self.get("/")+ self.assertEqual((home.status, home.getheader("Location")), (302, "/view"))++ def test_a_missing_page_is_reported_plainly(self):+ with mock.patch.object(view, "PAGE_DIR", os.path.join(self.data, "nowhere")), mock.patch.dict(view._page, clear=True):+ r, body = self.get("/view")+ self.assertEqual(r.status, 500)+ self.assertIn("missing from this install", json.loads(body)["error"])++ def test_large_replies_are_compressed_for_browsers_only(self):+ for i in range(30):+ self.alice.send(chat="bob", text=f"message {i} " * 10)+ body = json.dumps({"chat": self.alice.chats()["chats"][0]["id"]}).encode()+ conn = http.client.HTTPConnection("127.0.0.1", self.server.httpd.server_address[1], timeout=10)+ conn.request("POST", "/v1/view_messages", body, {"Authorization": f"Bearer {self.view.token}",+ "Accept-Encoding": "gzip, br"})+ r = conn.getresponse()+ self.assertEqual(r.getheader("Content-Encoding"), "gzip")+ self.assertEqual(len(json.loads(gzip.decompress(r.read()))["messages"]), 31)+ conn.close()+ self.assertEqual(len(self.view.view_messages(chat=json.loads(body)["chat"])["messages"]), 31)+++class ReadOnly(unittest.TestCase):+ def setUp(self):+ self.data = tempfile.mkdtemp(prefix="swarmchat-view-")+ server = Server(self.data)+ alice = server.user("alice", "Alice")+ server.user("bob", "Bob")+ alice.send(chat="bob", text="kept on disk")+ with open(os.path.join(self.data, "chats", alice.chats()["chats"][0]["id"], "log", "000001.jsonl"), "ab") as f:+ f.write(b'{"seq": 99, "half a line') # a crash in the middle of a write+ server.stop()++ def tearDown(self):+ shutil.rmtree(self.data)++ def test_a_folder_is_shown_without_a_single_change(self):+ before = folder_state(self.data)+ httpd, store = make_server(config.load(data=self.data, port=0), read_only=True)+ threading.Thread(target=httpd.serve_forever, kwargs={"poll_interval": 0.05}, daemon=True).start()+ try:+ view = Client(httpd.url, store.view_token)+ snap = view.view_snapshot()+ self.assertTrue(snap["server"]["read_only"])+ chat = snap["chats"][0]["id"]+ self.assertEqual(view.view_messages(chat=chat)["messages"][-1]["text"], "kept on disk")+ with open(os.path.join(self.data, "admin.token")) as f:+ admin = Client(httpd.url, f.read().strip())+ with self.assertRaises(ClientError) as e:+ admin.user_add(id="eve")+ self.assertEqual(e.exception.status, 403)+ with self.assertRaises(ClientError):+ admin.view_snapshot() # only this process's own token opens it+ view.close()+ admin.close()+ finally:+ httpd.shutdown()+ httpd.server_close()+ store.close()+ self.assertEqual(folder_state(self.data), before)++ def test_the_command_uses_a_running_server_or_else_the_folder(self):+ env = {**os.environ, "PYTHONPATH": ROOT, "SWARMCHAT_CONFIG_DIR": self.data + "/config"}+ proc = subprocess.Popen([sys.executable, "-m", "swarmchat", "view", "--data", self.data, "--no-open"], env=env,+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, text=True)+ try:+ proc.stdout.readline()+ url = proc.stdout.readline().split("Viewer: ", 1)[1].strip()+ base, token = url.split("/view#token=")+ view = Client(base, token)+ self.assertEqual(view.view_snapshot()["server"]["read_only"], True)+ view.close()+ finally:+ proc.terminate()+ proc.wait(10)+ proc.stdout.close()+ server = Server(self.data)+ try:+ r = subprocess.run([sys.executable, "-m", "swarmchat", "view", "--data", self.data, "--no-open"], env=env,+ capture_output=True, text=True, timeout=30)+ self.assertEqual(r.returncode, 0, r.stderr)+ with open(os.path.join(self.data, "view.token")) as f:+ self.assertEqual(r.stdout.strip(), f"Viewer: {server.url}/view#token={f.read().strip()}")+ finally:+ server.stop()+++if __name__ == "__main__":+ unittest.main()