Skip to content

A team of long-lived bots for DeepSeek Harness, with main bots, bot-to-bot messages and group chats.

Stop disposed learning passes from saving stale state

Fei2411b1b55dbeac37
diff --cc ds-bot/src/host/learning.js
index a13b1f7d,00000000..ba755e4a
mode 100644,000000..100644
--- a/ds-bot/src/host/learning.js
+++ b/ds-bot/src/host/learning.js
@@@ -1,414 -1,0 +1,427 @@@
+// Learning from work. After a long turn a Bot's chat is marked for review. When the
+// team has been idle for a while, and at most every few hours, one background pass
+// reads what was said since the last pass and asks the Bot's model whether the work
+// taught a skill worth keeping. Most work teaches nothing, so most passes propose
+// nothing. A proposal never takes effect by itself: it waits as a card in the Bot's
+// chat until the user approves it, and a declined one is remembered so the same idea
+// does not come back.
+import { createHash, randomUUID } from 'node:crypto'
+import { readFileSync } from 'node:fs'
+import { mkdir, writeFile } from 'node:fs/promises'
+import { dirname, join } from 'node:path'
+import { replaceFile } from './files.js'
+import { looksLikeKey, looksSecret, writtenIn } from './memory.js'
+import { validateCapabilityName } from './capabilities.js'
+
+export const LEARNING = {
+ minTurnMs: 3 * 60_000,
+ idleMs: 10 * 60_000,
+ intervalMs: 12 * 60 * 60_000,
+ tickMs: 60_000,
+ readMs: 15_000,
+ textMax: 24_000,
+ userTextMax: 8_000,
+ userLineMax: 2_000,
+ botLineMax: 1_200,
+ learnedMax: 5,
+ descriptionMax: 300,
+ contentMax: 8_000,
+ declinedMax: 20,
+}
+
+const clip = (text, max) => (text.length > max ? `${text.slice(0, max)} […]` : text)
+export const hashOf = text => createHash('sha256').update(String(text ?? '')).digest('hex').slice(0, 16)
+
+// The answer of the review's model: one JSON object, perhaps in a code fence.
+export function parseJson(text) {
+ const raw = String(text ?? '').trim()
+ // A skill's content may hold code fences of its own, so read the whole answer first
+ // and fall back to a fenced block only when that fails.
+ const fenced = /^```(?:json)?\s*([\s\S]*?)```\s*$/.exec(raw)?.[1]
+ for (const candidate of [raw, fenced]) {
+ if (candidate === undefined) continue
+ const start = candidate.indexOf('{')
+ const end = candidate.lastIndexOf('}')
+ if (start === -1 || end < start) continue
+ try {
+ const value = JSON.parse(candidate.slice(start, end + 1))
+ if (value && typeof value === 'object' && !Array.isArray(value)) return value
+ } catch {}
+ }
+ return undefined
+}
+
+// ---------------------------------------------------------------------------
+// The skill review.
+
+export function skillSystem() {
+ return [
+ 'You review a stretch of finished work by an AI Bot and decide whether it taught a skill worth keeping.',
+ 'A skill is a short document that a future Bot loads when a task of the same class comes up. Strong models already do most tasks well without one, so a weak or narrow skill only adds noise. Most work teaches nothing new: answer "none" unless every test below passes.',
+ '',
+ 'Tests:',
+ '1. Reusable: tasks of this class will come up again in later, different work. A rule that only fits this task, file, bug, or pull request fails.',
+ '2. Missing knowledge: it gives constraints or context that a capable model does not have or gets wrong: requirements and corrections the user stated, project or tool conventions found by trial and error, pitfalls that cost time, how this user wants the result checked or delivered. General know-how fails, and so does anything that reading the code or the docs shows at once.',
+ '3. Earned: the work shows it. The user asked for it or corrected the Bot, or a non-obvious way was found after one that failed. Never invent rules the work did not show.',
+ '4. Not covered: no listed skill already says it. When a learned skill of this Bot covers the class, update that skill instead of making a new one. Also update a learned skill that the work showed to be wrong, outdated, or missing a step.',
+ '',
+ 'Never put in a skill: one-off facts, task progress, file lists, dates, version, pull request or issue numbers, pasted errors or logs, facts about the user that are not about how to work (memory keeps those), or keys, passwords, and tokens.',
+ 'Write rules that stand alone, not the story of this work: a reason says why the rule matters, never when or how it was broken ("the user corrected this before", "version 1.2 missed it"). Examples must be general, not the files, values, or fixes of this task.',
+ '',
+ 'Form:',
+ '- name: the class of task, lowercase letters, digits, and single hyphens, 32 characters at most, e.g. "release-checks", never "fix-login-bug".',
+ '- description: one sentence, 200 characters at most, that says when to use the skill. Be concrete about the trigger.',
+ '- content: Markdown, 600 words at most. The rules as short imperative lines, each with a brief reason when it is not obvious; numbered steps only where order matters; how to check the work is done. Keep what is still true when you update, and remove what the work showed to be wrong.',
+ '',
+ 'Answer with one JSON object only, one of:',
+ '{"action":"none","reason":"<one sentence>"}',
+ '{"action":"create","name":"…","description":"…","content":"…","why":"<one sentence: what in this work shows the skill is needed>"}',
+ '{"action":"update","name":"<a learned skill listed>","description":"…","content":"<the whole new content>","why":"<one sentence>"}',
+ ].join('\n')
+}
+
+// `learned`: this Bot's learned skills, in full. `library`: every other skill on the
+// team, by name and description. `declined`: proposals the user turned down.
+export function skillRequest({ bot, lines, learned = [], library = [], declined = [], language, limit = LEARNING.learnedMax }) {
+ const parts = [`The Bot: ${bot.name}${bot.role ? `, ${bot.role}` : ''}.`]
+ parts.push(learned.length === 0
+ ? 'This Bot has no learned skills yet.'
+ : `This Bot's learned skills (${learned.length} of ${limit}${learned.length >= limit ? '; at the limit, only update one' : ''}):\n${learned.map(skill => `<skill name="${skill.name}">\ndescription: ${skill.description}\n${skill.content}\n</skill>`).join('\n')}`)
+ if (library.length > 0) parts.push(`Other skills on the team:\n${library.map(skill => `- ${skill.name}: ${skill.description}`).join('\n')}`)
+ if (declined.length > 0) parts.push(`Proposals the user declined. Do not propose them again unless this work shows something new:\n${declined.map(entry => `- ${entry.name}: ${entry.description}`).join('\n')}`)
+ if (language) parts.push(`Write the description and the content in ${language}.`)
+ parts.push(`The work, oldest first ("User:" lines are the user's own words):\n<work>\n${lines.join('\n')}\n</work>`)
+ return parts.join('\n\n')
+}
+
+// What keeps a proposal from the user, or undefined when it may be shown.
+// `skills`: the team's skills by name; `learned`: names this Bot learned.
+// Skills often talk about code such as `password == input`; read keys everywhere, but
+// password wording only outside code.
+const holdsSecret = text => looksLikeKey(text) || looksSecret(text.replace(/```[\s\S]*?```/g, ' ').replace(/`[^`\n]*`/g, ' '))
+
+export function skillProblem(proposal, { skills, learned, limit = LEARNING.learnedMax }) {
+ if (proposal === undefined) return 'the answer is not a JSON object'
+ const { action } = proposal
+ if (action === 'none') return undefined
+ if (action !== 'create' && action !== 'update') return `unknown action ${String(action)}`
+ let name
+ try {
+ name = validateCapabilityName(proposal.name)
+ } catch (error) {
+ return error.message
+ }
+ const description = typeof proposal.description === 'string' ? proposal.description.trim() : ''
+ const content = typeof proposal.content === 'string' ? proposal.content.trim() : ''
+ const why = typeof proposal.why === 'string' ? proposal.why.trim() : ''
+ if (description === '' || description.length > LEARNING.descriptionMax) return `the description must have 1 to ${LEARNING.descriptionMax} characters`
+ if (content === '' || content.length > LEARNING.contentMax) return `the content must have 1 to ${LEARNING.contentMax} characters`
+ if (holdsSecret(description) || holdsSecret(content) || holdsSecret(why)) return 'it looks like it holds a secret'
+ const existing = Object.hasOwn(skills, name) ? skills[name] : undefined
+ if (action === 'create') {
+ if (existing !== undefined) return `${name} already exists`
+ if (learned.length >= limit) return `the Bot already keeps ${limit} learned skills`
+ } else {
+ if (existing === undefined || existing.kind !== 'skill' || !learned.includes(name)) return `${name} is not a learned skill of this Bot`
+ if (existing.content.trim() === content && existing.description.trim() === description) return 'the update changes nothing'
+ }
+ return undefined
+}
+
+// ---------------------------------------------------------------------------
+
+export function install(rt) {
+ const { ctx, config, home, state } = rt
+ const settings = { ...LEARNING, ...(config.learningTimes ?? {}) }
+ const on = config.learning !== false && config.memory !== false
+ const view = () => ({})
+ if (!on) {
+ Object.assign(rt, { learningView: view, learningEndpoint: async () => { throw new Error('Learning is turned off on this host') }, learningRun: async () => 0 })
+ return
+ }
+
+ const root = join(home, 'learning')
+ const path = join(root, 'state.json')
+ // `pending`: Bot id → the chat Sessions with a long turn since the last pass.
+ // `cursors`: Session id → the newest line a pass has read. `proposals`: waiting
+ // for the user, by id. `learned`: skill names learned by each Bot.
+ let saved
+ const data = () => {
+ if (saved !== undefined) return saved
+ try {
+ saved = JSON.parse(readFileSync(path, 'utf8'))
+ } catch (error) {
+ if (error?.code !== 'ENOENT') rt.warn('ds-bot: unreadable learning state %s: %s', path, error?.message ?? error)
+ saved = {}
+ }
+ saved.pending ??= {}
+ saved.cursors ??= {}
+ saved.proposals ??= {}
+ saved.declined ??= []
+ saved.learned ??= {}
+ return saved
+ }
++ let stopped = false
+ let writing = Promise.resolve()
+ const persist = () => {
++ if (stopped) return writing
+ const text = JSON.stringify(data(), null, 1)
+ writing = writing.then(async () => {
+ await mkdir(dirname(path), { recursive: true })
+ const temp = `${path}.${process.pid}.tmp`
+ await writeFile(temp, text, 'utf8')
+ await replaceFile(temp, path)
+ }).catch(error => rt.warn('ds-bot: saving the learning state failed: %s', error?.message ?? error))
+ return writing
+ }
+
+ // A turn in a Bot's own chat that ran long marks the chat for the next pass.
+ const started = new Map()
+ const noted = new Map()
+ let lastActivity = Date.now()
+ ctx.on('agent/status', ({ agent, status }) => {
++ if (stopped) return
+ const id = agent?.id
+ if (id === undefined || !rt.isBotSession(id)) return
+ lastActivity = Date.now()
+ if (status !== 'idle') {
+ if (!started.has(id)) started.set(id, Date.now())
+ return
+ }
+ const since = started.get(id)
+ started.delete(id)
+ const botId = rt.chatOwners.get(id)
+ if (since === undefined || botId === undefined || Date.now() - since < settings.minTurnMs) return
+ const pending = data().pending
+ pending[botId] = [...new Set([...(pending[botId] ?? []), id])]
+ noted.set(id, (noted.get(id) ?? 0) + 1)
+ void persist()
+ })
+
+ // After the text budget, keep scanning to the cursor for user corrections.
+ async function linesSince(sessionId) {
+ const cursor = data().cursors[sessionId]
+ const kept = []
+ const userLines = []
+ let used = 0
+ let userUsed = 0
+ let overBudget = false
+ let through
+ for await (const line of rt.partLines(sessionId, AbortSignal.timeout(settings.readMs), { pages: Infinity })) {
+ through ??= line.seq
+ if (cursor !== undefined && line.seq <= cursor) break
+ const text = clip(line.text, line.who === 'user' ? settings.userLineMax : settings.botLineMax)
+ if (!overBudget && used + text.length <= settings.textMax) {
+ kept.push(text)
+ used += text.length
+ continue
+ }
+ overBudget = true
+ if (text.startsWith('User: ') && userUsed + text.length <= settings.userTextMax) {
+ userLines.push(text)
+ userUsed += text.length
+ }
+ }
+ return { lines: [...kept, ...userLines].reverse(), through }
+ }
+
+ const skillsOf = () => Object.fromEntries(Object.entries(state.capabilities?.items ?? {}).filter(([, item]) => item.kind === 'skill'))
+ const learnedOf = bot => (data().learned[bot.id] ?? []).filter(name => skillsOf()[name] !== undefined && (bot.capabilities ?? []).includes(name))
+ const waitingFor = botId => Object.values(data().proposals).filter(entry => entry.botId === botId)
+
+ async function reviewSkill(bot, lines) {
+ const skills = skillsOf()
+ const learned = learnedOf(bot)
+ const library = Object.values(skills).filter(item => !learned.includes(item.name)).map(item => ({ name: item.name, description: item.description }))
+ const text = skillRequest({
+ bot,
+ lines,
+ learned: learned.map(name => skills[name]),
+ library,
+ declined: data().declined,
+ language: writtenIn(lines.filter(line => line.startsWith('User: ')).map(line => line.slice(6)), rt.uiLanguage?.()),
+ limit: settings.learnedMax,
+ })
+ const reply = await rt.botShot(bot, skillSystem(), text, { sessionId: rt.chatOf(bot.id), kind: 'memory' })
+ const proposal = parseJson(reply)
+ if (['description', 'content', 'why'].some(key => {
+ const value = proposal?.[key]
+ return typeof value === 'string' && rt.redactSecrets(value) !== value
+ })) {
+ rt.warn('ds-bot: the skill review of %s proposed content containing a saved secret', bot.name)
+ return undefined
+ }
+ const problem = skillProblem(proposal, { skills, learned, limit: settings.learnedMax })
+ if (problem !== undefined) {
+ rt.warn('ds-bot: the skill review of %s proposed nothing usable: %s', bot.name, problem)
+ return undefined
+ }
+ if (proposal.action === 'none') return undefined
+ const name = proposal.name.trim()
+ const previous = proposal.action === 'update' ? skills[name] : undefined
+ return {
+ id: randomUUID(),
+ kind: 'skill',
+ action: proposal.action,
+ botId: bot.id,
+ at: Date.now(),
+ name,
+ description: proposal.description.trim(),
+ content: proposal.content.trim(),
+ why: typeof proposal.why === 'string' ? clip(proposal.why.trim(), 300) : '',
+ ...(previous ? {
+ previous: { description: previous.description, content: previous.content },
+ base: hashOf(`${previous.description}\n${previous.content}`),
+ } : {}),
+ }
+ }
+
+ // One pass over every Bot with a long turn since the last one. A Bot with a
+ // proposal still waiting is skipped: its chat is read again in the next pass.
+ let running
+ async function run() {
++ if (stopped) return 0
+ if (running) return running
+ running = (async () => {
+ const store = data()
+ store.lastRun = Date.now()
+ let made = 0
+ for (const [botId, pendingSessions] of Object.entries(store.pending)) {
++ if (stopped) break
+ const bot = rt.botOf(botId)
+ if (bot === undefined) {
+ delete store.pending[botId]
+ continue
+ }
+ if (waitingFor(botId).length > 0) continue
+ const sessions = [...pendingSessions]
+ const notedAtStart = new Map(sessions.map(sessionId => [sessionId, noted.get(sessionId)]))
+ try {
+ const read = []
+ for (const sessionId of sessions) read.push({ sessionId, ...await linesSince(sessionId) })
++ if (stopped) return made
+ const lines = read.flatMap(entry => entry.lines)
+ const proposal = lines.length === 0 ? undefined : await reviewSkill(bot, lines)
++ if (stopped) return made
+ if (proposal !== undefined) {
+ store.proposals[proposal.id] = proposal
+ made += 1
+ }
+ for (const entry of read) if (entry.through !== undefined) store.cursors[entry.sessionId] = entry.through
+ const remaining = (store.pending[botId] ?? []).filter(sessionId =>
+ !notedAtStart.has(sessionId) || noted.get(sessionId) !== notedAtStart.get(sessionId))
+ if (remaining.length === 0) delete store.pending[botId]
+ else store.pending[botId] = remaining
+ } catch (error) {
+ rt.warn('ds-bot: the skill review of %s failed: %s', bot.name, error?.message ?? error)
+ continue
+ }
+ }
+ await persist()
+ return made
+ })().finally(() => { running = undefined })
+ return running
+ }
+
+ const due = () => {
++ if (stopped) return false
+ const store = data()
+ if (Object.keys(store.pending).length === 0 || rt.readOnly() !== undefined) return false
+ if (Object.keys(rt.workView()).length > 0 || Date.now() - lastActivity < settings.idleMs) return false
+ return store.lastRun === undefined || Date.now() - store.lastRun >= settings.intervalMs
+ }
+ ctx.effect(() => {
+ const timer = setInterval(() => {
+ if (due()) void run().catch(error => rt.warn('ds-bot: learning pass failed: %s', error?.message ?? error))
+ }, settings.tickMs)
+ timer.unref?.()
- return () => clearInterval(timer)
++ return () => {
++ stopped = true
++ clearInterval(timer)
++ // A new host may take over after disposal; finish only writes already queued.
++ return writing
++ }
+ }, 'ds-bot: learning')
+
+ // The user's answer to a card.
+ async function answer({ id, approve }) {
+ const store = data()
+ const proposal = Object.hasOwn(store.proposals, id) ? store.proposals[id] : undefined
+ if (proposal === undefined) throw Object.assign(new Error('This proposal is no longer open'), { code: 'bot/stale' })
+ if (approve !== true) {
+ delete store.proposals[id]
+ store.declined = [...store.declined, { name: proposal.name, description: proposal.description, at: Date.now() }].slice(-settings.declinedMax)
+ await persist()
+ return { applied: false }
+ }
+ // A failed approval keeps the card, so the user can still decline it.
+ await applySkill(proposal)
+ delete store.proposals[id]
+ await persist()
+ return { applied: true }
+ }
+
+ async function applySkill(proposal) {
+ const bot = rt.botOf(proposal.botId)
+ if (bot === undefined) throw new Error('The Bot that learned this skill is gone')
+ const current = skillsOf()[proposal.name]
+ const store = data()
+ if (proposal.action === 'create') {
+ if (proposal.saved) {
+ if (current === undefined || current.description !== proposal.description || current.content !== proposal.content) {
+ throw new Error(`${proposal.name} changed since this was proposed`)
+ }
+ store.learned[bot.id] = [...new Set([...(store.learned[bot.id] ?? []), current.name])]
+ await persist()
+ } else {
+ if (current !== undefined) throw new Error(`A skill named ${proposal.name} exists now`)
+ const item = await rt.capabilitySave({ name: proposal.name, kind: 'skill', description: proposal.description, content: proposal.content }, { botId: bot.id, ui: false })
+ proposal.saved = true
+ store.learned[bot.id] = [...new Set([...(store.learned[bot.id] ?? []), item.name])]
+ await persist()
+ }
+ } else if (proposal.action === 'update') {
+ if (current === undefined || hashOf(`${current.description}\n${current.content}`) !== proposal.base) throw new Error(`${proposal.name} changed since this was proposed`)
+ const item = await rt.capabilitySave({ name: proposal.name, kind: 'skill', description: proposal.description, content: proposal.content }, { botId: bot.id, ui: false })
+ store.learned[bot.id] = [...new Set([...(store.learned[bot.id] ?? []), item.name])]
+ await persist()
+ }
+ if (!(bot.capabilities ?? []).includes(proposal.name)) await rt.capabilityEndpoint('bot-capability', { botId: bot.id, name: proposal.name, attached: true })
+ }
+
+ async function learningEndpoint(endpoint, body) {
+ if (endpoint === 'learning-answer') return answer({ id: String(body.id ?? ''), approve: body.approve === true })
+ throw new Error(`Unknown endpoint ${endpoint}`)
+ }
+
+ async function removeLearned(name) {
+ const learned = data().learned
+ const deleted = String(name ?? '')
+ for (const [botId, names] of Object.entries(learned)) {
+ learned[botId] = (Array.isArray(names) ? names : []).filter(entry => entry !== deleted)
+ }
+ await persist()
+ }
+
+ // Per Bot, its proposals waiting for the user.
+ const learningView = () => {
+ const byBot = {}
+ for (const entry of Object.values(data().proposals)) (byBot[entry.botId] ??= []).push(entry)
+ return byBot
+ }
+
+ const capabilityDelete = rt.capabilityDelete
+ rt.capabilityDelete = async (...args) => {
+ const result = await capabilityDelete(...args)
+ await removeLearned(args[0])
+ return result
+ }
+ const capabilityEndpoint = rt.capabilityEndpoint
+ rt.capabilityEndpoint = async (endpoint, body, ...args) => {
+ const result = await capabilityEndpoint(endpoint, body, ...args)
+ if (endpoint === 'capability-delete') await removeLearned(body?.name)
+ return result
+ }
+
+ Object.assign(rt, { learningView, learningEndpoint, learningRun: run })
+}
diff --cc ds-bot/test/learning.test.js
index 2043ad1b,00000000..a1e3456c
mode 100644,000000..100644
--- a/ds-bot/test/learning.test.js
+++ b/ds-bot/test/learning.test.js
@@@ -1,555 -1,0 +1,576 @@@
+import assert from 'node:assert/strict'
+import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
+import { tmpdir } from 'node:os'
+import { join } from 'node:path'
+import { test } from 'node:test'
+import { createRuntime } from '../index.js'
+import { LEARNING, parseJson, skillProblem } from '../src/host/learning.js'
+import { fakeContext } from './fake-ctx.js'
+
+const TEAM = {
+ version: 1,
+ mainBotIds: ['m1'],
+ revision: 1,
+ whaleStored: true,
+ bots: {
+ m1: { id: 'm1', sessionId: 'm1', name: 'Chief', role: 'Chief of Staff', model: 'deepseek/v4', instructions: '', capabilities: [], createdAt: 1 },
+ b2: { id: 'b2', sessionId: 'b2', name: 'Writer', role: 'Writing', model: 'deepseek/v4', instructions: '', capabilities: [], createdAt: 2 },
+ },
+ rooms: {},
+ capabilities: { items: {} },
+ exchanges: [],
+ exchangeTurns: {},
+ questions: {},
+}
+
+const ownChat = [
+ { event: { seq: 10, type: 'user/message', data: { source: { kind: 'user' }, content: [{ type: 'text', text: 'Please review this work' }] } } },
+ { event: { seq: 11, type: 'assistant/message', data: { message: { content: [{ type: 'text', text: 'I finished the work.' }] } } } },
+]
+
+const createReply = JSON.stringify({
+ action: 'create',
+ name: 'release-checks',
+ description: 'Use when checking a release.',
+ content: 'Run the release checks.',
+ why: 'The repeated work showed that the checks need to be kept.',
+})
+const noneReply = JSON.stringify({ action: 'none', reason: 'This work taught nothing reusable.' })
+
+function start({ replies = [], config = {}, history = {}, seed = TEAM, learningSeed = {} } = {}) {
+ const home = mkdtempSync(join(tmpdir(), 'ds-bot-learning-'))
+ writeFileSync(join(home, 'state.json'), JSON.stringify(JSON.parse(JSON.stringify(seed))))
+ if (Object.keys(learningSeed).length > 0) {
+ mkdirSync(join(home, 'learning'), { recursive: true })
+ writeFileSync(join(home, 'learning', 'state.json'), JSON.stringify(learningSeed))
+ }
+ const ctx = fakeContext()
+ const requests = []
+ Object.assign(ctx, {
+ llm: {
+ listProviders: () => [{ id: 'deepseek', name: 'DeepSeek' }],
+ listModels: async () => [{ id: 'v4' }],
+ resolveModelInfo: async () => ({ reasoning: { efforts: [{ id: 'off' }, { id: 'high' }] } }),
+ async* stream(request) {
+ requests.push(request)
+ const reply = await replies.shift()
+ if (reply instanceof Error) throw reply
+ const out = typeof reply === 'object' && reply !== null ? reply : { text: reply ?? '' }
+ yield { type: 'text-delta', index: 0, text: out.text ?? '' }
+ yield { type: 'finish', reason: { kind: out.finish ?? 'stop', ...(out.failure ? { failure: { message: out.failure } } : {}) } }
+ },
+ },
+ sessionController: {
+ resolveAgent: async sessionId => ({
+ agent: {
+ id: sessionId,
+ session: {
+ id: sessionId,
+ seq: (history[sessionId] ?? ownChat).at(-1).event.seq + 1,
+ requestHeader: () => ({ config: { provider: 'deepseek', model: 'v4' } }),
+ },
+ },
+ }),
+ selectModel: async () => {},
+ page: async ({ address, beforeSeq, maxMessages }) => {
+ const all = (history[address.sessionId] ?? ownChat).filter(record => beforeSeq === undefined || record.event.seq < beforeSeq)
+ const records = maxMessages === undefined ? all : all.slice(-maxMessages)
+ return { records, hasMore: records.length < all.length }
+ },
+ },
+ sessions: { messageProjections: [] },
+ workspaceRegistry: { archiveSession: async () => {} },
+ })
+ let route
+ const register = ctx.connection.fetch.register
+ ctx.connection.fetch.register = options => {
+ route = options
+ return register(options)
+ }
+ const settings = {
+ memoryReviewDelayMs: 60_000,
+ ...config,
+ learningTimes: { minTurnMs: 0, idleMs: 0, intervalMs: 0, tickMs: 10, ...config.learningTimes },
+ }
+ const rt = createRuntime(ctx, { home, ...settings })
+ const call = async (endpoint, payload = {}) => {
+ const response = await route.fetch(new Request('http://localhost/api/bot', {
+ method: 'POST',
+ body: JSON.stringify({ endpoint, payload }),
+ }))
+ return response.json()
+ }
+ const learningPath = join(home, 'learning', 'state.json')
+ const learningState = () => existsSync(learningPath) ? JSON.parse(readFileSync(learningPath, 'utf8')) : {}
+ const teamState = () => JSON.parse(readFileSync(join(home, 'state.json'), 'utf8'))
+ const stop = async () => {
+ await ctx.dispose()
+ rmSync(home, { recursive: true, force: true, maxRetries: 5 })
+ }
+ return { ctx, home, call, requests, history, learningPath, learningState, teamState, rt, stop }
+}
+
+const settle = () => new Promise(resolve => setTimeout(resolve, 20))
+
+async function until(check, message = 'condition did not hold within 4 s') {
+ for (let tries = 0; tries < 200; tries += 1) {
+ if (await check()) return
+ await settle()
+ }
+ assert.fail(message)
+}
+
+async function startRuntime(options) {
+ const runtime = start(options)
+ await runtime.call('state')
+ return runtime
+}
+
+async function finishTurn(ctx, { id = 'b2', duration = 0 } = {}) {
+ const { agent } = await ctx.sessionController.resolveAgent(id)
+ ctx.emit('agent/status', { agent, status: 'running' })
+ if (duration > 0) await new Promise(resolve => setTimeout(resolve, duration))
+ ctx.emit('agent/status', { agent, status: 'idle' })
+}
+
+function addNextTurn(history) {
+ history.b2.push(
+ { event: { seq: 12, type: 'user/message', data: { source: { kind: 'user' }, content: [{ type: 'text', text: 'Please check the release process too.' }] } } },
+ { event: { seq: 13, type: 'assistant/message', data: { message: { content: [{ type: 'text', text: 'The release process is checked.' }] } } } },
+ )
+}
+
++test('a disposed learning pass cannot overwrite the next host state with a late model reply', async t => {
++ let finishModel
++ const reply = new Promise(resolve => { finishModel = resolve })
++ const runtime = await startRuntime({ replies: [reply], config: { learningTimes: { tickMs: 100_000 } } })
++ t.after(() => runtime.stop())
++ await finishTurn(runtime.ctx)
++ await until(() => runtime.learningState().pending?.b2?.includes('b2'))
++ const pass = runtime.rt.learningRun()
++ await until(() => runtime.requests.length === 1)
++ await runtime.ctx.dispose()
++
++ const next = { pending: { m1: ['m1'] }, proposals: { next: { id: 'next', botId: 'm1', name: 'next-host-skill' } } }
++ writeFileSync(runtime.learningPath, JSON.stringify(next))
++ finishModel(createReply)
++ await pass
++ assert.deepEqual(runtime.learningState(), next)
++ assert.deepEqual(runtime.rt.learningView(), {})
++ assert.equal(await runtime.rt.learningRun(), 0)
++ assert.equal(runtime.requests.length, 1)
++})
++
+test('parseJson accepts fenced and unfenced objects and ignores garbage', () => {
+ assert.deepEqual(parseJson('```json\n{"action":"none"}\n```'), { action: 'none' })
+ assert.deepEqual(parseJson('Answer: {"action":"none"}'), { action: 'none' })
+ const answer = JSON.stringify({ action: 'create', content: 'Use this:\n```js\nvalue\n```' })
+ const expected = JSON.parse(answer)
+ assert.deepEqual(parseJson(answer), expected)
+ assert.deepEqual(parseJson('```json\n' + answer + '\n```'), expected)
+ assert.equal(parseJson('No JSON was returned.'), undefined)
+ assert.equal(parseJson('```json\n{"action":\n```'), undefined)
+})
+
+test('skillProblem validates actions, learned-skill limits, and secret-looking content', () => {
+ const proposal = {
+ action: 'create',
+ name: 'release-checks',
+ description: 'Use when checking a release.',
+ content: 'Run the release checks.',
+ }
+ assert.equal(skillProblem({ action: 'none' }, { skills: {}, learned: [] }), undefined)
+ assert.match(skillProblem({ ...proposal, name: 'Bad_Name' }, { skills: {}, learned: [] }), /capability name/)
+ assert.match(skillProblem(proposal, { skills: { 'release-checks': { kind: 'skill' } }, learned: [] }), /already exists/)
+ assert.match(skillProblem(proposal, { skills: {}, learned: ['one', 'two'], limit: 2 }), /learned skills/)
+ assert.match(skillProblem({ ...proposal, action: 'update' }, { skills: { 'release-checks': { kind: 'skill', content: 'Old', description: 'Old' } }, learned: [] }), /not a learned skill/)
+ assert.match(skillProblem({ ...proposal, action: 'update' }, {
+ skills: { 'release-checks': { kind: 'skill', content: proposal.content, description: proposal.description } },
+ learned: ['release-checks'],
+ }), /changes nothing/)
+ assert.match(skillProblem({ ...proposal, content: 'The root password is hunter2' }, { skills: {}, learned: [] }), /secret/)
+ assert.match(skillProblem({ ...proposal, why: 'The root password is hunter2' }, { skills: {}, learned: [] }), /secret/)
+ assert.equal(skillProblem({ ...proposal, content: 'Compare `password == input` with a timing-safe check' }, { skills: {}, learned: [] }), undefined)
+ assert.match(skillProblem({ ...proposal, description: 'x'.repeat(LEARNING.descriptionMax + 1) }, { skills: {}, learned: [] }), /description/)
+})
+
+test('short turns do not mark a Bot chat; a longer turn marks it for review', async t => {
+ const short = await startRuntime({ config: { learningTimes: { minTurnMs: 60_000, idleMs: 60_000 } } })
+ t.after(() => short.stop())
+ await finishTurn(short.ctx)
+ await settle()
+ assert.equal(existsSync(short.learningPath), false)
+
+ const long = await startRuntime({ config: { learningTimes: { minTurnMs: 5, idleMs: 60_000 } } })
+ t.after(() => long.stop())
+ await finishTurn(long.ctx, { duration: 15 })
+ await until(() => long.learningState().pending?.b2?.includes('b2'))
+ assert.deepEqual(long.learningState().pending.b2, ['b2'])
+})
+
+test('a none reply makes no proposal and advances the chat cursor', async t => {
+ const runtime = await startRuntime({ replies: [noneReply] })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await until(() => runtime.learningState().cursors?.b2 === 11 && runtime.learningState().pending?.b2 === undefined)
+ assert.equal(runtime.requests.length, 1)
+ assert.deepEqual((await runtime.call('state')).value.learning ?? {}, {})
+})
+
+test('long turns ending during review remain pending for the next pass', async t => {
+ const history = { b2: [...ownChat], b3: [...ownChat] }
+ const runtime = await startRuntime({
+ history,
+ replies: [noneReply, noneReply],
+ config: { learningTimes: { tickMs: 100_000 } },
+ })
+ t.after(() => runtime.stop())
+ const load = runtime.rt.load
+ runtime.rt.load = async (...args) => {
+ const result = await load(...args)
+ runtime.rt.chatOwners.set('b3', 'b2')
+ return result
+ }
+ runtime.rt.chatOwners.set('b3', 'b2')
+ const b3Statuses = []
+ runtime.ctx.on('agent/status', ({ agent, status }) => {
+ if (agent?.id === 'b3') b3Statuses.push(status)
+ })
+ assert.equal(runtime.rt.chatOwners.get('b3'), 'b2')
+ assert.equal(runtime.rt.isBotSession('b3'), true)
+ await finishTurn(runtime.ctx)
+ await until(() => runtime.learningState().pending?.b2?.includes('b2'))
+
+ const botShot = runtime.rt.botShot
+ let endedDuringReview = false
+ runtime.rt.botShot = async (...args) => {
+ if (!endedDuringReview) {
+ endedDuringReview = true
+ addNextTurn(history)
+ history.b3.push(
+ { event: { seq: 12, type: 'user/message', data: { source: { kind: 'user' }, content: [{ type: 'text', text: 'Please keep this new work too.' }] } } },
+ { event: { seq: 13, type: 'assistant/message', data: { message: { content: [{ type: 'text', text: 'The second chat has new work.' }] } } } },
+ )
+ await finishTurn(runtime.ctx, { id: 'b2' })
+ await finishTurn(runtime.ctx, { id: 'b3' })
+ assert.deepEqual(b3Statuses, ['running', 'idle'])
+ }
+ return botShot(...args)
+ }
+
+ await runtime.rt.learningRun()
+
+ assert.deepEqual(runtime.learningState().pending.b2, ['b2', 'b3'])
+ assert.equal(runtime.learningState().cursors.b2, 11)
+
+ await runtime.rt.learningRun()
+
+ assert.equal(runtime.requests.length, 2)
+ const request = runtime.requests[1].messages[0].content[0].text
+ assert.match(request, /Please check the release process too\./)
+ assert.match(request, /Please keep this new work too\./)
+ assert.equal(runtime.learningState().pending.b2, undefined)
+})
+
+test('approval ownership survives a normal skill edit and remains learned on review', async t => {
+ const history = { b2: [...ownChat] }
+ const updateReply = JSON.stringify({
+ action: 'update',
+ name: 'release-checks',
+ description: 'Use when checking any release.',
+ content: 'Run the release checks.',
+ why: 'The work showed the scope should be broader.',
+ })
+ const runtime = await startRuntime({
+ history,
+ replies: [createReply, updateReply],
+ config: { learningTimes: { tickMs: 100_000 } },
+ })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await runtime.rt.learningRun()
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = (await runtime.call('state')).value.learning.b2[0]
+ assert.equal(proposal.name, 'release-checks')
+
+ addNextTurn(history)
+ await finishTurn(runtime.ctx)
+ await until(() => runtime.learningState().pending?.b2?.includes('b2'))
+ await settle()
+ assert.equal(runtime.requests.length, 1)
+ const answer = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(answer.value.applied, true)
+ await runtime.call('capability-save', {
+ name: 'release-checks',
+ kind: 'skill',
+ description: 'Use when checking a release candidate.',
+ content: 'Run the release checks.',
+ })
+ await runtime.rt.learningRun()
+ await until(() => runtime.requests.length === 2)
+ await until(() => Object.values(runtime.learningState().proposals ?? {}).some(entry => entry.action === 'update'))
+ const saved = runtime.teamState()
+ assert.equal(saved.capabilities.items['release-checks'].learned, undefined)
+ assert.ok(saved.bots.b2.capabilities.includes('release-checks'))
+ assert.deepEqual(runtime.learningState().learned.b2, ['release-checks'])
+ assert.match(runtime.requests[1].messages[0].content[0].text, /This Bot's learned skills[\s\S]*Use when checking a release candidate\./)
+})
+
+test('a failed create attachment can be retried without resaving or losing the proposal', async t => {
+ const runtime = await startRuntime({ replies: [createReply] })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = Object.values(runtime.learningState().proposals)[0]
+ const attach = runtime.rt.capabilityEndpoint
+ let fail = true
+ runtime.rt.capabilityEndpoint = async (...args) => {
+ if (fail) {
+ fail = false
+ throw new Error('temporary attachment failure')
+ }
+ return attach(...args)
+ }
+
+ const first = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(first.ok, false)
+ assert.equal(runtime.learningState().proposals[proposal.id].saved, true)
+ assert.ok(runtime.learningState().learned.b2.includes('release-checks'))
+ assert.ok(runtime.learningState().proposals[proposal.id])
+
+ const second = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(second.value.applied, true)
+ assert.equal(runtime.learningState().proposals[proposal.id], undefined)
+ assert.ok(runtime.teamState().bots.b2.capabilities.includes('release-checks'))
+})
+
+test('a recreated approved skill is learned when its saved proposal is retried', async t => {
+ const runtime = await startRuntime({ replies: [createReply] })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = Object.values(runtime.learningState().proposals)[0]
+ const endpoint = runtime.rt.capabilityEndpoint
+ let fail = true
+ runtime.rt.capabilityEndpoint = async (name, body, ...args) => {
+ if (name === 'bot-capability' && fail) {
+ fail = false
+ throw new Error('temporary attachment failure')
+ }
+ return endpoint(name, body, ...args)
+ }
+
+ const first = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(first.ok, false)
+ assert.equal(runtime.learningState().proposals[proposal.id].saved, true)
+ assert.deepEqual(runtime.learningState().learned.b2, ['release-checks'])
+
+ const deleted = await runtime.call('capability-delete', { name: 'release-checks' })
+ assert.equal(deleted.ok, true)
+ assert.deepEqual(runtime.learningState().learned.b2, [])
+ await runtime.call('capability-save', {
+ name: 'release-checks',
+ kind: 'skill',
+ description: 'Use when checking a release.',
+ content: 'Run the release checks.',
+ })
+
+ const retry = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(retry.value.applied, true)
+ assert.equal(runtime.learningState().proposals[proposal.id], undefined)
+ assert.deepEqual(runtime.learningState().learned.b2, ['release-checks'])
+ assert.ok(runtime.teamState().bots.b2.capabilities.includes('release-checks'))
+})
+
+test('a failed create retry refuses a skill edited after it was saved', async t => {
+ const runtime = await startRuntime({ replies: [createReply] })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = Object.values(runtime.learningState().proposals)[0]
+ const endpoint = runtime.rt.capabilityEndpoint
+ let fail = true
+ runtime.rt.capabilityEndpoint = async (name, body, ...args) => {
+ if (name === 'bot-capability' && fail) {
+ fail = false
+ throw new Error('temporary attachment failure')
+ }
+ return endpoint(name, body, ...args)
+ }
+
+ const first = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(first.ok, false)
+ assert.equal(runtime.learningState().proposals[proposal.id].saved, true)
+ const edit = await runtime.call('capability-save', {
+ name: 'release-checks',
+ kind: 'skill',
+ description: 'Use when checking a release.',
+ content: 'Run the revised release checks.',
+ })
+ assert.equal(edit.ok, true)
+
+ const retry = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(retry.ok, false)
+ assert.match(JSON.stringify(retry), /release-checks changed since this was proposed/)
+ assert.ok(runtime.learningState().proposals[proposal.id])
+ assert.equal(runtime.teamState().capabilities.items['release-checks'].content, 'Run the revised release checks.')
+ assert.equal(runtime.teamState().bots.b2.capabilities.includes('release-checks'), false)
+})
+
+test('deleting a learned skill clears ownership before a same-name skill is recreated', async t => {
+ const history = { b2: [...ownChat] }
+ const runtime = await startRuntime({
+ history,
+ replies: [createReply, noneReply],
+ config: { learningTimes: { tickMs: 100_000 } },
+ })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await runtime.rt.learningRun()
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = Object.values(runtime.learningState().proposals)[0]
+ const answer = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(answer.value.applied, true)
+ assert.deepEqual(runtime.learningState().learned.b2, ['release-checks'])
+
+ const deleted = await runtime.call('capability-delete', { name: 'release-checks' })
+ assert.equal(deleted.ok, true)
+ assert.deepEqual(runtime.learningState().learned.b2, [])
+ await runtime.call('capability-save', {
+ name: 'release-checks',
+ kind: 'skill',
+ description: 'Use when checking a release.',
+ content: 'Run the release checks.',
+ })
+ await runtime.call('bot-capability', { botId: 'b2', name: 'release-checks', attached: true })
+ assert.ok(runtime.teamState().bots.b2.capabilities.includes('release-checks'))
+
+ addNextTurn(history)
+ await finishTurn(runtime.ctx)
+ await runtime.rt.learningRun()
+ assert.equal(runtime.requests.length, 2)
+ const request = runtime.requests[1].messages[0].content[0].text
+ assert.match(request, /This Bot has no learned skills yet\./)
+ assert.match(request, /Other skills on the team:\n- release-checks:/)
+ assert.deepEqual(runtime.learningState().learned.b2, [])
+})
+
+test('declined proposals are recorded and included in the next skill request', async t => {
+ const history = { b2: [...ownChat] }
+ const runtime = await startRuntime({ history, replies: [createReply, noneReply] })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = (await runtime.call('state')).value.learning.b2[0]
+
+ addNextTurn(history)
+ await finishTurn(runtime.ctx)
+ await until(() => runtime.learningState().pending?.b2?.includes('b2'))
+ await settle()
+ assert.equal(runtime.requests.length, 1)
+
+ const answer = await runtime.call('learning-answer', { id: proposal.id, approve: false })
+ assert.equal(answer.value.applied, false)
+ await until(() => runtime.learningState().cursors?.b2 === 13 && runtime.learningState().pending?.b2 === undefined)
+ assert.equal(runtime.teamState().capabilities.items['release-checks'], undefined)
+ assert.equal(runtime.learningState().declined[0].name, 'release-checks')
+ assert.match(runtime.requests[1].messages[0].content[0].text, /Proposals the user declined[\s\S]*- release-checks: Use when checking a release\./)
+})
+
+test('an update approval fails if the learned skill description alone changed', async t => {
+ const seed = JSON.parse(JSON.stringify(TEAM))
+ seed.bots.b2.capabilities = ['release-checks']
+ seed.capabilities.items['release-checks'] = {
+ name: 'release-checks',
+ kind: 'skill',
+ description: 'Use when checking a release.',
+ content: 'Run the original release checks.',
+ createdBy: 'b2',
+ createdAt: 3,
+ updatedAt: 3,
+ learned: true,
+ }
+ const updateReply = JSON.stringify({
+ action: 'update',
+ name: 'release-checks',
+ description: 'Use when checking a release.',
+ content: 'Run the new release checks.',
+ why: 'The work showed an additional step.',
+ })
+ const runtime = await startRuntime({ seed, replies: [updateReply], learningSeed: { learned: { b2: ['release-checks'] } } })
+ t.after(() => runtime.stop())
+ await finishTurn(runtime.ctx)
+ await until(() => Object.keys(runtime.learningState().proposals ?? {}).length === 1)
+ const proposal = (await runtime.call('state')).value.learning.b2[0]
+
+ await runtime.call('capability-save', {
+ name: 'release-checks',
+ kind: 'skill',
+ description: 'Use when checking a release candidate.',
+ content: 'Run the original release checks.',
+ })
+ const answer = await runtime.call('learning-answer', { id: proposal.id, approve: true })
+ assert.equal(answer.ok, false)
+ assert.match(JSON.stringify(answer), /release-checks changed since this was proposed/)
+ assert.equal(runtime.teamState().capabilities.items['release-checks'].description, 'Use when checking a release candidate.')
+ assert.equal(Object.keys(runtime.learningState().proposals).length, 1)
+ const declined = await runtime.call('learning-answer', { id: proposal.id, approve: false })
+ assert.equal(declined.ok, true)
+ assert.equal(Object.keys(runtime.learningState().proposals).length, 0)
+})
+
+test('a saved secret in a proposed skill is not shown as a proposal', async t => {
+ const secretValue = 'saved-secret-value-37'
+ const runtime = await startRuntime({
+ replies: [JSON.stringify({
+ action: 'create',
+ name: 'release-checks',
+ description: 'Use when checking a release.',
+ content: `Run with ${secretValue}.`,
+ why: 'The work showed it is needed.',
+ })],
+ })
+ t.after(() => runtime.stop())
+ await runtime.call('secret-set', { name: 'LEARNING_TEST', value: secretValue })
+ await finishTurn(runtime.ctx)
+ await until(() => runtime.learningState().cursors?.b2 === 11)
+ assert.deepEqual(runtime.learningState().proposals, {})
+})
+
+test('user corrections remain in long chat reviews after the text budget', async t => {
+ const runtime = await startRuntime({ replies: [noneReply], config: { learningTimes: { tickMs: 100_000 } } })
+ t.after(() => runtime.stop())
+ const lines = Array.from({ length: 30 }, (_, index) => ({
+ seq: 41 - index,
+ who: 'assistant',
+ text: `Bot: ${'x'.repeat(1_200)}`,
+ }))
+ lines.push({ seq: 10, who: 'user', text: 'User: Keep the correction even when the Bot writes a long answer.' })
+ runtime.rt.partLines = async function* () {
+ yield* lines
+ }
+ await finishTurn(runtime.ctx)
+ await runtime.rt.learningRun()
+ assert.match(runtime.requests[0].messages[0].content[0].text, /User: Keep the correction even when the Bot writes a long answer\./)
+})
+
+test('a failed chat read keeps that Bot pending while other Bots are reviewed', async t => {
+ const seed = JSON.parse(JSON.stringify(TEAM))
+ seed.bots.b3 = { id: 'b3', sessionId: 'b3', name: 'Coder', role: 'Coding', model: 'deepseek/v4', instructions: '', capabilities: [], createdAt: 3 }
+ const runtime = await startRuntime({ seed, replies: [noneReply], config: { learningTimes: { tickMs: 100_000 } } })
+ t.after(() => runtime.stop())
+ runtime.rt.partLines = async function* (sessionId) {
+ if (sessionId === 'b2') throw new Error('session read failed')
+ yield { seq: 11, who: 'user', text: 'User: Review this work' }
+ }
+ await finishTurn(runtime.ctx, { id: 'b2' })
+ await finishTurn(runtime.ctx, { id: 'b3' })
+ await until(() => runtime.learningState().pending?.b2?.length && runtime.learningState().pending?.b3?.length)
+
+ await runtime.rt.learningRun()
+
+ assert.deepEqual(runtime.learningState().pending.b2, ['b2'])
+ assert.equal(runtime.learningState().pending.b3, undefined)
+ assert.equal(runtime.learningState().cursors.b3, 11)
+ assert.equal(runtime.requests.length, 1)
+})