Self-hosted project management for devs and their clients: tickets, time tracking, docs and meetings, with an AI assistant that cites its sources. AGPL-3.0.
Ruby agplai-assistantdockerdsgvogdprhotwireissue-trackerjira-alternativeknowledge-basellmmcp-servermeeting-notesopen-sourceproject-managementredmine-alternativeruby-on-railsself-hostedticketingtime-tracking
Fix review round 1 of #83: refuse a NUL, skip rows beyond the total, pin the order
Guido Boyke78cd89e4594e
diff --git a/CLAUDE.md b/CLAUDE.mdindex ebd5d0a..7e27ed4 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -864,9 +864,10 @@ Rows only through the per-type policy scope: no title and no count of a type theread; a reader who may open the project but read no knowledge gets an empty list. Without `q`(or blank) REST lists every readable row; MCP keeps `q` required. `type` other than `note` or`document`, and `page`/`per_page` that are no whole number from 1 to the maximum (`per_page` at-most 100, where MCP clamps) answer 422 `invalid_input`, checked before the project. The project-gate is MCP's (token reach and `ProjectAccessService`): missing, malformed, beyond the reach,-not openable (an archived project for a non-admin included) and foreign answer the same 404.+most 100, where MCP clamps) and a NUL in `q` or `type` answer 422 `invalid_input`, checked+before the project; an absent or empty `page`/`per_page` is the default. A page beyond the total+reads no rows. The project gate is MCP's (token reach and `ProjectAccessService`): missing,+malformed, beyond the reach, not openable (an archived project for a non-admin included) and foreign answer the same 404.Guards: `spec/requests/api/v1/knowledge_spec.rb`, `spec/services/knowledge/lookup_spec.rb`.`GET /api/v1/organisations/:organisation_id/people` and `.../people/:id` (People S3b) readdiff --git a/app/controllers/api/v1/knowledge_controller.rb b/app/controllers/api/v1/knowledge_controller.rbindex 9e51226..b252fd5 100644--- a/app/controllers/api/v1/knowledge_controller.rb+++ b/app/controllers/api/v1/knowledge_controller.rb@@ -10,8 +10,9 @@ module Api# knowledge area without a search term. MCP keeps `q` required.# * Paging: a page or per_page that is no whole number from 1 to the maximum is refused# with 422, where MCP clamps: a program would get fewer rows than it asked for without- # being told. Absent or blank means the default. Checked before the project is looked up,- # so the 422 does not depend on whether the project exists.+ # being told. An absent or empty value means the default; a NUL in q or type is refused.+ # Checked before the project is looked up, so the 422 does not depend on whether the+ # project exists.# * The project gate is the one of MCP (Mcp::ToolCall#authorized_project): the token's reach# and the owner's access to the project. A missing project, a malformed id, a project# beyond the reach, one the owner cannot open and one of another organisation answer thediff --git a/app/services/knowledge/lookup.rb b/app/services/knowledge/lookup.rbindex 4cc8a56..0e12ad1 100644--- a/app/services/knowledge/lookup.rb+++ b/app/services/knowledge/lookup.rb@@ -42,7 +42,7 @@ module Knowledge# Reads the options of a lookup. An unknown type is refused on both surfaces. Page and# per_page differ by surface: MCP has always read garbage as the default and clamped a# large per_page (`strict: false`); REST refuses what it cannot read (`strict: true`),- # where absent or blank still means the default.+ # where an absent or empty value still means the default.## @param args [Hash] symbol keys q, type, page, per_page# @raise [InvalidOption] with a message naming the argument@@ -76,12 +76,18 @@ module Knowledge# @param options [Options] blank query = every readable row# @return [Result]def page(options)- Result.new(records: page_records(options), page: options.page, per_page: options.per_page,- total: total(options))+ total = total(options)+ Result.new(records: page_records(options, total), page: options.page, per_page: options.per_page,+ total: total)end+ # A NUL is refused before strip, which would drop one at the edges: inside the text it+ # reaches the database adapter, which raises (a 500 instead of a 422).def self.text_arg(value, name, strict)- raise InvalidOption, "#{name} must be text" if strict && !value.nil? && !value.is_a?(String)+ if strict && !value.nil?+ raise InvalidOption, "#{name} must be text" unless value.is_a?(String)+ raise InvalidOption, "#{name} must not contain NUL" if value.include?("\0")+ endvalue.to_s.stripend@@ -140,7 +146,13 @@ module Knowledge# MATCHES the Ruby merge below, so rows with an identical updated_at are cut by the LIMIT# in the order the merge expects; without it a tie across a page boundary could drop or# repeat a row.- def page_records(options)+ #+ # A page beyond the total reads no rows: each arm's LIMIT grows with the page, and a large+ # page would otherwise load the whole readable knowledge of the project to slice off nothing.+ def page_records(options, total)+ offset = (options.page - 1) * options.per_page+ return [] if offset >= total+limit = options.page * options.per_pagerecords = type_keys(options).flat_map do |key|scope = arm(key, options.query)@@ -148,7 +160,7 @@ module Knowledgescope.reorder(table[:updated_at].desc, table[:id].asc).limit(limit).to_aendpage_records = records.sort_by { |record| [-record.updated_at.to_f, record.id] }- .slice((options.page - 1) * options.per_page, options.per_page) || []+ .slice(offset, options.per_page) || []preload_titles(page_records)page_recordsenddiff --git a/spec/requests/api/v1/knowledge_spec.rb b/spec/requests/api/v1/knowledge_spec.rbindex e815b75..2a47e82 100644--- a/spec/requests/api/v1/knowledge_spec.rb+++ b/spec/requests/api/v1/knowledge_spec.rb@@ -33,20 +33,20 @@ RSpec.describe 'REST knowledge', :ci_guard, type: :request dolet(:reader) { member }- def note(title, at:)+ def note(title, at:, id: nil)Prosopite.pause doActsAsTenant.with_tenant(organisation) do- record = create(:note, project: project, organisation: organisation, title: title)+ record = create(:note, { project: project, organisation: organisation, title: title, id: id }.compact)record.update_columns(updated_at: at)recordendendend- def document(title, at:)+ def document(title, at:, id: nil)Prosopite.pause doActsAsTenant.with_tenant(organisation) do- doc = create(:document, project: project, organisation: organisation)+ doc = create(:document, { project: project, organisation: organisation, id: id }.compact)version = create(:document_version, document: doc, organisation: organisation, title: title)doc.update!(current_version: version)doc.update_columns(updated_at: at)@@ -137,16 +137,24 @@ RSpec.describe 'REST knowledge', :ci_guard, type: :request doexpect(get_json(reader, q: ' ')['meta']['total']).to eq(3)end+ # The tied rows are the newest, so they fill the first pages. Fixed ids interleave the types+ # (d1 n2 d3 n4 n5) and the notes are inserted against that order: without the id tiebreak+ # in SQL or in the merge, page 1 is no longer d1, n2.it 'pages across both types without repeating or losing a row of equal updated_at' do- at = 5.days.ago.change(usec: 0)- tied = [note('Tie 1', at: at), document('Tie 2', at: at), note('Tie 3', at: at)]+ at = 1.hour.ago.change(usec: 0)+ ids = (1..5).index_with { |n| format('00000000-0000-4000-8000-%012d', n) }+ note('n5', at: at, id: ids[5])+ note('n4', at: at, id: ids[4])+ note('n2', at: at, id: ids[2])+ document('d3', at: at, id: ids[3])+ document('d1', at: at, id: ids[1])token = token_for(reader)- second = get_json(reader, { page: 2, per_page: 2 }, token: token)['data'].pluck('id')- third = get_json(reader, { page: 3, per_page: 2 }, token: token)['data'].pluck('id')+ first = get_json(reader, { page: 1, per_page: 2 }, token: token)['data'].pluck('title')+ second = get_json(reader, { page: 2, per_page: 2 }, token: token)['data'].pluck('title')+ third = get_json(reader, { page: 3, per_page: 2 }, token: token)['data'].pluck('title')- # Pages 1 holds minutes and the checklist; the runbook, then the three ties in id order.- expect(second + third).to eq([runbook.id] + tied.map(&:id).sort)+ expect([first, second, third]).to eq([%w[d1 n2], %w[d3 n4], ['n5', minutes.title]])endend@@ -265,6 +273,11 @@ RSpec.describe 'REST knowledge', :ci_guard, type: :request doexpect_invalid_input(/per_page/)end+ it 'refuses a NUL in q, which would otherwise reach the database' do+ get "/api/v1/projects/#{project.id}/knowledge?q=a%00b", headers: { 'Authorization' => "Bearer #{token_for(reader)}" }+ expect_invalid_input(/q/)+ end+it 'refuses a q given as a list' doget "/api/v1/projects/#{project.id}/knowledge?q[]=a", headers: { 'Authorization' => "Bearer #{token_for(reader)}" }expect_invalid_input(/q/)diff --git a/spec/services/knowledge/lookup_spec.rb b/spec/services/knowledge/lookup_spec.rbindex 7ab2b50..05d8404 100644--- a/spec/services/knowledge/lookup_spec.rb+++ b/spec/services/knowledge/lookup_spec.rb@@ -34,14 +34,14 @@ RSpec.describe Knowledge::Lookup, :ci_guard docreate(:project_permission, { user: reader, project: project }.merge(areas))end- def note(title, at: nil)- record = create(:note, project: project, organisation: organisation, title: title)+ def note(title, at: nil, id: nil)+ record = create(:note, { project: project, organisation: organisation, title: title, id: id }.compact)record.update_columns(updated_at: at) if atrecordend- def document(title, at: nil)- doc = create(:document, project: project, organisation: organisation)+ def document(title, at: nil, id: nil)+ doc = create(:document, { project: project, organisation: organisation, id: id }.compact)version = create(:document_version, document: doc, organisation: organisation, title: title)doc.update!(current_version: version)doc.update_columns(updated_at: at) if at@@ -100,6 +100,18 @@ RSpec.describe Knowledge::Lookup, :ci_guard doendend+ it 'caps page at 100,000 (it bounds the LIMIT of each arm)' do+ expect(described_class::MAX_PAGE).to eq(100_000)+ expect(strict(page: '100000').page).to eq(100_000)+ end++ it 'refuses a NUL in q or type, also at the edges where strip would drop it' do+ ["a\0b", "\0", "a\0"].each do |value|+ expect { strict(q: value) }.to raise_error(described_class::InvalidOption, /q/), "q #{value.inspect}"+ end+ expect { strict(type: "note\0") }.to raise_error(described_class::InvalidOption, /type/)+ end+it 'refuses a page beyond MAX_PAGE' doexpect { strict(page: (described_class::MAX_PAGE + 1).to_s) }.to raise_error(described_class::InvalidOption, /page/)end@@ -157,23 +169,64 @@ RSpec.describe Knowledge::Lookup, :ci_guard doexpect(titles(lookup.page(options(q: 'back\\slash')))).to eq(['back\\slash'])end- # Five rows share one updated_at and straddle the page boundary of a merge over two- # tables. Walking the pages must hand out every row exactly once, in id order.+ # Five rows share one updated_at across both tables, with fixed ids so that the id order+ # interleaves the types (d1 n2 d3 n4 n5) and the notes are inserted against it. A per-arm+ # LIMIT without the SQL id tiebreak returns n5, n4 first; a Ruby merge without the id+ # tiebreak puts the note arm before the document arm. Either way page 1 is no longer d1, n2.it 'neither repeats nor loses a row at a page boundary with equal updated_at' doat = 1.day.ago.change(usec: 0)- # Setup only: five records from one line read as an N+1 to Prosopite.- rows = Prosopite.pause do- [note('t1', at: at), document('t2', at: at), note('t3', at: at), document('t4', at: at), note('t5', at: at)]+ ids = (1..5).index_with { |n| format('00000000-0000-4000-8000-%012d', n) }+ # Setup only: several records from one block read as an N+1 to Prosopite.+ Prosopite.pause do+ note('n5', at: at, id: ids[5])+ note('n4', at: at, id: ids[4])+ note('n2', at: at, id: ids[2])+ document('d3', at: at, id: ids[3])+ document('d1', at: at, id: ids[1])endfirst = lookup.page(options(page: 1, per_page: 2))second = lookup.page(options(page: 2, per_page: 2))third = lookup.page(options(page: 3, per_page: 2))- expect((first.records + second.records + third.records).map(&:id)).to eq(rows.map(&:id).sort)+ expect(titles(first)).to eq(%w[d1 n2])+ expect(titles(second)).to eq(%w[d3 n4])+ expect(titles(third)).to eq(%w[n5])expect(third.total).to eq(5)end+ # Rows one microsecond apart in one second: SQL orders them by the microseconds, so the+ # merge has to as well. Ids run against the time order, so a merge that compared whole+ # seconds and fell back to the id would turn the order round.+ it 'orders rows within one second by their microseconds' do+ base = 1.day.ago.change(usec: 0)+ Prosopite.pause do+ note('oldest', at: base + 0.000001, id: '00000000-0000-4000-8000-000000000001')+ note('middle', at: base + 0.000002, id: '00000000-0000-4000-8000-000000000002')+ document('newest', at: base + 0.000003, id: '00000000-0000-4000-8000-000000000003')+ end++ first = lookup.page(options(page: 1, per_page: 1))+ second = lookup.page(options(page: 2, per_page: 1))+ third = lookup.page(options(page: 3, per_page: 1))++ expect(titles(first) + titles(second) + titles(third)).to eq(%w[newest middle oldest])+ end++ it 'reads no rows for a page beyond the total, only the counts' do+ note('Only')+ statements = []+ callback = ->(*, payload) { statements << payload[:sql] }++ result = ActiveSupport::Notifications.subscribed(callback, 'sql.active_record') do+ lookup.page(options(page: 50, per_page: 10))+ end++ expect(result.records).to eq([])+ expect(result.total).to eq(1)+ expect(statements.grep(/\ASELECT "(notes|documents)"\.\*/)).to be_empty+ end+it 'returns an empty page beyond the last one, with the full total' donote('Only')