Skip to content

Self-hosted project management for devs and their clients: tickets, time tracking, docs and meetings, with an AI assistant that cites its sources. AGPL-3.0.

Fix review round 1 of #83: refuse a NUL, skip rows beyond the total, pin the order

Guido Boyke78cd89e4594e
diff --git a/CLAUDE.md b/CLAUDE.md
index ebd5d0a..7e27ed4 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -864,9 +864,10 @@ Rows only through the per-type policy scope: no title and no count of a type the
read; a reader who may open the project but read no knowledge gets an empty list. Without `q`
(or blank) REST lists every readable row; MCP keeps `q` required. `type` other than `note` or
`document`, and `page`/`per_page` that are no whole number from 1 to the maximum (`per_page` at
-most 100, where MCP clamps) answer 422 `invalid_input`, checked before the project. The project
-gate is MCP's (token reach and `ProjectAccessService`): missing, malformed, beyond the reach,
-not openable (an archived project for a non-admin included) and foreign answer the same 404.
+most 100, where MCP clamps) and a NUL in `q` or `type` answer 422 `invalid_input`, checked
+before the project; an absent or empty `page`/`per_page` is the default. A page beyond the total
+reads no rows. The project gate is MCP's (token reach and `ProjectAccessService`): missing,
+malformed, beyond the reach, not openable (an archived project for a non-admin included) and foreign answer the same 404.
Guards: `spec/requests/api/v1/knowledge_spec.rb`, `spec/services/knowledge/lookup_spec.rb`.
`GET /api/v1/organisations/:organisation_id/people` and `.../people/:id` (People S3b) read
diff --git a/app/controllers/api/v1/knowledge_controller.rb b/app/controllers/api/v1/knowledge_controller.rb
index 9e51226..b252fd5 100644
--- a/app/controllers/api/v1/knowledge_controller.rb
+++ b/app/controllers/api/v1/knowledge_controller.rb
@@ -10,8 +10,9 @@ module Api
# knowledge area without a search term. MCP keeps `q` required.
# * Paging: a page or per_page that is no whole number from 1 to the maximum is refused
# with 422, where MCP clamps: a program would get fewer rows than it asked for without
- # being told. Absent or blank means the default. Checked before the project is looked up,
- # so the 422 does not depend on whether the project exists.
+ # being told. An absent or empty value means the default; a NUL in q or type is refused.
+ # Checked before the project is looked up, so the 422 does not depend on whether the
+ # project exists.
# * The project gate is the one of MCP (Mcp::ToolCall#authorized_project): the token's reach
# and the owner's access to the project. A missing project, a malformed id, a project
# beyond the reach, one the owner cannot open and one of another organisation answer the
diff --git a/app/services/knowledge/lookup.rb b/app/services/knowledge/lookup.rb
index 4cc8a56..0e12ad1 100644
--- a/app/services/knowledge/lookup.rb
+++ b/app/services/knowledge/lookup.rb
@@ -42,7 +42,7 @@ module Knowledge
# Reads the options of a lookup. An unknown type is refused on both surfaces. Page and
# per_page differ by surface: MCP has always read garbage as the default and clamped a
# large per_page (`strict: false`); REST refuses what it cannot read (`strict: true`),
- # where absent or blank still means the default.
+ # where an absent or empty value still means the default.
#
# @param args [Hash] symbol keys q, type, page, per_page
# @raise [InvalidOption] with a message naming the argument
@@ -76,12 +76,18 @@ module Knowledge
# @param options [Options] blank query = every readable row
# @return [Result]
def page(options)
- Result.new(records: page_records(options), page: options.page, per_page: options.per_page,
- total: total(options))
+ total = total(options)
+ Result.new(records: page_records(options, total), page: options.page, per_page: options.per_page,
+ total: total)
end
+ # A NUL is refused before strip, which would drop one at the edges: inside the text it
+ # reaches the database adapter, which raises (a 500 instead of a 422).
def self.text_arg(value, name, strict)
- raise InvalidOption, "#{name} must be text" if strict && !value.nil? && !value.is_a?(String)
+ if strict && !value.nil?
+ raise InvalidOption, "#{name} must be text" unless value.is_a?(String)
+ raise InvalidOption, "#{name} must not contain NUL" if value.include?("\0")
+ end
value.to_s.strip
end
@@ -140,7 +146,13 @@ module Knowledge
# MATCHES the Ruby merge below, so rows with an identical updated_at are cut by the LIMIT
# in the order the merge expects; without it a tie across a page boundary could drop or
# repeat a row.
- def page_records(options)
+ #
+ # A page beyond the total reads no rows: each arm's LIMIT grows with the page, and a large
+ # page would otherwise load the whole readable knowledge of the project to slice off nothing.
+ def page_records(options, total)
+ offset = (options.page - 1) * options.per_page
+ return [] if offset >= total
+
limit = options.page * options.per_page
records = type_keys(options).flat_map do |key|
scope = arm(key, options.query)
@@ -148,7 +160,7 @@ module Knowledge
scope.reorder(table[:updated_at].desc, table[:id].asc).limit(limit).to_a
end
page_records = records.sort_by { |record| [-record.updated_at.to_f, record.id] }
- .slice((options.page - 1) * options.per_page, options.per_page) || []
+ .slice(offset, options.per_page) || []
preload_titles(page_records)
page_records
end
diff --git a/spec/requests/api/v1/knowledge_spec.rb b/spec/requests/api/v1/knowledge_spec.rb
index e815b75..2a47e82 100644
--- a/spec/requests/api/v1/knowledge_spec.rb
+++ b/spec/requests/api/v1/knowledge_spec.rb
@@ -33,20 +33,20 @@ RSpec.describe 'REST knowledge', :ci_guard, type: :request do
let(:reader) { member }
- def note(title, at:)
+ def note(title, at:, id: nil)
Prosopite.pause do
ActsAsTenant.with_tenant(organisation) do
- record = create(:note, project: project, organisation: organisation, title: title)
+ record = create(:note, { project: project, organisation: organisation, title: title, id: id }.compact)
record.update_columns(updated_at: at)
record
end
end
end
- def document(title, at:)
+ def document(title, at:, id: nil)
Prosopite.pause do
ActsAsTenant.with_tenant(organisation) do
- doc = create(:document, project: project, organisation: organisation)
+ doc = create(:document, { project: project, organisation: organisation, id: id }.compact)
version = create(:document_version, document: doc, organisation: organisation, title: title)
doc.update!(current_version: version)
doc.update_columns(updated_at: at)
@@ -137,16 +137,24 @@ RSpec.describe 'REST knowledge', :ci_guard, type: :request do
expect(get_json(reader, q: ' ')['meta']['total']).to eq(3)
end
+ # The tied rows are the newest, so they fill the first pages. Fixed ids interleave the types
+ # (d1 n2 d3 n4 n5) and the notes are inserted against that order: without the id tiebreak
+ # in SQL or in the merge, page 1 is no longer d1, n2.
it 'pages across both types without repeating or losing a row of equal updated_at' do
- at = 5.days.ago.change(usec: 0)
- tied = [note('Tie 1', at: at), document('Tie 2', at: at), note('Tie 3', at: at)]
+ at = 1.hour.ago.change(usec: 0)
+ ids = (1..5).index_with { |n| format('00000000-0000-4000-8000-%012d', n) }
+ note('n5', at: at, id: ids[5])
+ note('n4', at: at, id: ids[4])
+ note('n2', at: at, id: ids[2])
+ document('d3', at: at, id: ids[3])
+ document('d1', at: at, id: ids[1])
token = token_for(reader)
- second = get_json(reader, { page: 2, per_page: 2 }, token: token)['data'].pluck('id')
- third = get_json(reader, { page: 3, per_page: 2 }, token: token)['data'].pluck('id')
+ first = get_json(reader, { page: 1, per_page: 2 }, token: token)['data'].pluck('title')
+ second = get_json(reader, { page: 2, per_page: 2 }, token: token)['data'].pluck('title')
+ third = get_json(reader, { page: 3, per_page: 2 }, token: token)['data'].pluck('title')
- # Pages 1 holds minutes and the checklist; the runbook, then the three ties in id order.
- expect(second + third).to eq([runbook.id] + tied.map(&:id).sort)
+ expect([first, second, third]).to eq([%w[d1 n2], %w[d3 n4], ['n5', minutes.title]])
end
end
@@ -265,6 +273,11 @@ RSpec.describe 'REST knowledge', :ci_guard, type: :request do
expect_invalid_input(/per_page/)
end
+ it 'refuses a NUL in q, which would otherwise reach the database' do
+ get "/api/v1/projects/#{project.id}/knowledge?q=a%00b", headers: { 'Authorization' => "Bearer #{token_for(reader)}" }
+ expect_invalid_input(/q/)
+ end
+
it 'refuses a q given as a list' do
get "/api/v1/projects/#{project.id}/knowledge?q[]=a", headers: { 'Authorization' => "Bearer #{token_for(reader)}" }
expect_invalid_input(/q/)
diff --git a/spec/services/knowledge/lookup_spec.rb b/spec/services/knowledge/lookup_spec.rb
index 7ab2b50..05d8404 100644
--- a/spec/services/knowledge/lookup_spec.rb
+++ b/spec/services/knowledge/lookup_spec.rb
@@ -34,14 +34,14 @@ RSpec.describe Knowledge::Lookup, :ci_guard do
create(:project_permission, { user: reader, project: project }.merge(areas))
end
- def note(title, at: nil)
- record = create(:note, project: project, organisation: organisation, title: title)
+ def note(title, at: nil, id: nil)
+ record = create(:note, { project: project, organisation: organisation, title: title, id: id }.compact)
record.update_columns(updated_at: at) if at
record
end
- def document(title, at: nil)
- doc = create(:document, project: project, organisation: organisation)
+ def document(title, at: nil, id: nil)
+ doc = create(:document, { project: project, organisation: organisation, id: id }.compact)
version = create(:document_version, document: doc, organisation: organisation, title: title)
doc.update!(current_version: version)
doc.update_columns(updated_at: at) if at
@@ -100,6 +100,18 @@ RSpec.describe Knowledge::Lookup, :ci_guard do
end
end
+ it 'caps page at 100,000 (it bounds the LIMIT of each arm)' do
+ expect(described_class::MAX_PAGE).to eq(100_000)
+ expect(strict(page: '100000').page).to eq(100_000)
+ end
+
+ it 'refuses a NUL in q or type, also at the edges where strip would drop it' do
+ ["a\0b", "\0", "a\0"].each do |value|
+ expect { strict(q: value) }.to raise_error(described_class::InvalidOption, /q/), "q #{value.inspect}"
+ end
+ expect { strict(type: "note\0") }.to raise_error(described_class::InvalidOption, /type/)
+ end
+
it 'refuses a page beyond MAX_PAGE' do
expect { strict(page: (described_class::MAX_PAGE + 1).to_s) }.to raise_error(described_class::InvalidOption, /page/)
end
@@ -157,23 +169,64 @@ RSpec.describe Knowledge::Lookup, :ci_guard do
expect(titles(lookup.page(options(q: 'back\\slash')))).to eq(['back\\slash'])
end
- # Five rows share one updated_at and straddle the page boundary of a merge over two
- # tables. Walking the pages must hand out every row exactly once, in id order.
+ # Five rows share one updated_at across both tables, with fixed ids so that the id order
+ # interleaves the types (d1 n2 d3 n4 n5) and the notes are inserted against it. A per-arm
+ # LIMIT without the SQL id tiebreak returns n5, n4 first; a Ruby merge without the id
+ # tiebreak puts the note arm before the document arm. Either way page 1 is no longer d1, n2.
it 'neither repeats nor loses a row at a page boundary with equal updated_at' do
at = 1.day.ago.change(usec: 0)
- # Setup only: five records from one line read as an N+1 to Prosopite.
- rows = Prosopite.pause do
- [note('t1', at: at), document('t2', at: at), note('t3', at: at), document('t4', at: at), note('t5', at: at)]
+ ids = (1..5).index_with { |n| format('00000000-0000-4000-8000-%012d', n) }
+ # Setup only: several records from one block read as an N+1 to Prosopite.
+ Prosopite.pause do
+ note('n5', at: at, id: ids[5])
+ note('n4', at: at, id: ids[4])
+ note('n2', at: at, id: ids[2])
+ document('d3', at: at, id: ids[3])
+ document('d1', at: at, id: ids[1])
end
first = lookup.page(options(page: 1, per_page: 2))
second = lookup.page(options(page: 2, per_page: 2))
third = lookup.page(options(page: 3, per_page: 2))
- expect((first.records + second.records + third.records).map(&:id)).to eq(rows.map(&:id).sort)
+ expect(titles(first)).to eq(%w[d1 n2])
+ expect(titles(second)).to eq(%w[d3 n4])
+ expect(titles(third)).to eq(%w[n5])
expect(third.total).to eq(5)
end
+ # Rows one microsecond apart in one second: SQL orders them by the microseconds, so the
+ # merge has to as well. Ids run against the time order, so a merge that compared whole
+ # seconds and fell back to the id would turn the order round.
+ it 'orders rows within one second by their microseconds' do
+ base = 1.day.ago.change(usec: 0)
+ Prosopite.pause do
+ note('oldest', at: base + 0.000001, id: '00000000-0000-4000-8000-000000000001')
+ note('middle', at: base + 0.000002, id: '00000000-0000-4000-8000-000000000002')
+ document('newest', at: base + 0.000003, id: '00000000-0000-4000-8000-000000000003')
+ end
+
+ first = lookup.page(options(page: 1, per_page: 1))
+ second = lookup.page(options(page: 2, per_page: 1))
+ third = lookup.page(options(page: 3, per_page: 1))
+
+ expect(titles(first) + titles(second) + titles(third)).to eq(%w[newest middle oldest])
+ end
+
+ it 'reads no rows for a page beyond the total, only the counts' do
+ note('Only')
+ statements = []
+ callback = ->(*, payload) { statements << payload[:sql] }
+
+ result = ActiveSupport::Notifications.subscribed(callback, 'sql.active_record') do
+ lookup.page(options(page: 50, per_page: 10))
+ end
+
+ expect(result.records).to eq([])
+ expect(result.total).to eq(1)
+ expect(statements.grep(/\ASELECT "(notes|documents)"\.\*/)).to be_empty
+ end
+
it 'returns an empty page beyond the last one, with the full total' do
note('Only')