Skip to content

Self-hosted project management for devs and their clients: tickets, time tracking, docs and meetings, with an AI assistant that cites its sources. AGPL-3.0.

Refuse unstorable Idempotency-Keys and widen the error-body guard

Guido Boyke5efde7ed1c73
diff --git a/CLAUDE.md b/CLAUDE.md
index df725ed..39f3085 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -793,14 +793,15 @@ feature by feature):
`unauthorized`); an unmapped actor code is `refused` with the old value as `reason`. Every
body comes from `Api::V1::ErrorBody` through `BaseController#render_error`; guard
`spec/architecture/api_v1_error_body_guard_spec.rb`. Answers outside the controllers carry
- no `code`: the Rack::Attack 429 (plain text) and a 500 from the exceptions app. The 401 of `/api/mcp` (same concern,
+ no `code`: the Rack::Attack 429 (plain text), a 500 from the exceptions app, and a 400 for a
+ query string Rack cannot decode (`?x=%`; raised in middleware before any controller). The 401 of `/api/mcp` (same concern,
`render_unauthorized`) keeps its old body.
* **`Idempotency-Key`** on every write (`idempotent :create` in the controller,
`Api::V1::Idempotent`): the same key and request replays the first 2xx answer (status and
body), another request answers 422 `idempotency_key_reused`. Same table and store as MCP
(`Mcp::IdempotencyStore`); the REST fingerprint includes method and path, so a key never
crosses surfaces. Runs inside authentication and the read-only gate. A key is at most 255
- bytes (longer: 422 `invalid_idempotency_key`, nothing written). A replay checks no rights
+ bytes of valid UTF-8 without NUL (else: 422 `invalid_idempotency_key`, nothing written). A replay checks no rights
(it repeats what the same token already got; keys do not expire), and a retry must send
the same bytes. Every write route must be `idempotent`
(`spec/architecture/api_v1_writes_idempotent_guard_spec.rb`).
diff --git a/app/controllers/concerns/api/v1/idempotent.rb b/app/controllers/concerns/api/v1/idempotent.rb
index 637357e..6accc2d 100644
--- a/app/controllers/concerns/api/v1/idempotent.rb
+++ b/app/controllers/concerns/api/v1/idempotent.rb
@@ -22,7 +22,11 @@ module Api
# same token already received, nothing it has not seen; a new key goes
# through every check. Keys do not expire. A retry must send the same bytes:
# the fingerprint sorts only the top-level keys, so another encoding of the
- # same request is a mismatch (safe: it writes nothing).
+ # same request is a mismatch (safe: it writes nothing). With a key the body is
+ # parsed even where the action ignores it, so malformed JSON answers 400 there.
+ # A multipart body is never repeatable (the upload's temp path is in the
+ # fingerprint); reduce files to name, size and digest when an upload endpoint
+ # arrives.
module Idempotent
extend ActiveSupport::Concern
@@ -53,7 +57,7 @@ module Api
def with_idempotency_key
key = request.headers[HEADER].to_s.strip.presence
return yield unless key
- return render_invalid_idempotency_key if key.bytesize > MAX_KEY_LENGTH
+ return render_invalid_idempotency_key unless storable_key?(key)
store = Mcp::IdempotencyStore.new(token: current_token, tool_name: "rest:#{controller_path}##{action_name}",
key: key, args: idempotency_fingerprint_args)
@@ -65,6 +69,13 @@ module Api
store.remember({ status: response.status, body: response.body }.to_json) if response.successful?
end
+ # Within the index (bytes) and storable in a PostgreSQL text column: valid
+ # UTF-8 without NUL, which the database refuses with an error otherwise.
+ def storable_key?(key)
+ utf8 = key.dup.force_encoding(Encoding::UTF_8)
+ key.bytesize <= MAX_KEY_LENGTH && utf8.valid_encoding? && !utf8.include?("\0")
+ end
+
def idempotency_fingerprint_args
{ 'method' => request.request_method, 'path' => request.path,
'params' => request.query_parameters.merge(request.request_parameters) }
@@ -82,7 +93,8 @@ module Api
def render_invalid_idempotency_key
skip_authorization
render_error(error: INVALID_KEY, code: 'invalid_input',
- message: "#{HEADER} is longer than #{MAX_KEY_LENGTH} bytes", status: :unprocessable_entity)
+ message: "#{HEADER} must be valid UTF-8 without NUL, at most #{MAX_KEY_LENGTH} bytes",
+ status: :unprocessable_entity)
end
def render_idempotency_mismatch
diff --git a/spec/architecture/api_v1_error_body_guard_spec.rb b/spec/architecture/api_v1_error_body_guard_spec.rb
index e546fe4..e95723f 100644
--- a/spec/architecture/api_v1_error_body_guard_spec.rb
+++ b/spec/architecture/api_v1_error_body_guard_spec.rb
@@ -13,17 +13,33 @@ module ApiV1ErrorBodyGuard
GLOBS = %w[app/controllers/api/v1/**/*.rb app/controllers/concerns/api/v1/**/*.rb].freeze
OWNER = 'app/controllers/concerns/api/v1/error_body.rb'
- ERROR_STATUS = 'bad_request|unauthorized|forbidden|not_found|unprocessable_entity|unprocessable_content|' \
- 'conflict|internal_server_error|[45]\\d\\d'
+ # The two renders whose status is a variable by design, each named by the
+ # line it may contain. Anything else in these files is still checked.
+ ALLOWED_LINES = {
+ 'app/controllers/api/v1/base_controller.rb' =>
+ 'render json: Api::V1::ErrorBody.build(error: error, code: code, **extra), status: status',
+ 'app/controllers/concerns/api/v1/idempotent.rb' =>
+ "render json: answer.fetch('body'), status: answer.fetch('status')"
+ }.freeze
- # Any of: a render with an `error`/`errors` key (one line or wrapped), a
- # string `"error" =>` key, a bodyless `head` with an error status, or a plain
- # `render` (not `render_error`) that sets an error status on its own line.
+ # Every error status Rack knows by name, and any 4xx/5xx number.
+ ERROR_STATUS = (Rack::Utils::SYMBOL_TO_STATUS_CODE.select { |_name, code| code >= 400 }.keys.map(&:to_s) +
+ ['[45]\\d\\d']).join('|')
+ SUCCESS_STATUS = 'ok|created|accepted|no_content|2\\d\\d'
+
+ # Any of:
+ # * a render with an `error`/`errors` key, one line or wrapped;
+ # * a string or symbol key: `"error" =>`, `"error":`, `:error =>`;
+ # * a bodyless `head` with an error status;
+ # * a plain `render` (not `render_error`) whose `status:`, on its line or a
+ # continuation line after a comma, is anything but a success status: an
+ # error symbol, a number, or a variable such as a status lookup.
HAND_BUILT = Regexp.union(
/render\s*\(?\s*json:\s*\{[^}]*\berrors?:/m,
- /["']errors?["']\s*=>/,
+ /["']errors?["']\s*(?:=>|:)/,
+ /:errors?\s*=>/,
/\bhead\s*\(?\s*:?(?:#{ERROR_STATUS})\b/,
- /\brender(?!_)\b[^\n]*\bstatus:\s*:?(?:#{ERROR_STATUS})\b/
+ /\brender(?!_)\b(?:[^\n]*,[ \t]*\n)*[^\n]*\bstatus:[ \t]*(?![ \t])(?!:?(?:#{SUCCESS_STATUS})\b)/
)
def self.offences
@@ -31,7 +47,9 @@ module ApiV1ErrorBodyGuard
relative = path.relative_path_from(Rails.root).to_s
next if relative == OWNER
- relative if path.read.match?(HAND_BUILT)
+ source = path.read
+ source = source.sub(ALLOWED_LINES[relative], '') if ALLOWED_LINES.key?(relative)
+ relative if source.match?(HAND_BUILT)
end
end
end
@@ -52,10 +70,22 @@ RSpec.describe 'API V1 error body guard', :ci_guard do
expect('head :forbidden').to match(ApiV1ErrorBodyGuard::HAND_BUILT)
expect('render json: { data: rows }').not_to match(ApiV1ErrorBodyGuard::HAND_BUILT)
expect('render json: { data: row }, status: :created').not_to match(ApiV1ErrorBodyGuard::HAND_BUILT)
+ expect("render json: { message: x },\n status: :forbidden").to match(ApiV1ErrorBodyGuard::HAND_BUILT)
+ expect('render json: { "error": "x" }, status: http_status').to match(ApiV1ErrorBodyGuard::HAND_BUILT)
+ expect("render json: { :error => 'x' }").to match(ApiV1ErrorBodyGuard::HAND_BUILT)
+ expect('render json: x, status: ACTOR_ERROR_STATUS.fetch(c)').to match(ApiV1ErrorBodyGuard::HAND_BUILT)
+ expect('head :too_many_requests').to match(ApiV1ErrorBodyGuard::HAND_BUILT)
+ expect("render json: { data: row },\n status: :created").not_to match(ApiV1ErrorBodyGuard::HAND_BUILT)
expect('head :ok').not_to match(ApiV1ErrorBodyGuard::HAND_BUILT)
expect("render_error(error: 'forbidden', status: :forbidden)").not_to match(ApiV1ErrorBodyGuard::HAND_BUILT)
end
+ it 'still finds each allowed line, so an allowance cannot outlive its code' do
+ ApiV1ErrorBodyGuard::ALLOWED_LINES.each do |file, line|
+ expect(Rails.root.join(file).read).to include(line), "#{file} no longer contains the allowed render"
+ end
+ end
+
it 'leaves every error body to render_error' do
expect(ApiV1ErrorBodyGuard.offences).to be_empty
end
diff --git a/spec/requests/api/v1/idempotency_spec.rb b/spec/requests/api/v1/idempotency_spec.rb
index 3b1e9d4..fd50352 100644
--- a/spec/requests/api/v1/idempotency_spec.rb
+++ b/spec/requests/api/v1/idempotency_spec.rb
@@ -167,6 +167,29 @@ RSpec.describe 'API V1 idempotency', :ci_guard, type: :request do
expect(response.parsed_body).to include('error' => 'invalid_idempotency_key', 'code' => 'invalid_input')
end
+ # Bytes, not characters: 128 umlauts are 256 bytes.
+ it 'counts the limit in bytes' do
+ expect { comment('Hallo', key: 'ä' * 128) }.not_to(change { posts_count })
+ expect(response).to have_http_status(:unprocessable_content)
+ expect { comment('Hallo', key: "#{'ä' * 127}k") }.to change { posts_count }.by(1)
+ end
+
+ # PostgreSQL refuses both in a text column; they used to answer a 500.
+ def expect_invalid_key
+ expect(response).to have_http_status(:unprocessable_content)
+ expect(response.parsed_body).to include('error' => 'invalid_idempotency_key', 'code' => 'invalid_input')
+ end
+
+ it 'refuses a key that is not valid UTF-8, writing nothing' do
+ expect { comment('Hallo', key: "k\xFFx".b) }.not_to(change { posts_count })
+ expect_invalid_key
+ end
+
+ it 'refuses a key with a NUL byte, writing nothing' do
+ expect { comment('Hallo', key: "k\u0000x") }.not_to(change { posts_count })
+ expect_invalid_key
+ end
+
it 'accepts a key of exactly the limit' do
key = 'k' * Api::V1::Idempotent::MAX_KEY_LENGTH
expect { 2.times { comment('Hallo', key: key) } }.to change { posts_count }.by(1)