A platform for community discussion. Free, open, simple.
MT: Test extraction and placeholder resolution as a round trip (#43750)
Gerhard Schlagerec62b815b05a
diff --git a/migrations/importer/Gemfile b/migrations/importer/Gemfileindex 7cdbfc79231..a3281abaa44 100644--- a/migrations/importer/Gemfile+++ b/migrations/importer/Gemfile@@ -7,6 +7,9 @@ gem "migrations-core", path: "../core"gemspecgroup :test do+ # Exercise the resolver against rows produced by the real extractor.+ gem "migrations-converters", path: "../converters"+gem "rspec"gem "rspec-multi-mock"gem "mocha"diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/code.md b/migrations/importer/spec/fixtures/extraction_round_trip/code.mdnew file mode 100644index 00000000000..e3854adacb8--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/code.md@@ -0,0 +1,13 @@+Nothing below is an embed.++```+@alice #support :partyparrot: upload://abc123XYZ.png+[quote="alice, post:3, topic:42"]+https://forum.example.com/community/t/how-to-fix-it/123+```++ @alice in an indented block+ ++Inline: `@alice`, `#support`, `:partyparrot:`, ``,+`https://forum.example.com/community/faq` and `0123456789abcdef0123456789abcdef01234567`.diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/emoji.md b/migrations/importer/spec/fixtures/extraction_round_trip/emoji.mdnew file mode 100644index 00000000000..c31867c92de--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/emoji.md@@ -0,0 +1,3 @@+Ship it :partyparrot: — the standard :smile: is not a custom emoji.++In code it is text: `:partyparrot:`.diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/entities.md b/migrations/importer/spec/fixtures/extraction_round_trip/entities.mdnew file mode 100644index 00000000000..696adf84a55--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/entities.md@@ -0,0 +1,3 @@+Entities sit next to embeds: & and & before @alice and #support.++AT&T ships :partyparrot: as well — and so does [faq](https://old.example.com/faq).diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/hashtags.md b/migrations/importer/spec/fixtures/extraction_round_trip/hashtags.mdnew file mode 100644index 00000000000..9af6ffcab32--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/hashtags.md@@ -0,0 +1,9 @@+Filed under #support and #support:billing, tagged #howto.++An unknown #nosuchthing stays literal, and so does a bare `#`.++Inline `#support` is code:++```text+#support:billing #howto+```diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/links.md b/migrations/importer/spec/fixtures/extraction_round_trip/links.mdnew file mode 100644index 00000000000..a2d8b3cf6c2--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/links.md@@ -0,0 +1,17 @@+Front page: https://forum.example.com/community++The FAQ on the current host: [faq](https://forum.example.com/community/faq)++The same page on the domain the site used before: [faq](https://old.example.com/faq)++Spelled out twice: [https://old.example.com/about](https://old.example.com/about)++A topic, relative and inside the prefix: [read this](/community/t/how-to-fix-it/123 "worth it")++A path outside the prefix belongs to another app: /t/how-to-fix-it/123++A bare internal URL: https://forum.example.com/community/t/how-to-fix-it/123/4++A user page on the former domain: <https://old.example.com/u/alice>++Somewhere else entirely: [elsewhere](https://blog.example.net/posts/1)diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/mentions.md b/migrations/importer/spec/fixtures/extraction_round_trip/mentions.mdnew file mode 100644index 00000000000..e20bc3d480b--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/mentions.md@@ -0,0 +1,11 @@+Ping @alice and @bob about the release.++@staff should weigh in, and @here is everyone reading along.++A name the source does not have stays literal: @nobody.++Inline `@alice` is code, and so is the block below.++```text+@alice @staff @here+```diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/mixed.md b/migrations/importer/spec/fixtures/extraction_round_trip/mixed.mdnew file mode 100644index 00000000000..ed4498a0202--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/mixed.md@@ -0,0 +1,14 @@+@alice filed this under #support:billing :partyparrot:++[quote="bob, post:2, topic:42"]+Earlier, in the same topic.+[/quote]++++Details in [the guide](/community/t/how-to-fix-it/123) and on the+[FAQ](https://old.example.com/faq), plus https://blog.example.net/posts/1.++```+@alice #support:billing :partyparrot: upload://abc123XYZ.png+```diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/multiline_label.md b/migrations/importer/spec/fixtures/extraction_round_trip/multiline_label.mdnew file mode 100644index 00000000000..694303251c0--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/multiline_label.md@@ -0,0 +1,7 @@+A link whose label+[spans+two lines](/community/t/how-to-fix-it/123) inside one paragraph.++And an image whose alt text+ too.diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/quotes.md b/migrations/importer/spec/fixtures/extraction_round_trip/quotes.mdnew file mode 100644index 00000000000..caa23341384--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/quotes.md@@ -0,0 +1,9 @@+[quote="alice, post:3, topic:42"]+The quoted text is carried through as plain text.+[/quote]++[quote="bob, post:2"]+No topic here — it is the one this post lives in.+[/quote]++Both of those need the source posts to resolve.diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/references.md b/migrations/importer/spec/fixtures/extraction_round_trip/references.mdnew file mode 100644index 00000000000..21e9a3a5333--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/references.md@@ -0,0 +1,5 @@+Read the [guide][1], look at the ![shot][2], and see the [old faq][3].++[1]: /community/t/how-to-fix-it/123+[2]: upload://abc123XYZ.png+[3]: <https://old.example.com/faq>diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/refused_route.md b/migrations/importer/spec/fixtures/extraction_round_trip/refused_route.mdnew file mode 100644index 00000000000..4c5f28f0756--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/refused_route.md@@ -0,0 +1 @@+The topic you linked as [>>>>>](https://forum.example.com/community/t//209) does not load.diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/uploads.md b/migrations/importer/spec/fixtures/extraction_round_trip/uploads.mdnew file mode 100644index 00000000000..373d48483ff--- /dev/null+++ b/migrations/importer/spec/fixtures/extraction_round_trip/uploads.md@@ -0,0 +1,13 @@+++[report.pdf|attachment](upload://Zm9vYmFy.pdf) (1.2 MB)++A failed image paste keeps its hybrid label: [image|281x500](upload://YmFyYmF6.jpeg)++Full URL on the source host:+++Hotlinked from somewhere else:+++A short URL: https://forum.example.com/community/uploads/short-url/21.jpegdiff --git a/migrations/importer/spec/integration/extraction_round_trip_spec.rb b/migrations/importer/spec/integration/extraction_round_trip_spec.rbnew file mode 100644index 00000000000..a240e54843e--- /dev/null+++ b/migrations/importer/spec/integration/extraction_round_trip_spec.rb@@ -0,0 +1,126 @@+# frozen_string_literal: true++require "migrations-converters"++# The extractor replaces every reference in a post body with a placeholder token+# and records an embed row for it; the importer's resolver puts the tokens back.+# With nothing mapped, the two must cancel out: the resolved body equals the+# extractor's normalized input, except for links to the source site, whose+# origin always becomes the destination's base URL.+#+# The `PlaceholderResolver` specs cover the resolver alone against hand-written+# rows, so a recorder-side mistake — a wrong `url_offset`, a missing+# `original_markdown`, a token minted but never written — cannot show up there.+# It shows up here, because the rows come from a real extraction.+RSpec.describe "markdown extraction round trip" do+ include_context "with extraction round trip"++ let(:fixtures) { ExtractionRoundTrip.round_trip_bodies }++ describe "resolution with nothing mapped" do+ ExtractionRoundTrip.round_trip_bodies.each_key do |name|+ it "restores the #{name.tr("_", " ")} body, rewriting only source-site origins" do+ outcome = round_trip(fixtures.slice(name))++ expect(outcome.resolved[name]).to eq(outcome.expected[name])+ end+ end++ # Every construct the fixtures carry must survive the scanner intact;+ # otherwise the round trip above would be checking verbatim passthrough.+ it "extracts without refusing any fixture" do+ outcome = round_trip(fixtures)++ expect(outcome.refusals).to be_empty+ end++ # A fixture set that stopped producing rows would satisfy the round trip+ # with an extractor that recorded nothing at all.+ it "covers every construct the extractor defers" do+ outcome = round_trip(fixtures)+ kinds =+ outcome.placeholders.values.flatten.map { |token| Migrations::Placeholder.kind(token) }++ expect(kinds.uniq).to match_array(%w[emoji hashtag link mention quote upload])+ end++ # The origin rewrite is the one difference the contract allows, so it has to+ # actually happen — and only on the source site's own links.+ it "moves a source-site link onto the destination base URL" do+ name = "links"+ outcome = round_trip(fixtures.slice(name))++ expect(outcome.resolved[name]).not_to eq(fixtures[name])+ expect(outcome.resolved[name]).to include("#{ExtractionRoundTrip::BASE_URL}/faq")+ expect(outcome.resolved[name]).to include("[elsewhere](https://blog.example.net/posts/1)")+ end+ end++ describe "placeholder bookkeeping" do+ it "leaves no token without an embed row" do+ outcome = round_trip(fixtures)++ expect(outcome.orphans).to be_empty+ end++ it "puts every minted token into the extracted body exactly once" do+ outcome = round_trip(fixtures)++ miscounted =+ outcome.placeholders.flat_map do |name, tokens|+ tokens+ .reject { |token| outcome.extracted[name].scan(token).size == 1 }+ .map { |token| "#{name}: #{token.inspect}" }+ end++ expect(miscounted).to be_empty+ end++ it "leaves no token in the resolved body" do+ outcome = round_trip(fixtures)++ leftovers = outcome.resolved.select { |_name, body| Migrations::Placeholder.include?(body) }++ expect(leftovers.keys).to be_empty+ end+ end++ describe "batched and per-body scanning" do+ it "extracts and resolves a batch exactly as it does one body at a time" do+ batched = round_trip(fixtures, batched: true)+ per_body = round_trip(fixtures, batched: false)++ expect(batched.extracted).to eq(per_body.extracted)+ expect(batched.resolved).to eq(per_body.resolved)+ expect(batched.refusals).to eq(per_body.refusals)+ end++ # A body that had to be scanned again inside `extract_prepared` means the+ # batch call dropped it, which the identical results above would hide. The+ # engine-bound count guards the other way: without it a fixture set that+ # never reaches the engine would satisfy this trivially.+ it "scans no body again outside the batch" do+ outcome = round_trip(fixtures, batched: true)++ expect(outcome.engine_bound).to be > 1+ expect(outcome.live_scans).to eq(0)+ end+ end++ describe "a body the extractor refuses" do+ let(:name) { ExtractionRoundTrip::REFUSED_FIXTURE }+ let(:body) { ExtractionRoundTrip.fixtures.fetch(name) }++ # A refusal keeps the body out of the contract rather than breaking it: the+ # bytes are handed on untouched, and the cause is reported so a conversion+ # can list what still needs resolving.+ it "hands the body on verbatim and reports the cause" do+ outcome = round_trip({ name => body })++ expect(outcome.extracted[name]).to eq(body)+ expect(outcome.resolved[name]).to eq(body)+ expect(outcome.placeholders[name]).to be_empty+ expect(outcome.refusals).to eq(invalid_internal_route: 1)+ end+ end+enddiff --git a/migrations/importer/spec/support/extraction_round_trip_context.rb b/migrations/importer/spec/support/extraction_round_trip_context.rbnew file mode 100644index 00000000000..af1f057931f--- /dev/null+++ b/migrations/importer/spec/support/extraction_round_trip_context.rb@@ -0,0 +1,321 @@+# frozen_string_literal: true++require "tmpdir"++# Scaffolding for the extraction round-trip spec: the fixture bodies, the source+# site the extractor is configured for, a maps object where nothing resolves,+# and the runner that drives extraction and resolution against each other.+module ExtractionRoundTrip+ FIXTURES_PATH = File.expand_path("../fixtures/extraction_round_trip", __dir__)++ # The source site: one host on a subdirectory install plus the domain it used+ # before, so absolute links on either spelling, prefixed relative links and+ # foreign hosts all occur in the fixtures.+ SOURCE_HOST = "forum.example.com"+ SOURCE_PREFIX = "/community"+ FORMER_HOST = "old.example.com"+ HOSTS = { SOURCE_HOST => SOURCE_PREFIX, FORMER_HOST => nil }.freeze++ # The destination's base URL. Unreachable on purpose: a link only carries it+ # when resolution put it there.+ BASE_URL = "https://unresolved.invalid"++ # Every fixture body is a post in this topic, so a quote naming only a+ # `post:` has a topic to be completed from.+ TOPIC_ID = 42++ MENTION_NAMES = %w[alice all bob here staff].freeze+ GROUP_NAMES = %w[staff].freeze+ HASHTAG_NAMES = %w[howto support support:billing].freeze+ CUSTOM_EMOJI_NAMES = %w[partyparrot].freeze++ # The fixture that is expected to defeat the extractor; it is asserted on+ # separately, not round-tripped.+ REFUSED_FIXTURE = "refused_route"++ # CR line endings take a different path through the scanner, so one body is+ # rewritten here rather than stored — a checkout normalizes line endings.+ CRLF_SOURCE = "mixed"+ CRLF_FIXTURE = "mixed_crlf"++ # What one run of the fixture set produced; each value is keyed by fixture+ # name, except the tallies.+ Outcome =+ Data.define(+ :extracted,+ :expected,+ :resolved,+ :placeholders,+ :refusals,+ :orphans,+ :engine_bound,+ :live_scans,+ )++ def self.fixtures+ @fixtures ||=+ Dir[File.join(FIXTURES_PATH, "*.md")]+ .sort+ .to_h { |path| [File.basename(path, ".md"), File.read(path, encoding: Encoding::UTF_8)] }+ .freeze+ end++ # The bodies the round-trip properties hold for, in a stable order.+ def self.round_trip_bodies+ @round_trip_bodies ||=+ fixtures+ .except(REFUSED_FIXTURE)+ .merge(CRLF_FIXTURE => fixtures.fetch(CRLF_SOURCE).gsub("\n", "\r\n"))+ .freeze+ end++ # Building an engine context means a V8 isolate evaluating the whole engine+ # bundle, so the file shares one; contexts are stateless across scans.+ def self.markdown_engine+ @markdown_engine ||=+ Migrations::Converters::MarkdownEngine::Context.new(+ bundle: Migrations::Converters::MarkdownEngine::Bundle.load_or_build,+ config:+ Migrations::Converters::MarkdownEngine::Config.new(+ source_settings: {+ "unicode_usernames" => true,+ },+ category_slugs: HASHTAG_NAMES,+ tag_names: HASHTAG_NAMES,+ custom_emoji_names: CUSTOM_EMOJI_NAMES,+ ),+ )+ end++ def self.close_markdown_engine+ @markdown_engine&.close+ @markdown_engine = nil+ end++ # Every lookup misses, so resolution has nothing but each row's recorded+ # source to fall back on. A link to the source site is the exception the+ # contract allows: its origin always becomes the destination's.+ class AllMissMaps+ %i[+ badge+ category_id+ category_slug_path+ emoji_name+ event_markdown+ group_name+ poll_markdown+ post+ tag_id+ tag_name+ topic_id+ upload+ upload_markdown+ user+ ].each { |name| define_method(name) { |_key| nil } }++ def base_url+ BASE_URL+ end++ def here_mention+ "here"+ end+ end+end++RSpec.configure { |config| config.after(:suite) { ExtractionRoundTrip.close_markdown_engine } }++RSpec.shared_context "with extraction round trip" do+ let(:enums) { Migrations::Database::IntermediateDB::Enums }+ let(:owner_type) { enums::EmbedOwner::POST }++ # Small enough that the fixture set spans several engine calls.+ let(:batch_size) { 4 }++ # Extracts each body into a real buffer, writes the embed rows the way a+ # converter step does, and resolves the whole set back in one call.+ #+ # @param batched [Boolean] whether the engine-bound bodies go through+ # `scan_batches` or are scanned one at a time inside `extract_prepared`.+ def round_trip(bodies, batched: true)+ with_intermediate_db { |db| extract_and_resolve(bodies, db, batched:) }+ end++ def extract_and_resolve(bodies, intermediate_db, batched:)+ refusals = Hash.new(0)+ # A fixed nonce makes the tokens of two runs comparable.+ buffer =+ Migrations::Converters::EmbedBuffer.new(+ owner_type:,+ placeholder: Migrations::Placeholder.new(nonce: "roundtrip"),+ )+ extractor = build_extractor(buffer, refusals)++ names = bodies.keys+ prepared =+ bodies.map.with_index do |(_name, raw), index|+ extractor.prepare(raw:, id: index + 1, topic_id: ExtractionRoundTrip::TOPIC_ID)+ end+ scan_data = batched ? extractor.scan_batches(prepared, max_posts: batch_size) : {}++ extracted = {}+ expected = {}+ placeholders = {}+ engine_bound = 0+ live_scans = 0+ items = []++ prepared.each_with_index do |body, index|+ name = names[index]+ data = scan_data[body.id]+ if body.engine_bound?+ engine_bound += 1+ live_scans += 1 if data.nil?+ end++ buffer.clear+ output = extractor.extract_prepared(body, scan_data: data)+ buffer.write_for(body.id) unless buffer.empty?++ extracted[name] = output+ expected[name] = expected_all_miss_body(output, buffer)+ placeholders[name] = buffer.placeholders+ items << { id: body.id, raw: output }+ end++ orphans = []+ resolver =+ Migrations::Importer::PlaceholderResolver.new(+ intermediate_db,+ ExtractionRoundTrip::AllMissMaps.new,+ owner_type:,+ unresolved_embeds: [],+ orphan_placeholders: orphans,+ )+ resolved = resolver.resolve_all(items)++ ExtractionRoundTrip::Outcome.new(+ extracted:,+ expected:,+ resolved: names.each_with_index.to_h { |name, index| [name, resolved[index + 1]] },+ placeholders:,+ refusals:,+ orphans:,+ engine_bound:,+ live_scans:,+ )+ end++ def build_extractor(buffer, refusals)+ Migrations::Converters::Discourse::RawExtractor.new(+ embeds: buffer,+ mention_names: compact_set(ExtractionRoundTrip::MENTION_NAMES),+ hashtag_names: compact_set(ExtractionRoundTrip::HASHTAG_NAMES),+ custom_emoji_names: ExtractionRoundTrip::CUSTOM_EMOJI_NAMES,+ markdown_engine: ExtractionRoundTrip.markdown_engine,+ internal_link_hosts: ExtractionRoundTrip::HOSTS,+ internal_link_base_prefix: ExtractionRoundTrip::SOURCE_PREFIX,+ mention_classifier:+ Migrations::Converters::Discourse::MentionClassifier.new(+ here_mention: "here",+ group_names: ExtractionRoundTrip::GROUP_NAMES,+ ),+ on_engine_refusal: ->(cause, _detail) { refusals[cause] += 1 },+ )+ end++ def compact_set(names)+ Migrations::CompactStringSet.new(+ names.map { |name| Migrations::NameNormalizer.normalize(name) },+ )+ end++ def with_intermediate_db+ Dir.mktmpdir("extraction-round-trip") do |dir|+ db_path = File.join(dir, "intermediate.db")+ Migrations::Database.migrate(+ db_path,+ migrations_path: Migrations::Database::INTERMEDIATE_DB_SCHEMA_PATH,+ )+ database = Migrations::Database.connect(db_path)+ Migrations::Database::IntermediateDB.setup(database)+ yield database+ ensure+ Migrations::Database::IntermediateDB.setup(nil)+ database&.close+ end+ end++ # The byte-exact expectation for one body: the extracted raw with every token+ # put back the way an all-miss resolution has to put it back.+ def expected_all_miss_body(output, buffer)+ embed_rows(buffer).reduce(output) do |expected, (kind, row)|+ replacement = expected_replacement(kind, row)+ # The block form keeps a `\` in a snippet from being read as a+ # backreference.+ replacement ? expected.sub(row[:placeholder]) { replacement } : expected+ end+ end++ def expected_replacement(kind, row)+ if kind == :link && row[:target_type] == enums::LinkTarget::SITE+ expected_site_markup(row)+ elsif kind == :emoji+ # An emoji row carries no snippet: its name is the source spelling.+ ":#{row[:name]}:"+ else+ row[:original_markdown]+ end+ end++ # A link to the source site has no entity to miss — only its origin moves —+ # so the expectation splices the rewritten destination itself. The arithmetic+ # is independent of the resolver's: the path comes from the row's own URL+ # spelling rather than from `target_suffix`, so a wrongly recorded suffix+ # still fails. A row without a usable span returns nil, which leaves the token+ # in the expectation and fails the comparison loudly.+ def expected_site_markup(row)+ original = row[:original_markdown]+ offset = row[:url_offset]+ return nil if original.nil? || offset.nil?++ length = row[:url].bytesize+ return nil if offset < 0 || offset + length > original.bytesize++ spans = [offset]+ label = row[:label_url_offset]+ spans << label if label && label != offset && label >= 0 && label + length <= original.bytesize++ url = expected_site_url(row)+ expected = original.dup+ spans.sort.reverse_each { |span| expected.bytesplice(span, length, url) }+ expected+ end++ def expected_site_url(row)+ origin = Migrations::Converters::Discourse::MarkdownScanner::UrlOrigin+ host, rest = origin.split(row[:url])+ hosts = ExtractionRoundTrip::HOSTS+ path =+ if host && hosts.key?(host)+ origin.path_within_prefix(rest, hosts[host])+ else+ row[:target_suffix]+ end++ "#{ExtractionRoundTrip::BASE_URL}#{path}"+ end++ def embed_rows(buffer)+ {+ quote: buffer.quotes,+ link: buffer.links,+ mention: buffer.mentions,+ hashtag: buffer.hashtags,+ emoji: buffer.emojis,+ poll: buffer.polls,+ event: buffer.events,+ upload: buffer.uploads,+ }.flat_map { |kind, rows| rows.map { |row| [kind, row] } }+ end+end