Skip to content

A platform for community discussion. Free, open, simple.

MT: Test extraction and placeholder resolution as a round trip (#43750)

Gerhard Schlagerec62b815b05a
diff --git a/migrations/importer/Gemfile b/migrations/importer/Gemfile
index 7cdbfc79231..a3281abaa44 100644
--- a/migrations/importer/Gemfile
+++ b/migrations/importer/Gemfile
@@ -7,6 +7,9 @@ gem "migrations-core", path: "../core"
gemspec
group :test do
+ # Exercise the resolver against rows produced by the real extractor.
+ gem "migrations-converters", path: "../converters"
+
gem "rspec"
gem "rspec-multi-mock"
gem "mocha"
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/code.md b/migrations/importer/spec/fixtures/extraction_round_trip/code.md
new file mode 100644
index 00000000000..e3854adacb8
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/code.md
@@ -0,0 +1,13 @@
+Nothing below is an embed.
+
+```
+@alice #support :partyparrot: upload://abc123XYZ.png
+[quote="alice, post:3, topic:42"]
+https://forum.example.com/community/t/how-to-fix-it/123
+```
+
+ @alice in an indented block
+ ![x](upload://abc123XYZ.png)
+
+Inline: `@alice`, `#support`, `:partyparrot:`, `![x](upload://abc123XYZ.png)`,
+`https://forum.example.com/community/faq` and `0123456789abcdef0123456789abcdef01234567`.
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/emoji.md b/migrations/importer/spec/fixtures/extraction_round_trip/emoji.md
new file mode 100644
index 00000000000..c31867c92de
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/emoji.md
@@ -0,0 +1,3 @@
+Ship it :partyparrot: — the standard :smile: is not a custom emoji.
+
+In code it is text: `:partyparrot:`.
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/entities.md b/migrations/importer/spec/fixtures/extraction_round_trip/entities.md
new file mode 100644
index 00000000000..696adf84a55
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/entities.md
@@ -0,0 +1,3 @@
+Entities sit next to embeds: & and & before @alice and #support.
+
+AT&T ships :partyparrot: as well — and so does [faq](https://old.example.com/faq).
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/hashtags.md b/migrations/importer/spec/fixtures/extraction_round_trip/hashtags.md
new file mode 100644
index 00000000000..9af6ffcab32
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/hashtags.md
@@ -0,0 +1,9 @@
+Filed under #support and #support:billing, tagged #howto.
+
+An unknown #nosuchthing stays literal, and so does a bare `#`.
+
+Inline `#support` is code:
+
+```text
+#support:billing #howto
+```
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/links.md b/migrations/importer/spec/fixtures/extraction_round_trip/links.md
new file mode 100644
index 00000000000..a2d8b3cf6c2
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/links.md
@@ -0,0 +1,17 @@
+Front page: https://forum.example.com/community
+
+The FAQ on the current host: [faq](https://forum.example.com/community/faq)
+
+The same page on the domain the site used before: [faq](https://old.example.com/faq)
+
+Spelled out twice: [https://old.example.com/about](https://old.example.com/about)
+
+A topic, relative and inside the prefix: [read this](/community/t/how-to-fix-it/123 "worth it")
+
+A path outside the prefix belongs to another app: /t/how-to-fix-it/123
+
+A bare internal URL: https://forum.example.com/community/t/how-to-fix-it/123/4
+
+A user page on the former domain: <https://old.example.com/u/alice>
+
+Somewhere else entirely: [elsewhere](https://blog.example.net/posts/1)
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/mentions.md b/migrations/importer/spec/fixtures/extraction_round_trip/mentions.md
new file mode 100644
index 00000000000..e20bc3d480b
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/mentions.md
@@ -0,0 +1,11 @@
+Ping @alice and @bob about the release.
+
+@staff should weigh in, and @here is everyone reading along.
+
+A name the source does not have stays literal: @nobody.
+
+Inline `@alice` is code, and so is the block below.
+
+```text
+@alice @staff @here
+```
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/mixed.md b/migrations/importer/spec/fixtures/extraction_round_trip/mixed.md
new file mode 100644
index 00000000000..ed4498a0202
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/mixed.md
@@ -0,0 +1,14 @@
+@alice filed this under #support:billing :partyparrot:
+
+[quote="bob, post:2, topic:42"]
+Earlier, in the same topic.
+[/quote]
+
+![screenshot|690x388](upload://abc123XYZ.png)
+
+Details in [the guide](/community/t/how-to-fix-it/123) and on the
+[FAQ](https://old.example.com/faq), plus https://blog.example.net/posts/1.
+
+```
+@alice #support:billing :partyparrot: upload://abc123XYZ.png
+```
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/multiline_label.md b/migrations/importer/spec/fixtures/extraction_round_trip/multiline_label.md
new file mode 100644
index 00000000000..694303251c0
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/multiline_label.md
@@ -0,0 +1,7 @@
+A link whose label
+[spans
+two lines](/community/t/how-to-fix-it/123) inside one paragraph.
+
+And an image whose alt text
+![wraps
+around](upload://abc123XYZ.png) too.
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/quotes.md b/migrations/importer/spec/fixtures/extraction_round_trip/quotes.md
new file mode 100644
index 00000000000..caa23341384
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/quotes.md
@@ -0,0 +1,9 @@
+[quote="alice, post:3, topic:42"]
+The quoted text is carried through as plain text.
+[/quote]
+
+[quote="bob, post:2"]
+No topic here — it is the one this post lives in.
+[/quote]
+
+Both of those need the source posts to resolve.
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/references.md b/migrations/importer/spec/fixtures/extraction_round_trip/references.md
new file mode 100644
index 00000000000..21e9a3a5333
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/references.md
@@ -0,0 +1,5 @@
+Read the [guide][1], look at the ![shot][2], and see the [old faq][3].
+
+[1]: /community/t/how-to-fix-it/123
+[2]: upload://abc123XYZ.png
+[3]: <https://old.example.com/faq>
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/refused_route.md b/migrations/importer/spec/fixtures/extraction_round_trip/refused_route.md
new file mode 100644
index 00000000000..4c5f28f0756
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/refused_route.md
@@ -0,0 +1 @@
+The topic you linked as [>>>>>](https://forum.example.com/community/t//209) does not load.
diff --git a/migrations/importer/spec/fixtures/extraction_round_trip/uploads.md b/migrations/importer/spec/fixtures/extraction_round_trip/uploads.md
new file mode 100644
index 00000000000..373d48483ff
--- /dev/null
+++ b/migrations/importer/spec/fixtures/extraction_round_trip/uploads.md
@@ -0,0 +1,13 @@
+![screenshot|690x388](upload://abc123XYZ.png)
+
+[report.pdf|attachment](upload://Zm9vYmFy.pdf) (1.2 MB)
+
+A failed image paste keeps its hybrid label: [image|281x500](upload://YmFyYmF6.jpeg)
+
+Full URL on the source host:
+![full](https://forum.example.com/community/uploads/default/original/2X/a/ab/0123456789abcdef0123456789abcdef01234567.png)
+
+Hotlinked from somewhere else:
+![foreign](https://cdn.example.net/uploads/default/original/2X/a/ab/0123456789abcdef0123456789abcdef01234567.png)
+
+A short URL: https://forum.example.com/community/uploads/short-url/21.jpeg
diff --git a/migrations/importer/spec/integration/extraction_round_trip_spec.rb b/migrations/importer/spec/integration/extraction_round_trip_spec.rb
new file mode 100644
index 00000000000..a240e54843e
--- /dev/null
+++ b/migrations/importer/spec/integration/extraction_round_trip_spec.rb
@@ -0,0 +1,126 @@
+# frozen_string_literal: true
+
+require "migrations-converters"
+
+# The extractor replaces every reference in a post body with a placeholder token
+# and records an embed row for it; the importer's resolver puts the tokens back.
+# With nothing mapped, the two must cancel out: the resolved body equals the
+# extractor's normalized input, except for links to the source site, whose
+# origin always becomes the destination's base URL.
+#
+# The `PlaceholderResolver` specs cover the resolver alone against hand-written
+# rows, so a recorder-side mistake — a wrong `url_offset`, a missing
+# `original_markdown`, a token minted but never written — cannot show up there.
+# It shows up here, because the rows come from a real extraction.
+RSpec.describe "markdown extraction round trip" do
+ include_context "with extraction round trip"
+
+ let(:fixtures) { ExtractionRoundTrip.round_trip_bodies }
+
+ describe "resolution with nothing mapped" do
+ ExtractionRoundTrip.round_trip_bodies.each_key do |name|
+ it "restores the #{name.tr("_", " ")} body, rewriting only source-site origins" do
+ outcome = round_trip(fixtures.slice(name))
+
+ expect(outcome.resolved[name]).to eq(outcome.expected[name])
+ end
+ end
+
+ # Every construct the fixtures carry must survive the scanner intact;
+ # otherwise the round trip above would be checking verbatim passthrough.
+ it "extracts without refusing any fixture" do
+ outcome = round_trip(fixtures)
+
+ expect(outcome.refusals).to be_empty
+ end
+
+ # A fixture set that stopped producing rows would satisfy the round trip
+ # with an extractor that recorded nothing at all.
+ it "covers every construct the extractor defers" do
+ outcome = round_trip(fixtures)
+ kinds =
+ outcome.placeholders.values.flatten.map { |token| Migrations::Placeholder.kind(token) }
+
+ expect(kinds.uniq).to match_array(%w[emoji hashtag link mention quote upload])
+ end
+
+ # The origin rewrite is the one difference the contract allows, so it has to
+ # actually happen — and only on the source site's own links.
+ it "moves a source-site link onto the destination base URL" do
+ name = "links"
+ outcome = round_trip(fixtures.slice(name))
+
+ expect(outcome.resolved[name]).not_to eq(fixtures[name])
+ expect(outcome.resolved[name]).to include("#{ExtractionRoundTrip::BASE_URL}/faq")
+ expect(outcome.resolved[name]).to include("[elsewhere](https://blog.example.net/posts/1)")
+ end
+ end
+
+ describe "placeholder bookkeeping" do
+ it "leaves no token without an embed row" do
+ outcome = round_trip(fixtures)
+
+ expect(outcome.orphans).to be_empty
+ end
+
+ it "puts every minted token into the extracted body exactly once" do
+ outcome = round_trip(fixtures)
+
+ miscounted =
+ outcome.placeholders.flat_map do |name, tokens|
+ tokens
+ .reject { |token| outcome.extracted[name].scan(token).size == 1 }
+ .map { |token| "#{name}: #{token.inspect}" }
+ end
+
+ expect(miscounted).to be_empty
+ end
+
+ it "leaves no token in the resolved body" do
+ outcome = round_trip(fixtures)
+
+ leftovers = outcome.resolved.select { |_name, body| Migrations::Placeholder.include?(body) }
+
+ expect(leftovers.keys).to be_empty
+ end
+ end
+
+ describe "batched and per-body scanning" do
+ it "extracts and resolves a batch exactly as it does one body at a time" do
+ batched = round_trip(fixtures, batched: true)
+ per_body = round_trip(fixtures, batched: false)
+
+ expect(batched.extracted).to eq(per_body.extracted)
+ expect(batched.resolved).to eq(per_body.resolved)
+ expect(batched.refusals).to eq(per_body.refusals)
+ end
+
+ # A body that had to be scanned again inside `extract_prepared` means the
+ # batch call dropped it, which the identical results above would hide. The
+ # engine-bound count guards the other way: without it a fixture set that
+ # never reaches the engine would satisfy this trivially.
+ it "scans no body again outside the batch" do
+ outcome = round_trip(fixtures, batched: true)
+
+ expect(outcome.engine_bound).to be > 1
+ expect(outcome.live_scans).to eq(0)
+ end
+ end
+
+ describe "a body the extractor refuses" do
+ let(:name) { ExtractionRoundTrip::REFUSED_FIXTURE }
+ let(:body) { ExtractionRoundTrip.fixtures.fetch(name) }
+
+ # A refusal keeps the body out of the contract rather than breaking it: the
+ # bytes are handed on untouched, and the cause is reported so a conversion
+ # can list what still needs resolving.
+ it "hands the body on verbatim and reports the cause" do
+ outcome = round_trip({ name => body })
+
+ expect(outcome.extracted[name]).to eq(body)
+ expect(outcome.resolved[name]).to eq(body)
+ expect(outcome.placeholders[name]).to be_empty
+ expect(outcome.refusals).to eq(invalid_internal_route: 1)
+ end
+ end
+end
diff --git a/migrations/importer/spec/support/extraction_round_trip_context.rb b/migrations/importer/spec/support/extraction_round_trip_context.rb
new file mode 100644
index 00000000000..af1f057931f
--- /dev/null
+++ b/migrations/importer/spec/support/extraction_round_trip_context.rb
@@ -0,0 +1,321 @@
+# frozen_string_literal: true
+
+require "tmpdir"
+
+# Scaffolding for the extraction round-trip spec: the fixture bodies, the source
+# site the extractor is configured for, a maps object where nothing resolves,
+# and the runner that drives extraction and resolution against each other.
+module ExtractionRoundTrip
+ FIXTURES_PATH = File.expand_path("../fixtures/extraction_round_trip", __dir__)
+
+ # The source site: one host on a subdirectory install plus the domain it used
+ # before, so absolute links on either spelling, prefixed relative links and
+ # foreign hosts all occur in the fixtures.
+ SOURCE_HOST = "forum.example.com"
+ SOURCE_PREFIX = "/community"
+ FORMER_HOST = "old.example.com"
+ HOSTS = { SOURCE_HOST => SOURCE_PREFIX, FORMER_HOST => nil }.freeze
+
+ # The destination's base URL. Unreachable on purpose: a link only carries it
+ # when resolution put it there.
+ BASE_URL = "https://unresolved.invalid"
+
+ # Every fixture body is a post in this topic, so a quote naming only a
+ # `post:` has a topic to be completed from.
+ TOPIC_ID = 42
+
+ MENTION_NAMES = %w[alice all bob here staff].freeze
+ GROUP_NAMES = %w[staff].freeze
+ HASHTAG_NAMES = %w[howto support support:billing].freeze
+ CUSTOM_EMOJI_NAMES = %w[partyparrot].freeze
+
+ # The fixture that is expected to defeat the extractor; it is asserted on
+ # separately, not round-tripped.
+ REFUSED_FIXTURE = "refused_route"
+
+ # CR line endings take a different path through the scanner, so one body is
+ # rewritten here rather than stored — a checkout normalizes line endings.
+ CRLF_SOURCE = "mixed"
+ CRLF_FIXTURE = "mixed_crlf"
+
+ # What one run of the fixture set produced; each value is keyed by fixture
+ # name, except the tallies.
+ Outcome =
+ Data.define(
+ :extracted,
+ :expected,
+ :resolved,
+ :placeholders,
+ :refusals,
+ :orphans,
+ :engine_bound,
+ :live_scans,
+ )
+
+ def self.fixtures
+ @fixtures ||=
+ Dir[File.join(FIXTURES_PATH, "*.md")]
+ .sort
+ .to_h { |path| [File.basename(path, ".md"), File.read(path, encoding: Encoding::UTF_8)] }
+ .freeze
+ end
+
+ # The bodies the round-trip properties hold for, in a stable order.
+ def self.round_trip_bodies
+ @round_trip_bodies ||=
+ fixtures
+ .except(REFUSED_FIXTURE)
+ .merge(CRLF_FIXTURE => fixtures.fetch(CRLF_SOURCE).gsub("\n", "\r\n"))
+ .freeze
+ end
+
+ # Building an engine context means a V8 isolate evaluating the whole engine
+ # bundle, so the file shares one; contexts are stateless across scans.
+ def self.markdown_engine
+ @markdown_engine ||=
+ Migrations::Converters::MarkdownEngine::Context.new(
+ bundle: Migrations::Converters::MarkdownEngine::Bundle.load_or_build,
+ config:
+ Migrations::Converters::MarkdownEngine::Config.new(
+ source_settings: {
+ "unicode_usernames" => true,
+ },
+ category_slugs: HASHTAG_NAMES,
+ tag_names: HASHTAG_NAMES,
+ custom_emoji_names: CUSTOM_EMOJI_NAMES,
+ ),
+ )
+ end
+
+ def self.close_markdown_engine
+ @markdown_engine&.close
+ @markdown_engine = nil
+ end
+
+ # Every lookup misses, so resolution has nothing but each row's recorded
+ # source to fall back on. A link to the source site is the exception the
+ # contract allows: its origin always becomes the destination's.
+ class AllMissMaps
+ %i[
+ badge
+ category_id
+ category_slug_path
+ emoji_name
+ event_markdown
+ group_name
+ poll_markdown
+ post
+ tag_id
+ tag_name
+ topic_id
+ upload
+ upload_markdown
+ user
+ ].each { |name| define_method(name) { |_key| nil } }
+
+ def base_url
+ BASE_URL
+ end
+
+ def here_mention
+ "here"
+ end
+ end
+end
+
+RSpec.configure { |config| config.after(:suite) { ExtractionRoundTrip.close_markdown_engine } }
+
+RSpec.shared_context "with extraction round trip" do
+ let(:enums) { Migrations::Database::IntermediateDB::Enums }
+ let(:owner_type) { enums::EmbedOwner::POST }
+
+ # Small enough that the fixture set spans several engine calls.
+ let(:batch_size) { 4 }
+
+ # Extracts each body into a real buffer, writes the embed rows the way a
+ # converter step does, and resolves the whole set back in one call.
+ #
+ # @param batched [Boolean] whether the engine-bound bodies go through
+ # `scan_batches` or are scanned one at a time inside `extract_prepared`.
+ def round_trip(bodies, batched: true)
+ with_intermediate_db { |db| extract_and_resolve(bodies, db, batched:) }
+ end
+
+ def extract_and_resolve(bodies, intermediate_db, batched:)
+ refusals = Hash.new(0)
+ # A fixed nonce makes the tokens of two runs comparable.
+ buffer =
+ Migrations::Converters::EmbedBuffer.new(
+ owner_type:,
+ placeholder: Migrations::Placeholder.new(nonce: "roundtrip"),
+ )
+ extractor = build_extractor(buffer, refusals)
+
+ names = bodies.keys
+ prepared =
+ bodies.map.with_index do |(_name, raw), index|
+ extractor.prepare(raw:, id: index + 1, topic_id: ExtractionRoundTrip::TOPIC_ID)
+ end
+ scan_data = batched ? extractor.scan_batches(prepared, max_posts: batch_size) : {}
+
+ extracted = {}
+ expected = {}
+ placeholders = {}
+ engine_bound = 0
+ live_scans = 0
+ items = []
+
+ prepared.each_with_index do |body, index|
+ name = names[index]
+ data = scan_data[body.id]
+ if body.engine_bound?
+ engine_bound += 1
+ live_scans += 1 if data.nil?
+ end
+
+ buffer.clear
+ output = extractor.extract_prepared(body, scan_data: data)
+ buffer.write_for(body.id) unless buffer.empty?
+
+ extracted[name] = output
+ expected[name] = expected_all_miss_body(output, buffer)
+ placeholders[name] = buffer.placeholders
+ items << { id: body.id, raw: output }
+ end
+
+ orphans = []
+ resolver =
+ Migrations::Importer::PlaceholderResolver.new(
+ intermediate_db,
+ ExtractionRoundTrip::AllMissMaps.new,
+ owner_type:,
+ unresolved_embeds: [],
+ orphan_placeholders: orphans,
+ )
+ resolved = resolver.resolve_all(items)
+
+ ExtractionRoundTrip::Outcome.new(
+ extracted:,
+ expected:,
+ resolved: names.each_with_index.to_h { |name, index| [name, resolved[index + 1]] },
+ placeholders:,
+ refusals:,
+ orphans:,
+ engine_bound:,
+ live_scans:,
+ )
+ end
+
+ def build_extractor(buffer, refusals)
+ Migrations::Converters::Discourse::RawExtractor.new(
+ embeds: buffer,
+ mention_names: compact_set(ExtractionRoundTrip::MENTION_NAMES),
+ hashtag_names: compact_set(ExtractionRoundTrip::HASHTAG_NAMES),
+ custom_emoji_names: ExtractionRoundTrip::CUSTOM_EMOJI_NAMES,
+ markdown_engine: ExtractionRoundTrip.markdown_engine,
+ internal_link_hosts: ExtractionRoundTrip::HOSTS,
+ internal_link_base_prefix: ExtractionRoundTrip::SOURCE_PREFIX,
+ mention_classifier:
+ Migrations::Converters::Discourse::MentionClassifier.new(
+ here_mention: "here",
+ group_names: ExtractionRoundTrip::GROUP_NAMES,
+ ),
+ on_engine_refusal: ->(cause, _detail) { refusals[cause] += 1 },
+ )
+ end
+
+ def compact_set(names)
+ Migrations::CompactStringSet.new(
+ names.map { |name| Migrations::NameNormalizer.normalize(name) },
+ )
+ end
+
+ def with_intermediate_db
+ Dir.mktmpdir("extraction-round-trip") do |dir|
+ db_path = File.join(dir, "intermediate.db")
+ Migrations::Database.migrate(
+ db_path,
+ migrations_path: Migrations::Database::INTERMEDIATE_DB_SCHEMA_PATH,
+ )
+ database = Migrations::Database.connect(db_path)
+ Migrations::Database::IntermediateDB.setup(database)
+ yield database
+ ensure
+ Migrations::Database::IntermediateDB.setup(nil)
+ database&.close
+ end
+ end
+
+ # The byte-exact expectation for one body: the extracted raw with every token
+ # put back the way an all-miss resolution has to put it back.
+ def expected_all_miss_body(output, buffer)
+ embed_rows(buffer).reduce(output) do |expected, (kind, row)|
+ replacement = expected_replacement(kind, row)
+ # The block form keeps a `\` in a snippet from being read as a
+ # backreference.
+ replacement ? expected.sub(row[:placeholder]) { replacement } : expected
+ end
+ end
+
+ def expected_replacement(kind, row)
+ if kind == :link && row[:target_type] == enums::LinkTarget::SITE
+ expected_site_markup(row)
+ elsif kind == :emoji
+ # An emoji row carries no snippet: its name is the source spelling.
+ ":#{row[:name]}:"
+ else
+ row[:original_markdown]
+ end
+ end
+
+ # A link to the source site has no entity to miss — only its origin moves —
+ # so the expectation splices the rewritten destination itself. The arithmetic
+ # is independent of the resolver's: the path comes from the row's own URL
+ # spelling rather than from `target_suffix`, so a wrongly recorded suffix
+ # still fails. A row without a usable span returns nil, which leaves the token
+ # in the expectation and fails the comparison loudly.
+ def expected_site_markup(row)
+ original = row[:original_markdown]
+ offset = row[:url_offset]
+ return nil if original.nil? || offset.nil?
+
+ length = row[:url].bytesize
+ return nil if offset < 0 || offset + length > original.bytesize
+
+ spans = [offset]
+ label = row[:label_url_offset]
+ spans << label if label && label != offset && label >= 0 && label + length <= original.bytesize
+
+ url = expected_site_url(row)
+ expected = original.dup
+ spans.sort.reverse_each { |span| expected.bytesplice(span, length, url) }
+ expected
+ end
+
+ def expected_site_url(row)
+ origin = Migrations::Converters::Discourse::MarkdownScanner::UrlOrigin
+ host, rest = origin.split(row[:url])
+ hosts = ExtractionRoundTrip::HOSTS
+ path =
+ if host && hosts.key?(host)
+ origin.path_within_prefix(rest, hosts[host])
+ else
+ row[:target_suffix]
+ end
+
+ "#{ExtractionRoundTrip::BASE_URL}#{path}"
+ end
+
+ def embed_rows(buffer)
+ {
+ quote: buffer.quotes,
+ link: buffer.links,
+ mention: buffer.mentions,
+ hashtag: buffer.hashtags,
+ emoji: buffer.emojis,
+ poll: buffer.polls,
+ event: buffer.events,
+ upload: buffer.uploads,
+ }.flat_map { |kind, rows| rows.map { |row| [kind, row] } }
+ end
+end