Skip to content

armgit / Filee

Public Imported from GitHub
0

Free, open-source file converter. Convert 263 formats offline by drag & drop — images, PDF, Word, Excel, PowerPoint, HWP/HWPX, video, audio, e-books and archives. No upload, no Office needed.

Text: CP949 on Korean Windows for any valid CP949, no charset= sniffing (#496 review)

cre8ive-marketinge748cc3289fc
diff --git a/docs/ENGINES.md b/docs/ENGINES.md
index 9c55b8eb..c09f275d 100644
--- a/docs/ENGINES.md
+++ b/docs/ENGINES.md
@@ -340,7 +340,7 @@ Readers turn the source into a small document model (`Hwp/Hwpx/HwpxModel.cs`) an
rows, repeat on every page, and rows above the print titles come once before them. The page is A4 with narrow margins,
landscape when the columns are much wider than portrait, and the table is scaled to the page width like
Excel's "Fit all columns on one page". CSV is read with its delimiter (comma, semicolon, tab) and encoding
- (UTF-8 / UTF-16 with BOM; else CP949 when it looks Korean, as Korean Excel saves it; else the PC's code page). Not converted: charts, pictures, conditional
+ (UTF-8 / UTF-16 with BOM; else CP949 when it looks Korean or on Korean Windows, as Korean Excel saves it; else the PC's code page). Not converted: charts, pictures, conditional
formatting, print areas and page breaks.
- **PPTX** (`Hwp/Hwpx/Pptx`): one page per slide, sized like the slide, with every object floating at its place:
placeholders inherit position and text style from the layout and master, text with bullets and numbering,
@@ -348,8 +348,9 @@ Readers turn the source into a small document model (`Hwp/Hwpx/HwpxModel.cs`) an
tables with PowerPoint's default table style, groups, master decorations and the slide background. Not
converted: charts, SmartArt, rotation, effects (shadows, glow), picture cropping, animations, notes and hidden
slides.
-- **TXT** needs no reader (one paragraph per line; UTF-8, UTF-16, CP949 when the bytes look Korean, else the PC's
- code page: GBK on Chinese Windows, Shift-JIS on Japanese, Windows-1252 on Western, as subtitles are read).
+- **TXT** needs no reader (one paragraph per line; UTF-8, UTF-16, CP949 when the bytes look Korean or, on Korean
+ Windows, are valid CP949, else the PC's code page: GBK on Chinese Windows, Shift-JIS on Japanese, Windows-1252 on
+ Western, as subtitles are read; a `charset=` in the text isn't followed).
- **Markdown** is parsed with Markdig (`MarkdownReader.cs`): headings, emphasis, code, lists (1. / a. / i. with
start numbers), task lists, quotes, tables with spans, links, images, footnotes and math. No Pandoc needed.
- **PDF** is read with PdfPig (see *PDF reader* below); DOCM / DOTX / DOTM, XLSM / XLTX and PPTM / POTX / POTM / PPSX
@@ -414,7 +415,7 @@ the text the spreadsheet shows, the value behind it and its Excel number format.
| XLSX | own reader: values, number formats, fonts, fills, borders, alignment, merges, widths, heights, hidden rows / columns / sheets, frozen rows, print titles (rows) | every sheet, numbers and dates as numbers with their formats, booleans, styles, merges, widths, heights, hidden rows / columns, frozen rows |
| XLS (Excel 97–2003, also Excel 5/95) | ExcelDataReader (MIT): values, number formats, merges, horizontal / vertical alignment; widths, heights and hidden rows / columns from the BIFF records (`XlsLayout.cs`) | no (LibreOffice) |
| ODS | own reader (`content.xml`, `styles.xml`, `settings.xml`): repeated rows / columns (the empty repeats up to the end of the sheet are skipped), spans, every value type, the text Calc shows (`text:p`), data styles as Excel codes, cell styles, widths, heights, hidden rows / columns / sheets, frozen and header rows | every sheet, typed cells (float, percentage, currency, date, time, boolean) with data styles and the shown text, styles, merges, widths, heights, hidden rows / columns, frozen rows |
-| CSV / TSV | delimiter (CSV: comma, semicolon or tab; TSV: tab) and encoding (UTF-8 / UTF-16 with BOM, CP949 when it looks Korean, else the PC's code page) | UTF-8 with BOM, CRLF, one file per sheet |
+| CSV / TSV | delimiter (CSV: comma, semicolon or tab; TSV: tab) and encoding (UTF-8 / UTF-16 with BOM, CP949 when it looks Korean or on Korean Windows, else the PC's code page) | UTF-8 with BOM, CRLF, one file per sheet |
- Number formats: Excel codes and OpenDocument data styles are converted both ways (`OdsNumberStyles.cs`) for
digits, decimals, grouping, thousands scaling, percent, scientific, currency symbols, literal text, colours,
diff --git a/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs b/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs
index 07e5fb00..a5dca8f5 100644
--- a/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs
+++ b/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs
@@ -2,6 +2,7 @@
// (HwpxConverter) and the DOCX writer (DocxWriterConverter) take their input formats from here, so a new reader is one
// line in BuiltIn and both writers (and every route through them) pick it up.
+using System.Globalization;
using System.Text;
using System.Text.Json.Nodes;
using Filee.Engines.Ebooks;
@@ -176,8 +177,13 @@ internal static class DocumentReaders
/// <summary>
/// Text in whatever encoding its PC used, the way subtitles are read (<see cref="Subtitles.SubtitleEncoding"/>): by
- /// its byte order mark, as UTF-8 when it is valid UTF-8, as CP949 when the bytes look Korean, else in the PC's own
- /// code page (GBK on Chinese Windows, #491; Shift-JIS on Japanese; Windows-1252 on Western).
+ /// its byte order mark, as UTF-8 when it is valid UTF-8, as CP949 when the bytes look Korean or, on Korean Windows,
+ /// are valid CP949; else in the PC's own code page (GBK on Chinese Windows, #491; Shift-JIS on Japanese; Windows-1252
+ /// on Western). A <c>charset=</c> in the text is content here, not the file's encoding, so it isn't followed.
/// </summary>
- internal static string DecodeText(byte[] bytes) => Subtitles.SubtitleEncoding.Decode(bytes);
+ internal static string DecodeText(byte[] bytes) => DecodeText(bytes, CultureInfo.CurrentCulture.TextInfo.ANSICodePage);
+
+ /// <summary><see cref="DecodeText(byte[])"/> with the PC's ANSI code page given (for tests).</summary>
+ internal static string DecodeText(byte[] bytes, int ansiCodePage) =>
+ Subtitles.SubtitleEncoding.Decode(bytes, ansiCodePage, declaredCharset: false);
}
diff --git a/src/Filee.Engines/Subtitles/SubtitleEncoding.cs b/src/Filee.Engines/Subtitles/SubtitleEncoding.cs
index f2708505..e35ede70 100644
--- a/src/Filee.Engines/Subtitles/SubtitleEncoding.cs
+++ b/src/Filee.Engines/Subtitles/SubtitleEncoding.cs
@@ -13,13 +13,18 @@ internal static partial class SubtitleEncoding
{
/// <summary>
/// The text of a subtitle file: by its byte order mark; as UTF-8 when the bytes are valid UTF-8; by the charset the
- /// file declares (SAMI, TTML); as CP949 when the bytes look like Korean (EUC-KR / CP949); as the PC's code page on
- /// Chinese and Japanese Windows; else as the PC's Western code page.
+ /// file declares (SAMI, TTML); as CP949 when the bytes look like Korean (EUC-KR / CP949), or on Korean Windows when
+ /// they are valid CP949 (symbols and Hanja too); as the PC's code page on Chinese and Japanese Windows; else as the
+ /// PC's Western code page.
/// </summary>
public static string Decode(byte[] bytes) => Decode(bytes, CultureInfo.CurrentCulture.TextInfo.ANSICodePage);
- /// <summary><see cref="Decode(byte[])"/> with the PC's ANSI code page given (for tests).</summary>
- internal static string Decode(byte[] bytes, int ansiCodePage)
+ /// <summary>
+ /// <see cref="Decode(byte[])"/> with the PC's ANSI code page given (for tests). <paramref name="declaredCharset"/>
+ /// false ignores a <c>charset=</c> / <c>encoding=</c> in the text: in a TXT, Markdown or CSV file that is content
+ /// (a line of code, a quoted HTML tag), not what the file is saved in.
+ /// </summary>
+ internal static string Decode(byte[] bytes, int ansiCodePage, bool declaredCharset = true)
{
if (bytes is [0xEF, 0xBB, 0xBF, ..])
return new UTF8Encoding(false).GetString(bytes, 3, bytes.Length - 3);
@@ -37,15 +42,32 @@ internal static partial class SubtitleEncoding
}
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
- if (Declared(bytes) is { } declared)
+ if (declaredCharset && Declared(bytes) is { } declared)
return declared.GetString(bytes);
- if (LooksKorean(bytes))
+ if (LooksKorean(bytes) || (ansiCodePage == 949 && IsCp949(bytes)))
return Encoding.GetEncoding(949).GetString(bytes);
if (ansiCodePage is 932 or 936 or 950)
return Encoding.GetEncoding(ansiCodePage).GetString(bytes);
return Encoding.GetEncoding(ansiCodePage is > 0 and not (949 or 1361) and not 65001 ? ansiCodePage : 1252).GetString(bytes);
}
+ /// <summary>
+ /// True when every byte decodes as CP949 without a substitute. Western text fails here (an accented letter followed
+ /// by a space or punctuation is no CP949 pair), so a Korean PC still reads a Windows-1252 file as Windows-1252.
+ /// </summary>
+ private static bool IsCp949(byte[] bytes)
+ {
+ try
+ {
+ Encoding.GetEncoding(949, EncoderFallback.ReplacementFallback, DecoderFallback.ExceptionFallback).GetString(bytes);
+ return true;
+ }
+ catch (DecoderFallbackException)
+ {
+ return false;
+ }
+ }
+
/// <summary>The encoding named by <c>charset=…</c> (SAMI's HTML) or <c>encoding="…"</c> (XML) near the start.</summary>
private static Encoding? Declared(byte[] bytes)
{
diff --git a/tests/Filee.Engines.Tests/HwpxWriterTests.cs b/tests/Filee.Engines.Tests/HwpxWriterTests.cs
index 1b456d57..ef92c7a8 100644
--- a/tests/Filee.Engines.Tests/HwpxWriterTests.cs
+++ b/tests/Filee.Engines.Tests/HwpxWriterTests.cs
@@ -134,18 +134,34 @@ public class HwpxWriterTests(EngineFixture fx) : IClassFixture<EngineFixture>
Assert.Equal(Text, HwpxConverter.DecodeText(Encoding.Unicode.GetPreamble().Concat(Encoding.Unicode.GetBytes(Text)).ToArray()));
}
- [Fact]
- public void Chinese_text_files_are_not_read_as_korean()
+ [Theory]
+ [InlineData(936, "我是一个学生,这是我的文件。")] // #491: GBK, "ANSI" on Chinese Windows
+ [InlineData(950, "這是繁體中文的檔案。")]
+ [InlineData(932, "これは日本語のファイルです。")]
+ [InlineData(1252, "Café, déjà vu à l'été")]
+ public void Text_files_are_read_in_the_pcs_own_code_page(int codePage, string text)
{
- // #491: a GBK ("ANSI" on Chinese Windows) TXT came out as Hangul. It is read like a subtitle now: GBK on a
- // Chinese PC, never CP949 unless the bytes look Korean.
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
- var gbk = Encoding.GetEncoding(936).GetBytes("我是一个学生,这是我的文件。");
+ Assert.Equal(text, DocumentReaders.DecodeText(Encoding.GetEncoding(codePage).GetBytes(text), codePage));
+ }
- var text = HwpxConverter.DecodeText(gbk);
+ [Theory]
+ [InlineData("■ Item one\n※ Note\n● Point ①")]
+ [InlineData("大韓民國 憲法 第一條")]
+ [InlineData("한글 메모 <meta charset=\"iso-8859-1\"> encoding=\"cp1252\"")]
+ public void Korean_windows_reads_any_valid_cp949_text_as_cp949(string text)
+ {
+ // Symbols and Hanja aren't "mostly Hangul", and a charset= in the text is content, not the file's encoding.
+ Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
+ Assert.Equal(text, DocumentReaders.DecodeText(Encoding.GetEncoding(949).GetBytes(text), 949));
+ }
- Assert.Equal(Filee.Engines.Subtitles.SubtitleEncoding.Decode(gbk), text);
- Assert.DoesNotContain(text, c => c is >= '가' and <= '힣');
+ [Fact]
+ public void Korean_windows_still_reads_a_western_file_as_windows_1252()
+ {
+ Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
+ const string Text = "Café, déjà vu à l'été. Señor Müller";
+ Assert.Equal(Text, DocumentReaders.DecodeText(Encoding.GetEncoding(1252).GetBytes(Text), 949));
}
[Fact]