Skip to content

armgit / Filee

Public Imported from GitHub
0

Free, open-source file converter. Convert 263 formats offline by drag & drop — images, PDF, Word, Excel, PowerPoint, HWP/HWPX, video, audio, e-books and archives. No upload, no Office needed.

Read ANSI text in the PC's own code page, not always as Korean (#491)

cre8ive-marketinga6da88225d59
diff --git a/docs/ENGINES.md b/docs/ENGINES.md
index 616f8222..f1a0cf3e 100644
--- a/docs/ENGINES.md
+++ b/docs/ENGINES.md
@@ -333,7 +333,7 @@ Readers turn the source into a small document model (`Hwp/Hwpx/HwpxModel.cs`) an
rows, repeat on every page, and rows above the print titles come once before them. The page is A4 with narrow margins,
landscape when the columns are much wider than portrait, and the table is scaled to the page width like
Excel's "Fit all columns on one page". CSV is read with its delimiter (comma, semicolon, tab) and encoding
- (UTF-8 / UTF-16 with BOM, else CP949 as saved by Korean Excel). Not converted: charts, pictures, conditional
+ (UTF-8 / UTF-16 with BOM; else CP949 when it looks Korean, as Korean Excel saves it; else the PC's code page). Not converted: charts, pictures, conditional
formatting, print areas and page breaks.
- **PPTX** (`Hwp/Hwpx/Pptx`): one page per slide, sized like the slide, with every object floating at its place:
placeholders inherit position and text style from the layout and master, text with bullets and numbering,
@@ -341,7 +341,8 @@ Readers turn the source into a small document model (`Hwp/Hwpx/HwpxModel.cs`) an
tables with PowerPoint's default table style, groups, master decorations and the slide background. Not
converted: charts, SmartArt, rotation, effects (shadows, glow), picture cropping, animations, notes and hidden
slides.
-- **TXT** needs no reader (one paragraph per line; UTF-8, UTF-16 or the Korean code page CP949).
+- **TXT** needs no reader (one paragraph per line; UTF-8, UTF-16, CP949 when the bytes look Korean, else the PC's
+ code page: GBK on Chinese Windows, Shift-JIS on Japanese, Windows-1252 on Western, as subtitles are read).
- **Markdown** is parsed with Markdig (`MarkdownReader.cs`): headings, emphasis, code, lists (1. / a. / i. with
start numbers), task lists, quotes, tables with spans, links, images, footnotes and math. No Pandoc needed.
- **PDF** is read with PdfPig (see *PDF reader* below); DOCM / DOTX / DOTM, XLSM / XLTX and PPTM / POTX / POTM / PPSX
@@ -406,7 +407,7 @@ the text the spreadsheet shows, the value behind it and its Excel number format.
| XLSX | own reader: values, number formats, fonts, fills, borders, alignment, merges, widths, heights, hidden rows / columns / sheets, frozen rows, print titles (rows) | every sheet, numbers and dates as numbers with their formats, booleans, styles, merges, widths, heights, hidden rows / columns, frozen rows |
| XLS (Excel 97–2003, also Excel 5/95) | ExcelDataReader (MIT): values, number formats, merges, horizontal / vertical alignment; widths, heights and hidden rows / columns from the BIFF records (`XlsLayout.cs`) | no (LibreOffice) |
| ODS | own reader (`content.xml`, `styles.xml`, `settings.xml`): repeated rows / columns (the empty repeats up to the end of the sheet are skipped), spans, every value type, the text Calc shows (`text:p`), data styles as Excel codes, cell styles, widths, heights, hidden rows / columns / sheets, frozen and header rows | every sheet, typed cells (float, percentage, currency, date, time, boolean) with data styles and the shown text, styles, merges, widths, heights, hidden rows / columns, frozen rows |
-| CSV / TSV | delimiter (CSV: comma, semicolon or tab; TSV: tab) and encoding (UTF-8 / UTF-16 with BOM, else CP949) | UTF-8 with BOM, CRLF, one file per sheet |
+| CSV / TSV | delimiter (CSV: comma, semicolon or tab; TSV: tab) and encoding (UTF-8 / UTF-16 with BOM, CP949 when it looks Korean, else the PC's code page) | UTF-8 with BOM, CRLF, one file per sheet |
- Number formats: Excel codes and OpenDocument data styles are converted both ways (`OdsNumberStyles.cs`) for
digits, decimals, grouping, thousands scaling, percent, scientific, currency symbols, literal text, colours,
diff --git a/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs b/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs
index b98e78d0..6a4c27a9 100644
--- a/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs
+++ b/src/Filee.Engines/Hwp/Hwpx/DocumentReaders.cs
@@ -170,20 +170,10 @@ internal static class DocumentReaders
return document;
}
- /// <summary>UTF-8 (with or without BOM) or UTF-16 with BOM; anything else is read as CP949, common for Korean text.</summary>
- internal static string DecodeText(byte[] bytes)
- {
- if (bytes is [0xFF, 0xFE, ..] or [0xFE, 0xFF, ..])
- return new StreamReader(new MemoryStream(bytes), detectEncodingFromByteOrderMarks: true).ReadToEnd();
- try
- {
- var text = new UTF8Encoding(false, throwOnInvalidBytes: true).GetString(bytes);
- return text.Length > 0 && text[0] == '' ? text[1..] : text;
- }
- catch (DecoderFallbackException)
- {
- Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
- return Encoding.GetEncoding(949).GetString(bytes);
- }
- }
+ /// <summary>
+ /// Text in whatever encoding its PC used, the way subtitles are read (<see cref="Subtitles.SubtitleEncoding"/>): by
+ /// its byte order mark, as UTF-8 when it is valid UTF-8, as CP949 when the bytes look Korean, else in the PC's own
+ /// code page (GBK on Chinese Windows, #491; Shift-JIS on Japanese; Windows-1252 on Western).
+ /// </summary>
+ internal static string DecodeText(byte[] bytes) => Subtitles.SubtitleEncoding.Decode(bytes);
}
diff --git a/src/Filee.Engines/Hwp/Hwpx/HwpxConverter.cs b/src/Filee.Engines/Hwp/Hwpx/HwpxConverter.cs
index bf6f570e..6ceeb648 100644
--- a/src/Filee.Engines/Hwp/Hwpx/HwpxConverter.cs
+++ b/src/Filee.Engines/Hwp/Hwpx/HwpxConverter.cs
@@ -59,6 +59,6 @@ public sealed class HwpxConverter : IConverter
/// <summary>Plain text: one paragraph per line (see <see cref="DocumentReaders.TextDocument"/>).</summary>
internal static HDocument TextDocument(byte[] bytes) => DocumentReaders.TextDocument(bytes);
- /// <summary>UTF-8, UTF-16 with BOM or CP949 (see <see cref="DocumentReaders.DecodeText"/>).</summary>
+ /// <summary>Text in the encoding its PC used (see <see cref="DocumentReaders.DecodeText"/>).</summary>
internal static string DecodeText(byte[] bytes) => DocumentReaders.DecodeText(bytes);
}
diff --git a/tests/Filee.Engines.Tests/HwpxWriterTests.cs b/tests/Filee.Engines.Tests/HwpxWriterTests.cs
index 99cd3504..1b456d57 100644
--- a/tests/Filee.Engines.Tests/HwpxWriterTests.cs
+++ b/tests/Filee.Engines.Tests/HwpxWriterTests.cs
@@ -134,6 +134,20 @@ public class HwpxWriterTests(EngineFixture fx) : IClassFixture<EngineFixture>
Assert.Equal(Text, HwpxConverter.DecodeText(Encoding.Unicode.GetPreamble().Concat(Encoding.Unicode.GetBytes(Text)).ToArray()));
}
+ [Fact]
+ public void Chinese_text_files_are_not_read_as_korean()
+ {
+ // #491: a GBK ("ANSI" on Chinese Windows) TXT came out as Hangul. It is read like a subtitle now: GBK on a
+ // Chinese PC, never CP949 unless the bytes look Korean.
+ Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
+ var gbk = Encoding.GetEncoding(936).GetBytes("我是一个学生,这是我的文件。");
+
+ var text = HwpxConverter.DecodeText(gbk);
+
+ Assert.Equal(Filee.Engines.Subtitles.SubtitleEncoding.Decode(gbk), text);
+ Assert.DoesNotContain(text, c => c is >= '가' and <= '힣');
+ }
+
[Fact]
public void Docx_is_written_by_the_built_in_writer_in_one_step()
{