The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
fix: address review feedback on Bengali language support
Md Kishor Morol45a9780118b0
diff --git a/agents/bengali-reviewer.md b/agents/bengali-reviewer.mdindex 2e03fe64c..48734338b 100644--- a/agents/bengali-reviewer.md+++ b/agents/bengali-reviewer.md@@ -1,7 +1,7 @@---name: bengali-reviewerdescription: Reviews code handling Bengali (Bangla) text for Unicode correctness, proper normalization, script-aware processing, and internationalization best practices. Use when code processes, displays, or stores Bengali text.-tools: Read, Grep, Glob, Bash+tools: Read, Grep, Globmodel: sonnet---@@ -24,7 +24,7 @@ When invoked:2. **Check Unicode correctness** — Verify NFC normalization, correct handling of conjunct consonants (যুক্তবর্ণ), vowel signs (কার), and Hasanta (্).3. **Validate text processing** — Check tokenization, search, sort, and comparison operations for Bengali-awareness.4. **Review rendering and display** — Ensure proper font support, line breaking, and input handling for Bengali script.-5. **Check data layer** — Verify database columns use UTF-8/UTF-8MB4, API responses include proper charset headers, and file I/O specifies encoding.+5. **Check data layer** — Verify database columns use Unicode-capable encoding (utf8mb4 for MySQL, UTF-8 for PostgreSQL/SQLite), API responses include proper charset headers (`charset=utf-8`), and file I/O specifies encoding.6. **Report findings** — Use the output format below. Only report issues you are confident about (>80% sure it is a real problem).## Confidence-Based Filtering@@ -91,7 +91,9 @@ zero rows and verdict `APPROVE`.- **Missing NFC normalization before storage** — Bengali text stored withoutnormalization will cause duplicate records and failed lookups.- **Non-Unicode database encoding** — MySQL columns using `latin1` or `ascii`- charset for Bengali text will silently corrupt or truncate data.+ charset instead of `utf8mb4` for Bengali text will silently corrupt or+ truncate data. For PostgreSQL, verify UTF-8 database encoding. For SQLite,+ ensure text is handled as UTF-8.- **File I/O without encoding** — Reading/writing Bengali text files withoutexplicit UTF-8 encoding on platforms where the default is not UTF-8.@@ -130,9 +132,11 @@ if re.match(r'^[a-zA-Z\u0980-\u09FF]+$', name):### MEDIUM — Quality Issues-- **Zero-width characters not stripped** — Web-scraped Bengali text often contains- ZWNJ (U+200C), ZWJ (U+200D), and other invisible characters that cause- matching failures if not cleaned.+- **Unwanted zero-width characters** — Web-scraped Bengali text often contains+ invisible characters like ZWSP (U+200B) and BOM (U+FEFF) that cause matching+ failures. Note: preserve ZWNJ (U+200C) and ZWJ (U+200D) by default as they+ are orthographically significant for Bengali conjuncts — only remove them when+ an explicit input policy identifies the sequence as invalid.- **Hardcoded strings** — Bengali UI text embedded in source code instead ofi18n resource files.- **Missing locale in formatting** — Dates, numbers, and currencies displayed@@ -167,21 +171,25 @@ Fix: Use grapheme cluster segmentation via `regex.findall(r"\X", text)` or### Summary Format-End every review with:+End every review with a summary table. Calculate each severity count from your+actual findings and derive the verdict accordingly. Example format:```## Bengali Text Review Summary| Severity | Count | Status ||----------|-------|--------|-| CRITICAL | 0 | pass |-| HIGH | 2 | warn |-| MEDIUM | 1 | info |-| LOW | 1 | note |+| CRITICAL | <n> | pass/fail |+| HIGH | <n> | pass/warn |+| MEDIUM | <n> | pass/info |+| LOW | <n> | pass/note |-Verdict: WARNING — 2 HIGH issues should be resolved before merge.+Verdict: APPROVE / WARNING / BLOCK — <reasoning based on actual findings>```+A clean review with zero findings across all severities is valid — use+verdict `APPROVE` with status `pass` for all rows.+## Approval Criteria- **Approve**: No CRITICAL or HIGH issues, including clean reviews with zerodiff --git a/docs/bn/README.md b/docs/bn/README.mdindex feb4771ea..c5c2cb216 100644--- a/docs/bn/README.md+++ b/docs/bn/README.md@@ -1,4 +1,4 @@-**ভাষা:** [English](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | বাংলা+**ভাষা:** [English](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [Español](../es/README.md) | বাংলা# Everything Claude Code@@ -15,7 +15,7 @@**Language / ভাষা / 语言 / 語言 / 언어 / Dil / Язык / Ngôn ngữ**-[**English**](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [বাংলা](README.md)+[**English**](../../README.md) | [Português (Brasil)](../pt-BR/README.md) | [简体中文](../../README.zh-CN.md) | [繁體中文](../zh-TW/README.md) | [日本語](../ja-JP/README.md) | [한국어](../ko-KR/README.md) | [Türkçe](../tr/README.md) | [Русский](../ru/README.md) | [Tiếng Việt](../vi-VN/README.md) | [ไทย](../th/README.md) | [Deutsch](../de-DE/README.md) | [Español](../es/README.md) | [বাংলা](README.md)</div>diff --git a/skills/bengali-nlp/SKILL.md b/skills/bengali-nlp/SKILL.mdindex 020e785f9..8a3dd4fd1 100644--- a/skills/bengali-nlp/SKILL.md+++ b/skills/bengali-nlp/SKILL.md@@ -27,8 +27,8 @@ Bengali script occupies U+0980–U+09FF. Key character classes:```U+0981–U+0983 Chandrabindu, Anusvara, Visarga-U+0985–U+0990 Independent vowels (অ আ ই ঈ উ ঊ ঋ এ ঐ)-U+0993–U+09B0 Consonants (ও–র)+U+0985–U+0994 Independent vowels (অ আ ই ঈ উ ঊ ঋ ঌ এ ঐ ও ঔ)+U+0995–U+09B0 Consonants (ক–র)U+09B2 লU+09B6–U+09B9 শ ষ স হU+09BC Nukta@@ -195,23 +195,20 @@ BANGLA_STOP_WORDS = {Bengali has a defined sort order (স্বরবর্ণ before ব্যঞ্জনবর্ণ). Use locale-aware sorting:```python-import locale--# Set Bengali locale if available+# Preferred: use PyICU for reliable Bengali collation (pip install PyICU)try:- locale.setlocale(locale.LC_COLLATE, "bn_BD.UTF-8")- sorted_words = sorted(words, key=locale.strxfrm)-except locale.Error:- # Fallback: use ICU via PyICUimport icucollator = icu.Collator.createInstance(icu.Locale("bn_BD"))sorted_words = sorted(words, key=collator.getSortKey)+except ImportError:+ # Fallback: Unicode code point order (not linguistically perfect)+ sorted_words = sorted(words)``````javascript// JavaScript — Intl.Collator handles Bengali sort orderconst collator = new Intl.Collator('bn-BD');-words.sort(collator.compare);+const sortedWords = [...words].sort(collator.compare);```### Search with Normalization@@ -232,6 +229,11 @@ def bangla_search(query: str, corpus: list[str]) -> list[str]:### Prompt Engineering for Bengali+**Important:** Bengali text, Banglish input, and web-scraped content should be+treated as untrusted user data. Always separate system instructions from+user-provided content with clear delimiters, and validate before executing+tool calls or destructive actions.+```python# Use Bengali system prompts for Bengali-targeted appssystem_prompt = (@@ -245,6 +247,13 @@ system_prompt_mixed = ("respond in Bengali. When they write in English, respond in English. ""For technical terms, you may use English words within Bengali sentences.")++# Always delimit user-provided content+def build_prompt(system: str, user_input: str) -> str:+ return f"{system}+---USER INPUT---+{user_input}+---END INPUT---"```### Handling Banglish (Mixed Bengali-English)@@ -277,8 +286,8 @@ def prepare_bangla_dataset(texts: list[str]) -> list[str]:for text in texts:# Normalize Unicodetext = unicodedata.normalize("NFC", text)- # Remove zero-width characters (common in web-scraped Bengali)- text = re.sub(r'[\u200b-\u200f\u2028-\u202f\ufeff]', '', text)+ # Remove unwanted zero-width characters but preserve ZWNJ/ZWJ (important for conjuncts)+ text = re.sub(r'[\u200b\u200e\u200f\u2028-\u202f\ufeff]', '', text)# Normalize whitespacetext = re.sub(r'\s+', ' ', text).strip()if text:@@ -296,7 +305,10 @@ CREATE TABLE bengali_content (body TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci);--- PostgreSQL: UTF-8 is the default, but set collation for sorting+-- PostgreSQL: UTF-8 is the default. Verify ICU collation availability first:+-- SELECT collname FROM pg_collation WHERE collname LIKE 'bn%';+-- If missing, create it (requires ICU-enabled PostgreSQL build):+-- CREATE COLLATION IF NOT EXISTS "bn-BD-x-icu" (provider = icu, locale = 'bn-BD');CREATE TABLE bengali_content (id SERIAL PRIMARY KEY,title TEXT COLLATE "bn-BD-x-icu",@@ -338,7 +350,7 @@ if text.isdigit(): ...# GOOD — handles bothimport re-if re.match(r'^[\d০-৯]+$', text): ...+if re.fullmatch(r'[0-9০-৯]+', text): ...```### Don't: Use ASCII Transliteration When Unicode Is Available@@ -357,6 +369,6 @@ title = "বাংলা ভাষা"# BAD — invisible characters cause matching failurestext = scraped_html.get_text()-# GOOD — strip zero-width characters after extraction-text = re.sub(r'[\u200b-\u200f\u2028-\u202f\ufeff]', '', scraped_html.get_text())+# GOOD — strip unwanted zero-width chars, preserving ZWNJ (U+200C) and ZWJ (U+200D)+text = re.sub(r'[\u200b\u200e\u200f\u2028-\u202f\ufeff]', '', scraped_html.get_text())```