Text normalization and cleanup

normalize

normalize(text: str, form: UnicodeForm = 'NFC') -> str

Return text in the requested Unicode normalization form.

Parameters:

Name	Type	Description	Default
text	str	Input string.	required
form	UnicodeForm	One of "NFC", "NFD", "NFKC", "NFKD". Defaults to "NFC" (the right choice for Indic Unicode).	'NFC'

Returns:

Type	Description
str	The normalized string.

Raises:

Type	Description
ValueError	If form is not one of the four standard forms.

clean

clean(
    text: str, options: CleanOptions | None = None
) -> str

Return text with common Odia-specific anomalies removed.

Always applied (unless disabled in options):

• Unicode normalization (NFC by default).
• ZWJ / ZWNJ stripping.
• Whitespace collapsing.

Off by default:

• Latin ↔ Odia digit conversion.

Parameters:

Name	Type	Description	Default
text	str	Input string.	required
options	CleanOptions | None	Cleanup configuration. None uses DEFAULT_CLEAN.	None

Returns:

Type	Description
str	The cleaned string. clean(clean(x)) == clean(x) for any x.

CleanOptions dataclass

Options for clean.

Attributes:

Name	Type	Description
form	UnicodeForm	Unicode normalization form applied first. Defaults to "NFC".
strip_zwj	bool	Drop zero-width joiner (U+200D).
strip_zwnj	bool	Drop zero-width non-joiner (U+200C).
collapse_whitespace	bool	Replace runs of whitespace with a single space and trim leading/trailing whitespace.
latin_to_odia_digits	bool	Convert ASCII digits 0-9 to Odia digits.
odia_to_latin_digits	bool	Convert Odia digits ୦-୯ to ASCII digits. Mutually exclusive with latin_to_odia_digits.