Tokenizing, stopwords and language detection

ud

Tokenizer implementation

word_tokenizer classmethod

word_tokenizer(text: str) -> list[str]

Split the text into words

sentence_tokenizer classmethod

sentence_tokenizer(text: str) -> list[str]

Split the text into sentences at ।, ॥, ? and !, keeping each terminator.

remove_stopwords classmethod

remove_stopwords(
    text: str | list[str],
    get_str: Literal[False] = False,
    stopwords: Container[str] | None = None,
) -> list[str]

remove_stopwords(
    text: str | list[str],
    get_str: Literal[True],
    stopwords: Container[str] | None = None,
) -> str

remove_stopwords(
    text: str | list[str],
    get_str: bool,
    stopwords: Container[str] | None = None,
) -> list[str] | str

remove_stopwords(
    text: str | list[str],
    get_str: bool = False,
    stopwords: Container[str] | None = None,
) -> list[str] | str

Remove frequently used words from the text.

Parameters:

Name	Type	Description	Default
text	str | list[str]	It can take both tokens and text string as input.	required
get_str	bool	Whether the output should be a single string instead of a list of tokens.	False
stopwords	Container[str] | None	Optional custom container. Anything that supports token in stopwords works: openodia.Stopwords, frozenset[str], set[str], list[str], etc. Defaults to the bundled openodia.STOPWORDS.	None

detect_language classmethod

detect_language(
    text: str, threshold: float = 0.5
) -> dict[str, Any]

Detect language if it is Odia or non-Odia

Parameters:

Name	Type	Description	Default
text	str	text to detect language	required
threshold	float	confidence score which if crossed will be determined as Odia	0.5

stopwords

Customizable stopword lists for Odia text.

The package's bundled list is exposed as openodia.STOPWORDS (a frozenset). For pipelines that need to extend / shrink / swap the list per call, use Stopwords.

Examples:

>>> from openodia import Stopwords
>>> sw = Stopwords.default()
>>> "ଓ" in sw
True
>>> sw.add("ପ୍ରାୟ").remove("ଓ")
>>> "ଓ" in sw
False

Stopwords

A mutable, per-pipeline stopword list.

Internally backed by a set[str] so membership tests are O(1).

Parameters:

Name	Type	Description	Default
words	Iterable[str]	Initial words. Strings are added as-is (each as one stopword).	()

default classmethod

default() -> Stopwords

Return a new instance seeded with the package's bundled list.

from_file classmethod

from_file(
    path: Path | str, encoding: str = "utf-8-sig"
) -> Stopwords

Load stopwords from a text file (one word per line).

Blank lines and lines starting with # are skipped.

Parameters:

Name	Type	Description	Default
path	Path | str	File to read.	required
encoding	str	File encoding. Defaults to UTF-8, with a leading BOM ignored.	'utf-8-sig'

Raises:

Type	Description
FileNotFoundError	If path does not exist.

from_corpus classmethod

from_corpus(
    tokens: Iterable[str], top_n: int = 100
) -> Stopwords

Derive a stopword list from the highest-frequency tokens.

Parameters:

Name	Type	Description	Default
tokens	Iterable[str]	Pre-tokenized input.	required
top_n	int	How many of the most common tokens to keep. Must be ≥ 1.	100

Raises:

Type	Description
ValueError	If top_n is not positive.

add

add(words: str | Iterable[str]) -> Stopwords

Add one or more words. Returns self for chaining.

remove

remove(words: str | Iterable[str]) -> Stopwords

Drop one or more words. Missing words are ignored.

Returns self for chaining.

save

save(path: Path | str, encoding: str = 'utf-8') -> None

Write stopwords to a file, sorted, one per line.

coverage

coverage(tokens: Iterable[str]) -> float

Fraction of tokens that are stopwords.

Returns 0.0 for an empty input.