Corpus statistics

stats

Corpus statistics utilities.

Examples:

>>> from openodia import FreqDist, ngrams, collocations, cooccurrence
>>> tokens = ["ରାମ", "ସୀତା", "ରାମ", "ଲକ୍ଷ୍ମଣ"]
>>> FreqDist(tokens).most_common(2)
[('ରାମ', 2), ('ସୀତା', 1)]
>>> list(ngrams(tokens, 2))
[('ରାମ', 'ସୀତା'), ('ସୀତା', 'ରାମ'), ('ରାମ', 'ଲକ୍ଷ୍ମଣ')]

FreqDist

Bases: Counter[str]

Token frequency distribution backed by collections.Counter.

Adds three light convenience methods:

• hapaxes — words occurring exactly once.
• entropy — Shannon entropy of the distribution in bits.
• ttr — type-token ratio (vocabulary / total count).

All other Counter operations (most_common, +, &, ...) work unchanged.

total_count property

total_count: int

Sum of all counts (i.e. token count, not vocabulary size).

ttr property

ttr: float

Type-token ratio. 0.0 for an empty distribution.

hapaxes

hapaxes() -> list[str]

Words occurring exactly once, in insertion order.

entropy

entropy() -> float

Shannon entropy of the distribution, in bits.

Returns 0.0 for an empty distribution.

collocations

collocations(
    tokens_or_text: Sequence[str] | str,
    top_k: int = 20,
    min_count: int = 2,
    score: CollocationScore = "pmi",
) -> list[tuple[tuple[str, str], float]]

Top-scoring bigram collocations.

Uses pointwise mutual information (PMI), log2(P(w1, w2) / (P(w1) * P(w2))).

Parameters:

Name	Type	Description	Default
tokens_or_text	Sequence[str] | str	Pre-tokenized sequence or raw text.	required
top_k	int	Maximum number of collocations to return.	20
min_count	int	Bigrams occurring fewer than this many times are skipped — PMI is unstable on very rare pairs. Must be ≥ 1.	2
score	CollocationScore	Scoring method. Only "pmi" is supported in v1.	'pmi'

Returns:

Type	Description
list[tuple[tuple[str, str], float]]	List of ((w1, w2), score) sorted descending by score.
list[tuple[tuple[str, str], float]]	Empty when input is too short or no bigram meets min_count.

Raises:

Type	Description
ValueError	If top_k < 1, min_count < 1, or score is unknown.

cooccurrence

cooccurrence(
    tokens_or_text: Sequence[str] | str,
    window: int = 5,
    symmetric: bool = True,
) -> Counter[tuple[str, ...]]

Count word co-occurrences within a sliding window.

Parameters:

Name	Type	Description	Default
tokens_or_text	Sequence[str] | str	Pre-tokenized sequence or raw text.	required
window	int	Maximum number of positions between two tokens for them to be considered co-occurring. Must be ≥ 1.	5
symmetric	bool	If True (default), the returned keys are sorted tuples and order is ignored — ("a", "b") and ("b", "a") collapse. If False, keys are (earlier, later) so direction is preserved.	True

Returns:

Type	Description
Counter[tuple[str, ...]]	collections.Counter mapping pair → co-occurrence count.

Raises:

Type	Description
ValueError	If window < 1.

ngrams

ngrams(
    tokens_or_text: Sequence[str] | str, n: int
) -> Iterator[tuple[str, ...]]

Yield overlapping n-grams from tokens_or_text.

Parameters:

Name	Type	Description	Default
tokens_or_text	Sequence[str] | str	Pre-tokenized list/sequence, or a raw string (in which case the package's word tokenizer is applied).	required
n	int	N-gram size. Must be ≥ 1.	required

Yields:

Type	Description
str	Tuples of length n. Yields nothing if the token count is below
...	n.

Raises:

Type	Description
ValueError	If n < 1.