core_lens.base.namespaces.stats

Statistical analysis namespace for core_lens Result objects.

Exceptions

CorrelationError

Raised when correlation cannot be computed (e.g. fewer than 2 columns).

Classes

CorrelateMethod	Correlation methods.
TestMethod	Statistical hypothesis testing methods.
ChangeMethod	Methods for calculating change over time.
AnomalyCrossMethod	Methods for detecting cross-sectional anomalies.
AnomalyTsMethod	Methods for detecting time-series anomalies.
SimilarityMethod	Methods for calculating similarity or distance.
StatsNamespace	Statistical methods on result.stats.*.

Module Contents

class core_lens.base.namespaces.stats.CorrelateMethod(*args, **kwds)

Bases: enum.Enum

Correlation methods.

Variables:

• PEARSON – Pearson correlation coefficient.

• SPEARMAN – Spearman rank correlation.

• KENDALL – Kendall Tau correlation.

PEARSON = 'pearson'

SPEARMAN = 'spearman'

KENDALL = 'kendall'

class core_lens.base.namespaces.stats.TestMethod(*args, **kwds)

Bases: enum.Enum

Statistical hypothesis testing methods.

Variables:

• T_TEST – Student’s t-test.

• MANN_WHITNEY – Mann-Whitney U test.

• WILCOXON – Wilcoxon signed-rank test.

• KS – Kolmogorov-Smirnov test.

• CHI_SQUARE – Chi-square test.

__test__ = False

T_TEST = 't-test'

MANN_WHITNEY = 'mann-whitney'

WILCOXON = 'wilcoxon'

KS = 'ks'

CHI_SQUARE = 'chi-square'

class core_lens.base.namespaces.stats.ChangeMethod(*args, **kwds)

Bases: enum.Enum

Methods for calculating change over time.

Variables:

• ABSOLUTE – Absolute difference between periods.

• PERCENTAGE – Percentage change between periods.

• TREND – Linear trend over time.

ABSOLUTE = 'absolute'

PERCENTAGE = 'percentage'

TREND = 'trend'

class core_lens.base.namespaces.stats.AnomalyCrossMethod(*args, **kwds)

Bases: enum.Enum

Methods for detecting cross-sectional anomalies.

Variables:

• ZSCORE – Z-score method.

• IQR – Interquartile range method.

• PERCENTILE – Percentile-based method.

• THRESHOLD – Fixed threshold method.

ZSCORE = 'zscore'

IQR = 'iqr'

PERCENTILE = 'percentile'

THRESHOLD = 'threshold'

class core_lens.base.namespaces.stats.AnomalyTsMethod(*args, **kwds)

Bases: enum.Enum

Methods for detecting time-series anomalies.

Variables:

• STL – Seasonal-Trend decomposition using LOESS.

• CUSUM – Cumulative sum control chart.

• MAD – Median Absolute Deviation.

STL = 'stl'

CUSUM = 'cusum'

MAD = 'mad'

class core_lens.base.namespaces.stats.SimilarityMethod(*args, **kwds)

Bases: enum.Enum

Methods for calculating similarity or distance.

Variables:

• EUCLIDEAN – Euclidean distance.

• COSINE – Cosine similarity.

• MAHALANOBIS – Mahalanobis distance.

• MANHATTAN – Manhattan distance.

EUCLIDEAN = 'euclidean'

COSINE = 'cosine'

MAHALANOBIS = 'mahalanobis'

MANHATTAN = 'manhattan'

exception core_lens.base.namespaces.stats.CorrelationError

Bases: ValueError

Raised when correlation cannot be computed (e.g. fewer than 2 columns).

class core_lens.base.namespaces.stats.StatsNamespace(result: core_lens.base.result.Result)

Statistical methods on result.stats.*.

All methods return a fresh Result. Computed values always go in data; method parameters go in metadata.

describe(columns: list[str] | None = None, by: str = 'column') → core_lens.base.result.Result

Per-column or per-entity descriptive statistics.

Uses polars’ in-built methods for mean, std, min, max, quantiles etc.

Parameters:

• columns (list[str] | None, optional) – Numeric columns to describe. Defaults to all numeric cols.

• by (str, optional) – "column" (one row per column, design default) or "entity" (one row per entity with mean/std per column).

Returns:

New Result with descriptive stats.

Return type:

Result

Raises:

ValueError – If by is not \"column\" or \"entity\".

correlate(columns: list[str], method: CorrelateMethod = CorrelateMethod.PEARSON, across: str = 'entity', group_by: str | None = None, min_obs: int = 3) → core_lens.base.result.Result

Compute pairwise correlations between columns.

Correlations are computed with scipy.stats and returned as a Result whose data is a Polars DataFrame. Two modes are supported:

• Pooled (group_by is None): all rows are treated as one

observation set and a single correlation is computed per column pair. * Per-entity (group_by set): the result is grouped by the given key column (e.g. "mws_id") and one correlation is computed per entity across its remaining rows (typically its time series). This is what enables questions such as “which MWS show the strongest dependence of cropping intensity on annual rainfall?”.

Parameters:

• columns – Column names to correlate. Must contain at least two entries. In pooled mode every pair is computed; in per-entity mode the first two columns are used as the pair.

• method – The correlation coefficient to compute. One of CorrelateMethod (PEARSON, SPEARMAN, KENDALL).

• across – "entity" or "time". Recorded in result metadata to describe the intended axis of the relationship.

• group_by – Optional key column to compute one correlation per group (e.g. "mws_id"). When None, the pooled behaviour is preserved for backward compatibility.

• min_obs – Minimum number of non-null observations required per group (per-entity mode) or overall (pooled mode) before a correlation is computed. Groups below this threshold are skipped.

Returns:

A Result whose data

DataFrame has the following columns:

• Pooled mode: column_a | column_b | correlation | p_value

• Per-entity mode: {group_by} | correlation | p_value | n_obs

Geometry is dropped (has_geometry=False).

Return type:

Result

Raises:

• CorrelationError – If fewer than two columns are supplied.

• ValueError – If method is not a valid CorrelateMethod, or if group_by is set but the column is not present in the underlying data.

Examples

Pooled correlation across all MWS in the AoI:

result = aoi.mws.annual.stats.correlate(
    columns=["dw_precipitation", "ci_cropping_intensity"],
    method=CorrelateMethod.PEARSON,
)
print(result.data.collect(engine="in-memory"))

Per-MWS correlation, ranked by strength of dependence:

top5 = (
    aoi.mws.annual.stats
    .correlate(
        columns=["dw_precipitation", "ci_cropping_intensity"],
        method=CorrelateMethod.PEARSON,
        group_by="mws_id",
        min_obs=3,
    )
    .data
    .with_columns(pl.col("correlation").abs().alias("abs_corr"))
    .sort("abs_corr", descending=True)
    .select("mws_id", "correlation", "p_value", "n_obs")
    .limit(5)
    .collect(engine="in-memory")
)
print(top5)

test(column: str, groups: str | None = None, periods: list[tuple[int, int]] | None = None, against: float | None = None, method: TestMethod | None = None, significance_level: float = 0.05) → core_lens.base.result.Result

Hypothesis test in three modes: group-based, period-based, single-sample.

Parameters:

• column (str) – Numeric column to test.

• groups (str | None, optional) – Categorical column to split groups on.

• periods (list[tuple[int, int]] | None, optional) – List of (from_year, to_year) period tuples.

• against (float | None, optional) – Reference value for a one-sample test.

• method (TestMethod | None, optional) – A TestMethod enum value. Auto-selected via Shapiro-Wilk if None.

• significance_level (float, optional) – Alpha level for significant flag (default 0.05).

Returns:

Result whose data has per-group descriptive stats group | n | mean | std | median and metadata with statistic, p_value, significant.

Return type:

Result

Raises:

ValueError – If none of groups, periods, against supplied, or if method is not recognised.

Under the hood:

Uses scipy.stats functions for statistical testing. - shapiro is used for normality testing to auto-select parametric vs non-parametric methods. - Parametric tests: ttest_1samp, ttest_ind. - Non-parametric tests: wilcoxon, mannwhitneyu, ks_2samp.

change(column: str, from_period: int, to_period: int, method: ChangeMethod = ChangeMethod.ABSOLUTE) → core_lens.base.result.Result

Change between two time periods per entity.

Parameters:

• column (str) – Value column to compute change for.

• from_period (int) – Start year/period integer.

• to_period (int) – End year/period integer.

• method (ChangeMethod, optional) – A ChangeMethod enum value.

Returns:

For absolute / percentage: data has key_col | value_from | value_to | change | pct_change. For trend: key_col | slope | r_squared | direction.

Return type:

Result

Raises:

ValueError – If method is not recognised or year column absent.

Under the hood:

• Absolute and Percentage methods use native Polars expressions.

• Trend method uses scipy.stats.linregress to perform linear regression over the time period.

anomaly(column: str, mode: str, method: AnomalyCrossMethod | AnomalyTsMethod, baseline: tuple[int, int] | None = None, target_years: tuple[int, int] | None = None, threshold: float = 2.0) → core_lens.base.result.Result

Anomaly detection in cross-sectional or timeseries mode.

Parameters:

• column (str) – Value column to analyse.

• mode (str) – "cross_sectional" or "timeseries".

• method (AnomalyCrossMethod | AnomalyTsMethod) – Enum value for detection method.

• baseline (tuple[int, int] | None, optional) – (from_year, to_year) inclusive, used as the reference/”normal” period. Required for timeseries; optional for cross-sectional.

• target_years (tuple[int, int] | None, optional) – (from_year, to_year) inclusive, years actually scored. Timeseries: defaults to “everything after baseline[1]” if omitted. Cross-sectional: defaults to all rows. Must not overlap baseline.

• threshold (float, optional) – Sigma / score threshold for anomaly flag (default 2.0).

Returns:

data has key_col | anomaly_score | is_anomaly (cross-sectional) or key_col | <time_col> | anomaly_score | is_anomaly (timeseries, baseline excluded, restricted to target_years if given). <time_col> is the finer-grained date/period column when the data is sub-annual, falling back to plain year for annual data. metadata includes n_entities_dropped.

Return type:

Result

Raises:

ValueError – If mode/method invalid, observation count too low, or baseline and target_years overlap.

Notes

STL fitting scope. For AnomalyTsMethod.STL the decomposition (trend + seasonal + remainder) is fit on the entire series of each entity, baseline and target years together. baseline is used only to calibrate the scale (standard deviation) of the remainder; target_years selects which points are scored. Scores are remainder / baseline_remainder_std.

Because target observations take part in the fit, the trend and seasonal components can partly absorb target-period anomalies, shrinking the remainder and lowering the score. The effect grows with the size and duration of the anomaly; a sustained shift can be absorbed almost entirely into the trend. Scores therefore measure deviation from a locally fitted trend and seasonal pattern, not from the baseline’s “normal” behaviour, and the method is conservative: it may miss anomalies but is unlikely to over-flag them.

A stricter alternative is to fit on the baseline only and forecast the target period (e.g. statsmodels.tsa.forecasting.stl.STLForecast), scoring forecast errors against the baseline error spread. This is not implemented.

For sub-annual data, residuals are matched to rows by the finer time column (date/period), not by year, so each observation gets its own score.

similarity(target: str, columns: dict[str, Any], method: SimilarityMethod = SimilarityMethod.EUCLIDEAN, top_n: int = 10) → core_lens.base.result.Result

Find entities most similar to target across columns.

Parameters:

• target – Key value of the reference entity.

• columns –

  Mapping {column_name: None | (resolution, filter_dict)}.

  • None — use the column directly from result.data.

  • (resolution, filter_dict) — fetch from the entity’s file at the given resolution ("static", "annual", "sub_annual"). Supported filter_dict keys:

    • "year": int | (int, int) — year equality or range.

    • "season": str — season label ("kharif" etc.).

    • "agg": str — time-aggregation before joining ("mean" default, "sum", "min", "max").

• method – Distance metric. A SimilarityMethod enum value.

• top_n – Number of most-similar entities to return (default 10).

Returns:

Result with key_col | similarity_score | rank.

Raises:

ValueError – If method is invalid, no columns can be resolved, or target is not found.

Under the hood:

• Euclidean, Manhattan, and Cosine distances are computed using optimized NumPy operations.

• Mahalanobis distance uses scipy.spatial.distance.mahalanobis.