superstats.diagnostics.metrics#

Numerical diagnostics for posterior estimates over time.

Functions

`calibration_error_per_step`(estimates, targets)	Marginal calibration error per step and parameter.
`correlation_per_step`(estimates, targets[, ...])	Pearson correlation between point estimates and true values, per step and parameter.
`nrmse_per_step`(estimates, targets[, aggregation])	Normalized RMSE between posterior samples and targets, per step and parameter.
`posterior_contraction_per_step`(estimates, ...)	Posterior contraction per step and parameter.

superstats.diagnostics.metrics.calibration_error_per_step(estimates, targets, resolution=20, aggregation=<function median>, min_quantile=0.005, max_quantile=0.995)[source]#

Marginal calibration error per step and parameter.

Computes an aggregate score for the marginal calibration error over an ensemble of approximate posteriors, per step (time step). The calibration error is given as the aggregate (e.g. median) of the absolute deviation between an alpha-CI and the relative number of inliers from estimates, over multiple alphas in (0, 1).

Parameters:

estimatesnp.ndarray of shape (num_sim, num_samples, num_steps, num_params): Posterior samples.
targetsnp.ndarray of shape (num_sim, num_steps, num_params): Ground-truth parameter values.
resolutionint, optional, default: 20: Number of credibility intervals (CIs) to consider.
aggregationcallable(), optional, default: np.median: Function used to aggregate the per-alpha calibration errors. Typically np.mean or np.median.
min_quantilefloat in (0, 1), optional, default: 0.005: Minimum posterior quantile to consider.
max_quantilefloat in (0, 1), optional, default: 0.995: Maximum posterior quantile to consider.

Returns:

calibration_errornp.ndarray of shape (num_steps, num_params) -: aggregated calibration error, per step and parameter

Parameters:

estimates (ndarray)
targets (ndarray)
resolution (int)
aggregation (Callable)
min_quantile (float)
max_quantile (float)

Return type:

ndarray

superstats.diagnostics.metrics.correlation_per_step(estimates, targets, aggregation=<function median>)[source]#

Pearson correlation between point estimates and true values, per step and parameter.

Posterior samples are first collapsed to a point estimate per simulation, step, and parameter (using aggregation), then the Pearson correlation between those point estimates and targets is computed across simulations.

Parameters:

estimatesnp.ndarray of shape (num_sim, num_samples, num_steps, num_params): Posterior samples.
targetsnp.ndarray of shape (num_sim, num_steps, num_params): Ground-truth parameter values.
aggregationcallable(), optional, default: np.median: Function used to collapse posterior samples into a point estimate per simulation, step, and parameter. Typically np.mean or np.median.

Returns:

correlationnp.ndarray of shape (num_steps, num_params) - Pearson: correlation per step and parameter

Parameters:

estimates (ndarray)
targets (ndarray)
aggregation (Callable)

Return type:

ndarray

superstats.diagnostics.metrics.nrmse_per_step(estimates, targets, aggregation=<function median>)[source]#

Normalized RMSE between posterior samples and targets, per step and parameter.

RMSE is computed across posterior draws (not aggregated first) for each simulation, then normalized by a prior-only bootstrap RMSE aggregated across simulations. This follows the “prior” normalization scheme: 0 indicates a maximally informative posterior (point mass at ground truth), 1 indicates a non-informative posterior (equivalent to the prior). The per-simulation ratios are then aggregated over simulations (using aggregation, mirroring the bayesflow convention of aggregating the final metric with the same function used for the normalizer), yielding one value per step and parameter.

Parameters:

estimatesnp.ndarray of shape (num_sim, num_samples, num_steps, num_params): Posterior samples per simulation.
targetsnp.ndarray of shape (num_sim, num_steps, num_params): Target parameter trajectories (themselves prior draws, in a simulation-based calibration setting).
aggregationcallable(), optional, default: np.median: Function used to aggregate both the prior-only bootstrap RMSE (for the normalizer) and the final per-simulation nRMSE values across simulations. Typically np.mean or np.median.

Returns:

nrmsenp.ndarray of shape (num_steps, num_params) - RMSE across: posterior draws, normalized by the aggregated prior-only bootstrap RMSE, aggregated across simulations

Parameters:

estimates (ndarray)
targets (ndarray)
aggregation (Callable)

Return type:

ndarray

superstats.diagnostics.metrics.posterior_contraction_per_step(estimates, targets, aggregation=<function median>)[source]#

Posterior contraction per step and parameter.

Computes 1 minus the ratio of posterior to prior variance (using the unbiased/sample variance, ddof=1) for each simulation, step, and parameter, clipped to [0, 1], then aggregates across simulations (using aggregation) to yield one value per step and parameter. Matches the bayesflow posterior_contraction metric, extended over an additional time-step axis.

Parameters:

estimatesnp.ndarray of shape (num_sim, num_samples, num_steps, num_params): Posterior samples.
targetsnp.ndarray of shape (num_sim, num_steps, num_params): Ground-truth parameter values, used to estimate the prior variance per step and parameter.
aggregationcallable(), optional, default: np.median: Function used to aggregate the per-simulation contraction values across simulations. Typically np.mean or np.median.

Returns:

contractionnp.ndarray of shape (num_steps, num_params) - 1 minus: the ratio of posterior to prior variance, per step and parameter, clipped to [0, 1], aggregated across simulations

Parameters:

estimates (ndarray)
targets (ndarray)
aggregation (Callable)

Return type:

ndarray