-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Description
Hi team,
While going through the downstream validation pipeline for the neural-lam
probabilistic forecasting track (issue mllam/neural-lam#62), I noticed
that mllam-verification currently only covers deterministic
statistics like rmse and mae.
pyproject.toml already pulls scores>=1.2.0 which exposes exactly
what's needed for ensemble evaluation:
scores.probability.crps_for_ensemble— for member-indexed ensemble
outputs, which is the format neural-lam's datastore already uses
consistently (ensemble_memberdimension inweather_dataset.py
anddatastore/base.py)scores.plotdata.rank_histogram— for Talagrand diagram evaluation
of ensemble calibartion
Two concrete additions that would follow the existing architecture exactly:
- A
crps()function instatistics.pywrapping
scores.probability.crps_for_ensembleviacompute_pipeline_statistic
— same pattern as howrmsewrapsscores.continuous.rmse - A
plot_rank_histogram()inplot.pywrapping
scores.plotdata.rank_histogram— following theplot_single_metric_timeseries
structure
No new dependancies needed. Both functions already exist in the
pinned scores version.
If this makes sense I'll go ahead and implement it otherwise just
let me know and I'll close the issue.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels