feat: add MetOp AMSU-A and AVHRR Level 1B data sources

[NickGeneva]

Description

Add MetOpAMSUA and MetOpAVHRR DataFrameSource implementations for EUMETSAT MetOp satellite Level 1B products, accessed via the eumdac library.

Data source details

Property	MetOpAMSUA	MetOpAVHRR
Source type	DataFrameSource	DataFrameSource
Collection ID	EO:EUM:DAT:METOP:AMSUL1	EO:EUM:DAT:METOP:AVHRRL1
Format	EPS native binary (custom parser)	EPS native binary (custom parser)
Channels	14 (23.8–57.3 GHz microwave)	6 (0.63–12.0 µm visible/IR)
Spatial resolution	~50 km at nadir	~20 km (103 EPS tie points per scan line)
Temporal coverage	2006-present	2006-present
Region	Global (LEO polar orbit, ~14 orbits/day)	Global (LEO polar orbit)
Authentication	EUMETSAT API key/secret (env vars)	EUMETSAT API key/secret (env vars)

Implementation highlights

• MetOpAMSUA: Custom binary parser for EPS native MDR-1B records (3464 bytes/scan). Extracts calibrated radiances, converts to brightness temperatures via Planck inverse function, with full geolocation and satellite geometry. Channel 15 (89 GHz) excluded due to ~97% missing data in L1B products.
• MetOpAVHRR: Custom binary parser for EPS native format (~907 MB/orbit). Parses GIADR-radiance for calibration coefficients, reads MDR SCENE_RADIANCES at 103 navigation tie points per scan line. Visible channels calibrated to reflectance (%), thermal channels to brightness temperature (K). Handles 3a/3b channel switching via FRAME_INDICATOR bit 16. ~120x faster than satpy approach (1.2s vs ~150s per file). Validation script is below, about 10x faster to load file than satpy (which is GPL anyway).
• Both sources use eumdac for OAuth2-authenticated data access to the EUMETSAT Data Store.
• All binary format offsets and calibration formulas taken from public EUMETSAT Product Format Specification documents linked in docstrings
• MetOpAMSUALexicon and MetOpAVHRRLexicon with 20 new E2STUDIO_VOCAB entries.
• 24 non-network unit tests (mock + binary parser unit tests), 8 network integration tests.

Data licensing

License: EUMETSAT Data Policy
URL: https://www.eumetsat.int/data-policy

Full operational data freely available. Open access for both commercial and non-commercial use under EUMETSAT's data policy established by Member States. Complies with WMO data sharing principles.

Dependencies added

Package	Version	License	License URL	Reason
eumdac	>=3.1.0	MIT	LICENSE.txt	EUMETSAT Data Access Client for OAuth2 auth and product search/download

No GPL dependencies. Both AMSU-A and AVHRR parsers are clean-room implementations using only struct, numpy, and pandas.

Performance

Source	File size	Parse time (subsample=16)	Observations
MetOpAMSUA	~2.6 MB	<0.1s	~23k/orbit
MetOpAVHRR	~907 MB	1.2s (6 channels)	~1M/orbit

Files changed

File	Description
earth2studio/data/metop_amsua.py	MetOpAMSUA DataFrameSource (custom EPS binary parser)
earth2studio/data/metop_avhrr.py	MetOpAVHRR DataFrameSource (custom EPS binary parser)
earth2studio/lexicon/metop.py	Lexicon classes for both instruments
earth2studio/lexicon/base.py	20 new E2STUDIO_VOCAB entries
test/data/test_metop_amsua.py	13 tests (9 mock/unit + 4 network)
test/data/test_metop_avhrr.py	19 tests (15 mock/unit + 4 network)
earth2studio/data/__init__.py	Register new sources
earth2studio/lexicon/__init__.py	Register new lexicons
docs/modules/datasources_dataframe.rst	Documentation entries
pyproject.toml + uv.lock	New dependency (eumdac)
CHANGELOG.md	Release notes

Sanity-check plots

AMSU-A (channels 4, 9, 14 — 2025-01-15)

AVHRR (channels 4, 5 — 2025-01-15)

Sanity-check script (not committed)

"""Sanity-check plots for MetOp AMSU-A and AVHRR data sources."""
import os
from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import cartopy.crs as ccrs
import cartopy.feature as cfeature

os.environ["EUMETSAT_CONSUMER_KEY"] = "YOUR_KEY"
os.environ["EUMETSAT_CONSUMER_SECRET"] = "YOUR_SECRET"

from earth2studio.data import MetOpAMSUA, MetOpAVHRR

# --- AMSU-A ---
ds_a = MetOpAMSUA(time_tolerance=timedelta(hours=2), cache=True, verbose=True)
df_a = ds_a(datetime(2025, 1, 15, 11), ["amsua04", "amsua09", "amsua14"])

fig, axes = plt.subplots(1, 3, figsize=(18, 6),
                         subplot_kw={"projection": ccrs.Robinson()})
for ax, var in zip(axes, ["amsua04", "amsua09", "amsua14"]):
    sub = df_a[df_a["variable"] == var]
    sc = ax.scatter(sub["lon"], sub["lat"], c=sub["observation"],
                    s=1, transform=ccrs.PlateCarree(), cmap="RdYlBu_r")
    ax.add_feature(cfeature.COASTLINE, linewidth=0.5)
    ax.set_global()
    ax.set_title(f"{var} (N={len(sub):,})")
    plt.colorbar(sc, ax=ax, shrink=0.6, label="BT (K)")
fig.suptitle("MetOp AMSU-A — 2025-01-15 11:00 UTC (±2h)", fontsize=14)
plt.tight_layout()
plt.savefig("sanity_check_metop_amsua.png", dpi=150)

# --- AVHRR ---
ds_v = MetOpAVHRR(time_tolerance=timedelta(hours=2), subsample=16,
                   cache=True, verbose=True)
df_v = ds_v(datetime(2025, 1, 15, 11), ["avhrr04", "avhrr05"])

fig, axes = plt.subplots(1, 2, figsize=(16, 6),
                         subplot_kw={"projection": ccrs.Robinson()})
for ax, var in zip(axes, ["avhrr04", "avhrr05"]):
    sub = df_v[df_v["variable"] == var]
    sc = ax.scatter(sub["lon"], sub["lat"], c=sub["observation"],
                    s=0.1, transform=ccrs.PlateCarree(), cmap="RdYlBu_r")
    ax.add_feature(cfeature.COASTLINE, linewidth=0.5)
    ax.set_global()
    ax.set_title(f"{var} (N={len(sub):,})")
    plt.colorbar(sc, ax=ax, shrink=0.6, label="BT (K)")
fig.suptitle("MetOp AVHRR — 2025-01-15 11:00 UTC (±2h)", fontsize=14)
plt.tight_layout()
plt.savefig("sanity_check_metop_avhrr.png", dpi=150)

validate_avhrr_satpy.py

[greptile-apps]

Greptile Summary

Adds MetOpAMSUA and MetOpAVHRR as new DataFrameSource implementations for EUMETSAT MetOp satellite Level 1B products, with corresponding lexicons, unit tests, and documentation. The custom EPS binary parser for AMSU-A is well-structured and thoroughly tested; the AVHRR path leverages satpy with a performance-oriented shortcut through private internals. All previously raised concerns have been addressed.

Confidence Score: 5/5

Safe to merge; all remaining findings are minor style/robustness suggestions that don't affect the primary data path.

All P0/P1 concerns from previous review rounds have been resolved. The three remaining comments are P2: a global Dask config mutation without a context manager, reliance on private satpy internals (acknowledged performance trade-off), and an unguarded ValueError in the MPHR sensing-time parser for malformed files. None of these affect normal operation with real EPS products.

earth2studio/data/metop_avhrr.py (dask.config.set and private satpy API access)

Important Files Changed

Filename	Overview
earth2studio/data/metop_amsua.py	New DataFrameSource with custom EPS binary parser; well-structured and tested. Minor: _parse_sensing_time can throw ValueError for malformed MPHR without being caught.
earth2studio/data/metop_avhrr.py	New DataFrameSource using satpy for AVHRR parsing; relies on private satpy internals for performance and sets global Dask config without context manager.
earth2studio/lexicon/metop.py	New lexicons for AMSU-A (14 channels) and AVHRR (6 channels); clean implementation following existing patterns.
test/data/test_metop_amsua.py	Good unit test coverage with synthetic binary builder; tests GRH parsing, MPHR parsing, BT conversion, and end-to-end mock fetch.
test/data/test_metop_avhrr.py	Mock-based tests cover end-to-end call, fields subset, empty response, and exception handling.

_{Reviews (2): Last reviewed commit: "fix: try/finally for temp cache cleanup ..." | Re-trigger Greptile}

[NickGeneva]

Sanity-Check Plots (cartopy Robinson projection)

MetOpAMSUA — channels 4, 9, 14

MetOpAVHRR — channels 4, 5

Test date: 2025-01-15 12:00 UTC, Metop-B, ±2h tolerance

• AMSU-A: 162,000 obs (3 vars × 54,000 obs/var), 4 orbit passes
• AVHRR: 556,406 obs (2 vars), 4 orbit passes, subsample=32

[NickGeneva]

/blossom-ci

[NickGeneva]

@greptile-apps

[NickGeneva]

/blossom-ci

[NickGeneva]

/blossom-ci

[NickGeneva]

/blossom-ci