Generic NetCDF NIRS Datasets

Status: Supported (scoped) · Vendor: Vendor-neutral · Extensions: .nc, .cdf · Feature flag: fmt-hdf5

NetCDF is a self-describing scientific container (classic and HDF5-backed) widely used for spectral and atmospheric datasets. This reader maps the common NIRS spectra + wavelengths schema plus a few schema-specific atmospheric / sun-photometer products, and recognises ANDI/MS chromatography containers so it can refuse them with a useful pointer.

Instruments & software

Vendor-neutral. The generic path targets NetCDF written by NIRS pipelines and research datasets. Additional schema-specific paths cover Microtops MAN aerosol series and (local-only) DOE ARM MFRSR and SURFSPECALB atmospheric products. Non-NIRS NetCDF such as weather datasets and the PyrNet pyranometer fixture are refused.

File structure

Detected by the .nc / .cdf extension; the schema is validated on read. The classic path decodes through the pure-Rust netcdf-reader crate, while HDF5-backed NetCDF4 containers are decoded through hdf5-reader (both gated behind fmt-hdf5). The reader also exposes the sidecar resolver (read_bytes_with_sidecars): the only true companion file is an optional ARM MFRSR <stem>.yaml QC sidecar, which is served by the resolver in the in-memory path and read from disk by open_path; its absence is silent.

The generic NIRS schema is a 2-D spectra variable shaped samples x wavelength with a 1-D wavelengths axis variable and optional 1-D target variables.

What nirs4all-formats extracts

  • Signals — the spectra variable for the generic schema; for the atmospheric paths, discovered channel sets (e.g. Microtops aot_<wavelength> series, ARM MFRSR hemispheric/diffuse/direct irradiance plus voltage and ratio channels, ARM SURFSPECALB surface_albedo).

  • Axis — the wavelength axis variable (or, for Microtops, a wavelength axis assembled by sorting the aot_<wavelength> variable names).

  • Targets — 1-D variables matching the sample dimension.

  • Metadata — global attributes under metadata.global_attributes when the pure-Rust stack can decode them; per-signal QC arrays for the ARM paths.

  • Provenance & warnings — source file + SHA-256; the ARM QC YAML is added as a qc_sidecar source, mapping suspect/incorrect time ranges to per-record arm_mfrsr_sidecar_* quality flags.

The reader emits one SpectralRecord per sample row, or per non-missing time row for the derived time-series products.

Variants & support status

Variant

Status

Notes

Generic spectra + wavelengths schema

Supported

One record per sample row.

Microtops MAN AOT NetCDF4/HDF5 series

Supported (scoped)

aot typed as aerosol optical thickness; aot_std as uncertainty.

ARM MFRSR b1 7-channel time series

Experimental (local-only)

Irradiance/voltage/ratio channels + optional QC YAML sidecar.

ARM SURFSPECALB 6-filter surface albedo

Experimental (local-only)

Reflectance-like surface_albedo; all-missing rows dropped.

ANDI/MS chromatography NetCDF

Detected / refused

Recognised by ANDI variable markers; refused as non-NIRS.

Real generic NIRS NetCDF schemas

Planned

Broader real-world schemas and QC handling still wanted.

Limitations & known gaps

  • ANDI/MS containers are detected via the strict NetCDF path and refused with a message pointing to pyteomics, PyMassSpec or pyOpenMS. A NetCDF4-classic file wrapping an HDF5 ANDI container is not re-checked in the HDF5 fallback, so it surfaces as an unsupported-schema error rather than the canonical pointer.

  • For some NetCDF4 layouts, hdf5-reader 0.5 mis-decodes the shared attribute heap; the reader then falls back to a generic contiguous-layout decoder keyed on standard HDF5 metadata (scanning fractal-heap hard-link records, resolving each candidate dataset’s contiguous layout, dataspace and datatype), emitting microtops_man_netcdf_contiguous_layout_fallback. That path recovers only fixed-length ASCII string attributes; variable-length and numeric scalar attributes are skipped, signalled by microtops_man_netcdf_global_attributes_byte_scan.

  • The ARM MFRSR and SURFSPECALB paths are validated locally only (ARM Data Use Policy); a redistributable MFR-7/MFRSR .OUT dump and broader ARM mapping are still wanted.

  • Generic NIRS NetCDF support needs real-world schemas with QC and multi-signal groups to harden it.

Reference readers

netcdf-reader, xarray, the netCDF library and the ARM act toolkit read the same files; ANDI/MS belongs to pyteomics / pyOpenMS. nirs4all-formats adds the NIRS schema validation, signal typing and provenance.

Samples & validation

Fixtures: samples/netcdf/synthetic_nirs.nc (50 records, nm axis, absorbance/protein) and samples/microtops/microtops_arc_msm114_2.nc (PANGAEA MSM114/2, CC-BY-4.0; 5 AOT channels + *_std), both golden-backed; the PyrNet fixture (samples/netcdf/pyrnet_to_l1a_output.nc) is a locked non-NIRS refusal. Local-only fixtures under samples_local/ cover the ARM MFRSR b1 file (4,320 observations x 7 filters, with QC YAML sidecar) and the ARM SURFSPECALB product (986 useful rows x 6 filters); ARM AOSMET remains a refusal. Probe reports netcdf-nirs at Confidence::Likely, or andi-ms-netcdf at Confidence::Definite for refused chromatography containers.