Generic HDF5 NIRS Datasets

Status: Supported (scoped) · Vendor: Vendor-neutral · Extensions: .h5, .hdf5 · Feature flag: fmt-hdf5

HDF5 is a general-purpose hierarchical container used by many NIRS pipelines, spectrometers and research datasets to store spectra alongside a wavelength (or wavenumber) axis, targets and free-form attributes. This is a schema-aware reader for the common “spectra + axis” layouts; it is deliberately conservative and refuses HDF5 files it cannot map confidently.

Instruments & software

Vendor-neutral. The reader targets HDF5 files written by NIRS instruments, conversion tools and analysis pipelines that follow the usual dataset naming conventions. Committed fixtures are synthetic, exercising single- and multi-signal layouts, nested groups, common dataset aliases and a transposed matrix orientation.

File structure

Detected by the HDF5 magic (\x89HDF\r\n\x1a\n) combined with an .h5 / .hdf5 extension; the NIRS schema itself is validated only on read. Decoding uses the pure-Rust hdf5-reader crate (gated behind the fmt-hdf5 feature), so it works on the no-filesystem wasm32-unknown-unknown target as well. The reader also routes through the sidecar resolver, so external-file and external-link references inside the container are followed via the same companion-file path.

The reader searches the root group first, then nested groups up to four levels deep, looking for:

  • a 2-D spectral dataset shaped samples x bands, or bands x samples when the axis length identifies the band dimension unambiguously;

  • a matching 1-D axis dataset (wavelengths, wavenumbers, wn, lambda, x_axis, and related *_nm / *_cm-1 aliases);

  • optional 1-D numeric target datasets matching the sample dimension.

What nirs4all-formats extracts

  • Signals — one signal per recognised spectral dataset. Multiple compatible datasets in the same group (e.g. absorbance and reflectance sharing one /wavelengths) are emitted as separate signals on each record; duplicate names get stable numeric suffixes. Dataset names such as spectra, absorbance, reflectance, transmittance, intensity, raw, counts or data are recognised, and the signal type is inferred from the name or a units attribute.

  • Axis — values from the axis dataset, with the unit taken from its units attribute and the kind (Wavelength / Wavenumber / Index) inferred from the dataset name and unit.

  • Targets — any 1-D numeric dataset that matches the sample count and is not the axis or a spectral dataset.

  • Metadata — root and group attributes, the source group path, the matrix orientation when non-default, and per-signal unit hints.

  • Provenance — source file + SHA-256, reader name and version.

The reader emits one SpectralRecord per sample row.

Variants & support status

Variant

Status

Notes

spectra + wavelengths, single signal

Supported

Canonical samples x bands layout.

Multiple compatible datasets in one group

Supported

Emitted as separate signals sharing the axis.

Common dataset aliases (absorbance, data, wn, …)

Supported

Name- and unit-based signal/axis typing.

Transposed bands x samples matrix

Supported

Accepted only when the axis length is unambiguous.

Real metadata-rich vendor schemas

Planned

Synthetic fixtures only; real-world schemas still wanted.

Limitations & known gaps

  • Dispatch is intentionally conservative: HDF5 files without a recognised 2-D spectral dataset and a matching 1-D axis are refused.

  • Ambiguous transposes (where rows and columns could both match the axis) are rejected rather than guessed.

  • FGI XML+HDF5 pairs are handled by the dedicated fgi-hdf5-xml reader; MATLAB v7.3 .mat files (matlab) and Allotrope ADF (allotrope-adf) use separate schema mappers even though their payloads are HDF5-backed.

  • Real instrument schemas with rich metadata, complex axes, non-trivial targets and heterogeneous group conventions are still needed to harden the mapping.

Reference readers

h5py, the hdf5-reader crate and PyTables (tables) open the same containers; nirs4all-formats adds axis detection, signal typing, target extraction and provenance on top.

Samples & validation

Fixtures live under samples/hdf5/ (synthetic_nirs.h5 — 50 records, two signals on a shared /wavelengths; generic_aliases_data_group.h5 — a bands x samples /data/absorbance with a cm-1 axis) and samples/fgi/. They are covered by golden summaries in crates/nirs4all-formats/tests/goldens/, and non-spectral refusals are locked in. The probe reports hdf5-nirs-container at Confidence::Likely.