Parquet NIRS Tables

Status: Supported · Vendor: Apache / generic · Extensions: .parquet · Feature flag: fmt-parquet

Native Rust reader for canonical tabular NIRS datasets stored as Apache Parquet, the columnar format used for efficient internal distribution of wide spectral tables. A spectral Parquet table has one row per sample and numeric wavelength-named columns.

Instruments & software

Vendor-neutral; written by Arrow / pyarrow, fastparquet, pandas.to_parquet and the nirs4all ParquetLoader. Useful as a compact distribution format for NIRS datasets.

File structure

Detected by the PAR1 magic plus the .parquet extension and opened through Arrow (with Zstd support). A table is accepted as spectral when:

  • its spectral columns are named by numeric wavelength values and typed float32 or float64;

  • there are at least 8 such columns;

  • the resulting axis is strictly ascending (these two checks reject generic Parquet files).

Non-spectral numeric columns become targets, and a sample_id / sample / id UTF-8 column becomes the identifier.

What nirs4all-formats extracts

  • Signals — one SpectralRecord per table row, each with a single absorbance signal (type Absorbance).

  • Axis — values from the numeric column names; unit nm, kind Wavelength.

  • Targets — numeric non-spectral columns (float32/float64/int32/int64, e.g. protein) become targets; nulls are preserved as null.

  • Metadata — the sample_id/sample/id string column maps to metadata.sample_id; row_index and a parquet summary (spectral/target column counts) are recorded.

  • Provenance — source file + SHA-256, reader name and version.

The same decode path serves both read_path (filesystem) and read_bytes / open_bytes (in-memory), so the reader is sidecar-free and works without a resolver.

Variants & support status

Variant

Status

Notes

Numeric-wavelength spectral table (float32/float64)

Supported

≥ 8 ascending wavelength columns.

Zstd-compressed Parquet

Supported

Default committed fixture is Zstd.

Numeric target + sample_id columns

Supported

Targets and identifier joined per row.

In-memory decode (open_bytes)

Supported

Same code path as filesystem reads.

Non-spectral Parquet (e.g. alltypes_plain.parquet)

Detected / refused

Refused as “not a NIRS spectral table”.

Limitations & known gaps

  • Schema metadata is not yet read for explicit units or signal type, so the signal is always typed Absorbance on a nm wavelength axis.

  • Compression variants beyond the committed Zstd fixture are added as needed.

  • Projection-based reading for very wide tables is not implemented; the reader materialises all spectral columns per batch.

Reference readers

pyarrow.parquet, fastparquet, pandas.read_parquet and the nirs4all ParquetLoader read the same tables. nirs4all-formats adds the spectral-schema validation, axis construction, target/metadata separation and provenance.

Samples & validation

Fixtures live under samples/parquet/, covered by golden summaries in crates/nirs4all-formats/tests/goldens/ (parquet_*): synthetic_nirs.parquet yields 50 records over 200 wavelength columns with a protein target, and alltypes_plain.parquet (the Apache sample) is refused as non-spectral. The probe reports format parquet-container at Confidence::Likely.