Data file viewer for TDMS, HDF5, MAT and Parquet

Formats the viewer reads

A TDMS file is a chain of segments. Each segment has a 28-byte lead-in, optional metadata (object paths such as /'Group'/'Channel', properties and raw data indexes) and a block of raw data that can be contiguous or interleaved. Later segments only repeat what changed, so the viewer walks every lead-in, builds the channel list from the metadata, and reads a channel’s bytes when you click it. Timestamps are decoded from the TDMS layout of seconds since 1904-01-01 UTC plus fractions in units of 2⁻⁶⁴ s, and a group’s timestamp channel becomes the time axis for its other channels. You don’t need the .tdms_index file, which only holds a copy of the metadata.

MATLAB’s v7.3 MAT-files are HDF5 files with a 512-byte MATLAB header, so they go through the HDF5 reader. The viewer flips their dimensions back to MATLAB order, which is why a matrix column shows up as M(:,1). Older v4, v6 and v7 files use MATLAB’s own Level 4 and Level 5 formats, and v7 compresses each variable with zlib.

How the viewer reads each format

When you drop a file, the browser hands the page a reference to it. The viewer reads byte ranges with Blob.slice, decodes them in JavaScript or WebAssembly, and draws the result on a canvas.

TDMS metadata is read first and channel data only on demand, so a 200 MB test file with 25 segments listed its channels in about a quarter of a second. HDF5 and v7.3 MAT-files open in h5wasm, the HDF5 C library compiled to WebAssembly, which runs in a background worker and reads through the same file reference without copying the file into memory. Parquet files are read with hyparquet, which takes the schema from the footer and decompresses only the columns you plot (snappy, gzip, zstd, brotli and lz4 are supported). Level 5 MAT-files and CSV files are read into memory whole.

Plotting long channels

A 10-million-sample channel has thousands of times more points than the chart has pixels. For each pixel column the viewer keeps the minimum and maximum sample in that slice of time and draws both, so spikes and dropouts stay visible. Dragging across the chart zooms in and recomputes the envelope from the full-resolution data. Once fewer than about three samples per pixel are visible, every sample is drawn. Double-click to zoom back out. The statistics row (min, max, mean, RMS, sample count, duration) always covers the whole channel, and Export CSV writes every sample of the ticked channels with a time column.

Opening the same files in Python

For scripted analysis, npTDMS reads TDMS files with the same waveform timing and DAQmx scaling:

from nptdms import TdmsFile

tdms = TdmsFile.read("run.tdms")      # TdmsFile.open() streams files larger than RAM
for group in tdms.groups():
    for channel in group.channels():
        print(group.name, channel.name, len(channel), channel.properties.get("unit_string"))
volts = tdms["Analog"]["Voltage"][:]
t = tdms["Analog"]["Voltage"].time_track()   # seconds, from wf_increment

HDF5 with h5py, where slicing a 2-D dataset reads one column without loading the rest:

import h5py

with h5py.File("run.h5", "r") as f:
    f.visit(print)                          # every group and dataset path
    x = f["sensors/accel/x"][:]
    y = f["sensors/accel/xyz"][:, 1]
    print(dict(f["sensors/accel/x"].attrs))

Parquet with pyarrow, reading only the columns you name:

import pyarrow.parquet as pq

print(pq.ParquetFile("run.parquet").schema_arrow)
table = pq.read_table("run.parquet", columns=["time", "a"])
df = table.to_pandas()

MAT-files with SciPy. loadmat reads v4 to v7 files and raises NotImplementedError on v7.3, which h5py opens directly:

from scipy.io import loadmat
import h5py

m = loadmat("run.mat", squeeze_me=True)
sig = m["sig"]

with h5py.File("run_v73.mat", "r") as f:
    M = f["M"][()].T                        # transpose back to MATLAB row/column order

Limits

DAQmx raw TDMS channels that need Thermocouple, RTD, strain, thermistor or table scaling are listed but not plotted yet. npTDMS converts them. HDF5 datasets with more than two dimensions, string and reference datasets, MATLAB cell arrays, struct arrays and sparse matrices are also listed without a plot. Complex data shows the real part. Each plotted channel sits in memory as a typed array, so a 50-million-sample float64 channel takes 400 MB. Level 5 MAT-files over 800 MB and text files over 1.5 GB are refused because they have to be read whole.

If you’re choosing a format for new recordings, the comparison of CSV, MATLAB, TDMS, Parquet and HDF5 covers size, speed and tool support, and opening TDMS files on a Mac walks through desktop and Python options. DAQAtlas, which is in development, exports runs to CSV, MAT, TDMS, Parquet and HDF5, so its recordings open here too. Join the list for early access when DAQAtlas releases.