Steps and pipelines#

See also

Build and adjust a pipeline builds a pipeline step by step and shows what every step draws.

Conventions#

Steps and pipelines follow scikit-learn:

  • all parameters are keyword arguments of the constructor, and nothing else happens there;

  • fit, transform and fit_transform are the only ways to process data; what fit learns is stored in attributes ending with _ (bads_, ica_, mapping_, …) together with qc_;

  • get_params / set_params (nested as step__param in a pipeline) and sklearn.base.clone() work as usual.

Pipelines accept a Raw object or a path (read with meeg_utils.io.read()), check before running that the output of each step is a valid input for the next, and record provenance (provenance_: software versions, system, parameters, duration and warnings of each step). They can be edited and sliced:

pipe.replace("line_noise", S.LineNoise(method="notch"))
pipe.insert_after("interpolate", "align", S.HeadAlign(dest))
pipe.remove("resample")
first_part = pipe[:3]          # a pipeline of the first three steps

Data integrity#

A step never drops channels, reorders them or shifts the time axis unless its class declares it (changes_channels / changes_times, e.g. Resample). first_samp, meas_date and annotations are preserved. Each step only processes the systems it has been validated for and raises otherwise. meeg_utils.testing.check_step() checks all of this, plus the figures; every built-in step passes it, and custom steps can use it in their tests.

Available steps#

Step

What it does

Figures (plot_kinds)

Filter

FIR/IIR band-, high- or low-pass (MNE defaults).

response, psd

Resample

Change the sampling rate.

LineNoise

ZapLine-plus (default), ZapLine or notch, per channel type; QC measures suppression, distortion and over-/under-cleaning.

psd

BadChannels

PREP (EEG) or Maxwell (Neuromag, CTF) detection; marks info["bads"].

sensors, scores

Interpolate

Repair bad channels (spline for EEG, field mapping for MEG).

sensors

Reference

EEG reference; CTF gradient compensation grade.

ICA

ICA with ICLabel / MEGnet labels; removes components whose artifact probability reaches threshold; relabel for manual review.

components, labels, properties*, overlay*

HeadAlign

Map MEG data to another head position (e.g. the average over runs).

positions

Maxwell

SSS / tSSS (Neuromag): removes external interference, rebuilds bad MEG channels, optional movement compensation (head_pos="chpi") and transformation to a destination head position.

psd, head_positions†, displacement†

HFC

OPM: homogeneous (or harmonic) field correction; warns on single-axis arrays, where it also removes brain signal.

psd

Regression

Regress reference sensors (KIT/Ricoh, BTi, Artemis123, CTF grade 0) or EOG/ECG channels out of the data.

psd

BadSegments

Annotate bad stretches (BAD_amplitude, BAD_flat, BAD_muscle); later steps skip them. Amplitude thresholds are learned (autoreject) or given.

segments

ASR

Artifact subspace reconstruction of high-amplitude bursts (mne-denoise; cutoff 20 as EEGLAB).

reconstruction, psd

SNS

Sensor noise suppression: removes noise specific to single sensors (mne-denoise).

psd

BridgedElectrodes

Detect and interpolate bridged EEG electrodes.

topomap, distances

DropChannels

Remove named channels (e.g. mastoids before re-referencing).

Epoch

Raw → Epochs around events (annotations or stimulus channel); event codes fixed at fit; optional metadata from BIDS events.tsv.

drop_log, evoked

FixedLengthEpochs

Raw → consecutive epochs of equal length (resting state); drops those overlapping BAD annotations.

drop_log

Baseline

Baseline correction (mean, ratio, percent, z-score); refuses per-epoch z-scores on baselines too short to estimate a standard deviation, and offers a pooled scale.

AutoReject

Learned rejection thresholds (autoreject): repair channels and drop epochs (local), or one threshold per channel type (global).

reject_log, thresholds

* needs data: step.plot(inst=...). † only with movement compensation.

Figures#

step.plot() draws every figure that needs no data, from summaries kept on the fitted step; step.plot(inst=raw) adds the others; step.plot("psd", **kwargs) draws one kind, passing kwargs to the underlying MNE function. Figures are returned, never shown. For a pipeline, pipe.plot(inst=raw) replays the fitted steps so that every step plots the data it actually received.

MEG and EEG recorded together#

Most steps already treat channel types separately (filters, line noise, bad channels). When a type needs its own steps, branch with ByChannelType, the counterpart of scikit-learn’s ColumnTransformer:

pipe = meu.Pipeline([
    ("sss", S.Maxwell(st_duration=10.0)),
    ("by_type", S.ByChannelType({
        "meg": [("ica", S.ICA(picks="meg", labeler="megnet"))],
        "eeg": [("bads", S.BadChannels("prep")), ("interpolate", S.Interpolate()),
                ("ref", S.Reference("average")), ("ica", S.ICA(picks="eeg"))],
    })),
])
pipe.set_params(by_type__eeg__ica__n_components=15)

Each branch runs on a copy holding only its channels and the results are written back, so every channel stays in the data. Steps that change the channels or the time axis (Resample, Epoch) belong outside the branches.

Several runs#

Each run is processed on its own: bad channels, line noise and ICA differ between runs. For MEG, map the runs to a common head position before combining them:

dest = meu.io.average_dev_head_t(run_paths)   # weighted by good duration
pipe.insert_after("ica", "align", S.HeadAlign(dest))

HeadAlign uses minimum-norm field mapping; in simulations it lowers the error caused by a 10 mm / 5° head movement from 32-49 % to 1-3 %. For Neuromag data, S.Maxwell(destination=dest) does the same as part of Maxwell filtering.

After epoching each run, combine them:

epoch = meu.Pipeline([("epoch", S.Epoch("stimulus", -0.2, 0.8)), ("ar", S.AutoReject())])
runs = [epoch.fit_transform(run) for run in preprocessed_runs]
epochs = meu.epochs.combine(runs)              # bad channels: union across runs

meeg_utils.epochs.combine() checks that the runs share channels, sampling rate and epoch times, matches events by name (unifying their codes), and, for MEG, that their head positions differ by less than 2 mm (otherwise align them first).