.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_tutorials/plot_40_bids_datasets.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_tutorials_plot_40_bids_datasets.py: .. _tut-datasets: Whole BIDS datasets =================== For a study, the same steps run on every recording of a BIDS dataset: check them all, preprocess every run, epoch it, and combine the runs of each session. :class:`meeg_utils.Dataset` selects the recordings and :func:`meeg_utils.process_dataset` takes them through every stage, resuming an interrupted batch and collecting failures. The ``meu`` command does the same from a terminal. We first write MNE's ``sample`` recording as a small BIDS dataset, with two runs (its two halves), the empty-room recording and the Neuromag calibration files. .. GENERATED FROM PYTHON SOURCE LINES 18-20 .. code-block:: Python :dedent: 1 .. GENERATED FROM PYTHON SOURCE LINES 23-64 .. code-block:: Python import tempfile from pathlib import Path import mne from mne_bids import BIDSPath, write_meg_calibration, write_meg_crosstalk, write_raw_bids import meeg_utils as meu from meeg_utils import steps as S mne.set_log_level("ERROR") data_path = mne.datasets.sample.data_path() meg_dir = data_path / "MEG" / "sample" raw = mne.io.read_raw_fif(meg_dir / "sample_audvis_raw.fif").load_data() raw.info["line_freq"] = 60.0 event_id = {"auditory/left": 1, "auditory/right": 2, "visual/left": 3, "visual/right": 4} all_events = {**event_id, "smiley": 5, "button": 32} bids_root = Path(tempfile.mkdtemp()) / "bids" for run, (tmin, tmax) in enumerate([(0, 138), (138, None)], start=1): part = raw.copy().crop(tmin, tmax) part_events = mne.find_events(part) path = BIDSPath(subject="01", session="01", task="audvis", run=run, root=bids_root) write_raw_bids( part, path, events=part_events, event_id=all_events, format="FIF", allow_preload=True, overwrite=True, ) # fmt: skip er = mne.io.read_raw_fif(meg_dir / "ernoise_raw.fif").load_data() er.info["line_freq"] = 60.0 er_path = BIDSPath( subject="emptyroom", session=er.info["meas_date"].strftime("%Y%m%d"), task="noise", root=bids_root, ) # fmt: skip write_raw_bids(er, er_path, format="FIF", allow_preload=True, overwrite=True) meg_path = BIDSPath(subject="01", session="01", datatype="meg", root=bids_root) write_meg_crosstalk(data_path / "SSS" / "ct_sparse_mgh.fif", meg_path) write_meg_calibration(data_path / "SSS" / "sss_cal_mgh.dat", meg_path) sorted(str(p.relative_to(bids_root)) for p in bids_root.rglob("*") if p.is_file())[:14] .. rst-class:: sphx-glr-script-out .. code-block:: none ['README', 'dataset_description.json', 'participants.json', 'participants.tsv', 'sub-01/ses-01/meg/sub-01_ses-01_acq-calibration_meg.dat', 'sub-01/ses-01/meg/sub-01_ses-01_acq-crosstalk_meg.fif', 'sub-01/ses-01/meg/sub-01_ses-01_coordsystem.json', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_channels.tsv', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_events.json', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_events.tsv', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_meg.fif', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_meg.json', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-2_channels.tsv', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-2_events.json'] .. GENERATED FROM PYTHON SOURCE LINES 65-70 Select recordings ----------------- :class:`~meeg_utils.Dataset` selects recordings by BIDS entities and groups the runs of each session and task. Empty-room recordings are left out unless asked for: .. GENERATED FROM PYTHON SOURCE LINES 70-73 .. code-block:: Python ds = meu.Dataset(bids_root, datatype="meg") ds.table() .. rst-class:: sphx-glr-script-out .. code-block:: none [{'subject': '01', 'session': '01', 'task': 'audvis', 'acquisition': None, 'run': '1', 'datatype': 'meg', 'group': 'sub-01_ses-01_task-audvis_meg', 'path': '/tmp/tmpf3ax6brr/bids/sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_meg.fif'}, {'subject': '01', 'session': '01', 'task': 'audvis', 'acquisition': None, 'run': '2', 'datatype': 'meg', 'group': 'sub-01_ses-01_task-audvis_meg', 'path': '/tmp/tmpf3ax6brr/bids/sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-2_meg.fif'}] .. GENERATED FROM PYTHON SOURCE LINES 74-79 Check them all -------------- :func:`~meeg_utils.qc.inspect_dataset` runs the acquisition checks on every recording. In a BIDS dataset the checks also use the sidecars, the calibration files and the session's empty room: .. GENERATED FROM PYTHON SOURCE LINES 79-82 .. code-block:: Python result = meu.qc.inspect_dataset(bids_root) result .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 83-86 One row per recording and check: grey is measured only, green passed, amber and red need attention. With five or more recordings, a dot marks a recording that stands out from the rest of the dataset on a metric: .. GENERATED FROM PYTHON SOURCE LINES 86-88 .. code-block:: Python figs = result.plot() .. image-sg:: /auto_tutorials/images/sphx_glr_plot_40_bids_datasets_001.png :alt: QC per recording (grey measured only, green ok, amber warn, red fail; ● outlier) :srcset: /auto_tutorials/images/sphx_glr_plot_40_bids_datasets_001.png, /auto_tutorials/images/sphx_glr_plot_40_bids_datasets_001_2_00x.png 2.00x :class: sphx-glr-single-img .. GENERATED FROM PYTHON SOURCE LINES 89-92 .. code-block:: Python first = next(iter(result.reports.values())) print(first.checks["empty_room"].metrics_["mag"]) .. rst-class:: sphx-glr-script-out .. code-block:: none {'noise_floor': 3.989, 'unit': 'fT/√Hz', 'recording_above_db': 8.17, 'outlier_channels': ['MEG 0321', 'MEG 0411', 'MEG 0541', 'MEG 0631', 'MEG 2211', 'MEG 2441']} .. GENERATED FROM PYTHON SOURCE LINES 93-100 Process every run ----------------- Preprocessing runs on every run with its own copy of the pipeline; the epochs stage runs on each preprocessed run, after mapping the MEG runs of a session to their average head position; then the runs are combined. Here the preprocessing is the front of the ``meg-erp`` preset (SSS, with the calibration files found in the dataset), to keep the tutorial short: .. GENERATED FROM PYTHON SOURCE LINES 100-118 .. code-block:: Python preprocessing = meu.Pipeline( [ ("bads", S.BadChannels("maxwell")), ("sss", S.Maxwell()), ("highpass", S.Filter(0.1, None)), ("resample", S.Resample(250.0)), ] ) epochs = meu.Pipeline( [ ("lowpass", S.Filter(None, 40.0)), ("epoch", S.Epoch(list(event_id), tmin=-0.2, tmax=0.5)), ] ) deriv_root = bids_root / "derivatives" / "meu" batch = meu.process_dataset(ds, deriv_root, preprocessing=preprocessing, epochs=epochs) batch .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 119-121 Every output is a BIDS derivative with a JSON sidecar holding the configuration, software versions, warnings and QC metrics of every step: .. GENERATED FROM PYTHON SOURCE LINES 121-123 .. code-block:: Python sorted(str(p.relative_to(deriv_root)) for p in deriv_root.rglob("*.fif")) .. rst-class:: sphx-glr-script-out .. code-block:: none ['sub-01/ses-01/meg/sub-01_ses-01_task-audvis_desc-epochs_epo.fif', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_desc-epochs_epo.fif', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_desc-preproc_meg.fif', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-2_desc-epochs_epo.fif', 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-2_desc-preproc_meg.fif'] .. GENERATED FROM PYTHON SOURCE LINES 124-132 Calling :func:`~meeg_utils.process_dataset` again skips what exists, so an interrupted batch resumes and failed runs are retried. ``batch.failed`` lists the failures with their errors. Review the results ------------------ :func:`meeg_utils.report.summarize` collects the QC metrics of every derivative into one table, with the warnings of every run: .. GENERATED FROM PYTHON SOURCE LINES 132-135 .. code-block:: Python summary = meu.report.summarize(deriv_root, desc="preproc") summary.to_records() .. rst-class:: sphx-glr-script-out .. code-block:: none [{'file': 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-1_desc-preproc_meg.fif', 'n_warnings': 0, 'outlier_metrics': '', 'bads.meg.n_bads': 1.0, 'bads.meg.fraction': 0.0032679738562091504, 'sss.n_basis': 72.0, 'sss.projs_removed': 3.0, 'sss.power_change_db.mag': -8.76, 'sss.power_change_db.grad': -2.25, 'highpass.highpass': 0.10000000149011612, 'highpass.lowpass': 172.17630004882812, 'resample.sfreq_before': 600.614990234375, 'resample.sfreq_after': 250.0}, {'file': 'sub-01/ses-01/meg/sub-01_ses-01_task-audvis_run-2_desc-preproc_meg.fif', 'n_warnings': 0, 'outlier_metrics': '', 'bads.meg.n_bads': 0.0, 'bads.meg.fraction': 0.0, 'sss.n_basis': 72.0, 'sss.projs_removed': 3.0, 'sss.power_change_db.mag': -9.58, 'sss.power_change_db.grad': -1.77, 'highpass.highpass': 0.10000000149011612, 'highpass.lowpass': 172.17630004882812, 'resample.sfreq_before': 600.614990234375, 'resample.sfreq_after': 250.0}] .. GENERATED FROM PYTHON SOURCE LINES 136-138 and :func:`meeg_utils.report.from_derivative` rebuilds the report of a single output (pipeline, parameters, QC metrics, provenance and the data): .. GENERATED FROM PYTHON SOURCE LINES 138-141 .. code-block:: Python fname = next(deriv_root.rglob("*_desc-preproc_meg.fif")) meu.report.from_derivative(fname) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 142-151 The same from the command line ------------------------------ .. code-block:: bash meu qc bids/ --out qc/ meu run --preset meg-erp --epochs \ --option "event_id=[auditory/left, auditory/right, visual/left, visual/right]" \ --sources bids/ --out bids/derivatives/meu meu report bids/derivatives/meu --desc preproc .. rst-class:: sphx-glr-timing **Total running time of the script:** (2 minutes 24.727 seconds) .. _sphx_glr_download_auto_tutorials_plot_40_bids_datasets.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_40_bids_datasets.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_40_bids_datasets.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_40_bids_datasets.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_