Extract vision features with VneuroTK data#

data.vision.extract_from() uses the image database bound by BaseData.configure(). Features are stored once per unique stimulus; indexing produces arrays aligned to VisionData.output_order and therefore to the recording’s trial order.

This notebook requires a configured data object from the neural-data notebook and a selected model from the vision-model notebook. The model backend may require optional dependencies and cached or downloadable assets; the documentation build does not execute these cells.

Extract into the recording#

import numpy as np
import torch

import vneurotk as vtk

rng = np.random.default_rng(0)
stim_ids = np.array(["image-1", "image-2", "image-1"])
data = vtk.BaseData.for_continuous(
    neuro=np.arange(80, dtype=float).reshape(20, 4),
    neuro_info={"ch_names": ["a", "b", "c", "d"], "sfreq": 10.0},
)
data.configure(
    vision_onsets=np.array([2, 8, 14]),
    stim_ids=stim_ids,
    vision_db={stim_id: rng.integers(0, 256, (64, 64, 3), dtype=np.uint8) for stim_id in np.unique(stim_ids)},
    trial_window=[-1, 3],
)
device = "cuda" if torch.cuda.is_available() else "cpu"
model = vtk.VisionModel(
    "facebook/dinov2-base",
    backend="transformers",
    device=device,
)
model.set_selector(module_name="layernorm")
data.vision.extract_from(model, batch_size=2)
data.vision.meta

Repeating extraction with the same model is a no-op. If only selected modules are missing, existing records retain their provenance and newly extracted records receive current provenance. overwrite=True replaces each selected record together with its provenance.

Index trial-aligned features#

first_name = data.vision.meta.iloc[0]["module_name"]
trial_aligned = data.vision[first_name]
trial_aligned.shape

mask = data.vision.meta["module_type"] == data.vision.meta.iloc[0]["module_type"]
selected = data.vision[mask]
# One match returns an ndarray; multiple matches return VisualRepresentations.
type(selected)

Add another model or overwrite selected records#

# model2 = vtk.VisionModel("resnet50.a1_in1k", backend="timm", device=device)
# model2.set_selector(module_name="global_pool")
# data.vision.extract_from(model2, batch_size=4)

# To intentionally refresh records selected on model:
# data.vision.extract_from(model, batch_size=4, overwrite=True)

Persist features with neural data#

BaseData.save() stores neural data, trial configuration, visual representations, and structured extraction provenance in one HDF5 recording.

from pathlib import Path

import vneurotk as vtk

output = vtk.VTKPath(Path("outputs"), subject="01", task="demo")
# data.save(output)
# loaded = vtk.read(output)