Extract vision features from images#

VisionModel.extract() accepts a {stimulus_id: image} mapping independently of BaseData and always returns VisualRepresentations. Indexing by a module name or integer returns one VisualRepresentation; boolean-mask indexing always returns a VisualRepresentations collection, including zero or one match.

This workflow requires a configured model from the vision-model notebook. Model construction may require network access; the documentation build does not execute these cells.

Prepare a bounded image mapping#

import numpy as np
import torch

import vneurotk as vtk

rng = np.random.default_rng(0)
images = {f"image-{index}": rng.integers(0, 256, (64, 64, 3), dtype=np.uint8) for index in range(4)}
device = "cuda" if torch.cuda.is_available() else "cpu"
model = vtk.VisionModel(
    "facebook/dinov2-base",
    backend="transformers",
    device=device,
)
model.set_selector(module_name="layernorm")

Extract selected representations#

representations = model.extract(images, batch_size=4)
representations.meta

Inspect arrays and provenance#

first = representations[0]
array = representations.numpy(first.module_name)
provenance = first.provenance
provenance.backend, provenance.model_id, array.shape

ExtractionProvenance records locally available revision and preprocessing metadata, selector configuration, dtype, device, dependency versions, and the VneuroTK writer version. Unknown values stay "unknown"; provenance discovery does not make registry requests.

An optional digest can be supplied when the caller has computed it over the ordered stimulus content:

with_digest = model.extract(
    images,
    batch_size=4,
    stimulus_content_hash="sha256:<caller-computed-digest>",
)
with_digest[0].provenance.stimulus_content_hash

Select modules and stimuli#

one_module = representations[representations.meta["module_name"] == first.module_name]
first_again = one_module[0]
first_two_stimuli = representations.select(list(representations.stim_ids[:2]))