Vision model extraction#

VisionModel provides one interface across the transformers, timm, and thingsvision backends. Construction can require optional dependencies, network access, and model assets; run these cells in a suitable local environment.

Choose a backend#

Model IDs pass to the selected backend unchanged; VneuroTK does not substitute another backend or model.

import numpy as np
import torch

import vneurotk as vtk

device = "cuda" if torch.cuda.is_available() else "cpu"
model = vtk.VisionModel("facebook/dinov2-base", backend="transformers", device=device)

Select modules#

Replace the default block-level selector by exact module type, exact name, their union, a selector object, or a custom list from model.module_list.

model.set_selector(
    module_type="Dinov2Layer",
    module_name="layernorm",
)
selected_names = model.module_names

Extract a stimulus mapping#

Batch input is a {stimulus_id: image} mapping so activation rows retain identity. extract() returns VisualRepresentations, with one VisualRepresentation per selected module.

rng = np.random.default_rng(0)
images = {f"image-{i}": rng.integers(0, 256, (64, 64, 3), dtype=np.uint8) for i in range(4)}
representations = model.extract(images, batch_size=4, show_progress=False)

Index arrays and provenance#

A string or integer selects one VisualRepresentation. A boolean mask always returns a VisualRepresentations collection, even for one match. Use .array or .numpy(module_name) for activations and .select(ids) to align every module to a stimulus subset.

first = representations[0]
activation = representations.numpy(first.module_name)
matching = representations[representations.meta["module_name"] == first.module_name]
first_two = representations.select(list(representations.stim_ids[:2]))
provenance_record = first.provenance.to_dict()
assert provenance_record["backend"] == "transformers"
assert provenance_record["model_id"] == model.model_id

ExtractionProvenance records locally discoverable revision, preprocessing, selector, dependency versions, dtype, device, and writer version. Unavailable values remain "unknown"; callers may supply a stimulus-content digest to extract().