Vision

DNN vision representation module.

VisionData

class vneurotk.vision.data.VisionData(output_order, vision_db=None)

Aligned storage and view for Visual Representations within a Recording.

Stores unique-stimulus activations internally as a dict[(model_id, module_name), VisualRepresentation] and re-indexes to output_order (typically BaseData.trial_stim_ids) at read time.

All storage, HDF5 persistence, and trial-aligned view logic live here; there is no separate inner storage class.

Attributes:

dbAny

Original stimulus image database.

output_ordernp.ndarray

Sequence of stimulus IDs defining the desired output ordering.

metapd.DataFrame

DataFrame with columns model, module_type, module_name, shape.

Parameters:

• output_order (np.ndarray)

• vision_db (Any)

add(visual_representations, overwrite=False)

Add unique source records after validating stimulus coverage.

Parameters:

visual_representationsVisualRepresentations

overwritebool

Parameters:

• visual_representations (VisualRepresentations)

• overwrite (bool)

Return type:

None

attach_db(db)

Set or replace the image database attached to this VisionData.

Parameters:

dbAny

Parameters:

db (Any)

Return type:

None

by_module(name, model=None)

Return the output-order-aligned activation array for name.

Parameters:

namestr

modelstr or None

Disambiguates when multiple models share the same module name.

Returns:

np.ndarray

Shape (n_output_order_items, ...).

Parameters:

• name (str)

• model (str | None)

Return type:

ndarray

property db: Any

Original stimulus image database.

dump(f, group_name='vision_store', storage_options=None)

Serialize stored records to an HDF5 group.

Parameters:

fh5py.File

group_namestr

Parameters:

• f (h5py.File)

• group_name (str)

• storage_options (Any)

Return type:

None

extract_from(model, vision_db=None, *, batch_size=32, overwrite=False, stimulus_content_hash=None)

Extract DNN features and store them.

Accepts either a pre-built VisionModel or a model-id string (in which case a VisionModel is built internally with default settings — use VisionModel directly for custom backends/selectors).

Parameters:

modelVisionModel or str

A configured VisionModel, or a model-id string like "facebook/dinov2-base" / "resnet50". When a string is given, backend defaults to "transformers", device to "cpu", and the default BlockLevelSelector is used.

vision_dbAny, optional

Image source {stim_id: image}. Uses the already-attached db if None.

batch_sizeint

Images per forward pass. Default 32.

overwritebool

Replace existing records for the same (model_id, module_name) key.

stimulus_content_hashstr or None

Optional caller-computed digest attached to records extracted in this call.

Raises:

RuntimeError

If no image source is available.

Parameters:

• model (Any)

• vision_db (Any)

• batch_size (int)

• overwrite (bool)

• stimulus_content_hash (str | None)

Return type:

None

classmethod from_h5(f, output_order, group_name='vision_store', vision_db=None, fpath=None, file_identity=None)

Reconstruct from an HDF5 group.

Parameters:

fh5py.File

output_ordernp.ndarray

group_namestr

vision_dbAny, optional

fpathPath or str, optional

File path used to create lazy array loaders. When provided, activation arrays are not loaded into memory until first access.

Returns:

VisionData

Parameters:

• f (h5py.File)

• output_order (np.ndarray)

• group_name (str)

• vision_db (Any)

• fpath (Any)

• file_identity (Any)

Return type:

VisionData

property has_visual_representations: bool

Whether any VisualRepresentations have been stored.

property meta: DataFrame

DataFrame with columns model, module_type, module_name, shape.

property output_order: ndarray

Sequence of stimulus IDs defining the desired output ordering.

VisionModel

class vneurotk.vision.model.base.VisionModel(model_id, backend='transformers', selector=None, device='cpu', pretrained=True)

Unified interface for extracting DNN activations from images.

Composes a BaseBackend and a ModuleSelector. Activations are returned as-is; any further processing (pooling, embedding, etc.) is left to the user.

Parameters:

model_idstr

Model identifier passed directly to the backend, e.g. "facebook/dinov2-base" (transformers) or "resnet50" (timm).

backendstr

Backend to use: "transformers" (default), "timm", or "thingsvision".

selectorModuleSelector or None

Layer selection strategy. Defaults to BlockLevelSelector.

devicestr

Inference device (default "cpu").

pretrainedbool

Load pretrained weights (default True).

Parameters:

• model_id (str)

• backend (str)

• selector (ModuleSelector | None)

• device (str)

• pretrained (bool)

extract(image, batch_size=32, show_progress=True, *, stimulus_content_hash=None)

Extract DNN activations for one image or a collection of stimuli.

Parameters:

imagePIL.Image.Image or np.ndarray or str or Path or dict

Single image → n_sample=1, stim_id=0. dict mapping stim_ids to images → n_sample=len(dict). String / pathlib.Path values are opened automatically. list is not accepted — use a dict with explicit stim IDs to preserve alignment with BaseData.trial_stim_ids.

batch_sizeint

Number of images per GPU forward pass. Default 32. Ignored for single-image input.

show_progressbool

Display a tqdm progress bar over batches. Automatically suppressed for single-image input. Default True.

stimulus_content_hashstr or None

Optional caller-computed digest of the ordered stimulus content. VneuroTK does not read or hash images implicitly.

Returns:

VisualRepresentations

Parameters:

• image (Any)

• batch_size (int)

• show_progress (bool)

• stimulus_content_hash (str | None)

Return type:

VisualRepresentations

extract_for_modules(images, module_names, batch_size, show_progress=True, *, stimulus_content_hash=None)

Extract activations for a subset of modules without altering state.

Temporarily re-registers hooks for module_names, runs extraction, then restores the original hook configuration.

Parameters:

imagesdict

{stim_id: image} mapping.

module_nameslist[str]

Subset of module names to extract.

batch_sizeint

Images per forward pass.

show_progressbool

Show tqdm progress bar.

stimulus_content_hashstr or None

Optional caller-computed stimulus content digest.

Returns:

VisualRepresentations

Parameters:

• images (dict)

• module_names (list[str])

• batch_size (int)

• show_progress (bool)

• stimulus_content_hash (str | None)

Return type:

VisualRepresentations

classmethod from_model(model, backend, selector=None, provenance=None)

Build a VisionModel from an already-loaded model.

Parameters:

modelnn.Module

Pre-loaded PyTorch model.

backendBaseBackend

Backend instance with model already assigned.

selectorModuleSelector or None

Defaults to BlockLevelSelector.

provenanceExtractionProvenance or None

Explicit metadata for a caller-supplied model. If omitted, locally discoverable backend metadata is used and unavailable fields remain "unknown".

Returns:

VisionModel

Parameters:

• model (Any)

• backend (BaseBackend)

• selector (ModuleSelector | None)

• provenance (ExtractionProvenance | None)

Return type:

VisionModel

property model_id: str

Model identifier (e.g. 'facebook/dinov2-base', 'resnet50').

property module_list: list

All modules available in the loaded model.

Returns:

list[ModuleInfo]

One entry per named module, ordered as model.named_modules(). Each entry exposes .name, .module_type, .depth, and .n_params.

property module_names: list[str]

Names of all currently hooked modules.

print_modules(max_depth=None, console=None)

Print a tree-style summary of all model modules.

Parameters:

max_depthint or None

Maximum nesting depth to display. None shows all levels.

consolerich.console.Console or None

Rich console to use for output. Pass Console(record=True) to capture the output for SVG / HTML export.

Parameters:

• max_depth (int | None)

• console (Any)

Return type:

None

property provenance: ExtractionProvenance

Base extraction provenance, excluding an optional stimulus hash.

classmethod register_backend(name, backend_cls)

Register a custom backend class under name.

Parameters:

namestr

Key used in the backend= argument of VisionModel.

backend_clstype[BaseBackend]

Backend class to register. Must be a concrete subclass of BaseBackend.

Parameters:

• name (str)

• backend_cls (type[BaseBackend])

Return type:

None

set_selector(selector=None, *, module_type=None, module_name=None)

Replace the layer selector and re-register hooks.

Accepts the following forms (combinable):

• set_selector(BlockLevelSelector()) — explicit selector object

• set_selector(["layer.0", "layer.1"]) — list of module names / ModuleInfo

• set_selector(module_type="Dinov2Layer") — all modules of that type

• set_selector(module_type=["Dinov2Layer", "LayerNorm"]) — multiple types

• set_selector(module_name="encoder.layer.3") — single module by name

• set_selector(module_name=["enc.0", "enc.6"]) — multiple names

• set_selector(module_type="Dinov2Layer", module_name="layernorm") — union of both filters

selector and (module_type / module_name) are mutually exclusive.

Parameters:

selectorModuleSelector, list, or None

Explicit selector object or list of module names / ModuleInfo objects.

module_typestr, list[str], or None

Hook all modules whose module_type is in this set.

module_namestr, list[str], or None

Hook modules whose name is in this set (exact match).

Raises:

ValueError

If no arguments are supplied, selector is combined with filters, or the resulting module list is empty.

Parameters:

• selector (ModuleSelector | list | None)

• module_type (str | list[str] | None)

• module_name (str | list[str] | None)

Return type:

None

Backend interface

Concrete backends are normally selected through VisionModel; BaseBackend documents the interface implemented by each backend.

class vneurotk.vision.model.backend.base.BaseBackend(device='cpu')

Abstract base for all feature-extraction backends.

Subclasses implement load(), preprocess(), forward(), and get_model_meta(). Hook management and module enumeration are provided here and shared.

Parameters:

devicestr or torch.device

Device for inference (default "cpu").

Parameters:

device (str | torch.device)

class vneurotk.vision.model.backend.transformers_backend.TransformersBackend(device='cpu')

Backend powered by HuggingFace transformers.

Supports any vision model loadable via AutoModel.from_pretrained(). CLIP and SigLIP models are detected by name and loaded with the appropriate model class; all hook management uses hookable_model.

Parameters:

devicestr or torch.device

Inference device (default "cpu").

Parameters:

device (str | torch.device)

class vneurotk.vision.model.backend.timm_backend.TimmBackend(device='cpu')

Backend powered by the timm library.

Any model available via timm.create_model() is supported. Preprocessing uses the model’s registered data config so no ImageNet mean/std are hard-coded.

Parameters:

devicestr or torch.device

Inference device (default "cpu").

Parameters:

device (str | torch.device)

class vneurotk.vision.model.backend.thingsvision_backend.ThingsVisionBackend(source='torchvision', device='cpu')

Backend powered by the thingsvision library.

thingsvision must be installed before instantiating this class; a missing installation raises ImportError immediately (fail-fast).

Parameters:

sourcestr

Model source string for thingsvision, e.g. "timm" or "torchvision".

devicestr or torch.device

Inference device (default "cpu").

Parameters:

• source (str)

• device (str | torch.device)

Model and module utilities

vneurotk.vision.model.base.print_modules(layers, max_depth=None, console=None)

Print a tree-style summary of model layers.

Parameters:

layerslist[ModuleInfo]

Module list returned by VisionModel.module_list.

max_depthint or None

Maximum nesting depth to display. None shows all levels.

consolerich.console.Console or None

Rich console to use for output. Pass Console(record=True) to capture the output for SVG / HTML export.

Parameters:

• layers (list)

• max_depth (int | None)

• console (Any)

Return type:

None

class vneurotk.vision._cache.CachedModel(model_id, source, size_bytes, last_used)

Metadata for a locally cached model.

Parameters:

model_idstr

Model identifier or filename.

sourceSource

Cache origin: "transformers", "timm", "torch", "clip", or "keras".

size_bytesint

Total size on disk in bytes.

last_useddatetime

Last access time.

Parameters:

• model_id (str)

• source (Literal['transformers', 'timm', 'torch', 'clip', 'keras'])

• size_bytes (int)

• last_used (datetime)

vneurotk.vision._cache.find_cached_models(hf_cache_dir=None)

Return all locally cached models across known cache locations.

Parameters:

hf_cache_dirstr or Path or None

HuggingFace hub cache directory. Defaults to huggingface_hub.constants.HF_HUB_CACHE, which respects HF_HUB_CACHE and HF_HOME environment variables.

Returns:

list[CachedModel]

One entry per cached model, sorted by source then model_id.

Parameters:

hf_cache_dir (str | Path | None)

Return type:

list[CachedModel]

vneurotk.vision._cache.print_cached_models(models=None, hf_cache_dir=None, console=None)

Print a coloured summary of locally cached models.

Parameters:

modelslist[CachedModel] or None

Pre-fetched list from find_cached_models(). If None, calls find_cached_models() automatically.

hf_cache_dirstr or Path or None

Forwarded to find_cached_models() when models is None.

consolerich.console.Console or None

Rich console to use for output. Pass Console(record=True) to capture the output for SVG / HTML export. Defaults to a new Console() that writes to stdout.

Parameters:

• models (list[CachedModel] | None)

• hf_cache_dir (str | Path | None)

• console (Console | None)

Return type:

None

Module Selectors

class vneurotk.vision.model.selector.ModuleSelector

Abstract base class for layer selection strategies.

Subclasses implement select(), which receives a list of ModuleInfo objects and returns an ordered list of module name strings to hook.

describe()

Return a stable, human-readable selector description.

Return type:

str

abstractmethod select(modules)

Return layer names to hook.

Parameters:

moduleslist[ModuleInfo]

All named modules enumerated by the backend, as returned by enumerate_modules().

Returns:

list[str]

Ordered module names to register hooks on.

Parameters:

modules (list[ModuleInfo])

Return type:

list[str]

class vneurotk.vision.model.selector.BlockLevelSelector(max_depth=2, include_patterns=None, arch_patterns=None)

Select major block-level modules appropriate for the architecture.

Uses regex patterns matched against module names. Architecture patterns are tried in order; the first match wins. Falls back to top-level children (depth == 1) if no pattern matches.

Parameters:

max_depthint

Maximum nesting depth to include (default 2). Controls how deeply nested sub-blocks are included.

include_patternslist[str] or None

Additional regex patterns to include alongside defaults.

Parameters:

• max_depth (int)

• include_patterns (list[str] | None)

• arch_patterns (list[tuple[str, int]] | None)

classmethod default_patterns()

Return a copy of the built-in architecture patterns.

Returns:

list[tuple[str, int]]

Each element is (regex_pattern, max_depth). Mutating the returned list does not affect the class default.

Return type:

list[tuple[str, int]]

describe()

Return the configured depth and pattern lists.

Return type:

str

select(modules)

Select block-level layers from modules.

Parameters:

moduleslist[ModuleInfo]

Returns:

list[str]

Parameters:

modules (list[ModuleInfo])

Return type:

list[str]

class vneurotk.vision.model.selector.AllLeafSelector(exclude_types=None)

Select all leaf modules (modules with no children).

Parameters:

exclude_typestuple[type, …] or None

Module types to skip. Defaults to activation and regularization layers that carry no representational content.

Parameters:

exclude_types (tuple | None)

describe()

Return excluded module type names in stable order.

Return type:

str

select(modules)

Return names of all non-excluded leaf modules.

Parameters:

moduleslist[ModuleInfo]

Returns:

list[str]

Parameters:

modules (list[ModuleInfo])

Return type:

list[str]

class vneurotk.vision.model.selector.CustomSelector(layer_names)

Use an explicit user-supplied list of layer names.

Parameters:

layer_nameslist[str] or list[ModuleInfo]

Exact module names as they appear in model.named_modules(), or ModuleInfo objects as returned by VisionModel.module_list.

Raises:

ValueError

During select() if any name is not found in the module list.

Parameters:

layer_names (list)

describe()

Return the explicitly selected module names.

Return type:

str

select(modules)

Validate and return the configured layer names.

Parameters:

moduleslist[ModuleInfo]

Returns:

list[str]

Raises:

ValueError

If any layer name is absent from modules.

Parameters:

modules (list[ModuleInfo])

Return type:

list[str]

Representations

class vneurotk.vision.representation.visual_representations.VisualRepresentations(representations)

Collection of VisualRepresentation objects.

Returned by extract(). Supports DataFrame-style filtering via boolean masks on meta.

Parameters:

representationslist[VisualRepresentation]

Ordered list of atomic activation records.

Parameters:

representations (list[VisualRepresentation])

Examples

>>> visual_representations = model.extract(images)
>>> meta = visual_representations.meta
>>> subset = visual_representations[meta["module_type"] == "Dinov2Layer"]

by_module(name, model=None)

Return the VisualRepresentation for name.

Parameters:

namestr

Module name.

modelstr or None

Model identifier to disambiguate when multiple records share the same module name.

Raises:

KeyError

If name is not found, or is ambiguous and model was not given.

Parameters:

• name (str)

• model (str | None)

Return type:

VisualRepresentation

filter(mask)

Return a subset filtered by a 1-D boolean mask over meta rows.

Parameters:

maskpd.Series or np.ndarray of bool

Aligned to meta rows.

Parameters:

mask (Series | ndarray)

Return type:

VisualRepresentations

property meta: DataFrame

DataFrame with columns model, module_type, module_name, shape.

property module_names: list[str]

Module names of all contained records.

property n_stim: int

Number of stimuli (from first record, or 0 if empty).

numpy(layer)

Return activation array for layer, shape (n_stim, ...).

Parameters:

layerstr

Module name.

Parameters:

layer (str)

Return type:

ndarray

select(ids)

Return a subset of stimuli by their IDs across all records.

Parameters:

idslist or np.ndarray

Stimulus IDs to keep.

Parameters:

ids (list | ndarray)

Return type:

VisualRepresentations

select_by_index(indices)

Return a subset of stimuli by positional index across all records.

Parameters:

indiceslist or np.ndarray

Integer indices.

Parameters:

indices (list | ndarray)

Return type:

VisualRepresentations

property stim_ids: tuple

Stimulus IDs shared by all records.

to_tensor(layer)

Return activations for layer as a PyTorch tensor.

Parameters:

layerstr

Module name.

Parameters:

layer (str)

Return type:

Any

class vneurotk.vision.representation.visual_representations.VisualRepresentation(model, module_name, module_type, stim_ids, array=None, *, array_loader=None, shape=None, provenance=None, _allow_repeated_stim_ids=False)

Atomic activation record: one model × one module.

Parameters:

modelstr

Model identifier, e.g. "facebook/dinov2-base".

module_namestr

Module name as from named_modules(), e.g. "encoder.layer.11".

module_typestr

Class name of the module, e.g. "Dinov2Layer".

stim_idslist

Ordered stimulus identifiers corresponding to the first axis of array.

arraynp.ndarray or None

Activation array of shape (n_stim, ...). Mutually exclusive with array_loader; one of the two must be provided.

array_loadercallable or None

Zero-argument callable that returns the activation array on first access. Used for lazy loading from HDF5. Mutually exclusive with array.

shapetuple or None

Pre-computed shape to return from shape without triggering array loading. Required when array_loader is given; ignored otherwise.

Parameters:

• model (str)

• module_name (str)

• module_type (str)

• stim_ids (list)

• array (np.ndarray | None)

• array_loader (Callable[[], np.ndarray] | None)

• shape (tuple | None)

• provenance (ExtractionProvenance | None)

• _allow_repeated_stim_ids (bool)

property array: ndarray

Activation array, loaded lazily from HDF5 if constructed with array_loader.

property n_stim: int

Number of stimuli.

select(ids)

Return a subset of stimuli by their IDs.

Parameters:

idslist or np.ndarray

Stimulus IDs to select.

Returns:

VisualRepresentation

Parameters:

ids (list | ndarray)

Return type:

VisualRepresentation

property shape: tuple

Shape of the activation array (n_stim, ...).

Image Source

class vneurotk.vision.image_source.ImageSource(*args, **kwargs)

Protocol for any mapping from stimulus ID to image data.

Both StimulusSet and LazyH5Dict satisfy this protocol, as does a plain dict. Callers that accept stimulus images should annotate their parameter as ImageSource rather than enumerating concrete types.

Metadata

class vneurotk.vision.meta.ExtractionProvenance(backend='unknown', model_id='unknown', model_revision='unknown', pretrained='unknown', preprocessing='unknown', selector='unknown', dependency_versions=<factory>, dtype='unknown', device='unknown', writer_version='unknown', stimulus_content_hash=None)

Reproducibility metadata for one feature-extraction result.

The record deliberately uses the literal string "unknown" for metadata that cannot be discovered locally. This makes missing information explicit without requiring a model registry lookup or other network access.

Parameters:

backendstr

Backend that executed the model.

model_idstr

Backend-native model identifier.

model_revisionstr

Locally available model revision/commit, or "unknown".

pretrainedbool or str

Whether pretrained weights were requested, or "unknown".

preprocessingstr

Stable description of the processor or preprocessing transform.

selectorstr

Stable description of the module selector.

dependency_versionsmapping

Locally installed dependency versions. Missing versions are explicit as "unknown".

dtypestr

Model parameter dtype, or "unknown".

devicestr

Device used for inference, or "unknown".

writer_versionstr

VneuroTK version that created this provenance record.

stimulus_content_hashstr or None

Optional sha256:... digest of the ordered stimulus mapping.

Parameters:

• backend (str)

• model_id (str)

• model_revision (str)

• pretrained (bool | str)

• preprocessing (str)

• selector (str)

• dependency_versions (Mapping[str, str])

• dtype (str)

• device (str)

• writer_version (str)

• stimulus_content_hash (str | None)

classmethod from_dict(value)

Construct from a serialized mapping.

Absent fields are interpreted as explicit unknowns, which also makes the reader tolerant of early or manually-authored schema-1 records.

Parameters:

value (Mapping[str, Any])

Return type:

ExtractionProvenance

classmethod from_json(value)

Deserialize a record produced by to_json().

Parameters:

value (str | bytes)

Return type:

ExtractionProvenance

to_dict()

Return the complete, serialization-stable mapping representation.

Return type:

dict[str, Any]

to_json()

Serialize deterministically as compact UTF-8-safe JSON.

Return type:

str

classmethod unknown(*, model_id='unknown')

Return an explicit unknown record, retaining a known model ID.

Parameters:

model_id (str)

Return type:

ExtractionProvenance

class vneurotk.vision.meta.ModelInfo(model_id, backend)

Basic metadata for a loaded model.

Parameters:

model_idstr

Model identifier passed to the backend, e.g. "facebook/dinov2-base" or "resnet50".

backendstr

Backend used: "timm", "transformers", or "thingsvision".

Parameters:

• model_id (str)

• backend (str)

class vneurotk.vision.meta.ModuleInfo(name, module_type, depth, n_params=0, is_leaf=False, param_shapes=<factory>)

Metadata for an enumerated module.

Parameters:

namestr

Module name as from named_modules().

module_typestr

Class name of the module.

depthint

Nesting depth in the module tree.

n_paramsint

Total number of parameters in this module (including children).

is_leafbool

True if the module has no child modules (suitable for direct hooking).

param_shapesdict[str, tuple]

Shape of each directly-owned parameter (empty for container modules). E.g. {"weight": (768, 768), "bias": (768,)}.

Parameters:

• name (str)

• module_type (str)

• depth (int)

• n_params (int)

• is_leaf (bool)

• param_shapes (dict[str, tuple])