{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "vision_union-01",
   "metadata": {},
   "source": "# Extract vision features with VneuroTK data\n\n`data.vision.extract_from()` uses the image database bound by `BaseData.configure()`. Features are stored once per unique stimulus; indexing produces arrays aligned to `VisionData.output_order` and therefore to the recording's trial order.\n\nThis notebook requires a configured `data` object from the {doc}`neural-data notebook <data>` and a selected `model` from the {doc}`vision-model notebook <vision_models>`. The model backend may require optional dependencies and cached or downloadable assets; the documentation build does not execute these cells."
  },
  {
   "cell_type": "markdown",
   "id": "vision_union-02",
   "metadata": {},
   "source": [
    "## Extract into the recording"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vision_union-03",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import torch\n",
    "\n",
    "import vneurotk as vtk\n",
    "\n",
    "rng = np.random.default_rng(0)\n",
    "stim_ids = np.array([\"image-1\", \"image-2\", \"image-1\"])\n",
    "data = vtk.BaseData.for_continuous(\n",
    "    neuro=np.arange(80, dtype=float).reshape(20, 4),\n",
    "    neuro_info={\"ch_names\": [\"a\", \"b\", \"c\", \"d\"], \"sfreq\": 10.0},\n",
    ")\n",
    "data.configure(\n",
    "    vision_onsets=np.array([2, 8, 14]),\n",
    "    stim_ids=stim_ids,\n",
    "    vision_db={stim_id: rng.integers(0, 256, (64, 64, 3), dtype=np.uint8) for stim_id in np.unique(stim_ids)},\n",
    "    trial_window=[-1, 3],\n",
    ")\n",
    "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
    "model = vtk.VisionModel(\n",
    "    \"facebook/dinov2-base\",\n",
    "    backend=\"transformers\",\n",
    "    device=device,\n",
    ")\n",
    "model.set_selector(module_name=\"layernorm\")\n",
    "data.vision.extract_from(model, batch_size=2)\n",
    "data.vision.meta"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vision_union-04",
   "metadata": {},
   "source": [
    "Repeating extraction with the same model is a no-op. If only selected modules are missing, existing records retain their provenance and newly extracted records receive current provenance. `overwrite=True` replaces each selected record together with its provenance."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vision_union-05",
   "metadata": {},
   "source": [
    "## Index trial-aligned features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vision_union-06",
   "metadata": {},
   "outputs": [],
   "source": [
    "first_name = data.vision.meta.iloc[0][\"module_name\"]\n",
    "trial_aligned = data.vision[first_name]\n",
    "trial_aligned.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vision_union-07",
   "metadata": {},
   "outputs": [],
   "source": [
    "mask = data.vision.meta[\"module_type\"] == data.vision.meta.iloc[0][\"module_type\"]\n",
    "selected = data.vision[mask]\n",
    "# One match returns an ndarray; multiple matches return VisualRepresentations.\n",
    "type(selected)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vision_union-08",
   "metadata": {},
   "source": [
    "## Add another model or overwrite selected records"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vision_union-09",
   "metadata": {},
   "outputs": [],
   "source": [
    "# model2 = vtk.VisionModel(\"resnet50.a1_in1k\", backend=\"timm\", device=device)\n",
    "# model2.set_selector(module_name=\"global_pool\")\n",
    "# data.vision.extract_from(model2, batch_size=4)\n",
    "\n",
    "# To intentionally refresh records selected on model:\n",
    "# data.vision.extract_from(model, batch_size=4, overwrite=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vision_union-10",
   "metadata": {},
   "source": [
    "## Persist features with neural data\n",
    "\n",
    "`BaseData.save()` stores neural data, trial configuration, visual representations, and structured extraction provenance in one HDF5 recording."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vision_union-11",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import vneurotk as vtk\n",
    "\n",
    "output = vtk.VTKPath(Path(\"outputs\"), subject=\"01\", task=\"demo\")\n",
    "# data.save(output)\n",
    "# loaded = vtk.read(output)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vision_union-12",
   "metadata": {},
   "source": "## Related documentation\n\n- {doc}`Build VneuroTK neural data <data>`\n- {doc}`Standalone vision extraction <vision_alone>`\n- [VneuroTK HDF5 format](../format/hdf5.md)\n- [Vision API](../api/vision.md)"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}