Welcome to the In Silico Framework (ISF)

ISF is a multi-scale simulation environment for the generation, simulation, and analysis of neurobiologically tractable single cell and network-level simulations.

image1

Packages in ISF

Packages in this repository you will most likely directly interact with

Module

Description

singlecell_input_mapper

Module for generating anatomical models, i.e. determining number and location of synapses and number and location of presynaptic cells

single_cell_parser

High level interface to the NEURON simulator providing methods to perform single cell simulations with synaptic input. The anatomical constraints of the synaptic input are provided by the single_cell_input_mapper module.

simrun

High level interface to the single_cell_parser module providing methods for common simulation tasks. It also provides methods for building reduced models mimicing the full compartmental model.

data_base

Flexible database whose API mimics a python dictionary. It provides efficient and scalable methods to store and access simulation results at a terrabyte scale. It also generates metadata, indicating when the data was put in the database and the exact version of the in_silico_framework that was used at this timepoint. Simulation results from the single_cell_parser module can be imported and converted to a high performance binary format.

How should I interact with these packages?

The recommended way is to use the Interface module which provides the API necessary to perform simulation tasks.

[1]:

import Interface as I

[INFO] ISF: Current version: heads/tutorials+0.g2c5677a12.dirty
[INFO] ISF: Current pid: 1788200

Warning: no DISPLAY environment variable.
--No graphics will be displayed.

[INFO] mechanisms: Loaded mechanisms in NEURON namespace.
[ATTENTION] ISF: The source folder has uncommited changes!



[INFO] ISF: Loaded modules with __version__ attribute are:
IPython: 8.18.1, Interface: heads/tutorials+0.g2c5677a12.dirty, PIL: 11.3.0, _brotli: 1.1.0, _csv: 1.0, _ctypes: 1.1.0, _curses: b'2.2', _decimal: 1.70, argparse: 1.1, blosc: 1.11.3, bluepyopt: 1.9.126, brotli: 1.1.0, certifi: 2025.08.03, charset_normalizer: 3.4.3, click: 7.1.2, cloudpickle: 3.1.1, colorama: 0.4.6, comm: 0.2.3, csv: 1.0, ctypes: 1.1.0, cycler: 0.12.1, cytoolz: 1.0.1, dash: 2.18.2, dask: 2.30.0, dateutil: 2.9.0.post0, deap: 1.4, debugpy: 1.8.16, decimal: 1.70, decorator: 5.2.1, defusedxml: 0.7.1, distributed: 2.30.0, distutils: 3.9.23, entrypoints: 0.4, exceptiongroup: 1.3.0, executing: 2.2.0, fasteners: 0.17.3, flask: 1.1.4, fsspec: 2025.7.0, future: 1.0.0, greenlet: 3.2.4, idna: 3.10, ipaddress: 1.0, ipykernel: 6.29.5, ipywidgets: 8.1.7, isf_pandas_msgpack: 0.4.0, itsdangerous: 1.1.0, jedi: 0.19.2, jinja2: 2.11.3, joblib: 1.5.1, json: 2.0.9, jupyter_client: 7.3.4, jupyter_core: 5.8.1, kiwisolver: 1.4.7, logging: 0.5.1.2, markupsafe: 2.0.1, matplotlib: 3.3.2, msgpack: 1.1.1, neuron: 7.8.2+, numcodecs: 0.12.1, numexpr: 2.9.0, numpy: 1.19.2, packaging: 25.0, pandas: 1.1.3, parso: 0.8.4, past: 1.0.0, pexpect: 4.9.0, pickleshare: 0.7.5, platform: 1.0.8, platformdirs: 4.3.8, plotly: 5.24.1, prompt_toolkit: 3.0.51, psutil: 7.0.0, ptyprocess: 0.7.0, pure_eval: 0.2.3, pydevd: 3.2.3, pygments: 2.19.2, pyparsing: 3.2.3, pytz: 2025.2, re: 2.2.1, requests: 2.32.5, scipy: 1.5.2, seaborn: 0.12.2, six: 1.17.0, sklearn: 0.23.2, socketserver: 0.4, socks: 1.7.1, sortedcontainers: 2.4.0, stack_data: 0.6.3, statsmodels: 0.12.1, tables: 3.8.0, tblib: 3.1.0, tlz: 1.0.1, toolz: 1.0.0, tqdm: 4.67.1, traitlets: 5.14.3, urllib3: 2.5.0, wcwidth: 0.2.13, werkzeug: 1.0.1, yaml: 6.0.2, zarr: 2.15.0, zlib: 1.0, zmq: 27.0.2, zstandard: 0.23.0

Now, you can access top-level API directly:

[2]:

I.scp # single_cell_parser package
I.sca # single_cell_analyzer package
I.DataBase # main class of isf_data_base
I.map_singlecell_inputs # compute anatomical model for a given cell morphology in barrel cortex
I.simrun_run_new_simulations # default function for running single cell simulations with well constrained synaptic input
I.db_init_simrun_general.init # default method to initialize a model data base with existing simulation results
I.db_init_simrun_general.optimize # converts the data to speed optimized compressed binary format
I.synapse_activation_binning_dask # parallelized binning of synapse activation data
I.rm_get_kernel # create reduced lda model from simulation data

[2]:

<module 'simrun.reduced_model.get_kernel' from '/gpfs/soma_fs/scratch/meulemeester/project_src/in_silico_framework/simrun/reduced_model/get_kernel.py'>

Use autocompletion of IPython to find the other modules. To quickly inspect the documentation from within the IPython or Jupyter session, use a questionmark:

[3]:

I.db_init_simrun_general.init?

Signature:
I.db_init_simrun_general.init(
    db,
    simresult_path,
    core=True,
    synapse_activation=True,
    dendritic_voltage_traces=True,
    parameterfiles=True,
    spike_times=True,
    repartition=True,
    scheduler=None,
    rewrite_in_optimized_format=True,
    dendritic_spike_times=True,
    dendritic_spike_times_threshold=-30.0,
    client=None,
    check_health=False,
    n_chunks=5000,
    paramfile_copy_config=None,
    voltage_traces=None,
    burst_times=None,
    dumper=None,
)
Docstring:
Initialize a database with simulation data.

Use this function to load simulation data generated with the simrun module
into a :class:`~data_base.DataBase`.

Most configuration options you would want to change on a init-by-init basis are given here as keyword arguments.
Additional options can be configured in :mod:`data_base.db_initializers.load_simrun_general.config`

Args:
    db (:class:`~data_base.DataBase`):
        The database to store the simulation results in.
    core (bool, optional):
        Parse and write the core data to the database: voltage traces, metadata, sim_trial_index and filelist.
    spike_times (bool, optional):
        Parse and write the spike times into the database.
        See also: :func:`data_base.analyze.spike_detection.spike_detection`
    dendritic_voltage_traces (bool, optional):
        Parse and write the dendritic voltage traces to the database.
    dendritic_spike_times (bool, optional):
        Parse and write the dendritic spike times to the database.
    dendritic_spike_times_threshold (float, optional):
        Threshold for the dendritic spike times in :math:`mV`. Default is :math:`-30 mV`.
    synapse_activation (bool, optional):
        Parse and write the synapse activation data to the database.
    parameterfiles (bool, optional):
        Resolve and copy the parameterfiles used in each simulation to the database.
        Since this copies all parameterfiels required to rerun simulations, this makes the database self-containing and portable.
        You could then remove the original raw simulation data, provided you have selected all desired results in the keyword arguments here.
    rewrite_in_optimized_format (bool, optional):
        If True (default): data is converted to a high performance binary format
        If False: the db only contains links (pickled objects) to the actual simulation data folder
        and will not work if the data folder is deleted or moved or transferred to another machine.
    repartition (bool|int):
        If ``int``, the voltage trace dataframes will be partitioned to be of this length (approximately).
        If ``True``, the voltage trace dataframes will be partitioned to be of a default length: :attr:`~data_base.db_initializers.load_simrun_general.data_parsing.DEFAULT_VT_PARTITION_SIZE`
        If ``False``, the voltage trace dataframe will not be repartitioned, and the dask dataframe will be one ``.csv`` file per partition.
    n_chunks (int, optional):
        Amount of partitions to split the :ref:`syn_activation_format` and :ref:`spike_times_format` dataframes into.
        Default is :math:`5000`.
    paramfile_copy_config (dict, optional):
        Dictionary containing configuration on how to organise parameterfiles in the database. Options are:

        - ``copy_method`` (str): Which copy strategy to use.
          Must be either ``"hash_rename"`` or ``"remount"``.
          ``"hash_rename"`` will rename all parameterfiles to a hash of their content. Useful when you want all parameter files in one folder and avoid fielname clashes.
          ``"remount"`` will preserve the relative directory structure of the parameterfiles per file category (see below). Useful when parameterfiles are already organized.
        - "neup" (str): Target directory name of :ref:`cell_parameters_format`. Default is ``"parameterfiles_folder"``
        - "netp" (str): Target directory name of :ref:`network_parameters_format`. Default is ``"parameterfiles_folder"``
        - "morph" (str): Target directory name of :ref:`morphology_file_format` files. Default is ``"parameterfiles_folder"``
        - "syn" (str): Target directory name of :ref:`syn_file_format` files. Default is ``"parameterfiles_folder"``
        - "con" (str): Target directory name of :ref:`con_file_format` files. Default is ``"parameterfiles_folder"``
        - "recsites" (str): Target directory name of recordingsites (``.landmarkAscii``). Default is ``"parameterfiles_folder"``

    client (:class:`distributed.Client`, optional):
        Distributed Client object for parallel parsing of anything that isn't a dask dataframe.
    scheduler (:class:`distributed.Client`, optional)
        Scheduler to use for parallellized parsing of dask dataframes.
        can e.g. be simply the ``distributed.Client.get`` method.
        Default is ``None``.


Note:
    Note the difference between *amount* of partitions (:param:`n_chunks`) and target partition *size* (:param:`repartition`)

.. deprecated:: 0.2.0
    The :paramref:`burst_times` argument is deprecated and will be removed in a future version.

.. versionadded:: 0.5.0
   The keyword argument :param:`repartition` now accepts integers in addition to booleans.
   Integers reflect the target size of each voltage trace dataframe partition. Booleans reflect either
   ``True`` for a default lenght, or ``False`` for no repartitioning (i.e. one ``.csv`` file per partition)

.. deprecated:: 0.2.0
    The :param:`burst_times` argument is deprecated and will be removed in a future version.

.. deprecated:: 0.5.0
   The :param:`dumper` argument is deprecated and will be removed in a future version.
   Dumpers are configured in the centralized :mod:`~data_base.db_initializers.load_simrun_general.config` module.

.. deprecated:: 0.5.0
   The :param:`voltage_traces` is deprecated and will be removed in a future version.
   Voltage traces are always built when :param:`core` is ``True``.
File:      /gpfs/soma_fs/scratch/meulemeester/project_src/in_silico_framework/data_base/db_initializers/load_simrun_general/__init__.py
Type:      function

Example simulation

As an example to showcase what ISF can do, we will:

  1. Simulate a reconstructed neuron morphology with known neuron parameters

    • … to in vivo like synaptic input conditions defined in network parameters

    • … parallellized using a scheduler, to highlight ISF’s parallellization and HPC capabilities

    • … and save the results to a DataBase

  2. Parse the simulation data to an optimized format, to highlight the capabilities of the DataBase package

  3. Visualize the simulation using the visualize package

How to build and interact with these parameters will be covered in the next tutorials.

1. Simulating a neuron model under in vivo synaptic input conditions

[4]:

from pathlib import Path
from getting_started import getting_started_dir

tutorial_output_dir = f"{Path.home()}/isf_tutorial_output"  # <-- Change this to your desired output directory
parent_db = I.DataBase(tutorial_output_dir)
db = parent_db.create_sub_db("intro_to_isf")
# Create new folder for results
db.create_managed_folder('simrun', raise_ = False)
raw_sim_output_dirname = db['simrun'].join('example')

example_data_dir = Path(getting_started_dir) / "example_data"
neuron_param = I.scp.build_parameters(example_data_dir / 'biophysical_constraints' / '86_C2_center.param')
network_param = I.scp.build_parameters(example_data_dir / 'functional_constraints' / 'network.param')

[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] client: Connection dropped: socket connection broken
[WARNING] client: Transition to CONNECTING
[WARNING] client: Session has expired

[5]:

nSweeps = 1     # number of consecutive simulation trials per process
nprocs = 1      # number of processes simulating that task in parallel
tStop = 300     # stop the simulation at 300ms

d = I.simrun_run_new_simulations(
    neuron_param,
    network_param,
    dirPrefix = raw_sim_output_dirname,
    nSweeps = nSweeps,
    nprocs = nprocs,
    tStop = tStop,
    silent = False
)

We now have a list of delayed functions. These contain the instructions to run the simulation, but ar enot yet executed. We can use these delayed functions to organize the parallellization of the execution. A common way of running these in parallel is to launch a dask scheduler on some remote machine that has enough CPU power, and pass these delayed functions to that scheduler. This can be done using the dask Client. The Client is your contact point to the remote scheduler.

[6]:

dask_client = I.get_client()

[7]:

# execute the delayed function on the remote scheduler, save the progress as a variable
pending_results = dask_client.compute([d])

You can use dask’s dashboard to inspect the progress:

[8]:

dask_client

[8]:

Client

Cluster

  • Workers: 8
  • Cores: 8
  • Memory: 128.85 GB
 
[9]:

from dask.distributed import wait
wait(pending_results)  # wait until simulations are done before continuing

[9]:

DoneAndNotDoneFutures(done={<Future: finished, type: builtins.list, key: list-4b1fe0c9-1b99-404a-bf37-0a8c7629c175>}, not_done=set())

The example above only had a single pair of neuron and network parameters, and only a single simulation run for those parameters. For most use cases, you will likely want to change those parameters and run multiple simulations per parameter set to capture biological variability. For example:

  • Different network parameters capture different synapse locations and activations on the neuron model

  • Different neuron parameters capture different morphologies and biophysical properties of the neuron model

  • Multiple simulation runs capture the trial-to-trial variability, sampling the stochasticity of some parameters like synapse activation probabilities

2. Parsing simulation data

Simulation output is written out in simple .csv format. This is very useful for simulations in the exploratory stage. However, for high-performant large-scale simulations that are intended to extensively explore biological diversity, you probably want something more performant that ASCII csv.

For this purpose, the data_base package provides data parsers that aggregate raw csv output from these simulations that convert them to an optimized format (currently this is msgpack with blosc compression)

[13]:

I.db_init_simrun_general.init(
    db=db,
    simresult_path=raw_sim_output_dirname,
    client=dask_client,
    paramfile_copy_config={"copy_method": "hash_rename"}  # rename parameterfiles to a hash - useful when running only a single simulation
)

[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.

We can inspect what’s in the database:

[14]:

db

[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.
[WARNING] isf_data_base: ISF has uncommitted changes - reproducing results may not be possible.

[14]:

<data_base.isf_data_base.ISFDataBase object at 0x153932ebf670>
Located at /gpfs/soma_fs/home/meulemeester/isf_tutorial_output/intro_to_isf/db
db
├── cell_activation
├── voltage_traces
├── simresult_path
├── metadata
├── parameterfiles
├── filelist
├── spike_times
├── simrun
├── dendritic_spike_times
│   └── 000_-30.0
├── dendritic_recordings
│   └── 000
├── synapse_activation
├── parameterfiles_folder
└── sim_trial_index

3. Visualizing the simulation

We can create a Cell object from each simulation trial: a dataclass containing the simulation results.

[15]:

cell = I.simrun_simtrial_to_cell_object(
    db,
    sim_trial_index = db['sim_trial_index'][0]
)

[25]:

from matplotlib.pyplot import cm

sa = db["synapse_activation"]
unique_syn_types = I.np.unique(sa["synapse_type"])

cmv = I.CellMorphologyVisualizer(
    cell,
    t_start=245-25,
    t_stop=245+25,
    t_step=0.2
)

synapse_colors = cm.rainbow(range(len(unique_syn_types)))
cmv.population_to_color_dict = dict(zip(unique_syn_types, synapse_colors))

images_path = str(db._basedir / 'C2_stim_animation_3d')
cmv.animation(
    images_path=images_path,
    color="voltage",
    show_synapses=True,
    client=dask_client,  # parallellize the creation of frames
);


Once Loop Reflect