❭ API reference ❭ data_base ❭ db_initializers ❭

load_simrun_general¶

Parse Raw simulation results generated with simrun and write to a DataBase

The output format of simrun are Raw simulation results: a nested folder structure with .csv and/or .npz files. This module provides functions to gather and parse this data to pandas and dask dataframes. It merges all trials in a single dataframe. This saves IO time, disk space, and is strongly recommended for HPC systems and other shared filesystems in general, as it reduces the amount of inodes required.

After running init(), a database is created containing the following keys:

Key	Description
`simresult_path`	Filepath to the raw simulation output of `simrun`
`filelist`	List containing paths to all original somatic voltage trace files.
`sim_trial_index`	The simulation trial indices as a pandas Series.
`metadata`	A metadata dataframe out of sim_trial_indices
`voltage_traces`	Dask dataframe containing the somatic voltage traces
`parameterfiles_cell_folder`	A `ManagedFolder` containing the Cell parameters file, renamed to its file hash.
`parameterfiles_network_folder`	A `ManagedFolder` containing the Network parameters file, renamed to its file hash.
`parameterfiles`	A pandas dataframe containing the original paths of the parameter files and their hashes.
`synapse_activation`	Dask dataframe containing the parsed Synapse activation data.
`cell_activation`	Dask dataframe containing the parsed Presynaptic spike times.
`dendritic_recordings`	Subdatabase containing the membrane voltage at the recording sites specified in the Cell parameters as a dask dataframe.
`dendritic_spike_times`	Subdatabase containing the spike times at the recording sites specified in the Cell parameters as a dask dataframe.
`spike_times`	Dask dataframe containing the spike times of the postsynaptic cell for all trials.

If you initialize the database with rewrite_in_optimized_format=True (default), the keys are written as dask dataframes to whichever format is configured as the optimized format (see config). If rewrite_in_optimized_format=False instead, these keys are pickled dask dataframes, containing the instructions to build the dataframe, not the data itself.

Warning

Using pickle as a data format is useful for fast intermediate analysis, but strongly discouraged for long-term storage. These pickle files are instructions to build the data, not the data itself. The instructions contain absolute paths to the original data files, which invalidate once they are moved or deleted. They have a hard dependency on the loader functions, which invalidate if those are renamed or moved in the source code.

Hint

Individual data keys can afterwards be re-optimized to other data formats by calling optimize() on specific database keys. This is useful to convert intermediate pickle data to long-term stsorage formats.

Example:

>>> paramfile_copy_config = {
...     "copy_method": "remount",
...     "neup" : "parameterfiles_folder",
...     "netp" : "parameterfiles_folder",
...     "morph" : "morphology_folder",
...     "syn" : "parameterfiles_folder",
...     "con" : "parameterfiles_folder",
...     "recsites" : "parameterfiles_folder"
... }
>>> simresult_path = '/path/to/raw/simrun/output/folder'
>>> db = I.DataBase("db_parsed_data")
>>> client = distributed.Client("localhost:8786")
>>> I.db_init_simrun_general.init(
... db = db,
... simresult_path = p,
... core = True,
... repartition = 500,
... parameterfiles = True,
... synapse_activation = True,
... n_chunks = 5000,
... dendritic_voltage_traces = True,
... spike_times = True,
... dendritic_spike_times = False,
... rewrite_in_optimized_format = True,
... client = client,
... check_health = True,
... paramfile_copy_config = paramfile_copy_config
... )

After initialization, you can access the data from the data_base in the following manner:

>>> db['synapse_activation']
<synapse activation dataframe>
>>> db['cell_activation']
<cell activation dataframe>
>>> db['voltage_traces']
<voltage traces dataframe>
>>> db['spike_times']
<spike times dataframe>

Functions¶

`init`(db, simresult_path, core, synapse_activation, dendritic_voltage_traces, parameterfiles, spike_times, repartition, scheduler, rewrite_in_optimized_format, dendritic_spike_times, dendritic_spike_times_threshold, client, check_health, n_chunks, paramfile_copy_config, voltage_traces, burst_times, dumper)	Initialize a database with simulation data.
`add_dendritic_voltage_traces`(db, rewrite_in_optimized_format, dendritic_spike_times, repartition, dendritic_spike_times_threshold, scheduler, client)	Add dendritic voltage traces to the database.
`add_dendritic_spike_times`(db, dendritic_spike_times_threshold)	Add dendritic spike times to the database.
`optimize`(db, dumper, select, scheduler, repartition, categorized, client)	Rewrite existing data with a new dumper.
`load_initialized_cell_and_evokedNW_from_db`(db, sti, allPoints, reconnect_synapses)	Load and set up the cell and network from the database.

Attributes¶

`logger`	-
`DEFAULT_PARAMFILE_COPY_CONFIG`	-

Modules¶

`builders`	Pipelines for building database keys containing results from `simrun`.
`config`	Central configuration for simrun-initializing databases.
`data_parsing`	-
`file_handling`	-
`filepath_resolution`	-
`health`	-
`metadata_utils`	-
`param_file_parser`	-
`reoptimize`	Re-optimize a database with a new dumper
`utils`	-

Documentation unclear, incomplete, broken or wrong? Let us know