sim_db for Python¶

Minimal Example using Python¶

A parameter file called params_mininal_python_example.txt is located in the sim_db/examples/ directory in the source code. The file contains the following:

name (string): minimal_python_example

run_command (string): python root/examples/minimal_example.py

param1 (string): "Minimal Python example is running."

param2 (int): 42

A python script called minimal_example.py and is found in the same directory:

import sim_db # 'sim_db/src/' have been include in the path.

# Open database and write some initial metadata to database.
sim_database = sim_db.SimDB()

# Read parameters from database.
param1 = sim_database.read("param1") # String
param2 = sim_database.read("param2") # Integer

# Print param1 just to show that the example is running.
print(param1)

# Write final metadata to database and close connection.
sim_database.close()

Add the those simulations parameters to the sim_db database and run the simulation from the sim_db/examples/ directory with:

$ sim_db add_and_run -f params_minimal_python_example.txt

Extensive Example using Python¶

A parameter file called params_extensive_python_example.txt is found in the sim_db/examples/ directory in the source code. This parameter file contains all the possible types available in addition to some comments:

This is a comment, as any line without a colon is a comment.
# Adding a hashtag to the start of a comment line, make the comment easier to recognize.

# The name parameter is highly recommended to include.
name (string): extensive_python_example

# It is also recommended to include a description to further explain the intention of 
# the simulation.
description (string): Extensive Python example to demonstrate most features in sim_db.

run_command (string): python root/examples/extensive_example.py

# A parameter is added for each of the avaiable types.
param1_extensive (int): 3
param2_extensive (float): -0.5e10
param3_extensive (string): "Extensive Python example is running."
param4_extensive (bool): True
param5_extensive (int array): [1, 2, 3]
param6_extensive (float array): [1.5, 2.5, 3.5]
param7_extensive (string array): ["a", "b", "c"]
param8_extensive (bool array): [True, False, True]

# Include parameters from another parameter file.
include_parameter_file: root/examples/extra_params_example.txt

# Change a parameter value from the included parameter file to demonstrate that
# it is the last parameter value that count for a given parameter name. 
extra_param1 (int): 9

The line in the parameter file starting with include_parameter_file: will be substituted with the contain of the specified extra_params_example.txt file, found in the same directory:

# Extra parameters included in the extensive examples.

extra_param1 (int): 7
extra_param2 (string): "Extra params added."
extra_param3 (bool): False

extensive_example.py is also found in the same directory:

import sim_db # 'sim_db/' have been included in the path.

# Open database and write some initial metadata to database.
sim_database = sim_db.SimDB()

# Read parameters from database.
param1 = sim_database.read("param1_extensive") # Integer
param2 = sim_database.read("param2_extensive") # Float
param3 = sim_database.read("param3_extensive") # String 
param4 = sim_database.read("param4_extensive") # Bool 
param5 = sim_database.read("param5_extensive") # List of integers 
param6 = sim_database.read("param6_extensive") # List of floats 
param7 = sim_database.read("param7_extensive") # List of strings 
param8 = sim_database.read("param8_extensive") # List of bools 

# Demonstrate that the simulation is running.
print(param3)

# Write to database.
sim_database.write("example_result_1", param1, type_of_value="int")
sim_database.write("example_result_2", param2, type_of_value="float")
sim_database.write("example_result_3", param3, type_of_value="string")
sim_database.write("example_result_4", param4, type_of_value="bool")
sim_database.write("example_result_5", param5, type_of_value="int array")
sim_database.write("example_result_6", param6, type_of_value="float array")
sim_database.write("example_result_7", param7, type_of_value="string array")
sim_database.write("example_result_8", param8, type_of_value="bool array")

# Make unique subdirectory for storing results and write its name to database.
results = np.array(param6)
name_results_dir = sim_database.unique_results_dir("root/examples/results")
np.savetxt(name_results_dir + "/results.txt", results)

# Check if column exists in database.
is_column_in_database = sim_database.column_exists("column_not_in_database")

# Check is column is empty and then set it to empty.
sim_database.is_empty("example_result_1")
sim_database.set_empty("example_result_1")

# Get the 'ID' of the connected simulation and the path to the root directory.
db_id = sim_database.get_id()
path_proj_root = sim_database.get_path_proj_root()

# Write final metadata to the database and close the connection.
sim_database.close()

# Add an empty simulation to database, open connection and write to it.
sim_database_2 = sim_db.add_empty_sim(False)
sim_database_2.write("param1_extensive", 7, type_of_value="int")

# Delete simulation from the database.
sim_database_2.delete_from_database()

# Close connection to the database.
sim_database_2.close()

Add the those simulations parameters to the sim_db database and run the simulation from the sim_db/examples/ directory with:

$ sdb add_and_run -f params_extensive_python_example.txt

Python API Referance¶

Read and write parameters, results and metadata to the ‘sim_db’ database.

class SimDB(store_metadata=True, db_id=None, rank=None, only_write_on_rank=0)¶

To interact with the sim_db database.

For an actuall simulation it should be initialised at the very start of the simulation (with ‘store_metadata’ set to True) and closed with close() at the very end of the simulation. This must be done to add the corrrect metadata.

For multithreading/multiprocessing each thread/process MUST have its own connection (instance of this class) and MUST provide it with its rank.

__init__(store_metadata=True, db_id=None, rank=None, only_write_on_rank=0)¶

Connect to the sim_db database.

Parameters

store_metadata (bool) – If False, no metadata is added to the database. Typically used when postprocessing (visualizing) data from a simulation.
db_id (int) – ID number of the simulation parameters in the sim_db database. If it is ‘None’, then it is read from the argument passed to the program after option ‘–id’.
rank (int) – Number identifing the calling process and/or thread. (Typically the MPI or OpenMP rank.) If provided, only the ‘rank’ matching ‘only_write_on_rank’ will write to the database to avoid too much concurrent writing to the database. Single process and threaded programs may ignore this, while multithreading/multiprocessing programs need to provide it.
only_write_on_rank (int) – Number identifing the only process/thread that will write to the database. Only used if ‘rank’ is provided.

read(column, check_type_is='')¶

Read parameter in ‘column’ from the database.

Return None if parameter is empty.

Parameters

column (str) – Name of the column the parameter is read from.
check_type_is – Throws ValueError if type does not match ‘check_type_is’.The valid types the strings ‘int’, ‘float’, ‘bool’, ‘string’ and ‘int/float/bool/string array’ or the types int, float, bool, str and list.

Raises

ColumnError – If column do not exists.
ValueError – If return type does not match ‘check_type_is’.
sqlite3.OperationalError – Waited more than 5 seconds to read from the database, because other threads/processes are busy writing to it. Way too much concurrent writing is done and it indicates an design error in the user program.

write(column, value, type_of_value='', only_if_empty=False)¶

Write value to ‘column’ in the database.

If ‘column’ does not exists, a new is added.

If value is None and type_of_value is not set, the entry under ‘column’ is set to empty.

For multithreaded and multiprocess programs only a single will process/thread write to the database to avoid too much concurrent writing to the database. This is as long as the ‘rank’ was passed to SimDB under initialisation.

Parameters

column (str) – Name of the column the parameter is read from.
value – New value of the specified entry in the database.
type_of_value (str or type) – Needed if column does note exists or if value is empty list. The valid types the strings ‘int’, ‘float’, ‘bool’, ‘string’ and ‘int/float/bool/string array’ or the types int, float, bool and str.
only_if_empty (bool) – If True, it will only write to the database if the simulation’s entry under ‘column’ is empty.

Raises

ValueError – If column exists, but type does not match, or empty list is passed without type_of_value given.

unique_results_dir(path_directory)¶

Get path to subdirectory in ‘path_directory’ unique to simulation.

The subdirectory will be named ‘date_time_name_id’ and is intended to store results in. If ‘results_dir’ in the database is empty, a new and unique directory is created and the path stored in ‘results_dir’. Otherwise the path in ‘results_dir’ is just returned.

Parameters: path_directory (str) – Path to directory of which to make a subdirectory. If ‘path_directory’ starts with ‘root/’, that part will be replaced by the full path of the root directory of the project.
Returns: Full path to new subdirectory.
Return type: str

column_exists(column)¶

Return True if column is a column in the database.

Raises: sqlite3.OperationalError – Waited more than 5 seconds to read from the database, because other threads/processes are busy writing to it. Way too much concurrent writing is done and it indicates an design error in the user program.

is_empty(column)¶

Return True if entry in the database under ‘column’ is empty.

Raises: sqlite3.OperationalError – Waited more than 5 seconds to read from the database, because other threads/processes are busy writing to it. Way too much concurrent writing is done and it indicates an design error in the user program.

set_empty(column)¶: Set entry under ‘column’ in the database to empty.

get_id()¶: Return ‘ID’ of the connected simulation.

get_path_proj_root()¶

Return the path to the root directory of the project.

The project’s root directory is assumed to be where the ‘.sim_db/’ directory is located.

update_sha1_executables(paths_executables)¶

Update the ‘sha1_executable’ column in the database.

Sets the entry to the sha1 of all the executables. The order will affect the value.

Parameters: paths_executables ([str]) – List of full paths to executables.
Raises: sqlite3.OperationalError – Waited more than 5 seconds to write to the database, because other threads/processes are busy writing to it. Way too much concurrent writing is done and it indicates an design error in the user program.

delete_from_database()¶

Delete simulation from database.

Raises: sqlite3.OperationalError – Waited more than 5 seconds to write to the database, because other threads/processes are busy writing to it. Way too much concurrent writing is done and it indicates an design error in the user program.

close()¶: Closes connection to sim_db database and add metadata.

add_empty_sim(store_metadata=False)¶

Add an empty entry into the database and SimDB connected to it.

Parameters: store_metadata (bool) – If False, no metadata is added to the database. Typically used when postprocessing (visualizing) data from a simulation.