Manager

DataManager

class DataManager.DataManager(pipeline: Any, database_config: Optional[BaseDatabaseConfig] = None, environment_config: Optional[BaseEnvironmentConfig] = None, session: str = 'sessions/default', new_session: bool = True, produce_data: bool = True, batch_size: int = 1)

close() → None

Launch the closing procedure of the DataManager.

connect_handler(handler: DatabaseHandler) → None

Add a new DatabaseHandler to the list of handlers of the DatabaseManager.

Parameters

handler – New handler to register.

get_data(epoch: int = 0, animate: bool = True, load_samples: bool = True) → None

Fetch data from the EnvironmentManager or the DatabaseManager according to the context.

Parameters

• epoch – Current epoch number.

• animate – Allow EnvironmentManager to trigger a step itself in order to generate a new sample.

• load_samples – If True, trigger a sample loading from the Database.

get_prediction(instance_id: int) → None

Get a Network prediction for the specified Environment instance.

load_sample() → List[int]

Load a sample from the Database.

Returns

Index of the loaded line.

property nb_environment: Optional[int]

Get the number of Environments managed by the EnvironmentManager.

property normalization: Dict[str, List[float]]

Get the normalization coefficients computed by the DatabaseManager.

DatabaseManager

class DatabaseManager.DatabaseManager(database_config: Optional[BaseDatabaseConfig] = None, data_manager: Optional[Any] = None, pipeline: str = '', session: str = 'sessions/default', new_session: bool = True, produce_data: bool = True)

add_data(data_lines: Optional[List[int]] = None) → None

Manage new lines adding in the Database.

Parameters

data_lines – Indices of the newly added lines.

change_mode(mode: str) → None

Change the current Database mode.

Parameters

mode – Name of the Database mode.

close()

Launch the closing procedure of the DatabaseManager.

compute_normalization() → Dict[str, List[float]]

Compute the mean and the standard deviation of all the training samples for each data field.

connect_handler(handler: DatabaseHandler) → None

Add and init a new DatabaseHandler to the list.

Parameters

handler – New DatabaseHandler.

create_partition() → None

Create a new partition of the Database.

get_data(batch_size: int) → List[List[int]]

Select a batch of indices to read in the Database.

Parameters

batch_size – Number of sample in a single batch.

get_data_shapes() → Dict[str, List[int]]

Get the shape of data Fields.

get_database_architecture() → Dict[str, List[str]]

Get the Tables and Fields structure of the Database.

get_nb_samples() → None

Get the number of sample in each partition.

get_partition_names() → List[List[str]]

Get the list of partition paths of the Database for the current mode.

get_partition_objects() → List[Database]

Get the list of partitions of the Database for the current mode.

index_samples() → None

Create a new indexing list of samples. Samples are identified by [partition_id, line_id].

load_directory(rename_partitions: bool = False) → None

Get the Database information from the json file (partitions, samples, etc). Load all the partitions or create one if necessary.

Parameters

rename_partitions – If True, the existing partitions should be renamed to match the session name.

static load_partitions_fields(partition: Database, fields: List[str]) → Dict[str, ndarray]

Load all the samples from a Field of a Table in the Database.

Parameters

• partition – Database partition to load.

• fields – Data Fields to get.

property normalization: Optional[Dict[str, List[float]]]

Get the normalization coefficients.

remove_empty_partitions()

Remove every empty partitions of the Database.

search_partitions_info() → None

Get the information about the Database manually if the json file is not found.

update_json() → None

Update the JSON info file with the current Database information.

update_normalization(data_lines: List[int]) → Dict[str, List[float]]

Update the mean and the standard deviation of all the training samples with newly added samples for each data field.

Parameters

data_lines – Indices of the newly added lines.

EnvironmentManager

class EnvironmentManager.EnvironmentManager(environment_config: BaseEnvironmentConfig, data_manager: Optional[Any] = None, pipeline: str = '', session: str = 'sessions/default', produce_data: bool = True, batch_size: int = 1)

close() → None

Launch the closing procedure of the EnvironmentManager.

update_visualizer(instance: int) → None

Update the Visualizer.

Parameters

instance – Index of the Environment render to update.

NetworkManager

class NetworkManager.NetworkManager(network_config: BaseNetworkConfig, pipeline: str = '', session: str = 'sessions/default', new_session: bool = True)

close() → None

Launch the closing procedure of the NetworkManager.

compute_online_prediction(instance_id: int, normalization: Optional[Dict[str, List[float]]] = None) → None

Make a prediction with the data passed as argument.

Parameters

• instance_id – Index of the Environment instance to provide a prediction.

• normalization – Normalization coefficients.

compute_prediction_and_loss(optimize: bool, data_lines: List[List[int]], normalization: Optional[Dict[str, List[float]]] = None) → Dict[str, float]

Make a prediction with the data passed as argument, optimize or not the network

Parameters

• optimize – If true, run a backward propagation.

• data_lines – Batch of indices of samples in the Database.

• normalization – Normalization coefficients.

Returns

The prediction and the associated loss value

get_database_handler() → DatabaseHandler

Get the DatabaseHandler of the NetworkManager.

link_clients(nb_clients: Optional[int] = None) → None

Update the data Exchange Database with a new line for each TcpIpClient.

Parameters

nb_clients – Number of Clients to connect.

load_network(which_network: int = -1) → None

Load an existing set of parameters of the Network.

Parameters

which_network – If several sets of parameters were saved, specify which one to load.

classmethod normalize_data(data: ndarray, normalization: List[float], reverse: bool = False) → ndarray

Apply or unapply normalization following current standard score.

Parameters

• data – Data to normalize.

• normalization – Normalization coefficients.

• reverse – If True, apply normalization; if False, unapply normalization.

Returns

Data with applied or misapplied normalization.

save_network(last_save: bool = False) → None

Save the current set of parameters of the Network.

Parameters

last_save – If True, the Network training is done then give a special denomination.

StatsManager

class StatsManager.StatsManager(session: str, keep_losses: bool = False)

add_3D_mesh(tag: str, vertices: ndarray, colors: Optional[ndarray] = None, faces: Optional[ndarray] = None, b_n_3: bool = False, config_dict: Optional[Dict[Any, Any]] = None) → None

Add 3D Mesh cloud to tensorboard framework.

Parameters

• tag (str) – Data identifier

• vertices (ndarray) – List of the 3D coordinates of vertices.

• colors (Optional[ndarray]) – Colors for each vertex

• faces (Optional[ndarray]) – Indices of vertices within each triangle.

• b_n_3 (bool) – Data is in the format [batch_size, number_of_nodes, 3]

• config_dict (Optional[Dict[Any, Any]]) – Dictionary with ThreeJS classes names and configuration.

add_3D_point_cloud(tag: str, vertices: ndarray, colors: Optional[ndarray] = None, b_n_3: bool = False, config_dict: Optional[Dict[Any, Any]] = None) → None

Add 3D point cloud to tensorboard framework

Parameters

• tag (str) – Data identifier

• vertices (DataContainer) – List of the 3D coordinates of vertices.

• colors (DataContainer) – Colors for each vertex

• b_n_3 (bool) – Data is in the format [batch_size, number_of_nodes, 3]

• config_dict (Optional[Dict[Any, Any]]) – Dictionary with ThreeJS classes names and configuration.

add_custom_scalar(tag: str, value: float, count: int) → None

Add a custom scalar to tensorboard framework.

Parameters

• tag (str) – Graph name

• value (float) – Value to store

• count (int) – ID of the value

add_custom_scalar_full(tag: str, value: float, count: int) → None

Add a custom scalar to tensorboard framework. Also compute mean and variance.

Parameters

• tag (str) – Graph name

• value (float) – Value to store

• count (int) – ID of the value

add_network_weight_grad(network: Any, count: int, save_weights: bool = False, save_gradients: bool = True) → None

Add network weights and gradiant if specified to tensorboard framework.

Parameters

• network (BaseNetwork) – Network you want to display

• count (int) – ID of the sample

• save_weights (bool) – If True will save weights to tensorboard

• save_gradients (bool) – If True will save gradient to tensorboard

add_test_loss(value: float, count: int) → None

Add test loss to tensorboard framework. Also compute mean and variance.

Parameters

• value (float) – Value to store

• count (int) – ID of the value

add_test_loss_OOB(value: float, count: int) → None

Add out of bound test loss to tensorboard framework. Also compute mean and variance.

Parameters

• value (float) – Value to store

• count (int) – ID of the value

add_train_batch_loss(value: float, count: int) → None

Add batch loss to tensorboard framework. Also compute mean and variance.

Parameters

• value (float) – Value to store

• count (int) – ID of the value

add_train_epoch_loss(value: float, count: int) → None

Add epoch loss to tensorboard framework. Also compute mean and variance.

Parameters

• value (float) – Value to store

• count (int) – ID of the value

add_train_test_batch_loss(train_value: float, test_value: float, count: int) → None

Add train and test batch loss to tensorboard framework.

Parameters

• train_value (float) – Value of the training batch

• test_value (float) – Value of the testing batch

• count (int) – ID of the value

add_values_multi_plot(graph_name: str, tags: Iterable, values: Iterable, counts: Iterable) → None

Plot multiples value on the same graph

Parameters

• graph_name (str) – Name of the graph

• tags (Iterable) – Iterable containing the names of the values

• values (Iterable) – Iterable containing the value

• counts (Iterable) – ID of the plots

close() → None

Launch the closing procedure of the StatsManager.

update_mean_get_var(index: int, value: float, count: int) → Optional[ndarray]

Update mean and return the variance of the selected value

Parameters

• value (float) – Value to add in the computation of the mean

• index (int) – Target that is updated by the value

• count (int) – ID of the value

Returns

The mean and var values over the last 50 elements