Dataset

DatabaseHandler

class DatabaseHandler.DatabaseHandler(on_init_handler: Optional[Callable] = None, on_partitions_handler: Optional[Callable] = None)

add_batch(table_name: str, batch: Dict[str, List[Any]]) → None

Add a batch of data in a Database.

Parameters

• table_name – Name of the Table.

• batch – New lines of the Table.

add_data(table_name: str, data: Dict[str, Any]) → Union[int, List[int]]

Add a new line of data in a Database.

Parameters

• table_name – Name of the Table.

• data – New line of the Table.

create_fields(table_name: str, fields: Union[List[Tuple[str, Type]], Tuple[str, Type]]) → None

Create new Fields in a Table from one of the Databases.

Parameters

• table_name – Name of the Table.

• fields – Field or list of Fields names and types.

get_database_dir() → str

Get the database repository of the session.

get_exchange() → Database

Get the exchange Database.

get_fields(table_name: str) → List[str]

Get the list of Fields in a Table.

Parameters

table_name – Name of the Table.

get_line(table_name: str, line_id: Union[int, List[int]], fields: Optional[Union[str, List[str]]] = None) → Dict[str, Any]

Get a line of data from a Database.

Parameters

• table_name – Name of the Table.

• line_id – Index of the line to get.

• fields – Data fields to extract.

get_lines(table_name: str, lines_id: List[List[int]], fields: Optional[Union[str, List[str]]] = None) → Dict[str, Any]

Get lines of data from a Database.

Parameters

• table_name – Name of the Table.

• lines_id – Indices of the lines to get.

• fields – Data fields to extract.

get_partitions() → List[Database]

Get the storing Database partitions.

init(storing_partitions: List[Database], exchange_db: Database) → None

Initialize the list of the partitions.

Parameters

• storing_partitions – List of the storing Database partitions.

• exchange_db – Exchange Database.

init_remote(storing_partitions: List[List[str]], exchange_db: List[str]) → None

Initialize the list of partitions in remote DatabaseHandlers.

Parameters

• storing_partitions – List of paths to the storing Database partitions.

• exchange_db – Path to the exchange Database.

load() → None

Load the Database partitions stored by the component.

update(table_name: str, data: Dict[str, Any], line_id: Union[int, List[int]]) → None

Update a line in a Database.

Parameters

• table_name – Name of the Table.

• data – Updated line of the Table.

• line_id – Index of the line to edit.

update_list_partitions(partition: Database) → None

Add a new storing partition to the list.

Parameters

partition – New storing partition to add.

update_list_partitions_remote(partition: List[str]) → None

Add a new storing partition to the list in remote DatabaseHandler.

Parameters

partition – Path to the new storing partition.

BaseDatabaseConfig

class BaseDatabaseConfig.BaseDatabaseConfig(existing_dir: Optional[str] = None, mode: Optional[str] = None, max_file_size: Optional[float] = None, shuffle: bool = False, normalize: bool = False, recompute_normalization: bool = False)