ml_grid.pipeline.parallel_execute

Exceptions

ModelExecutionError

Initialize self. See help(type(self)) for accurate signature.

Classes

ParallelModelExecutor

Initialize the parallel executor.

Functions

create_parallel_executor(→ ParallelModelExecutor)	Factory function to create a parallel executor.
execute_with_parallel_support(→ Tuple[List[List[Any]], ...)	Execute models with parallel processing support.
aggregate_parallel_results(→ Tuple[List[List[Any]], float])	Aggregate results from parallel model execution.

Module Contents

exception ml_grid.pipeline.parallel_execute.ModelExecutionError

Bases: Exception

Initialize self. See help(type(self)) for accurate signature.

class ml_grid.pipeline.parallel_execute.ParallelModelExecutor(n_jobs: int | None = None, verbose: int = 0)

Initialize the parallel executor.

Parameters:

• n_jobs – Number of parallel jobs. If None, uses all available CPU cores.

• verbose – Verbosity level (0=quiet, 1=minimal, 2=detailed)

logger

verbose = 0

gpu_model_locks: Dict[str, bool]

cpu_pool_lock

manager

results_queue

error_queue

global_result_lock

execute_models_parallel(arg_list: List[Tuple], shared_data: Dict[str, Any], timeout: float | None = None, max_retries: int = 2) → Tuple[List[Tuple], List[Dict]]

Execute all models in parallel.

Parameters:

• arg_list – List of argument tuples for each model

• shared_data – Dictionary containing shared data (X_train, y_train, etc.)

• timeout – Global timeout for the entire execution

• max_retries – Maximum retry attempts for failed models

Returns:

Tuple of (success_results, error_results)

execute_models_joblib_parallel(arg_list: List[Tuple], shared_data: Dict[str, Any], timeout: float | None = None, batch_size: int = 'auto') → Tuple[List[Tuple], List[Dict]]

Execute all models in parallel using joblib.

This approach is more efficient for many small tasks and reduces process creation overhead.

Parameters:

• arg_list – List of argument tuples for each model

• shared_data – Dictionary containing shared data

• timeout – Timeout per model execution

• batch_size – Batch size for joblib (auto, int, or float)

Returns:

Tuple of (success_results, error_results)

execute_models_multiprocessing(arg_list: List[Tuple], shared_data: Dict[str, Any], timeout: float | None = None) → Tuple[List[Tuple], List[Dict]]

Execute all models using multiprocessing Pool.

Best for heavy models with significant computation per model. Uses fork server or spawn start method for better isolation.

Parameters:

• arg_list – List of argument tuples for each model

• shared_data – Dictionary containing shared data

• timeout – Timeout per model execution

Returns:

Tuple of (success_results, error_results)

close()

Cleanup resources.

ml_grid.pipeline.parallel_execute.create_parallel_executor(n_jobs: int | None = None, method: str = 'auto', verbose: int = 0) → ParallelModelExecutor

Factory function to create a parallel executor.

Parameters:

• n_jobs – Number of parallel jobs

• method – Execution method (‘joblib’, ‘multiprocessing’, or ‘auto’)

• verbose – Verbosity level

Returns:

Configured ParallelModelExecutor instance

ml_grid.pipeline.parallel_execute.execute_with_parallel_support(run_instance: Any, arg_list: List[Tuple], n_jobs: int | None = None, verbose: int = 0) → Tuple[List[List[Any]], float]

Execute models with parallel processing support.

This function replaces the sequential execution in main.py:execute()

Parameters:

• run_instance – The run instance for accessing global parameters

• arg_list – List of argument tuples for each model

• n_jobs – Number of parallel jobs (None = auto-detect)

• verbose – Verbosity level

Returns:

Tuple of (model_error_list, highest_score)

ml_grid.pipeline.parallel_execute.aggregate_parallel_results(success_results: List[Tuple], error_results: List[Dict], arg_list: List[Tuple]) → Tuple[List[List[Any]], float]

Aggregate results from parallel model execution.

Parameters:

• success_results – List of successful execution results

• error_results – List of failed execution metadata

• arg_list – Original argument list

Returns:

Tuple of (model_error_list, highest_score)