ml_grid.pipeline.parallel_execute
=================================

.. py:module:: ml_grid.pipeline.parallel_execute


Exceptions
----------

.. autoapisummary::

   ml_grid.pipeline.parallel_execute.ModelExecutionError


Classes
-------

.. autoapisummary::

   ml_grid.pipeline.parallel_execute.ParallelModelExecutor


Functions
---------

.. autoapisummary::

   ml_grid.pipeline.parallel_execute.create_parallel_executor
   ml_grid.pipeline.parallel_execute.execute_with_parallel_support
   ml_grid.pipeline.parallel_execute.aggregate_parallel_results


Module Contents
---------------

.. py:exception:: ModelExecutionError

   Bases: :py:obj:`Exception`


   Initialize self.  See help(type(self)) for accurate signature.


.. py:class:: ParallelModelExecutor(n_jobs: Optional[int] = None, verbose: int = 0)

   Initialize the parallel executor.

   :param n_jobs: Number of parallel jobs. If None, uses all available CPU cores.
   :param verbose: Verbosity level (0=quiet, 1=minimal, 2=detailed)


   .. py:attribute:: logger


   .. py:attribute:: verbose
      :value: 0



   .. py:attribute:: gpu_model_locks
      :type:  Dict[str, bool]


   .. py:attribute:: cpu_pool_lock


   .. py:attribute:: manager


   .. py:attribute:: results_queue


   .. py:attribute:: error_queue


   .. py:attribute:: global_result_lock


   .. py:method:: execute_models_parallel(arg_list: List[Tuple], shared_data: Dict[str, Any], timeout: Optional[float] = None, max_retries: int = 2) -> Tuple[List[Tuple], List[Dict]]

      Execute all models in parallel.

      :param arg_list: List of argument tuples for each model
      :param shared_data: Dictionary containing shared data (X_train, y_train, etc.)
      :param timeout: Global timeout for the entire execution
      :param max_retries: Maximum retry attempts for failed models

      :returns: Tuple of (success_results, error_results)



   .. py:method:: execute_models_joblib_parallel(arg_list: List[Tuple], shared_data: Dict[str, Any], timeout: Optional[float] = None, batch_size: int = 'auto') -> Tuple[List[Tuple], List[Dict]]

      Execute all models in parallel using joblib.

      This approach is more efficient for many small tasks and reduces
      process creation overhead.

      :param arg_list: List of argument tuples for each model
      :param shared_data: Dictionary containing shared data
      :param timeout: Timeout per model execution
      :param batch_size: Batch size for joblib (auto, int, or float)

      :returns: Tuple of (success_results, error_results)



   .. py:method:: execute_models_multiprocessing(arg_list: List[Tuple], shared_data: Dict[str, Any], timeout: Optional[float] = None) -> Tuple[List[Tuple], List[Dict]]

      Execute all models using multiprocessing Pool.

      Best for heavy models with significant computation per model.
      Uses fork server or spawn start method for better isolation.

      :param arg_list: List of argument tuples for each model
      :param shared_data: Dictionary containing shared data
      :param timeout: Timeout per model execution

      :returns: Tuple of (success_results, error_results)



   .. py:method:: close()

      Cleanup resources.



.. py:function:: create_parallel_executor(n_jobs: Optional[int] = None, method: str = 'auto', verbose: int = 0) -> ParallelModelExecutor

   Factory function to create a parallel executor.

   :param n_jobs: Number of parallel jobs
   :param method: Execution method ('joblib', 'multiprocessing', or 'auto')
   :param verbose: Verbosity level

   :returns: Configured ParallelModelExecutor instance


.. py:function:: execute_with_parallel_support(run_instance: Any, arg_list: List[Tuple], n_jobs: Optional[int] = None, verbose: int = 0) -> Tuple[List[List[Any]], float]

   Execute models with parallel processing support.

   This function replaces the sequential execution in main.py:execute()

   :param run_instance: The run instance for accessing global parameters
   :param arg_list: List of argument tuples for each model
   :param n_jobs: Number of parallel jobs (None = auto-detect)
   :param verbose: Verbosity level

   :returns: Tuple of (model_error_list, highest_score)


.. py:function:: aggregate_parallel_results(success_results: List[Tuple], error_results: List[Dict], arg_list: List[Tuple]) -> Tuple[List[List[Any]], float]

   Aggregate results from parallel model execution.

   :param success_results: List of successful execution results
   :param error_results: List of failed execution metadata
   :param arg_list: Original argument list

   :returns: Tuple of (model_error_list, highest_score)