wait_for_pool_results#

caf.toolkit.concurrency.multiprocessing_wrapper.wait_for_pool_results(results, terminate_process_event, result_timeout, pbar_kwargs=None)[source]#

Wait for and grab results from a multiprocessing Pool.

Aims to return all results once processes have complete. Will throw an error if terminate_process_event is set, or result_timeout is reached.

Parameters:

  • results (list[ApplyResult[_T]]) – A list of multiprocessing.pool.AsyncResult. The results to wait for.

  • terminate_process_event (Event) – A multiprocessing.Event. This event should get set if an error occurs and the multiprocessing.Pool (which is generating the results) needs to terminate.

  • result_timeout (int) – How many seconds to wait for all results before throwing an error.

  • pbar_kwargs (Optional[dict[str, Any]]) – A dictionary of keyword arguments to pass into a tqdm progress bar. Will be used as tqdm.tqdm(**pbar_kwargs)

Returns:

A list of the return values collected from results. May not be in the same order as received results.

Return type:

results_out

Raises:

  • multiprocessing.ProcessError: – Will be raised in the following cases: - If terminate_process_event - If an error is discovered in one of the processes - If one or more of the results are lost when retrieving results

  • TimeoutError: – If the seconds spent waiting for results is greater than result_timeout