- mantidimaging.core.parallel.shared.create_partial(func, fwd_function, **kwargs)[source]¶
Create a partial using functools.partial, to forward the kwargs to the parallel execution of imap.
If you seem to be getting nans, check if the correct fwd_function is set!
func – Function that will be executed
fwd_function – The function will be forwarded through function.
kwargs – kwargs to forward to the function func that will be executed
The decorated forwarded function, ready for further execution
- mantidimaging.core.parallel.shared.execute(partial_func: functools.partial, num_operations: int, progress=None, msg: str = '', cores=None) → None[source]¶
Executes a function in parallel with shared memory between the processes.
The array must have been created using parallel.utility.create_shared_array(shape, dtype).
If the input array IS NOT a shared array, the data will NOT BE CHANGED!
The reason for that is that the processes don’t work on the data, but on a copy.
When they process it and return the result, THE RESULT IS NOT ASSIGNED BACK TO REPLACE THE ORIGINAL, it is discarded.
imap_unordered gives the images back in random order
map and map_async do not improve speed performance
imap seems to be the best choice
Using _ in the for _ enumerate is slightly faster, because the tuple from enumerate isn’t unpacked, and thus some time is saved.
From performance tests, the chunksize doesn’t seem to make much of a difference, but having larger chunks usually led to slower performance:
Shape: (50,512,512) 1 chunk 3.06s 2 chunks 3.05s 3 chunks 3.07s 4 chunks 3.06s 5 chunks 3.16s 6 chunks 3.06s 7 chunks 3.058s 8 chunks 3.25s 9 chunks 3.45s
partial_func – A function constructed using create_partial
num_operations – The expected number of operations - should match the number of images being processed Also used to set the number of progress steps
cores – number of cores that the processing will use
progress – Progress instance to use for progress reporting (optional)
msg – Message to be shown on the progress bar