Usage¶
To use Parallel TQDM in a project:
import pqdm
There very basic usage is running pqdm
on an Iterable
whose elements are
directly supported by the Callable
passed to pqdm
:
from pqdm.processes import pqdm
# If you want threads instead:
# from pqdm.threads import pqdm
args = [1, 2, 3, 4, 5]
# args = range(1,6) would also work
def square(a):
return a*a
result = pqdm(args, square, n_jobs=2)
Everytime you run pqdm
with more than one job (i.e. n_jobs > 1
) you will
need to make a decision about the backend used, the standard options from Python’s
concurrent.futures
library are:
- threads: share memory with the main process, subject to GIL, low benefit on CPU heavy tasks, best for IO tasks or tasks involving external systems,
- processes: best for CPU heavy tasks, GIL-free, have an IO overhead due to lack of shared memory and need to dump/pickle data
With both backends pqdm
supports bounded variants, which are semaphore guarded. For
more information on processes vs threads see the concurent futures documentation or Brendant
Fortuner’s medium post.
Different ways to pass arguments to function¶
By default pqdm
assumes your function can handle the element taken from Iterable
,
you may however want to have other kinds of situations:
- an
Iterable[Dict[str, Any]]
which you would like to pass as named arguments to a function, - or
Iterable[Union[List, Tuple]]
which you would like to pass as positional arguments to a function
The library supports both variants, here’s how a named argument should be passed:
from pqdm.processes import pqdm # If you want threads instead: # from pqdm.threads import pqdm args = [ {'a': 1, 'b': 2}, {'a': 2, 'b': 3}, {'a': 3, 'b': 4}, {'a': 4, 'b': 5} ] def multiply(a, b): return a*b result = pqdm(args, multiply, n_jobs=2, argument_type='kwargs') # result is [2, 6, 12, 20]
and for the positional arguments:
from pqdm.processes import pqdm # If you want threads instead: # from pqdm.threads import pqdm args = [[1, 2], [2, 3], [3, 4], [4, 5]], def multiply(a, b): return a*b result = pqdm(args, multiply, n_jobs=2, argument_type='args') # result is [2, 6, 12, 20]
Passing arguments to tqdm¶
You may want to change tqdm
output, for this reason any option not handle by the
Executor
class from concurrent.futures
is passed to tqdm
.
from pqdm.processes import pqdm # If you want threads instead: # from pqdm.threads import pqdm args = [1, 2, 3, 4, 5] def square(a): return a*a result = pqdm(args, square, n_jobs=2, desc='Squaring elements', unit='el')
Changing the tqdm_class¶
In some use cases you might want to use a custom tqdm class. By default the tqdm.auto
class is used, which should select either a html-based tqdm for notebooks or a command
line tqdm.
However other tqdm classes exists, let’s for example assume you have a discord channel and want to use the tqdm.contrib.discord class, just use the following:
from pqdm.processes import pqdm from tqdm.contrib.discord import tqdm as tqdm_discord # If you want threads instead: # from pqdm.threads import pqdm args = [1, 2, 3, 4, 5] def square(a): return a*a result = pqdm( args, square, n_jobs=2, tqdm_class=tqdm_discord, # tqdm_discord kwargs token='{token}', channel_id='{channel_id}', # base tqdm kwargs desc='Squaring elements', unit='el' )