Jug: A Task-Based Parallelization Framework

Jug allows you to write code that is broken up into tasks and run different tasks on different processors.

It uses the filesystem to communicate between processes and works correctly over NFS, so you can coordinate processes on different machines.

Jug is a pure Python implementation and should work on any platform.

Python versions 3.7 and above are supported.

Website: http://luispedro.org/software/jug

Documentation: https://jug.readthedocs.org/

Video: On vimeo or showmedo

Mailing List: https://groups.google.com/group/jug-users

Testimonials

"I've been using jug with great success to distribute the running of a reasonably large set of parameter combinations" - Andreas Longva

Install

You can install Jug with pip:

pip install Jug

or use, if you are using conda, you can install jug from conda-forge using the following commands:

conda config --add channels conda-forge
conda install jug

Citation

If you use Jug to generate results for a scientific publication, please cite

Coelho, L.P., (2017). Jug: Software for Parallel Reproducible Computation in Python. Journal of Open Research Software. 5(1), p.30.

https://doi.org/10.5334/jors.161

Short Example

Here is a one minute example. Save the following to a file called primes.py (if you have installed jug, you can obtain a slightly longer version of this example by running jug demo on the command line):

from jug import TaskGenerator
from time import sleep

@TaskGenerator
def is_prime(n):
    sleep(1.)
    for j in range(2,n-1):
        if (n % j) == 0:
            return False
    return True

primes100 = [is_prime(n) for n in range(2,101)]

This is a brute-force way to find all the prime numbers up to 100. Of course, this is only for didactical purposes, normally you would use a better method. Similarly, the sleep function is so that it does not run too fast. Still, it illustrates the basic functionality of Jug for embarassingly parallel problems.

Type jug status primes.py to get:

Task name                  Waiting       Ready    Finished     Running
----------------------------------------------------------------------
primes.is_prime                  0          99           0           0
......................................................................
Total:                           0          99           0           0

This tells you that you have 99 tasks called primes.is_prime ready to run. So run jug execute primes.py &. You can even run multiple instances in the background (if you have multiple cores, for example). After starting 4 instances and waiting a few seconds, you can check the status again (with jug status primes.py):

Task name                  Waiting       Ready    Finished     Running
----------------------------------------------------------------------
primes.is_prime                  0          63          32           4
......................................................................
Total:                           0          63          32           4

Now you have 32 tasks finished, 4 running, and 63 still ready. Eventually, they will all finish and you can inspect the results with jug shell primes.py. This will give you an ipython shell. The primes100 variable is available, but it is an ugly list of jug.Task objects. To get the actual value, you call the value function:

In [1]: primes100 = value(primes100)

In [2]: primes100[:10]
Out[2]: [True, True, False, True, False, True, False, False, False, True]

What's New

Unreleased

User-visible improvements

• Support project-local configuration files (.jugrc or jugrc). Jug now walks up the directory tree from the current working directory (up to the git project root) looking for local configuration files. See :doc:`configuration` for details.

Version 2.4.0

Released 8 May 2025

User-visible improvements

• Adds support for lambda functions in Tasklets
• Adds NoHash class to disable hashing for some arguments. This is in jug.unsafe as it can be used to "fool" Jug, but it can be useful when there are nuisance arguments that are not relevant for the task (e.g., number of threads)
• jug.file_store: create files with better permissions

Internal improvements

• Convert to pyproject.toml for building

Bugfixes

• Better error detection for permission problems
• Bugfix when using local imports and jug pack

Drops support for versions of Python older than 3.7. Technically, it should still work, but they are too old to test in Github CI, so we will not support them.

Version 2.3.1

Released 5 November 2023

• Update for Python 3.12

Version 2.3.0

Released 25 June 2023

• jug shell: Add get_filtered_tasks()
• jug: Fix jug --version (which had been broken in the refactoring to use subcommands)
• jug shell: Fix message in jug shell when there are no dependencies (it would repeatedly print the message stating this will only be run once)
• jug pack: Make it much faster to invalidate elements
• file_store: ensure that the temporary directory exists
• Drops support for Python 3.4

For older version see ChangeLog file or the full history.