Parallelize agents

Parallelize sweep agents on multi-core or multi-GPU machines.

Parallelize your sweep agents on a multi-core or multi-GPU machine to run multiple sweep runs at the same time. Parallelization reduces the time needed to explore your hyperparameter space. This page shows you how to launch parallel agents on multi-CPU and multi-GPU machines so you can use all available compute on a single host.

Before you start, you must initialize your sweep. For more information, see Initialize sweeps.

Parallelize on a multi-CPU machine

The following tabs describe how to run multiple sweep agents in parallel on the same machine. Parallel agents are most useful on a machine with multiple CPU cores, where each agent can run on its own core. Choose the tab for the CLI or a Jupyter Notebook based on your workflow.

CLI

Use the wandb agent command to parallelize your sweep agent across multiple CPUs with the terminal. Provide the sweep ID that W&B returned when you initialized the sweep.

1. Open more than one terminal window on your local machine.

2. Copy and paste the following code snippet and replace [SWEEP-ID] with your sweep ID:

   wandb agent [SWEEP-ID]
   

Jupyter Notebook

Use the W&B Python SDK library to parallelize your sweep agent across multiple CPUs within Jupyter Notebooks. Ensure you have the sweep ID that W&B returned when you initialized the sweep. Also provide the name of the function the sweep executes for the function parameter:

1. Open more than one Jupyter Notebook.

2. Copy and paste the sweep ID on multiple Jupyter Notebooks to parallelize a sweep. For example, if you have the sweep ID stored in a variable called sweep_id and the name of the function is function_name, paste the following code snippet on multiple Jupyter Notebooks to parallelize your sweep:

   wandb.agent(sweep_id=sweep_id, function=function_name)
   

Parallelize on a multi-GPU machine

The following procedure describes how to run sweep agents in parallel across multiple GPUs on the same machine, using a terminal with CUDA Toolkit. This procedure requires a machine with more than one GPU. Use CUDA_VISIBLE_DEVICES to assign each wandb agent to a different GPU so the agents run in parallel without competing for the same device.

For example, suppose you have two NVIDIA GPUs on your local machine. Replace [SWEEP-ID] in each command with the sweep ID that W&B returns when you initialize a sweep:

1. In one terminal, set CUDA_VISIBLE_DEVICES to 0 and start an agent:

   CUDA_VISIBLE_DEVICES=0 wandb agent [SWEEP-ID]
   

2. In a second terminal, set CUDA_VISIBLE_DEVICES to 1 and start another agent with the same sweep ID:

   CUDA_VISIBLE_DEVICES=1 wandb agent [SWEEP-ID]