# Run training script in DDP using GLOO

**URL:** <https://discuss.huggingface.co/t/run-training-script-in-ddp-using-gloo/15406>\
**Category:** Intermediate\
**Created:** [March 6, 2022, 1:27pm UTC](https://discuss.huggingface.co/t/run-training-script-in-ddp-using-gloo/15406 "2022-03-06T13:27:42Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![kiroul](https://avatars.discourse-cdn.com/v4/letter/k/bb73d2/32.png) [@kiroul](https://discuss.huggingface.co/u/kiroul)\
**Post date:** [March 6, 2022, 1:27pm UTC](https://discuss.huggingface.co/t/run-training-script-in-ddp-using-gloo/15406/1 "2022-03-06T13:27:42Z")

</div>

Hello,

I am desperately trying to run my training script in DDP using GLOO (on windows). I managed to do it on a single machine having 2 GPUs running the following command:

launch.py  
–nproc\_per\_node=2  
my\_script.py

As NLCC is not available on windows I had to tweak the ‘setup\_devices’ method of ‘training\_args.py’ and write:

torch.distributed.init\_process\_group(backend=“nccl”) → torch.distributed.init\_process\_group(backend=“gloo”)

along with the ‘distributed\_concat’ in ‘trainer\_pt\_utils.py’:

dist.all\_gather(output\_tensors, tensor) → dist.all\_gather(output\_tensors, tensor if len(tensor.shape) \> 0 else tensor[None])

With those 2 changes I can run my script using DPP and GLOO backend on my 2GPU windows machine.  
Following that I am now trying to run the script on 2 windows machines with 2 GPU each. From what I have understand I need to run the same script on both machine. Hence I used the following command on the 2 machines:

…torch\distributed\launch.py  
–nproc\_per\_node=2  
–nnodes=2  
–node\_rank=0  
–master\_addr=“10.73.8.68”  
–master\_port=1234  
my\_script.py

and …torch\distributed\launch.py

–nproc\_per\_node=2  
–nnodes=2  
–node\_rank=1  
–master\_addr=“10.73.8.68”  
–master\_port=1234  
my\_script.py

When doing so script start to be executed on both machine (loading the dataset on both sides) but then nothing happens and the scripts don t seem to go beyond the instruction:

torch.distributed.init\_process\_group(backend=“gloo”)

in training\_args.py and just never ends…  
Do you have any idea if what I am doing makes sense and how I should do to run my script in DDP on 2 windows machines with 2 GPU (using GLOO backend) ?

Thank you.

---

<div class="post-metadata">

**Author:** ![brando](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/brando/32/30114_2.png) [@brando](https://discuss.huggingface.co/u/brando)\
**Post date:** [August 17, 2022, 3:02pm UTC](https://discuss.huggingface.co/t/run-training-script-in-ddp-using-gloo/15406/2 "2022-08-17T15:02:09Z")

</div>

did you manage to solve your issue? Mind sharing? 🙂