Add _rank_not_in_group to idist

[sadra-barikbin]

Hi there!

To add _rank_not_in_group to idist.utils.

[vfdev-5]

Thanks for the PR @sadra-barikbin !

[vfdev-5]

Let's remove this as it was for typing only, right?

[sadra-barikbin]

@vfdev-5 , is XLA's _do_new_group correct?

def _do_new_group(self, ranks: List[int], **kwargs: Any) -> Any:
            return [ranks] # Shouldn't it return `list(ranks)` ?

[vfdev-5]

@vfdev-5 , is XLA's _do_new_group correct?

def _do_new_group(self, ranks: List[int], **kwargs: Any) -> Any:
            return [ranks] # Shouldn't it return `list(ranks)` ?

On xla group is a list of list of ranks, for example: https://pytorch.org/xla/release/r2.6/learn/api-guide.html#torch_xla.core.xla_model.all_gather

[vfdev-5]

Failing test:

  >       gloo_hvd_executor(_test_distrib_all_gather_group, (device,), np=np, do_init=True)

   [0]<stderr>:  File "/home/runner/work/ignite/ignite/tests/ignite/distributed/utils/__init__.py", line 228, in _test_distrib_all_gather_group
  [0]<stderr>:    res = idist.all_gather(t, group=group)
  [0]<stderr>:          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [0]<stderr>:  File "/home/runner/work/ignite/ignite/ignite/distributed/utils.py", line 435, in all_gather
  [0]<stderr>:    return _model.all_gather(tensor, group=group)
  [0]<stderr>:           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [0]<stderr>:  File "/home/runner/work/ignite/ignite/ignite/distributed/comp_models/base.py", line 224, in all_gather
  [0]<stderr>:    return self._collective_op(tensor, self._do_all_gather, group=group)
  [1]<stderr>:User function raise error: horovod_torch_allgather_async_torch_LongTensor(): incompatible function arguments. The following argument types are supported:
  [1]<stderr>:    1. (arg0: torch.Tensor, arg1: torch.Tensor, arg2: str, arg3: int) -> int

[vfdev-5]

We have this method coded incorrectly:

def _do_new_group(self, ranks: List[int], **kwargs: Any) -> Any:
    return hvd.ProcessSet(ranks)

it should be

def _do_new_group(self, ranks: List[int], **kwargs: Any) -> Any:
    return hvd.add_process_set(ranks)

and we need to set HOROVOD_DYNAMIC_PROCESS_SETS=1 env var.

I'll commit some updates to this PR.

[vfdev-5]

Failures seems not to be related to the PR's feature.
Let's fix them later.