Do i still need to set kv_store in using mxnet? why?

[ZHAIXINGZHAIYUE]

Do i still need to set kv_store in using mxnet? why?

[ymjiang]

You don't need to. MXNet-BytePS's implementation bypasses kvstore.

[ZHAIXINGZHAIYUE]

您好，如果我在使用 mxnet native 进行分布式训练的时候，也单独设置scheduler节点， server节点，worker节点，让它们运行在单独的服务器上。这样的话，使用byteps mxnet 与使用 mxnet native 进行训练，效率上会有差异吗？如果有的话，差异主要来自哪里？谢谢。

[ymjiang]

There will be performance difference even if using the same setup as you said. We did many performance optimizations on BytePS. For example, compared to mxnet native, BytePS-mxnet eliminates some extra copy. BytePS also supports RDMA, which is obviously faster than mxnet-native TCP. We will have a technical report talking about these optimizations in the future.

[ZHAIXINGZHAIYUE]

thank you

[bobzhuyb]

Below are some numbers. The following experiments are performed on a public cloud with 20 Gbps networks. Each machine has 8 Tesla V100 16GB GPUs (with NVLink-enabled). The batch size is 32 for each GPU, and we use fp32 training. We calculate the "total images per second" as the metric.

[ZHAIXINGZHAIYUE]

@bobzhuyb the number of server is same? between mxnet-native and mxnet-byteps.

[bobzhuyb]

Yes. You can try them yourself. The original ps-lite implementation is pretty poor -- it is slower than Horovod, let alone BytePS.

[ZHAIXINGZHAIYUE]

你好，我这里还有一个问题。在使用原版mxnet 进行分布式训练的时候，不时的会遇到Check failed: (my_node_.port) != (-1) bind failed, 在byteps中，这个问题还存在吗？

[ymjiang]

@ZHAIXINGZHAIYUE I believe you won't have that problem if you configure byteps correctly. We never meet this when using byteps.