we run ScaleIO (currently 1.32) on XenServer 6.5 for our Hyperconverged "Cloud" product.
As we were looking for hardware to expand the current setup, we realized that Infiniband hardware can be had for (much) less money than even 10G ethernet (40G ethernet not even a consideration). A posting on a "home-server" forum seemed to underline the benefits of going this route, especially if you run an all-flash setup (which we hope to be able to do).
Has anyone run such a setup?
Obviously, the HPC-community has more knowledge and experience with IB (Infiniband) and the potential pitfalls - but we more we look at it, the more the modern cloud-setups resemble the setups that massive-parallel computer-clusters already had a decade ago..