Network of moderate speed

[fivemack]

Infiniband HCAs like

http://www.ebay.co.uk/itm/HP-452372-...item53f8db23ab

are ubiquitously available on ebay, so I've bought two and connected them with a

http://www.ebay.co.uk/itm/INFINIBAND...item27d73d7981

sudo modprobe ib_umad
sudo modprobe mlx_ib
sudo modprobe ib_ipoib

on both machines, then

sudo ifconfig ib1 10.10.10.1 netmask 255.255.255.0

on one and

sudo ifconfig ib1 10.10.10.2 netmask 255.255.255.0

on the other, and Robert appears to be some relative of your father:

pumpkin@pumpkin:~$ netperf -H 10.10.10.2
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.10.2 () port 0 AF_INET : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec  

 87380  16384  16384    10.00    12408.11

There is perceptible CPU usage involved with the connection (in fact, probably just with netperf); the throughput with both ends running two gnfs-lasieve4I14e processes on each core drops to 7384Mbps.

It's still not to be sniffed at for less than £100 including postage.

Is there anything complicated I need to do to set up an MPI cluster between the two machines using the nice fast interconnect?

[EdH]

Originally Posted by fivemack

Is there anything complicated I need to do to set up an MPI cluster between the two machines using the nice fast interconnect?

Did this ever get answered?

I found the following site helpful in setting up my openmpi tests, (but you probably already know all of the following):

Setting up a Beowulf Cluster Using Open MPI on Linux

I have found that I need to have the exact same openmpi version (therefore, linux distribution) on all the machines for success. For the following reason, all the machines also need to have the same architecture. I have been using the msieve binary of the host machine, placed in the host machine's folder that is accessed by all the others, which, through sshfs are linked to the host machine's folder:

They all have the same account name and basic folder structure:

The host has /home/math85/Math/folder, which has all the programs/files/subdirectories.

The slaves have /home/math85/Math/folder, which is empty until I use sshfs to link in the host's folder.

I hope this is helpful...