The Ubiquiti ES-16-XG seems to be what I need, and I've just ordered one: 12 SFP+ ports, four 1Gbit/10Gbit RJ45 ports. Enough to keep me in moderately-quickly-interconnected compute nodes for as long as 10Gbit SFP+ Ethernet cards are readily available on eBay, and in the not-entirely-impossible event that 10GbaseT becomes standard on motherboards :)

I have a switch; I have four direct-attach cables.

When I plug the cable into the switch and the server (on which I've put Ubuntu 16.04.1), I see a message from ixgbe on the server saying 'SFP of type 3 detected'; when I unplug the cable from the switch, there is a message in the switch's log (which I can see because I have a gigabit cable plugged into the switch) saying 'SFP+ removed from port 2'.

But other than that, nothing happens - no link light on the switch, no light of any kind on the back of the server, no sign of a packet hopping its little packetty way across the network.

This does not strike me as terribly promising, and I'm not sure what the next step in diagnosis is; I can take the switch and the cables over to a friend who has known-good switches and cables to try swapping things over against, but I'm not at all sure what I can do on the server.

[QUOTE=fivemack;445486]This does not strike me as terribly promising, and I'm not sure what the next step in diagnosis is; I can take the switch and the cables over to a friend who has known-good switches and cables to try swapping things over against, but I'm not at all sure what I can do on the server.[/QUOTE]

That's what I'd try... make sure the switch/cables are working okay with known-good NICs before going on.

I've been avoiding 10Gb on my side for a while just because I want to wait and see how it all shakes out. 10Gbase-T would be ideal since the per-port cost seems like it'd be the most reasonable. I can think back to when the only way to get gigabit+ speeds was with expensive fiber, and now you can pick up a cheapo Gbase-T switch just about anywhere. The same will happen to 10Gb, hopefully.

You mentioned using direct-attach cables... I wasn't entirely sure from what you'd said previously about the type of cabling you were going to be using and what your transceivers were. Sounds like it's not fiber at any rate. I thought SFP+ direct-attach was a specific implementation so it may depend on whether the servers and switch both support it? Not really sure about all that.

Tom, look what you have missed.

[url]http://www.bargainhardware.co.uk/dedicated-cloud-server-rack-39x-foxconn-servers-544-e5-2670-cores-5tb-ram/[/url]

[QUOTE=pinhodecarlos;445534]Tom, look what you have missed.

[url]http://www.bargainhardware.co.uk/dedicated-cloud-server-rack-39x-foxconn-servers-544-e5-2670-cores-5tb-ram/[/url][/QUOTE]

They tried to sell me one of those when I rang up to ask about buying only four :)

But I don't have enough electricity supply to the shed to power them all, and I need thirty-nine computers no more than I need thirty-nine weasels braided into my beard.

[QUOTE=Madpoo;445494]That's what I'd try... make sure the switch/cables are working okay with known-good NICs before going on.

I've been avoiding 10Gb on my side for a while just because I want to wait and see how it all shakes out. 10Gbase-T would be ideal since the per-port cost seems like it'd be the most reasonable. I can think back to when the only way to get gigabit+ speeds was with expensive fiber, and now you can pick up a cheapo Gbase-T switch just about anywhere. The same will happen to 10Gb, hopefully.[/quote]

I'm a bit suspicious of 10Gbase-T because of the latency and the per-port power consumption - you're doing deep DSP magic to get that speed over those cables, which translates to several microseconds of latency and 5W per port per end. It'll be even worse for 25Gbase-T which is said to be coming soon.

A colleague who's into quite serious networking suspects that 5Gbase-T, which doesn't need to be quite so aggressive in the signal processing, probably will appear routinely on desktop motherboards in 2018 or so.

Well, that was interesting
 

After an evening of experiments with my switch, my cables, a known-good switch and some known-good cables, it was concluded that the switch is not a happy piece of equipment.

With a known-good 0.5-metre cable, my switch can establish a link with either of the servers from any of its twelve ports, and this link has near-enough zero packet loss.

With any of my cables (Amphenol AHSI 3-metre cables), I can link the ports on two servers and get near-enough zero packet loss. If I attach any of my cables to any of the ports on a Cisco switch, I get zero packet loss.

If I use my cables and my switch, I do not get a link at all if I use ports 1, 2, 3, 4, 9, 11 or 12 on the switch. With one particular one of the cables I get a link from switch port 10 only if I use the cable in one orientation. With another of the cables I cannot use switch port 7 or 10. In all configurations I get between 0.2% and 0.8% packet loss.

I suspect the designers of the switch may have done rather more cost optimisation than they should in the high-speed signalling circuitry. I don't know whether I should return it for a replacement, or return it for a refund and buy a new switch from a company better at making switches.

Better news about BargainHardware machines
 

They work fine with Ubuntu-16.04 and an Anker USB3-to-gigabit-Ethernet cable, though the file transfer rate isn't much above 20MB per second; I suspect the ports are USB2.

Despite what it says in Intel's manuals for the X520 network cards which use the same chipset, you *can* put an SFP-to-1GBaseT transceiver in the front port of the Foxconn machines and get gigabit ethernet out; these transceivers are £16 from Amazon, which is only slightly more expensive than the USB3 connector, and are likely to work a lot better. So for £300 for the machine, £25 for a 60G SSD off eBay, £16 for a transceiver, you get yourself about the same performance as seven cores of i7/4790K. The machines are really quite loud.