Is this an item of commerce?
 

I'm looking for an eight-port 10GbE switch with SFP+ connectors, because buying a pile of

[url]http://www.bargainhardware.co.uk/cheap-e5-2600-sixteen-core-cloud-foxconn-server-configure-to-order/[/url]

and connecting them with gigabit-over-copper seems a bit silly.

A source of SFP+ cables which don't cost a hundred pounds each would also be quite useful - at the moment it looks as if it's cheaper to get SFP+-form-factor SR transceivers and SC-at-each-end optic fibres, and I can't really believe that a system involving two sets of high-tech lasers ought to be cheaper than two metres of even fairly fancy copper wire.

Well, 10GbE switches aren't exactly cheap, but it looks like you can get them and the cables for (just) less than the cost of the 8 computers connected to it:

Here is a 12-port version:
[url]https://www.cdw.com/shop/products/D-Link-Web-Smart-DXS-1210-12SC-switch-12-ports-managed-desktop-rac/3733028.aspx[/url]

And some direct attach copper cables:
[url]http://www.cablesondemand.com/category/SFP%2B+CBL/URvars/Catalog/Library/InfoManage/SFP+_CABLES_(DIRECT_ATTACH).htm[/url]

I didn't do much shopping so its probable they could come slightly cheaper somewhere else.

[url]http://www.bargainhardware.co.uk/cisco-catalyst-3750e-24td-s-24-ethernet-port-switch-ws-c3750e-24td-s/[/url]

and

[url]http://www.bargainhardware.co.uk/intel-x520-da2-dual-port-10gbe-ethernet-cna-10gb-sfp-with-two-sfp-cables/[/url]

and

[url]http://www.bargainhardware.co.uk/sfp-cable-cisco-twinax-direct-attached-copper-sfp-h10gb-cu3m-10gbe-dell-j564n/[/url]

Monoprice has cables for $25: [url]http://www.monoprice.com/product?p_id=12051[/url]

[QUOTE=pinhodecarlos;444299][url]http://www.bargainhardware.co.uk/cisco-catalyst-3750e-24td-s-24-ethernet-port-switch-ws-c3750e-24td-s/[/url][/QUOTE]

That's a 1000BaseT switch, not 10Gbit.

[quote][url]http://www.bargainhardware.co.uk/intel-x520-da2-dual-port-10gbe-ethernet-cna-10gb-sfp-with-two-sfp-cables/[/url][/quote]

The machines already have dual 10Gbit SFP+ ethernet on the motherboard

[quote][url]http://www.bargainhardware.co.uk/sfp-cable-cisco-twinax-direct-attached-copper-sfp-h10gb-cu3m-10gbe-dell-j564n/[/url][/QUOTE]

But that one's useful.

I, err, may have ordered a 42U rack for my shed and four of the bargainhardware machines to put in it. £20 per 2GHz Sandy Bridge core with 4GB RAM is not to be sniffed at; I wasn't sad paying £90 per 1.9GHz Opteron core with 1365MB RAM five years ago.

[QUOTE=fivemack;444305]
I, err, may have ordered a 42U rack for my shed and four of the bargainhardware machines to put in it. £20 per 2GHz Sandy Bridge core with 4GB RAM is not to be sniffed at; I wasn't sad paying £90 per 1.9GHz Opteron core with 1365MB RAM five years ago.[/QUOTE]

:smile:

I love how you can just go and do this kind of stuff. In my house there are several layers of committee and management review boards (all of which are solely staffed and bottlenecked by the same individual), each with a different approval process that changes each time I encounter it, that must be traversed before any new computer equipment may be purchased.

[QUOTE=bsquared;444308]:smile:

I love how you can just go and do this kind of stuff. In my house there are several layers of committee and management review boards (all of which are solely staffed and bottlenecked by the same individual), each with a different approval process that changes each time I encounter it, that must be traversed before any new computer equipment may be purchased.[/QUOTE]

[I]Down to Gehenna or up to the Throne,
He travels the fastest who travels alone.[/I]

-- Rudyard Kipling, [I]The Winners[/I]

[QUOTE=fivemack;444291]I'm looking for an eight-port 10GbE switch with SFP+ connectors, because buying a pile of

[URL]http://www.bargainhardware.co.uk/cheap-e5-2600-sixteen-core-cloud-foxconn-server-configure-to-order/[/URL]

and connecting them with gigabit-over-copper seems a bit silly.

A source of SFP+ cables which don't cost a hundred pounds each would also be quite useful - at the moment it looks as if it's cheaper to get SFP+-form-factor SR transceivers and SC-at-each-end optic fibres, and I can't really believe that a system involving two sets of high-tech lasers ought to be cheaper than two metres of even fairly fancy copper wire.[/QUOTE]

Tom, have you bought them? Really looking forward to hearing from you about this rack servers.

[QUOTE=pinhodecarlos;444471]Tom, have you bought them? Really looking forward to hearing from you about this rack servers.[/QUOTE]

They should be arriving on Tuesday. Just put in the auxiliary orders for five kettle leads, the weird mains-plug-to-IEC60309 converter (usually used for plugging in motor-homes), gigabit switch, six patch cables in different colours, C14-to-mains-socket converter to plug the gigabit switch into ... debating slightly what to do for disc, probably a four-pack of 120GB SSDs is the answer since I'm likely to want to do NFS filtering, which will be unbearably slow from USB2 stick.

Good news! The servers have arrived!

Further good news! I will never run out of Ethernet patch cables or C13-to-C14 mains leads again, since the movers did not remove the 84 patch cables and 19 mains leads velcroed to the rack before delivery!

Indifferent news! I now have a much more visceral understanding of palletised delivery.

More indifferent news! The rack advertised as 'weighs 20kg' on the Web site actually weighs 60kg. The very best data-centre commissioning engineers probably arrange that at no stage in the commissioning process does the rack need to be manhandled across long damp grass.

Decidedly unfortunate news! The building I intend to install the rack in has ceiling-rafters which descend to less than 42U from the floor.

Post pictures please.:max:

[QUOTE=fivemack;444783]The very best data-centre commissioning engineers probably arrange that at no stage in the commissioning process does the rack need to be manhandled across long damp grass.
[/QUOTE]
:davar55: Hehe... :tu:
[QUOTE]
Decidedly unfortunate news! The building I intend to install the rack in has ceiling-rafters which descend to less than 42U from the floor.[/QUOTE]Is there any room more than 42U wide in at least one direction? Then problem solved... :razz:

Pictures
 

1U servers; Rack with house for scale; back of rack showing all the free cables it came with; sad sideways rack is sad.

More relevantly to competent people, or at least people who have not decided that a 42U rack might be a sensible thing to own: whilst the spec document said "2x SFP+ Ports (82599 10 Gigabit Network Connection)" when I ordered them, there is actually only one usable SFP+ port - the metalwork has two side by side, but only one of them has a connector at the back. The spec document has now been changed and I have received a small discount for the thing not being as advertised.

This is a pity; I was hoping to connect the four machines as A-B C-D A-C B-D, which is all that linear algebra needs, and use Linux routing tables to make it possible to talk from A to D, but this obviously doesn't work with only one port per machine.

[QUOTE=fivemack;444796]1U servers; Rack with house for scale; back of rack showing all the free cables it came with; sad sideways rack is sad.[/QUOTE]Reminds me of my first full-time paid job or, rather, a year or so before I got paid.

While a grad student I worked with a small start-up in its very early stages. The company found premises in a building which had been a donkey stable. The hayloft became a large shared office and the room where the donkey lived was converted into the machine room. The expression on the DEC sales critter was wonderful when he realised that a shiny new VAX 11/750 (then about £100K list price) was to be installed in a room which then still had the drainage channels designed to keep the donkey's feet out of the donkey's excreta. There was even some straw left in odd corners. Of course, all was clean and level by the time the VAX was delivered.

[QUOTE=fivemack;444798]

This is a pity; I was hoping to connect the four machines as A-B C-D A-C B-D, which is all that linear algebra needs, and use Linux routing tables to make it possible to talk from A to D, but this obviously doesn't work with only one port per machine.[/QUOTE]

So what's your next step? Think you edited one of your messages with regards to MPI solution to adapt or I was half sleeping and read it somewhere else.

[QUOTE=fivemack;444783]Good news! The servers have arrived!
...
Decidedly unfortunate news! The building I intend to install the rack in has ceiling-rafters which descend to less than 42U from the floor.[/QUOTE]

They make "baby racks" that are a mere 14U or some variation around that. They're the type of thing you're likely to find in a utility closet or something, hosting a few switches or whatever and they didn't bother with a 2-post telco rack even.

Might find a decent used one...

Be thankful you didn't wind up with a taller 48U rack, I guess.

I hope you find some clever way of getting it upright where you planned... worst case I guess you can do it sideways but it sure takes the fun out of sliding servers in and out. Could be worse though.

[QUOTE=Madpoo;444922]They make "baby racks" that are a mere 14U or some variation around that. They're the type of thing you're likely to find in a utility closet or something, hosting a few switches or whatever and they didn't bother with a 2-post telco rack even.

Might find a decent used one...

Be thankful you didn't wind up with a taller 48U rack, I guess.

I hope you find some clever way of getting it upright where you planned... worst case I guess you can do it sideways but it sure takes the fun out of sliding servers in and out. Could be worse though.[/QUOTE]Or find someone with a hacksaw and a MIG welding kit.

[QUOTE=xilman;444929]Or find someone with a hacksaw and a MIG welding kit.[/QUOTE]

I know plenty of people with hacksaws, but I don't know hobbyist welders and it feels a bit much welding on rather thin painted steel plate to ask someone to do as a favour. My current kludge plan would be to cut it off at about 30U height, and then cut bits of sturdy timber to size and bolt them around the uprights to keep the uprights vertical and parallel; is there a fundamental flaw in that?

[QUOTE=pinhodecarlos;444845]So what's your next step? Think you edited one of your messages with regards to MPI solution to adapt or I was half sleeping and read it somewhere else.[/QUOTE]

I edited the message to suggest connecting the machines in pairs, but then realised I had returned all the SFP+ cables for a refund.

OK, now I have a new and exciting problem.

The servers don't have conventional-Ethernet hardware connected to the processor, their only network port is the 10Gbit SFP+ one and the Ethernet port at the front connects only to the (quite fancy) lights-out management server.

What do I need to attach four of them to my current 1Gbit network? My suspicion is quite an expensive switch; it seems that the 82559 10Gbit ethernet controller will [I]not[/I] step down to 1Gbit if I plug a 1Gbit SFP->1000BaseT into its SFP+ port, but that the sort of switches that Cisco sell [i]will[/i] do that, but that those switches cost more than the cluster.

Does anyone here know about enterprise networking to the point of being able to answer authoritatively whether [url]http://www.tp-link.com/en/products/details/cat-40_T1700G-28TQ.html[/url] treats the 10GE slots uniformly with the other ports, or whether it expects to route only *from* the gigabit ports *to* the 10GE SFP+ ones, rather than between the 10GE SFP+ ones and from 10GE SFP+ to gigabit?

Many inexpensive switches have two SPF+ ports. You could get two of those.

If the USB ports are USB3, you can get decent gigabit Ethernet dongles made by StarTech (I've had excellent luck with that brand in particular).

The Ubiquiti ES-16-XG seems to be what I need, and I've just ordered one: 12 SFP+ ports, four 1Gbit/10Gbit RJ45 ports. Enough to keep me in moderately-quickly-interconnected compute nodes for as long as 10Gbit SFP+ Ethernet cards are readily available on eBay, and in the not-entirely-impossible event that 10GbaseT becomes standard on motherboards :)

I have a switch; I have four direct-attach cables.

When I plug the cable into the switch and the server (on which I've put Ubuntu 16.04.1), I see a message from ixgbe on the server saying 'SFP of type 3 detected'; when I unplug the cable from the switch, there is a message in the switch's log (which I can see because I have a gigabit cable plugged into the switch) saying 'SFP+ removed from port 2'.

But other than that, nothing happens - no link light on the switch, no light of any kind on the back of the server, no sign of a packet hopping its little packetty way across the network.

This does not strike me as terribly promising, and I'm not sure what the next step in diagnosis is; I can take the switch and the cables over to a friend who has known-good switches and cables to try swapping things over against, but I'm not at all sure what I can do on the server.

[QUOTE=fivemack;445486]This does not strike me as terribly promising, and I'm not sure what the next step in diagnosis is; I can take the switch and the cables over to a friend who has known-good switches and cables to try swapping things over against, but I'm not at all sure what I can do on the server.[/QUOTE]

That's what I'd try... make sure the switch/cables are working okay with known-good NICs before going on.

I've been avoiding 10Gb on my side for a while just because I want to wait and see how it all shakes out. 10Gbase-T would be ideal since the per-port cost seems like it'd be the most reasonable. I can think back to when the only way to get gigabit+ speeds was with expensive fiber, and now you can pick up a cheapo Gbase-T switch just about anywhere. The same will happen to 10Gb, hopefully.

You mentioned using direct-attach cables... I wasn't entirely sure from what you'd said previously about the type of cabling you were going to be using and what your transceivers were. Sounds like it's not fiber at any rate. I thought SFP+ direct-attach was a specific implementation so it may depend on whether the servers and switch both support it? Not really sure about all that.

Tom, look what you have missed.

[url]http://www.bargainhardware.co.uk/dedicated-cloud-server-rack-39x-foxconn-servers-544-e5-2670-cores-5tb-ram/[/url]

[QUOTE=pinhodecarlos;445534]Tom, look what you have missed.

[url]http://www.bargainhardware.co.uk/dedicated-cloud-server-rack-39x-foxconn-servers-544-e5-2670-cores-5tb-ram/[/url][/QUOTE]

They tried to sell me one of those when I rang up to ask about buying only four :)

But I don't have enough electricity supply to the shed to power them all, and I need thirty-nine computers no more than I need thirty-nine weasels braided into my beard.

[QUOTE=Madpoo;445494]That's what I'd try... make sure the switch/cables are working okay with known-good NICs before going on.

I've been avoiding 10Gb on my side for a while just because I want to wait and see how it all shakes out. 10Gbase-T would be ideal since the per-port cost seems like it'd be the most reasonable. I can think back to when the only way to get gigabit+ speeds was with expensive fiber, and now you can pick up a cheapo Gbase-T switch just about anywhere. The same will happen to 10Gb, hopefully.[/quote]

I'm a bit suspicious of 10Gbase-T because of the latency and the per-port power consumption - you're doing deep DSP magic to get that speed over those cables, which translates to several microseconds of latency and 5W per port per end. It'll be even worse for 25Gbase-T which is said to be coming soon.

A colleague who's into quite serious networking suspects that 5Gbase-T, which doesn't need to be quite so aggressive in the signal processing, probably will appear routinely on desktop motherboards in 2018 or so.

Well, that was interesting
 

After an evening of experiments with my switch, my cables, a known-good switch and some known-good cables, it was concluded that the switch is not a happy piece of equipment.

With a known-good 0.5-metre cable, my switch can establish a link with either of the servers from any of its twelve ports, and this link has near-enough zero packet loss.

With any of my cables (Amphenol AHSI 3-metre cables), I can link the ports on two servers and get near-enough zero packet loss. If I attach any of my cables to any of the ports on a Cisco switch, I get zero packet loss.

If I use my cables and my switch, I do not get a link at all if I use ports 1, 2, 3, 4, 9, 11 or 12 on the switch. With one particular one of the cables I get a link from switch port 10 only if I use the cable in one orientation. With another of the cables I cannot use switch port 7 or 10. In all configurations I get between 0.2% and 0.8% packet loss.

I suspect the designers of the switch may have done rather more cost optimisation than they should in the high-speed signalling circuitry. I don't know whether I should return it for a replacement, or return it for a refund and buy a new switch from a company better at making switches.

Better news about BargainHardware machines
 

They work fine with Ubuntu-16.04 and an Anker USB3-to-gigabit-Ethernet cable, though the file transfer rate isn't much above 20MB per second; I suspect the ports are USB2.

Despite what it says in Intel's manuals for the X520 network cards which use the same chipset, you *can* put an SFP-to-1GBaseT transceiver in the front port of the Foxconn machines and get gigabit ethernet out; these transceivers are £16 from Amazon, which is only slightly more expensive than the USB3 connector, and are likely to work a lot better. So for £300 for the machine, £25 for a 60G SSD off eBay, £16 for a transceiver, you get yourself about the same performance as seven cores of i7/4790K. The machines are really quite loud.