Interesting GPU heat issue...
 

Interesting GPU heat issue on one of my nodes. This is i7 2600k@4.5GHz with 2x GTX580s.

I have one card here that's instance 0 and 1. (top two mfaktc windows - the gpuz in the foreground/left)

Note that their times are twice that of the bottom 2 instances on the other GPU (gpuz on the right/background)

Looking at gpuz stats the slower card is running 97degC (ish). It seems the card is doing 50% of the work output that it should be doing. If I close the mfaktc windows and restart when it's cooler, iteration times are normal.

Yes I recognize this is running 'hot'. :)

If you look at the fan speed on the problem card - it seems all over the place.

Anyone recommend a windows app to control GPU fan speeds?

-- Craig

[QUOTE=nucleon;286109]Interesting GPU heat issue on one of my nodes. This is i7 2600k@4.5GHz with 2x GTX580s.

I have one card here that's instance 0 and 1. (top two mfaktc windows - the gpuz in the foreground/left)

Note that their times are twice that of the bottom 2 instances on the other GPU (gpuz on the right/background)

Looking at gpuz stats the slower card is running 97degC (ish). It seems the card is doing 50% of the work output that it should be doing. If I close the mfaktc windows and restart when it's cooler, iteration times are normal.

Yes I recognize this is running 'hot'. :)

If you look at the fan speed on the problem card - it seems all over the place.

Anyone recommend a windows app to control GPU fan speeds?

-- Craig[/QUOTE]

You could try MSI Afterburner. It lets you define the fan response curve.

If the frequency shown wasn't so high, I might have thought that the card had throttled because of the high temp. EDIT: But I don't know what frequency it should be running at.
[URL]http://event.msi.com/vga/afterburner/download.htm[/URL]

EDIT2: I use Afterburner even though my GTX 460 is by Gigabyte.

[QUOTE=nucleon;286109]Interesting GPU heat issue on one of my nodes.
<...>
Anyone recommend a windows app to control GPU fan speeds?
[/QUOTE]
.
That is exactly the issue I was describing, with the GPU card in the middle getting hotter then the other, for the simple reason that airflow is not right, this card is getting all the heat from the CPU (above) and the second GPU (below) and the space in between is not the most "fortunate". I have about the same config, I described somewhere around. What mobo/gpu is that? (brand). If they are Asus (like for example Maximus Extreme Z or Sabertooth with ENGTX580/590/mars) then Asus has [B]very nice[/B] "control tools" to deal with fans, overclocking, etc. (SmartDoctor, GPUTweak, iTracker, etc), which I am using, but the best way would be (as I mentioned) going for watercooling for the card in the middle, at least (I was waiting for Santa, but he was not so eager to come this Christmas... or at least, not with that present... maybe in a month or two, he will come for Chinese New Year, or Thai Songkran, depends of what savings we can make...:smile:). I said "at least", because once you got the pumps, tubes, radiators, blabla, there is only a small step to buy two more plates and two splitters and have watercooling for all three (both gpu's and the cpu).

[QUOTE=kladner;286114]You could try MSI Afterburner. It lets you define the fan response curve.

If the frequency shown wasn't so high, I might have thought that the card had throttled because of the high temp. EDIT: But I don't know what frequency it should be running at.
[URL]http://event.msi.com/vga/afterburner/download.htm[/URL]

EDIT2: I use Afterburner even though my GTX 460 is by Gigabyte.[/QUOTE]
Seconded. Though mine is MSI, I've seen in many places that it's more-or-less compatible with everything.

nVidia reports 772 core clock standard, so that's fine.
[url]http://www.geforce.com/Hardware/GPUs/geforce-gtx-580/specifications[/url]

I think LaurV has really put the finger on it (assuming that this card is sandwiched.) The fact that stopping the app and letting the card cool results in better performance really does suggest some kind of throttling.

These are exotic situations with multiple high-wattage devices packed in so closely. Once again, my airflow instincts are twitching big time. But you have a pretty good idea of such things. Try to get more air in between is all I can say.

We can see here the fan has pretty much died now.

I've reduced it to one instance for the moment until I can get a fix organized. Like you said more airflow needed.

It's a shame $500 of GPU power boils down to a fan assembly that costs maybe $20-50 all up.

-- Craig

[QUOTE=kladner;286114]You could try MSI Afterburner. It lets you define the fan response curve.

If the frequency shown wasn't so high, I might have thought that the card had throttled because of the high temp. EDIT: But I don't know what frequency it should be running at.
[URL]http://event.msi.com/vga/afterburner/download.htm[/URL]

EDIT2: I use Afterburner even though my GTX 460 is by Gigabyte.[/QUOTE]

Cheers - I've tried afterburner. No luck increasing the fan speed. It looks like the fan has seized. It's happened on another card I had.

-- Craig

[QUOTE=nucleon;286143]Cheers - I've tried afterburner. No luck increasing the fan speed. It looks like the fan has seized. It's happened on another card I had.

-- Craig[/QUOTE]

How long did the cards run before you had the fan problem? I'm not looking forward to this happening to me.

[QUOTE=nucleon;286140]We can see here the fan has pretty much died now.

I've reduced it to one instance for the moment until I can get a fix organized. Like you said more airflow needed.

It's a shame $500 of GPU power boils down to a fan assembly that costs maybe $20-50 all up.

-- Craig[/QUOTE]

I failed to notice previously (or put 2 and 2 together) that the fan controller was constantly ~85% while the RPM varied. :redface:

Besides seconding Chuck's question as to run time before failure, I am curious who made the card and what approx temp it was running before the fan went south. Is it a single fan design? Is it "in the middle" as LaurV put it? I am wondering now if the ambient temp it was running in helped take down the fan. (Not that that is any excuse.)

It is a damned shame that the manufacturer didn't put a more reliable part in, considering how vital that part is. I'd be pretty surprised if a ~70mm fan (or 2!) cost them even $20 at wholesale. The most charitable thing I can say is that maybe this was just bad luck as to the particular fan you ended up with.

My condolences, and hopes that you get a quick and painle$$ resolution.

[QUOTE=kladner;286170]I failed to notice previously (or put 2 and 2 together) that the fan controller was constantly ~85% while the RPM varied. :redface:

Besides seconding Chuck's question as to run time before failure, I am curious who made the card and what approx temp it was running before the fan went south. Is it a single fan design? Is it "in the middle" as LaurV put it? I am wondering now if the ambient temp it was running in helped take down the fan. (Not that that is any excuse.)

It is a damned shame that the manufacturer didn't put a more reliable part in, considering how vital that part is. I'd be pretty surprised if a ~70mm fan (or 2!) cost them even $20 at wholesale. The most charitable thing I can say is that maybe this was just bad luck as to the particular fan you ended up with.

My condolences, and hopes that you get a quick and painle$$ resolution.[/QUOTE]

I think that luck has a lot to do with it.

I looked up when I purchased 'them'.

I bought 2x of the same model (MSI GTX580 Twin Frozr II OC) - 7/3/11 and 28/3/11. Now both models have fans that aren't working anymore. You could possibly google what they looked like (dual fans). I'm spewing as they advertised that they were pretty good for cooling.

I've been pretty much running them 24*7 since I've bought them. It was running high 90degC for a few weeks now. Looks like the fans have had problems for a while my guess.

If I use my finger to rotate the blades, I can feel obvious friction. They've 'siezed up' somehow.

-- Craig