Restarting a process after it is hung? - Page 2

Xyzzy · 2014-02-14, 21:25

This is weird, but we think the order in which the GPUs are installed causes this issue.

(Perhaps Linux deals with card order improperly? Windows worked flawlessly.)

We have three GPUs in the system now.

We carefully monitor temperatures and never allow the GPUs to exceed 70°C.

We will report back in a few days with our results.

EdH · 2014-02-14, 21:48

Quote:

Originally Posted by Xyzzy

This is weird, but we think the order in which the GPUs are installed causes this issue.

(Perhaps Linux deals with card order improperly? Windows worked flawlessly.)

We have three GPUs in the system now.

We carefully monitor temperatures and never allow the GPUs to exceed 70°C.

We will report back in a few days with our results.

I have had issues with linux not assigning USB devices in the same order during reboot. Is this a similar issue?

Xyzzy · 2014-02-15, 22:33

We figured out that (for some reason) the 660 "owns" the "gpu 0" slot no matter what slot we put it in, and even if we do not use it as the primary display, or as a display at all.

(The 660 and TITAN are double-width cards so we can only use them in the top two slots. We have a 430 that is a single-width card that we put into the third slot.)

Now that we have tested the system with the 660 in the slot closest to the CPU everything has worked perfectly.

The 660 was the card that was stalling, and it has not done so since we tried the current order, so we will probably keep things just the way they are.

chalsall · 2014-02-15, 22:39

Quote:

Originally Posted by Xyzzy

The 660 was the card that was stalling, and it has not done so since we tried the current order, so we will probably keep things just the way they are.

Keep observing...

Just because you change something (or reboot), and it appears to fix the problem, it doesn't necessarily mean the problem is fixed. You need many samples before you can be confident (not sure, but confident).

blip · 2014-02-16, 22:00

I happen to have the same issue with a 590: One or the other of the two cores just hangs after a while. Both do CUDALucas. (Ubuntu 13.10).

chalsall · 2014-02-16, 23:00

Quote:

Originally Posted by blip

I happen to have the same issue with a 590: One or the other of the two cores just hangs after a while. Both do CUDALucas. (Ubuntu 13.10).

This suggests strongly that the drivers are at fault, rather than the programs.

No big surprise, really....

Xyzzy · 2014-02-16, 23:15

No hangs yet from the system. Our fingers are crossed!

Code:

$ w
 17:14:27 up 2 days,  2:32,  6 users,  load average: 2.45, 2.82, 3.21

blip · 2014-02-16, 23:40

Quote:

Originally Posted by chalsall

This suggests strongly that the drivers are at fault, rather than the programs.

No big surprise, really....

Code:

nvidia-smi
Mon Feb 17 00:34:53 2014       
+------------------------------------------------------+                       
| NVIDIA-SMI 5.319.32   Driver Version: 319.32         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 590     Off  | 0000:03:00.0     N/A |                  N/A |
|  0%   91C  N/A     N/A /  N/A |      153MB /  1535MB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 590     Off  | 0000:04:00.0     N/A |                  N/A |
| 88%   89C  N/A     N/A /  N/A |      153MB /  1535MB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 650     Off  | 0000:09:00.0     N/A |                  N/A |
| 18%   46C  N/A     N/A /  N/A |       46MB /  2047MB |     N/A      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Compute processes:                                               GPU Memory |
|  GPU       PID  Process name                                     Usage      |
|=============================================================================|
|    0            Not Supported                                               |
|    1            Not Supported                                               |
|    2            Not Supported                                               |
+-----------------------------------------------------------------------------+

(quite hot, I know...)

(Where) Is there a newer/better driver for Ubuntu 13.10? (331.??)

kracker · 2014-02-17, 01:57

Um... Get it from nvidia's website?

blip · 2014-02-17, 08:20

Quote:

Originally Posted by kracker

Um... Get it from nvidia's website?

In my case:

Code:

sudo add-apt-repository ppa:xorg-edgers/ppa
sudo apt-get update
sudo apt-get install nvidia-331

Xyzzy · 2014-02-19, 17:47

We need to reboot for a kernel update. So far no problems!

Code:

$ w
 11:47:12 up 4 days, 21:05,  6 users,  load average: 4.10, 4.18, 4.15

2014-02-16, 23:15	#18
Xyzzy "Mike" Aug 2002 8224₁₀ Posts	No hangs yet from the system. Our fingers are crossed! Code: $ w 17:14:27 up 2 days, 2:32, 6 users, load average: 2.45, 2.82, 3.21

2014-02-19, 17:47	#22
Xyzzy "Mike" Aug 2002 2⁵·257 Posts	We need to reboot for a kernel update. So far no problems! Code: $ w 11:47:12 up 4 days, 21:05, 6 users, load average: 4.10, 4.18, 4.15

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Trouble restarting large job	fivemack	Msieve	4	2018-01-04 01:13
assignment restarting prob	isaac1204	Information & Answers	2	2017-07-20 17:26
restarting nfs linear algebra	cubaq	YAFU	2	2017-04-02 11:35
Well hung parliaments	davieddy	Soap Box	0	2010-08-23 13:43
Stop p95 or llr before restarting?	Joshua2	Software	6	2005-05-16 16:36

2014-02-14, 21:25	#12
Xyzzy "Mike" Aug 2002 2020₁₆ Posts	This is weird, but we think the order in which the GPUs are installed causes this issue. (Perhaps Linux deals with card order improperly? Windows worked flawlessly.) We have three GPUs in the system now. We carefully monitor temperatures and never allow the GPUs to exceed 70°C. We will report back in a few days with our results.

2014-02-15, 22:33	#14
Xyzzy "Mike" Aug 2002 20040₈ Posts	We figured out that (for some reason) the 660 "owns" the "gpu 0" slot no matter what slot we put it in, and even if we do not use it as the primary display, or as a display at all. (The 660 and TITAN are double-width cards so we can only use them in the top two slots. We have a 430 that is a single-width card that we put into the third slot.) Now that we have tested the system with the 660 in the slot closest to the CPU everything has worked perfectly. The 660 was the card that was stalling, and it has not done so since we tried the current order, so we will probably keep things just the way they are.

2014-02-16, 22:00	#16
blip Jan 2014 146₁₀ Posts	I happen to have the same issue with a 590: One or the other of the two cores just hangs after a while. Both do CUDALucas. (Ubuntu 13.10).

2014-02-17, 01:57	#20
kracker "Mr. Meeseeks" Jan 2012 California, USA 2³×271 Posts	Um... Get it from nvidia's website?