Restarting a process after it is hung?

[chris2be8]

Originally Posted by blip

|   0  GeForce GTX 590     Off  | 0000:03:00.0     N/A |                  N/A |
|  0%   91C  N/A     N/A /  N/A |      153MB /  1535MB |     N/A      Default |

(quite hot, I know...)

Why is the fan for GPU 0 stopped even though it's at 91C? Have a look to see if it's turning. If you are lucky it's blocked by something so will start turning if you free it.

Chris

[Xyzzy]

Why is the fan for GPU 0 stopped even though it's at 91C?

It might be that the 590 is two GPUs on one card, with one fan?

[blip]

That is correct: It has only one fan.

[chris2be8]

OK , that explains it.

My GTX 560 Ti has two fans in the body, but nvidia-smi only reports one speed. I assume they both turn at the same speed.

Chris

[TheMawn]

Originally Posted by chris2be8

OK , that explains it.

My GTX 560 Ti has two fans in the body, but nvidia-smi only reports one speed. I assume they both turn at the same speed.

Chris

Likely the voltage / PWM is fed to both but RPM is only taken from one fan.

A fairly common doodad for builders is a case fan Y-splitter cable (especially with CPU heatsinks designed for two fans) and some bad manufacturers didn't think to cut one the rotation speed wire on one of the ends. Unless the rotations are in perfect synchronization (+/- impossible) the RPMs add up and cause some serious confusion for the motherboard and user who see a fan spinning at 4000 RPM.

And then the good manufacturers who remove the third pin get flack from the self-proclaimed gurus about having missing pins

(Lots of dumb mistakes like that on Newegg from "5/5 tech knowledge" users... my favourite is a guy who "tested hundreds of fans in his life and has a masters in fluid dynamics complaining that there's no arrow showing you the direction of flow)

[blip]

I am still having the issue with cudalucas stalling on this machine after a while (several hours).

mfaktc is running fine on the same card other chip (GTX590 with two chips).

Swapping processes between chips does not help.

Current setup:

Linux 3.11.0-18-generic (Ubuntu)

| NVIDIA-SMI 334.21 Driver Version: 334.21 |

both mfaktc and CUDAlucas compiled with

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2013 NVIDIA Corporation
Built on Sat_Jan_25_17:33:19_PST_2014
Cuda compilation tools, release 6.0, V6.0.1

[preben s]

I have had this problem for long time and I have a good hint on what is triggering it.

I have have very bad connection to the Internet, and the problem ocurred "only" when Chromium or Firefox (and to some extent my computer; it helped to throw out the crappy kmail) became unresponsive because of failing connection.

Got new phone lines drawn and no problems since then.

I made a script to run mfaktc under:

#!/usr/bin/sh

./mfaktc&
B=`nvidia-smi -q -d TEMPERATURE|grep Gpu|cut -c39,40`
while true; do
  A=$B; sleep 30
  B=`nvidia-smi -q -d TEMPERATURE|grep Gpu|cut -c39,40`
  C=`expr \( $A - $B \)`
  if  [ $C -ge 3 -o $B -lt 50 ]; then 
    kill `ps -e |grep mfaktc|grep -v sh|cut -d\  -f1`
    wait
    ./mfaktc&
  fi
done

It catches the stalling state and restarts rapidly.

[preben s]

Sorry, I posted the code above without testing enough.
This time it is better tested.

#!/usr/bin/sh

# If this script is named other than mfaktc.sh,
# the script below should be changed accordingly.

# It is assumed that the only process named mfaktc is
# the one started by this script.

# The temperature limits used here seems to work fine
# down to 30 seconds sleep time under 35 C environment
# temperature. Tested on a GTX 780 @ 1006 Mhz.

# Using the programs cut and tr is chosen for lean-ness.
# awk (gawk) could be used instead of cut/tr.
# Also sed could be used instead of tr.

# "man kill" states that kill can be used to find the
# pid of a named process. If somebody can make that to work,
# the script could be simplified a little.


echo `date` > start.log
./mfaktc&
B=`nvidia-smi -q -d TEMPERATURE|grep Gpu|cut -c39,40`
while true; do
  A=$B; sleep 120
  B=`nvidia-smi -q -d TEMPERATURE|grep Gpu|cut -c39,40`
  C=`expr \( $A - $B \)`
  if  [ $C -ge 3 -o $B -lt 50 ]; then
    PID=`ps -e |grep mfaktc|grep -v sh|cut -c1-5|tr ' ' '0'`
    if [ $PID ]; then kill $PID; sleep 2; fi
    PID=`ps -e |grep mfaktc|grep -v sh|cut -c1-5|tr ' ' '0'`
    if [ $PID ]; then kill $PID; fi
    wait
    echo `date` > start.log
    ./mfaktc&
  fi
done