Gpuowl / Linux question

[Prime95]

For the second time in a week I woke up to find all the gpuowls on a Linux box hung. Dmesg reports this:

[183465.976255] Restoring PASID 32768 queues
[183465.976348] Restoring PASID 32768 queues
[265226.254716] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring page0 timeout, signaled seq=49658, emitted seq=49660
[265226.254769] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
[265226.254775] amdgpu 0000:04:00.0: GPU reset begin!
[265226.254782] Evicting PASID 32769 queues

Do any experts have a good suggestion on how to either prevent this from happening AND/OR have gpuowl recover properly AND/OR a way to detect the hung condition, terminate the gpuowls, and restart the gpuowls?

I'll be leaving for an extended trip and hope to have a remedy in place before I go.

[paulunderwood]

Originally Posted by Prime95

For the second time in a week I woke up to find all the gpuowls on a Linux box hung. Dmesg reports this:

[183465.976255] Restoring PASID 32768 queues
[183465.976348] Restoring PASID 32768 queues
[265226.254716] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring page0 timeout, signaled seq=49658, emitted seq=49660
[265226.254769] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
[265226.254775] amdgpu 0000:04:00.0: GPU reset begin!
[265226.254782] Evicting PASID 32769 queues

Do any experts have a good suggestion on how to either prevent this from happening AND/OR have gpuowl recover properly AND/OR a way to detect the hung condition, terminate the gpuowls, and restart the gpuowls?

I'll be leaving for an extended trip and hope to have a remedy in place before I go.

Passing amdgpu.noretry=0 to he kernel might help. Adding it to the approriate grub line GRUB_CMDLINE_LINUX_DEFAULT (space delimited) and running update-grub will make it permanent for the next boot. See this for more details

I seem to have an old kernel:

uname -a
Linux honeypot9 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

[axn]

Originally Posted by Prime95

a way to detect the hung condition

Checking the age of the checkpoint / result file is probably the easiest way to detect no-progress condition. A simple script that runs every x minutes, checks the date of latest update in the folder, and if it is too old, killall the process and restart it.

[preda]

Originally Posted by Prime95

For the second time in a week I woke up to find all the gpuowls on a Linux box hung. Dmesg reports this:

[183465.976255] Restoring PASID 32768 queues
[183465.976348] Restoring PASID 32768 queues
[265226.254716] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* ring page0 timeout, signaled seq=49658, emitted seq=49660
[265226.254769] [drm:amdgpu_job_timedout [amdgpu]] *ERROR* Process information: process  pid 0 thread  pid 0
[265226.254775] amdgpu 0000:04:00.0: GPU reset begin!
[265226.254782] Evicting PASID 32769 queues

Do any experts have a good suggestion on how to either prevent this from happening AND/OR have gpuowl recover properly AND/OR a way to detect the hung condition, terminate the gpuowls, and restart the gpuowls?

I'll be leaving for an extended trip and hope to have a remedy in place before I go.

Is the OS responding when this happens? is it possible to kill the gpuowl processes? (e.g. with ctrl-C, kill, kill -9). If you [re]start a gpuowl in that state, does it work?

[ATH]

Originally Posted by Prime95

I'll be leaving for an extended trip and hope to have a remedy in place before I go.

Time to prepare for double checking a new prime

(we need a "hope" smiley or "cross your fingers" smiley)

[Prime95]

Originally Posted by preda

Is the OS responding when this happens? is it possible to kill the gpuowl processes? (e.g. with ctrl-C, kill, kill -9). If you [re]start a gpuowl in that state, does it work?

The OS does respond. gpuowl does not react to ^C. Killing the gpuowls and restarting does work.

[Prime95]

Originally Posted by axn

Checking the age of the checkpoint / result file is probably the easiest way to detect no-progress condition. A simple script that runs every x minutes, checks the date of latest update in the folder, and if it is too old, killall the process and restart it.

This looks like a great option.

My linux-fu is poor and I'm a little lazy to google right now. What would the syntax be for:

if (gpuowl.log has not been updated in the last hour) {
killall gpuowl
}

I can do the crontab entry and restarting is the same as the start-at-reboot code.

[kriesel]

An application and OS agnostic general case code snippet would be great. I have an instance of CUDAPm1 on Windows that restarts by batch file when it exits, but lately sometimes it produces no output into the redirected log file for nearly a day, not even the usual startup prints. Then I kill it and relaunch it manually and it seems to be fine for a while. Preferably it would be perl that could be compiled in Indigostar's perl2exe. http://www.indigostar.com/perl2exe/

Something along the lines of if this specifiable file path/name (such as a process's log file)'s last-modification date is older than this settable age, kill the process that has it open for append and relaunch. Or if the last saved checkpoint file is older than a settable age.
Support for a list of files and folders to be separately checked and individually processed. (I set up with a separate folder per running instance.)

I'm still working on a general monitor and results gathering application for several gpu apps. Will consider adding this functionality to it. More likely in the short term it will be a separate creation.

[paulunderwood]

Originally Posted by Prime95

This looks like a great option.

My linux-fu is poor and I'm a little lazy to google right now. What would the syntax be for:

if (gpuowl.log has not been updated in the last hour) {
killall gpuowl
}

I can do the crontab entry and restarting is the same as the start-at-reboot code.

If you can hack this not-so-great code...

if (( $(date +%s) > $(stat -c %Y -- examplefile.txt ) + 3600 )); then killall gpuowl; echo "hi" ;  fi;

[kriesel]

In Windows, it's tasklist and taskkill. There are various filters in them.
In George's case it seems all gpuowl instances are to be killed and restarted.
It gets trickier if wanting to determine which pid corresponds to one hung gpu app to kill and restart, among multiple processes running the same app but on different gpus or in different folders, or different app names. Roll through a list and see which one has the relevant log file open?
Is there a Windows command line equivalent to linux lsof, that would show which process id has which log file open?

[kriesel]

Originally Posted by kriesel

Is there a Windows command line equivalent to linux lsof, that would show which process id has which log file open?

Yes, an addon.
https://docs.microsoft.com/en-us/sys...wnloads/handle
Run the program once interactively, to read and accept the license terms, that will pop up separately, before attempting to use it in any sort of script, as the license is programmed to be a showstopper until accepted.

C:\Users\ken\Documents>handle64 gpuowl.log

Nthandle v4.22 - Handle viewer
Copyright (C) 1997-2019 Mark Russinovich
Sysinternals - www.sysinternals.com

gpuowl-win.exe     pid: 5488   type: File            98: C:\msys64\home\ken\gpuowl-compile\gpuowl-v6.11-99-gdd8527b\gpuowl.log
gpuowl-win.exe     pid: 5768   type: File            70: C:\msys64\home\ken\gpuowl-compile\v6.11-104-g91ef9a8\rx550\gpuowl.log

 C:\Users\ken\Documents>

And console redirection to a file suffices too, for those applications that don't have built-in logging:

C:\Users\User\My Documents\starfish>handle64 cudapm1.txt

Nthandle v4.22 - Handle viewer
Copyright (C) 1997-2019 Mark Russinovich
Sysinternals - www.sysinternals.com

cmd.exe            pid: 12396  type: File            58: C:\Users\User\Documents\pm1-k4000\cudapm1.txt
CUDAPm1_win64_20130923_CUDA_55.exe pid: 1908   type: File            58: C:\Users\User\Documents\pm1-k4000\cudapm1.txt

C:\Users\User\My Documents\starfish>

I think that's the last piece needed, for a general purpose tool to identify, kill, and restart any GIMPS gpu app that's stalled, on Windows or on linux.