mersenneforum.org CUDA driver disappeared after patch installation + kernel update
 Register FAQ Search Today's Posts Mark Forums Read

 2012-12-18, 17:54 #1 Graff     Jul 2006 USA (UT-5) via UK (UT) 22·59 Posts CUDA driver disappeared after patch installation + kernel update Unbelievable. I had a running CUDA installation on one of my machine for about two days. Then I installed some patches and a new Linux kernel. After rebooting, mfatkc didn't run. After digging around a bit, I tried nvidia-smi -a and got: FATAL: Module nvidia not found. NVIDIA: failed to load the NVIDIA kernel module. NVIDIA-SMI has failed because it couldn't communicate with NVIDIA driver. Make sure that latest NVIDIA driver is installed and running. !!!! It was there before I installed the new kernel. # dir /dev/nvidiactl dir: cannot access /dev/nvidiactl: No such file or directory I assume I need to reinstall CUDA. Has anyone else experienced this? Gareth
 2012-12-18, 19:11 #2 ckdo     Dec 2007 Cleves, Germany 2×5×53 Posts Happens with every kernel update on Ubuntu.
 2012-12-18, 19:14 #3 Dubslow Basketry That Evening!     "Bunslow the Bold" Jun 2011 40
2012-12-18, 19:54   #4
Graff

Jul 2006
USA (UT-5) via UK (UT)

22×59 Posts

Quote:
 Originally Posted by Dubslow For Ubuntu at least, there's a repository you can add that will auto-update drivers via the package manager. I'll try and find it. Edit: http://www.ubuntuupdates.org/ppa/ubuntu-x-swat Code: sudo add-apt-repository ppa:ubuntu-x-swat/x-updates sudo apt-get update sudo apt-get install nvidia-current
Thanks. That worked. My GPU is functioning again. The installed driver
is now 304.64, which is older than the 310.19 I was running earlier. But
I guess that isn't a problem.
I assume I'll need to do all three commands after each kernel "upgrade"?

Gareth

Last fiddled with by Graff on 2012-12-18 at 20:01

2012-12-18, 20:24   #5
henryzz
Just call me Henry

"David"
Sep 2007
Cambridge (GMT/BST)

585310 Posts

Quote:
 Originally Posted by Graff Thanks. That worked. My GPU is functioning again. The installed driver is now 304.64, which is older than the 310.19 I was running earlier. But I guess that isn't a problem. I assume I'll need to do all three commands after each kernel "upgrade"? Gareth
Just the last one should do.

2012-12-19, 00:25   #6
Dubslow

"Bunslow the Bold"
Jun 2011
40<A<43 -89<O<-88

3·29·83 Posts

Quote:
 Originally Posted by Graff Thanks. That worked. My GPU is functioning again. The installed driver is now 304.64, which is older than the 310.19 I was running earlier. But I guess that isn't a problem. I assume I'll need to do all three commands after each kernel "upgrade"? Gareth
Not quite.

sudo apt-get upgrade will tell your package manager to update all the packages on your system.

If you're in Ubuntu (I'm guessing you are) then the graphical interface is called "Update Manager". Any updates to the kernel, drivers, or anything else, will be handled by the Update Manager. Since the drivers are now managed by the package system, you shouldn't need to do anything at all after any kernel upgrades, since those are also handled by the package manager. Any updates to the driver itself will appear on the list of packages that need updating, whenever the Update Manager pops up like that.

In other (simpler) words, the drivers are now a part of the same system that updates the kernel, and all other installed packages on your system. It will make sure that all packages work after any updates.

2012-12-27, 21:52   #7
Graff

Jul 2006
USA (UT-5) via UK (UT)

22×59 Posts

Quote:
 Originally Posted by Dubslow If you're in Ubuntu (I'm guessing you are) then the
Yes, Ubuntu 12.04 LTS.

Quote:
 Originally Posted by Dubslow In other (simpler) words, the drivers are now a part of the same system that updates the kernel, and all other installed packages on your system. It will make sure that all packages work after any updates.
Thanks for that info, I hope that is the case.

I've just had to restart both of my GPU-equipped machines and upon reboot
mfaktc fails to run on both machines:

Code:
./mfaktc.exe
mfaktc v0.19 (64bit built)
...
CUDA version info
binary compiled for CUDA  4.20
CUDA runtime version      0.0
CUDA driver version       4350.57
ERROR: CUDA runtime version must match the CUDA toolkit version used during compile!
nvidia-smi -a tells me:

Code:
NVIDIA: could not open the device file /dev/nvidiactl (No such file or directory).
NVIDIA-SMI has failed because it couldn't communicate with NVIDIA driver. Make sure that latest NVIDIA driver is installed and running.
Attempting to reinstall the driver tells me I already have the latest
driver. lshw indicates that the GPU card is using the nvidia driver:

Code:
configuration: driver=nvidia latency=0
printenv | grep cuda shows the correct entries in LD_LIBRARY_PATH
and PATH.

So what am I missing? Why isn't my CUDA setup being maintained
across reboots/power cycles? Is this really normal behavior????

Gareth

2012-12-27, 21:58   #8
Graff

Jul 2006
USA (UT-5) via UK (UT)

22×59 Posts

Quote:
 Originally Posted by Graff So what am I missing? Why isn't my CUDA setup being maintained across reboots/power cycles? Is this really normal behavior????
Just tried another reboot. This time ran the nvidia-smi -a command with sudo.
Normal output resulted! Was able to get mfaktc running.
Will now try this on the other machine.

Same thing, no joy until I ran sudo nvidia-smi -a.

Gareth

Last fiddled with by Graff on 2012-12-27 at 22:05 Reason: Tried it on the other machine...

2012-12-27, 22:08   #9
chalsall
If I May

"Chris Halsall"
Sep 2002

952010 Posts

Quote:
 Originally Posted by Graff So what am I missing? Why isn't my CUDA setup being maintained across reboots/power cycles? Is this really normal behavior????

I very recently had a similar situation. I upgraded the kernel on one of my CentOS-64 installations, and suddenly mfaktc failed.

Trying to access the nVidia Unix drivers via lynx and wget failed.

Thankfully I had another system I could use, and a flash drive, on site. So I was able to download the latest driver and run the installation script which compiled the latest driver against the just installed kernel.

This would be funny if it wasn't so sad....

 2012-12-27, 22:10 #10 Dubslow Basketry That Evening!     "Bunslow the Bold" Jun 2011 40
2012-12-28, 03:01   #11
chalsall
If I May

"Chris Halsall"
Sep 2002

224608 Posts

Quote:
 Originally Posted by Dubslow Huh, yes that is really bizzare behavior. Another thing to try is sudo apt-get upgrade, though I'm not sure that would help.
Actually, that's not really bizzare.

The nVidia drivers are proprietary code. So you are suppose to download them yourself each and every time. Then run the script to recompile the driver against your current kernel.

Welcome to freedom... Even though you payed for all the hardware, you still have to jump through hoops to run said hardware using free software....

 Similar Threads Thread Thread Starter Forum Replies Last Post Stargate38 Operation Billion Digits 2 2017-05-08 18:53 Manpowre GPU Computing 43 2013-08-22 12:28 moebius Linux 1 2012-02-20 03:17 10metreh Aliquot Sequences 3 2010-02-15 15:57 mdettweiler Forum Feedback 3 2008-02-21 09:20

All times are UTC. The time now is 07:36.

Sun Apr 11 07:36:16 UTC 2021 up 3 days, 2:17, 1 user, load averages: 1.89, 1.61, 1.50