How to run MFAKTC in Linux? - Page 5

Runtime Error · 2020-01-22, 16:05

Hi, unfortunately these give the same error.

Quote:

CUDA version info
binary compiled for CUDA 10.0
CUDA runtime version 4414.89
CUDA driver version 10.10
ERROR: CUDA runtime version must match the CUDA toolkit version used during compile!

Also, I had been under the impression that the CUDA driver version above was from the system over here. However, if I print loaded modules, it says it is cuda/10.0. But that could be me confusing modules for drivers again.

Quote:

Currently Loaded Modules:
1) slurm/18.08 2) xalt/2.7 3) cuda/10.0

Thank you for your efforts, I imagine this is more than you bargained for when you replied to this thread. Thank you!

paulunderwood · 2020-01-22, 17:43

This is a last ditch attempt. These are compiled over ssh to a virtual instance of Centos 6 using the cuda 10.1 toolkit -- note X was not running in the virtual instance and this could be crucial to a successful compile; Without hardware rearrangement -- which I am reluctant to do at the moment -- it maybe insuffucient resources.

Judging from the file sizes they are the same as I posted before, which give a message about runtime (or driver??) having to be the same version as the compiler, when they are the same!!

Runtime Error · 2020-01-22, 18:26

Hi Paul,

Your suspicions are correct. Unfortunately, I'm still getting the same error. You've done a ton of work on this, and I'm pretty amazed by the community here. Thank you. I have reached out to the folks here to see if we can update things. If there is progress, I will let you know. Thanks a million!!!!

Quote:

CUDA version info
binary compiled for CUDA 10.10
CUDA runtime version 4414.65
CUDA driver version 10.10
ERROR: CUDA runtime version must match the CUDA toolkit version used during compile!

Also, here is output from a command you asked me about previously

Quote:

$ lspci | grep NVIDIA
1a:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2 16GB] (rev a1)
1c:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2 16GB] (rev a1)
1d:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2 16GB] (rev a1)
1e:00.0 3D controller: NVIDIA Corporation GV100GL [Tesla V100 SXM2 16GB] (rev a1)

Quote:

$ lspci | grep VGA
05:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller (rev 04)

Or if I send the job to an 80k:

Quote:

$ lspci | grep NVIDIA
06:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)
07:00.0 3D controller: NVIDIA Corporation GK210GL [Tesla K80] (rev a1)

Quote:

$ lspci | grep VGA
0e:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. G200eR2 (rev 01)

paulunderwood · 2020-01-22, 18:49

From this page... please run this command:

Code:

nvidia-smi | grep "Driver Version" | awk '{print $6}'

Curiously, when I run mfaktc on my nvidia card -- installed with nvdia through Debian's apt-get -- I get this information:

Code:

CUDA version info
  binary compiled for CUDA  8.0
  CUDA runtime version      8.0
  CUDA driver version       9.10

So the "runtime versions" you gave are dubious.

I think it would be best if your admin installs a toolkit for you to compile your own mfaktc.

Runtime Error · 2020-01-23, 17:45

Hi Paul,

Unfortunately that command gives me no output (not even an error); it just prints blank. Thank you for your help. I will look into compiling my own in the future. Thanks!

kriesel · 2020-01-23, 20:35

Quote:

Originally Posted by Runtime Error

Hi Paul,

Unfortunately that command gives me no output (not even an error); it just prints blank. Thank you for your help. I will look into compiling my own in the future. Thanks!

Does nvidia-smi by itself run and give any output?
Here's an example output from a recent Google Colaboratory session

Code:

Thu Jan 23 20:40:53 2020       
+-----------------------------------------------------------------------------+ 
| NVIDIA-SMI 440.44       Driver Version: 418.67       CUDA Version: 10.1     | 
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC | 
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. | 
|===============================+======================+======================| 
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |  
| N/A   34C    P8     9W /  70W |      0MiB / 15079MiB |      0%      Default |   
+-------------------------------+----------------------+----------------------+
                                                 
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      | 
|=============================================================================| 
|  No running processes found                                                 | 
+-----------------------------------------------------------------------------+

Runtime Error · 2020-01-24, 15:21

Hi Kriesel,

Unfortunately, nvidia-smi on its own produces the same nothingness. Thank you for the suggestion, though!

kriesel · 2020-01-24, 15:57

Quote:

Originally Posted by Runtime Error

Hi Kriesel,

Unfortunately, nvidia-smi on its own produces the same nothingness. Thank you for the suggestion, though!

It gets grep and awk out of the way and goes for a basic function. I expect a local compile won't help if nvidia-smi won't run for you.
On Windows, inability to run nvidia-smi is indicative of a short list of issues. The ones I can think of at the moment are:
a) a path issue or other command syntax issue. Find the nvidia-smi image on the system and try again with specification of a known-good path/nvidia-smi.
b) Permissions issues
c) Nvidia-smi is not installed on the system, and maybe neither are the rest of the NVIDIA gpu software package (drivers etc.)
d) No NVIDIA gpu hardware installed, which would explain b).

chris2be8 · 2020-01-24, 16:37

Try the following commands to track down why nvidia-smi is saying nothing:
nvidia-smi
echo $?
nvidia-smi -h
which nvidia-smi
The last should produce something like:
/usr/bin/nvidia-smi
Replace that with whatever you get in the following command:
ls -l /usr/bin/nvidia-smi
If that's a synlink repeat on wherever it points to. Run the following on the end of the chain.
file /usr/bin/nvidia-smi

That should give you something to go on. Post output here if you are still stuck. Also try man nvidia-smi to get the manual page for it.

Chris

2020-01-22, 17:43	#46
paulunderwood Sep 2002 Database er0rr 5×937 Posts	This is a last ditch attempt. These are compiled over ssh to a virtual instance of Centos 6 using the cuda 10.1 toolkit -- note X was not running in the virtual instance and this could be crucial to a successful compile; Without hardware rearrangement -- which I am reluctant to do at the moment -- it maybe insuffucient resources. Judging from the file sizes they are the same as I posted before, which give a message about runtime (or driver??) having to be the same version as the compiler, when they are the same!! Last fiddled with by paulunderwood on 2020-01-22 at 19:30

2020-01-22, 18:49	#48
paulunderwood Sep 2002 Database er0rr 5·937 Posts	From this page... please run this command: Code: nvidia-smi \| grep "Driver Version" \| awk '{print $6}' Curiously, when I run mfaktc on my nvidia card -- installed with nvdia through Debian's apt-get -- I get this information: Code: CUDA version info binary compiled for CUDA 8.0 CUDA runtime version 8.0 CUDA driver version 9.10 So the "runtime versions" you gave are dubious. I think it would be best if your admin installs a toolkit for you to compile your own mfaktc. Last fiddled with by paulunderwood on 2020-01-22 at 19:36

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
mfaktc, linux & laptop temperatures	ric	GPU Computing	8	2017-08-31 20:01
mfaktc on Linux and misfit on Windows	bgbeuning	GPU Computing	3	2016-01-25 05:20
mfaktc on a Mac	bayanne	GPU Computing	0	2013-10-18 09:59
mfaktc (0.20)	fairsky	Software	9	2013-09-24 12:58
mfaktc	tichy	GPU Computing	4	2010-12-03 21:51

2020-01-23, 17:45	#49
Runtime Error Sep 2017 USA 2×5×19 Posts	Hi Paul, Unfortunately that command gives me no output (not even an error); it just prints blank. Thank you for your help. I will look into compiling my own in the future. Thanks!

2020-01-24, 15:21	#51
Runtime Error Sep 2017 USA 2·5·19 Posts	Hi Kriesel, Unfortunately, nvidia-smi on its own produces the same nothingness. Thank you for the suggestion, though!

2020-01-24, 16:37	#53
chris2be8 Sep 2009 2⁵·7·11 Posts	Try the following commands to track down why nvidia-smi is saying nothing: nvidia-smi echo $? nvidia-smi -h which nvidia-smi The last should produce something like: /usr/bin/nvidia-smi Replace that with whatever you get in the following command: ls -l /usr/bin/nvidia-smi If that's a synlink repeat on wherever it points to. Run the following on the end of the chain. file /usr/bin/nvidia-smi That should give you something to go on. Post output here if you are still stuck. Also try man nvidia-smi to get the manual page for it. Chris