Msieve with MPI block Lanczos

[R.D. Silverman]

Originally Posted by jasonp

This is with OpenMPI. I don't have the means to get more than one machine working on this at home, so it doesn't matter a great deal which one I use. I can see the handling of signals varying from one set of MPI middleware to the other; maybe I can even configure mpirun to do what I want. The modified LA does explicitly abort after a checkpoint is written, to force the other instances to shut down. (Yes, when I say 'head node' I mean rank 0, the one that handles all tasks besides the matrix multiply)

Greg, do you see a performance difference using fewer MPI process but more than one thread per process?

Bob, are you sure your laptop isn't throttling once the memory controller gets pushed hard enough?

No, I am not sure.

[R.D. Silverman]

Originally Posted by R.D. Silverman

No, I am not sure.

Indeed. How would one find out while the system is up?
I don't know of any tools that would tell me.

[frmky]

Originally Posted by jasonp

Now you can restart an MPI run with a different number of processes without having to rebuild the matrix.

Great! That will help with testing. Rebuilding the matrix each time grew tiresome quickly.

Originally Posted by jasonp

A problem I noticed on my local machine is that mpirun apparently catches signals, so that executing mpirun and hitting Ctrl-C does not make the LA write a checkpoint. Sending SIGINT directly to the msieve 'head node' process (i.e. the one whose working set is largest :) does seem to work however.

I noticed the same here, but rather than log into the correct compute node, I just got into the habit of waiting for a checkpoint update, then SIGKILLing the head node process (SIGTERM wasn't strong enough).

I've started the LA on 11,287+ using 16 MPI processes (2/node by slot) on our cluster. It's a matrix with 9.3M columns and weight 1142 million. It has an ETA of 186 hours. In just over a week, we should have the solution.

[frmky]

Originally Posted by jasonp

Greg, do you see a performance difference using fewer MPI process but more than one thread per process?

I only tried that once, and it crashed. I'm using OpenMPI, but I'm not sure which version. I had lots of other benchmarking to do, so I didn't try to diagnose it.

[R.D. Silverman]

Originally Posted by R.D. Silverman

Indeed. How would one find out while the system is up?
I don't know of any tools that would tell me.

I do know how to add code to the linear algebra code that will tell me
the current clock rate, but that is rather hard to do while the code
is running.

[R.D. Silverman]

Originally Posted by frmky

I've started the LA on 11,287+ using 16 MPI processes (2/node by slot) on our cluster. It's a matrix with 9.3M columns and weight 1142 million. It has an ETA of 186 hours. In just over a week, we should have the solution.

Whereas my laptop is taking more than 4 times as long on 7.9M columns
and weight 542M. Note that my code is only single threaded.

[henryzz]

Originally Posted by R.D. Silverman

I do know how to add code to the linear algebra code that will tell me
the current clock rate, but that is rather hard to do while the code
is running.

CPU-Z will tell the clockrate and should indicate throttling.

[Batalov]

Originally Posted by R.D. Silverman

I do know how to add code to the linear algebra code that will tell me
the current clock rate, but that is rather hard to do while the code
is running.

You may use PC Wizard (also for various temperatures, settings)... maybe TMonitor/perfMonitor is even better -- this is from the same group as CPU-Z.

[R.D. Silverman]

Originally Posted by Batalov

You may use PC Wizard (also for various temperatures, settings)... maybe TMonitor/perfMonitor is even better -- this is from the same group as CPU-Z.

Much obliged. I installed it. My system is not throttling the clock.
It is running at full speed.

[jasonp]

Originally Posted by R.D. Silverman

Whereas my laptop is taking more than 4 times as long on 7.9M columns
and weight 542M. Note that my code is only single threaded.

It could be that nothing is going wrong; multiplying by twice as many nonzeros and about 17% more matrix columns, then dividing by 4 for the efficiency difference between msieve's LA and the CWI LA code (as reported by Paul a while ago), and dividing by 4 to account for the sqrt(N)-like scaling using 16 MPI processes, we should be seeing a runtime difference of a factor of about 6.5x. The fact that you see only 4x slower may be a compliment :)

Do you get dramatically different runtimes from slightly smaller matrices that do fit in 2GB? Maybe windows is playing games with PAE to allow a larger VM space once the working set size exceeds 2GB.

[R.D. Silverman]

Originally Posted by jasonp

It could be that nothing is going wrong; multiplying by twice as many nonzeros and about 17% more matrix columns, then dividing by 4 for the efficiency difference between msieve's LA and the CWI LA code (as reported by Paul a while ago),

I was not aware that the msieve code was 4x faster.
Do you know the cause of the difference?

Maybe I can improve the CWI code.

and dividing by 4 to account for the sqrt(N)-like scaling using 16 MPI processes, we should be seeing a runtime difference of a factor of about 6.5x. The fact that you see only 4x slower may be a compliment :)

Do you get dramatically different runtimes from slightly smaller matrices that do fit in 2GB? Maybe windows is playing games with PAE to allow a larger VM space once the working set size exceeds 2GB.

Yes, I get much faster times for matrices that fit in 2GB. I had a
matrix of 7.1M rows of about the same density that fit in 2G. It took
only 14 days. Theory would predict (7.1/7.9)^2 times ~ 81% as long.