Memory for Matrix: 2,1538M

[R.D. Silverman]

I have finished building a matrix for 2,1538M, but once again
I am running into memory limitations.....

The matrix fits in 1.8G, but my biggest available machine
only has 2G and the process is thrashing.......

The matrix had 8.56M rows and 8.565M columns with a
weight of 431.7M.

I am going to do some more sieving over the weekend
and hope I can bring it down just a bit smaller.

This is a pain because I had already started sieving 3,517+.
Now I have to switch back. I will also have to rebuild the
matrix; another pain.

[Minty]

Hi Bob,
I'm a big fan of all the work that you and others that post on this forum do. How is 2,1538M going?

I might be able to help out, but it largely depends on the task in hand along with my hardware suitablility. I have a Vista (32-bit) Core 2 Quad with 3GB DDR2 memory (dual channel mode), plenty of free HD space and broadband. May I ask a few questions:

1) What software do you use to solve the LA (do you have vista capable binaries)?
2) Is 3GB enough to accomplish the task?
3) Is this fire and forget, or does the software require parameter tweaking based on observed outputs (i.e. can a relative novice run it?)
4) Can the analysis be paused/continued, or is 1 complete run desirable?
5) And now the killer - approximately how long do you think this might/could take on a single core @2.4GHz - are we talking possibly 12-24 hours, or 2-3 weeks continuous (it's probably difficult to say - length of string and all, but I'm more interested if it's a night's processing rather than several days)

Best wishes,
Minty

[R.D. Silverman]

Originally Posted by Minty

Hi Bob,
I'm a big fan of all the work that you and others that post on this forum do. How is 2,1538M going?

Huh? My previous post gave its exact status...........

I generated another 4M relations over the weekend
and am rebuilding the matrix now. I expect that it will fit
easily within 2Gb and will take ~3 weeks to solve. (single threaded)

[Minty]

I was of course enquiring as to how the weekend work had gone to gauge whether you needed a PC with >2GB. Sounds like you've got everything under control - at 3 weeks this is certainly an undertaking if only in electricity costs.
I wish you all the best!

[R.D. Silverman]

Originally Posted by R.D. Silverman

I have finished building a matrix for 2,1538M, but once again
I am running into memory limitations.....

The matrix fits in 1.8G, but my biggest available machine
only has 2G and the process is thrashing.......

The matrix had 8.56M rows and 8.565M columns with a
weight of 431.7M.

I am going to do some more sieving over the weekend
and hope I can bring it down just a bit smaller.

This is a pain because I had already started sieving 3,517+.
Now I have to switch back. I will also have to rebuild the
matrix; another pain.

I rebuilt the matrix after more sieving. The matrix now has 7.8million
but the weight increased to 446.8 million. I Also built a slightly
larger matrix with similar weight.

The code reads in the matrix without problem. It requires
1.78Gbytes of real memory. The code then attempts to malloc
some auxiliary vectors and the malloc FAILS ---> the code will
not run.

[xilman]

Originally Posted by R.D. Silverman

I rebuilt the matrix after more sieving. The matrix now has 7.8million
but the weight increased to 446.8 million. I Also built a slightly
larger matrix with similar weight.

The code reads in the matrix without problem. It requires
1.78Gbytes of real memory. The code then attempts to malloc
some auxiliary vectors and the malloc FAILS ---> the code will
not run.

Have you rebooted with the /3gb switch and/or in safe mode?

Windows puts the system DLLs near the top of the address space. It's possible for malloc() to fail because of the fragmented virtual address space even though there's enough physical memory to hold everything.

Paul

[R.D. Silverman]

Originally Posted by xilman

Have you rebooted with the /3gb switch and/or in safe mode?

Windows puts the system DLLs near the top of the address space. It's possible for malloc() to fail because of the fragmented virtual address space even though there's enough physical memory to hold everything.

Paul

Owing to managerial restrictions, I am not *allowed* to modify
the OS to accomodate the /3GB switch. IT insists that it can't
maintain systems unless everyone has the same configuration......

[xilman]

Originally Posted by R.D. Silverman

Owing to managerial restrictions, I am not *allowed* to modify
the OS to accomodate the /3GB switch. IT insists that it can't
maintain systems unless everyone has the same configuration......

In that case, you may be screwed unless running in safe mode gives you just enough space to continue.

My earlier offer still stands.

Paul
P.S. Having worked in IT support for many years, I have some sympathy with your IT people...

[R.D. Silverman]

Originally Posted by xilman

In that case, you may be screwed unless running in safe mode gives you just enough space to continue.

My earlier offer still stands.

Paul
P.S. Having worked in IT support for many years, I have some sympathy with your IT people...

Hi,

I will send you the final matrix and polynomial file via snail-mail...
Remind me (via private email) of your snal mail address.

Bob