Linear algebra reservations, progress and results

[debrouxl]

Reserved C190_147_34 for you.

You can run remdups4, but you don't need to

[Xyzzy]

We found some of our old notes in our email archive, so we were able to compile msieve easily. We did not include ECM support but we did include zlib support.

Is there a test that we can run to verify our installation?

Thanks!

[fivemack]

If you want a slightly smaller post-processing example you could download http://www.chiark.greenend.org.uk/~twomack/i7test.zip (it's 650MB long, and it contains an msieve.exe file which you should ignore).

Once it starts producing output like

linear algebra completed 3704502 of 13827899 dimensions (26.8%, ETA 60h 2m)

you can be confident it's working, and stop it and go on to the real task.

[Xyzzy]

It works!

[Xyzzy]

PS - We didn't specify a target density for that job. How do we know what target density to specify for future jobs?

Thanks!

[VBCurtis]

Originally Posted by Xyzzy

PS - We didn't specify a target density for that job. How do we know what target density to specify for future jobs?

Thanks!

120 is the usual first target to try for these jobs. If a matrix doesn't build and msieve exits, try 116 or 112 or 110. Rarely, a number will not have enough relations to build at 110; 100 is occasionally required. The experienced users seem to use multiples of 4, but I don't know why (say, 108 instead of 110, 116 instead of 115, etc); perhaps lowering target-density by 4s increases the chance the matrix builds enough to be worth trying the postprocessing run again?

The higher the target density, the smaller and faster the matrix, but more relations are needed. Since relation-gathering is highly parallel while matrix-solving is not, NFS@home usually gathers enough extra relations to make 112 or 120 density work, in order to use the post-processor machines more efficiently. Matrix-solving rises roughly with the square of matrix size, so a 10M by 10M matrix will take about 4 times as long as a 5M by 5M matrix.

On my personal projects, where my CPU time is spent on relation-gathering as well as matrix-solving, I use 90 for 31-bit large-prime projects, and 100 for 32-bit. I haven't tried densities higher than 100 on my own projects yet to see if the time saved in matrix-solving exceeds the time spent gathering the extra relations.

The projects done by the 15e siever at NFS@home sometimes don't fit in 16GB, thus your hardware is somewhat rare in its ability to solve the bigger tasks that folks like me cannot.

[VictordeHolland]

C200_118_77 is in linear algebra and should be ready on Friday.

Reserving C202_127_58 next.

[Xyzzy]

Argument: With 'remdups4', if you have spare memory, use a silly high DIM value.

Data:

$ cat C190_147_34.dat | wc -l
217639878

$ time cat C190_147_34.dat | ./remdups4 1 > /dev/null
Found 215444357 unique, 2195472 duplicate, and 49 bad relations.
Largest dimension used: 20 of 20
Average dimension used: 20.0 of 20
*** Some redundant relations may have been retained (increase DIM)
*** 208890897 (quasi-unique) relations were not hashed

real 3m6.848s; user 2m53.868s; sys 0m26.940s; 55.5m resident memory used

$ time cat C190_147_34.dat | ./remdups4 10 > /dev/null
Found 215444357 unique, 2195472 duplicate, and 49 bad relations.
Largest dimension used: 20 of 20
Average dimension used: 20.0 of 20
*** Some redundant relations may have been retained (increase DIM)
*** 208890897 (quasi-unique) relations were not hashed

real 3m21.408s; user 3m6.720s; sys 0m28.112s; 55.5m resident memory used

$ time cat C190_147_34.dat | ./remdups4 100 > /dev/null
Found 207823761 unique, 9816068 duplicate, and 49 bad relations.
Largest dimension used: 100 of 100
Average dimension used: 100.0 of 100
*** Some redundant relations may have been retained (increase DIM)
*** 175056461 (quasi-unique) relations were not hashed

real 4m13.093s; user 3m55.896s; sys 0m31.168s; 255.5m resident memory used

$ time cat C190_147_34.dat | ./remdups4 1000 > /dev/null
Found 188705028 unique, 28934801 duplicate, and 49 bad relations.
Largest dimension used: 697 of 1000
Average dimension used: 575.9 of 1000

real 5m16.653s; user 4m55.908s; sys 0m36.748s; 2.446g resident memory used

$ time cat C190_147_34.dat | ./remdups4 10000 > /dev/null
Found 188705028 unique, 28934801 duplicate, and 49 bad relations.
Largest dimension used: 697 of 10000
Average dimension used: 575.9 of 10000

real 5m31.164s; user 5m9.708s; sys 0m35.472s; 2.638g resident memory used

[RichD]

General rule of thumb is 5 times the number of 1M relations.

In your example 1080 should be more than adequate.

Edit: Didn't scroll down to see your entire post. It looks like 700 would have been the ideal number.

[chris2be8]

Is there a reason to use cat? Why not:
wc -l C190_147_34.dat
time ./remdups4 1 <C190_147_34.dat > /dev/null

And \time (to run /usr/bin/time instead of the builtin in bash would tell you the number of major and minor page faults as well. Or "\time -v command" for everything time can tell you about resources used by the command.

Chris

[Xyzzy]

Originally Posted by chris2be8

Is there a reason to use cat?

Our mind works from left to right, especially when we work quickly. Fancy redirection stuff is cool but it doesn't "flow" for us.

(Bad habits are hard to quit!)

Originally Posted by chris2be8

And \time (to run /usr/bin/time instead of the builtin in bash would tell you the number of major and minor page faults as well. Or "\time -v command" for everything time can tell you about resources used by the command.

Very interesting!