mersenneforum.org  

Go Back   mersenneforum.org > Other Stuff > Archived Projects > NFSNET Discussion

 
 
Thread Tools
Old 2003-11-05, 18:04   #1
xilman
Bamboozled!
 
xilman's Avatar
 
"π’‰Ίπ’ŒŒπ’‡·π’†·π’€­"
May 2003
Down not across

2·3·5·337 Posts
Default Post processing for 2,757-

A brief update on how we are getting on with the factorization of 2,757-, aka M757, since the sieving finished just over three weeks ago.

The raw data came to around 4 gigabytes, so manipulating files of that sort of size was not trivial.

Fifteen days was spent filtering out the useful data from the useless and then boiling down the former. The merging stage reduced what would have been a matrix with 40 million rows and columns to one which has "only" 7,522,873 rows and 7,525,487 columns.

The linear algebra to find linear dependencies in this7.5M matrix began on 28th October and is still going, 8 days later. Progress is mostly steady (we had a few interruptions for various reasons which lost us about 8 hours running in total) and is expected to take another three weeks or so.

The linear algebra alone will take over a cpu-year though this is, of course, only a fraction of the resources the sieving used.

When the linear algebra has finished, it should be a matter of a few hours before we know the factors.


Paul
xilman is offline  
Old 2003-11-05, 20:32   #2
ET_
Banned
 
ET_'s Avatar
 
"Luigi"
Aug 2002
Team Italia

25·149 Posts
Default

Just out of curiosity, may I ask how can you manage data of that size? I mean, what software and resources are needed to handle that huge mass of data?

Luigi

Last fiddled with by ET_ on 2003-11-05 at 20:33
ET_ is offline  
Old 2003-11-06, 12:21   #3
xilman
Bamboozled!
 
xilman's Avatar
 
"π’‰Ίπ’ŒŒπ’‡·π’†·π’€­"
May 2003
Down not across

2·3·5·337 Posts
Default

Quote:
Originally posted by ET_
Just out of curiosity, may I ask how can you manage data of that size? I mean, what software and resources are needed to handle that huge mass of data?

Luigi
The data was brought from Austin to Cambridge by sftp over the open ethernet. Richard has ADSL and 24/7 connectivity; Microsoft Research has adequate capacity ;-) It was transferred in several over-night sessions.

My workstation is a fairly ordinary 2.5GHz box with a 40G disk and a rather larger than average 1G of RAM. It runs XP Pro and so has the NTFS filesystem which supports files much larger than the ones needed here. The 40G disk is a bit limiting and I'll be installing a commodity 160G scratch disk next week. For now, I've been keeping stuff compressed when not needed and I've been dumping files onto other machines for temporary storage.

One stage of the postprocessing needs a large amount of memory, so I used a cluster node. Each node has 2G of RAM and the filter run used 1900M of memory, so it only just fitted.

Summary: the only essentials are a filesystem that supports large files, a few dozen gigabytes of disk and a decent amount of RAM.
Oh, and a degree of patience 8-)

The large memory filter run could have been avoided, but at the cost of greater inefficiency later on, so the entire post-processing up to but excluding the linear algebra could have been performed on a commodity PC upgraded to 1G RAM.


Paul
xilman is offline  
Old 2003-11-06, 14:23   #4
ET_
Banned
 
ET_'s Avatar
 
"Luigi"
Aug 2002
Team Italia

25×149 Posts
Default

:surprised

Thanks Paul!

Luigi
ET_ is offline  
 

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
16e Post Processing Progress pinhodecarlos NFS@Home 8 2018-11-28 13:45
NFS@Home Post-Processing Rack Build pinhodecarlos NFS@Home 1 2016-09-27 12:34
Crash doing large post-processing job wombatman Msieve 22 2013-12-04 01:37
Post-Processing Fails at Cycle Optimization wombatman Msieve 3 2013-10-12 04:51
Update on 7^254+1 post processing dleclair NFSNET Discussion 4 2005-04-05 09:51

All times are UTC. The time now is 13:10.

Wed Oct 21 13:10:16 UTC 2020 up 41 days, 10:21, 1 user, load averages: 1.39, 1.48, 1.54

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.