Crash doing large post-processing job

[wombatman]

Try the prebuilt dll from here: ftp://sourceware.org/pub/pthreads-wi...lease/dll/x64/

If that doesn't work, I actually got another version compiled in MinGW-64 that I'll attach to this thread tomorrow along with the dll that works on mine.

[swellman]

That dll seemed to do the trick! msieve now runs, though it will be a couple of days before I can really shake it out.

Thanks.

[wombatman]

You bet. And if you want a copy of msieve that can read straight from zipped file (i.e., with ZLIB compiled in), let me know. That's the one I put together in MinGW-64.

Seems like my issues were with that one particular relations file. How very strange.

[WraithX]

I'd like one that has ZLIB compiled in. I'm about to do a test run of post-processing a LOT of relations and would rather not uncompress them all. Could you post the exe and any needed dll's?

edit: Also, what svn will this be? Can you make one from the latest msieve svn? (at least 945?)

[wombatman]

I'll have to post it when I get home tonight (flying home from seeing my family for Thanksgiving). I think the SVN is 945 or 946, but I'll see when I get home.

[swellman]

FWIW the current executable has SVN 946 in its title.

Been using it for post processing 31 bit jobs for NFS@Home with great success. But I've recently started running the relations through remdups first to avoid overwhelming msieve with errors. Oddities in the relations data file seems to be an unfortunate side effect of sieving via BOINC.

Just curious - how many relations will you be processing?

[WraithX]

Originally Posted by swellman

FWIW the current executable has SVN 946 in its title.

Been using it for post processing 31 bit jobs for NFS@Home with great success. But I've recently started running the relations through remdups first to avoid overwhelming msieve with errors. Oddities in the relations data file seems to be an unfortunate side effect of sieving via BOINC.

Just curious - how many relations will you be processing?

Good to know about the MSVC svn version.

I wish I could run these rels through remdups (or remdups4) first. But I'm not sure if there is a version that works on windows. Do you know of one?

Also, I'm not sieving via BOINC, I've set up my own php web page that hands out assignments to my sieving machines. Those machines are running a custom python script (largely based on Brian's factmsieve.py) to get work from that web page.

I'm glad you asked me how many relations I'll be processing. In the process of finding out for each gz file, I would run "zcat rel_xyz.gz | wc -l", this actually turned up some gzip errors in some of my archives. The errors were "invalid compressed data--format violated". Not sure why I got the errors, but I may have to go back and re-sieve those ranges. So, I've sieved from 10M-400M. The problem ranges were in 163M, 342M, 343M, 344M. Excluding those, I currently have a total of 352.5M relations (around 22.1GB). (I also have some from 500M-563M, but those are on another machine far away, so I can't include those in this count) However, I'm not sure if this is enough for the C210 that I'm working on. So, I'm going to run a test pass of the post-processing to see if I have enough or if I should finish the 400M-500M range, too.

[swellman]

Originally Posted by WraithX

Good to know about the MSVC svn version.

I wish I could run these rels through remdups (or remdups4) first. But I'm not sure if there is a version that works on windows. Do you know of one?

Try this.

I'm glad you asked me how many relations I'll be processing. In the process of finding out for each gz file, I would run "zcat rel_xyz.gz | wc -l", this actually turned up some gzip errors in some of my archives. The errors were "invalid compressed data--format violated". Not sure why I got the errors, but I may have to go back and re-sieve those ranges. So, I've sieved from 10M-400M. The problem ranges were in 163M, 342M, 343M, 344M. Excluding those, I currently have a total of 352.5M relations (around 22.1GB). (I also have some from 500M-563M, but those are on another machine far away, so I can't include those in this count) However, I'm not sure if this is enough for the C210 that I'm working on. So, I'm going to run a test pass of the post-processing to see if I have enough or if I should finish the 400M-500M range, too.

How many bits is your job? 31 bit jobs require ~235M raw (i.e. pre-filtered) relations, at least in the NFS@Home factorizations I've post processed recently. YMMV of course.

[WraithX]

Originally Posted by swellman

Try this.

How many bits is your job? 31 bit jobs require ~235M raw (i.e. pre-filtered) relations, at least in the NFS@Home factorizations I've post processed recently. YMMV of course.

The post says that is a 32-bit binary. Will that have any problems with a > 22GB file?

I guess you could say this is a 32/33 bit job. Here are my sieving parameters that I'm using with the 16e siever:
rlim: 500000000
alim: 500000000
lpbr: 32
lpba: 33
mfbr: 64
mfba: 96
rlambda: 2.7
alambda: 3.7

[swellman]

Originally Posted by WraithX

The post says that is a 32-bit binary. Will that have any problems with a > 22GB file?

I've run files >25Gb through it with no problems, despite my system being 64-bit Win 7. It only takes ~20 minutes for remdups to chew through files of that size.

I guess you could say this is a 32/33 bit job. Here are my sieving parameters that I'm using with the 16e siever:
rlim: 500000000
alim: 500000000
lpbr: 32
lpba: 33
mfbr: 64
mfba: 96
rlambda: 2.7
alambda: 3.7

It's a good sized job, well outside my experience. I've read about the different factor base technique but I can't remember how many relations you may need. North of 400M? Maybe some of the folks who work on big iron can tell you more.

ETA: there is this thread with a similar sized job.

[WraithX]

Originally Posted by swellman

I've run files >25Gb through it with no problems, despite my system being 64-bit Win 7. It only takes ~20 minutes for remdups to chew through files of that size.

It's a good sized job, well outside my experience. I've read about the different factor base technique but I can't remember how many relations you may need. North of 400M? Maybe some of the folks who work on big iron can tell you more.

ETA: there is this thread with a similar sized job.

Well, I had compiled a version of remdups4 a while back and tried it out today, with:
zcat msieve.dat.gz | remdups4 1500 -v > msieve.dat
This reduced my 356M relations down to 272M unique. So, this works for me on these large files.

I downloaded the msvc msieve svn946 that wombatman provided earlier. When i ran that on the msieve.dat file, it finally came back and told me it needed 1e6 more relations. :( So close! Just kidding, I know I'll need a lot more than 1e6, but I can dream, right?

And thanks for the link to that other factorization job. At worst, I may be only half-way through this factorization. I'm hoping I can get away with collecting only 400M, or 450M, or 500M unique rels. We'll see how it goes as time goes by.