Crash doing large post-processing job

wombatman · 2013-10-22, 04:56

I'm working on post-processing of W_2_736 for NFS@Home. When reading the relations in, there's a host of error -1 and -11, but it ultimately reads in ~124.5M relations. It goes through all of the reduction steps and end up around 9M or so. The crash occurs at the full merge step with exception C00000374, which is a heap corruption error.

This is with MSieve 1.52 SVN 945, which I have used to factor other smaller numbers without issue. I watched the amount of RAM used, and it never rose above ~2.6GB out of 8 GB available, so it shouldn't be a memory issue. I'm currently using an SVN900 exe to see if that works. Any ideas what might be the cause of this crash?

wombatman · 2013-10-22, 16:17

Running with the SVN900 version didn't work either. Running again with SVN945 to try and get an idea of where the error occurs with VS.

jasonp · 2013-10-22, 17:29

2.6GB is suspiciously close to a 3GB virtual address limit, though if a memory allocation failed then you would be notified with a console message. A crash early in the filtering is a sign of a problem that others have reported on larger problems, which I haven't had the time to investigate.

wombatman · 2013-10-22, 17:32

Stupid question perhaps, but does that 3GB limit apply on 64-bit systems? MSieve correctly identifies that there are ~8GB RAM available for use.

wombatman · 2013-10-22, 19:42

The crash is related to ntdll.dll (according to the error given), which is apparently 32-bit in both the system32 and syswow64 directory on my Windows 7. Maybe that has something to do with it as well.

wombatman · 2013-10-22, 21:50

Last bit of information to add: I've tried SVN 923, 945, and 946. In each case, MSieve gets to 1.66GB of RAM being used during the full merge step when it errors out. This is independent of total RAM being used outside of MSieve (anywhere from 1 to 2.5GB).

henryzz · 2013-10-23, 07:28

If you are using a 32-bit dll then my understanding is that the binary must be 32-bit.
Your binary needs to be 64-bit.

jasonp · 2013-10-23, 11:30

Depending on the exact libraries that are linked in when the binary is built, MSVC can still give you a binary with 32-bit limitations even though the binary itself will use 64-bit instructions.

wombatman · 2013-10-23, 13:05

Yeah, biggest problem is there doesn't seem to be a 64-bit version of ntdll.dll on my system (neither system32 nor syswow64 has one), so it may be that MinGW-64 can only link to that 32-bit one. I'll keep investigating and also try a VS-compilation to see if that gives a proper 64-bit version.

wombatman · 2013-10-23, 22:02

Success! Compiling a 64-bit MSieve in Visual Studio has gotten me past the error point and allowed linear algebra to start. The file is attached. It was compiled on a Corei7 processor without ECM or CUDA. It's basically just for doing the -nc step.

swellman · 2013-11-08, 01:13

I've tried running your executable and hit a problem. An error box pops up

"The program can't start because pthreadVC2.dll is missing from your computer. Try reinstalling the program to fix this problem."

A search of my hard drive locates 3 copies of the missing dll, all 89.3 kB, all dated 8/20/2010. Once I copied one of these files to the msieve directory I got a new error message.

"The application was unable to start correctly (0xc000007b). Click OK to close the application."

Is this a registry issue? Any suggestions?

2013-10-22, 04:56	#1
wombatman I moo ablest echo power! May 2013 1741₁₀ Posts	Crash doing large post-processing job I'm working on post-processing of W_2_736 for NFS@Home. When reading the relations in, there's a host of error -1 and -11, but it ultimately reads in ~124.5M relations. It goes through all of the reduction steps and end up around 9M or so. The crash occurs at the full merge step with exception C00000374, which is a heap corruption error. This is with MSieve 1.52 SVN 945, which I have used to factor other smaller numbers without issue. I watched the amount of RAM used, and it never rose above ~2.6GB out of 8 GB available, so it shouldn't be a memory issue. I'm currently using an SVN900 exe to see if that works. Any ideas what might be the cause of this crash?

2013-10-23, 13:05	#9
wombatman I moo ablest echo power! May 2013 1,741 Posts	Yeah, biggest problem is there doesn't seem to be a 64-bit version of ntdll.dll on my system (neither system32 nor syswow64 has one), so it may be that MinGW-64 can only link to that 32-bit one. I'll keep investigating and also try a VS-compilation to see if that gives a proper 64-bit version. Last fiddled with by wombatman on 2013-10-23 at 13:06

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
16e Post Processing Progress	pinhodecarlos	NFS@Home	8	2018-11-28 13:45
NFS@Home Post-Processing Rack Build	pinhodecarlos	NFS@Home	1	2016-09-27 12:34
Post-Processing Fails at Cycle Optimization	wombatman	Msieve	3	2013-10-12 04:51
Update on 7^254+1 post processing	dleclair	NFSNET Discussion	4	2005-04-05 09:51
Post processing for 2,757-	xilman	NFSNET Discussion	3	2003-11-06 14:23

2013-10-22, 16:17	#2
wombatman I moo ablest echo power! May 2013 1,741 Posts	Running with the SVN900 version didn't work either. Running again with SVN945 to try and get an idea of where the error occurs with VS.

2013-10-22, 17:29	#3
jasonp Tribal Bullet Oct 2004 3³·131 Posts	2.6GB is suspiciously close to a 3GB virtual address limit, though if a memory allocation failed then you would be notified with a console message. A crash early in the filtering is a sign of a problem that others have reported on larger problems, which I haven't had the time to investigate.

2013-10-22, 17:32	#4
wombatman I moo ablest echo power! May 2013 3315₈ Posts	Stupid question perhaps, but does that 3GB limit apply on 64-bit systems? MSieve correctly identifies that there are ~8GB RAM available for use.

2013-10-22, 19:42	#5
wombatman I moo ablest echo power! May 2013 1,741 Posts	The crash is related to ntdll.dll (according to the error given), which is apparently 32-bit in both the system32 and syswow64 directory on my Windows 7. Maybe that has something to do with it as well.

2013-10-22, 21:50	#6
wombatman I moo ablest echo power! May 2013 11011001101₂ Posts	Last bit of information to add: I've tried SVN 923, 945, and 946. In each case, MSieve gets to 1.66GB of RAM being used during the full merge step when it errors out. This is independent of total RAM being used outside of MSieve (anywhere from 1 to 2.5GB).

2013-10-23, 07:28	#7
henryzz Just call me Henry "David" Sep 2007 Cambridge (GMT/BST) 5,857 Posts	If you are using a 32-bit dll then my understanding is that the binary must be 32-bit. Your binary needs to be 64-bit.

2013-10-23, 11:30	#8
jasonp Tribal Bullet Oct 2004 3³×131 Posts	Depending on the exact libraries that are linked in when the binary is built, MSVC can still give you a binary with 32-bit limitations even though the binary itself will use 64-bit instructions.

2013-11-08, 01:13	#11
swellman Jun 2012 2⁵·3·31 Posts	I've tried running your executable and hit a problem. An error box pops up "The program can't start because pthreadVC2.dll is missing from your computer. Try reinstalling the program to fix this problem." A search of my hard drive locates 3 copies of the missing dll, all 89.3 kB, all dated 8/20/2010. Once I copied one of these files to the msieve directory I got a new error message. "The application was unable to start correctly (0xc000007b). Click OK to close the application." Is this a registry issue? Any suggestions?