fix integer overflow with deg 4 in sieve_lattice()

[jrk]

In stage1_sieve.c:

/* structures for storing arithmetic progressions. Rational
   leading coeffs of NFS polynomials are assumed to be the 
   product of two groups of factors p, each of size at most 32
   bits (32 bits is enough for 512-bit factorizations), and 
   candidates must satisfy a condition modulo p^2 */

#define MAX_P ((uint64)(-1))

The comment is wrong, since now degree>=5 can use p up to 64 bits.

But, for degree==4, max p is still 32 bits.

		if (degree == 4) {
			CUDA_TRY(cuModuleGetGlobal(&L.gpu_p_array, 
					NULL, gpu_module64, "pbatch"))
			done = sieve_lattice_gpu_deg4_64(obj, &L,
				&sieve_small, &sieve_large,
				(uint32)small_p_min, 
				(uint32)small_p_max,
				(uint32)large_p_min, 
				(uint32)large_p_max,
				gpu_info, gpu_kernel64);
		}

Here on degree==4, the factor p is assumed 32 bits but is bounded by MAX_P in sieve_lattice(), leading to a potential overflow and incorrect bounds being used.

The attached patch does a simple fix for this.

[jrk]

Forgot to mention, this is about msieve-GPU only.

[jrk]

Maybe it would be better to instead have different MAX_P defined for the different degrees. (uint32)(-1) for degree==4 and (uint64)(-1) for larger degrees.

[jrk]

Originally Posted by jrk

Maybe it would be better to instead have different MAX_P defined for the different degrees. (uint32)(-1) for degree==4 and (uint64)(-1) for larger degrees.

I like this patch better than the earlier one. See attached.

[jasonp]

Originally Posted by jrk

I like this patch better than the earlier one. See attached.

Thanks, this is an inconsistency that I missed. Your changes will be in SVN soon (I'm merging the GPU branch into mainline)