20091207, 22:00  #1 
May 2008
3·5·73 Posts 
fix integer overflow with deg 4 in sieve_lattice()
In stage1_sieve.c:
Code:
/* structures for storing arithmetic progressions. Rational leading coeffs of NFS polynomials are assumed to be the product of two groups of factors p, each of size at most 32 bits (32 bits is enough for 512bit factorizations), and candidates must satisfy a condition modulo p^2 */ #define MAX_P ((uint64)(1)) But, for degree==4, max p is still 32 bits. Code:
if (degree == 4) { CUDA_TRY(cuModuleGetGlobal(&L.gpu_p_array, NULL, gpu_module64, "pbatch")) done = sieve_lattice_gpu_deg4_64(obj, &L, &sieve_small, &sieve_large, (uint32)small_p_min, (uint32)small_p_max, (uint32)large_p_min, (uint32)large_p_max, gpu_info, gpu_kernel64); } The attached patch does a simple fix for this. 
20091207, 22:01  #2 
May 2008
3·5·73 Posts 
Forgot to mention, this is about msieveGPU only.

20091207, 23:03  #3 
May 2008
3×5×73 Posts 
Maybe it would be better to instead have different MAX_P defined for the different degrees. (uint32)(1) for degree==4 and (uint64)(1) for larger degrees.

20091207, 23:19  #4 
May 2008
3·5·73 Posts 
I like this patch better than the earlier one. See attached.

20091211, 15:44  #5 
Tribal Bullet
Oct 2004
3537_{10} Posts 

Thread Tools  
Similar Threads  
Thread  Thread Starter  Forum  Replies  Last Post 
k*b^n+/c where b is an integer greater than 2 and c is an integer from 1 to b1  jasong  Miscellaneous Math  5  20160424 03:40 
The techniques to guard buffers against overflow  SarK0Y  Programming  0  20141023 21:20 
ubasic: overflow?  Andi47  Programming  7  20090227 00:20 
Always an integer.  mfgoode  Puzzles  18  20070713 18:03 
Integer FFT  nevarcds  Math  4  20040728 19:14 