Need msieve.dat file?

Greenk12 · 2008-12-31, 11:33

Hello!

Now, i'm working to parallel block lanczos in Msieve (parallel a matrix vector multiplication in Block Lanczos) . But i don't have a matrix that enough big to test my program.

My matrix that i'm using to test my program has size 55000 x 56000 (i have matrix after run msieve with 95digit), but it is small because after i run funtions

mul_MxN_x_Nx64();

it took about 0.3s. It is small, so i can't test my program.

Could you send me a bigger matrix (after run msieve), or send me msieve.dat file ( and number that you use to factoring )?

I can use your msieve.dat file and your number to generate a matrix for me.

Thanks for your reading.

Please, help me.

Ahh i lost my mind.

In Msieve, my matrix that i received is store follow format :
Each of colums in matrix that store value rowth ( position that has value is non zero on a row).

Could you show me how to convert Msieve matrix to follow format:
Each of rows in matrix that store value columth (position that has value is non zero on a colum).

I wrote a functions convert Msieve matrix format into my matrix format. But it is not convenience. Could you show me modify code in Msieve to generate a matrix format that i described ( each of rows in matrix that store value columth)?

henryzz · 2008-12-31, 11:37

is this qs or nfs

Greenk12 · 2008-12-31, 11:45

Quote:

Originally Posted by henryzz

is this qs or nfs

It is mpqs.

jasonp · 2008-12-31, 13:26

See this thread for much more information on the binary format used for msieve matrices. QS and NFS both use the same Lanczos code and in principle can read the same matrix format from disk, though only the NFS code dumps matrices (see gnfs/gf2.c for matrix file IO).

I have the relations from about 10 large NFS jobs that I use for testing. The largest matrix these will generate is about 10 million columns, and the smallest is about 200k columns (the latter is still 275MB of compressed relations). Others here can offer matrices of size almost 20M, and some have high-speed network pipes that could get you such matrices much faster than I could with my puny residential DSL.

Greenk12 · 2008-12-31, 14:05

Quote:

Originally Posted by jasonp

See this thread for much more information on the binary format used for msieve matrices. QS and NFS both use the same Lanczos code and in principle can read the same matrix format from disk, though only the NFS code dumps matrices (see gnfs/gf2.c for matrix file IO).

I have the relations from about 10 large NFS jobs that I use for testing. The largest matrix these will generate is about 10 million columns, and the smallest is about 200k columns (the latter is still 275MB of compressed relations). Others here can offer matrices of size almost 20M, and some have high-speed network pipes that could get you such matrices much faster than I could with my puny residential DSL.

Thank you Jasonp. Could you send me a file matrix 200k columns? I want to check my program with it?

My network is slow. Could you up load your matrix on mediafire.com or rapidshare.com?.

My email is : binhsuse@gmail.com

Thank you very much.

Greenk12 · 2008-12-31, 15:18

Hello Jasonp.

After i read dump_matrix() and read_matrix() functions in gnfs/gf2.c(Msieve), i don't understand how could you read a large matrix into memory on computer.

If my computer has 2GB ram, i can store a matrix that what is size?. If i have larger matrix, could i use your read_matrix() function to load all it into memory on computer?

I'm not good, so please help me.

jasonp · 2008-12-31, 16:21

Can you be more specific about what you don't understand? A machine with 2GB of memory can probably fit 1.3GB worth of matrix file, and the extra vectors that block Lanczos needs would consume another ~200MB at that size. That is a fairly big matrix, perhaps 2-3 million columns worth. Linear algebra on a matrix that size needs a little less than one day if you use multiple threads, so that is a convenient problem size for larger tests.

The largest matrix msieve has tackled needed about 8GB of memory. You can fit that in one machine with the existing code, or modify the code to work across multiple machines, but the aggregate amount of memory needed is about that much. The existing code does assume that all matrix entries will fit in memory on one machine. Many here have discussed the possibility of making the code cluster-aware, but so few msieve users have an actual cluster that other tasks have higher priority.

I can upload my smallest matrix later (the matrix is about a 50MB binary file) but note that a 210k size matrix takes less than 8 minutes to solve on a low-end core2 system, and matrix multiplies at that size are so fast that it actually is not worth using multiple threads.

Greg, can you leave a bigger matrix somewhere at Fullerton like you used to do for me?

Greenk12 · 2008-12-31, 17:44

Thank you Jasonp, i see many problems that before i don't understand.

Could you up load some your matrix file for me?

I want to try it with some different sizes. Then i can realize my problem in my program.

I don't have much experiences in programming, especially parallel programming, so do you know who or what document that i should know to improve my knowledge?

I'm a student, I come from Vietnam, my english is not good but i'm interested in factorizing (i will try explain my idea clearly as much as i can).

So please help me, thank you very much.

jasonp · 2009-01-02, 00:32

Until someone makes a larger file available, here is a link to the matrix generated from my test 100-digit number; the download is 42MB and the unzipped files are about 61MB. The matrix is entirely unremarkable, of size 210k with 91 nonzeros per column on average. Both the matrix file and the underlying cycle file with relation numbers is provided, though you don't really need the latter if you only want to run the linear algebra.

Note that without the 500MB of relations that generated this matrix, an unmodified msieve binary will not be able to run the linear algebra, so you should modify the code to do what you want.

Binh, your english is far better than my Vietnamese :)

Greenk12 · 2009-01-03, 07:57

Hello Jasonp!

I got your matrix file, after unzipped files i have a folder includes:

old [folder]
msieve.dat.cyc
msieve.dat.mat
msieve.fb

i use your read_matrix() functions in gnfs/gf2.c, after i read it, i print it on screen :

Quote:

.....
cols[210719] - weight 6 73579 80722 90934 108479 123624 209012
cols[210720] - weight 6 73579 80722 90934 108479 123624 209012
cols[210721] - weight 6 73579 80722 90934 108479 123624 209012
cols[210722] - weight 6 73579 80722 90934 108479 123624 209012
cols[210723] - weight 6 73579 80722 90934 108479 123624 209012
cols[210724] - weight 6 73579 80722 90934 108479 123624 209012
cols[210725] - weight 6 73579 80722 90934 108479 123624 209012
cols[210726] - weight 6 73579 80722 90934 108479 123624 209012
cols[210727] - weight 6 73579 80722 90934 108479 123624 209012
nrows_out: 210528
ncols_out: 210728
num_dense_rows_out: 84

Is this your data in matrix file you sent it to me? Is it correct?
I wonder why every colum's data are same?

if i use your matrix file and above data, could i run your Block Lanczos?

Data in msieve.dat.cyc and msieve.fb, should i need it to run your Block Lanczos?

This is code i used to print la_col_t *cols:

Quote:

for(i = 0; i < ncols; i++)
{
printf("cols[%d] - weight %d ", i, cols->weight);
for ( k = 0; k < cols->weight; k++)
{
printf(" %d ", cols->data[k]);
}
printf("\n");
}

Thank you for sending file and helping me.

jasonp · 2009-01-03, 16:26

The parameters you read in look correct, but the matrix columns are supposed to have different entries in them. If you have your own code that sets up an array of la_col_t structures, reads the matrix, and then calls the top-level Lanczos routine, then you only need the .mat file.

Good luck with your studies; let us know if you find out anything exciting, and be sure to test your ideas on problems larger than this one.

2008-12-31, 11:33	#1
Greenk12 Oct 2008 14₁₀ Posts	Need msieve.dat file? Hello! Now, i'm working to parallel block lanczos in Msieve (parallel a matrix vector multiplication in Block Lanczos) . But i don't have a matrix that enough big to test my program. My matrix that i'm using to test my program has size 55000 x 56000 (i have matrix after run msieve with 95digit), but it is small because after i run funtions mul_MxN_x_Nx64(); it took about 0.3s. It is small, so i can't test my program. Could you send me a bigger matrix (after run msieve), or send me msieve.dat file ( and number that you use to factoring )? I can use your msieve.dat file and your number to generate a matrix for me. Thanks for your reading. Please, help me. Ahh i lost my mind. In Msieve, my matrix that i received is store follow format : Each of colums in matrix that store value rowth ( position that has value is non zero on a row). Could you show me how to convert Msieve matrix to follow format: Each of rows in matrix that store value columth (position that has value is non zero on a colum). I wrote a functions convert Msieve matrix format into my matrix format. But it is not convenience. Could you show me modify code in Msieve to generate a matrix format that i described ( each of rows in matrix that store value columth)? Last fiddled with by Greenk12 on 2008-12-31 at 11:44

2008-12-31, 16:21	#7
jasonp Tribal Bullet Oct 2004 3,541 Posts	Can you be more specific about what you don't understand? A machine with 2GB of memory can probably fit 1.3GB worth of matrix file, and the extra vectors that block Lanczos needs would consume another ~200MB at that size. That is a fairly big matrix, perhaps 2-3 million columns worth. Linear algebra on a matrix that size needs a little less than one day if you use multiple threads, so that is a convenient problem size for larger tests. The largest matrix msieve has tackled needed about 8GB of memory. You can fit that in one machine with the existing code, or modify the code to work across multiple machines, but the aggregate amount of memory needed is about that much. The existing code does assume that all matrix entries will fit in memory on one machine. Many here have discussed the possibility of making the code cluster-aware, but so few msieve users have an actual cluster that other tasks have higher priority. I can upload my smallest matrix later (the matrix is about a 50MB binary file) but note that a 210k size matrix takes less than 8 minutes to solve on a low-end core2 system, and matrix multiplies at that size are so fast that it actually is not worth using multiple threads. Greg, can you leave a bigger matrix somewhere at Fullerton like you used to do for me? Last fiddled with by jasonp on 2008-12-31 at 16:38

2009-01-02, 00:32	#9
jasonp Tribal Bullet Oct 2004 3,541 Posts	Until someone makes a larger file available, here is a link to the matrix generated from my test 100-digit number; the download is 42MB and the unzipped files are about 61MB. The matrix is entirely unremarkable, of size 210k with 91 nonzeros per column on average. Both the matrix file and the underlying cycle file with relation numbers is provided, though you don't really need the latter if you only want to run the linear algebra. Note that without the 500MB of relations that generated this matrix, an unmodified msieve binary will not be able to run the linear algebra, so you should modify the code to do what you want. Binh, your english is far better than my Vietnamese :) Last fiddled with by jasonp on 2009-01-02 at 04:00

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
P-1 File Name Bug	NBtarheel_33	Software	2	2010-06-06 08:29
Msieve: deleted the log file	Zeta-Flux	Factoring	4	2010-04-20 08:27
Dat File?	Xentar	Sierpinski/Riesel Base 5	10	2008-06-23 02:09
FILE vs. MDB	HiddenWarrior	Software	1	2004-02-19 04:38
How to change Pxxxx.001 file to Pxxxx file?	edorajh	Software	3	2003-12-31 12:30

2008-12-31, 11:37	#2
henryzz Just call me Henry "David" Sep 2007 Cambridge (GMT/BST) 2³·3·5·7² Posts	is this qs or nfs

2008-12-31, 13:26	#4
jasonp Tribal Bullet Oct 2004 3,541 Posts	See this thread for much more information on the binary format used for msieve matrices. QS and NFS both use the same Lanczos code and in principle can read the same matrix format from disk, though only the NFS code dumps matrices (see gnfs/gf2.c for matrix file IO). I have the relations from about 10 large NFS jobs that I use for testing. The largest matrix these will generate is about 10 million columns, and the smallest is about 200k columns (the latter is still 275MB of compressed relations). Others here can offer matrices of size almost 20M, and some have high-speed network pipes that could get you such matrices much faster than I could with my puny residential DSL.

2008-12-31, 15:18	#6
Greenk12 Oct 2008 E₁₆ Posts	Hello Jasonp. After i read dump_matrix() and read_matrix() functions in gnfs/gf2.c(Msieve), i don't understand how could you read a large matrix into memory on computer. If my computer has 2GB ram, i can store a matrix that what is size?. If i have larger matrix, could i use your read_matrix() function to load all it into memory on computer? I'm not good, so please help me.

2008-12-31, 17:44	#8
Greenk12 Oct 2008 2·7 Posts	Thank you Jasonp, i see many problems that before i don't understand. Could you up load some your matrix file for me? I want to try it with some different sizes. Then i can realize my problem in my program. I don't have much experiences in programming, especially parallel programming, so do you know who or what document that i should know to improve my knowledge? I'm a student, I come from Vietnam, my english is not good but i'm interested in factorizing (i will try explain my idea clearly as much as i can). So please help me, thank you very much.

2009-01-03, 16:26	#11
jasonp Tribal Bullet Oct 2004 3,541 Posts	The parameters you read in look correct, but the matrix columns are supposed to have different entries in them. If you have your own code that sets up an array of la_col_t structures, reads the matrix, and then calls the top-level Lanczos routine, then you only need the .mat file. Good luck with your studies; let us know if you find out anything exciting, and be sure to test your ideas on problems larger than this one.