Msieve 1.53 feedback

[VBCurtis]

Alfred-
He is not commenting on your troubles in any way. This is a feedback thread, he's providing feedback that perhaps the default target-density should be higher than 70.

It seems to me the density that leads to shortest project length rises with difficulty. 70 appears best for GNFS up to mid 140s and SNFS up to 200ish. At the risk of asking for too much, perhaps default density could be a function of project difficulty or large-prime size?

On the other hand, a world where defaults "just work" for projects up to a day or two, while larger projects require tweaking such settings for best performance isn't such a bad world.

If default density could be a function of difficulty, perhaps (GNFS size - 65) with a min of 70 and max of 120 would reduce our need to invoke the target-density command line flag.

[pinhodecarlos]

Is there away you could send me the dat file, ini and fb files?

[Alfred]

Originally Posted by pinhodecarlos

Is there away you could send me the dat file, ini and fb files?

Yes, I think there is a way.
I can split the dat file to smaller pieces, say 10,000,000 relations per part.
Then upload (send) the pieces to whatever you want.

In any case, thank you for your great offering.

[pinhodecarlos]

Originally Posted by Alfred

Yes, I think there is a way.
I can split the dat file to smaller pieces, say 10,000,000 relations per part.
Then upload (send) the pieces to whatever you want.

In any case, thank you for your great offering.

Can you make a torrent file?

[Alfred]

Originally Posted by pinhodecarlos

Can you make a torrent file?

No.

msieve succeeded.
I used the previous set of relations, but removed the duplicates and
reordered the set of unique relations (simply using python's sort() function).
This set of unique relations is identical with the previous set.

@VBCurtis
I understand (more or less).
Thank you for your response.

[wreck]

While doing the test postprocessing of 10^343-1 by Kurt,
there is a strange behavior when using msieve_svn988, it
complains "matrix not dense enough, retrying" repeatly.
The svn967 do -nc2 normally.
The command when use svn988 is
miseve -s -i -l -nf -v -t 12 -p

The command when use svn967 is
miseve -s -i -l -nf -v -t 12 -p "target_density=070"

Both postprocessing log and polynomial file are attached.

[jasonp]

This is very strange: for the high-memory singleton removal pass, the 967 log has

Sun Jun 12 21:29:51 2016  begin with 634064226 relations and 599307596 unique ideals
Sun Jun 12 21:50:18 2016  reduce to 347280625 relations and 277146609 ideals in 20 passes

while the 988 log has

Sun Jun 12 01:27:11 2016  begin with 630234936 relations and 341529243 unique ideals
Sun Jun 12 01:27:54 2016  reduce to 433544069 relations and 836522 ideals in 10 passes

This looks like the large dataset bug, the 988 version has only half of the unique ideals that it is supposed to have. Do you get better behavior if you manually remove duplicates from the dataset and re-run the filtering? With the number of relations you have, you should be able to try a larger target density and reduce the matrix solve time.

The smaller number of ideals makes the intermediate files generated by the 988 filtering much larger, and in that case the filtering tries to read the dataset in several passes to keep it in memory. But because you really have many more ideals than that, the partial datasets all fail.

[wreck]

I give some time to investigate the difference of these two postprocessing
logs, the first line not the same is at
commencing in-memory singleton removal
That line does not appear before "keeping xxx ideals with weight <= 200"
at the log988, but it does appear before "keeping xxx ideals with weight
<= 200" at log966.
Then I search the reason why it does not appear at log988.
This line is print at first sentence in function filter_purge_singletons_core.

There are three places call filter_purge_singletons_core(), they are:
1. filter_purge_cliques at line 646 in common/filter/clique.c
2. filter_read_lp_file at line 134 in common/filter/singleton.c
3. filter_read_lp_file at line 243 in common/filter/singleton.c

It is likely that the "filter_read_lp_file at line 134 in common/filter/singleton.c"
should call first.
Before "filter_read_lp_file at line 134 in common/filter/singleton.c", there is one line,
filter_read_lp_file_1pass(obj, filter, 200);

I think the
"keeping 341529243 ideals with weight <= 200, target excess is 299982897"
is print in function filter_read_lp_file_1pass.
Most some other normal postprocessing log does not print this line,
so it is possible there exist some issues in this function.
The function filter_read_lp_file_1pass is seems only called once , and in the problem log,
the sentecnes

	if (i != j) {
		logprintf(obj, "keeping %u ideals with weight <= %u, "
				"target excess is %u\n",
				j, max_ideal_weight, 
				filter->target_excess);

is executed,
maybe it could give some help to print out some sample of

if (counts[i] == 0 || counts[i] > max_ideal_weight)

to see why there i is not equal to j.

In the function get_file_size

		char name_gz[256];
		sprintf(name_gz, "%s.gz", name);

changed to

		char name_gz[256];
		memset(name_gz, 0, sizeof(name_gz));
		sprintf(name_gz, "%s.gz", name);

seems better.
I can not understand why this function ' s return value is

		return (tmp.st_size / 11) * 20;

the number 11 and 20 seems two magic numbers.

That's all I can got after some hours review of the msieve code.

I saw the commands "manually remove duplicates" some years ago but
have forgot it , could you tell me how to do that?

We are going to sieve 700M unique rels, and then test different
target_density to choose the best one.

[jasonp]

The GGNFS tree on sourceforge has useful utilities here.

[pinhodecarlos]

I have a question with regards to the LA phase. Now that UK temperatures are dropping and so far I've been running msieve with 4 threads under LA phase, I was wondering in running LA with 6 threads or even 7. Will I see a gain on it? I've always worked with less than 4 threads due to heat constrain. Just asking although I can stop my current job and try it with 7 threads.

Thanks you.

Carlos

[henryzz]

Originally Posted by pinhodecarlos

I have a question with regards to the LA phase. Now that UK temperatures are dropping and so far I've been running msieve with 4 threads under LA phase, I was wondering in running LA with 6 threads or even 7. Will I see a gain on it? I've always worked with less than 4 threads due to heat constrain. Just asking although I can stop my current job and try it with 7 threads.

Thanks you.

Carlos

As far as I am aware it is up to experimentation. At one point I think it may have been faster to use two mpi instances on a cpu. I don't know whether that is still/ever was true. I don't think it is anymore.
It certainly is cooling down in the UK this week. Unplesantly cold cycling today.