double check server online
 

The DC server is now online for the public.
After a long testing phase with the help of Joe_O and VJS I decided to make the DC server public.

The purpose of the DC server is to check if we have problems with some ranges or machines or maybe even a problem with heat in the summer that gives us bad results.

To do this at the moment the server will hand out data from one k and will try to bring this k to the level of the first pass.

If we find a pattern for an error the server will be switched to make detailed tests in that range.

So you can expect that the size of the tests will change dramatically sometimes. For example the effort is at 2.19M at the moment but within the next 200 tests there will be another set of small tests around 1.6-1.8M.

If we find no interesting pattern the next step for the DC server will be to bring all k up to n=2M and see if that brings us more information.

To make it clear. We do not expect that we have a missed prime at the moment!!!!!
So the first priority for testing is still sieving (manual and BOINC based) and first pass PRP tests via llrnet, BOINC and manual reservation.

As the tests on the DC server are significant smaller at the moment this might be an oportunity for older machines.

For the stats freaks. The point value of a second pass test is identical to the value of the coresponding first pass test.

So now for the important part if you still want to help.
To run DC tests you need a clean copy of llrnet. You can do this by either downloading a new one or if you want you can also make a copy of your local llrnet installation. Attention you must run the DC effort in a seperate folder!!!
When you want to use a local copy make sure that you delete the following files from the folder: workfile.txt, tosend.txt, zXXXXX

Now open llr-clientconfig.txt and change/check the following entries:

server = "pspnet.no-ip.info"
port = 13984

-- username
username = "your username"

The only difference to the normal PSP server is the entry for port.
If you have problems with the DNS (which happens on some linux distributions) you can try the following:
server = "62.75.240.56"
At the moment i can not check if that really helps with the latest distros but it has in the past.


There are also online stats available for the DC effort:

[url]http://www.psp-project.de/llrnetstatsdc.php[/url]

FYI,

Most of that double checking was done using a heavily overclock q6600 which produced quite a few errors earlier. This was later sorted out with the help of Lars constantly sending me results back and forth.

With one core on a q6600 your going to produce at least 10 results per day.

It wouldn't take much to take the 1st place crown away from me at this point.

Second place is also a friends q6600 that he ran for a few weeks all four cores.

Stats near the bottom of this page.

[url]http://www.psp-project.de/stats.html[/url]

server = "62.75.240.56"
port = 13984
username = "IronBits"
One core of a Q6600, stock, working here now. :)

Small update on the DC progress.

After going public we have tested 110 pairs and within these pairs
we have a very good error rate.

We found 20 tests that needed a third test but i found out that
all of them had incompatilbe results due to usage of the old client
for the first test.

So within the first 110 tests we had [COLOR="SeaGreen"][B]NO[/B][/COLOR] real error.

Update on the DC front.
June again brought a set of false positives due to usage of the older client.

There were no other errors so no reason to change the search strategy.

Beginning of this month there is a set of errors that needs some more investigation.

Thanks to everybody who did help to create those results.

I have activated a set of around 850 tests in the range from 2M to 2.6M which look like having an error pattern. I do not expect many errors but better to remove all these tests now.

When we continue with k=168451 it is possible that we will find out that the error is still there at higher ranges.

We left the first zone which contained an error pattern. When cleaning up we did not find a prime but the high number of wrong results show that it was a good decission to retest the suspicious results. Thanks to all who did help.

In the next stage we will let our main dc test "k" run to 3.0M and look if the error continues and if so make another cleanup run.

In the background I am checking two other smaller possible error pattern but so far it is not clear if it is again a false positive or not.

One last question:

Do you like to be informed about the internal work this way or is it not so interesting and I can stop posting about it? Or what additional information would you like to see?

[quote=ltd;138167]One last question:

Do you like to be informed about the internal work this way or is it not so interisting and I can stop posting about it? Or what additional information would you like to see?[/quote]
Well, I for one find internal info like this quite interesting. :smile: It's always nice for users to know what's going on behind the scenes.

[quote=Anonymous;138170]It's always nice for users to know what's going on behind the scenes.[/quote]+1

As you all are so fast running the DC server I had to change my mind in one point. The result from the last day show that the error pattern stops around 2.62M so i decided to make clean up run immediately.

To make sure that we run all the problematic tests I have activated tests up to 2.64M.

Once more thanks to everybody for the continued contribution.

The next regular update.

After we left the region with the high error rate due to tests comming from one user there was another set of tests that needed a closer look.

But this time it was a false positive. There was only a set of tests that was done with the older client which had incompatible residues. The rest of the tests in this group will be double checked with the older client.

If we do not count the groups mentioned above our error rate is still less then 2%.

It has been a very long time since the last update.
I wanted to share with all of you the error rates found within the testrange for k=168451 and n>3M and n<3.95M (with some small gaps)

After cleaning up false positives (That is pairs that are done using clients that give residues that are not comparable) we have the following numbers.

Within 1161 tests we had 6 mismatches without good reason so far. That means that we have an errorrate of only 0.52% for that range!!!!!!!!!

Great news! Thanks for the update! :)

LTD,

That is really exceptional news :smile:
Almost too good to believe :surprised:

A recent post in Mersenne in another section showed error rates of 3-5% depending on the n.

Note the following:
- one error any time during the test will create a residue mismatch.
- Mersenne tests are much longer therefore much more suseptable to errors

I still maintain that we should continue with the doublechecking of one k up to the firstpass level.

Major reason is we did find some errors and that error rate should acutally increase with n. Lets not get caught behind the 8-ball when it comes to error testing.


Great news is that from your calculations we do not have to increase the entire secondpass level beyond the 1.5M that its currently at.

I will spend more of my CPU resources on the one k.

Opyrt, and others, Lars sent me a file just a few minutes ago. We will see how much of a dent I can put into the k from 5.5M up. So hopefully once secondpass reaches around 5.5M we will have another large jump.

[QUOTE]I still maintain that we should continue with the doublechecking of one k up to the firstpass level.

Major reason is we did find some errors and that error rate should acutally increase with n. Lets not get caught behind the 8-ball when it comes to error testing.[/QUOTE]

That is then reason why I stated somewhere else that I am very interested to see the developement of the error rate above 4M and up to first pass. Depending on the real numbers we can then decide how to procede.

Depending on the results I think there will be at least four differnt ways to react:

1. Error rate stays very low: procede with the "one K" solution until the error rate increases
2. "Higher" error rate which has an error pattern. DC all pairs with the same pattern and then fall back to "one k"
3. higher error rate without any signifcant pattern.
I see two ways to react here:
a. Start a complete DC in the region the error rate increased.
b. Start a complete DC for all pairs.

Due to the high error rate from 2 user accounts I decided to add all results from these users to the dc queue. We will have an additional set of 260 tests in the range n>3.7M and n<4.15M. The chances that one of these tests contains a prime are minimal but better sure the sorry.

I want to invite everybody with a spare system to help in a short sprint to clear these tests. With an additional 10 Cores running 24/7 we should be able to finish these tests in around 10 days.

Lars,

Personally I think its great that you can pick these things out with the DC effort.

Good job Lars! :-)

I see the first of the tests have already been handed out from the server.
Do you inform the users with the invalid tests? I personally would like to know if my computsers were producing garbage @100% CPU 24/7. :-)

If the user is still active I send a mail informing about the problem but in this case both users are not active anymore and the results are around a year old. So I decided not to contact them.

About puting resources to DC testing:

My i7 is now up and runing. 4 virtual cores run DC on the llrnet server. The other 4 virtual cores run different projects which use more RAM to see if that is also stable. With this configuration I get 14.x ms per iteration.
When I switch off the additional task to have only the 4 cores work on llr without using the HT I am down to 10ms per iteration. When the real CPU cooler arives I will see if some overclocking is possible.

Will we be granted double check credits for these tests? Even though they are actually triple check? :-)

Not that it's important, I'm just curious.

All test in the DC queue give DC credits.

News from our extra DC run. No prime so far but it is good that we redue the ranges. Specialy the results from one of the users are worse then I expected. There are between 20-30% errors from this machine!!!!

[quote=ltd;163494]News from our extra DC run. No prime so far but it is good that we redue the ranges. Specialy the results from one of the users are worse then I expected. There are between 20-30% errors from this machine!!!![/quote]

20-30% is alot. This is one of the reasons I would like to see our second pass n closer to the first pass n. Then we could have found this while the user was actually crunching and told him/her that something was wrong.
Depending on a few things, I might be able to focus more on second pass this year. :-)

A possibly silly question that might already have been answered: What happens if I get a WU from the second pass server which contains a kn-pair that I was the one who crunched when it was first pass? Is it valid or discarded?

Also: Thank you for keeping us updated! I really appreciate your reports on how things are going "behind the scenes". :-)

Your result is valid if you received it from the second pass server.

DC server is handing out k=168451 work again.
Here some preliminary results from our special run with around 20 results still open.

We had to do 272 suspicious tests.
From these tests 76 gave wrong results.
So far I ran a third test for 37 of these tests and every time the DC result was the correct one.

Thanks to everybody for helping.
By the way at the moment we have to run another 1900 tests before we catch up with first pass testing. That does not include VJS manual reservation.

Another update:

We are still removing candidates faster then new ones are created.
When you look here:
[url]http://www.psp-project.de/llrnetstatsdc.php[/url]
and see that the number of pairs in work queue stays below 1400 that means that there are no candidates left to fill the queue anymore.

Second thing. I am not contributing to the llrnet DC effort at the moment due to the fact that I am doing some tests on a suspicious account. It produced an error rate of around 10%. As I also need some work for my offline clients I decided to do all the work on my own and not to feed the tests into the normal queue.

Thanks for the update.

I'll take a break from double checking now. Will be back later. :-)

arrgh...

just when we were holding gound :no:

we would really like to get this to within a few hundred tests.

I understand that other things take precedence at times so I thank you very much for your help.

Personally still cruching away at the DC que and a couple other 100 results offline.

[quote=VJS;167503]arrgh...

just when we were holding gound :no:

we would really like to get this to within a few hundred tests.

I understand that other things take precedence at times so I thank you very much for your help.

Personally still cruching away at the DC que and a couple other 100 results offline.[/quote]

I'll try to explain...

The reason I first started participating in this project was only to give my team a boost on the DC-Vault ladder. The only credits that count towards DC-Vault are first pass credits.

Then I started to find this project interresting, so I decided to help the project with doing second pass. But I also really want to find a prime, so I will probably be on and off second pass in the future. I will be running only FP when I'm feeling greedy, and throw some cores on SP when I'm feeling guilty for not helping out... ;-)

However... I will be back on SP, I promise. :-)

I know and thanks.

There was some discussion in the past about vaulting secondpass...

I think a few questions needed to be answered.

One was will the points ever be combined and I think the answer to this is a strong NO.

The others were stats related but I think everything is now the same for first and secondpass.

Anyways vaulting might give secondpass a good push.

In any case thanks and don't feel guilty I appreciated all your help, just a little lonely here... LOL [url]http://www.psp-project.de/llrnetstatsdc.php[/url]

[quote=VJS;167870]One was will the points ever be combined and I think the answer to this is a strong NO.[/quote]

Why is that? :-)

They seem to be using the same score calculation, and there can never be too much focus on second pass as the only thing that will happen is that the queue empties (for the k we are checking) and people will have to make sure to switch to FP.

The only reason I go for a strong NO is that I think ARS will regain the first position... ;-)

Edit: I don't mean that these statistics should not exist by themselves. I was thinking that there is one for FP, one for SP and one combined.

I can't remeber the exact reason but if they were going to be combined why are they not currently combined.

Perhaps we could have someone make a decision for the project and we will stick with that.

Although I do like your suggestion of a combined score... keeping track of # of tests seperate yet a total score (points) reported in one place.

I think this is something Lars and the mods should come up with.

Hi,

there are three reasons why there are no combined stats at the moment.

1. I have to create some query that combines the stats from the different data sources or write a script that does it and I have no good idea.
2. When I asked if there is interest to have combined stats there was no positive reaction. (two or three years ago)
3. I am lazy.

[QUOTE=opyrt;167887]
The only reason I go for a strong NO is that I think ARS will regain the first position... ;-) [/QUOTE]

Not to Worry! Team Norway would retain the first posititon.

In fact, the first 23 teams would retain their positions. Only 3 teams would move positions:
Grid Spike and the The Belgian Crunchers would exchange positions and Ultimate Chaos would drop down one position to make room for BOINC.BE which only has second pass points.

Can't someone keep VJS company? He is the only one that has returned second pass work so far this month: [url]http://www.psp-project.de/llrnetstatsdc.php[/url]

Actually from a points stand point, now would be the time to combine the two scores.

Norway would still be in the lead by about 1/4 million points.

As joe pointed out there would be little other changes to the vault scoring.

Those teams affected wouldn't really need much additonal effort to regain their lost rank either.

My major concern for keeping the stats seperate is not score related.

As long as we could keep track of the tests done in each of the firstpass and secondpass (which is more important) and keep those number of tests done seperate that would be great.

As far as scoring and points awarded those could be combined. They already are in SoB for example, a project that closely matches this one.

[COLOR="Red"][SIZE="7"]Attention!!!!![/SIZE][/COLOR]

The double check DC llrnet server has been taken offline.
The reason for that is that it has been replaced by a new prpnet server.
There are no connect informations at the moment as the setup is tested in a closed beta at the moment. There is still some work to be done before the statistics transfer works again and I have to check the automatic restart and .........

I keep you informed about the progress.

[quote=VJS;167503]I understand that other things take precedence at times so I thank you very much for your help.

Personally still cruching away at the DC que and a couple other 100 results offline.[/quote]

As soon as ltd find the time to make me a DC reservation (I've sent him a pm), I'll start DC-testing again. Hopefully we'll be able to catch up this time! :smile:

On monday I did send a range to you .
I used the mail account that you put into your PM.
If that does not reach you please give me an information and I will send it again.

One more info about the dc server.
So far the tests run without any problems. But there are some more things to be done before we can go public.

1. I have to update the server to the latest version
2. For internal reason I have to change the installation path
3. The result transfer from ther server to the main DB does not work correctly at the moment. Has to work before we go public.

When this is done the next step will be a first pass server and then begin to phase out the old llrnet server in the next two or three months.

I can not promise to fix the three main points this weekend as on friday I will go to Frankfurt to see the BOSS live again.:jokedrum::beer::party:

Who knows how long it takes to regenerate from such a concert?:smile::grin:

[quote=ltd;179370]On monday I did send a range to you .
I used the mail account that you put into your PM.[/quote]

My bad. The google apps spam filter decided to grab your email. Found it now though, thanks alot! :)

[quote=ltd;179372]I can not promise to fix the three main points this weekend as on friday I will go to Frankfurt to see the BOSS live again.
Who knows how long it takes to regenerate from such a concert?:smile::grin:[/quote]

I have several friends who have seen him on this tour, and they won't stop talking about it! :smile:
He also got a great review in Q Magazine.

Data transfer script runs at least in a manual mode. But I do not like the code.
I still have to decide if I will stay with it or build something better. Problem is the real life. In the next nearly two months I will be several times out of town for my job. So no promises about the progress.

The double check llrnet status page ([URL]http://psp-project.de/llrnetstatsdc.php[/URL]) says there are (currently) 912 pairs in queue. Is that number equal to what we need to process to catch up with first pass? :)

Sorry for the late answer. The link you rever to is an artifact from the old llrnet server but it is still at least telling half of the story.
The number you see there is the amount assigned to the prpnet server. In the future I will have to create a new statistics page as the feeding of the server is different.

The total amount of open tests is larger due to the fact that there are also two manual ranges assigned to different users. I think you know one of them. :wink:

So the total amount is 1491.

Thanks for your answer. Hopefully we will be able to reduce the number of outstanding second pass tests a bit this year. :)

Hey Lars,

How are things going on the DC front?

Update to the latest release is finished.
Changing of the paths is finished.
Data transfer from the server to the main DB is also working.

Next step is to create some download package with the client software like it is available at primegrid. (I have allready asked and they allow me to grap their package and modify it for our use)

If somebody allready has the prpnet client and wants to help you only have to modify the following entries in the prpclient.ini
[CODE]
email=your@mail

// userid= is a REQUIRED field that will be used by the server
// to report on stats, etc. without having to reveal the user's
// e-mail address.
userid=place the PSP nickname here

// This value differtiates clients using the same e-mail ID
clientid=1

server=PSPtestDC:100:1:[url]www.psp-project.de:7101[/url]
[/CODE]

By the way if everything works as I expect there will be a prpnet server for the first pass available this weekend!!!!!!!! I only have to wait for the first result to be returned and check if everything is booked as expected.

[quote=VJS;183022]How are things going on the DC front?[/quote]

Also, I've added 8-10 cores that are running offline DC tests. They have mostly done clean-up ranges so far.

Just a thought... Could it be an idea to reverse the queue on the double check prpnet server so that it offers the newest tests first? Then we would be able to notify participants that are producing garbage a lot earlier...

Interesting idea but the server does not offer a posibility to revers the order at the moment.

opyrt,

Yup good idea in theory but consider the following.

The lower the n the more likely its prime, the higher the n the more likely there is an error. Not sure if these too are a wash or not...

In anycase the best senario is to get the one k approach to the firstpass level.

With one double check k at firstpass level it will ultimately accomplish what your talking about.

I think we will all be supprised once Lars has all of the double check stats on-line. I have been pounding away with somewhat of a top down approach like you talked about with 8 cores, also there is a chunk in there that's already complete...

I'm going to send in another 30 or so tests, at that point (time permitting) perhaps lars can make a comment about what's left. In the mean time please put as much effort as you can into getting this mini one k project accomplished.

Jason

[quote=VJS;183542]The lower the n the more likely its prime, the higher the n the more likely there is an error. Not sure if these too are a wash or not...

In anycase the best senario is to get the one k approach to the firstpass level.

With one double check k at firstpass level it will ultimately accomplish what your talking about.[/quote]

Agreed. I did not mean that we should step away from the one k strategy. I meant we could revert the queue so that we don't end up with alot of candidates to rerun because someone was hammering away with an overclocked computer. I'm currently re-running 130 tests from one user that mostly produced garbage...
That said... We would still have to finish the gap as you said. :)

[quote]I think we will all be supprised once Lars has all of the double check stats on-line. I have been pounding away with somewhat of a top down approach like you talked about with 8 cores, also there is a chunk in there that's already complete...[/quote]

I did not know that. Well as long as both you and I are double checking fresh results (n>7M) we should be able to see if there are any rotten eggs quite quickly! :)

-Kai

I am looking forward to getting the prpnet servers up. Once that happens I can shut down the one I have here and get my machines back to the daily stats. Prpnet should allow to easily split time between first and second pass with a single system or setting some up for first and others second.


S.

I don't think I did too much or anything above 7M, I worked from 7M down. SOunds like Lars is sending you what I find/have found.

I don't really even know where I currently am on it have a few runnig around 5.8M some 6.3M. With splitting stuff up etc etc etc, all I know is I'm finally getting close to done. I have another 40 now to send to Lars.

He seems pretty busy lately so I havn't bothered him much on sending another batch or updating what I have (less factors etc).

My main goal right now is bring that k up to firstpass.

Jason

Another update to what is going on at DC.

First of all the number of open test for k=168451 till we are at first pass level has droped to 1443 ( 20.7 it was 1491)

Second the following work is in progress:
1. Second pass server is again online to the public (k=168451)
2. there is manual work assigned to VJS and opyrt (k=168451)
3. Joe O and ltd run third pass tests on data where the second pass did not match
4. opyrt runs second pass tests for data where the third pass tests have shown a error pattern
5. I am bringing the overall dc level to 1.7M at the moment and depending on the other dc results I will either run DC tests on error pattern or increase the lower bound to 2M.

A very ineteresting result of the DC tests is that we had several bad result sets in the range between 4m and 5M but so far from 5.1M to 5.5M we are back to extrem low error rates.

has droped to 1443 ( 20.7 it was 1491) :sad:

I though we were closer than than...

As far as the pattern goes around 4-5M any ideas?
A prime would certainly help out the lower limits...

[QUOTE=VJS;183721]has droped to 1443 ( 20.7 it was 1491) :sad:

I though we were closer than than...

As far as the pattern goes around 4-5M any ideas?
A prime would certainly help out the lower limits...[/QUOTE]

To get the drop in perspective I have to add the following number We drop to this point even with the first pass adding 62 new tests to the queue.

For the 4-5M range we had at least two user with very bad results and one where we make triple checks at the moment to find out more about the quality of the tests.

For one of the users already all his tests have been redone for the second we have 130 tests left and for the third one this would mean at first 140 tests below 5M and if we later find that we should redo everything another 640 tests above 5.4M. But for now I hope it will only be the low range as this results seem to be from manual reservations while most of the high range seems to be done by llrnet which typicaly means different PCs.

I've done alot of quite new (n>7M1) second pass tests lately, and good news everyone! None of them proved any errors!

Keep up the good work!

THats great!!!

Lars sent me some more results I looked at waht i've been doing lately and tried to figure out whats left of my group.

THe simple answer is :censored: I think I'm just going to send in whatI have and start cruching a new range.

I'm doing some stuff twice (tripples) and not sure if I've done others, ARRGH!!!

Anyways god news is we are past due for a prime. NO I dno't know anything but it's the season. Here is hoping we find something soon.

[quote=VJS;185942]Lars sent me some more results I looked at waht i've been doing lately and tried to figure out whats left of my group.

THe simple answer is :censored: I think I'm just going to send in whatI have and start cruching a new range.

I'm doing some stuff twice (tripples) and not sure if I've done others, ARRGH!!![/quote]

I know the feeling! At first you have a couple of ranges running, and everything is fine. Then they start to run out, so you get two new ones. And then it's already confusing! :D

[quote]Anyways god news is we are past due for a prime. NO I dno't know anything but it's the season. Here is hoping we find something soon.[/quote]

Hear, hear! I feel we deserve a prime now. We've put alot of effort into this project the last year. :)

Has anyone tried offering something to the ancient prime gods? ;-)

[quote]Has anyone tried offering something to the ancient prime gods? ;-) [/quote]

Do you mean besides the ritualistic animal sacrifices every second Tuesday of the month?
:deadhorse:

Other than that and my first born, No.

On a good note, I noticed we took a 1M increase in the secondpass que.

Currently crunching numbers around 6.6M in the dcllrque.

ltd: How many second pass candidates are pending now?

VJS: Looking at the daily first pass stats, I see a lot of people sabotaging our second pass effort!
:raman: :grin:

Before I imported the huge sets of results today we were down to 1236 but now we a back again to 1356.

[quote=ltd;187965]Before I imported the huge sets of results today we were down to 1236 but now we a back again to 1356.[/quote]

Well, atleast we're moving in the right direction. And my two second pass ranges are way past half ways. :)

Another update from the DC front.

The good news is that the error rate is still surprisingly low. Congratulation to all contributors you have very stable equipment.

Due to the fact that lots of people have joined into first pass the catch up process is very slow. We have 1198 open second pass tests for k=168451 at the moment and there are no more open tests from systems with bad results.

[quote=ltd;191873]Due to the fact that lots of people have joined into first pass the catch up process is very slow.[/quote]

I'm pretty sure that most of these have joined because of the DC-Vault. Currently, only first pass score is counted towards the DC-Vault rank. If you want more focus on second pass, I can ask the DC-Vault stats master to combine the first pass and second pass scores. Just let me know. :smile:

No let us stay as we are. First pass is significant more important then dc as the chances to find a prime are much higher.

Thanks to several ranges with manual reservation and also lots of work done on the prpnet server we are down to 1022 tests now.

We are closing in on first pass but it is still a very long way to go.

Another update about the number of open tests.

The good news we have only 804 tests open for the k=168451 range.

The bad news we have the need to retest 200 tests from one user with some problem within some ranges. It might be that 140 tests would be sufficient but the start and endpoints of the bad ranges are a little bit unclear so we will have to run the whole set.

Correction: After writing my post I received another big resultset from a manual tester (thanks opyrt:tu:) and now we are down to 750 tests for k=168451.

[QUOTE=ltd;194557]The good news we have only 804 tests open for the k=168451 range.

The bad news we have the need to retest 200 tests from one user with some problem within some ranges.[/QUOTE]

I'll say it again Lars, the good news is that you found a user requiring a 200 test retest.

Yeah it sucks that there are errors but proves the method, who knows could be a prime in there.

[quote=VJS;194812]I'll say it again Lars, the good news is that you found a user requiring a 200 test retest.[/quote]

The really good news is that the user still had the logfiles, so we'll only have to retest the candidates from the faulty computer, not all tests from that user during that time window! :smile:

We are down to 623 lines for k=168451 and at the same time have brought k=156511 to 4M and also run several tests for suspicious results.

It looks like we will have to decide how to proceed in January.
We could either run dc on all remaining k or let the DC rest till we have some more test again and concentrate on first pass.

Then I have the offer from Primegrid to help us to bring DC to 5M for all K.
Here I am still unsure what way I should go. On one hand with these tests we will be "sure" (as sure as you can be without a real factor) that we have no missed primes. But n the other hand the error rate is so low so far that it might be better to run first pass tests to find a new prime first before using resources to make dc tests.

All comes down to the question how confident are we not to have missed a prime. So far the overall error rate is quite low (<2%) with most of the errors concentrated to special users/machines.

Any opinions how we should proceed?

Maybe you DC folks have reduced the error rate down to < 1% by finding this error pattern already?
On the other hand when calculating all candidates that have been tested up to 5M I would not be sure if there are hidden prime in there. Even if the error rate is only 0,5%. If we have the privilige having Primegrid helping here, I will wote for DC to 5M for all k. Then we do not have to worry if there are missing candidates in there......unless we have missed some already in sieving stage.

That makes me wonder if this errors that are already found, and are more or less eliminated to special users/machines. Did this machines do some sieve-works ?

buggy machines are the best machines to have in sieve.
If they make error they only miss factors.

If a buggey machine reports a factor which is incorrect, that factor would only remove a test. However all factors are check to be legit before a test is removed from the system.

I'll let Lars comment about which are which; I think there is probably equal probablility between heavy overclocks and flaky cheap memory from (dell, hp, walmart, etc...). producing errors.

As far as what level, unless you go project wide there is no way to keep double check at the correct level. I think our accorch is currently the best alternative.

If PG wants to DC all k's up to 5M, I'm all for that.

When it comes to strategy, I think we should try to keep k=168451 as close to first pass as possible... and with the current progress in second pass, it's only a matter of time before we catch up.

Having PG doing strictyly double checks is a waste of resources. Especially up to 5M. If we missed a prime, I think that it is above 5M, not below. Besides, we could double check up to 5M ourselves. Let PG continue the fight in first pass. Remember, they double check as they go. This is very important. This is one reason for our low error rate. All the PG results come in double checked and do not add to our error rate.
I also think that we should keep k=168451 as close to first pass as possible. I would go further, and bring k=156511 up to first pass and keep it there. Then when we find a prime for k=168451 we don't have to start from scratch with a new k. We will start a new k, but we will have a k in position to continue the job of quickly monitoring the quality of work while we bring up another k for a backup.

To bring k=156511 to first pass will need another 2200 test from 4M to first pass.

About unstable PCs and sieving. An unstable sieving PC can only miss a factor but can not insert wrong factors into the database. Every factor returned is checked if it is really a factor before it is booked. This is possible due to the fact that checking if a factor is valid only takes milliseconds.

So the worst that can happen if a unstable PC is used for sieving is that we have to run a prp test due to the missed factor. There is no way to miss a prime due to wrong sieving.

An unstable PC used for PRP testing on the other hand can lead to much more trouble as there is only one way to find out if a residue is valid and that is doing a double check which needs the same time as the original one. Also a wrong residue can lead to a missed prime which in the end mean a lot of wasted resources.

To decide if we can be confident about our results how about the following compromise. Lets have Primegrid run 2M-3M (1.7M-2M is manual reserved) as doublecheck and see if the results are really as good as we think. After that we can reevaluate if we should continue to 5M or not.

That's not a bad idea if PG agree's... I didn't think there results were compatible for some reason.

BTW the <2M should be completed shortly.

[QUOTE=ltd;198150]To decide if we can be confident about our results how about the following compromise. Lets have Primegrid run 2M-3M (1.7M-2M is manual reserved) as doublecheck and see if the results are really as good as we think. After that we can reevaluate if we should continue to 5M or not.[/QUOTE]

The problem is that even if 2M-3M shows no or low errors, that does not tell us anything about 3M-5M. We really need to go to 5M, or even higher. And yes, there is probably a prime hiding below 8M. Maybe even two.

The attached graph is a subtle variation on the one I posted in the milestone thread. The lighter green is recent 2nd pass activity and the lighter blue is recent 1st pass activity. The light gray is tests eliminated by sieving. The dark gray is test that have not yet been done.

What about taking 3 candidate through 4M-6M (or around 5M and going in both direction) and see if the error rate are increasing around 5M ? Maybe to little gab to test if we like to see increasing error rate after 5M ?
Anyway I am not going to try to be the wise guy here.... so as usual I will just keep crunching whatever decision will be made

What about taking all the candidates from 5800000 to 6100000?
That would also help find "error prone machines" that are still contributing.

[quote=ltd;198150]To decide if we can be confident about our results how about the following compromise. Lets have Primegrid run 2M-3M (1.7M-2M is manual reserved) as doublecheck and see if the results are really as good as we think. After that we can reevaluate if we should continue to 5M or not.[/quote]

2M-3M is inserted and will be sent out from now on PrimeGrid.

There is still some higher n left in buffer , but it is running now.

Lennart

[QUOTE=Lennart;199911]2M-3M is inserted and will be sent out from now on PrimeGrid.

There is still some higher n left in buffer , but it is running now.

Lennart[/QUOTE]

[CODE]n Range Total Tasks Done Tasks Unprepared Tasks Primes Found
2M - 3M 14215 0 (0%) 14208 0
[/CODE]

They may be "inserted" but they don't look ready to run.

[quote=Joe O;199913][code]n Range Total Tasks Done Tasks Unprepared Tasks Primes Found
2M - 3M 14215 0 (0%) 14208 0
[/code]They may be "inserted" but they don't look ready to run.[/quote]

That means only 7 is sent out to buffer and buffer is set to 100 wu.

There is still 9M wu left in buffer.

You will see many going out tomorrow.

Lennart

This plot only shows the outstanding double check work. PrimeGrid's current work is not shown but will be soon.

As I made a small mistake sometime the next tasks for the DC server are quite low. For some reason I did not send several tests below 2M to VJS when he reserved the range below 2M for manual DC testing. But when it came to create the data for primegrid I was thinking that I had send those tests. So we now have a set of tests that need to be done to close that gap.

Have fun with a burst of short tests.

Primegrid is out handing out the work with the lower N. That means all the double checks will get done before they finish the 9M range they were working on. There are a lot of tests but they are a lot quicker then the current tests.

It would be interesting to see if sending PG a double check range at the same time they do one of their challenges. Lots of tests done in a short time and short run times per test.


S.

[quote=Sloth;200127]It would be interesting to see if sending PG a double check range at the same time they do one of their challenges. Lots of tests done in a short time and short run times per test.[/quote]

I agree! Probably more fun for the participants as well. :smile:

BTW: the PrimeGrid progress can be seen here: [URL]http://www.primegrid.com/stats_psp_llr.php[/URL]

The short test for a challenge might be a good idea.

I am somewhat with Joe on the keeping primegrid to the firstpass tests only as a rule.

The 2-3M test is not a bad idea and am certainly for it for one reason only... the perspective of the 2-3m test is showing if our current 1 or 2K approach is working.

I would say keep PG on firstpasses unless for some reason they request the short tests... especially since they are doing their own doublechecking.

The long tests for the challenge are nice since it really helps move the front edge of the tests forward. Also the long tests make life easier on the PG servers. Even the short tests are still 3-4 hours so server load might not be an issue for PG. Some of their other challenges are tests that are 30 min long.

I assume PG is doing their normal double checks now so the double check stuff they are doing now will actually be triple checks. If everything works out and no major problems are found I would like to see PG stay on first pass. That way we get first and second pass done and taken care of and do not waste time on a 3rd pass test.

PG will only do a single DC test if the residue matches, but will fire off a 3rd (and potentially more) test until a matching residue is received.

[quote=Sloth;200275]I assume PG is doing their normal double checks now so the double check stuff they are doing now will actually be triple checks. If everything works out and no major problems are found I would like to see PG stay on first pass. That way we get first and second pass done and taken care of and do not waste time on a 3rd pass test.[/quote]

As Vato said, PG is not running double tests on all these double check cadidates. They are still quorum 2, but the residues from PSP first pass have been inserted as dummy results into the PrimeGrid system. So if one test is ran and it matches the result from here, no further tests are done on that candidate.

What a difference a week makes: [CODE][n Range Total Tasks Done Tasks Unprepared Tasks Primes Found
2M - 3M 14215 9516 (66.94%) 1992 0[/CODE]

They've done (and are doing) a really great job for the project! Absolutely fantastic! :smile:

Moved on to 3M-4M now...

[I]n[/I] Range Total Tasks Done Tasks Unprepared Tasks Primes Found
2M - 3M 14215 11515 (81.01%) 0 0
3M - 4M 2967 0 (0%) 2842 0

Any day now!
 

[CODE]n Range Total Tasks Done Tasks Unprepared Tasks Primes Found
2M - 3M 14215 14196 (99.87%) 0 0
3M - 4M 15803 15791 (99.92%) 0 0
4M - 5M 19905 17863 (89.74%) 444 0
5M - 6M 10487 10487 (100%) 0 0
6M - 7M 6877 6877 (100%) 0 0
7M - 8M 11702 11702 (100%) 0 0
8M - 9M 9626 9608 (99.81%) 0 0
9M - 10M 974 541 (55.54%) 416 0
Total: 860 0
[/CODE]