Testing....

[mdettweiler]

Originally Posted by gd_barnes

The version that I posted yesterday is the latest version. Correct? lol It is that latest version that I ran my big stress test on yesterday. It was about halfway through the stress test that I changed a prime residue from 16 x's to a single digit of "0" like the Windows client. What I want to test today is the same script for the cancellation of pairs and the problem with the pairs not processed at the beginning and end of the file by the server.

Ah, right, duh. BTW, I tested the cancellation function about half an hour ago on Windows and it seems it worked perfectly.

[kar_bon]

Originally Posted by gd_barnes

What change is needed to accomplish what you are talking about?

Edit: The code in the Windows client is the same. Please enlighten me.

see my note in that post: only in the 'old' llrnet.lua file, when LLRnet also do the primetest. this

	 result, residue = primeTest(t, format("%s %s", k, n))
	 --result, residue = 0, "0"

	 -- check user interruption
	 if stopCheck() then
	    return -- return with no error
	 end

is what i see in the lua. the result/residue set to 0 commented out for testing.
but it seems, when a prime is found, the residue was not set to "0" in primeTest properly, so then my code!

[kar_bon]

Originally Posted by gd_barnes

Don't you ever sleep? lol

I'm going to need some help with this. I'm not clear on where in llrserver.lua that it goes. Can you post an updated llrserver.lua file with this change in it?

just awaken and perparing for work. perhaps i can do that then.

try a 'search for...' in an editor

[gd_barnes]

Mr. former maintenance programmer still at your service here.

Not that it should ever happen but I've now fixed the problem when a pair has a small factor, which caused some subsequent pairs not to be sent (I believe the remainder of the batch). What was important about this, though, is that I've now verified that that is exactly what was causing several pairs at the beginning of each of my runs to not be processed to the server. Checking my latest run confirmed that all pairs at the beginning of the file were processed.

What I ended up doing is putting the term "factored" in the residue when there was a small factor. I had to add an elsif to that section to do a separate parsing of the line looking for the word "factor".

An important discovery: I've confirmed that the fact that the last few pairs aren't being sent when the server dries is a bug in the Linux client script. It processes them all in the client but because the server has dried, it can't retrieve more pairs. And if it cannot retrieve more pairs, it doesn't want to send that final batch. The fact is, it should be sending those pairs BEFORE attempting to retrieve additional pairs. That is the issue.

What this confirms is what Max encountered where he had to send one final "bogus" pair/result to the server in order to get the final pairs to process was unrelated to the pruning. The fact is, the results never got there because they were sitting in tosend.txt. Once it was able to process one final bogus pair, it sent them all except the bogus pair, which is exactly what was needed.

Karsten, on the Windows client, have you run tests where you've dried out a server? If so, was there any problem with it sending the final pairs to the server? In other words, did all of the pairs in knpairs.txt clear themselves out with an automatic pruning?

Bottom line:
6 errors corrected
1 error to go


I suspect this last error will be the most difficult for me to fix because it involves the structure and timing of when things are processed in the script; which may require a complete change in the order of things; which could result in several other things being impacted. If there's one thing I will do; it is be VERY careful and test the heck out of it if I think I have it right.


Gary

[kar_bon]

Originally Posted by gd_barnes

An important discovery: I've confirmed that the fact that the last few pairs aren't being sent when the server dries is a bug in the Linux client script. It processes them all in the client but because the server has dried, it can't retrieve more pairs. And if it cannot retrieve more pairs, it doesn't want to send that final batch. The fact is, it should be sending those pairs BEFORE attempting to retrieve additional pairs. That is the issue.

What this confirms is what Max encountered where he had to send one final "bogus" pair/result to the server in order to get the final pairs to process was unrelated to the pruning. The fact is, the results never got there because they were sitting in tosend.txt. Once it was able to process one final bogus pair, it sent them all except the bogus pair, which is exactly what was needed.

Karsten, on the Windows client, have you run tests where you've dried out a server? If so, was there any problem with it sending the final pairs to the server? In other words, did all of the pairs in knpairs.txt clear themselves out with an automatic pruning?

I suspect this last error will be the most difficult for me to fix because it involves the structure and timing of when things are processed in the script; which may require a complete change in the order of things; which could result in several other things being impacted. If there's one thing I will do; it is be VERY careful and test the heck out of it if I think I have it right.

- my script does the cLLR-work first
- when done, convert lresults.txt into tosend.txt
- call LLRnet
- if no (new) workfile.txt exist, goto ERROR (no connection)

i've not tested this with a real server, so perhaps we could do this later today.

but:
when i process a workfile on my offline PC (same script), the script will end with that conncection error and the tosend.txt remains.
i then only start llrnet.exe (folder on stick into online PC): LLRnet will first getting new pairs and then submitting the tosend.txt!

to test:
what happens when LLRnet dried and calling llrnet.exe with a tosend.txt there

looking at llrnet.lua the function DialogWithServer:
(some unimportant lines deleted)

function DialogWithServer()
(...)
   -- first check if we have some job to work on
  (...)
   -- if no unfinished job, then ask a new job to the server
   if not t or not k or not n then
      --print("Requesting new job from the server ...")
      t, k, n = GetPair()
      changed = t and k and n
   end
(...)
	 print(format("Fetching %d additional Work Unit(s) ...", num))
(...)

   if changed then
      WriteWorkfile(t, k, n, more)
      SendStructureToAllGUIs("WUs", more, 1)
   end

   SendAllResults()
(...)
end

SendAllResults done at last, so perhaps put this line at first here and test again!

so if the script is ok, it's at LLRnet not sending the last results when the server dried.

@Gary: you can test this easily, too.
a folder with normal testing, 2nd folder with some pairs done but no connection so only the tosend.txt exist.
let the first folder dry out the server and then start the script/llrnet.exe (2 tests!) in the 2nd folder!
i'm home again in about 7 hours from now so can't test it (and got no dried server ).

[gd_barnes]

OK guys, after much effort, I've been unable to fix the problem with the server not returning the final few pairs as the server is near drying. Oddly, I even switched the order of things and it would show that it had sent the final tosend.txt pairs to the server but the server would never accept them. In other words, they just never showed up in the server results.txt or rejected.txt. It's like they just went off into cyberspace somewhere.

I suppose we could live with that at NPLB and just make sure a server never dries but there's one problem: I checked it against the current client that we run in "production". It returns all pairs perfectly. So despite correcting a couple of bugs that LLRnet previously had, we have introduced a new bug that it did not previously have, which I find difficult to stomach.

Karsten, by your explanation here, I suspect the Windows client will have the same problem.

So we have some work to do.

In a little while, I will edit this post to attach the latest do.pl Linux script. Besides the factoring issue that I've fixed, I've tweaked the order of when tosend.txt can be processed and not made it dependent on getting a workfile.txt file from LLRnet. Now it will immediately check for tosend.txt when starting the script and send the pairs immediately. This is good if the server goes down, your machine has subsequently processed and formatted the pairs into tosend.txt, and you decide to stop your client. Then after the server comes back up and you start your client back up, it will send them all right away. That closely simulates what LLRnet currently does.

I've also tweaked the timing of when a time stamp is written to lresults_hist stating when pairs were sent to the server. Previously that time stamp was written when lresults was added to lresults_hist. That was incorrect because the results have not actually been sent to the server at that point. It is now written immediately after the time in which they are actually sent.


Gary

[kar_bon]

Originally Posted by gd_barnes

I suppose we could live with that at NPLB and just make sure a server never dries but there's one problem: I checked it against the current client that we run in "production". It returns all pairs perfectly. So despite correcting a couple of bugs that LLRnet previously had, we have introduced a new bug that it did not previously have, which I find difficult to stomach.

Karsten, by your explanation here, I suspect the Windows client will have the same problem.

So we have some work to do.

i've nothing changed in client.lua (only 3 lines with output) or client_server.lua nor llrserver.lua. so the error has to be on the client side.

i see in do.pl the function doLLRnet() is called in 2 different places.
only just before the end of the main loop, the tosend.txt will be deleted (and in cancel-function of course). i call llrnet.exe only once.

try to test as i mentioned above!

[gd_barnes]

Karsten,

I was somewhat unable to follow what you meant about testing in the last post. I'm extremely bleary eyed now after an all night testing session. But in looking at what I could understand, it appears that I'm doing very close to what you are suggesting. My tests tonight have involved one core running about 200 small pairs over and over.

I do have some good news finally:

I was able to get it down to where the server is only not receiving the final about 1/2 of a batch of pairs at the very end of the file. This amounted to not receiving the final 2 pairs instead of 7 pairs with the cache set to 5. I found that it was erroring out on the last batch when it had less than 5 pairs when it was trying to display "WU 3/5 xx xxxxx", which caused the next-to-last batch to not be received by the server.

What I did was comment out 2 lines in the llrnet.lua file. That caused it to stop the error message and allowed it to process the next-to-last batch. Now it's only the final batch (in this case 2 pairs) that it won't accept.

Attached are the modified llrnet.lua file and do.pl script.

Changes:
1. Commented out 2 lines in llrnet.lua where it was displaying unneeded info. about what WU's it was fetching. This was causing an error when less than a full batch was remaining on the server.

2. Corrected the problem with a pair that has a small factor. The residue will contain the simple word "factored" in it. Currently LLRnet just puts the residue of the previous pair in there, which is a bug...the same bug it does on a prime and that can cause a problem if the first pair in the file is a prime.

3. Moved the timing of when the time stamp referencing pairs being submitted to the server is written to the lresults_hist file. It was previously written several lines of code before they were actually submitted, which in some cases meant that it was being written when the results were not actually received by the server due to other issues.

4. Something you had mentioned that I had earlier concluded: I changed the execution of doLLRnet() at the end of the main loop to simply run LLRnet once. The problem was that doLLRnet() requires that at least one pair be fetched because it requires the existence of the workfile.txt in order to exit the loop. On the next-to-last or last batch, that was causing a problem. It could even happen sooner if many clients are connected. The DoLLRnet() requiring the existing of workfile.txt in order to exit should only be executed when the main intent of it is to fetch pairs. That is now done only one time towards the beginning of the main loop.

Based on what you said, we are now more synced up on #4.

Note that I opted against moving the processing of tosend.txt to the top of the main loop like I had suggested that I would do earlier. I thought of a couple of scenarios where that would not be desirable.

Please take a look at the logic where I handle a small factor. LLRnet as it currently exists incorrectly handles it by leaving the residue from the previous pair in there. But our scripts were worse than that because they would put the file header in there and end up skipping several subsequent pairs. You should probably code for that in the Windows client.

Stats:
8 errors corrected (including new problems #1 & #4 above. #2 was already counted and #3 is a nitpick more than a bug.)
1 still to go
:-)


Gary

[kar_bon]

i had a quick look at your llr.lua:

you can put the printings (print"Fetching...) below the test, if GetPair was ok, like:

before

	    t, k, n = GetPair()
	    print(format("  Fetching WU #%d/%d: %s %s",i,WUCacheSize,k,n))
	    if t and k and n then
	       tinsert(more, { k = k, n = n })
	    else
	       break
	    end

after

	    t, k, n = GetPair()
	    if t and k and n then
               print(format("  Fetching WU #%d/%d: %s %s",i,WUCacheSize,k,n))
	       tinsert(more, { k = k, n = n })
	    else
               print("No more pairs on the server!")
	       break
	    end

and perhaps the message, no pairs available on the server.

try it. i have to look for that at home, there're several points to do so!

testing:
- getting new pairs when server is already dried
- getting the last pairs from the server, not enough then

[mdettweiler]

Originally Posted by gd_barnes

Karsten,

I was somewhat unable to follow what you meant about testing in the last post. I'm extremely bleary eyed now after an all night testing session. But in looking at what I could understand, it appears that I'm doing very close to what you are suggesting. My tests tonight have involved one core running about 200 small pairs over and over.

I do have some good news finally:

I was able to get it down to where the server is only not receiving the final about 1/2 of a batch of pairs at the very end of the file. This amounted to not receiving the final 2 pairs instead of 7 pairs with the cache set to 5. I found that it was erroring out on the last batch when it had less than 5 pairs when it was trying to display "WU 3/5 xx xxxxx", which caused the next-to-last batch to not be received by the server.

What I did was comment out 2 lines in the llrnet.lua file. That caused it to stop the error message and allowed it to process the next-to-last batch. Now it's only the final batch (in this case 2 pairs) that it won't accept.

Attached are the modified llrnet.lua file and do.pl script.

Changes:
1. Commented out 2 lines in llrnet.lua where it was displaying unneeded info. about what WU's it was fetching. This was causing an error when less than a full batch was remaining on the server.

2. Corrected the problem with a pair that has a small factor. The residue will contain the simple word "factored" in it. Currently LLRnet just puts the residue of the previous pair in there, which is a bug...the same bug it does on a prime and that can cause a problem if the first pair in the file is a prime.

3. Moved the timing of when the time stamp referencing pairs being submitted to the server is written to the lresults_hist file. It was previously written several lines of code before they were actually submitted, which in some cases meant that it was being written when the results were not actually received by the server due to other issues.

4. Something you had mentioned that I had earlier concluded: I changed the execution of doLLRnet() at the end of the main loop to simply run LLRnet once. The problem was that doLLRnet() requires that at least one pair be fetched because it requires the existence of the workfile.txt in order to exit the loop. On the next-to-last or last batch, that was causing a problem. It could even happen sooner if many clients are connected. The DoLLRnet() requiring the existing of workfile.txt in order to exit should only be executed when the main intent of it is to fetch pairs. That is now done only one time towards the beginning of the main loop.

Based on what you said, we are now more synced up on #4.

Note that I opted against moving the processing of tosend.txt to the top of the main loop like I had suggested that I would do earlier. I thought of a couple of scenarios where that would not be desirable.

Please take a look at the logic where I handle a small factor. LLRnet as it currently exists incorrectly handles it by leaving the residue from the previous pair in there. But our scripts were worse than that because they would put the file header in there and end up skipping several subsequent pairs. You should probably code for that in the Windows client.

Stats:
8 errors corrected (including new problems #1 & #4 above. #2 was already counted and #3 is a nitpick more than a bug.)
1 still to go
:-)


Gary

Nice work--I read through the code and it looks good. I've uploaded the latest packages to the following links due to problems with the noprimeleftehind.net server:

http://nplb-gb1.no-ip.org/sieves/llr...0.61-win32.zip
http://nplb-gb1.no-ip.org/sieves/llr...61-linux32.zip

[kar_bon]

Status of WIN-script V0.63

i've just uploaded a new version of the WIN-script with the latest lua-files (link see post #1).

llrnet.lua:
- no errors when printing "Fetching ..." new pairs -> included again

do.bat:
- displaying important settings from llr-clientconfig.txt
- trying to connect to server if not responding (every 60 seconds) forever

llr-clientconfig.txt:
- reduced to mostly needed settings, all others will set in lua-files before reading these

ToDo:
- parsing several options in batch-mode is very annoying and several lines per option would be needed with error-finding. so i will do it with setting an option at top of the do.bat with the set-command from DOS. every user can put his own settings (ON or OFF) in there.

- option OutputIterations (10000 by default)
- option primes.txt in same folder or folder above (above by default)
- option beep on prime / batch ended (beep by default)

- to test: dried server /dry when receiving pairs

do_tosend.awk:
- support pairs with small factors found or probable primes

more work to do now as thought!