Rally Nov. 21st-23rd

[gd_barnes]

Originally Posted by mdettweiler

Okay, cool, that might just do the trick. However, you may want to do, say, 1/4 on C443 and 3/4 on G4000....I believe Carlos had mentioned once before that since he's got a Windows server he didn't want too terribly many cores on his server.

OK. How about 32 cores on port 4000 and 10 (including 2 slower) on port 443? 2 slower cores are already running on port 443.

Darn, why didn't this happen on port 5000 at the end of the 3rd drive? Was Lennart not running a Proxy server then?

BTW, I was able to connect to port 400 and it cleared out one pair each on my 2 slower cores (Windows desktop) after I had changed the --once=1 line to uncomment it. Now it has cleared its WuCache on port 400 and is running port 443.

I need to go downstairs and check the quads now to see if they've begun processing port 400 again. If so, I'll stop them, clear out the cache, and move them to ports 443 and 4000.


Gary

[mdettweiler]

Originally Posted by IronBits

Max Connections was set at 60, and it handled everything thrown at it, until the proxy showed up

Hmm...maybe you could try 500? 1000? I don't know--is there a maximum theoretical upper limit for this?

[em99010pepe]

Forget tunning the timings, it's a proxy bug. All cores must be spread through the other servers.

Gary, 10 is ok for C443. Just checked 2 already got work as you said.

Carlos

[IronBits]

It has nothing to do with it, the proxy runs through the sockets at a rate of 10 per second until it runs out of sockets.
Makes no difference if I set it to 1,000, the end result is always the same, runs out of sockets and dies.
Until he turns that thing off, we are screwed... I could go into my router and ignore his IP I think.

[mdettweiler]

Originally Posted by gd_barnes

OK. How about 32 cores on port 4000 and 10 (including 2 slower) on port 443? 2 slower cores are already running on port 443.

Darn, why didn't this happen on port 5000 at the end of the 3rd drive? Was Lennart not running a Proxy server then?

BTW, I was able to connect to port 400 and it cleared out one pair each on my 2 slower cores (Windows desktop) after I had changed the --once=1 line to uncomment it. Now it has cleared its WuCache on port 400 and is running port 443.

I need to go downstairs and check the quads now to see if they've begun processing port 400 again. If so, I'll stop them, clear out the cache, and move them to ports 443 and 4000.


Gary

Yes, I think this is the first time Lennart is using a proxy. The main reason why he wanted to set up a proxy was because it was getting just too hard to manually stop/start 60 or 70 individual LLRnet clients.

Hey, wait a minute, I just had a thought! How about Lennart simply set his proxy to cache, say, 50 k/n pairs at one time--well below any of our servers' maxConnections limit--and thus, theoretically, his proxy should be no worse than *one* caching LLRnet client set to 50 k/n pairs at once (which is not a problem)? Yes, his proxy would be connecting to the server more often, but that's fine, as long as it's in more manageable, individual chunks--as is evidenced by how the servers survive him just fine when he connects without a proxy.

[IronBits]

Beyond and I went through all of this before to find out what the problem was and it came down to the proxy and server not being able to talk properly.
You can't fix this one Max.
What he could do is run multiple proxy servers, with no more than 25 clients per proxy.
I just watched his proxy eat 100 sockets in under 2 seconds.

[IronBits]

You can also see another problem the proxy causes by looking at the rejected.txt file.
http://nplb.ironbits.net/rejected_400.txt

I need to get some sleep, been up all night...
if he doesn't kill his proxy server, then my server will not be able to function until it does ...

[mdettweiler]

Originally Posted by IronBits

Beyond and I went through all of this before to find out what the problem was and it came down to the proxy and server not being able to talk properly.
You can't fix this one Max.
What he could do is run multiple proxy servers, with no more than 25 clients per proxy.
I just watched his proxy eat 100 sockets in under 2 seconds.

Hmm...I see. That's why I was thinking that if he set his proxy to cache, say, 50 k/n pairs at a time (instead of 1000 or whatever he's got), then it would communicate more often, but in smaller bits, thus giving the server a chance to clean up its sockets before it connects again.

Anyway, might help (band-aid fix, at least), might not--I suggested it over at the PrimeGrid chat room thingie, hopefully Lennart will see it shortly and give it a try. And then if that doesn't help, as a last resort he can go back to managing his cores manually, without a proxy.

[gd_barnes]

Yep, all of my quads were sleeping too. Argh! I just now killed a couple of them, changed them to once=1 to clear out the current pair on port 400, and restarted those. No luck...can't even clear out the single pair in the cache.

I know where Lennart is coming from. I now have to change 42 total cores over and the way I have things set up, I try to keep different servers in different folders and make sure they have different GUI port numbers so they don't step on each other when I'm trying to clear one out while starting on the other. So it's not quite the quick process of copying ant then cutting and pasting. This promises to be an hour task or more. If I had 68 cores or more, I'd likely spend 2 hours.

So there is definitely a need for a proxy server. If there is some compromise where the heavy hitters only cache 25 pairs at a time for their proxy server, then let's work with them on setting up that kind of thing.


Gary

[IronBits]

Lennart, it is not your fault! It's the communication between the proxy and the server.

My server will be down and out for a while, because I have to get some sleep.
I'll check on the situation when I get up from my nap and hopefully it will over by then.

Lennart, you can put 25 fast cores on a proxy to see if that helps...

[mdettweiler]

Originally Posted by IronBits

Lennart, it is not your fault! It's the communication between the proxy and the server.

My server will be down and out for a while, because I have to get some sleep.
I'll check on the situation when I get up from my nap and hopefully it will over by then.

Lennart, you can put 25 fast cores on a proxy to see if that helps...

Hey, I just had another idea: maybe somebody (i.e. somebody with at least a modicum of LUA programming language) could code up some small code to insert in place of whatever makes up the LLRnet proxy server's "send a dump of results to the server" code, that essentially does the following:

for(i = 0; i < *number of results in tosend-proxy.txt*; i++) {
send 50 results to server;
wait 10 seconds;
}

Essentially the above pseudo-pseudocode transmogrified into LUA.

Anyway, gotta run...