Rally Nov. 21st-23rd

[mdettweiler]

Originally Posted by IronBits

That's what I thought it was. llrnet server can't handle proxy to server very well at all.
We are toast till he knocks off some of them cores. Beyond used to do that to me as well, and he had to stop doing it ...

Yet, ironically enough, we survived Lennart just fine when he put them on "normally", i.e. without a proxy...maybe that's the solution?

[gd_barnes]

Well, bummer.

I'll start shifting shortly. Perhaps half on 443 and half on 4000. Carlos, I'm assuming you can handle ~20 clients coming your way.

One more thing: All of my cores (42) currently have a single k/n pair that has processed and is ready to be sent to port 4000. What I'll do is kill the processes on all of those, change the --once=1 line to remove the comment and restart them. Whenver your server is ready again, it will then just send those completed pairs back to you and not request any more work. At the same time, I'll have those same cores running ports 443 and 4000.

I'm thinking without my 42 cores on port 400, Lennart and everyone else should be OK. I'm assuming he has 68 on there. Even if he has up to 100, we've tested with both him and I on port 5000 before where he had 68 cores running and we had no problems.


Gary

[em99010pepe]

Originally Posted by IronBits

That's what I thought it was. llrnet server can't handle proxy to server very well at all.
We are toast till he knocks off some of them cores. Beyond used to do that to me as well, and he had to stop doing it ...

That was so annoying when he was constantly breaking my server down. We should put an announce to advise people not to run a proxy server when they have more than 30 cores.

Gary, my server has probably less than 15 cores connected, you can bring 4 quad cores to it. I already slept so I'll stay here controlling things. Taking all of you cores from IB400 won't resolve things because Lennart proxy will keep dumping and crashing the server.

Carlos

[IronBits]

the proxy server marches right up the sockets till it runs out.
I changed it to 150 and within 30 seconds it dies

connection request from e325955b:2342 (socket 149)
broken pipe
recv error res=0, errno=32
recv error res=0, errno=32
recv error res=0, errno=32
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 149)
connection request from e325955b:2344 (socket 149)
connection request from e325955b:2346 (socket 150)
connection request from e325955b:2349 (socket 151)
connection request from 879e2544:54186 (socket 152)
Proposing pair 587/556366 to MyDogBuster
broken pipe
recv error res=0, errno=32
recv error res=0, errno=32
recv error res=0, errno=32
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 151)
Proposing pair 717/556366 to MyDogBuster
connection closed (socket 152)
connection request from e325955b:2353 (socket 151)
connection request from e325955b:2355 (socket 152)
connection request from e325955b:2357 (socket 153)
recv error res=-1, errno=104
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 153)
connection request from e325955b:2359 (socket 153)
connection request from e325955b:2363 (socket 154)
connection request from e325955b:2361 (socket 155)
connection request from 75242c52:2085 (socket 156)

[mdettweiler]

Originally Posted by gd_barnes

Well, bummer.

I'll start shifting shortly. Perhaps half on 443 and half on 4000. Carlos, I'm assuming you can handle ~20 clients coming your way.

One more thing: All of my cores (42) currently have a single k/n pair that has processed and is ready to be sent to port 4000. What I'll do is kill the processes on all of those, change the --once=1 line to remove the comment and restart them. Whenver your server is ready again, it will then just send those completed pairs back to you and not request any more work. At the same time, I'll have those same cores running ports 443 and 4000.

I'm thinking without my 42 cores on port 400, Lennart and everyone else should be OK. I'm assuming he has 68 on there. Even if he has up to 100, we've tested with both him and I on port 5000 before where he had 68 cores running and we had no problems.


Gary

Okay, cool, that might just do the trick. However, you may want to do, say, 1/4 on C443 and 3/4 on G4000....I believe Carlos had mentioned once before that since he's got a Windows server he didn't want too terribly many cores on his server.

[em99010pepe]

Originally Posted by mdettweiler

Yet, ironically enough, we survived Lennart just fine when he put them on "normally", i.e. without a proxy...maybe that's the solution?

Correct.

[mdettweiler]

Originally Posted by IronBits

the proxy server marches right up the sockets till it runs out.
I changed it to 150 and within 30 seconds it dies

connection request from e325955b:2342 (socket 149)
broken pipe
recv error res=0, errno=32
recv error res=0, errno=32
recv error res=0, errno=32
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 149)
connection request from e325955b:2344 (socket 149)
connection request from e325955b:2346 (socket 150)
connection request from e325955b:2349 (socket 151)
connection request from 879e2544:54186 (socket 152)
Proposing pair 587/556366 to MyDogBuster
broken pipe
recv error res=0, errno=32
recv error res=0, errno=32
recv error res=0, errno=32
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 151)
Proposing pair 717/556366 to MyDogBuster
connection closed (socket 152)
connection request from e325955b:2353 (socket 151)
connection request from e325955b:2355 (socket 152)
connection request from e325955b:2357 (socket 153)
recv error res=-1, errno=104
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 153)
connection request from e325955b:2359 (socket 153)
connection request from e325955b:2363 (socket 154)
connection request from e325955b:2361 (socket 155)
connection request from 75242c52:2085 (socket 156)

So, just curious, what was your maxConnections set to before you changed it to 150? I have G4000 set to 300 (though it's never actually gotten anywhere near there).

I've got a hunch that as long as your maxConnections is at least just a bit higher than the amount Lennart is caching at a time, that might give us just enough room to squeak by when his proxy hits the server.

[em99010pepe]

PrimeGrid has an online chat, I'll see if Lennart is there.

[mdettweiler]

Originally Posted by em99010pepe

Correct.

I guess the problem is that the whole concept of how the proxy server works was designed for the "original" LLRnet setup at Riesel Sieve, which has always worked with much, much bigger numbers than these, so even for someone with as many cores as Lennart the load wouldn't be too bad...still, even then if I remember correctly they set up a separate server for one of their heaviest hitters at one point. Presumably he was just so "heavy" that he was knocking out even their custom-made high-resiliency servers even at high n-levels.

[IronBits]

I upped it to 250 and within about 1 minute, the proxy server locks me up.
Everyone might as well go someplace else until Lennart stops using a proxy server to connect to here. It will never run with more than ~50 cores on a proxy server to my server, that's what Beyond and I figured out when he was doing it. Yes, I can handle tons of single clients, but there is something fishy about a proxy to server communications problem.

Proposing pair 777/556372 to gd_barnes
Proposing pair 929/556372 to gd_barnes
connection closed (socket 253)
connection closed (socket 254)
recv error res=-1, errno=104
recv error res=0, errno=104
recv error res=0, errno=104
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 252)
connection request from 26d85e46:43116 (socket 252)
Proposing pair 749/556372 to gd_barnes
connection closed (socket 252)
connection request from e325955b:4331 (socket 252)
connection request from e325955b:4333 (socket 253)
connection request from e325955b:4337 (socket 254)
connection request from e325955b:4335 (socket 255)
recv error res=-1, errno=104
recv error res=0, errno=104
broken pipe
recv error res=0, errno=32
shutdown: errno 107
connection closed (socket 255)
connection request from 26d85e46:43661 (socket 255)
connection request from e325955b:4339 (socket 256)
Proposing pair 987/556372 to gd_barnes
connection closed (socket 255)
connection request from e325955b:4341 (socket 255)

[IronBits]

Max Connections was set at 60, and it handled everything thrown at it, until the proxy showed up

I haven't gone to bed yet, but I'll hang in there for a little bit to see if we can get Lennart to kill his proxy.
I can take the clients fine, just no proxy servers please.

I'll leave the server down for a little while, maybe his proxy will give up and roll over and die.
That's the best I can hope for right now to alleviate the problem.