Cable Internet question

[Prime95]

Sigh, now I'm confused.

I prefer the default installation of prime95 leads to trouble-free operation. I don't want to annoy users. Many are unsophisticated and won't know to tweak these settings. I don't need the headache of fielding email complaints about prime95 slowing down their Internet. Some won't bother with email, they just stop running prime95.

OTOH, I don't want to cripple the server.

I had just coded up the default of 0.25Mbps default upload speed.

Each chunk is 1MB in size (bigger if not limiting upload bandwidth). I'm not sure if the TCP connection is kept alive between chunks. If it is, would there be any advantage in creating a new connection with each chunk?

@Mark: Yes, resume upload is supported.

[preda]

Originally Posted by Prime95

Each chunk is 1MB in size (bigger if not limiting upload bandwidth). I'm not sure if the TCP connection is kept alive between chunks. If it is, would there be any advantage in creating a new connection with each chunk?

That may be an idea -- upload each chunk (1MB) fast, and breathe for a few seconds in between chunks, closing the TCP connection during this "rest" interval (which I think is what happens by default, i.e. the connection is not kept alive). (probably this would work better with HTTP vs. HTTPS, where I imagine HTTPS has a more expensive connection establishment).

OTOH on GpuOwl's upload script I'm going to try "connection keep-alive" between chunks to speed up the upload.

Maybe this is the idea: upload all proofs of the day at 3am each night, without throttling.

[Uncwilly]

My suggestion is allow the user to designate what their "3 AM" is. If the computer is at a work site that requires machines to be shut down at the end of the day, a hard coded time would be troublesome. Or for people that shutdown laptops, etc.

[Prime95]

Originally Posted by preda

T
Maybe this is the idea: upload all proofs of the day at 3am each night, without throttling.

I did some quick googling at it looks like Windows Server is supposed to have no troubles with 50,000+ active TCP connections. Aaron would knoe the definitive answer.

There will be an option for time interval prime95 is allowed to do its uploads. The problem with setting this to 2AM to 5AM by default is the user that turns his computer off at night never uploads his proofs.

[PhilF]

Originally Posted by Prime95

I did some quick googling at it looks like Windows Server is supposed to have no troubles with 50,000+ active TCP connections. Aaron would knoe the definitive answer.

Tens of thousands of simultaneous open TCP connections aren't really a problem. In my opinion you shouldn't use it as a reason to make decisions concerning software defaults.

[Mark Rose]

Originally Posted by preda

That may be an idea -- upload each chunk (1MB) fast, and breathe for a few seconds in between chunks, closing the TCP connection during this "rest" interval (which I think is what happens by default, i.e. the connection is not kept alive). (probably this would work better with HTTP vs. HTTPS, where I imagine HTTPS has a more expensive connection establishment).

There are two potential problems with this. The first is on the client side, where saturating upload is going to delay TCP/ACK and UDP packet. Forget having a VOIP call or doing online gaming. This will quickly lead to Prime95 being removed from networks for having abusive uploading behaviour.

The second comes into play if a load balancer or NAT is used. Each TCP connection requires a unique tuple of source port and IP to differentiate it from other connections. Since strange things can happen, like packet retransmission, there is a time during which TCP tuples cannot be reused, generally one minute. This effectively limits the maximum connection throughput from a load balancer IP to back end service to about 1000 connections/second. The same could also happen on a corporate network behind a NAT gateway, where if uploads were scheduled to happen at one time at night, causing tuple exhaustion between the NAT and PrimeNet. So it's better to make fewer connections per second.

The other concern is resource usage on the server when dealing with thousands of connections. We run many servers that maintain 50k connections at work, all also proxied through Nginx. The TCP buffers even though tuned to be small, still consume gigabytes of memory. So that's the argument for uploading as fast as possible, to reduce memory usage. Any reasonably written HTTP server can handle 50k connections on a single core machine; the c10k problem was solved over a decade ago.


I do think being to pick an upload time as well as a bandwidth limit is ideal. Even better than picking a bandwidth limit, would be implementing CoDel in the app, (since Windows and Mac haven't gotten around to implementing it), a flow control algorithm that monitors round trip latency to avoid saturating buffers, which is the problem George ran into trying to watch Netflix and getting his TCP ACK packets delayed.