Planning & Coordination for 2,2330L

[lukerichards]

Originally Posted by VBCurtis

He changed from an always-on instance to an at-will 24-hr-max instance. Much cheaper; we're speculating about why it's also slower.

Have got to the bottom of this...

instance-1 was running 4 threads on an 8-cpu machine, so while throughput per-thread increased, there were 4 unused cores for the entire time it was running. So essentially, the workrate of this instance is twice what it actually was in terms of rate-per-core.

Having done a few tests of various different configurations of threads over the weekend, have switched back to 8-threaded work.

Perhaps I would get more out of 2 clients each running 4 threads, but right now I can't be doing with adjusting my set up to account for this.

[VBCurtis]

Originally Posted by SethTro

Would it be better for me to use wall clock? or to guess at number of cores? or to remove rels/CPU-sec?

Also 15%!

I think rels/CPU-sec is useful. Our trusty GGNFS siever provides sec/rel for each single-threaded process; yours is the reciprocal, just as useful for comparing machine speeds. Relations per wall-clock-sec is also interesting, as that's what each user should be trying to optimize.

Thanks for the HT speed correction; I estimated 20% improvement in previous personal tests on smaller CADO jobs, and also misremembered your measurements.

[pinhodecarlos]

Those 15% were related to the project progress, not HT....lol 20% is accurate on increase of output with HT on.

[VBCurtis]

I see, thanks Carlos!

The server seems to have ceased sending workunits; a couple of my clients are stalled, while others are still crunching. The terminal housing the CADO server lost contact, but trying to restart the server gives me an error that it's already running.

Hopefully I'll have this fixed in a few minutes; it's not obvious to me how to kill the server process.
EDIT: success! I killed python processes randomly until CADO let me restart. It's back up now.

[lukerichards]

Originally Posted by SethTro

maybe time since last workunit?

How about a tab with the same table as "Client stats" but where clients are grouped by user?

Because of my testing of various configurations I've got over half a dozen different clients on the main table, but I'd be interested to see my overall contribution without having to add it up manually.

[SethTro]

Originally Posted by lukerichards

How about a tab with the same table as "Client stats" but where clients are grouped by user?

Because of my testing of various configurations I've got over half a dozen different clients on the main table, but I'd be interested to see my overall contribution without having to add it up manually.

Vebis has ~10 clients (vebis.1, vebis.2, ...) so I joined all of those to vebis.<X>

I'm going to join all your clients into lukerichards.<comp> for the main tab and then add a new tab for all clients

[SethTro]

I added an individual client tab.

@VBCurtis if you have logs from a past run would you mind if I also visualize them?

I'd like to supporting displaying multiple factoring efforts.

[fivemack]

The server seems to have been unhappy since 21/6 1543Z and is still unhappy at 1613Z

Would there be any possibility of hosting the server in the AWS cloud, or is twelve cents a gigabyte impractical? 2.7 billion relations at 78 bytes per relation.gz is fifty dollars assuming you have to pay once to get them in and again to get them out again

[RichD]

Another thought, in addition to the above, can we get another server going for the second half of the project to run simultaneously? We could pull in small memory machines in addition to having a backup connection should one server go AWOL.

[VBCurtis]

The server is back up; each time CADO has crashed has been the software, not the hardware; I have not rebooted the system, and my other SSH connections to the same machine are still live.

This time, as last, the server instance was still running; restarting produced an error, and I had to manually kill python instances to get CADO to reboot.

Apologies for the delay in giving it a kick; I was in Vegas for 3 days, checking twice a day remotely via Seth's webpage but hadn't checked since this morning.

As for moving it to AWS, I have zero experience with cloud use. If that is the desired pathway, someone else will have to manage the rest of the job. Similarly, I am not confident with merging the relations from two CADO runs- else I would have started the I=16 and I=15 runs at nearly the same time. On the bright side, yield so far is much higher than my test-sieves indicated, so we will be able to transition to I=15 earlier than I expected (say, 180M or 200M rather than 220-250M).

[fivemack]

Originally Posted by VBCurtis

The server is back up; each time CADO has crashed has been the software, not the hardware; I have not rebooted the system, and my other SSH connections to the same machine are still live.

This time, as last, the server instance was still running; restarting produced an error, and I had to manually kill python instances to get CADO to reboot.

Apologies for the delay in giving it a kick; I was in Vegas for 3 days, checking twice a day remotely via Seth's webpage but hadn't checked since this morning.

As for moving it to AWS, I have zero experience with cloud use. If that is the desired pathway, someone else will have to manage the rest of the job. Similarly, I am not confident with merging the relations from two CADO runs- else I would have started the I=16 and I=15 runs at nearly the same time. On the bright side, yield so far is much higher than my test-sieves indicated, so we will be able to transition to I=15 earlier than I expected (say, 180M or 200M rather than 220-250M).

I agree that if the software itself is unreliable there's no point in running it in the cloud.