[QUOTE=Uncwilly;276581]You can unassign them. Log-in to your PrimeNet account. Click on Assignments. You will be presented with a list of your current assignments. Check the box next to those that you wish to drop. Click the box at the bottom that says "Unreserve checked exponents".[/QUOTE]

To clarify, click on "Assignments" under "My Account", not "Assignments" under "Manual Testing".

Thank you very much, it works.

GPU to 72 version 0.3 now available to assign work...
 

OK...

The above proposed "GPU to 72 Tool" is now ready and able to assign work to "GPU Workers". And after only three days of "spidering", it has already collected 641 exponents below 50M which have been TFed to less than 71. (457 to less than 70; 152 to less than 69.)

The tool can be accessed at [url]http://gpu.mersenne.info/[/url]

Please give it a try, and let me know if anyone sees any issues, or have suggestions.

Additional reports will be made available over the next couple of days, and then I will implement a "DC Work Type". After that, I will enable functionality to enable "P-1 Workers" to request low exponents which have not had any P-1 work done on them.

Lastly, I plan to add a feature to allow "owners" of exponents to request GPU work to be done on the exponents by those with the "fire power".

[QUOTE=chalsall;276621]OK...

The above proposed "GPU to 72 Tool" is now ready and able to assign work to "GPU Workers". And after only three days of "spidering", it has already collected 641 exponents below 50M which have been TFed to less than 71. (457 to less than 70; 152 to less than 69.)

The tool can be accessed at [url]http://gpu.mersenne.info/[/url]

Please give it a try, and let me know if anyone sees any issues, or have suggestions.

Additional reports will be made available over the next couple of days, and then I will implement a "DC Work Type". After that, I will enable functionality to enable "P-1 Workers" to request low exponents which have not had any P-1 work done on them.

Lastly, I plan to add a feature to allow "owners" of exponents to request GPU work to be done on the exponents by those with the "fire power".[/QUOTE]
Sweet!
I do believe that James Heinrich already has something similar for P-1: [url]http://mersenne-aries.sili.net/p1small.php[/url]
Otherwise awesome!

Perfect! Signed up and grabbing work already. I'm assuming you will automatically release all exponents that are done to 72. There has to be a line where TF to 73 starts becoming optimal. For the moment, my suggestion is that we just add 4 to Primenet's default TF limits.
[CODE]47450000 2^68
58520000 2^69
[/CODE]

So do everything to 47450000 to 72 bits and everything to 58520000 to 73 bits. We can tinker with these once we know how much horsepower we have and how long GPU-TF vs. LL takes. Thoughts?

[QUOTE=garo;276632]Perfect! Signed up and grabbing work already. I'm assuming you will automatically release all exponents that are done to 72. There has to be a line where TF to 73 starts becoming optimal. For the moment, my suggestion is that we just add 4 to Primenet's default TF limits.
[CODE]47450000 2^68
58520000 2^69
[/CODE]

So do everything to 47450000 to 72 bits and everything to 58520000 to 73 bits. We can tinker with these once we know how much horsepower we have and how long GPU-TF vs. LL takes. Thoughts?[/QUOTE]
I'd say start with three extra bits, and then reevaluate the fourth.

[QUOTE=garo;276632]Perfect! Signed up and grabbing work already. I'm assuming you will automatically release all exponents that are done to 72. There has to be a line where TF to 73 starts becoming optimal. For the moment, my suggestion is that we just add 4 to Primenet's default TF limits.
[CODE]47450000 2^68
58520000 2^69
[/CODE]

So do everything to 47450000 to 72 bits and everything to 58520000 to 73 bits. We can tinker with these once we know how much horsepower we have and how long GPU-TF vs. LL takes. Thoughts?[/QUOTE]

Assuming 0.029 sec per iter, 2.25 test saved per factor. My back of the envelope calculation tell me that 73 bit is optimal on my system (GTX 560 ~ 37 min for 2^70 TF search).

[QUOTE=garo;276632]Perfect! Signed up and grabbing work already.[/QUOTE]

Yes, I saw that. Thanks; as you promised, you are my first beta tester.

[QUOTE=garo;276632]I'm assuming you will automatically release all exponents that are done to 72.[/QUOTE]

Yes.

[QUOTE=garo;276632]There has to be a line where TF to 73 starts becoming optimal. For the moment, my suggestion is that we just add 4 to Primenet's default TF limits.
[CODE]47450000 2^68
58520000 2^69
[/CODE]

So do everything to 47450000 to 72 bits and everything to 58520000 to 73 bits. We can tinker with these once we know how much horsepower we have and how long GPU-TF vs. LL takes. Thoughts?[/QUOTE]

My intention is to allow each Worker to vote for where the cut-offs should be, with a weight given based on each Worker's contribution.

Prime95, of course, would be given a serious weight in the vote even if he didn't contribute a single GPU cycle....

I like this...
 

I like it; I took five assignments to test it out. I still have 130 exponents Mr. P-1 gave me which I will finish up before switching over to this new process.

What about the double check TF work ckdo and his assistants (I am one) are working on? Will there be any way to integrate that, or perhaps have a separate version of this process?

[QUOTE=Dubslow;276635]I'd say start with three extra bits, and then reevaluate the fourth.[/QUOTE]
Nah. Don't pussyfoot around.

As with any important engineering project, testing
to destruction is called for:smile:

David

Unfinished work
 

How will this tool handle (heaven forbid) unfinished assignments? Will there be some time limit?

[QUOTE=Chuck;276651]How will this tool handle (heaven forbid) unfinished assignments? Will there be some time limit?[/QUOTE]

On the [url]http://gpu.mersenne.info/account/getassignments/[/url] page it says that assignments are valid for one month, but can be extended.

[QUOTE=Chuck;276646]What about the double check TF work ckdo and his assistants (I am one) are working on? Will there be any way to integrate that, or perhaps have a separate version of this process?[/QUOTE]

That is trivial to implement.

But I want to get the LL system fully implemented first. The system's data definitions and logic are keeping future DC work (and P-1 work) in mind.

[QUOTE=chalsall;276658]On the [URL]http://gpu.mersenne.info/account/getassignments/[/URL] page it says that assignments are valid for one month, but can be extended.[/QUOTE]

Oops, I wasn't very observant.

[QUOTE=Chuck;276646]I like it; I took five assignments to test it out.[/QUOTE]

Thanks. And yes, I saw that.

[QUOTE=Chuck;276646]I still have 130 exponents Mr. P-1 gave me which I will finish up before switching over to this new process.[/QUOTE]

It would help me if you did at least one of the assignments the GPU to 72 system gave you immediately, and PM me the exponent once it had been completed.

This would help me ensure the observation sub-system is working correctly. (Sad to say, I don't myself have a GPU. But it is definitely going to be in the next order for my next workstation.)

Thanks.

I am happy with three extra bits to start with too. I got the first bunch of exponents to 70 but will grab some work to 72 as well. So far, its looking good.

[QUOTE=chalsall;276663]Thanks. And yes, I saw that.



It would help me if you did at least one of the assignments the GPU to 72 system gave you immediately, and PM me the exponent once it had been completed.

This would help me ensure the observation sub-system is working correctly. (Sad to say, I don't myself have a GPU. But it is definitely going to be in the next order for my next workstation.)

Thanks.[/QUOTE]
I'll do this with like two exponents.

[QUOTE=Dubslow;276671]I'll do this with like two exponents.[/QUOTE]

Thanks.

Truly. Much appreciated.

And I have reason to believe we both know (but no one else knows) the exponents we're talking about.

[QUOTE=chalsall;276663]Thanks. And yes, I saw that.



It would help me if you did at least one of the assignments the GPU to 72 system gave you immediately, and PM me the exponent once it had been completed.

This would help me ensure the observation sub-system is working correctly. (Sad to say, I don't myself have a GPU. But it is definitely going to be in the next order for my next workstation.)

Thanks.[/QUOTE]

I did put the five assignments at the top of my queue; this one has already finished.

[CODE]
no factor for M46849237 from 2^68 to 2^69 [mfaktc 0.17-Win barrett79_mul32]
no factor for M46849237 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M46849237 from 2^70 to 2^71 [mfaktc 0.17-Win barrett79_mul32]
[/CODE]

I am reporting it to PrimeNet now.

Time to process on 1/2 of a GTX580:

68—>69 16.5 min
69—>70 33 min
70—>71 65 min

So all of my GTX 460 is about half a GTX 580. (Though mine is barely saturated with one instance, SievePrimes=5000 and avg. wait is 100-150 us.)

[QUOTE=chalsall;276674]Thanks.

Truly. Much appreciated.

And I have reason to believe we both know (but no one else knows) the exponents we're talking about.[/QUOTE]

Wait. Forgot I do each bit level as a separate assignment. Half an hour. (21 minutes according to mfaktc, and then the last bit level of the second will be done.)
The first one has been completely reported.
Edit: Second exponent has also been reported.

[QUOTE=Dubslow;276684]So all of my GTX 460 is about half a GTX 580. (Though mine is barely saturated with one instance, SievePrimes=5000 and avg. wait is 100-150 us.)[/QUOTE]

The two instances of mfaktc actually represent an 80% total load.

[QUOTE=chalsall;276621]The above proposed "GPU to 72 Tool" is now ready and able to assign work to "GPU Workers". And after only three days of "spidering", it has already collected 641 exponents below 50M which have been TFed to less than 71. (457 to less than 70; 152 to less than 69.)

The tool can be accessed at [url]http://gpu.mersenne.info/[/url]

Please give it a try, and let me know if anyone sees any issues, or have suggestions.[/QUOTE]I didn't see anything of interest for those that don't sign-up. Is there any reports or data that might be displayed, to interest those that don't have a GPU at their disposal?

[QUOTE=Uncwilly;276696]I didn't see anything of interest for those that don't sign-up. Is there any reports or data that might be displayed, to interest those that don't have a GPU at their disposal?[/QUOTE]

Have patience, Grasshopper....

[QUOTE=Chuck;276693]The two instances of mfaktc actually represent an 80% total load.[/QUOTE]

Mine is at 90% load, so 90% of a 460 = 40% of a 580...

too lazy to do the rest.

Wait: 580 = 225% of a 460. Sound right?

Oh and chalsall, reported.

Reporting partial results
 

Will it cause any problems when partial results are reported? I report results to PrimeNet irregularly throughout the day, and sometimes not all bit levels will be finished. For example, I just reported

[CODE]
no factor for M46863503 from 2^68 to 2^69 [mfaktc 0.17-Win barrett79_mul32] by "Chuck"
no factor for M46863503 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32] by "Chuck"
[/CODE]

and I have not yet completed and reported bit level 71

[QUOTE=Dubslow;276698]Mine is at 90% load, so 90% of a 460 = 40% of a 580...

too lazy to do the rest.

Wait: 580 = 225% of a 460. Sound right?

Oh and chalsall, reported.[/QUOTE]

Fortunately I am too dim-witted to worry about all these details; I just have enough time and money to get a fast computer and run it 24 hours/day. I let you smart guys figure out all the details and complicated stuff!

[QUOTE=Chuck;276676]
68—>69 16.5 min
69—>70 33 min
70—>71 65 min[/QUOTE]

Do I detect some sort of progression here?

[QUOTE=davieddy;276701]Do I detect some sort of progression here?[/QUOTE]

I think maybe each bit level takes twice as long as the previous.

[QUOTE=Chuck;276703]I think maybe each bit level takes twice as long as the previous.[/QUOTE]
Indeed, he should know that, and knowing him, he does know that, but for whatever reason... I don't know.

[QUOTE=Dubslow;276705]Indeed, he should know that, and knowing him, he does know that, but for whatever reason... I don't know.[/QUOTE]

I think it was humor.....

[QUOTE=chalsall;276697]Have patience, Grasshopper....[/QUOTE]
No problem. I was only making a point of observation. Better to get it functional first, pretty second.

We will need to add this to the Wiki.

[QUOTE=Chuck;276708]I think it was humor.....[/QUOTE]
Aquieus or Vitreous?

[QUOTE=Dubslow;276698]Oh and chalsall, reported.[/QUOTE]

Thanks.

That helps. :smile:

Hi chalsall,

I have two exponents working, low 47M's, factored to 73. I don't think I'll see completion until morning, CDT US time. I'll notify you when they are uploaded.

If I might suggest it, a sign out link might be a good thing.

Thanks for making this possible.

[QUOTE=kladner;276723]Hi chalsall,

I have two exponents working, low 47M's, factored to 73. I don't think I'll see completion until morning, CDT US time. I'll notify you when they are uploaded.
[/QUOTE]

The results have been reported. The assignment and final mfaktc output are attached.

GPU to 72 version 0.4...
 

Thanks to everyone who has helped test the GPU to 72 system.

Thanks particularly to the five individuals who took work from the system, and then submitted results to PrimeNet, so I could be sure my Observation script was sane. (It was.)

Today I have created some publicly available reports to show the status of the system. Such as [URL="http://gpu.mersenne.info/reports/workers/"]Workers Progress[/URL] and [URL="http://gpu.mersenne.info/reports/available/"]Available Assignments[/URL].

I will shortly make additional reports available to signed in Workers, showing more details of their particular pledges and work.

Any chance we can get P-1 status in that second report?

[QUOTE=Dubslow;276839]Any chance we can get P-1 status in that second report?[/QUOTE]

Sure. P-1 status is already in the database; just a slight change to the SQL query.

Please see [URL="http://gpu.mersenne.info/reports/available/"]http://gpu.mersenne.info/reports/available/[/URL] for what (I hope) you asked for.

Now, a deeper question... Would Workers like to be able to request TF work on exponents which have not yet be P-1'ed?

GPU to 72 modus operandi
 

When does the process look for completed assignments? Is is just once per day at the nightly exponent expiration time, or several times throughout the day?

[QUOTE=Chuck;276869]When does the process look for completed assignments? Is is just once per day at the nightly exponent expiration time, or several times throughout the day?[/QUOTE]

The latter.

[QUOTE=chalsall;276855]Sure. P-1 status is already in the database; just a slight change to the SQL query.

Please see [URL="http://gpu.mersenne.info/reports/available/"]http://gpu.mersenne.info/reports/available/[/URL] for what (I hope) you asked for.

Now, a deeper question... Would Workers like to be able to request TF work on exponents which have not yet be P-1'ed?[/QUOTE]

Yes, that's what I was thinking, but looking at it now, it also occurs to me that it would be nice to see that yes/no info for each bit level, instead of composite for all bit levels. Is that possible somehow?

And also, yes and no to your question. No, because P-1 needs to be done regardless, but yes because extra TF bits can help mitigates the need, without eliminating the need entirely.
From another perspective, such an option wouldn't hurt, but it also wouldn't help much.

[QUOTE=Dubslow;276881]Yes, that's what I was thinking, but looking at it now, it also occurs to me that it would be nice to see that yes/no info for each bit level, instead of composite for all bit levels. Is that possible somehow?

And also, yes and no to your question. No, because P-1 needs to be done regardless, but yes because extra TF bits can help mitigates the need, without eliminating the need entirely.
From another perspective, such an option wouldn't hurt, but it also wouldn't help much.[/QUOTE]

You are the typical ICT client Dubslow... Ask for everything you can think of, and then complain when you only get what you asked for....

Eh? Complaining? Just a suggestion. If everybody was happy with what they had, nobody would progress. Intel and AMD wouldn't make better CPU's, and George wouldn't put any more effort into optimizing P95. Obviously I can't make the change myself, but better for you to see and reject then to not see at all.

[QUOTE=chalsall;276837]Today I have created some publicly available reports to show the status of the system. Such as [URL="http://gpu.mersenne.info/reports/workers/"]Workers Progress[/URL] and [URL="http://gpu.mersenne.info/reports/available/"]Available Assignments[/URL].[/QUOTE]Cool!:bounce:

Narrow column width
 

On the "View Assignments" page the column width allocated for "assigned" is very narrow, which makes each entry in the list three lines tall. The "View Completed" pages looks good.

Chuck

[QUOTE=Dubslow;276905]Eh? Complaining? Just a suggestion. If everybody was happy with what they had, nobody would progress. Intel and AMD wouldn't make better CPU's, and George wouldn't put any more effort into optimizing P95. Obviously I can't make the change myself, but better for you to see and reject then to not see at all.[/QUOTE]

Sorry Dubslow... I was trying to be funny... Obviously too long a day to have attempted humour...

Your suggestion is a good one, and has been implemented. There are now two sub-menus for the Available Assignments link... One for [URL="http://gpu.mersenne.info/reports/available/p-1/"]Prior P-1 Work Done[/URL], and one for [URL="http://gpu.mersenne.info/reports/available/nop-1/"]No Prior P-1 Work Done[/URL].

Ooooh, pretty, thanks :)

And it's always difficult on the emotionless blogotubes :)

Mr. P-1, do you still want results files, or just want me to tell you when the batch is done so you can unreserve?

Mr. P-1 my exponents from you are complete; I am getting assignments from GPU to 72 now. I have reported all I got from you.

[QUOTE=Chuck;277388]Mr. P-1 my exponents from you are complete; I am getting assignments from GPU to 72 now. I have reported all I got from you.[/QUOTE]

I can echo this question, with the caveat that I am still finishing off a few of the assignments I got from Mr. P-1, some of which I upped from 70 to 71. I have also added some from GPU to 72 just to make sure I keep the GPU churning one way or another.

But really, I have another question which might be pertinent for others.

In the last few days I got my old system back up and running with Ubuntu 11.1. With [URL="http://www.mersenneforum.org/member.php?u=252"]delta_t[/URL]'s kind assistance, I have it running headless with mprime. I'm even starting to get a few of my minimal Linux skills back. (It's been 5-6 years since I dealt with that.) The "old system" is an Opteron 180, dual core, 2x 128KB L1, 2x 1MB L2, 2.4GHz, with 3GB of RAM. At the moment it is running P-1 on both cores, with 2048MB allowed. The PrimeNet assigned exponents are both low 60M, and both workers are still on the first assignment.

Meanwhile, I have reconfigured my 1090T system (Win7-64, 8GB) to run 3 cores to feed 3x mfaktc, 2 cores doing P-1, and 1 core doing LL with a few partially completed assignments to finish up. Considerable experimenting has shown that the machine remains reasonably usable with mfaktc.ini set for AllowSleep=1 set for all instances. I should note that the GTX-460 is no longer being bothered with feeding the display. I got the onboard ATI back in service for that when I found that, even NOT running mfaktc that the 460 did not make a noticeable difference in screen performance relative to the ATI onboard HD 4290. I suppose that this takes the GTX-460 out of the loop as far as GPU assistance for Photoshop goes, though. I have also configured P95-64 to pause the first worker (LL) when Pshop is running.

In any case, regarding the Linux box, should I just continue with the large (60M) exponent P-1 assignments from PrimeNet? Would any of the wiser persons suggest a different allocation of resources overall?

Thanks!

EDIT: And one other question--should I be pushing all assignments (or at least PFs) to 72?

Kladner:
Sounds like a fine plan...although I'd push all my exponents by exactly one bit, so as to maximise the number of factors you find.

Thanks, Christenson!

[QUOTE=Dubslow;277290]Mr. P-1, do you still want results files, or just want me to tell you when the batch is done so you can unreserve?[/QUOTE]
.

[QUOTE=Dubslow;277290]Mr. P-1, do you still want results files, or just want me to tell you when the batch is done so you can unreserve?[/QUOTE]

Sorry for my lack of attention to this project these past few days. With chalsall taking over my role (thank heavens), I've turned my attention to other matters.

If you've done everything I've given you, then that's all I need to know.

[QUOTE=Mr. P-1;277783]Sorry for my lack of attention to this project these past few days. With chalsall taking over my role (thank heavens), I've turned my attention to other matters.

If you've done everything I've given you, then that's all I need to know.[/QUOTE]

I'll be finishing the tasks you assigned to me over the weekend, I'll let you know.

I just tried to get 100 assignments to 2^70, <53M from GPU to 72 for the machine that is about to finish one of P-1's jobs, but got only 93. "There are no more" is basically what it said.

However, the "Available Assignments" lists plenty of them for all categories. I'm not sure if I unchecked the "P-1 done" box, but for both P-1 done and not done there are hundreds of exps available in <69 and 69 ... is there some other limit? Or a display bug in the "Available Assignments" (I already tried reloading with no change)?

If we really cleared the available 2^69's - very good ...

Mr P-1:
I have temporarily lost access to the computer where I was completing your assignments. The group that owns it has been moving and my work has made me unable to meet up with them (on the required weekinight) to restore my after-hours access. There's a very good chance that a major part has been completed, but is simply waiting for me to come in and upload the "results.txt" file.

Ckdo: Chalsall's tool also seems to cover the 25-30M area...found out accidentally when looking for some low bit levels...I have finished the set you gave me, found one more factor.

[QUOTE=Christenson;278083]Mr P-1:
I have temporarily lost access to the computer where I was completing your assignments.[/QUOTE]

Such things happen. I'll wait until you regain access.

Hey Mr. P-1, either do you have a script to transform mfaktc results files to mfaktc worktodo files, or should I use awk or sed or something else to implement such a script?

[QUOTE=Christenson;278083]Ckdo: Chalsall's tool also seems to cover the 25-30M area...found out accidentally when looking for some low bit levels...I have finished the set you gave me, found one more factor.[/QUOTE]

Yup -- my spider is now grabbing expiring DCs which are less than 28M which have been TFed to less than 69 "bits".

Christenson, the fact that you (and some others) were assigned these by the Tool was actually a mistake -- I intended to only give them to those who explicitly asked for them (at [URL="http://gpu.mersenne.info/account/getassignments/dc/"]http://gpu.mersenne.info/account/getassignments/dc/[/URL]; available via a sub-menu on the "Get Assignments" menu).

If you, or anyone else, don't want to do the DC TF work assigned, I've also added a "Unreserve Assignments" function on the View Assignments report.

Chalsall-

Your ever-evolving spider becomes more amazing by the day. It's turning into a veritable Swiss Army Knife, or Leatherman Multipurpose Tool. It's been fun watching its development.

Thanks for all of your efforts!

Range limits for DC
 

ckdo was giving me exponents through the end of the 28M range; if fact I still have 603 of them still left to go. Should the DC TF range be extended another 1M?

Chuck

[QUOTE=chalsall;278169]Yup -- my spider is now grabbing expiring DCs which are less than 28M which have been TFed to less than 69 "bits".

Christenson, the fact that you (and some others) were assigned these by the Tool was actually a mistake -- I intended to only give them to those who explicitly asked for them.[/QUOTE]

Chalsall:
I wasn't upset..I was actually looking for the EASIEST (or most effective) TF assignments available...and all I need is one factor to have made a positive difference. Running a small batch of anything has a name for me: just for fun.:smile:...I'll have to go see if I have more than "no factor found" lines to turn in tonight.

The simplest (IMO) fix is to let the lower bound of the standard exponent range be 45M, as shown in the box where it's zero now. The most informative is to give approximate equivalent bit levels by exponent...That is, the 45M range at 72 bits is equivalent to the 25M range at 68 bits, taking into account that TF in the 25M range only knocks out 1 LL test when a factor is found, is twice as hard to do for the same bit level at 25M, and a quarter as much work is involved in the now-unneeded LL test at 25M in the first place.

****
That is, according to me, myself, and I,and noone else, the most effective GPU TF assignment is the one that will, on average, save the largest amount of LL testing for the least effort. That is, saving two LL tests counts double saving one LL test, and saving a 50M LL test counts double a 25M LL test. (Never mind RDS's NlogN*loglogN cost of multiplying, the change in loglogN is too small to matter). Likewise, a TF that takes double the number of candidates, because its exponent is half-sized, costs double, so halves the efficiency.

These are the *BEST* assignments to get out of your tool, 25 or 50 or 100 at a time!
****

And, before the music starts playing (let's have it nutcracker suite, please!) or someone gets the wrong idea, let me say that I like your tool and intend to continue using it, with or without upgrades.....

Does PrimeNet check factors that are submitted to it? I remember hearing that somewhere, but I want to make sure...

[QUOTE=Dubslow;278223]Does PrimeNet check factors that are submitted to it? I remember hearing that somewhere, but I want to make sure...[/QUOTE]Yes, It checks factors that are submitted to ensure that they are indeed factors of the associated exponent. The test takes a trivially small amount of time.

[QUOTE=Uncwilly;278225]Yes, It checks factors that are submitted to ensure that they are indeed factors of the associated exponent. The test takes a trivially small amount of time.[/QUOTE]
With any luck (ask Eric!) the exponent won't have a factor:smile:

David

[QUOTE=Dubslow;278154]Hey Mr. P-1, either do you have a script to transform mfaktc results files to mfaktc worktodo files, or should I use awk or sed or something else to implement such a script?[/QUOTE]

After a decent amount of gnashing, I got a basic awk script to work. The last problem is that the results were reported for each bit, and most of them were two bits of work. I want to be able to write them as one line of work, to give the worker the option of doing the bit levels together or not. I got something that checks the previous line and can write a combined worktodo line, but the problem is the previous line is still there, i.e.

Factor=59234982,69,70
Factor=59234982,69,71

My current script is attached.

Umm...dubslow...the right tool for this isn't sed, or awk...it's perl.

Perl is a full programming language, so you can form a hash of all the work done in the file, then retrieve it in exponent order and look at your variables/parameters to decide whether to print Factor=x,69,70 or Factor=x,69,71 as you please.

Note that a hash in perl is a native data structure, so you are only fooling around with using it, not trying to make it work.

[QUOTE=Christenson;278253]Umm...dubslow...the right tool for this isn't sed, or awk...it's perl.

Perl is a full programming language, so you can form a hash of all the work done in the file, then retrieve it in exponent order and look at your variables/parameters to decide whether to print Factor=x,69,70 or Factor=x,69,71 as you please.[/QUOTE]

I agree with Christenson, Dubslow. They don't call Perl "The Internet's Duct tape" for nothing...

If you sent me a sample input file, and the desired resultant output in another file, I could throw together a little Perl script for you in a few minutes.

As you might be able to tell by now, I do Perl and RegEx for a living.... :smile:

[QUOTE=Christenson;278253]Umm...dubslow...the right tool for this isn't sed, or awk...it's perl.[/QUOTE]

I disagree. This task lies right in AWK's sweet spot. Dublsow, however (no disrespect) didn't do it the AWK way.

[QUOTE]Perl is a full programming language, so you can form a hash of all the work done in the file, then retrieve it in exponent order and look at your variables/parameters to decide whether to print Factor=x,69,70 or Factor=x,69,71 as you please.[/QUOTE]

AWK can do this as well.

Here's my script which fixes his problem. It could easily be modified to output the results in exponent order.

[code]#! /bin/awk -f

BEGIN {
FS="[ M^]"
ORS="\r\n"
}

$5 == expo {
bitb = $11
next
}

$5 ~ /[0-9]./ {
if ( expo ) print "Factor=" expo "," bita "," bitb
bita = $8
bitb = $11
expo=$5
}

END {
if ( expo ) print "Factor=" expo "," bita "," bitb
}[/code]

[QUOTE=Mr. P-1;278301]I disagree. This task lies right in AWK's sweet spot. Dublsow, however (no disrespect) didn't do it the AWK way.[/QUOTE]

As we quickly learn in computer science and the maths (and probably other domains as well)...

There is often more than one way to solve a problem.... :smile:

To all of the above: I put that script together after about 30-60 minutes of going through [URL="http://www.grymoire.com/Unix/Awk.html"]here[/URL]. I don't know an ounce of programming besides. I literally know no perl, and the only awk I know is what I posted. My only experience in anything related is a novice level of bash, if that.

(Also I did think of a way to do what I wanted within what I know, but your method does look cleaner Mr. P-1.) But this will have to wait anyways, I need to go eat lunch. Also, see [url]http://www.mersenneforum.org/showthread.php?t=16228[/url] for why I'm doing this; there's around 2000 TF that needs to be done, and I'd like some help.

[QUOTE=Mr. P-1;278301]
[code]#!/bin/awk -f

BEGIN {
FS="[ M^]"
ORS="\r\n"
}

$5 == expo {
bitb = $11
next
}

$5 ~ /[0-9]./ {
if ( expo ) print "Factor=" expo "," bita "," bitb
bita = $8
bitb = $11
expo=$5
}

END {
if ( expo ) print "Factor=" expo "," bita "," bitb
}[/code][/QUOTE]
In support of learning something: $5 == expo and $5 ~ regex are conditions attached to running the following functions? ( expo ) only tests if that variable is defined? What does it test? And how exactly does 'next' work? (I can guess it's purpose but want to be sure of details.)

Also, what's a hash?
Also also, how could this be modified to output to exponent order? It gets harder and harder to compare lines further and further away, and to order it you'd need to check the whole file...

If the end of the week is not too late, then send (PM) me a bunch of, say, 100 to 500 of them, in mfaktc input format. I am going to try to play with mfaktc for the first time in my life during the coming weekend and I think that is better to have some input to test, i.e. to do some useful work instead of playing around, but I don't want to be forced to figure out how to transform a list of numbers into a format that mfaktc would understand (I know nothing about mfaktc yet). You save me of this pain and I save you of a part of exponents.

[QUOTE=Dubslow;278403]
Also, what's a hash?
[/QUOTE]

Imagine you have a million nice colored little balls in a bag and you see a new one with a nice red and blue pattern in the shop, but you only want to buy it if you do not have one like that already. Then you have to go home and see if you have it already, but you can't search your bag, ball by ball, this will take ages, but you also can't take the ball with you to compare. And remembering the color's pattern is impossible for a normal man. Then you invent a method (function, or hash) to codify the color patterns, in a way that is easy for you to remember, and in a way that can also be "sorted" at a glance. For example you associate a digit to each color, red=1, orange=2, etc, a digit to each pattern, triangles=1, diamonds=2, leaves=3, etc. Then use 3 digits for colors, from the one most dominant to the one less dominant, two digits for patterns. Then the ball 32147 is easy to remember. Then you go home and look in your big bag and sort your balls in 10 small bags, all balls starting with 1 in the first bag, all starting with 2 in the second, etc. Then you repeat for each small bag: put 10 boxes inside of each bag, and sort in the first box all the balls which have the second digit 1, put in the second bag all the balls having the second digit 2, etc. Do this for each small bag, eventually go one step deeper adding 10 little boxes inside of each box (that is inside of each bag) and sort for the third digit.

Congratulation, you just made a hash table. Now it is quite easy to search for a ball: from the first three digits you know exactly which small box to look into, and in that box there are not so many balls, you can look to them one by one.

The hash variables are not different. They make "hashes" of the lines automatically, and for each line they read (this is the new ball you see on the shop), the value is hashed and compared to what you already read. If the hash match, then the value is checked, and if it matches, then the function (pattern) is applied. This makes sense because checking the hash is much faster then checking the value/pattern/function/etc effectively. The method is used for complicate data that you can not sort (if you can sort it, then a quick search or binary search will be much faster) or for which the comparison is complicate (as comparing the patterns, you better make hashes -functions- of patterns and compare the hashes -values of the fucntions). In fact, it is exactly same amount of work as you said, "checking the whole file", but this is done faster, optimal, and/or with only one read, etc, because somebody else did the work already (implemented the hash variables), hash tables are short and can be kept in memory, etc.

For details [URL="http://en.wikipedia.org/wiki/Hash_table"]see here[/URL].

Laurv:
The mfaktc parser isn't difficult. It's in C, and straightforward. But don't put too much work into it, as I have a re-write waiting for me to test it that is a little smarter...I set it up so there is exactly one interpreter of the lines in worktodo.txt, and so assignments are beginning to be handled as structs. The subtlety here is that worktodo has to be edited when an assignment is done, and anything skipped over in finding the assignment has to be ignored silently.

Definitions, from me (and Chalsall can correct them):
Hash := associative hash := store (key, value) pairs retrievable by key := content-addressable storage
Regular Expression (Regex): Powerful syntax for dissecting and re-assembling strings as needed.

Chalsall's perl script would have been shorter than Mr P-1's awk script. I don't do perl often enough to do it without the book, being a classical C programmer during the day. Also, a "real" text editor would be able to be programmed quickly to do the editing you need on a one-time basis. I personally use Codewright, (now from Borland, has a small following) but this is not open source. It's worth learning enough emacs or other editor to learn how to do this, because you need this kind of editing from time to time.

Skinning a cat
 

[QUOTE=chalsall;278304]As we quickly learn in computer science and the maths (and probably other domains as well)...

There is often more than one way to solve a problem.... :smile:[/QUOTE]
I related this anecdote to Ernst the other week, and can't
resist retelling it now.

About 15 years ago I had to decompress JPEGs on a 386.
I devized a fast inverse discrete cosine transform which used
only 11 multiplies. I had written it out in BASIC.
Knowing that a guy called Chen had done similar, we
checked it out. It was displayed in C, and it was identical
(verbatim) to mine.

In the way smart colleagues do, my mate Steve casually
remarked:
"Just goes to show there's only one way to skin a cat."

Happy Days:smile:

David

Oh also Mr. P-1 my work that you assigned me a while ago was completed earlier today.

[QUOTE=Dubslow;278403]In support of learning something: $5 == expo and $5 ~ regex are conditions attached to running the following functions?[/quote]

Basically yes. In the vernacular of AWK, these are "patterns", while the associated function is the corresponding "action".

[QUOTE]( expo ) only tests if that variable is defined? What does it test?[/QUOTE]

It actually tests if it evaluates to true. Undefined evaluates to false. 0 is false. "" is false. any other number or non-blank string is true.

[QUOTE]And how exactly does 'next' work? (I can guess it's purpose but want to be sure of details.)[/QUOTE]

Proceed to the next line of input (or END if there is no more input) do not parse any more patterns or actions for this line.

[QUOTE]Also, what's a hash?[/QUOTE]

Most programming languages (and all useful ones!) allow you define arrays of values, usually indexed by small integers. An associative array or hash is similar, except that it can be indexed by arbitrary values defined by the programmer.

[QUOTE]Also also, how could this be modified to output to exponent order? It gets harder and harder to compare lines further and further away, and to order it you'd need to check the whole file...[/QUOTE]

No, you use a hash indexed by the exponent. You don't compare lines. You check to see if current exponent is already an index in the hash.

And here's the script so modifed. Admittedly asorta is a GAWK extension, so this would perhaps not be so easy with other versions.

[code]#! /bin/awk -f

BEGIN {
FS="[ M^]"
ORS="\r\n"
}

$5 in bita {
if (bita[$5] > $8) bita[$5] = $8
if (bitb[$5] < $11) bitb[$5] = $11
next
}

$5 ~ /[0-9]./ {
bita[$5] = $8
bitb[$5] = $11
}

END {
n = asorti(bita, expo)
for (i = 1; i <= n; i++) print "factor=" expo[i] "," bita[expo[i]] "," bitb[expo[i]]
}[/code]

I would be interested to see perl versions of both scripts.

[QUOTE=Mr. P-1;278500]Most programming languages (and all useful ones!) allow you define arrays of values, usually indexed by small integers. An associative array or hash is similar, except that it can be indexed by arbitrary values defined by the programmer[/QUOTE]

Indeed. Hash arrays (AKA Associative arrays) are an incredibly powerful tool in the programmer's tool chest.

If I can give a real world example (at least to we GIMPSers... :smile:)

Say you had a worktodo.txt file like (note the exponents are out of order, and the duplicate exponent "2"):

[CODE]Factor=N/A,11,7,11
Factor=N/A,2,1,2
Factor=N/A,7,5,7
Factor=N/A,5,4,5
Factor=N/A,2,3,4[/CODE]

But you preferred to do each "bit" level individually, and from the lowest exponent up. The following code could do the transform for you:

[CODE]#!/usr/bin/perl

while (<>) {
$_ =~ /Factor=([^,]*),(\d*),(\d*),(\d*)/;

$AID{$2} = $1;

if ($From{$2} > $3 || !defined($From{$2})) { $From{$2} = $3; }
if ($To{$2} < $4 || !defined($To{$2})) { $To{$2} = $4; }
}

sub sort_by_number { $a <=> $b; }

foreach $key (sort sort_by_number keys %AID) {
for ($cnt = $From{$key}; $cnt < $To{$key}; $cnt++) {
$NewTo = $cnt+1;
print "Factor=${AID{$key}},${key},${cnt},${NewTo}\n";
}
}[/CODE]

If you put this Perl code into a file called stepbystep.pl and made it "executable", you could then at the command line execute:

[CODE]./stepbystep.pl < worktodo.txt >new_worktodo.txt
[/CODE]

This would result in new_worktodo.txt containing:

[CODE]Factor=N/A,2,1,2
Factor=N/A,2,2,3
Factor=N/A,2,3,4
Factor=N/A,5,4,5
Factor=N/A,7,5,6
Factor=N/A,7,6,7
Factor=N/A,11,7,8
Factor=N/A,11,8,9
Factor=N/A,11,9,10
Factor=N/A,11,10,11
[/CODE]

Note that an Associated array (in Perl) is referenced by a % symbol (e.g. "%AID" in the above), but the value of the array based on the key is referenced by, in the above example, "$AID{$key}" (replace "$key" with whatever variable you'd like).

One of the reasons Associative arrays are so powerful, beyond the speed, is they are effectively "sparse arrays". As in, the memory consumed is a function of the number of elements, not the minimum and maximum values of the keys. In Perl, executing the two statements "$Array[1] = 1; $Array[1000000000]=1" will result in an "Out of memory error" on most computers because the interpreter will try to allocate an array with one billion rows only to store two values.

Does that help at all, or only confuse you more?

[QUOTE=Mr. P-1;278509]I would be interested to see perl versions of both scripts.[/QUOTE]

Please provide a sample input file, and I'd be happy to produce the Perl...

(I always enjoy language wars... :smile::smile::smile:)

[QUOTE=chalsall;278513]Please provide a sample input file, and I'd be happy to produce the Perl...

(I always enjoy language wars... :smile::smile::smile:)[/QUOTE]

[code]no factor for M51795493 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51796097 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51796097 from 2^70 to 2^71 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51813887 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51817247 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51827527 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51827537 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32][/code]

There could also be factor found lines that don't match this format, but there aren't in the file I have to hand. Just assume that if the fourth word is M followed by a number, then it's in this format.

[QUOTE=chalsall;278512]Indeed. Hash arrays (AKA Associative arrays) are an incredibly powerful tool in the programmer's tool chest.

If I can give a real world example (at least to we GIMPSers... :smile:)[/QUOTE]

Here's my AWK version of the same program:

[code]#! /usr/bin/awk -f

BEGIN {
FS=","
}

/^Factor=[^,]*,[0-9]*,[0-9]*,[0-9]*$/ {
if ( !($2 in from) || from[$2] > $3) from[$2] = $3
if ( !($2 in to) || to[$2] < $4) to[$2] = $4
}

END {
n = asorti(from, expo)
for (i = 1; i <= n; i++) for (j = from[expo[i]]; j < to[expo[i]]; j++) print "factor=" expo[i] "," j "," j+1
}[/code]

With the greatest of respect, I think it's more elegant than yours. There's no explicit "while" loop. AWK just "knows" that it has to loop on the input lines. Similarly there's no explicit reference to the input line. Again AWK just "knows" what the regexp has to match. Finally the "pattern" logic (the rexexp) is kept separate from the "action" logic, which you can't really do with perl.

Unfortunately the prog doesn't quite work. Here's the output:

[code]factor=11,7,8
factor=11,8,9
factor=11,9,10
factor=11,10,11
factor=2,1,2
factor=2,2,3
factor=2,3,4
factor=5,4,5
factor=7,5,6
factor=7,6,7[/code]

The problem is that AWK treats array indexes as strings, hence sorts "11" before "2". This is easy to fix:

[code]#! /usr/bin/awk -f

BEGIN {
FS=","
}

/^Factor=[^,]*,[0-9]*,[0-9]*,[0-9]*$/ {
expo[$2]=$2
if ( !($2 in from) || from[$2] > $3) from[$2] = $3
if ( !($2 in to) || to[$2] < $4) to[$2] = $4
}

END {
n = asort(expo)
for (i = 1; i <= n; i++) for (j = from[expo[i]]; j < to[expo[i]]; j++) print "factor=" expo[i] "," j "," j+1
}[/code]

However the mere fact that I've had to work around a langauge wart detracts from the sublime simplicity and elegance of the original.

[QUOTE=Mr. P-1;278548]no factor for M51795493 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51796097 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51796097 from 2^70 to 2^71 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51813887 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51817247 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51827527 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32]
no factor for M51827537 from 2^69 to 2^70 [mfaktc 0.17-Win barrett79_mul32][/QUOTE]

Given the above input, the following Perl code would produce the following output:

[CODE]#!/usr/bin/perl

while (<>) {
$_ =~ /.*M(\d*) from 2\^(\d.)* to 2\^(\d.)/;
$Exp{$1} = $1;
if ($From{$1} > $2 || !defined($From{$1})) { $From{$1} = $2; }
if ($To{$1} < $3 || !defined($To{$1})) { $To{$1} = $3; }
}

foreach $key (keys %Exp) {
print "Factor=${key},${From{$key}},${To{$key}}\n";
}[/CODE]

[CODE]Factor=51827527,69,70
Factor=51813887,69,70
Factor=51827537,69,70
Factor=51795493,69,70
Factor=51817247,69,70
Factor=51796097,69,71[/CODE]

[QUOTE=Mr. P-1;278568]With the greatest of respect, I think it's more elegant than yours. There's no explicit "while" loop. AWK just "knows" that it has to loop on the input lines. Similarly there's no explicit reference to the input line. Again AWK just "knows" what the regexp has to match. Finally the "pattern" logic (the rexexp) is kept separate from the "action" logic, which you can't really do with perl.[/QUOTE]

With the greatest of respect back, there is sometimes something to be said for readability vs. the fewest characters written. AWK is a declarative language. Much like Forth.

Perl is a procedural language. Much like C or C++ (or Java et al).

And, further, can you tell us all how you get a MySQL database handle in AWK?

[QUOTE=chalsall;278579]With the greatest of respect back, there is sometimes something to be said for readability vs. the fewest characters written.[/QUOTE]

Readability is in the eye of the reader. I find the AWK versions more readable than the perl ones, because I'm more familar with the language, and probably also because I'm more familiar with my own coding style

[QUOTE]AWK is a declarative language. Much like Forth.

Perl is a procedural language. Much like C or C++ (or Java et al).[/QUOTE]

I don't see a lot of difference between them. In your script:

[code]$_ =~ /.*M(\d*) from 2\^(\d.)* to 2\^(\d.)/;[/code]

if I understand it correctly, tell it how to recognise and parse the input lines it wants to work on. That's declarative. The rest is procedural. The difference with AWK is that it provides separate pattern (declarative) and action (procedural) spaces.

[QUOTE]And, further, can you tell us all how you get a MySQL database handle in AWK?[/QUOTE]

No idea. Never had to do that. Not sure why I'd want to with AWK. It sounds more like a job for perl.

I'm not here arguing that AWK is a better language than perl. I took issue with the assertion by Christenson that "the right tool for this isn't sed, or awk". I think I've shown that it is. Of course, perl is also a fine tool for this task too.

Christenson also said "Chalsall's perl script would have been shorter than Mr P-1's awk script." I don't think that's been demonstrated either. It looks to me as though our scripts both have pretty much the same components, except that you had to tell perl to do some things (such as loop on the input lines) that AWK does without being told.

But that's only useful if what AWK does without being told is what you want it to do. AWK was designed to be very good a performing a particular task, namely reading records (by default, lines) parsing them into fields (by default white-space delimited), and then manipulating these in various not-too-complicated ways. The closer the actual problem fits this paradigm, the better language choice AWK will be.

So I would say that the real difference between AWK and perl is not that one is declarative and one procedural, it's that one is special purpose, while the other is general purpose.

[QUOTE=Mr. P-1;278663]Christenson also said "Chalsall's perl script would have been shorter than Mr P-1's awk script." I don't think that's been demonstrated either. It looks to me as though our scripts both have pretty much the same components, except that you had to tell perl to do some things (such as loop on the input lines) that AWK does without being told.[/QUOTE]

You are correct. Although my script was only 12 lines vs. your 22, mine contained 299 characters while yours contained 302 (with indentation removed from both scripts). Basically, identical.

[QUOTE=Mr. P-1;278663]So I would say that the real difference between AWK and perl is not that one is declarative and one procedural, it's that one is special purpose, while the other is general purpose.[/QUOTE]

Agreed. :smile:

[QUOTE=chalsall;278687]You are correct. Although my script was only 12 lines vs. your 22, mine contained 299 characters while yours contained 302 (with indentation removed from both scripts). Basically, identical.

<snip>
Agreed. :smile:[/QUOTE]

Boo! You call this a language war. You sissies :) We want a real fight!!

[QUOTE=garo;278796]Boo! You call this a language war. You sissies :) We want a real fight!![/QUOTE]

The shortest possible code in our modern age is management speak:

"Turn this into that." :smile:

[QUOTE=Mr. P-1;278509]And here's the script so modifed. Admittedly asorta is a GAWK extension, so this would perhaps not be so easy with other versions.

[code]#! /bin/awk -f

BEGIN {
FS="[ M^]"
ORS="\r\n"
}

$5 in bita {
if (bita[$5] > $8) bita[$5] = $8
if (bitb[$5] < $11) bitb[$5] = $11
next
}

$5 ~ /[0-9]./ {
bita[$5] = $8
bitb[$5] = $11
}

END {
n = asorti(bita, expo)
for (i = 1; i <= n; i++) print "factor=" expo[i] "," bita[expo[i]] "," bitb[expo[i]]
}[/code][/QUOTE]

I've just spotted an error in the above. The pattern

[code]$5 ~ /[0-9]./[/code]

should have been

[code]$5 ~ /[0-9]+/[/code]

The same error is present in the earlier non-hashed script. dubslow, if you're actually using either of them, I suggest you correct it.

[QUOTE=chalsall;278802]The shortest possible code in our modern age is management speak:

"Turn this into that." :smile:[/QUOTE]

The ideal computer language has just one statement:

"Do what I want".

[QUOTE=chalsall;278687]You are correct. Although my script was only 12 lines vs. your 22, mine contained 299 characters while yours contained 302 (with indentation removed from both scripts). Basically, identical.[/QUOTE]

Mine contained a whole statement:

[code] ORS="\r\n"[/code]

taken from dublow's original, which as far as I can see is an adaptation to a Windows environment. It's not necessary in *nix. You also used a shorter variable name: "$To" versus "bita".

I could have merged the two tests. Instead of

[code]$5 in bita {
if (bita[$5] > $8) bita[$5] = $8
if (bitb[$5] < $11) bitb[$5] = $11
next
}

$5 ~ /[0-9]+/ {
bita[$5] = $8
bitb[$5] = $11
}[/code]

I could have written

[code]$5 ~ /[0-9]+/ {
if (! ($5 in bita) || bita[$5] > $8) bita[$5] = $8
if (! ($5 in bitb) || bitb[$5] < $11) bitb[$5] = $11
}[/code]

[QUOTE=Mr. P-1;278849]The ideal computer language has just one statement:

"Do what I want".[/QUOTE]
That is implemented already. The computers do what you told them to do, and not what you intended to tell them to do. :smile: (or what you believe that you told them to do).

Indeed, it is for Winblows. I was wondering about the regex, but what it output worked just fine, seeing as the only numbers to be parsed are the ones we need.

:blush:Perhaps it is time I learned awk...but what I would say of perl is:
1) It is a GENERAL tool. I'd argue that unless the user has a lot of work to do that the specific tool is really good at, the general tool is better. This is not the case here; this is a one-off that will soon be forgotten when the task is over.
2) Perl is designed as a conscious attempt to minimise the amount of programming effort for one-off problems.
3) If sed can do it, perl has a "sed" mode.

Now, let's fight!!!:boxer::rant::furious::glare::cmd::squash::deadhorse::raman::fight::dnftt::tank::ouch:
Anyone know what I should look at on the net besides "man awk"?

[url]http://www.grymoire.com/Unix/Awk.html[/url] is what I used to crap together my not-a-program.