[QUOTE=bsquared;274330]Any other thoughts/suggestions/ideas?[/QUOTE]

Another one: keep all data? That is, create new .dat/.job/.p/etc files for each run and never delete old ones at all. Obviously not very disk friendly, but I'll throw it out there. I suppose it could be an option, instead of default behavior... something like "-keepdata" from the command line.

[QUOTE=bsquared;274330] One alternative would be to refuse to do this and notify the user.[/QUOTE]

That is fine when yafu is used alone. In fact, everything is fine as long as yafu is used alone. I can take care (or other user) of preserving/deleting the files I want, and I know for sure better then anyone what files I would need and what work I want to do next, so I can rename, keep, delete, whatever. The problem starts when you combine it with some "wrapper" software as aliqueit.

For example, after you interrupt the aliqueit/yafu combination during a siqs or a nfs, there is no way to continue without losing some work, or plenty of time. If you just restart aliqueit, it will come again to ecm and redo all the work already done before interruption (no way to use -noecm switch, and if you use it in yafu.ini, it will be in efect for the subsequent calls, automated by aliqueit, where in fact you should WANT the ecm be done). Assuming you pass this one somehow (waiting, or playing with the files, like I use to do: I put a plan=noecm in ini file, then after the nfs starts, I edit the ini file, during the nfs is working the file is not locked by yafu, and I add a comment "$" in front of the "plan" line, so subsequent calls from aliqueit, for the next terms, will use the normal plan). Then you pass it, ok. Then nfs or siqs starts, and you start getting hundreds of messages like "please use -R switch if you like to resume", because he finds .dat files from the former (interrrupted) work, and he "refuses" to overwrite them, exactly as you said (this is happening already!).

Now ok, assuming you force it to do resuming, then you will have trouble for the next terms of aliqueit, when he will not delete the dat files, but add to them. So, any path you choose, is bad.

That is the problem. When yafu is used alone No problem, No claim, No argue. One can personally take care of his/her files. But as long as you add aliqueit into equation, things can go bad very fast.

I hope you understand my obfuscated explanations...

Having a -F flag will not help. First improvement that has to be done: if there are files (siqs or nfs) for the same job (same number) already on the folder, then ALL preliminary steps (rho, pp1, pm1, ecm) should be skipped. Eventually notify the user. This would speed up a lot working on aliquotes.

The second, if there are nfs.dat files for another job, rename them to nfs01.dat, nfs02.dat, etc, same as for job files, or siqs files. That would mean some other job was interrupted, and later on the user can decide whar to do with them. Eventually notify the user. The files (in case the user decide he don't need them) could be deleted at once from explorer, total commander, command prompt, etc, during the yafu window is continuing doing its (nfs or siqs) job. This would automatize a lot the "aliquoting" process, and no data would be lost. Keeping maximum 3 or 4 (or just 2?) "old sets" of .dat/.job/.fb files won't hurt the space on hdd. Just my two pence.

[QUOTE=bsquared;274330]
If yafu is told to work on a number, N, that is different from one or more existing files based on a different number, to go ahead and blow away all existing files before starting work on N. One alternative would be to refuse to do this and notify the user. Another alternative would be to look for a flag for permission - maybe something like "-F" for "Force new start", or something.[/QUOTE]

Or use different filenames/folders for different factorizations? This will allow multiple factorizations to share the same (parent) folder.

[QUOTE=LaurV;274337] The problem starts when you combine it with some "wrapper" software as aliqueit.
[/QUOTE]

Agreed.

[QUOTE=LaurV;274337]

If you just restart aliqueit, it will come again to ecm and redo all the work already done before interruption...
[/QUOTE]

There is a flag for this in aliqueit: "-e". That flag causes ecm to be skipped on the first term only (and proceed directly to siqs/nfs). Subsequent terms will be treated normally. Other people had had this problem before :smile:.


[QUOTE=LaurV;274337]

Now ok, assuming you force it to do resuming, then you will have trouble for the next terms of aliqueit, when he will not delete the dat files, but add to them. So, any path you choose, is bad.

[/QUOTE]

A normal completion of yafu's NFS should delete the .dat file, so you shouldn't have this problem as long as everything proceeds normally. Also, in order to keep the user from putting a "-R" switch permanently into the yafu.ini file, maybe aliqueit should have a "-R" switch telling it to resume the current iteration only (similar to -e).

[QUOTE=LaurV;274337]


Having a -F flag will not help. First improvement that has to be done: if there are files (siqs or nfs) for the same job (same number) already on the folder, then ALL preliminary steps (rho, pp1, pm1, ecm) should be skipped. Eventually notify the user. Shis would speed up a lot working on aliquotes.

[/QUOTE]

This is already true for siqs - if yafu's factor() routine (which is what aliqueit -y uses) sees that there is a valid siqs.dat file for the input, it should proceed immediately to siqs and skip all pretesting. That would be harder to implement for nfs detection... but that is what aliqueit's "-e" is for.
[QUOTE=LaurV;274337]
Just my two pence. [/QUOTE]

Thanks for the feedback! and for helping work through these issues.

[QUOTE=bsquared;274339]
There is a flag for this in aliqueit: "-e"
[/QUOTE]

This is an immensely valuable piece of information! Thanks a billion! I was wondering how nobody was thinking to that before! :D

[QUOTE=axn;274338]Or use different filenames/folders for different factorizations? This will allow multiple factorizations to share the same (parent) folder.[/QUOTE]

Good idea! Occasionally the user will have to do "housekeeping", but that is at the discretion of the user. It wouldn't be hard to scan the directory for folder names in order to look for matches of a given input. Not sure how common this use case would be, but that would allow for multiple "ongoing" factorizations of different numbers.

As for what to keep in the folders, I think there could be two levels. A "-keepinfo" flag will keep the .job/.p/.fb and .log files so that one could go back and review old jobs and/or restart from a previously found poly. A "-keepdata" flag will also keep the .dat file.

If neither of these flags is present, then the files will be put at the top level and overwritten (current approach).

How's this sound?

Great, on my opinion.

[QUOTE=bsquared;274342]Good idea! Occasionally the user will have to do "housekeeping", but that is at the discretion of the user. It wouldn't be hard to scan the directory for folder names in order to look for matches of a given input. Not sure how common this use case would be, but that would allow for multiple "ongoing" factorizations of different numbers.

As for what to keep in the folders, I think there could be two levels. A "-keepinfo" flag will keep the .job/.p/.fb and .log files so that one could go back and review old jobs and/or restart from a previously found poly. A "-keepdata" flag will also keep the .dat file.

If neither of these flags is present, then the files will be put at the top level and overwritten (current approach).

How's this sound?[/QUOTE]
It sounds excessively complex.:smile:

A better approach would be to always create a new directory for a new number and always delete the directory upon the completion of a number. No overwriting of files, no need for any new flags, no need for the user to do any housekeeping, and the user can count the number of unfinished factorizations by a directory listing. Simpler to use and simpler to implement.

Disagree with the deleting part. Maybe for a normal user there is not necessary to keep old history. But many times you want to look into old stuff, old polys, troubleshoots, etc. I would vote for the part with keeping the history and letting the user to do the janitorial job, at least if he opted so (by some switch). The default could be to delete all, overwrite all, I don't care, but if I want to do other strange things, why not have a switch for it?

Thanks for the feedback. I do like the idea of keeping it simple. I'll admit that I'm not the best at managing scope creep :)

Also in the back of my mind is a suggestion I got via PM to generate some sort of summary file for each NFS job, similar to what factMsieve does if I understand correctly.

To sum things up: some level of NFS data compartmentalization would be nice in order to prevent problems like what LaurV experienced, and some amount of archiving is also desired by multiple people. I'll see if I can come up with something to deal with each of these.

If you start putting your fingers into it, may I be allowed to suggest some other small improvements? :D For example, in pari, functions like issquare and ispower have two parameters, the second parameter is a pointer where the root will be returned in case it was computed in the process. Now I know that thing is not easy to do (variable number of parameters, and pointer) but maybe could the two functions set some global variable (last_computed_root, or some better name) with the value of the root, in case it was computed in the process, otherwise set that variable to zero (actually set it to zero at the function entry, then if the function pass all masking-bit story and come to extracting the root, set that variable to the root value). This would avoid calling again the square root function after testing (and founding it true) that a number is a square root, for example. Could be more other applications, too. At least it would speedup some of my batch processing stuff.. :D