GPU72's fetching/reporting API

[swl551]

MFAKTO/C, CudaLucas, CudaP1 do not implement any primenet interfacing at all.

What is their future for embedded primenet IO? Mfacktc readme says the following

- automatic primenet interaction (Eric Christenson is working on this) <- specification draft exists; on hold, Eric doesn't want to continue his efforts. :(
- this will greatly increase usability of mfaktc
- George Woltman agreed to include the so called "security module" in
mfaktc for a closed source version of mfaktc. I have to check license
options, GPL v3 does not allow to have parts of the program to be
closed source. Solution: I'll re-release under another license. This is
NOT the end of the GPL v3 version! I'll release future versions of
mfaktc under GPL v3! I want mfaktc being open source! The only
differences of the closed version will be the security module and the
license information.

Do we care? Do we want to provide an interface? Can they call a compiled .dll the prevents them from having to deal with closed source releases?

I know the long term vision is that those products will be native to Prime95.exe. (prime95 detects GPU and uses it). Is that 1 year away, 5 years or so far out it is irrelevant and we could help them now.

That basically restates my previous question: Who/what are the desired consumers of gpu72/api.

[Dubslow]

Originally Posted by chalsall

That would be very inappropriate. Scott K. was involved from the beginning -- I'm a relative newcomer.

I thought that was the most likely response, but I wanted to put it out there.

Originally Posted by chalsall

I do think, however, that something like a "Primenet Development Committee" being created would be a good thing for GIMPS.

As we've been saying for more than a year with little progress

Originally Posted by swl551

MFAKTO/C, CudaLucas, CudaP1 do not implement any primenet interfacing at all.

What is their future for embedded primenet IO? Mfacktc readme says the following


Do we care? Do we want to provide an interface? Can they call a compiled .dll the prevents them from having to deal with closed source releases?

I know the long term vision is that those products will be native to Prime95.exe. (prime95 detects GPU and uses it). Is that 1 year away, 5 years or so far out it is irrelevant and we could help them now.

That basically restates my previous question: Who/what are the desired consumers of gpu72/api.

Eric Christenson is, AFAICT, "afk" from this forum/GIMPS, so mfaktc PrimeNet interaction is going nowhere at the moment.

As for Prime95 natively supporting GPUs, the major problem is that much of the GPU code is more or less independent of the CPU code -- with entirely
different states of development and release cycles. So a new Prime95 version would have to be released any time there's a new CUDALucas or new mfakto or new mfaktc or new P-1, which would get very cumbersome very quickly. There aren't very many ways to make a modularized single executable either (or even if you do split it into multiple executables).

[Aramis Wyler]

Originally Posted by Dubslow

So a new Prime95 version would have to be released any time there's a new CUDALucas or new mfakto or new mfaktc or new P-1, which would get very cumbersome very quickly. There aren't very many ways to make a modularized single executable either (or even if you do split it into multiple executables).

It's true, times have changed. Prime95 is a powerhouse of a program, dealing with lots of different kinds of calculations in different orders/ranges and that is just the kind of work you get when you have a single programmer or a very small group of programmers working on a thing to do that. I think it's an incredible program, myself. Things are a world of different when you have a community of people working on different things together. I think the future of that work is a collections of simple command line programs (like mfactc, mfacto, others) specialized onto a specific kind of work. Mfactc and Mfacto don't compete, but they don't work together either. Different people use different versions based on their hardware, and that's ok. CudaLucas has few competitors. They all do their thing with inputs and outputs, and to bring the average joe user into the project what you need is an interface (Like misfit) and a suite of utilities (mfactc, mfacto, cudalucas, maybe something for cpus or physx cards, other new hardware) that the interface... interfaces with. Other people (like myself) might prefer to not have an interface at all, but just the collectuion of utilities tied together with a script or a set of cron entries or a batch file. It doesn't matter, and it's all good as long as the utilities are simple enough that they can work together.

All of the current ones that we use for gpus are simple enough that they can be managed. I think it would be troublesome if mfactc actually got the primenet api built into it, because that might hinder use of the utilities with others. What we need is a utility that simply manages the other utilities and treats them like a collection. Scott did a lot to fill that gap on the windows side, but I don't know of anything yet on the linux side.

In the end though, the prime95 app is huge. Until we have a suite of utilities that could supercede that for the average users, we need to comply with prime95/primenet's api. That doesn't mean that another api can't be written if someone had the will, but the new api would need to either translate it's output into primenet api code or rebuild all that code from scratch, which is harder to justify if we need to complete the primenet api anyway.

[Aramis Wyler]

Originally Posted by Aramis Wyler

... because the work requests/results/statements that we do agree with can simply be passed on for primenet to deal with.

I need to note, that a lot of this discussion (I think) is relevant specifically because we offer p-1, ll, and dc work from gpu72. If gpu72 were not offering that work from it's own stores but was instead letting primenet handle that work, the gpu72 api would become completely unhinged from primenet's api and could be anything.

(Personally I don't have a problem with the primenet api, but I think it is an important point.)

[chalsall]

Originally Posted by Aramis Wyler

I'd appreciate a restating of gpu72's mission statement, so that we're clear on that. Right now what I have is 'compress the curve(s)' and 'increase the factored bitdepth of candidates primenet releases'. I am proceeding as such, but it is always nice to have a formal and agreed apon mission statement.

OK... I've had some time to think about this, and talked to God...

GPU72's original "mission" was simply to semi-automatically coordinate those doing GPU TFing. Quickly, and to my great surprise, a great amount of "firepower" manifested. Additionally, people asked for other work types to be made available, first it was DCTFing, then P-1'ing, then DCing and LLing.

Over time, this became more and more automated. First with my Submission spider, then the proxy for DC, LL and P-1 work, then with Scott's MISFIT. But for Unix users, TF fetching still requires manual intervention (this will soon be solved as well).

As part of the latter solution, and MISFIT, I've built an API which uses a "pre-shared secret" instead of a "secret hash". These secrets will be on a per-user, per-machine (possibly per-GPU) basis. These will be created by GPU72, and accessed by the user when configuring a new "instance".

Basically what this does is transfer the "Trust relationship" from a closed-source algorithm to instead be based on each user. If the user demonstrates they are untrustworthy, their activity can be blocked.

Now, the exciting news... George has agreed to make GPU72 a "trusted client"!!!

What this means is we will soon be able to accept results via the GPU72 API, and it will then submit the results to Primenet on the user's behalf. No longer will the assignments be issued by GPU, but the results need to be sent to Primenet (although that will still be an option).

My medium-term hope is that this new (well, really a very slightly modified Primenet) API / authentication methodology will be implemented in mfaktx, CUDALucas, MISFIT etc, making my new Fetching spider obsolete. (I expect people will continue to use MISFIT for its load balancing and monitoring abilities.)

My long-term hope is this new methodology will be subsumed into Primenet, so that GPU72 no longer needs to be a intermediary. (GPU72 was always intended to be a temporary solution.)

I have some work to do for this, but most of the back-end (and the new spider) is finished. It's mostly the Web UI -- I'll try to devote some cycles to this this week.

Thoughts and comments welcome.

[swl551]

I think that changes your main players to TheJudger, Bdot, Dublsow.

I'll stand by to see where this goes.

[chalsall]

Originally Posted by swl551

I think that changes your main players to TheJudger, Bdot, Dublsow. I'll stand by to see where this goes.

I don't entirely agree. MISFIT users could benefit from the API's ability to report on what machine(s) have been assigned the work (including possible moves), estimated completion, last update, etc.

[chalsall]

Further to this discussion...

Now that George has granted GPU72 "trusted client" status, it is possible for the API to be anything we want. Full SOAP, REST, et al.

However, I question if this is needed, or a good idea.

In my mind, the current Primenet API is fully functional, with the ability to expand in the future if needed with additional message types (t=[\w\w]). The only issue (as already discussed, but I'll repeat it here) is the "secret hash" which has prevented Open Source code from using it.

Additionally, the more GPU72's API diverges from Primenet's, the less likely it will be adopted by George, Scott K., et al.

Thoughts? Comments? Pointing out fundamental flaws in my reasoning?

[frmky]

Originally Posted by chalsall

My medium-term hope is that this new (well, really a very slightly modified Primenet) API / authentication methodology will be implemented in mfaktx, CUDALucas, MISFIT etc, making my new Fetching spider obsolete.

It would be great if each of these tools can interact automatically with GPU72 with no manual intervention required. For CUDALucas, once owftheevil's bitshift version is fully tested and this API is implemented, it could be used for fully automated first time tests and double checks, including double checks of numbers previously run with CUDALucas.

[henryzz]

Personally it sound like you should abstain from changing any of the current API but add to it as necessary. Does this really put any restriction on what you can do?

[swl551]

Originally Posted by chalsall

Further to this discussion...

Now that George has granted GPU72 "trusted client" status, it is possible for the API to be anything we want. Full SOAP, REST, et al.

However, I question if this is needed, or a good idea.

In my mind, the current Primenet API is fully functional, with the ability to expand in the future if needed with additional message types (t=[\w\w]). The only issue (as already discussed, but I'll repeat it here) is the "secret hash" which has prevented Open Source code from using it.

Additionally, the more GPU72's API diverges from Primenet's, the less likely it will be adopted by George, Scott K., et al.

Thoughts? Comments? Pointing out fundamental flaws in my reasoning?

I can't say I fully understand the reason for the secret hash: Have the user authenticate as part of the API call, over an SSL connection.

For GPU tracking: Each client_x.ini file can contain a GPU_GUID, created by the client app and passed into the API.

Additionally I just have to ask. If the primenet API has been around for so long, and we want to maintain compatibility, why haven't these applications already embraced calling it. What will GPU72/api do that is such a game changer. (that is an honest question, not a challenge of possibilities or vision)


Scott