mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Software

Reply
 
Thread Tools
Old 2010-09-29, 18:33   #1
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

177816 Posts
Default ECMNet Feature Requests

I haven't touched ECMNet in about four years, so it's probably time for me to do a few updates to it. It doesn't appear that much needs to be done as I haven't gleaned much from what I've read in this forum (or others). I have a few things that I intend to do for the next release including:
  • Removing support for sbfactor. AFAIK, it is no longer used by anybody.
  • Using the socket code I've been using for PRPNet.
  • Using the mail code I've been using for PRPNet, which added SMTP authentication support.
  • Adding support for -m in the client.

I have a few other ideas for the next release, so I'm posting them here to see what users are interested in. These include:
  • Multi-thread the server
  • Use MySQL instead of the candidates file.
  • Allow client to grab multiple numbers to work on between communications with the server.

I'm open to other ideas. Please post them here.
rogue is offline   Reply With Quote
Old 2010-09-30, 06:20   #2
debrouxl
 
debrouxl's Avatar
 
Sep 2009

977 Posts
Default

Before giving up on MacOS X and installing my favorite Linux distro on my work's Mac, I remember having to make minor tweaks to the code and/or building system for the compilation on MacOS X to compile the *nix version (instead of the Windows version).

Quote:
Use MySQL instead of the candidates file.
The candidates file is easier to set up and edit than a MySQL server + database. It was pretty painless to set up an ECMNet server on my own computer to raise one of the XYYXF integers to t50.
Maybe, at least, sqlite could be used instead of MySQL ?
debrouxl is offline   Reply With Quote
Old 2010-09-30, 12:41   #3
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

23×751 Posts
Default

Quote:
Originally Posted by debrouxl View Post
Before giving up on MacOS X and installing my favorite Linux distro on my work's Mac, I remember having to make minor tweaks to the code and/or building system for the compilation on MacOS X to compile the *nix version (instead of the Windows version).

The candidates file is easier to set up and edit than a MySQL server + database. It was pretty painless to set up an ECMNet server on my own computer to raise one of the XYYXF integers to t50.
Maybe, at least, sqlite could be used instead of MySQL ?
I'll fix any compile issues on OS X before I release it.

I'm not familiar with sqlite. MySQL is really easy to get going and runs on multiple platforms. I (and many others) use it for PRPNet without any problems. Any database I use would have to support transactions, i.e. no auto-commit.
rogue is offline   Reply With Quote
Old 2010-09-30, 13:16   #4
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

23×751 Posts
Default

Quote:
Originally Posted by rogue View Post
I'll fix any compile issues on OS X before I release it.

I'm not familiar with sqlite. MySQL is really easy to get going and runs on multiple platforms. I (and many others) use it for PRPNet without any problems. Any database I use would have to support transactions, i.e. no auto-commit.
I did some further reading on sqlite and although it probably could work, the problem is that the PRPNet server is multi-threaded and I lock records within transaction allowing multiple clients to update concurrently. From what I have read, sqlite doesn't allow for record level locking which would cause clients to wait unnecessarily for work. ECMNet is a little different in that regard as there tends to be fewer clients.

On the other hand, sqlite might be an option for the client, which is apparently what it is more intended for, a client-side database. I don't see a need for a client-side database at this time.
rogue is offline   Reply With Quote
Old 2010-09-30, 14:06   #5
yoyo
 
yoyo's Avatar
 
Oct 2006
Berlin, Germany

3×197 Posts
Default

Quote:
Originally Posted by rogue View Post
  • Allow client to grab multiple numbers to work on between communications with the server.
This sounds nice. I'm interested in a feature to connect yoyo@home to different ecmnet servers. For this it is requied, that I can fetch e.g. 1000 curves for a number and send done curves and found factors days/weeks later back to the server.
E.g. I get a file with <server> <number> <requied curves>, than I create a number of Boinc Wus out of it. When parts of them are finished (days/weeks later) I send them back.
Of course I have to filter them if they fit to my requirements, because I support only a defined range of digit-length and B1.

yoyo
yoyo is online now   Reply With Quote
Old 2010-09-30, 14:23   #6
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

23×751 Posts
Default

Quote:
Originally Posted by yoyo View Post
This sounds nice. I'm interested in a feature to connect yoyo@home to different ecmnet servers. For this it is requied, that I can fetch e.g. 1000 curves for a number and send done curves and found factors days/weeks later back to the server.
E.g. I get a file with <server> <number> <required curves>, than I create a number of Boinc Wus out of it. When parts of them are finished (days/weeks later) I send them back.
Of course I have to filter them if they fit to my requirements, because I support only a defined range of digit-length and B1.
I don't know the architecture of yoyo@home. I would need to understand that better before I know what you are asking for. If desired, we can talk about this via e-mail and then I can put the results of said discussions into this thread.
rogue is offline   Reply With Quote
Old 2010-10-01, 16:40   #7
jyb
 
jyb's Avatar
 
Aug 2005
Seattle, WA

5×11×29 Posts
Default

Quote:
Originally Posted by rogue View Post
I haven't touched ECMNet in about four years, so it's probably time for me to do a few updates to it. It doesn't appear that much needs to be done as I haven't gleaned much from what I've read in this forum (or others).
[snip]
I'm open to other ideas. Please post them here.
When I use the ecmnet client, I pretty much always want to run multiple copies of it, all sharing most of their config information. In particular, I would like them to all be able to use the same config file. As far as I can tell, ecmnet makes this impossible.

I assume other people like to run multiple clients. What do you all do? Have a different directory for each client, each with a slightly different config file?
jyb is offline   Reply With Quote
Old 2010-10-01, 17:54   #8
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

23·751 Posts
Default

Quote:
Originally Posted by jyb View Post
When I use the ecmnet client, I pretty much always want to run multiple copies of it, all sharing most of their config information. In particular, I would like them to all be able to use the same config file. As far as I can tell, ecmnet makes this impossible.

I assume other people like to run multiple clients. What do you all do? Have a different directory for each client, each with a slightly different config file?
The problem is the the client does i/o to a set of work files that are named based upon the ini settings. With multiple copies of the client running from the same directory, they would overwrite one another. That is the problem that needs to be solved in order to allow what you want to happen. I could add a command line switch that would append a value to the work file name, but if you other ideas, I'm listening.
rogue is offline   Reply With Quote
Old 2010-10-01, 18:25   #9
jyb
 
jyb's Avatar
 
Aug 2005
Seattle, WA

5×11×29 Posts
Default

Quote:
Originally Posted by rogue View Post
The problem is the the client does i/o to a set of work files that are named based upon the ini settings. With multiple copies of the client running from the same directory, they would overwrite one another. That is the problem that needs to be solved in order to allow what you want to happen. I could add a command line switch that would append a value to the work file name, but if you other ideas, I'm listening.
Yes, in fact the problems are these:
  • The client always looks for "ecmclient.cfg" in the current directory, so all clients have to have the same current directory. That would be fine, but:
  • The log file names are hardcoded, so the clients's log and factors files will all stomp on each other.
  • The suffix for work file names is set in the config file, so all clients will have the same suffix, meaning that they will all stomp on each other.
  • The machine ID is set in the config file and can't be overridden, so different client processes can't have different IDs.

So it seems that the only way to run multiple clients is for each client to have its own directory containing a slightly different config file, and each client process has to run out of its directory. Needless to say, this is a giant pain.

So I made some changes in my local copy that allow multiple clients to share the same config file without any stomping. They can all therefore run from the same directory. The changes are:
  • Use the machine ID to identify each client.
  • Allow the machine ID to be overridden on the command line (with "-ID").
  • Append the machine ID string to the log and factors file names (e.g. ecmclient_myID.log).
  • Recognize a special token in the config file's server descriptor for the work file suffix. If the suffix is specified there as "$ID" then use the machine ID as the suffix. (E.g. work_myID.in).

So now I just set the suffix in the ecmclient.cfg to $ID, and I can fire off any number of clients from the same directory with "ecmclient -ID <string>" and they will all happily coexist (as long as each one has a different <string>, of course). I've been running it this way for the last year or so.

If you want my diffs, let me know.
jyb is offline   Reply With Quote
Old 2010-10-01, 19:04   #10
rogue
 
rogue's Avatar
 
"Mark"
Apr 2003
Between here and the

23×751 Posts
Default

Quote:
Originally Posted by jyb View Post
Yes, in fact the problems are these:
  • The client always looks for "ecmclient.cfg" in the current directory, so all clients have to have the same current directory. That would be fine, but:
  • The log file names are hardcoded, so the clients's log and factors files will all stomp on each other.
  • The suffix for work file names is set in the config file, so all clients will have the same suffix, meaning that they will all stomp on each other.
  • The machine ID is set in the config file and can't be overridden, so different client processes can't have different IDs.

So it seems that the only way to run multiple clients is for each client to have its own directory containing a slightly different config file, and each client process has to run out of its directory. Needless to say, this is a giant pain.

So I made some changes in my local copy that allow multiple clients to share the same config file without any stomping. They can all therefore run from the same directory. The changes are:
  • Use the machine ID to identify each client.
  • Allow the machine ID to be overridden on the command line (with "-ID").
  • Append the machine ID string to the log and factors file names (e.g. ecmclient_myID.log).
  • Recognize a special token in the config file's server descriptor for the work file suffix. If the suffix is specified there as "$ID" then use the machine ID as the suffix. (E.g. work_myID.in).

So now I just set the suffix in the ecmclient.cfg to $ID, and I can fire off any number of clients from the same directory with "ecmclient -ID <string>" and they will all happily coexist (as long as each one has a different <string>, of course). I've been running it this way for the last year or so.

If you want my diffs, let me know.
Why not? Just e-mail them to me.
rogue is offline   Reply With Quote
Old 2010-10-02, 10:42   #11
xilman
Bamboozled!
 
xilman's Avatar
 
"π’‰Ίπ’ŒŒπ’‡·π’†·π’€­"
May 2003
Down not across

237228 Posts
Default

Quote:
Originally Posted by jyb View Post
So it seems that the only way to run multiple clients is for each client to have its own directory containing a slightly different config file, and each client process has to run out of its directory. Needless to say, this is a giant pain.

So I made some changes in my local copy that allow multiple clients to share the same config file without any stomping. They can all therefore run from the same directory. The changes are:
  • Use the machine ID to identify each client.
  • Allow the machine ID to be overridden on the command line (with "-ID").
  • Append the machine ID string to the log and factors file names (e.g. ecmclient_myID.log).
  • Recognize a special token in the config file's server descriptor for the work file suffix. If the suffix is specified there as "$ID" then use the machine ID as the suffix. (E.g. work_myID.in).
I do it slightly differently, in that I use different directories but identically the same config files for each client.First create the directories (I call them cl1, cl2, ..., cln) and in one of them create a config file. Then hard link to that config file to the same name in each of the other directories.

Second create a shell script which contains lines:
Code:
#! /bin/sh
cd /path/to/cl1
/path/to/ecmclient &
cd /path/to/cl2
/path/to/ecmclient &
cd /path/to/cl3
/path/to/ecmclient &
...
cd /path/to/cln
/path/to/ecmclient &
Third, run the shell script. Because the config files are hard linked together, editing any one immediately ensures that all the others are changed at the same time.

If you wish, you could get clever and write the script as a loop which runs from 1 to $1, thereby starting a user-selected number of clients. but I couldn't be bothered. If you wanted to get really clever, you could write a script (probably easier in Perl than sh) which writes customized config files in each directory, creating the directory if required. Again, I couldn't be bothered but if there's interest I could write one easily enough.

Paul
xilman is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Featured request bsquared YAFU 260 2019-12-10 10:30
Requests? Xyzzy Forum Feedback 104 2017-04-02 22:20
Collaboration Requests robert44444uk Prime Gap Searches 2 2017-01-17 07:57
a few simple requests for v5 ixfd64 PrimeNet 44 2010-01-11 20:21
New Year requests masser Sierpinski/Riesel Base 5 22 2007-09-24 21:05

All times are UTC. The time now is 06:40.

Fri Nov 27 06:40:21 UTC 2020 up 78 days, 3:51, 4 users, load averages: 1.12, 1.21, 1.20

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.