ECMNet Feature Requests

[rogue]

I haven't touched ECMNet in about four years, so it's probably time for me to do a few updates to it. It doesn't appear that much needs to be done as I haven't gleaned much from what I've read in this forum (or others). I have a few things that I intend to do for the next release including:

• Removing support for sbfactor. AFAIK, it is no longer used by anybody.
• Using the socket code I've been using for PRPNet.
• Using the mail code I've been using for PRPNet, which added SMTP authentication support.
• Adding support for -m in the client.

I have a few other ideas for the next release, so I'm posting them here to see what users are interested in. These include:

• Multi-thread the server
• Use MySQL instead of the candidates file.
• Allow client to grab multiple numbers to work on between communications with the server.

I'm open to other ideas. Please post them here.

[debrouxl]

Before giving up on MacOS X and installing my favorite Linux distro on my work's Mac, I remember having to make minor tweaks to the code and/or building system for the compilation on MacOS X to compile the *nix version (instead of the Windows version).

Use MySQL instead of the candidates file.

The candidates file is easier to set up and edit than a MySQL server + database. It was pretty painless to set up an ECMNet server on my own computer to raise one of the XYYXF integers to t50.
Maybe, at least, sqlite could be used instead of MySQL ?

[rogue]

Originally Posted by debrouxl

Before giving up on MacOS X and installing my favorite Linux distro on my work's Mac, I remember having to make minor tweaks to the code and/or building system for the compilation on MacOS X to compile the *nix version (instead of the Windows version).

The candidates file is easier to set up and edit than a MySQL server + database. It was pretty painless to set up an ECMNet server on my own computer to raise one of the XYYXF integers to t50.
Maybe, at least, sqlite could be used instead of MySQL ?

I'll fix any compile issues on OS X before I release it.

I'm not familiar with sqlite. MySQL is really easy to get going and runs on multiple platforms. I (and many others) use it for PRPNet without any problems. Any database I use would have to support transactions, i.e. no auto-commit.

[rogue]

Originally Posted by rogue

I'll fix any compile issues on OS X before I release it.

I'm not familiar with sqlite. MySQL is really easy to get going and runs on multiple platforms. I (and many others) use it for PRPNet without any problems. Any database I use would have to support transactions, i.e. no auto-commit.

I did some further reading on sqlite and although it probably could work, the problem is that the PRPNet server is multi-threaded and I lock records within transaction allowing multiple clients to update concurrently. From what I have read, sqlite doesn't allow for record level locking which would cause clients to wait unnecessarily for work. ECMNet is a little different in that regard as there tends to be fewer clients.

On the other hand, sqlite might be an option for the client, which is apparently what it is more intended for, a client-side database. I don't see a need for a client-side database at this time.

[yoyo]

Originally Posted by rogue

• Allow client to grab multiple numbers to work on between communications with the server.

This sounds nice. I'm interested in a feature to connect yoyo@home to different ecmnet servers. For this it is requied, that I can fetch e.g. 1000 curves for a number and send done curves and found factors days/weeks later back to the server.
E.g. I get a file with <server> <number> <requied curves>, than I create a number of Boinc Wus out of it. When parts of them are finished (days/weeks later) I send them back.
Of course I have to filter them if they fit to my requirements, because I support only a defined range of digit-length and B1.

yoyo

[rogue]

Originally Posted by yoyo

This sounds nice. I'm interested in a feature to connect yoyo@home to different ecmnet servers. For this it is requied, that I can fetch e.g. 1000 curves for a number and send done curves and found factors days/weeks later back to the server.
E.g. I get a file with <server> <number> <required curves>, than I create a number of Boinc Wus out of it. When parts of them are finished (days/weeks later) I send them back.
Of course I have to filter them if they fit to my requirements, because I support only a defined range of digit-length and B1.

I don't know the architecture of yoyo@home. I would need to understand that better before I know what you are asking for. If desired, we can talk about this via e-mail and then I can put the results of said discussions into this thread.

[jyb]

Originally Posted by rogue

I haven't touched ECMNet in about four years, so it's probably time for me to do a few updates to it. It doesn't appear that much needs to be done as I haven't gleaned much from what I've read in this forum (or others).
[snip]
I'm open to other ideas. Please post them here.

When I use the ecmnet client, I pretty much always want to run multiple copies of it, all sharing most of their config information. In particular, I would like them to all be able to use the same config file. As far as I can tell, ecmnet makes this impossible.

I assume other people like to run multiple clients. What do you all do? Have a different directory for each client, each with a slightly different config file?

[rogue]

Originally Posted by jyb

When I use the ecmnet client, I pretty much always want to run multiple copies of it, all sharing most of their config information. In particular, I would like them to all be able to use the same config file. As far as I can tell, ecmnet makes this impossible.

I assume other people like to run multiple clients. What do you all do? Have a different directory for each client, each with a slightly different config file?

The problem is the the client does i/o to a set of work files that are named based upon the ini settings. With multiple copies of the client running from the same directory, they would overwrite one another. That is the problem that needs to be solved in order to allow what you want to happen. I could add a command line switch that would append a value to the work file name, but if you other ideas, I'm listening.

[jyb]

Originally Posted by rogue

The problem is the the client does i/o to a set of work files that are named based upon the ini settings. With multiple copies of the client running from the same directory, they would overwrite one another. That is the problem that needs to be solved in order to allow what you want to happen. I could add a command line switch that would append a value to the work file name, but if you other ideas, I'm listening.

Yes, in fact the problems are these:

• The client always looks for "ecmclient.cfg" in the current directory, so all clients have to have the same current directory. That would be fine, but:
• The log file names are hardcoded, so the clients's log and factors files will all stomp on each other.
• The suffix for work file names is set in the config file, so all clients will have the same suffix, meaning that they will all stomp on each other.
• The machine ID is set in the config file and can't be overridden, so different client processes can't have different IDs.

So it seems that the only way to run multiple clients is for each client to have its own directory containing a slightly different config file, and each client process has to run out of its directory. Needless to say, this is a giant pain.

So I made some changes in my local copy that allow multiple clients to share the same config file without any stomping. They can all therefore run from the same directory. The changes are:

• Use the machine ID to identify each client.
• Allow the machine ID to be overridden on the command line (with "-ID").
• Append the machine ID string to the log and factors file names (e.g. ecmclient_myID.log).
• Recognize a special token in the config file's server descriptor for the work file suffix. If the suffix is specified there as "$ID" then use the machine ID as the suffix. (E.g. work_myID.in).

So now I just set the suffix in the ecmclient.cfg to $ID, and I can fire off any number of clients from the same directory with "ecmclient -ID <string>" and they will all happily coexist (as long as each one has a different <string>, of course). I've been running it this way for the last year or so.

If you want my diffs, let me know.

[rogue]

Originally Posted by jyb

Yes, in fact the problems are these:

• The client always looks for "ecmclient.cfg" in the current directory, so all clients have to have the same current directory. That would be fine, but:
• The log file names are hardcoded, so the clients's log and factors files will all stomp on each other.
• The suffix for work file names is set in the config file, so all clients will have the same suffix, meaning that they will all stomp on each other.
• The machine ID is set in the config file and can't be overridden, so different client processes can't have different IDs.

So it seems that the only way to run multiple clients is for each client to have its own directory containing a slightly different config file, and each client process has to run out of its directory. Needless to say, this is a giant pain.

So I made some changes in my local copy that allow multiple clients to share the same config file without any stomping. They can all therefore run from the same directory. The changes are:

• Use the machine ID to identify each client.
• Allow the machine ID to be overridden on the command line (with "-ID").
• Append the machine ID string to the log and factors file names (e.g. ecmclient_myID.log).
• Recognize a special token in the config file's server descriptor for the work file suffix. If the suffix is specified there as "$ID" then use the machine ID as the suffix. (E.g. work_myID.in).

So now I just set the suffix in the ecmclient.cfg to $ID, and I can fire off any number of clients from the same directory with "ecmclient -ID <string>" and they will all happily coexist (as long as each one has a different <string>, of course). I've been running it this way for the last year or so.

If you want my diffs, let me know.

Why not? Just e-mail them to me.

[xilman]

Originally Posted by jyb

So it seems that the only way to run multiple clients is for each client to have its own directory containing a slightly different config file, and each client process has to run out of its directory. Needless to say, this is a giant pain.

So I made some changes in my local copy that allow multiple clients to share the same config file without any stomping. They can all therefore run from the same directory. The changes are:

• Use the machine ID to identify each client.
• Allow the machine ID to be overridden on the command line (with "-ID").
• Append the machine ID string to the log and factors file names (e.g. ecmclient_myID.log).
• Recognize a special token in the config file's server descriptor for the work file suffix. If the suffix is specified there as "$ID" then use the machine ID as the suffix. (E.g. work_myID.in).

I do it slightly differently, in that I use different directories but identically the same config files for each client.First create the directories (I call them cl1, cl2, ..., cln) and in one of them create a config file. Then hard link to that config file to the same name in each of the other directories.

Second create a shell script which contains lines:

#! /bin/sh
cd /path/to/cl1
/path/to/ecmclient &
cd /path/to/cl2
/path/to/ecmclient &
cd /path/to/cl3
/path/to/ecmclient &
...
cd /path/to/cln
/path/to/ecmclient &

Third, run the shell script. Because the config files are hard linked together, editing any one immediately ensures that all the others are changed at the same time.

If you wish, you could get clever and write the script as a loop which runs from 1 to $1, thereby starting a user-selected number of clients. but I couldn't be bothered. If you wanted to get really clever, you could write a script (probably easier in Perl than sh) which writes customized config files in each directory, creating the directory if required. Again, I couldn't be bothered but if there's interest I could write one easily enough.

Paul