mersenneforum.org 3,748+ c204 Smaller-but-Needed
 User Name Remember Me? Password
 Register FAQ Search Today's Posts Mark Forums Read

 2021-10-26, 21:08 #155 kruoli     "Oliver" Sep 2017 Porta Westfalica, DE 28×3 Posts Have a look, your addition is implemented. The script took nearly nine minutes (!) to generate the HTML file the last time. Before that, it was less than thirty seconds. I cannot see why your extension should change this so drastically. The range is not extracted correctly. It states letter n, space, hyphen, and then a part of the log file name. Thus, I invalidated it.
2021-10-27, 00:28   #156
EdH

"Ed Hall"
Dec 2009

2·29·71 Posts

Quote:
 Originally Posted by kruoli Have a look, your addition is implemented. The script took nearly nine minutes (!) to generate the HTML file the last time. Before that, it was less than thirty seconds. I cannot see why your extension should change this so drastically. The range is not extracted correctly. It states letter n, space, hyphen, and then a part of the log file name. Thus, I invalidated it.
I was working with a locally generated log. If you could, copy a section of your log that has a couple of my clients in it with a found relations line (or two) and post those in a code block in a PM. That will help me adjust the script for your log.

2021-10-27, 01:29   #157
EdH

"Ed Hall"
Dec 2009

100268 Posts

Quote:
 Originally Posted by kruoli . . . The script took nearly nine minutes (!) to generate the HTML file the last time. Before that, it was less than thirty seconds. I cannot see why your extension should change this so drastically. . .
That's because I was doing too much unneeded processing within the case block. I've moved that processing to later. After I adjust for the log differences, it should run much faster.

 2021-10-27, 12:34 #158 EdH     "Ed Hall" Dec 2009 Adirondack Mtns 2·29·71 Posts Good - the fix "seems" simple. I also knocked some processing time off the routine, but the changes might be confusing. I'll paste the two sections first and then a complete script that you can make your time modification to, in case that's easier. This should take the place of the previous case block: Code:  *"Lattice Sieving: Found"*) total=${line} found=${line:50:25} d=$(echo expr index "$found" d) found=${found:${d}+1} r=$(echo expr index "$found" r) found=${found:0:${r}-2} if [ $found -gt$maxfound ] then maxfound=$found rangemax=${total:100} fi if [ $found -lt$minfound ] then minfound=$found rangemin=${total:100} fi ;; And, this should be changed in the appropriate section below the case block: Code: rindh=$(echo expr index "$rangemax" -) rindp=$(echo expr index "${rangemax:${rindh}}" .) rangemax=${rangemax:${rindh}-${rindp}:${rindp}+${rindp}-1} rindh=$(echo expr index "$rangemin" -) rindp=$(echo expr index "${rangemin:${rindh}}" .) rangemin=${rangemin:${rindh}-${rindp}:${rindp}+${rindp}-1} echo "

$eta " >>TeamSieve.html echo " Maximum relations returned from single WU:${maxfound} ($rangemax) " >>TeamSieve.html echo " Minimum relations returned from single WU:${minfound} ($rangemin) " >>TeamSieve.html echo "" >>TeamSieve.html Here's my whole script ATM. There's some extra code for displaying the numbers of relations that doesn't include the echo for it. You can work on that if you wish, or I'll add that in later, in case it's of interest. I don't think it was in the original script, but I didn't check. You may have just removed that: Code: #!/bin/bash log="CADOhtml.log" maxfound=0 minfound=100000 echo "" >TeamSieve.html echo "" >>TeamSieve.html echo "" >>TeamSieve.html echo "Team Sieve data for 3,748+" >>TeamSieve.html echo "" >>TeamSieve.html echo "" >>TeamSieve.html echo " " >>TeamSieve.html echo " Current percentage done and ETA " >>TeamSieve.html exec <"$log" while read line do case $line in *"Lattice Sieving: Marking workunit"*) eta=${line:80} ;; *"Lattice Sieving: Found"*) total=${line} found=${line:50:25} d=$(echo expr index "$found" d) found=${found:${d}+1} r=$(echo expr index "$found" r) found=${found:0:${r}-2} if [ $found -gt$maxfound ] then maxfound=$found rangemax=${total:100} fi if [ $found -lt$minfound ] then minfound=$found rangemin=${total:100} fi ;; esac done k=$(echo expr index "$eta" k) eta=${eta:${k}+2:-1} total=${total##*w} div=$(echo expr index "$total" /) totalf=${total:0:${div}-1} totaln=${total:${div}} rindh=$(echo expr index "$rangemax" -) rindp=$(echo expr index "${rangemax:${rindh}}" .) rangemax=${rangemax:${rindh}-${rindp}:${rindp}+${rindp}-1} rindh=$(echo expr index "$rangemin" -) rindp=$(echo expr index "${rangemin:${rindh}}" .) rangemin=${rangemin:${rindh}-${rindp}:${rindp}+${rindp}-1} echo "$eta

" >>TeamSieve.html echo "

Maximum relations returned from single WU: ${maxfound} ($rangemax)

" >>TeamSieve.html echo "

Minimum relations returned from single WU: ${minfound} ($rangemin)

" >>TeamSieve.html echo "
" >>TeamSieve.html echo "" >>TeamSieve.html echo "" >>TeamSieve.html I really should make this more modular if we're going to add/edit much more. . .
2021-10-27, 12:50   #159
kruoli

"Oliver"
Sep 2017
Porta Westfalica, DE

28·3 Posts

The script now took 5m 50s, which means I definitely try to get "my version" (not with BASH) working ASAP. If Seth is faster, we might skip this step entirely.

Quote:
 Originally Posted by EdH There's some extra code for displaying the numbers of relations that doesn't include the echo for it.
In your current version neither; that was not in the original version and I have not removed it.

 2021-10-27, 13:02 #160 Gimarel   Apr 2010 33×7 Posts I think the loop Code: exec <"$log" while read line do ... done is to slow. I would use something like Code: line=$(grep "Lattice Sieving: Marking workunit" "$log" | tail -n 1) eta=${line:80} grep "Lattice Sieving: Found" "$log" | while read line do ... done and ommit the "case". But this doesn't work because of the pipe. Last fiddled with by Gimarel on 2021-10-27 at 13:04  2021-10-27, 22:11 #161 EdH "Ed Hall" Dec 2009 Adirondack Mtns 101616 Posts @Gimarel: I did use your grep suggestion for the ETA, but I'm trying something a little different for the Max/Min to see if it will speed things up at all. I've made a bunch of changes in the script and added comments, so here's the whole thing now: Code: #!/bin/bash log="CADOhtml.log" maxfound=0 minfound=100000 # Initialize HTML page echo "" >TeamSieve.html echo "" >>TeamSieve.html echo "" >>TeamSieve.html echo "Team Sieve data for 3,748+" >>TeamSieve.html echo "" >>TeamSieve.html echo "" >>TeamSieve.html echo " " >>TeamSieve.html echo " Current percentage done and ETA " >>TeamSieve.html # Harvest found relations data exec <"$log" while read line do case $line in *"Lattice Sieving: Found"*) linearr=($line) found="${linearr[6]}" if [$found -gt $maxfound ] then maxfound=$found rangemax=${line:100} fi if [$found -lt $minfound ] then minfound=$found rangemin=${line:100} fi ;; esac done # Harvest ETA data eta=$(grep "Lattice Sieving: Marking workunit" "$log" | tail -n 1) etaarr=($eta) echo "

${etaarr[10]:1}${etaarr[11]} ${etaarr[12]}${etaarr[13]} ${etaarr[14]}${etaarr[15]} ${etaarr[16]}${etaarr[17]:0:-1}

" >>TeamSieve.html # Add local generation time echo "

Local time of creating this report: " >>TeamSieve.html date >>TeamSieve.html echo "

" >>TeamSieve.html # Harvest Maximum and Minimum IFS="." rmaxarr=($rangemax) rminarr=($rangemin) unset IFS echo "

Maximum relations returned from single WU: ${maxfound} (${rmaxarr[2]})

" >>TeamSieve.html echo "

Minimum relations returned from single WU: ${minfound} (${rminarr[2]})

" >>TeamSieve.html # Finalize HTML page echo "
" >>TeamSieve.html echo "" >>TeamSieve.html echo "" >>TeamSieve.html Let me know of the timing change. If it's longer to run now, I'll look at other approaches. I'm also questioning the usefulness of the max/min ATM. I added in the local time data, too.
2021-10-27, 22:50   #162
VBCurtis

"Curtis"
Feb 2005
Riverside, CA

117238 Posts

Quote:
 Originally Posted by EdH I'm also questioning the usefulness of the max/min ATM. I added in the local time data, too.
I don't think the max has any value- I experimented with -override flag to see what could be overridden, and ran a client-day with A=31. Pretty sure no A=30 workunit will find 12,000 relations!

Kind of tempted to run a few with A=32 to see how high I can push that record. :)

2021-10-27, 23:57   #163
EdH

"Ed Hall"
Dec 2009

2·29·71 Posts

Quote:
 Originally Posted by VBCurtis I don't think the max has any value- I experimented with -override flag to see what could be overridden, and ran a client-day with A=31. Pretty sure no A=30 workunit will find 12,000 relations! Kind of tempted to run a few with A=32 to see how high I can push that record. :)
I kind of wondered about that number shown, compared to the minimum. I have had really small returns with the first range, but that was because it was smaller than the rest for whatever CADO was doing at the time. I did wonder about this run, though.

Still wondering what data would be of use. Time taken for WUs is of interest, but I'm doing that locally for my machines and we'd be displaying too much if it was done for all clients.

 2021-10-28, 00:11 #164 VBCurtis     "Curtis" Feb 2005 Riverside, CA 52×7×29 Posts Stats I find interesting as a job proceeds: total relations found ETA Q value of current workunit average yield for job current yield for job (say, past 100 or 400 workunits) time spent so far on job, in units of client-days or some such. Time spent carries little meaning in a team job like this, since various machines run at variable speeds.
2021-10-28, 01:27   #165
EdH

"Ed Hall"
Dec 2009

2×29×71 Posts

Quote:
 Originally Posted by VBCurtis Stats I find interesting as a job proceeds: total relations found ETA Q value of current workunit average yield for job current yield for job (say, past 100 or 400 workunits) time spent so far on job, in units of client-days or some such. Time spent carries little meaning in a team job like this, since various machines run at variable speeds.
I will look at this, but not sure how many I can easily implement other than the first, which I was working on, but quit. I'll revisit it. I did add something different I'll post separately about. Let's see if it has any interest.

 Similar Threads Thread Thread Starter Forum Replies Last Post fivemack Factoring 3 2017-09-19 08:52 skan YAFU 6 2013-02-26 13:57 akruppa Factoring 114 2012-08-20 14:01 fortega Data 2 2005-06-16 22:48 marc Factoring 6 2004-10-09 14:17

All times are UTC. The time now is 07:09.

Wed Dec 8 07:09:29 UTC 2021 up 138 days, 1:38, 1 user, load averages: 1.41, 1.37, 1.32