Output text in data format?
 

Hi,

I have a feature request. Some of you may have already seen this since I stupidly posted it in the wrong place :redface:

While I'm at it, I'll try to clarify a bit of what I meant since it wasn't understood very well.

In certain places on mersenne.org, when acquiring data, there is an option "print a simple text report." See the attachment. The query is limited to 50,000 lines, I would imagine because of the load it might place on the server.

With the server getting its upgrade, I was thinking it might be possible for this limit to be raised, but having 50,000 lines of text displayed in a browser is a bit cumbersome, and if the limit were extended to something like 250,000, it would only become worse.

On the other hand, I was thinking if there was an option to have the results uploaded to a report.txt, it might be feasible to have a very large amount of data quickly gathered into a format that's ready to be processed.

Is this clearer?

[QUOTE=TheMawn;380742]... having 50,000 lines of text displayed in a browser is a bit cumbersome, and if the limit were extended to something like 250,000, it would only become worse.[/QUOTE]IIRC the HTTP [url=http://tools.ietf.org/html/rfc2183]Content-Disposition[/url] header tells the browser the appropriate method of display/play/download.

[QUOTE=TheMawn;380742]Hi,

I have a feature request. Some of you may have already seen this since I stupidly posted it in the wrong place :redface:

While I'm at it, I'll try to clarify a bit of what I meant since it wasn't understood very well.

In certain places on mersenne.org, when acquiring data, there is an option "print a simple text report." See the attachment. The query is limited to 50,000 lines, I would imagine because of the load it might place on the server.

With the server getting its upgrade, I was thinking it might be possible for this limit to be raised, but having 50,000 lines of text displayed in a browser is a bit cumbersome, and if the limit were extended to something like 250,000, it would only become worse.

On the other hand, I was thinking if there was an option to have the results uploaded to a report.txt, it might be feasible to have a very large amount of data quickly gathered into a format that's ready to be processed.

Is this clearer?[/QUOTE]

I hadn't looked at that section in regards to performance. The new server seems to generate the data in a somewhat reasonable time, given the amount of content it spits out. It's a little CPU heavy during the query so that's probably a good candidate for some optimization down the road.

As far as getting the "print" button to generate a simmple text/plain output instead of formatted inside the page, that's a design question... it wouldn't make the query run any faster though so unless that could be addressed to handle larger sets of data, I don't know...

If not for increased performance, then maybe for convenience?

I can't imagine anyone ever queries 50,000 lines of text just to look at them in the web page. I usually copy it all into a results.txt anyway and then open Excel to process the data. I don't know how many people do that.

[QUOTE=TheMawn;380764]50,000 lines of text[/QUOTE]

Seems a touch demanding?

[QUOTE]I usually copy it all into a results.txt anyway and then open Excel to process the data. I don't know how many people do that.[/QUOTE]

I imagine you are really thinking of a [B]worktodo.txt[/B], though if that were true why wasn't the Exclude currently assigned exponents button checked?

Well, yes, the idea is to make a worktodo.txt. I was just making that picture to show what I was talking about. Normally yes assigned work would be excluded, but I fail to see how that is in any way relevant to my suggestion.

Anyway, if you want to persist, I can show you a completely different example. See the attachment.

If I wanted to access the data from this page instead, for whatever purposes, the query is limited to 1,000 exponents and it generates a big clunky web page. If mersenne.org just let me download a report.txt instead of displaying everything, it could save me a step.

[QUOTE=TheMawn;380816]See the attachment.

If I wanted to access the data from this page instead, for whatever purposes, the query is limited to 1,000 exponents[/QUOTE]

I don't see any attachment but I don't imagine that would be especially useful anyhow.
Perhaps if I had injected an emoticon my purpose for comment might be been clearer!
I don't for any moment imagine that you would take a shortcut to results.
I note that today results pages from mersenne.org are now offering 3000 lines instead of 1000 formerly, so perhaps more liberal limits to your particular kind of search might also get a rise.
I would think that limits are imposed to prevent server wastage.
I have been guilty of submitting a request where, by finger problem, the upper limit had one too many digits than intended, thus incurring a curtailed response.

I just don't get why nobody is understanding what my request is.

Instead of a giant page full of lines of text, how about the page lets me download a raw .txt file instead?

This was the page I was looking at previously. Again the data there isn't particularly useful to anyone as it is, being very messy. If I was going to process it for any reason, I would copy it all, paste it into a .txt file and then open it in excel and use some clever delimitation options to try to make it more readable.


Again it's limited to 1,000 exponents and if I wanted to look at 10,000 exponents, I would have to f*ck around with this nine more times.


If I have not yet made myself clear then I won't bother trying to explain it any more. And if I have, I would like to know if it would be a lot of work.

Hint: GPU72.com has the option to download worktodo.txt as a file instead of as a great big pile of crap on the page that needs to be manually copied and pasted.

[QUOTE=TheMawn;381200]download a raw .txt file instead?[/QUOTE]

Well that would be a design change isn't it. And that is what you seek. I do get that.

[QUOTE=TheMawn;381200]I just don't get why nobody is understanding what my request is.

Instead of a giant page full of lines of text, how about the page lets me download a raw .txt file instead?[/QUOTE]

I "get" what you are saying, trying to create a worktodo.txt file for mfaktc can be a pain. I end up highlighting the stuff on the web page then copy'n'paste to a text file using edit and then search and replace...

Bit of a fag, but doable.

Even nicer would be an option to directly build the worktodo ie you are asked for

start,number of exponents, end bit

[QUOTE=Gordon;381205]I "get" what you are saying, trying to create a worktodo.txt file for mfaktc can be a pain. I end up highlighting the stuff on the web page then copy'n'paste to a text file using edit and then search and replace...

Bit of a fag, but doable.
[/QUOTE]

My comprehension stops short of understanding what "bit of a fag" might mean.
My first guess is that might be some admission of difficulty.
I do not concede any difficulty editing mersenne.org/report_factoring_effort material for worktodo purposes.
Perhaps it is the the tool for the purpose that matters.
I tend to use TextPad.

[QUOTE=TheMawn;380764]I can't imagine anyone ever queries 50,000 lines of text just to look at them in the web page. I usually copy it all into a results.txt anyway and then open Excel to process the data. I don't know how many people do that.[/QUOTE]I used to do it by hand, until I found that Excel can grab the table all by itself. I set my spreadsheet up to grab 2 longish tables (4500-8000 rows) and then work from there. I have attached a picture for where to find it in the old menu system.

:geek:
[edit]BTW, you will want to uncheck text-only when you pass the URL to excel. And with the new server, it all takes me ~10 seconds once the spreadsheet is open, before making it all automated and copying everything by hand and sorting it out, it took about 3 minutes each time.[/edit]
[edit2]http://bit.ly/XHJdW4[/edit]

If you grab your expos and drop them to mersenne.ca, it can create a worktodo list for you. I however use programmer's notepad 2, to which I made a small macro with search and replace to get rid of the p-1 data.

[QUOTE=TheMawn;381200]I just don't get why nobody is understanding what my request is.
Instead of a giant page full of lines of text, how about the page lets me download a raw .txt file instead?[/QUOTE]I understand your request, but I hadn't seen this thread until now. Now that I know where/what change you're looking for I should be able to do something about it.

You should now have a download option. Let me know if it's not working as intended.
You can in theory choose the download option for either text or HTML output, but presumably you would usually want the text version.

I also increased the max-rows from 50k to 100k (for text, HTML output remains limited 10k)

If you want another checkbox to generate worktodo directly from that report let me know what you want it to look like, should be trivial to add.

[QUOTE=James Heinrich;381454]You should now have a download option. Let me know if it's not working as intended.
You can in theory choose the download option for either text or HTML output, but presumably you would usually want the text version.

I also increased the max-rows from 50k to 100k (for text, HTML output remains limited 10k)

If you want another checkbox to generate worktodo directly from that report let me know what you want it to look like, should be trivial to add.[/QUOTE]

I'm not pushing it.... but is it possible for a option to remove all P-1/ECM data from it?(pure TF data) :whistle:

[QUOTE=kracker;381455]I'm not pushing it.... but is it possible for a option to remove all P-1/ECM data from it?(pure TF data) :whistle:[/QUOTE]Do you mean remove the last 2 columns of the report (B1/B2), or remove exponents from the list that have had P-1/ECM done?

edit: if you meant remove B1/B2 columns... done :smile:
If you meant something else, please explain further.

[QUOTE=James Heinrich;381456]Do you mean remove the last 2 columns of the report (B1/B2), or remove exponents from the list that have had P-1/ECM done?

edit: if you meant remove B1/B2 columns... done :smile:
If you meant something else, please explain further.[/QUOTE]

Exactly what I was thinking, Thank you!! :smile:

No problem. If someone uses this report to generate worktodo for TF please let me know what you would like the output to be like. Specifically the target TF level -- should it be set to current+1, a fixed number, or a placeholder like "xx" so you can search-replace with whatever you want?

I say just use curl and awk ;D

Filling it with "from 1 to 10000" returns nothing. However, the "2 to 10000" returns the right list... (default when picked from the menu on the left, the box is filled with "1").

Now, if we are at it, (I mean you, James :razz:), can we have a "limited" number of rows for the "[URL="http://www.mersenne.org/results/default.php"]work done[/URL]" report? I mean, beside of "exponent range" boxes, just a "show last 100 results" box, where I can edit the 100, same as it is done for top producer lists?

I don't know if it makes too much sense from the server's point of view, here it depends how the query is organized, maybe it needs to search all the DB to get the last results, so it may *not* be less work for the server, but certainly [U]would be less data to transfer[/U], because we are usually interested only in the results form the last day, week, whatever, so if I feel "100" in the box and check the "don't show futile Tf work done" box, then I will only get 100 lines instead of 15k lines, i.e. transfer 10k instead of 1.5 megs, and yet have all the report I need.

Currently, the report gives us thousands of lines we don't need, and it is very slow if you check the boxen that exclude the "lot of text generation" (as TF results), because the DB is still searched for ages for very old LL results to fill in all those thousands of lines. It's worth thinking about. Most of us won't need to see the very old reports (and transfer hundreds of kilobytes of them) after every submission...

Beautiful, James. This is great. 100,000 rows is pretty nice, too.

Just to be clear, I didn't want my of my slight frustration directed at you. I was just having a hard time getting my point across but you got it perfectly.

(the result is interesting when simple-text only isn't checked: the report.txt can be renamed to .html and it generates the table nice and pwetty)


If you're thinking of taking this further for TF or whatever, be my guest but this is all I was asking for.

Having the 'Factor=N/A," at the start is helpful. Having the two commas after the current bit level is good because it makes it easy to find-and-replace if that's the path one wants to take. Having ",xx," would be one step better still. Having a space to enter the desired level is truly above and beyond.

Definite no on defaulting to bit-level+1 because not all of the exponents in the list are necessarily factored to the same level to start.


That's all I can think of, but everything so far is exactly what I was after and there are easy ways of doing the rest.

[QUOTE=LaurV;381480]Filling it with "from 1 to 10000" returns nothing.[/quote]Fixed, thanks.

[QUOTE=LaurV;381480]Now, if we are at it, (I mean you, James :razz:), can we have a "limited" number of rows for the "[URL="http://www.mersenne.org/results/default.php"]work done[/URL]" report?[/QUOTE]Sure, done.

[QUOTE=TheMawn;381481]the result is interesting when simple-text only isn't checked: the report.txt can be renamed to .html and it generates the table nice and pwetty[/QUOTE]Thanks, fixed -- it will now download as report.html if text-only is not checked. Proper MIME type now added too.

[QUOTE=TheMawn;381481]Having a space to enter the desired level is truly above and beyond.[/QUOTE]Enjoy the above-and-beyond. :smile:

[QUOTE=LaurV;381480]Now, if we are at it, (I mean you, James :razz:), can we have a "limited" number of rows for the "[URL="http://www.mersenne.org/results/default.php"]work done[/URL]" report? I mean, beside of "exponent range" boxes, just a "show last 100 results" box, where I can edit the 100, same as it is done for top producer lists? [/QUOTE]

Grrr. We lost my last update to this web page where the exponent is linked to the full report on the exponent.

What I really want is to display each LL result exponent in color. Red = known bad, yellow = suspect error code or mismatches another LL, green = known good, black = none of the above.

This will let me quickly see at a glance if a machine starts returning bad LL results.

[QUOTE=Prime95;381495]Grrr. We lost my last update to this web page where the exponent is linked to the full report on the exponent.[/QUOTE]Sorry George, I guess I missed re-mirroring my copy before making my changes. Do you have a local copy of your modified file? Please send it to me and I'll merge my changes and yours.

No problem, James. I fixed it on the server.

Beautiful! [URL="http://www.youtube.com/watch?v=e5WygWnzj3w"]Thanks a million[/URL] to both of you. I was wondering if my post didn't escape unobserved, I have this luck of posting at the end of the page just one minute before someone else (usually The Mawn :razz:) posts a new page, hehe...

About the colors, yes, that would be nice to have too, but it it not really necessary, as the matched/notmatched is not differentiated in the current report, it will require some new DB tricks/flag, etc....

[QUOTE=Prime95;381495]What I really want is to display each LL result exponent in color. Red = known bad, yellow = suspect error code or mismatches another LL, green = known good, black = none of the above.[/QUOTE]I'll leave that to you. After some experimenting I conclude I apparently haven't figured out either the database structure or how to properly do the query in MSSQL.

[QUOTE=LaurV;381514]I have this luck of posting at the end of the page just one minute before someone else (usually The Mawn :razz:) posts a new page, hehe...[/QUOTE]

It has a funny effect on me, because when I finish my post, it leaves me on the new page because that's where my post went. However, because your post is still "unread" (because I haven't actually seen it) when I hit "Go to First Unread" it brings me to yours, and then I see mine when I go to the next page.

Selectivity
 

[QUOTE=James Heinrich;381494]Enjoy the above-and-beyond. :smile:[/QUOTE]

These extensions now provide access to unassigned work via report_factoring_effort much more selective than assigning some work via manual_assignment.
The latter can't select by sought start levels, nor nominate a target level.
Isn't this a tiny bit unbalanced?

[QUOTE=snme2pm1;381604]These extensions now provide access to unassigned work via report_factoring_effort much more selective than assigning some work via manual_assignment.
The latter can't select by sought start levels, nor nominate a target level.
Isn't this a tiny bit unbalanced?[/QUOTE]I have just cleaned up the code for the manual-assignments page so it would now be easy to add some additional interface to solicit a TF bit range from the user, but for it to do anything the back-end get_assignment_within_range code would need to be updated, and that's in George's court. If he is willing to add additional optional parameters to filter by TF level then I could add that into the GUI.

[QUOTE=James Heinrich;381493]Fixed, thanks.[/QUOTE]
Well, not really. Now it works from 1, but when you select it from the menu on the left (result query/factoring limits), the box is filled in with... 0! So, I filled 10000 on the second box. Zero results returned. :razz:
I had to fill in a 1 in the first box (replacing the zero) to get the list.

Now please, don't fix it for zero in such a way that the default value will be "-1" hehe :razz:

[QUOTE=LaurV;381609]Well, not really.[/QUOTE]Let's try that again. Serves me right for trying to code at 3am... :sleep:

Haha, love to bother you!
In fact, just replacing the default text in that box with "2" would solve all the problems. Or, add an "if smaller than 2, replace it with 2". And for the second box, "if smaller than the first box, then replace it with first box". And that is all. For lazy guys like me who won't type over the default values, hehe.

edit: The listing works very nice, and I like the automatic exclusion of the assigned exponents (again, for lazy guys who wont click once more to select the checkbox, haha, wonderful!) (first my heart jumped, I thought that someone factorized 1277 and 1619, grrr :wink:)

edit2: now it works fine, thanks a lot, very nice and prompt job! Real-time support, hehe...

[QUOTE=Prime95;381495]Grrr.[/QUOTE]
Told ye I am contagious! :wink:

TF work gathered en-masse through the new interface is still off-the-record, I imagine? I see little value in adding reservations, particularly if we're good about unofficially reserving the stuff through the forum.

[QUOTE=TheMawn;381621]I see little value in adding reservations, particularly if we're good about unofficially reserving the stuff through the forum.[/QUOTE]

Your message does not allude to any prior post which might leave viewers confused.
Perhaps you are bothered or confused about my prior post mentioning that provisions for un-assigned work have now been enhanced, yet there is not a symmetric provision for gaining assigned work.
Don't get me wrong, I can see the place for soaking up masses of unreserved work, and I also engage in such thrusts.
I believe that the face of PrimeNet should be coherent in manner filtering candidates for a proposed purpose.
Recent advances here alter that balance more noticeably, introducing candidate filtering not offered universally.
I also see the merits of work being reserved, especially when within sight of the PrimeNet automatic allocation front.
I reckon that provisions for honorable assignment of work should be not inferior to the provisions for grabbing un-assigned work. however that seems to be the current situation.
Is it not preferable to encourage assignment of work for most people?
Edit, I acknowledge that filtering by TF level is not new for the report_factoring_effort information, but the greater accessibility is nevertheless significant.

I wasn't alluding to any particular post, no.

I was just making the observation that gathering lots of the 100M+ TF work has gotten a LOT easier now that the factoring limits page literally hands you the worktodo.txt exactly as you need it. I just wanted to make sure I hadn't missed anything and that we hadn't in any way added a reservations system, and that all the LMH stuff is still strictly free-for-all.

To be honest, I still don't understand what troubles you with my comment. I know very well the value of the reservations system for any LL, DC, and factoring work remotely near the wave fronts.

I was specifically saying that despite the huge facilitation to [B][I]gathering[/I][/B] LHM factoring work I see little value in adding a system to [B][I]reserve[/I][/B] that work. I feel like there isn't enough of an organized effort to warrant a reservations system for tens of thousands of factoring jobs. Anyone going into the 700M range, for example, understands that it's completely unregulated.

[QUOTE=snme2pm1;381209]My comprehension stops short of understanding what "bit of a fag" might mean.
My first guess is that might be some admission of difficulty.
I do not concede any difficulty editing mersenne.org/report_factoring_effort material for worktodo purposes.
Perhaps it is the the tool for the purpose that matters.
I tend to use TextPad.[/QUOTE]

bit of a fag = time consuming and tedious

[QUOTE=Gordon;381674]bit of a fag = time consuming and tedious[/QUOTE]
...in that context.:smile:

[QUOTE=Brian-E;381689]...in that context.:smile:[/QUOTE]

Thought I'd keep it simple...

bit of a fag = small part of a cigarette

[url]https://www.google.ca/search?q=fag+bearings&espv=2&source=lnms&tbm=isch&sa=X&ei=UeUAVNW1NdSkyATt74CgDg&ved=0CAYQ_AUoAQ&biw=1920&bih=955[/url]

[QUOTE=TheMawn;381653]I wasn't alluding to any particular post, no.
... I see little value in adding a system to [B][I]reserve[/I][/B] that work. I feel like there isn't enough of an organized effort to warrant a reservations system for tens of thousands of factoring jobs.[/QUOTE]

Ok, your worry that work units might suddenly and unnecessarily be reserved has presumably been sorted, yet I can't imagine how that fear was spawned.
I was at pains in my last message to agree that masses of unreserved work in outer reaches is not a bad thing, so I can't think why you imagined that I might be troubled about such business.
My position remains, that there is no symmetric mechanism for anybody to request assigned work subject to explored level.
I would have occasion to invoke such discrimination filter if it were available.
It is currently very awkward to assign work limited by depth in a non-virgin region where different contributors at different times have created a ragged edge of exploration.
I haven't subscribed, but I suspect that one can exercise such discrimination for assignments via GPU72.
I'm off tripping for a couple of weeks, Macau, Hong Kong, Beijing.
Done all that before, but Beijing will be very different to the last time I saw it.
I'll progress my reservations upon return, in the meantime my equipment will remain cold!

[QUOTE=Gordon;381694]bit of a fag[/QUOTE]
[URL="http://ro.wikipedia.org/wiki/Fag"]Only the leafs, or parts of the trunk too? [/URL]:razz:
(this reminds me of some chat program I used in the past which was replacing vulgar English words with inoffensive stuff, it was a big source of fun when newcomers joined, we used to talk in Romanian and we had to build strange grammar constructions to go around that "feature", one of the most bothering part was replacing the "[URL="https://translate.google.com/#ro/en/cum"]cum[/URL]", used a lot in Romanian interrogative :wink:)

[QUOTE=LaurV;381738][URL="http://ro.wikipedia.org/wiki/Fag"]Only the leafs, or parts of the trunk too? [/URL]:razz:
(this reminds me of some chat program I used in the past which was replacing vulgar English words with inoffensive stuff, it was a big source of fun when newcomers joined, we used to talk in Romanian and we had to build strange grammar constructions to go around that "feature", one of the most bothering part was replacing the "[URL="https://translate.google.com/#ro/en/cum"]cum[/URL]", used a lot in Romanian interrogative :wink:)[/QUOTE]

I could have [URL="http://global3.memecdn.com/fag-tiger_o_2057963.jpg"]linked something like this[/URL] or [URL="https://s3.amazonaws.com/webassets.ticketmob.com/LS/images/ShowPosters/YanezFQLogo.jpg"]this[/URL] but didn't want the THINKPOL to remove it as not being relevant...

[url]http://www.thefreedictionary.com/fag[/url]
[QUOTE]
fag 1 (f[IMG]http://img.tfd.com/hm/GIF/abreve.gif[/IMG]g)[I]n.[/I][B]1. [/B][B]a. [/B] [U]A student at a British public school who is required to perform menial tasks for a student in a higher class.[/U]
[B]b. [/B] A drudge.

[B]2. [/B] [I]Chiefly British[/I] Fatiguing or tedious work; drudgery.

[I]v.[/I] [B]fagged[/B], [B]fag·ging[/B], [B]fags[/B]
[I]v.[/I][I]intr.[/I][B]1. [/B] To work to exhaustion; toil.
[B]2. [/B] To function as the servant of another student in a British public school.

[I]v.[/I][I]tr.[/I] To exhaust; weary: Four hours on the tennis court fagged me out.

[From fag, [I]to droop (obsolete)[/I], perhaps from Middle English fagge; see [B] fag end[/B].]
fag 2 (f[IMG]http://img.tfd.com/hm/GIF/abreve.gif[/IMG]g)[I]n.[/I] [I]Slang[/I] A cigarette.

[Short for [URL="http://www.thefreedictionary.com/fag%20end"]fag end[/URL].]
fag 3 (f[IMG]http://img.tfd.com/hm/GIF/abreve.gif[/IMG]g)[I]n.[/I] [I]Offensive Slang[/I] Used as a disparaging term for a homosexual man.

[Short for [URL="http://www.thefreedictionary.com/faggot"]faggot[/URL].]
fag (fæɡ) [I]n[/I][B]1. [/B]a boring or wearisome task: it's a fag having to walk all that way.
[B]2. [/B] (Education) [I]Brit[/I] (esp formerly) a young public school boy who performs menial chores for an older boy or prefect

[I]vb[/I], [B]fags[/B], [B]fagging[/B] [I]or[/I] [B]fagged[/B][B]3. [/B] (when: [I]tr, often foll by out[/I]) to become or cause to become exhausted by hard toil or work
[B]4. [/B] ([I]usually intr[/I]) [I]Brit[/I] to do or cause to do menial chores in a public school: Brown fags for Lee.

[C18: of obscure origin]

[/QUOTE]