mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Data

Reply
 
Thread Tools
Old 2019-04-11, 07:58   #1
neomacdev
 
Apr 2019

2·32 Posts
Default Database of k-values for factors found without scrapping website?

Hi,

I have an idea I would like to research/test. Basically, I want to write a little program to analyze existing data that has been generated through this GIMPS project. Without going into detail here the minimal data I want to analyze would be a list containing the following values:

Mersenne exponent, k-value for the smallest known factor (if one is known)

It would be even better if I got get a data set that would include all the k-values for known factors of each Mersenne exponent, and some indication of what range trial factoring has been done, but that is not absolutely necessary for what I want to analyze. And I’d like to get my hands on as much data like this as possible — every exponent for which you know a factor would be fantastic!

Is there any way to get a copy of this data from the GIMPS project? I see it exists by browsing to individual exponent status report webpages on the site. But is it available to download anywhere without having to scrap the website and build my own database?

When I came here to the forum to ask this question, I searched a bit first and found a post entitled “Mass exponent reports - CSVs” from just last month that had a zip file of a csv file that provided some info on the top trial factoring level for each exponent. That's nice, but what I'm after is a data set that gives the k-value for the smallest known factor for each exponent. Is this obtainable without scraping the webpages and building my own database?

Thanks!
neomacdev is offline   Reply With Quote
Old 2019-04-11, 12:02   #2
ATH
Einyen
 
ATH's Avatar
 
Dec 2003
Denmark

22×17×41 Posts
Default

The smallest factor and the smallest k-value is easy: k=1 and the factor is 23=2*1*11+1 which is a factor of 2^11-1: https://mersenne.org/M11

A total of 1,655,600 factors have k=1.

Last time I downloaded all the GIMPS factors with small program was on Dec 19th 2018, so about 4 months ago, at that time there were 41,933,619 factors.

Here is a list of all the k-values from those factors:
gimps-klist20181219.zip

The file is ~190 MB to download and 714 MB unpacked (use https://www.7-zip.org/ ).

The format is:
<exponent>,<kvalue>

Last fiddled with by ATH on 2019-04-11 at 14:06
ATH is offline   Reply With Quote
Old 2019-04-11, 12:30   #3
neomacdev
 
Apr 2019

2·32 Posts
Default

Quote:
Originally Posted by ATH View Post
The smallest factor and the smallest k-value is easy: k=1 and the factor is 23=2*1*11+1 which is a factor of 2^11-1: https://mersenne.org/M11
ATH, thanks for your reply. Sorry if I wasn't perfectly clear enough in my message above. I didn't mean I want to find the smallest k-value that ever exists. I meant I wanted to get a list of exponents, and the smallest k-value of a valid factor for that M(exponent) if a factor is known.

Quote:
Originally Posted by ATH View Post
A total of 1,655,600 factors have k=1.
This is good information! And relates to what I wanted to study. Thank you!

Quote:
Originally Posted by ATH View Post
Last time I downloaded all the GIMPS factors with small program was on Dec 19th 2018, so about 4 months ago, at that time there were 41,933,619 factors.

Here is a list of all the k-values from those factors:
gimps-klist20181219.zip

The file is ~190 MB to download and 714 MB unpacked (use https://www.7-zip.org/ ).

The format is:
exponent,kvalue
Perfect! This is exactly what I needed! Thank you so much! I very much appreciate it. I didn't want to have to write something to scrape the website from scratch. If you would be willing, is there any chance you could share the program you used to scrape the website and generate this text file?

Thanks again!
neomacdev is offline   Reply With Quote
Old 2019-04-11, 12:49   #4
axn
 
axn's Avatar
 
Jun 2003

3·1,511 Posts
Default

Quote:
Originally Posted by neomacdev View Post
If you would be willing, is there any chance you could share the program you used to scrape the website and generate this text file?
Scraping the website for large data extraction is strongly discouraged as it puts unnecessary load on the servers. You can contact user madpoo who is the sysadmin with your exact request, and he will be able to either create a custom extract or point to an existing extract that will fulfill your need.
axn is offline   Reply With Quote
Old 2019-04-11, 12:57   #5
neomacdev
 
Apr 2019

2×32 Posts
Default

Quote:
Originally Posted by axn View Post
Scraping the website for large data extraction is strongly discouraged as it puts unnecessary load on the servers. You can contact user madpoo who is the sysadmin with your exact request, and he will be able to either create a custom extract or point to an existing extract that will fulfill your need.
Makes sense. (I wouldn't want to scrape the website anyway unless absolutely necessary.) Do you know if there is there a list or document of what "extracts" are available and/or how to access them?

Thanks.
neomacdev is offline   Reply With Quote
Old 2019-04-11, 14:06   #6
ATH
Einyen
 
ATH's Avatar
 
Dec 2003
Denmark

53448 Posts
Default

Here are the actual factors (not the k-values):
gimpsfactors20181219.zip (349 MB) (1033MB unpacked)
Format:
<exponent>,<factor>


Maybe we can get Madpoo to make the server create a full factor list every month maybe or every 3 months, if it is not too taxing, and if bandwidth requirements allows some people to download the list.

Last fiddled with by ATH on 2019-04-11 at 14:09
ATH is offline   Reply With Quote
Old 2019-04-11, 14:22   #7
GP2
 
GP2's Avatar
 
Sep 2003

2×1,289 Posts
Default

You can use the "Factors Found" report to ask for all factors found since 2018-12-19.

Select the "Print simple text report" checkbox. Unselect the "Display date found" checkbox.

That will be a much smaller file. Then merge it with ATH's, making sure to eliminate duplicates.

Obviously k = (factor − 1) / (2 * exponent)
GP2 is offline   Reply With Quote
Old 2019-04-11, 15:15   #8
neomacdev
 
Apr 2019

2×32 Posts
Default

Quote:
Originally Posted by GP2 View Post
You can use the "Factors Found" report to ask for all factors found since 2018-12-19.

Select the "Print simple text report" checkbox. Unselect the "Display date found" checkbox.

That will be a much smaller file. Then merge it with ATH's, making sure to eliminate duplicates.

Obviously k = (factor − 1) / (2 * exponent)
Excellent! Thanks for all the help everyone. Now on to doing some analysis... :-)
neomacdev is offline   Reply With Quote
Old 2019-04-11, 15:43   #9
ATH
Einyen
 
ATH's Avatar
 
Dec 2003
Denmark

22×17×41 Posts
Default

Quote:
Originally Posted by GP2 View Post
You can use the "Factors Found" report to ask for all factors found since 2018-12-19.

Select the "Print simple text report" checkbox. Unselect the "Display date found" checkbox.

That will be a much smaller file. Then merge it with ATH's, making sure to eliminate duplicates.

Obviously k = (factor − 1) / (2 * exponent)
I started downloading factors around 11am CET (10am UTC / 5 am EST), so maybe get factors from 2018-12-18 to make sure.
ATH is offline   Reply With Quote
Old 2019-04-12, 17:36   #10
neomacdev
 
Apr 2019

2·32 Posts
Default

Quote:
Originally Posted by ATH View Post
A total of 1,655,600 factors have k=1.
Quote:
Originally Posted by neomacdev View Post
This is good information! And relates to what I wanted to study. Thank you!
ATH,

Looking at the factors where k=1 is actually something I wanted to do. When you pointed out that ~1,655,600 out of ~41,933,619 known factors were generated from k=1 that peaked my interested and wanted to look at how those factors were distributed by exponent range. So I made a little plot and right away noticed something interesting...

I grouped the exponents into ranges from 1-999999, 1000000-1999999, ..., 900000000-999999999, and then counted the number of occurrences of factors for k=1 amongst the exponents in each range, and made a plot. (attached)

There is a really interesting transition at exponents around 5 million, 50 million, and 500 million, where the frequency of factors for k=1 sharply increases before starting to slowly decrease again. I can't think of any logical reason for this behavior, but it will be something fun to look at more closely & try to explain.
Attached Thumbnails
Click image for larger version

Name:	Plot1.png
Views:	48
Size:	36.1 KB
ID:	20199  

Last fiddled with by neomacdev on 2019-04-12 at 17:37
neomacdev is offline   Reply With Quote
Old 2019-04-12, 18:19   #11
ATH
Einyen
 
ATH's Avatar
 
Dec 2003
Denmark

AE416 Posts
Default

Very interesting. For k=1 the exponent p is a Sophie Germain prime: https://en.wikipedia.org/wiki/Sophie_Germain_prime

I did some checking of my own on the k-values several years ago, but I was quickly accused of doing "numerology" so I did not take it any further:
https://mersenneforum.org/showthread.php?t=19662

I assume you noticed k=2 (mod 4) has no factors because then 2kp+1 != +/- 1 (mod 8)

Last fiddled with by ATH on 2019-04-12 at 18:25
ATH is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Possible values for k (in Mersenne factors) James Heinrich Math 127 2019-09-23 06:22
Factors found before ECM starts MisterBitcoin YAFU 1 2018-08-10 16:58
No factors found aketilander PrimeNet 9 2011-05-17 11:32
C program to rapidly verify all factors in GIMPS database GP2 Programming 8 2005-01-03 07:49

All times are UTC. The time now is 00:47.

Thu Apr 2 00:47:03 UTC 2020 up 7 days, 22:20, 3 users, load averages: 0.84, 1.04, 1.14

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.