Database of k-values for factors found without scrapping website?

[neomacdev]

Hi,

I have an idea I would like to research/test. Basically, I want to write a little program to analyze existing data that has been generated through this GIMPS project. Without going into detail here the minimal data I want to analyze would be a list containing the following values:

Mersenne exponent, k-value for the smallest known factor (if one is known)

It would be even better if I got get a data set that would include all the k-values for known factors of each Mersenne exponent, and some indication of what range trial factoring has been done, but that is not absolutely necessary for what I want to analyze. And I’d like to get my hands on as much data like this as possible — every exponent for which you know a factor would be fantastic!

Is there any way to get a copy of this data from the GIMPS project? I see it exists by browsing to individual exponent status report webpages on the site. But is it available to download anywhere without having to scrap the website and build my own database?

When I came here to the forum to ask this question, I searched a bit first and found a post entitled “Mass exponent reports - CSVs” from just last month that had a zip file of a csv file that provided some info on the top trial factoring level for each exponent. That's nice, but what I'm after is a data set that gives the k-value for the smallest known factor for each exponent. Is this obtainable without scraping the webpages and building my own database?

Thanks!

[ATH]

The smallest factor and the smallest k-value is easy: k=1 and the factor is 23=2*1*11+1 which is a factor of 2^11-1: https://mersenne.org/M11

A total of 1,655,600 factors have k=1.

Last time I downloaded all the GIMPS factors with small program was on Dec 19th 2018, so about 4 months ago, at that time there were 41,933,619 factors.

Here is a list of all the k-values from those factors:
gimps-klist20181219.zip

The file is ~190 MB to download and 714 MB unpacked (use https://www.7-zip.org/ ).

The format is:
<exponent>,<kvalue>

[neomacdev]

Originally Posted by ATH

The smallest factor and the smallest k-value is easy: k=1 and the factor is 23=2*1*11+1 which is a factor of 2^11-1: https://mersenne.org/M11

ATH, thanks for your reply. Sorry if I wasn't perfectly clear enough in my message above. I didn't mean I want to find the smallest k-value that ever exists. I meant I wanted to get a list of exponents, and the smallest k-value of a valid factor for that M(exponent) if a factor is known.

Originally Posted by ATH

A total of 1,655,600 factors have k=1.

This is good information! And relates to what I wanted to study. Thank you!

Originally Posted by ATH

Last time I downloaded all the GIMPS factors with small program was on Dec 19th 2018, so about 4 months ago, at that time there were 41,933,619 factors.

Here is a list of all the k-values from those factors:
gimps-klist20181219.zip

The file is ~190 MB to download and 714 MB unpacked (use https://www.7-zip.org/ ).

The format is:
exponent,kvalue

Perfect! This is exactly what I needed! Thank you so much! I very much appreciate it. I didn't want to have to write something to scrape the website from scratch. If you would be willing, is there any chance you could share the program you used to scrape the website and generate this text file?

Thanks again!

[axn]

Originally Posted by neomacdev

If you would be willing, is there any chance you could share the program you used to scrape the website and generate this text file?

Scraping the website for large data extraction is strongly discouraged as it puts unnecessary load on the servers. You can contact user madpoo who is the sysadmin with your exact request, and he will be able to either create a custom extract or point to an existing extract that will fulfill your need.

[neomacdev]

Originally Posted by axn

Scraping the website for large data extraction is strongly discouraged as it puts unnecessary load on the servers. You can contact user madpoo who is the sysadmin with your exact request, and he will be able to either create a custom extract or point to an existing extract that will fulfill your need.

Makes sense. (I wouldn't want to scrape the website anyway unless absolutely necessary.) Do you know if there is there a list or document of what "extracts" are available and/or how to access them?

Thanks.

[ATH]

Here are the actual factors (not the k-values):
gimpsfactors20181219.zip (349 MB) (1033MB unpacked)
Format:
<exponent>,<factor>


Maybe we can get Madpoo to make the server create a full factor list every month maybe or every 3 months, if it is not too taxing, and if bandwidth requirements allows some people to download the list.

[GP2]

You can use the "Factors Found" report to ask for all factors found since 2018-12-19.

Select the "Print simple text report" checkbox. Unselect the "Display date found" checkbox.

That will be a much smaller file. Then merge it with ATH's, making sure to eliminate duplicates.

Obviously k = (factor − 1) / (2 * exponent)

[neomacdev]

Originally Posted by GP2

You can use the "Factors Found" report to ask for all factors found since 2018-12-19.

Select the "Print simple text report" checkbox. Unselect the "Display date found" checkbox.

That will be a much smaller file. Then merge it with ATH's, making sure to eliminate duplicates.

Obviously k = (factor − 1) / (2 * exponent)

Excellent! Thanks for all the help everyone. Now on to doing some analysis... :-)

[ATH]

Originally Posted by GP2

You can use the "Factors Found" report to ask for all factors found since 2018-12-19.

Select the "Print simple text report" checkbox. Unselect the "Display date found" checkbox.

That will be a much smaller file. Then merge it with ATH's, making sure to eliminate duplicates.

Obviously k = (factor − 1) / (2 * exponent)

I started downloading factors around 11am CET (10am UTC / 5 am EST), so maybe get factors from 2018-12-18 to make sure.

[neomacdev]

Originally Posted by ATH

A total of 1,655,600 factors have k=1.

Originally Posted by neomacdev

This is good information! And relates to what I wanted to study. Thank you!

ATH,

Looking at the factors where k=1 is actually something I wanted to do. When you pointed out that ~1,655,600 out of ~41,933,619 known factors were generated from k=1 that peaked my interested and wanted to look at how those factors were distributed by exponent range. So I made a little plot and right away noticed something interesting...

I grouped the exponents into ranges from 1-999999, 1000000-1999999, ..., 900000000-999999999, and then counted the number of occurrences of factors for k=1 amongst the exponents in each range, and made a plot. (attached)

There is a really interesting transition at exponents around 5 million, 50 million, and 500 million, where the frequency of factors for k=1 sharply increases before starting to slowly decrease again. I can't think of any logical reason for this behavior, but it will be something fun to look at more closely & try to explain.

[ATH]

Very interesting. For k=1 the exponent p is a Sophie Germain prime: https://en.wikipedia.org/wiki/Sophie_Germain_prime

I did some checking of my own on the k-values several years ago, but I was quickly accused of doing "numerology" so I did not take it any further:
https://mersenneforum.org/showthread.php?t=19662

I assume you noticed k=2 (mod 4) has no factors because then 2kp+1 != +/- 1 (mod 8)