Database of k-values for factors found without scrapping website?

[ATH]

I think you made a mistake because I do not get any spikes when plotting number of factors with k=1 in bins of 1M, and it would not fit with the heuristic estimate of the number of sophie germain primes below n: 1.32032 * n / (ln n)^2

[neomacdev]

Originally Posted by ATH

I think you made a mistake because I do not get any spikes when plotting number of factors with k=1 in bins of 1M...

Yes. The plot certainly seemed odd, I suspected there might be a problem. It turned out to be the line endings in the file. Your file must be from Windows it had (\r\n) CR+LF style line endings. When I read it into Sage w/ Python on my Mac that was causing a problem. Once I stripped out the line endings, then I get a plot very similar to yours. Thanks AGAIN.

[Madpoo]

Originally Posted by axn

Scraping the website for large data extraction is strongly discouraged as it puts unnecessary load on the servers. You can contact user madpoo who is the sysadmin with your exact request, and he will be able to either create a custom extract or point to an existing extract that will fulfill your need.

If the request won't take too much time, and you're willing to be patient until I can get to it, I can generally create a customized set of whatever data. Odds are you'll still get it before you can crawl it all from the website, and it'll be in some manageable form (csv, maybe xml, and I should play with exporting data as JSON at some point). And it keeps the server from getting hammered for days on end.

[kriesel]

Originally Posted by neomacdev

ATH,

Looking at the factors where k=1 is actually something I wanted to do. When you pointed out that ~1,655,600 out of ~41,933,619 known factors were generated from k=1 that peaked my interested and wanted to look at how those factors were distributed by exponent range. So I made a little plot and right away noticed something interesting...

I grouped the exponents into ranges from 1-999999, 1000000-1999999, ..., 900000000-999999999, and then counted the number of occurrences of factors for k=1 amongst the exponents in each range, and made a plot. (attached)

There is a really interesting transition at exponents around 5 million, 50 million, and 500 million, where the frequency of factors for k=1 sharply increases before starting to slowly decrease again. I can't think of any logical reason for this behavior, but it will be something fun to look at more closely & try to explain.

This has to do with how you chose the bins. The binning you chose is apparently for each left digit of the exponent in decimal, over multiple powers of ten. Replot for any equal ratio of bin limits, that is, for n bins covering a power of ten, the ratio high limit to low limit of each bin is the nth root of ten, and it will become a rather smooth flat curve I think, with some statistical fluctuation present. This is another example of Newcomb-Benford's law https://en.wikipedia.org/wiki/Benford%27s_law. You're plotting number of k=1 factors in the chosen intervals, for f=2kp+1. So naturally the frequency per left-digit bin transitions (discontinuities) are at p~10ⁿ/2. Those chosen bin widths are different on a log scale. log (1.9999../1.000) =~ 0.30103 while log (9.9999../9.0000...) only ~0.04576, ~6.58:1.

[Batalov]

Originally Posted by neomacdev

I can't think of any logical reason for this behavior, but it will be something fun to look at more closely & try to explain.

Your sampling size at these boundaries changes by a factor of 10, so the curve has sharp bumps by a factor of 10. (The curve of expected densities is roughly logarithmic. And if you remove sampling bias, it will be.)

[Uncwilly]

Originally Posted by Batalov

Your sampling size at these boundaries changes by a factor of 10, so the curve has sharp bumps by a factor of 10. (The curve of expected densities is roughly logarithmic. And if you remove sampling bias, it will be.)

Originally Posted by neomacdev

Your file must be from Windows it had (\r\n) CR+LF style line endings. When I read it into Sage w/ Python on my Mac that was causing a problem. Once I stripped out the line endings, then I get a plot very similar to yours. Thanks AGAIN.

Serge, you are late to the party. The problem was already found out.