mersenneforum.org  

Go Back   mersenneforum.org > Prime Search Projects > Prime Gap Searches

Reply
 
Thread Tools
Old 2019-11-18, 10:12   #144
henryzz
Just call me Henry
 
henryzz's Avatar
 
"David"
Sep 2007
Cambridge (GMT/BST)

575410 Posts
Default

Quote:
Originally Posted by robert44444uk View Post
If you want some history I have a zip file with 77 dated merit files going back to 2015 - that is the shorter of the two files kept by Dr Nicely.

The problem is the size of the zip file - 40.4 Mb

If someone could write some perl code that would just pick up the new records in each file then I am happy to run that to create a history file. I saved each of the files in the form meritsddmmyy.txt

An alternative to "self regulation" is for one of the group to act as gatekeeper. That person would receive the files and (i) run cglp again, in the case of new searcher or in the case of important records, (ii) carry out spot checks to ensure accuracy of the gaps in the case of established searchers, or review the cglp.out file that accompanies the new records before updating the record file (iv) recheck the record file from time to time for gaps that were initially unchecked. The gatekeeper role is the role that Thomas N actually played.
My suggestion is that anyone can add to a needs verification queue(just a separate file on github maybe). We can then have several people who can verify other's results and move them to the correct place. I would suggest steering clear of self-verifying or overloading one person.
henryzz is online now   Reply With Quote
Old 2019-11-18, 14:17   #145
rudy235
 
rudy235's Avatar
 
Jun 2015
Vallejo, CA/.

23·112 Posts
Default

Quote:
Originally Posted by robert44444uk View Post
Actually this is not the case. The first "C" refers to "Conventional" i.e. a gap between two prime numbers. The idea being that Thomas was hoping to also collect other types of gaps - for example our recent work on gaps between twins.

The "P" in the third place refers to whether either or both of the two bounding primes is probabilistic. Only where both bookends are certified was the "C" used.
Just as he said.
Quote:
The character in position 9 is the type of the gap; all of the gaps in this document are conventional (common, classic, standard, regular, ordinary, normal) prime gaps, indicated by the letter "C"; in other words…
rudy235 is offline   Reply With Quote
Old 2019-11-18, 15:10   #146
robert44444uk
 
robert44444uk's Avatar
 
Jun 2003
Oxford, UK

190810 Posts
Default

Quote:
Originally Posted by henryzz View Post
My suggestion is that anyone can add to a needs verification queue(just a separate file on github maybe). We can then have several people who can verify other's results and move them to the correct place. I would suggest steering clear of self-verifying or overloading one person.
That sounds like a good solution
robert44444uk is offline   Reply With Quote
Old 2019-11-23, 12:17   #147
gjhiggins
 
"Graham Higgins"
Nov 2019
UK

1316 Posts
Default interim progress

I've created a first-cut implementation of a repository-driven (maintained separately from the SQL INSERT gap list repository) web site that publishes prime gap list resources. It's not an ideal solution but it might work tolerably well within the constraints.

Commits to this github repository: https://github.com/gjhiggins/gjhiggins.github.io/ result in this static gh-pages website: https://gjhiggins.github.io/ (where, upon community consensus approval, "gjhiggins.github.io" will be replaced by "primegap-list-project.github.io"). The dynamics are done in-browser with javascript.

There isn't a completely seamless path from a file of SQL insert statements to a SQLite db that can be used by the website javascript, currently the SQL javascript library is loading the (8Mb) SQLite allgaps.db from where I committed it in the repository's _data folder.

See the post: https://gjhiggins.github.io/project/...st-in-sqlite3/ for how to access and work with the SQLite3 database of prime gap list data.

There is good documentation for this particular repos-as-website paradigm of Github Pages and Jekyll and the whole shebang is designed to have a low-ish tech barrier to entry.

News posts are quite straightforward, just copy an existing one, edit it to suit, save it as a new file and commit the new file to the repository. The web site will be automagically updated with the new post.

As an example, here's a rendering of one of Tom Nicely's paras as a post:
Code:
---
layout: post
author: Dr. Tom Nicely
category: record
title:  "New first known occurrence prime gap"            # Title used in HTML Meta, Post Header, Recent Posts list
tags: post                   # Tags to relate multiple topics to a post
excerpt: "Robert W. Smith has discovered a new first known occurrence prime gap of 203890"
---

On 08 May 2019, Robert W. Smith discovered a new first known occurrence prime gap of 203890 following the 2485-digit prime 140207∗5813#/46410 - 86644.

This gap has merit 35.640174363, the greatest merit for any known prime gap exceeding 26892.

On 08 July 2019, Smith also discovered a new first known occurrence prime gap of 614640 following the 10004-digit prime 281∗23173#/46410 - 267338.

This gap has merit 26.6845515588753865, the greatest merit for any known prime gap exceeding 556982.

An [extended table](https://www.mersenneforum.org/showpost.php?p=520996&postcount=97) of previous such gaps, due to Robert W. Smith and axn, is available on the Mersenne PGS forum.
And its corresponding rendering on the web site is: https://gjhiggins.github.io/record/2...nce-prime-gap/

There's more to do/discuss yet but (IMO) it's an encouraging start.

(I welcome bouquets or brickbats with equal equanimity.)

Cheers

Graham
gjhiggins is offline   Reply With Quote
Old 2019-11-24, 10:06   #148
robert44444uk
 
robert44444uk's Avatar
 
Jun 2003
Oxford, UK

77416 Posts
Default

This is amazing Graham. Congratulations on all your hard work.

There's a lot to absorb here and I am not familiar with any of the tools so I've got a bit of a learning curve to overcome.

One thing - much of the very large database size is taken up by large gap records that are not in one of the primorial forms. Dr Nicely overcame this issue by keeping he much smaller
merits file - most gap searchers download this to determine whether their new finds are records. Typically I find between 500 to 1000 records per month out of perhaps 1 million notable gaps found, so this is a key practical tool.

Perhaps this is already there - there is a lot to absorb from what you have done - again congrats!
robert44444uk is offline   Reply With Quote
Old 2019-11-25, 23:53   #149
gjhiggins
 
"Graham Higgins"
Nov 2019
UK

19 Posts
Default merits.txt

Quote:
Originally Posted by robert44444uk View Post
There's a lot to absorb here and I am not familiar with any of the tools so I've got a bit of a learning curve to overcome.
It's really just a "serving suggestion", the main focus is the lists and they don't need any immediate further maintenance, they'll get updated automagically when the database is updated.

The rest of the material is completely optional, I was riffing off've an eariler discussion in this forum (summarised in the "roadmap": https://gjhiggins.github.io/project/...oject-roadmap/). It's entirely up to the community how much, if any, of this is considered to be worth the effort of maintaining.

Quote:
the much smaller merits file - most gap searchers download this to determine whether their new finds are records ... this is a key practical tool.
It wasn't there, but it is now, automagically generated from the database. Unfortunately, it is constrained to be rendered in a web page which isn't the easiest way of accessing it.

A somewhat easier way is to d/l the sqlite3 database, open it with the sqlitebrowser app, pose a SQL query and save the results to a csv (comma-separated values) file which can be opened with a standard spreadsheet application (see illustration). It's not ideal but it will have the ahem, merit of being current.

(As regards the more concise primorial representation, if all the expanded primes could be rewritten in primorial form, the whole db could be recast as a much more convenient spreadsheet. I'm an ignoramus in this respect, is there a programmatic solution? As it is, that column blows out LibreOffice Calc's max cell limit.)

Cheers

Graham
Attached Thumbnails
Click image for larger version

Name:	sqlitebrowser-merits-to-csv.png
Views:	41
Size:	84.5 KB
ID:	21361  
gjhiggins is offline   Reply With Quote
Old 2019-11-26, 08:48   #150
robert44444uk
 
robert44444uk's Avatar
 
Jun 2003
Oxford, UK

111011101002 Posts
Default

Quote:
Originally Posted by gjhiggins View Post
It's really just a "serving suggestion", the main focus is the lists and they don't need any immediate further maintenance, they'll get updated automagically when the database is updated.

The rest of the material is completely optional, I was riffing off've an eariler discussion in this forum (summarised in the "roadmap": https://gjhiggins.github.io/project/...oject-roadmap/). It's entirely up to the community how much, if any, of this is considered to be worth the effort of maintaining.

It wasn't there, but it is now, automagically generated from the database. Unfortunately, it is constrained to be rendered in a web page which isn't the easiest way of accessing it.

A somewhat easier way is to d/l the sqlite3 database, open it with the sqlitebrowser app, pose a SQL query and save the results to a csv (comma-separated values) file which can be opened with a standard spreadsheet application (see illustration). It's not ideal but it will have the ahem, merit of being current.

(As regards the more concise primorial representation, if all the expanded primes could be rewritten in primorial form, the whole db could be recast as a much more convenient spreadsheet. I'm an ignoramus in this respect, is there a programmatic solution? As it is, that column blows out LibreOffice Calc's max cell limit.)

Cheers

Graham
Hi Graham

That's good that we can generate the simple list of merits. Question - Is the SQL query script for creating this csv available?

The world of gap searching has many angles, and using deficient primorials is only one possible route to finding large gaps. (A deficient primorial is of the form p#/x). Hence many of the awkward records cannot be put in this form. Maybe an idea is to have a separate database of these non standard records in their full form, and then each summarised within the authoritative database within the bounds of the field's maximum alphanumeric count. That would significantly reduce the size of the authoritative database.

I'm unclear on how to use the database. I have downloaded SQLite and the database and I can see the database within SQLite. Now what?

I need an idiot's guide!
  • How do I add records to the database? - I would need to do this in an automated way.
  • What format would the new records list need to be in to be accepted by the database?
  • How then does the database become the authoritative database? Do I have to upload the revised database?
  • Does the database check to see if it the added record is superior to the existing one and reject it if it does not meet that criteria?
  • Do records on the database drop off the database if they are beaten, or does the database just get larger?
  • I take it the database is not checking that the gap is correct, so we need an agreed way to ensure that uploading a rogue record can't happen.
  • Who can update the database? - worried about hackers or irresponsible checkers
  • What controls are in place if two people or more are updating the database at the same time?

Finally it seems a good opportunity to add a field for the month of discovery, blank for existing records, but an opportunity to add a useful field.

Last fiddled with by robert44444uk on 2019-11-26 at 11:50
robert44444uk is offline   Reply With Quote
Old 2019-11-27, 09:23   #151
robert44444uk
 
robert44444uk's Avatar
 
Jun 2003
Oxford, UK

22·32·53 Posts
Default

I was thinking about some of the controls that we would need to put into place to ensure that the database file was kept in good order. These rules largely mirror the Dr. Nicely approach to control.

Grateful for views on this:

1. Only people who are members of Mersenneforum should be allowed to submit new gaps, or have a sponsor who is a member
2. Only specific members will be allowed access to update the database - for this we might need volunteers
3. All gaps of less than 1e6 will be checked by the applicant using cglp4
4. Applicants wishing to have their gaps uploaded should post the cglp4.out file, and the required database upload file in correct format to this forum and request the database update
5. All gaps in excess of 1e6 will require sieve and prp checking evidence for all integers of the gap, including the bookends, posted to this forum, or made available to a checker, along with the correctly formatted upload file
6. Any gap that is a new candidate for the largest gap must have a second person carry out a 100% check before being accepted as the record gap
7. No gap will be accepted with merit <10, except for the record size gap
8. We will distinguish between finder, the software and program used, team members and/or helper attribution, in the database
robert44444uk is offline   Reply With Quote
Old 2019-11-28, 02:48   #152
gjhiggins
 
"Graham Higgins"
Nov 2019
UK

19 Posts
Default curating allgaps.dat

Quote:
Originally Posted by robert44444uk View Post
I need an idiot's guide!
Understood. There's a lot to be taken in.

I've made an attempt to expound the rationale and the approach:

https://gjhiggins.github.io/project/...g-allgaps-dat/

Cheers

Graham
gjhiggins is offline   Reply With Quote
Old 2019-11-28, 12:42   #153
robert44444uk
 
robert44444uk's Avatar
 
Jun 2003
Oxford, UK

22·32·53 Posts
Default

Again massive progress over the last 24 hours - thank you Graham.. I can see that it will be possible to use this solution, but other than that I'm not sure what to do.

The method though for doing things on GitHub is pretty counter-intuitive, and I, and expect others, will literally need to be walked though how to do things. I am feeling quite the idiot in need of the idiots guide.
  • I've opened up a GitHub account.
  • I have a cglp4.out file with some records from August 2019, which may or may not be records now, but were records according to the last Nicely merit file. They may now no longer be records because others may have already updated the records file.
  • I also have .txt file with the same records in Dr Nicely format.
  • I have downloaded SQLite.
So now I need to know, step by step, what to do.
robert44444uk is offline   Reply With Quote
Old 2019-11-28, 19:11   #154
MJansen
 
Jan 2018

2×19 Posts
Default

Hi Graham and Robert,


I did not have a chance to respond sooner, but both of you huge thanks for the additions and further developing the site! Really appreciated !


I am however in the same position as Robert and without more detailed expanation, I would not be able to add anything yet to the site.


May I suggest a practical test, based on a gap I found recently (the last gap with merit below 25 in the range 10.000-14.998:
13918 10645257227*547#/30 - 7804


Above is the format I would submit the gap to Thomas. Now if I would want to add this result to the new database, what do I need to do? Can you provide me with a step by step manual?


PS Would it be possible to add a text file with a few hundred gaps like these, to the site that can be read by the database? Or is it only possible to add one gap at the time?


Kind regards
Michiel Jansen




Quote:
Originally Posted by robert44444uk View Post
Again massive progress over the last 24 hours - thank you Graham.. I can see that it will be possible to use this solution, but other than that I'm not sure what to do.

The method though for doing things on GitHub is pretty counter-intuitive, and I, and expect others, will literally need to be walked though how to do things. I am feeling quite the idiot in need of the idiots guide.
  • I've opened up a GitHub account.
  • I have a cglp4.out file with some records from August 2019, which may or may not be records now, but were records according to the last Nicely merit file. They may now no longer be records because others may have already updated the records file.
  • I also have .txt file with the same records in Dr Nicely format.
  • I have downloaded SQLite.
So now I need to know, step by step, what to do.
MJansen is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
Prime gaps Terence Schraut Miscellaneous Math 10 2020-09-01 23:49
Welcome to the Prime Gaps Search Forum robert44444uk Prime Gap Searches 2 2019-09-23 01:00
Prime gaps and storage HellGauss Computer Science & Computational Number Theory 18 2015-11-16 14:21
Nicely done PrimeGrid - Record Woodall Prime axn Prime Cullen Prime 7 2007-09-03 08:48
Gaps and more gaps on <300 site gd_barnes Riesel Prime Search 11 2007-06-27 04:12

All times are UTC. The time now is 11:29.

Fri Dec 4 11:29:20 UTC 2020 up 1 day, 7:40, 0 users, load averages: 1.43, 1.62, 1.57

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.