Best RDBMS for arbitrarily large integers?

[Tyler Busby]

Hi! I wasn't sure where to ask, so I made a thread here. I was curious if anybody here had experience with using an RDBMS for storing things like large prime candidates to test, composites to factor, etc. My application would have a small enough scope to where I'd use the lightest-weight one possible (probably SQLite) if all things were equal. But it seems like the type of thing a certain RDBMS might have better support for (I'd guess postgres or maybe something I've never heard of).

Obviously you'd want to store large integers as more compact expressions and whatnot when possible. And I could, of course, spend all sorts of time implementing arbitrary integer features on top of blobs on a RDBMS system that doesn't really have great existing support, but I figured this is exactly the type of thing that someone has probably figured out before me.

I did attempt to see what RDBMS factordb is using, as I assume that's the most successful project close to what I'm hoping to implement to myself, but it doesn't seem to have been discussed anywhere that I could find (or on the website). And I'd bug Markus, but I already have a small feature request I've sent him, so I didn't want to bug him more :)

Thanks!

[paulunderwood]

https://www.sqlite.org/limits.html says 1,000,000,000 for text and blob.

[axn]

Postgres supports numbers upto 131k digits natively (i.e. you can do arithmetic with it).

Almost all DBs support LOBs (Large Objects) - both Binary and Character variants. Typical size is 2^31 bytes / characters. However, the DB only assists with storing -- any maths on it will need to be done outside.


I suspect FactorDB will be using either mysql or postgres, as these are the most popular "free" RDBMS.

[xilman]

I keep my factoring data in PostgreSQL.

If you wish, I could provide you with the schema in use.

[Tyler Busby]

Originally Posted by xilman

I keep my factoring data in PostgreSQL.

If you wish, I could provide you with the schema in use.

That would be lovely if you don't mind! I'm sure it would be a great starting point.

[chris2be8]

Originally Posted by Tyler Busby

I did attempt to see what RDBMS factordb is using, as I assume that's the most successful project close to what I'm hoping to implement to myself, but it doesn't seem to have been discussed anywhere that I could find (or on the website).

I've seen messages like:
1040: SQLSTATE[HY000] [1040] Too many connections
come out on occcasion. Duckduckgo searches suggest it's from mysql/mariadb.

But that might be overkill for you.

[Tyler Busby]

Originally Posted by chris2be8

I've seen messages like:
1040: SQLSTATE[HY000] [1040] Too many connections
come out on occcasion. Duckduckgo searches suggest it's from mysql/mariadb.

But that might be overkill for you.

Precisely the type of information leakage I was hoping somebody here had caught :) Good to know!

I've always had a fondness for postgres, so I may rise to that level of overkill if xilman's schema suits my project. But realistically, storing text/blobs in sqlite would probably be fine for anything I can imagine right now (basically just personal factoring queues, storing/sorting prime candidates for sieving/pfgw, possibly reimplementing some factordb-esque features to assist with those two tasks)

Further down the line, I would maybe want to have this software able to interface with ecmnet client/server software to coordinate multiple computers (probably just my own) factoring against this queue, as that seems like less overhead than going the boinc route.

I mostly want to moderately automate finding the most tractable composites relating to OEIS sequences that need factoring efforts, and hammer them with YAFU :)

This also stems from the desire to know how much ecm an arbitrary composite on factordb has had run against it (even if just by me on a different day that I've forgotten about). It doesn't seem likely Markus will implement this type of self reporting (and probably for good reason, it's a difficult trust problem), but is maybe something that a third party could track and display with an extension, if the ECM is done via a trusted framework.

An example workflow: factorize a number on factordb -> there's a remaining C200 -> "ecmdb" extension annotates the composite as having no tracked ECM on it -> click the annotation -> "ecmdb" spins up an ecmnet server (or similar) and your extension boots the client with the server info -> if you find the factor, "ecmdb" reports it to factordb. If you don't find the factor after x curves and quit, ecmdb updates its internal ecm tracking and shows that "progress" on the factordb annotation for other "ecmdb" extension users, who can also use the ecmnet server if it was already created, or if they'd prefer not to, they will at least not duplicate ecm efforts.

I'll likely not implement that entire idea any time soon, but wanted to at least think through the design out loud :) since I have the day off today, I'll probably just get started on a little sqlite version to track composites that I'm interested in, and take it from there.

[EdH]

If you're looking for in-house only ECM accross several machines, you might look at ecmpi. I use it for my local machines and have scripts that will step through levels and even have it implemented as the ECM method for Aliqueit, via aliqueit.ini. I have a brief "How I. . ." in the blog area, if interested. As to the more extensive scripts, they're knd of thrown together for my specific setup, but I might be able to give some basics for those as well.

[xilman]

Originally Posted by Tyler Busby

That would be lovely if you don't mind! I'm sure it would be a great starting point.

DM me with your email address.

I will provide some Perl scripts, one of which contains the schema and is used to initialize the DB. It will take me a little while to ensure that no secrets are revealed, such as the account names and passwords used.

I can also provide a populater pg_dump which well let you explore the database containing real data.

Paul

[Tyler Busby]

Originally Posted by xilman

DM me with your email address.

I will provide some Perl scripts, one of which contains the schema and is used to initialize the DB. It will take me a little while to ensure that no secrets are revealed, such as the account names and passwords used.

I can also provide a populater pg_dump which well let you explore the database containing real data.

Paul

That would be great, sent you my email! Thanks for doing this for me.

Originally Posted by EdH

If you're looking for in-house only ECM accross several machines, you might look at ecmpi. I use it for my local machines and have scripts that will step through levels and even have it implemented as the ECM method for Aliqueit, via aliqueit.ini. I have a brief "How I. . ." in the blog area, if interested. As to the more extensive scripts, they're knd of thrown together for my specific setup, but I might be able to give some basics for those as well.

Ah! That's a good recommendation as I didn't know about ecmpi. I'll keep it in mind going forward, but I think the computing power disparity between my best and second best machine is enough to where setting up ecmpi might be more trouble than it's worth until I have the rest of my system fleshed out. I do wonder if using ecmpi to coordinate simultaneous gpu and cpu ecm (from one machine) on a single composite would be a worthwhile endeavor, and if this amalgamation could be plugged into yafu for ecm.

[EdH]

I'm unfamiliar with ecmpi working with a GPU. Parallel processes could be scripted, but ecm.py or YAFU would be my choice for a single machine CPU tasking. GMP-ECM with CGBN would be my choice for GPU, but the GPU will only do stage 1. Stage 2 would still need to be done via CPU threads.

I actually have a GPU set up on a dual core, no hyper-threading, machine that sends the residues to machines with either 40/24/20 threads, depending on what's available at the time.

ETA: Just an additional note: I find my ecmpi cluster well outperforms my GPUs, even though my better GPU has 1536 nodes and my ecmpi cluster has ~ 200 nodes.