Best RDBMS for arbitrarily large integers?

[mrh]

I know you said RDBMS, but just in case you are interested... If you happen to be using golang, BadgerDB is impressive if you just need key/value pairs. I've also used AWS DynamoDB a number of times when I didn't want to worry about where to put the data. I found it easy to use.

[Tyler Busby]

Originally Posted by mrh

I know you said RDBMS, but just in case you are interested... If you happen to be using golang, BadgerDB is impressive if you just need key/value pairs. I've also used AWS DynamoDB a number of times when I didn't want to worry about where to put the data. I found it easy to use.

Thanks for the recommendation! I'm more of a python guy, but I'm glad to hear all recs regardless. As I hadn't heard of DynamoDB or BadgerDB until now.


I've managed to whip up a little sqlite schema along with some python scripts to manage it, just to prototype and see what works well and what doesn't. For the time being, pickling gmpy2 mpz objects and storing those in a unique blob column for values worked basically seamlessly. Just have to store num_digits and a separate unique PK as well, and most queries won't be too painful.

Moreso than anything, the parser/expression evaluator ended up being the more difficult piece of the puzzle (when trying to machine parse the wiki page of sequences needing factors), but after accidentally trying to calculate 2^50331648+1 a few times and running out of memory, I have a moderately fault tolerant parser for that page :^)

Converting ecm efforts of the form <curves>@<B1> into a t-level that can be passed to yafu was another difficult problem, I basically converted WraithX's ecmprobs javascript into python exactly and used that. Although come to think of it, yafu probably has some version of this transformation embedded in it I could've extracted.

The next piece is to determine the next "least work" composite to factor for, say, 5 t-levels. i.e. given many composites and associated t levels of ecm effort, which one will have the fastest completion of the next 5 t-levels?

For example, say you have these composites and ecm work done (and no other info) and want to find a factor as soon as possible:

C289    t30
C180    t35
C150    t40
C500    t20
C1239   t5

It doesn't seem easy to know best how to distribute your efforts. The heuristic I keep returning to is: only work on composites less than some size, probably cutoff around 300-500. Then work on the composite with the lowest existing work level. But once they all get to about t30-40, I'd think this would start to break down as the larger composites would take much longer than the smaller ones.

I think this could be solved with math used for WraithX's ecmprob's bottom graph, but that seems a little bit harder to translate into python, just because there's much more data. If I finish doing that, I think it might be worthwhile to create and release ecmprobs.py that can calculate useful transformations relating to ecm work, like [(curves,b1)]->t_level; (digits, t_level_diff, threads)->completion_time; as ecmprobs seems to be the only place you can directly access these types of non-trivial calculations right now.

[chalsall]

Originally Posted by Tyler Busby

Thanks for the recommendation! I'm more of a python guy...

As are most. I won't hire anyone who can't work with low-level C.

[Tyler Busby]

Originally Posted by chalsall

As are most. I won't hire anyone who can't work with low-level C.

C certainly has its place! I've been meaning to get back to writing more C for performance dependent/system level stuff (or maybe learn rust). But it's tough to beat python when you want to haphazardly tape together various executables/C bindings modules for fun like a mad scientist

I just mention preferring Python over Go as they occupy the same programming language stratum in my mind (although I'm sure some might object to that). Not that I'm only capable of programming in Python.

But Python's generators/coroutines mixed with modules like gmpy2/primesieve/numpy/sympy really make it feel like you can get 80% of the way to the performance of a super optimized C program with 20% of the effort. And I just haven't really felt the need to squeeze out the last bit of performance by switching to C for any specific subproblem. And when I do, it's usually already been written by someone in a program that's used widely here.

[chalsall]

Originally Posted by Tyler Busby

C certainly has its place.

Yes. As does handcrafted asm..

[mrh]

Originally Posted by chalsall

Yes. As does handcrafted asm..

I'm and old school asm/C person, started in the PDP-11 days. Recently took up using go, and quite enjoyed it. And I'm pretty grumpy...