mersenneforum.org  

Go Back   mersenneforum.org > Great Internet Mersenne Prime Search > Software

Reply
 
Thread Tools
Old 2020-02-02, 15:37   #1
jas
 
"Simon Josefsson"
Jan 2020
Stockholm

1816 Posts
Default Best settings for 2xE5-2673 machine?

I'm struggling with how to configure mprime the best. I have a machine with two E5-2673v3 CPUs. They have 12 cores each, so 24 cores, and multithreaded so /proc/cpuinfo shows 48 cpus. How many workers should I use? How many threads per worker? What give the best performance? The default seems to be 6 workers with 4 threads each. Is this the best?


Thanks,
Simon
jas is offline   Reply With Quote
Old 2020-02-03, 15:23   #2
tServo
 
tServo's Avatar
 
"Marv"
May 2009
near the Tannhäuser Gate

10011111012 Posts
Default

Quote:
Originally Posted by jas View Post
I'm struggling with how to configure mprime the best. I have a machine with two E5-2673v3 CPUs. They have 12 cores each, so 24 cores, and multithreaded so /proc/cpuinfo shows 48 cpus. How many workers should I use? How many threads per worker? What give the best performance? The default seems to be 6 workers with 4 threads each. Is this the best?


Thanks,
Simon
Since few, if any of us, have a setup similar to yours, the best thing to do is to try some experiments yourself. As a general rule, since mprime is memory bandwidth limited, you only need test a small number of workers. Start at 1 and gradually increase # cores. Also, turn hyperthreading off ( that is usually a bios setting ).
At some point, increasing # cores will reduce thruput. At that point, do the same test with 2 workers starting with 1 core and working up.
Repeat, rinse.

BTW, instead of incrementing # cores by 1 ( which would be tedious ), start at 2 and increment by 2.
tServo is offline   Reply With Quote
Old 2020-02-03, 17:50   #3
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

117608 Posts
Default

Quote:
Originally Posted by tServo View Post
Since few, if any of us, have a setup similar to yours, the best thing to do is to try some experiments yourself. As a general rule, since mprime is memory bandwidth limited, you only need test a small number of workers. Start at 1 and gradually increase # cores. Also, turn hyperthreading off ( that is usually a bios setting ).
At some point, increasing # cores will reduce thruput. At that point, do the same test with 2 workers starting with 1 core and working up.
Repeat, rinse.

BTW, instead of incrementing # cores by 1 ( which would be tedious ), start at 2 and increment by 2.
Agreed on benchmark the specific hardware. Even if someone had done the test on their own same cpu model hardware, that doesn't necessarily mean the ram sticks are the same speed or type.

Doesn't mprime automate benchmarking vs. fft length and worker count and whether hyperthreading is used or not, like prime95 does? Mprime's readme and undoc text files seem to indicate it does. The tables given at https://www.mersenneforum.org/showpo...18&postcount=4 and https://www.mersenneforum.org/showpo...19&postcount=5 for a very few cpu models are from individual automated benchmark runs each.

On hyperthread capable systems, generally not using hyperthreading in primality testing is faster, but turning it off in the BIOS is not necessary. Also I've found cases where hyperthreading fft benchmarks are faster (i7-4790 is one where it benchmarked faster on some fft lengths.) When doing other things also on the same system, there's an advantage to having hyperthreading available to run those incidental other activities without occasionally stopping an entire prime95 worker to free up a single cpu core.

Also note that there seems to be an advantage to using a number of cores per worker that is, or is a factor of, the number of cores per cpu package. For a dual 6-core, try 1,2,3,6 cores/worker, 12,6,4,2 workers. Four cores/worker will be slower, since it is 3 workers, and so at least one of the workers straddles the package interconnect which is slower than on-chip caching.
Attached Thumbnails
Click image for larger version

Name:	benchmark setup and launch.png
Views:	109
Size:	10.5 KB
ID:	21714  

Last fiddled with by kriesel on 2020-02-03 at 18:22
kriesel is online now   Reply With Quote
Old 2020-02-04, 11:36   #4
jas
 
"Simon Josefsson"
Jan 2020
Stockholm

1816 Posts
Default

Quote:
Originally Posted by tServo View Post
Since few, if any of us, have a setup similar to yours, the best thing to do is to try some experiments yourself. As a general rule, since mprime is memory bandwidth limited, you only need test a small number of workers. Start at 1 and gradually increase # cores. Also, turn hyperthreading off ( that is usually a bios setting ).
At some point, increasing # cores will reduce thruput. At that point, do the same test with 2 workers starting with 1 core and working up.
Repeat, rinse.

BTW, instead of incrementing # cores by 1 ( which would be tedious ), start at 2 and increment by 2.
Thank you -- I just found the Benchmarking option of the mprime program, so I'm running it now and it generates interesting data. I'm not sure how to interpret it yet though.

What FFT size is relevant to use? When running for real, I want it to attempt 100M+ primes. I assume the ideal settings may differ for a 2048k FFT and a 8192k FFT operation. Will 100M+ primes always use 4k FFT, always 8k FFT, or use all sizes?

How should I interpret outputs like below?

/Simon


Quote:
[Worker #1 Feb 4 10:15] Timing 2048K FFT, 12 cores, 1 worker. Average times: 5.10 ms. Total throughput: 195.99 iter/sec.
[Worker #1 Feb 4 10:15] Timing 2048K FFT, 12 cores, 2 workers. Average times: 7.88, 8.00 ms. Total throughput: 251.98 iter/sec.
[Worker #1 Feb 4 10:16] Timing 2048K FFT, 12 cores, 3 workers. Average times: 15.83, 15.83, 8.91 ms. Total throughput: 238.63 iter/sec.
[Worker #1 Feb 4 10:16] Timing 2048K FFT, 12 cores, 12 workers. Average times: 47.25, 47.24, 47.59, 47.23, 47.24, 47.24, 40.75, 39.98, 40.42, 40.32, 40.31, 39.88 ms. Total throughput: 275.83 iter/sec.
[Worker #1 Feb 4 10:16] Timing 2048K FFT, 12 cores hyperthreaded, 1 worker. Average times: 6.43 ms. Total throughput: 155.48 iter/sec.
[Worker #1 Feb 4 10:17] Timing 2048K FFT, 12 cores hyperthreaded, 2 workers. Average times: 10.61, 10.33 ms. Total throughput: 191.07 iter/sec.
[Worker #1 Feb 4 10:17] Timing 2048K FFT, 12 cores hyperthreaded, 3 workers. Average times: 19.34, 19.11, 10.52 ms. Total throughput: 199.11 iter/sec.
[Worker #1 Feb 4 10:17] Timing 2048K FFT, 12 cores hyperthreaded, 12 workers. Average times: 53.26, 54.25, 52.04, 53.28, 54.19, 53.64, 54.48, 54.07, 52.88, 53.98, 53.40, 53.64 ms. Total throughput: 223.94 iter/sec.
...
[Worker #1 Feb 4 10:54] Timing 4096K FFT, 12 cores, 1 worker. Average times: 11.07 ms. Total throughput: 90.34 iter/sec.
[Worker #1 Feb 4 10:54] Timing 4096K FFT, 12 cores, 2 workers. Average times: 17.54, 17.52 ms. Total throughput: 114.08 iter/sec.
[Worker #1 Feb 4 10:54] Timing 4096K FFT, 12 cores, 3 workers. Average times: 32.62, 32.57, 17.40 ms. Total throughput: 118.83 iter/sec.
[Worker #1 Feb 4 10:55] Timing 4096K FFT, 12 cores, 12 workers. Average times: 93.72, 92.12, 94.67, 94.92, 94.92, 94.33, 95.64, 94.42, 95.13, 95.70, 93.93, 94.53 ms. Total throughput: 126.99 iter/sec.
[Worker #1 Feb 4 10:55] Timing 4096K FFT, 12 cores hyperthreaded, 1 worker. Average times: 12.91 ms. Total throughput: 77.44 iter/sec.
[Worker #1 Feb 4 10:56] Timing 4096K FFT, 12 cores hyperthreaded, 2 workers. Average times: 23.04, 23.16 ms. Total throughput: 86.58 iter/sec.
[Worker #1 Feb 4 10:56] Timing 4096K FFT, 12 cores hyperthreaded, 3 workers. Average times: 45.54, 45.54, 22.99 ms. Total throughput: 87.42 iter/sec.
[Worker #1 Feb 4 10:56] Timing 4096K FFT, 12 cores hyperthreaded, 12 workers. Average times: 135.01, 131.99, 131.93, 131.53, 133.24, 131.95, 134.56, 132.71, 134.66, 131.63, 132.73, 134.58 ms. Total throughput: 90.20 iter/sec.
...
[Worker #1 Feb 4 11:47] Timing 8192K FFT, 12 cores, 1 worker. Average times: 23.57 ms. Total throughput: 42.43 iter/sec.
[Worker #1 Feb 4 11:47] Timing 8192K FFT, 12 cores, 2 workers. Average times: 40.77, 40.79 ms. Total throughput: 49.04 iter/sec.
[Worker #1 Feb 4 11:47] Timing 8192K FFT, 12 cores, 3 workers. Average times: 81.39, 81.39, 40.57 ms. Total throughput: 49.22 iter/sec.
[Worker #1 Feb 4 11:48] Timing 8192K FFT, 12 cores, 12 workers. Average times: 231.34, 222.30, 227.74, 230.12, 231.22, 228.06, 232.29, 236.49, 230.23, 233.18, 232.87, 232.83 ms. Total throughput: 52.02 iter/sec.
[Worker #1 Feb 4 11:49] Timing 8192K FFT, 12 cores hyperthreaded, 1 worker. Average times: 28.77 ms. Total throughput: 34.76 iter/sec.
[Worker #1 Feb 4 11:49] Timing 8192K FFT, 12 cores hyperthreaded, 2 workers. Average times: 51.27, 50.46 ms. Total throughput: 39.33 iter/sec.
[Worker #1 Feb 4 11:49] Timing 8192K FFT, 12 cores hyperthreaded, 3 workers. Average times: 100.34, 100.81, 51.34 ms. Total throughput: 39.36 iter/sec.
[Worker #1 Feb 4 11:50] Timing 8192K FFT, 12 cores hyperthreaded, 12 workers. Average times: 299.96, 301.02, 296.25, 294.32, 302.67, 297.46, 295.57, 299.05, 301.31, 295.53, 300.18, 294.27 ms. Total throughput: 40.25 iter/sec.
jas is offline   Reply With Quote
Old 2020-02-04, 13:04   #5
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

24×11×29 Posts
Default

Three workers is usually a poor choice on a dual-package system. (Really, any odd number > 1.)
Benchmark for 1,2,3,4,6,12 workers on your dual-6core system. It will be slow. But you don't have to do it very often.

If you plan to run 100Mdigit exponents, you'll need to benchmark up to at least 18432K fft length, perhaps 20480K. In general, the number of bits per fft word is limited to around 17-18, trending lower at larger exponents. A 100Mdigit exponent is ~333Mbits. So 333M/17 implies an fft length around 20M (20480K).

Use the # of workers that gives highest total throughput (system total iterations/second) for a given fft length, that you expect will be used by the exponents you run. Or pick the # of workers that gives a good compromise on the several fft lengths you may use, and stick with it. If the maximum throughput choice means time to complete a primality test is too long compared to assignment expiration, go to fewer workers. For ordinary prime hunting or double-checking, the best choices change with time as the GIMPS wavefront moves to larger exponents.

The results you posted indicate that using hyperthreading in mprime on your system for primality testing is not efficient. Prime95 default is to not enable hyperthreading for primality testing, because that is usually the case. Probably mprime is the same. Check.

I find it useful to put the benchmarking results into a spreadsheet for later easy reference. A column for maximum exponent for each fft length could be very useful.

Last fiddled with by kriesel on 2020-02-04 at 13:29
kriesel is online now   Reply With Quote
Old 2020-02-04, 13:25   #6
axn
 
axn's Avatar
 
Jun 2003

25×5×31 Posts
Default

Quote:
Originally Posted by kriesel View Post
Three workers is usually a poor choice on a dual-package system.
2673 is a 12-core machine, so I would guess that the benchmarks are only for 1 socket.

Working on 100M digits exponent, probably the best thing to do is to run one worker per socket -- to minimize the (months long) completion time for an exponent.

Of course, actual benchmarks for that FFT size (18M) will tell how much of a thruput loss it is.
axn is offline   Reply With Quote
Old 2020-02-04, 13:32   #7
kriesel
 
kriesel's Avatar
 
"TF79LL86GIMPS96gpu17"
Mar 2017
US midwest

24·11·29 Posts
Default

Quote:
Originally Posted by axn View Post
2673 is a 12-core machine, so I would guess that the benchmarks are only for 1 socket.
That's not how prime95 works. See https://www.mersenneforum.org/showpo...18&postcount=4

You're right that latency will be an issue. Also cache effectiveness is likely better with fewer workers at large data size each.

And run PRP with its far superior error detection and correction. Don't even consider LL.

Last fiddled with by kriesel on 2020-02-04 at 13:38
kriesel is online now   Reply With Quote
Old 2020-02-04, 13:56   #8
axn
 
axn's Avatar
 
Jun 2003

25×5×31 Posts
Default

Quote:
Originally Posted by kriesel View Post
That's not how prime95 works.
Yet, the benchmark is clearly for 12 cores, not 24. Hmmm....
axn is offline   Reply With Quote
Old 2020-02-04, 16:26   #9
jas
 
"Simon Josefsson"
Jan 2020
Stockholm

23·3 Posts
Default

Quote:
Originally Posted by kriesel View Post
Three workers is usually a poor choice on a dual-package system. (Really, any odd number > 1.)
Benchmark for 1,2,3,4,6,12 workers on your dual-6core system. It will be slow. But you don't have to do it very often.

If you plan to run 100Mdigit exponents, you'll need to benchmark up to at least 18432K fft length, perhaps 20480K. In general, the number of bits per fft word is limited to around 17-18, trending lower at larger exponents. A 100Mdigit exponent is ~333Mbits. So 333M/17 implies an fft length around 20M (20480K).

Use the # of workers that gives highest total throughput (system total iterations/second) for a given fft length, that you expect will be used by the exponents you run. Or pick the # of workers that gives a good compromise on the several fft lengths you may use, and stick with it. If the maximum throughput choice means time to complete a primality test is too long compared to assignment expiration, go to fewer workers. For ordinary prime hunting or double-checking, the best choices change with time as the GIMPS wavefront moves to larger exponents.

The results you posted indicate that using hyperthreading in mprime on your system for primality testing is not efficient. Prime95 default is to not enable hyperthreading for primality testing, because that is usually the case. Probably mprime is the same. Check.

I find it useful to put the benchmarking results into a spreadsheet for later easy reference. A column for maximum exponent for each fft length could be very useful.

Thank you for hints! I will run much larger FFT sizes. If anybody is interested, below is statistics for 2048k FFT on this machine with more core/worker combinations... Seems to peak at 22 core and 2 workers with 1498 iter/sec.


If anyone else has other observations, please share.

Code:
[Worker #1 Feb 4 12:33] Timing 2048K FFT, 1 core, 1 worker.  Average times: 10.32 ms.  Total throughput: 96.90 iter/sec.
[Worker #1 Feb 4 12:33] Timing 2048K FFT, 1 core hyperthreaded, 1 worker.  Average times:  9.89 ms.  Total throughput: 101.12 iter/sec.
[Worker #1 Feb 4 12:34] Timing 2048K FFT, 2 cores, 1 worker.  Average times:  5.55 ms.  Total throughput: 180.04 iter/sec.
[Worker #1 Feb 4 12:34] Timing 2048K FFT, 2 cores, 2 workers.  Average times: 10.52, 10.54 ms.  Total throughput: 189.96 iter/sec.
[Worker #1 Feb 4 12:34] Timing 2048K FFT, 2 cores hyperthreaded, 1 worker.  Average times:  5.42 ms.  Total throughput: 184.42 iter/sec.
[Worker #1 Feb 4 12:35] Timing 2048K FFT, 2 cores hyperthreaded, 2 workers.  Average times:  9.80,  9.84 ms.  Total throughput: 203.73 iter/sec.
[Worker #1 Feb 4 12:35] Timing 2048K FFT, 3 cores, 1 worker.  Average times:  4.01 ms.  Total throughput: 249.20 iter/sec.
[Worker #1 Feb 4 12:35] Timing 2048K FFT, 3 cores, 2 workers.  Average times:  5.60, 10.58 ms.  Total throughput: 273.04 iter/sec.
[Worker #1 Feb 4 12:36] Timing 2048K FFT, 3 cores, 3 workers.  Average times: 10.73, 10.68, 10.45 ms.  Total throughput: 282.46 iter/sec.
[Worker #1 Feb 4 12:36] Timing 2048K FFT, 3 cores hyperthreaded, 1 worker.  Average times:  3.71 ms.  Total throughput: 269.19 iter/sec.
[Worker #1 Feb 4 12:36] Timing 2048K FFT, 3 cores hyperthreaded, 2 workers.  Average times:  5.32,  9.97 ms.  Total throughput: 288.15 iter/sec.
[Worker #1 Feb 4 12:37] Timing 2048K FFT, 3 cores hyperthreaded, 3 workers.  Average times: 10.10, 10.42, 10.09 ms.  Total throughput: 294.08 iter/sec.
[Worker #1 Feb 4 12:37] Timing 2048K FFT, 4 cores, 1 worker.  Average times:  3.17 ms.  Total throughput: 315.21 iter/sec.
[Worker #1 Feb 4 12:38] Timing 2048K FFT, 4 cores, 2 workers.  Average times:  5.63,  5.55 ms.  Total throughput: 357.67 iter/sec.
[Worker #1 Feb 4 12:38] Timing 2048K FFT, 4 cores, 3 workers.  Average times: 10.82, 10.75,  5.61 ms.  Total throughput: 363.74 iter/sec.
[Worker #1 Feb 4 12:38] Timing 2048K FFT, 4 cores, 4 workers.  Average times: 10.96, 10.87, 10.73, 11.10 ms.  Total throughput: 366.60 iter/sec.
[Worker #1 Feb 4 12:39] Timing 2048K FFT, 4 cores hyperthreaded, 1 worker.  Average times:  3.06 ms.  Total throughput: 326.70 iter/sec.
[Worker #1 Feb 4 12:39] Timing 2048K FFT, 4 cores hyperthreaded, 2 workers.  Average times:  5.45,  5.38 ms.  Total throughput: 369.39 iter/sec.
[Worker #1 Feb 4 12:39] Timing 2048K FFT, 4 cores hyperthreaded, 3 workers.  Average times: 10.36, 10.53,  5.38 ms.  Total throughput: 377.52 iter/sec.
[Worker #1 Feb 4 12:40] Timing 2048K FFT, 4 cores hyperthreaded, 4 workers.  Average times: 10.77, 11.03, 11.43, 11.38 ms.  Total throughput: 358.87 iter/sec.
[Worker #1 Feb 4 12:40] Timing 2048K FFT, 5 cores, 1 worker.  Average times:  2.52 ms.  Total throughput: 396.38 iter/sec.
[Worker #1 Feb 4 12:40] Timing 2048K FFT, 5 cores, 2 workers.  Average times:  3.99,  5.63 ms.  Total throughput: 428.32 iter/sec.
[Worker #1 Feb 4 12:41] Timing 2048K FFT, 5 cores, 3 workers.  Average times: 11.79,  5.93,  5.57 ms.  Total throughput: 432.95 iter/sec.
[Worker #1 Feb 4 12:41] Timing 2048K FFT, 5 cores, 4 workers.  Average times: 11.90,  5.99, 11.33, 10.86 ms.  Total throughput: 431.30 iter/sec.
[Worker #1 Feb 4 12:42] Timing 2048K FFT, 5 cores, 5 workers.  Average times: 14.00, 14.08, 14.10, 10.82, 10.99 ms.  Total throughput: 396.88 iter/sec.
[Worker #1 Feb 4 12:42] Timing 2048K FFT, 5 cores hyperthreaded, 1 worker.  Average times:  2.49 ms.  Total throughput: 401.79 iter/sec.
[Worker #1 Feb 4 12:42] Timing 2048K FFT, 5 cores hyperthreaded, 2 workers.  Average times:  3.72,  5.41 ms.  Total throughput: 453.26 iter/sec.
[Worker #1 Feb 4 12:43] Timing 2048K FFT, 5 cores hyperthreaded, 3 workers.  Average times: 11.48,  5.80,  5.58 ms.  Total throughput: 438.78 iter/sec.
[Worker #1 Feb 4 12:43] Timing 2048K FFT, 5 cores hyperthreaded, 4 workers.  Average times: 12.17,  5.76, 10.56, 10.80 ms.  Total throughput: 443.17 iter/sec.
[Worker #1 Feb 4 12:43] Timing 2048K FFT, 5 cores hyperthreaded, 5 workers.  Average times: 11.77, 14.16, 15.00, 11.23, 10.68 ms.  Total throughput: 404.92 iter/sec.
[Worker #1 Feb 4 12:44] Timing 2048K FFT, 6 cores, 1 worker.  Average times:  2.18 ms.  Total throughput: 458.95 iter/sec.
[Worker #1 Feb 4 12:44] Timing 2048K FFT, 6 cores, 2 workers.  Average times:  4.05,  4.07 ms.  Total throughput: 492.75 iter/sec.
[Worker #1 Feb 4 12:45] Timing 2048K FFT, 6 cores, 3 workers.  Average times: 11.77,  6.01,  4.04 ms.  Total throughput: 498.78 iter/sec.
[Worker #1 Feb 4 12:45] Timing 2048K FFT, 6 cores, 4 workers.  Average times: 12.54,  5.91, 12.12,  5.90 ms.  Total throughput: 500.90 iter/sec.
[Worker #1 Feb 4 12:45] Timing 2048K FFT, 6 cores, 5 workers.  Average times: 15.04, 13.69, 14.99, 11.86,  6.00 ms.  Total throughput: 457.21 iter/sec.
[Worker #1 Feb 4 12:46] Timing 2048K FFT, 6 cores, 6 workers.  Average times: 14.42, 13.80, 14.39, 17.25, 17.25, 15.88 ms.  Total throughput: 390.24 iter/sec.
[Worker #1 Feb 4 12:46] Timing 2048K FFT, 6 cores hyperthreaded, 1 worker.  Average times:  2.11 ms.  Total throughput: 473.53 iter/sec.
[Worker #1 Feb 4 12:47] Timing 2048K FFT, 6 cores hyperthreaded, 2 workers.  Average times:  3.71,  3.73 ms.  Total throughput: 537.47 iter/sec.
[Worker #1 Feb 4 12:47] Timing 2048K FFT, 6 cores hyperthreaded, 3 workers.  Average times: 11.17,  5.82,  3.78 ms.  Total throughput: 525.79 iter/sec.
[Worker #1 Feb 4 12:47] Timing 2048K FFT, 6 cores hyperthreaded, 4 workers.  Average times: 12.38,  5.83, 12.28,  5.80 ms.  Total throughput: 506.33 iter/sec.
[Worker #1 Feb 4 12:48] Timing 2048K FFT, 6 cores hyperthreaded, 5 workers.  Average times: 12.88, 12.65, 12.27, 11.93,  5.81 ms.  Total throughput: 494.22 iter/sec.
[Worker #1 Feb 4 12:48] Timing 2048K FFT, 6 cores hyperthreaded, 6 workers.  Average times: 15.86, 16.48, 16.48, 27.50, 24.04, 34.78 ms.  Total throughput: 291.12 iter/sec.
[Worker #1 Feb 4 12:49] Timing 2048K FFT, 7 cores, 1 worker.  Average times:  1.87 ms.  Total throughput: 534.93 iter/sec.
[Worker #1 Feb 4 12:49] Timing 2048K FFT, 7 cores, 2 workers.  Average times:  3.28,  4.07 ms.  Total throughput: 550.62 iter/sec.
[Worker #1 Feb 4 12:49] Timing 2048K FFT, 7 cores, 3 workers.  Average times:  6.46,  6.31,  4.05 ms.  Total throughput: 560.36 iter/sec.
[Worker #1 Feb 4 12:50] Timing 2048K FFT, 7 cores, 4 workers.  Average times:  7.12,  6.78, 12.49,  6.04 ms.  Total throughput: 533.69 iter/sec.
[Worker #1 Feb 4 12:50] Timing 2048K FFT, 7 cores, 5 workers.  Average times: 15.85, 15.51,  6.92, 12.46,  5.96 ms.  Total throughput: 520.21 iter/sec.
[Worker #1 Feb 4 12:50] Timing 2048K FFT, 7 cores, 6 workers.  Average times: 14.04, 16.42,  8.60, 14.61, 16.00, 14.64 ms.  Total throughput: 447.63 iter/sec.
[Worker #1 Feb 4 12:51] Timing 2048K FFT, 7 cores, 7 workers.  Average times: 15.97, 20.54, 15.97, 20.52, 14.08, 15.44, 15.56 ms.  Total throughput: 422.73 iter/sec.
[Worker #1 Feb 4 12:51] Timing 2048K FFT, 7 cores hyperthreaded, 1 worker.  Average times:  1.83 ms.  Total throughput: 545.19 iter/sec.
[Worker #1 Feb 4 12:52] Timing 2048K FFT, 7 cores hyperthreaded, 2 workers.  Average times:  3.07,  3.75 ms.  Total throughput: 592.76 iter/sec.
[Worker #1 Feb 4 12:52] Timing 2048K FFT, 7 cores hyperthreaded, 3 workers.  Average times:  6.23,  5.98,  3.75 ms.  Total throughput: 594.59 iter/sec.
[Worker #1 Feb 4 12:52] Timing 2048K FFT, 7 cores hyperthreaded, 4 workers.  Average times:  7.40,  6.96, 12.15,  5.87 ms.  Total throughput: 531.50 iter/sec.
[Worker #1 Feb 4 12:53] Timing 2048K FFT, 7 cores hyperthreaded, 5 workers.  Average times: 15.97, 16.06,  7.23, 13.21,  6.30 ms.  Total throughput: 497.48 iter/sec.
[Worker #1 Feb 4 12:53] Timing 2048K FFT, 7 cores hyperthreaded, 6 workers.  Average times: 14.33, 16.43,  7.46, 14.39, 15.65, 18.64 ms.  Total throughput: 451.76 iter/sec.
[Worker #1 Feb 4 12:54] Timing 2048K FFT, 7 cores hyperthreaded, 7 workers.  Average times: 18.82, 16.53, 19.88, 19.16, 18.48, 13.77, 13.77 ms.  Total throughput: 415.47 iter/sec.
[Worker #1 Feb 4 12:54] Timing 2048K FFT, 8 cores, 1 worker.  Average times:  1.71 ms.  Total throughput: 585.46 iter/sec.
[Worker #1 Feb 4 12:54] Timing 2048K FFT, 8 cores, 2 workers.  Average times:  3.28,  3.25 ms.  Total throughput: 612.47 iter/sec.
[Worker #1 Feb 4 12:55] Timing 2048K FFT, 8 cores, 3 workers.  Average times:  6.20,  6.46,  3.18 ms.  Total throughput: 630.26 iter/sec.
[Worker #1 Feb 4 12:55] Timing 2048K FFT, 8 cores, 4 workers.  Average times:  8.36,  6.87,  6.63,  6.64 ms.  Total throughput: 566.64 iter/sec.
[Worker #1 Feb 4 12:55] Timing 2048K FFT, 8 cores, 5 workers.  Average times: 16.83, 14.73,  6.98,  6.58,  6.98 ms.  Total throughput: 565.89 iter/sec.
[Worker #1 Feb 4 12:56] Timing 2048K FFT, 8 cores, 6 workers.  Average times: 15.66, 15.82,  7.18, 16.03, 16.13,  7.02 ms.  Total throughput: 533.13 iter/sec.
[Worker #1 Feb 4 12:56] Timing 2048K FFT, 8 cores, 7 workers.  Average times: 18.06, 19.19, 19.20, 18.04, 15.55, 15.43,  7.36 ms.  Total throughput: 479.99 iter/sec.
[Worker #1 Feb 4 12:57] Timing 2048K FFT, 8 cores, 8 workers.  Average times: 18.65, 18.55, 18.54, 18.57, 20.04, 18.02, 17.92, 19.72 ms.  Total throughput: 427.23 iter/sec.
[Worker #1 Feb 4 12:57] Timing 2048K FFT, 8 cores hyperthreaded, 1 worker.  Average times:  1.63 ms.  Total throughput: 611.63 iter/sec.
[Worker #1 Feb 4 12:58] Timing 2048K FFT, 8 cores hyperthreaded, 2 workers.  Average times:  3.21,  3.22 ms.  Total throughput: 622.75 iter/sec.
[Worker #1 Feb 4 12:58] Timing 2048K FFT, 8 cores hyperthreaded, 3 workers.  Average times:  6.12,  6.19,  3.12 ms.  Total throughput: 645.45 iter/sec.
[Worker #1 Feb 4 12:58] Timing 2048K FFT, 8 cores hyperthreaded, 4 workers.  Average times:  7.20,  7.04,  9.16, 12.93 ms.  Total throughput: 467.65 iter/sec.
[Worker #1 Feb 4 12:59] Timing 2048K FFT, 8 cores hyperthreaded, 5 workers.  Average times: 16.23, 14.96,  6.98,  8.42,  6.13 ms.  Total throughput: 553.56 iter/sec.
[Worker #1 Feb 4 12:59] Timing 2048K FFT, 8 cores hyperthreaded, 6 workers.  Average times: 16.56, 16.62,  7.46, 13.86, 18.83,  6.63 ms.  Total throughput: 530.77 iter/sec.
[Worker #1 Feb 4 13:00] Timing 2048K FFT, 8 cores hyperthreaded, 7 workers.  Average times: 18.61, 21.32, 14.17, 20.97, 20.22, 16.94,  7.92 ms.  Total throughput: 453.67 iter/sec.
[Worker #1 Feb 4 13:00] Timing 2048K FFT, 8 cores hyperthreaded, 8 workers.  Average times: 20.53, 20.51, 20.51, 20.63, 19.43, 16.74, 18.79, 16.75 ms.  Total throughput: 418.79 iter/sec.
[Worker #1 Feb 4 13:00] Timing 2048K FFT, 9 cores, 1 worker.  Average times:  1.54 ms.  Total throughput: 647.43 iter/sec.
[Worker #1 Feb 4 13:01] Timing 2048K FFT, 9 cores, 2 workers.  Average times:  2.42,  3.21 ms.  Total throughput: 725.99 iter/sec.
[Worker #1 Feb 4 13:01] Timing 2048K FFT, 9 cores, 3 workers.  Average times:  6.53,  4.09,  3.27 ms.  Total throughput: 703.67 iter/sec.
[Worker #1 Feb 4 13:02] Timing 2048K FFT, 9 cores, 4 workers.  Average times:  6.94,  4.22,  7.05,  6.25 ms.  Total throughput: 682.80 iter/sec.
[Worker #1 Feb 4 13:02] Timing 2048K FFT, 9 cores, 5 workers.  Average times: 18.72,  9.46,  8.78,  8.19,  6.50 ms.  Total throughput: 549.02 iter/sec.
[Worker #1 Feb 4 13:02] Timing 2048K FFT, 9 cores, 6 workers.  Average times: 22.14, 10.49, 10.44, 15.89, 14.61,  8.79 ms.  Total throughput: 481.35 iter/sec.
[Worker #1 Feb 4 13:03] Timing 2048K FFT, 9 cores, 7 workers.  Average times: 26.26, 29.21, 15.84, 13.25, 17.22, 13.89, 10.65 ms.  Total throughput: 434.91 iter/sec.
[Worker #1 Feb 4 13:03] Timing 2048K FFT, 9 cores, 8 workers.  Average times: 16.61, 17.16, 16.19, 18.32, 22.35, 22.35, 22.68, 22.53 ms.  Total throughput: 412.82 iter/sec.
[Worker #1 Feb 4 13:04] Timing 2048K FFT, 9 cores hyperthreaded, 1 worker.  Average times:  1.50 ms.  Total throughput: 666.29 iter/sec.
[Worker #1 Feb 4 13:04] Timing 2048K FFT, 9 cores hyperthreaded, 2 workers.  Average times:  2.60,  3.23 ms.  Total throughput: 693.99 iter/sec.
[Worker #1 Feb 4 13:04] Timing 2048K FFT, 9 cores hyperthreaded, 3 workers.  Average times:  8.13,  4.42,  3.07 ms.  Total throughput: 674.96 iter/sec.
[Worker #1 Feb 4 13:05] Timing 2048K FFT, 9 cores hyperthreaded, 4 workers.  Average times:  7.02,  9.17,  7.01,  7.06 ms.  Total throughput: 535.62 iter/sec.
[Worker #1 Feb 4 13:05] Timing 2048K FFT, 9 cores hyperthreaded, 5 workers.  Average times: 23.54,  6.44, 11.64,  6.71,  6.68 ms.  Total throughput: 582.40 iter/sec.
[Worker #1 Feb 4 13:06] Timing 2048K FFT, 9 cores hyperthreaded, 6 workers.  Average times: 25.55, 17.47,  6.21, 15.38, 16.46,  7.36 ms.  Total throughput: 518.95 iter/sec.
[Worker #1 Feb 4 13:06] Timing 2048K FFT, 9 cores hyperthreaded, 7 workers.  Average times: 21.00, 22.40, 23.50, 10.23, 20.00, 15.49, 17.89 ms.  Total throughput: 402.98 iter/sec.
[Worker #1 Feb 4 13:06] Timing 2048K FFT, 9 cores hyperthreaded, 8 workers.  Average times: 36.42, 21.65, 38.30, 12.50, 19.35, 19.34, 15.21, 20.50 ms.  Total throughput: 397.67 iter/sec.
[Worker #1 Feb 4 13:07] Timing 2048K FFT, 10 cores, 1 worker.  Average times:  1.39 ms.  Total throughput: 719.43 iter/sec.
[Worker #1 Feb 4 13:07] Timing 2048K FFT, 10 cores, 2 workers.  Average times:  2.61,  2.68 ms.  Total throughput: 756.92 iter/sec.
[Worker #1 Feb 4 13:08] Timing 2048K FFT, 10 cores, 3 workers.  Average times:  6.42,  4.10,  2.41 ms.  Total throughput: 814.74 iter/sec.
[Worker #1 Feb 4 13:08] Timing 2048K FFT, 10 cores, 4 workers.  Average times:  7.17,  4.14,  7.81,  4.68 ms.  Total throughput: 722.55 iter/sec.
[Worker #1 Feb 4 13:08] Timing 2048K FFT, 10 cores, 5 workers.  Average times: 15.78,  9.49,  9.40,  8.75,  4.35 ms.  Total throughput: 619.07 iter/sec.
[Worker #1 Feb 4 13:09] Timing 2048K FFT, 10 cores, 6 workers.  Average times: 21.77, 10.35, 10.25, 15.07, 10.07, 16.60 ms.  Total throughput: 466.13 iter/sec.
[Worker #1 Feb 4 13:09] Timing 2048K FFT, 10 cores, 7 workers.  Average times: 18.94, 18.94, 21.66, 11.74, 14.64, 13.06, 21.28 ms.  Total throughput: 428.83 iter/sec.
[Worker #1 Feb 4 13:10] Timing 2048K FFT, 10 cores, 8 workers.  Average times: 21.07, 22.48, 22.28,  9.86, 24.01, 23.47, 22.41,  9.45 ms.  Total throughput: 472.99 iter/sec.
[Worker #1 Feb 4 13:10] Timing 2048K FFT, 10 cores, 10 workers.  Average times: 26.92, 20.61, 26.41, 20.44, 26.42, 20.78, 24.52, 26.77, 26.73, 24.85 ms.  Total throughput: 414.21 iter/sec.
[Worker #1 Feb 4 13:11] Timing 2048K FFT, 10 cores hyperthreaded, 1 worker.  Average times:  1.41 ms.  Total throughput: 708.45 iter/sec.
[Worker #1 Feb 4 13:11] Timing 2048K FFT, 10 cores hyperthreaded, 2 workers.  Average times:  2.49,  2.56 ms.  Total throughput: 792.57 iter/sec.
[Worker #1 Feb 4 13:11] Timing 2048K FFT, 10 cores hyperthreaded, 3 workers.  Average times:  6.70,  4.01,  2.50 ms.  Total throughput: 798.75 iter/sec.
[Worker #1 Feb 4 13:12] Timing 2048K FFT, 10 cores hyperthreaded, 4 workers.  Average times:  7.83,  4.05, 11.94,  4.37 ms.  Total throughput: 686.99 iter/sec.
[Worker #1 Feb 4 13:12] Timing 2048K FFT, 10 cores hyperthreaded, 5 workers.  Average times: 21.64, 10.14, 10.14,  6.59,  4.74 ms.  Total throughput: 606.13 iter/sec.
[Worker #1 Feb 4 13:12] Timing 2048K FFT, 10 cores hyperthreaded, 6 workers.  Average times: 32.80,  9.47,  7.35, 21.98, 10.10, 10.10 ms.  Total throughput: 515.86 iter/sec.
[Worker #1 Feb 4 13:13] Timing 2048K FFT, 10 cores hyperthreaded, 7 workers.  Average times: 27.97, 26.40, 28.00, 11.48, 14.50,  6.68, 21.71 ms.  Total throughput: 461.16 iter/sec.
[Worker #1 Feb 4 13:13] Timing 2048K FFT, 10 cores hyperthreaded, 8 workers.  Average times: 27.39, 27.45, 27.54, 10.67, 13.80, 13.78, 70.28, 29.88 ms.  Total throughput: 395.69 iter/sec.
[Worker #1 Feb 4 13:14] Timing 2048K FFT, 10 cores hyperthreaded, 10 workers.  Average times: 25.00, 22.08, 19.14, 31.37, 31.70, 24.97, 26.35, 24.97, 26.54, 24.97 ms.  Total throughput: 396.73 iter/sec.
[Worker #1 Feb 4 13:14] Timing 2048K FFT, 11 cores, 1 worker.  Average times:  1.34 ms.  Total throughput: 743.67 iter/sec.
[Worker #1 Feb 4 13:15] Timing 2048K FFT, 11 cores, 2 workers.  Average times:  2.23,  2.68 ms.  Total throughput: 821.39 iter/sec.
[Worker #1 Feb 4 13:15] Timing 2048K FFT, 11 cores, 3 workers.  Average times:  5.01,  4.25,  2.58 ms.  Total throughput: 822.58 iter/sec.
[Worker #1 Feb 4 13:15] Timing 2048K FFT, 11 cores, 4 workers.  Average times:  4.89,  4.88,  6.72,  4.23 ms.  Total throughput: 794.36 iter/sec.
[Worker #1 Feb 4 13:16] Timing 2048K FFT, 11 cores, 5 workers.  Average times: 12.62, 12.40,  9.71,  6.35, 10.86 ms.  Total throughput: 512.48 iter/sec.
[Worker #1 Feb 4 13:16] Timing 2048K FFT, 11 cores, 6 workers.  Average times: 14.55, 13.57, 13.49, 13.09, 39.50, 43.09 ms.  Total throughput: 341.47 iter/sec.
[Worker #1 Feb 4 13:17] Timing 2048K FFT, 11 cores, 7 workers.  Average times: 27.26, 27.37, 12.81, 13.09, 20.02,  8.88, 10.97 ms.  Total throughput: 481.31 iter/sec.
[Worker #1 Feb 4 13:17] Timing 2048K FFT, 11 cores, 8 workers.  Average times: 23.35, 22.31, 11.09, 23.15, 21.92, 23.44, 24.67, 10.15 ms.  Total throughput: 448.34 iter/sec.
[Worker #1 Feb 4 13:18] Timing 2048K FFT, 11 cores, 10 workers.  Average times: 22.37, 23.74, 26.99, 21.43, 17.21, 32.67, 24.39, 32.99, 32.98, 20.64 ms.  Total throughput: 409.35 iter/sec.
[Worker #1 Feb 4 13:18] Timing 2048K FFT, 11 cores hyperthreaded, 1 worker.  Average times:  1.37 ms.  Total throughput: 729.70 iter/sec.
[Worker #1 Feb 4 13:18] Timing 2048K FFT, 11 cores hyperthreaded, 2 workers.  Average times:  2.13,  2.49 ms.  Total throughput: 871.40 iter/sec.
[Worker #1 Feb 4 13:19] Timing 2048K FFT, 11 cores hyperthreaded, 3 workers.  Average times:  3.99,  5.11,  2.58 ms.  Total throughput: 833.95 iter/sec.
[Worker #1 Feb 4 13:19] Timing 2048K FFT, 11 cores hyperthreaded, 4 workers.  Average times:  4.66,  6.27,  7.35,  4.17 ms.  Total throughput: 749.99 iter/sec.
[Worker #1 Feb 4 13:19] Timing 2048K FFT, 11 cores hyperthreaded, 5 workers.  Average times: 12.53, 12.57,  7.69, 15.66,  4.81 ms.  Total throughput: 561.08 iter/sec.
[Worker #1 Feb 4 13:20] Timing 2048K FFT, 11 cores hyperthreaded, 6 workers.  Average times: 28.50, 12.23, 15.17, 20.14,  9.92,  9.91 ms.  Total throughput: 434.06 iter/sec.
[Worker #1 Feb 4 13:20] Timing 2048K FFT, 11 cores hyperthreaded, 7 workers.  Average times: 27.33, 27.41, 12.67, 12.65, 21.60, 10.45,  9.41 ms.  Total throughput: 479.35 iter/sec.
[Worker #1 Feb 4 13:21] Timing 2048K FFT, 11 cores hyperthreaded, 8 workers.  Average times: 32.82, 33.76, 15.29, 15.29, 27.32, 18.40, 18.40,  8.32 ms.  Total throughput: 456.36 iter/sec.
[Worker #1 Feb 4 13:21] Timing 2048K FFT, 11 cores hyperthreaded, 10 workers.  Average times: 30.28, 29.61, 28.82, 29.27, 13.11, 30.73, 23.01, 23.09, 23.24, 26.04 ms.  Total throughput: 412.64 iter/sec.
[Worker #1 Feb 4 13:22] Timing 2048K FFT, 12 cores, 1 worker.  Average times:  1.29 ms.  Total throughput: 773.03 iter/sec.
[Worker #1 Feb 4 13:22] Timing 2048K FFT, 12 cores, 2 workers.  Average times:  2.24,  2.23 ms.  Total throughput: 895.02 iter/sec.
[Worker #1 Feb 4 13:22] Timing 2048K FFT, 12 cores, 3 workers.  Average times:  4.45,  4.28,  2.21 ms.  Total throughput: 911.45 iter/sec.
[Worker #1 Feb 4 13:23] Timing 2048K FFT, 12 cores, 4 workers.  Average times:  4.71,  5.47,  5.05,  8.56 ms.  Total throughput: 709.64 iter/sec.
[Worker #1 Feb 4 13:23] Timing 2048K FFT, 12 cores, 5 workers.  Average times: 21.38, 11.64, 21.36,  4.64,  4.64 ms.  Total throughput: 610.24 iter/sec.
[Worker #1 Feb 4 13:24] Timing 2048K FFT, 12 cores, 6 workers.  Average times: 12.35, 12.35,  7.93, 35.23, 10.29, 33.97 ms.  Total throughput: 443.01 iter/sec.
[Worker #1 Feb 4 13:24] Timing 2048K FFT, 12 cores, 7 workers.  Average times: 27.89, 27.64, 13.35, 13.62, 17.37,  7.90, 17.58 ms.  Total throughput: 461.47 iter/sec.
[Worker #1 Feb 4 13:24] Timing 2048K FFT, 12 cores, 8 workers.  Average times: 16.50, 34.69, 19.38, 19.38, 16.33, 34.92, 13.16, 17.22 ms.  Total throughput: 416.56 iter/sec.
[Worker #1 Feb 4 13:25] Timing 2048K FFT, 12 cores, 10 workers.  Average times: 34.31, 22.58, 34.93, 32.01, 11.17, 31.78, 38.27, 31.75, 43.18,  8.55 ms.  Total throughput: 452.01 iter/sec.
[Worker #1 Feb 4 13:25] Timing 2048K FFT, 12 cores, 12 workers.  Average times: 29.77, 29.75, 29.87, 29.77, 29.41, 29.49, 29.69, 29.62, 29.64, 29.74, 29.60, 29.67 ms.  Total throughput: 404.47 iter/sec.
[Worker #1 Feb 4 13:26] Timing 2048K FFT, 12 cores hyperthreaded, 1 worker.  Average times:  1.33 ms.  Total throughput: 749.71 iter/sec.
[Worker #1 Feb 4 13:26] Timing 2048K FFT, 12 cores hyperthreaded, 2 workers.  Average times:  2.12,  2.11 ms.  Total throughput: 945.17 iter/sec.
[Worker #1 Feb 4 13:27] Timing 2048K FFT, 12 cores hyperthreaded, 3 workers.  Average times:  4.98,  4.98,  2.12 ms.  Total throughput: 873.92 iter/sec.
[Worker #1 Feb 4 13:27] Timing 2048K FFT, 12 cores hyperthreaded, 4 workers.  Average times:  5.89,  5.91,  6.58, 14.04 ms.  Total throughput: 562.05 iter/sec.
[Worker #1 Feb 4 13:27] Timing 2048K FFT, 12 cores hyperthreaded, 5 workers.  Average times: 12.76, 13.09,  7.58, 22.02, 14.32 ms.  Total throughput: 401.92 iter/sec.
[Worker #1 Feb 4 13:28] Timing 2048K FFT, 12 cores hyperthreaded, 6 workers.  Average times:  6.67, 44.47, 49.58, 15.54, 15.49, 16.22 ms.  Total throughput: 383.16 iter/sec.
[Worker #1 Feb 4 13:28] Timing 2048K FFT, 12 cores hyperthreaded, 7 workers.  Average times: 25.37, 19.35,  9.20, 11.63,  6.75, 24.51, 22.90 ms.  Total throughput: 518.37 iter/sec.
[Worker #1 Feb 4 13:29] Timing 2048K FFT, 12 cores hyperthreaded, 8 workers.  Average times: 14.03, 110.19, 42.50, 51.94, 25.48, 25.38, 12.43,  7.05 ms.  Total throughput: 424.11 iter/sec.
[Worker #1 Feb 4 13:29] Timing 2048K FFT, 12 cores hyperthreaded, 10 workers.  Average times: 47.34, 47.17, 20.42, 45.71,  9.46, 43.93, 22.20, 22.35, 27.81, 20.82 ms.  Total throughput: 415.42 iter/sec.
[Worker #1 Feb 4 13:30] Timing 2048K FFT, 12 cores hyperthreaded, 12 workers.  Average times: 29.07, 28.08, 29.00, 29.14, 28.69, 28.96, 31.88, 31.70, 31.89, 31.68, 29.82, 31.66 ms.  Total throughput: 399.16 iter/sec.
[Worker #1 Feb 4 13:30] Timing 2048K FFT, 13 cores, 1 worker.  Average times:  1.28 ms.  Total throughput: 782.43 iter/sec.
[Worker #1 Feb 4 13:30] Timing 2048K FFT, 13 cores, 2 workers.  Average times:  1.93,  2.19 ms.  Total throughput: 974.42 iter/sec.
[Worker #1 Feb 4 13:31] Timing 2048K FFT, 13 cores, 3 workers.  Average times:  4.52,  3.35,  2.01 ms.  Total throughput: 1016.15 iter/sec.
[Worker #1 Feb 4 13:31] Timing 2048K FFT, 13 cores, 4 workers.  Average times: 11.53,  5.06,  4.39,  5.06 ms.  Total throughput: 710.04 iter/sec.
[Worker #1 Feb 4 13:32] Timing 2048K FFT, 13 cores, 5 workers.  Average times: 13.63, 13.87,  4.48,  7.02,  4.68 ms.  Total throughput: 724.87 iter/sec.
[Worker #1 Feb 4 13:32] Timing 2048K FFT, 13 cores, 6 workers.  Average times: 13.51, 13.52,  8.36, 13.31, 13.31, 13.31 ms.  Total throughput: 492.97 iter/sec.
[Worker #1 Feb 4 13:32] Timing 2048K FFT, 13 cores, 7 workers.  Average times: 39.49, 19.23,  7.13, 19.23,  9.19, 17.87, 17.04 ms.  Total throughput: 493.04 iter/sec.
[Worker #1 Feb 4 13:33] Timing 2048K FFT, 13 cores, 8 workers.  Average times: 35.32, 20.56, 20.66, 12.22, 19.74, 34.54, 17.37, 16.80 ms.  Total throughput: 403.86 iter/sec.
[Worker #1 Feb 4 13:33] Timing 2048K FFT, 13 cores, 10 workers.  Average times: 32.95, 31.59, 31.67, 15.68, 15.57, 29.76, 29.39, 29.75, 29.78, 13.97 ms.  Total throughput: 428.00 iter/sec.
[Worker #1 Feb 4 13:34] Timing 2048K FFT, 13 cores, 12 workers.  Average times: 24.27, 37.65, 56.28, 22.57, 38.02, 28.08, 46.74, 23.85, 23.96, 46.72, 47.97, 23.84 ms.  Total throughput: 381.02 iter/sec.
[Worker #1 Feb 4 13:34] Timing 2048K FFT, 13 cores hyperthreaded, 1 worker.  Average times:  1.35 ms.  Total throughput: 740.24 iter/sec.
[Worker #1 Feb 4 13:35] Timing 2048K FFT, 13 cores hyperthreaded, 2 workers.  Average times:  1.87,  2.12 ms.  Total throughput: 1006.11 iter/sec.
[Worker #1 Feb 4 13:35] Timing 2048K FFT, 13 cores hyperthreaded, 3 workers.  Average times:  4.78,  3.15,  1.93 ms.  Total throughput: 1045.17 iter/sec.
[Worker #1 Feb 4 13:35] Timing 2048K FFT, 13 cores hyperthreaded, 4 workers.  Average times:  6.48,  3.81, 15.09, 12.02 ms.  Total throughput: 566.28 iter/sec.
[Worker #1 Feb 4 13:36] Timing 2048K FFT, 13 cores hyperthreaded, 5 workers.  Average times: 12.72, 31.70,  4.49,  5.92,  5.81 ms.  Total throughput: 674.26 iter/sec.
[Worker #1 Feb 4 13:36] Timing 2048K FFT, 13 cores hyperthreaded, 6 workers.  Average times: 14.75, 14.07,  4.74, 16.45, 16.17, 14.43 ms.  Total throughput: 541.69 iter/sec.
[Worker #1 Feb 4 13:37] Timing 2048K FFT, 13 cores hyperthreaded, 7 workers.  Average times: 31.43, 15.95, 13.42, 11.30, 20.76, 15.58, 21.24 ms.  Total throughput: 416.87 iter/sec.
[Worker #1 Feb 4 13:37] Timing 2048K FFT, 13 cores hyperthreaded, 8 workers.  Average times: 18.70, 27.09, 17.98, 33.55, 29.61, 28.99, 13.75, 12.22 ms.  Total throughput: 398.60 iter/sec.
[Worker #1 Feb 4 13:38] Timing 2048K FFT, 13 cores hyperthreaded, 10 workers.  Average times: 31.19, 37.01, 37.33, 17.48, 17.26, 59.44, 62.52, 31.94, 17.88, 10.71 ms.  Total throughput: 414.47 iter/sec.
[Worker #1 Feb 4 13:38] Timing 2048K FFT, 13 cores hyperthreaded, 12 workers.  Average times: 37.88, 35.73, 41.32, 32.33, 37.53, 15.84, 34.63, 38.04, 34.46, 33.71, 23.93, 30.27 ms.  Total throughput: 387.96 iter/sec.
[Worker #1 Feb 4 13:39] Timing 2048K FFT, 14 cores, 1 worker.  Average times:  1.24 ms.  Total throughput: 807.40 iter/sec.
[Worker #1 Feb 4 13:39] Timing 2048K FFT, 14 cores, 2 workers.  Average times:  1.95,  1.94 ms.  Total throughput: 1026.51 iter/sec.
[Worker #1 Feb 4 13:39] Timing 2048K FFT, 14 cores, 3 workers.  Average times:  4.59,  3.53,  1.92 ms.  Total throughput: 1020.95 iter/sec.
[Worker #1 Feb 4 13:40] Timing 2048K FFT, 14 cores, 4 workers.  Average times:  7.26,  3.70,  5.36,  3.68 ms.  Total throughput: 866.29 iter/sec.
[Worker #1 Feb 4 13:40] Timing 2048K FFT, 14 cores, 5 workers.  Average times:  8.09, 14.00, 10.51,  7.65,  3.84 ms.  Total throughput: 681.64 iter/sec.
[Worker #1 Feb 4 13:40] Timing 2048K FFT, 14 cores, 6 workers.  Average times:  8.26, 14.14,  8.74, 16.56, 47.72, 24.57 ms.  Total throughput: 428.16 iter/sec.
[Worker #1 Feb 4 13:41] Timing 2048K FFT, 14 cores, 7 workers.  Average times: 17.67, 43.39, 29.37, 39.54, 13.75, 14.24,  7.44 ms.  Total throughput: 416.40 iter/sec.
[Worker #1 Feb 4 13:41] Timing 2048K FFT, 14 cores, 8 workers.  Average times: 30.68, 22.08, 20.39, 10.52, 33.41, 16.80, 20.26, 14.51 ms.  Total throughput: 429.71 iter/sec.
[Worker #1 Feb 4 13:42] Timing 2048K FFT, 14 cores, 10 workers.  Average times: 34.71, 31.91, 33.11, 20.72, 15.30, 27.80, 28.61, 28.11, 19.69, 22.41 ms.  Total throughput: 405.86 iter/sec.
[Worker #1 Feb 4 13:42] Timing 2048K FFT, 14 cores, 12 workers.  Average times: 40.77, 40.55, 40.41, 40.19, 37.84, 18.51, 27.41, 26.29, 26.10, 27.97, 27.97, 28.34 ms.  Total throughput: 398.89 iter/sec.
[Worker #1 Feb 4 13:43] Timing 2048K FFT, 14 cores hyperthreaded, 1 worker.  Average times:  1.37 ms.  Total throughput: 731.72 iter/sec.
[Worker #1 Feb 4 13:43] Timing 2048K FFT, 14 cores hyperthreaded, 2 workers.  Average times:  1.87,  1.84 ms.  Total throughput: 1080.55 iter/sec.
[Worker #1 Feb 4 13:43] Timing 2048K FFT, 14 cores hyperthreaded, 3 workers.  Average times:  4.94,  3.32,  1.87 ms.  Total throughput: 1037.69 iter/sec.
[Worker #1 Feb 4 13:44] Timing 2048K FFT, 14 cores hyperthreaded, 4 workers.  Average times:  5.02,  5.00,  8.68,  4.45 ms.  Total throughput: 738.71 iter/sec.
[Worker #1 Feb 4 13:44] Timing 2048K FFT, 14 cores hyperthreaded, 5 workers.  Average times: 15.11, 15.16,  8.56,  5.62,  3.86 ms.  Total throughput: 686.09 iter/sec.
[Worker #1 Feb 4 13:45] Timing 2048K FFT, 14 cores hyperthreaded, 6 workers.  Average times:  8.96,  9.77, 21.61, 17.18, 18.23, 10.36 ms.  Total throughput: 469.76 iter/sec.
[Worker #1 Feb 4 13:45] Timing 2048K FFT, 14 cores hyperthreaded, 7 workers.  Average times: 20.09, 17.97, 17.94, 17.94, 43.13, 20.73, 10.31 ms.  Total throughput: 385.36 iter/sec.
[Worker #1 Feb 4 13:45] Timing 2048K FFT, 14 cores hyperthreaded, 8 workers.  Average times: 29.13, 13.89, 24.68, 21.23, 32.17,  7.84, 20.14, 20.08 ms.  Total throughput: 452.10 iter/sec.
[Worker #1 Feb 4 13:46] Timing 2048K FFT, 14 cores hyperthreaded, 10 workers.  Average times: 29.15, 30.15, 38.95, 17.23, 20.73, 30.24, 53.03, 52.36, 15.35, 20.38 ms.  Total throughput: 384.69 iter/sec.
[Worker #1 Feb 4 13:46] Timing 2048K FFT, 14 cores hyperthreaded, 12 workers.  Average times: 30.96, 32.16, 32.19, 32.17, 30.96, 15.47, 38.85, 38.72, 38.89, 38.84, 38.67, 18.41 ms.  Total throughput: 405.69 iter/sec.
[Worker #1 Feb 4 13:47] Timing 2048K FFT, 15 cores, 1 worker.  Average times:  1.24 ms.  Total throughput: 803.68 iter/sec.
[Worker #1 Feb 4 13:47] Timing 2048K FFT, 15 cores, 2 workers.  Average times:  1.69,  1.91 ms.  Total throughput: 1112.43 iter/sec.
[Worker #1 Feb 4 13:48] Timing 2048K FFT, 15 cores, 3 workers.  Average times:  3.98,  3.48,  1.95 ms.  Total throughput: 1052.96 iter/sec.
[Worker #1 Feb 4 13:48] Timing 2048K FFT, 15 cores, 4 workers.  Average times: 14.76,  9.23,  5.99,  3.54 ms.  Total throughput: 625.97 iter/sec.
[Worker #1 Feb 4 13:48] Timing 2048K FFT, 15 cores, 5 workers.  Average times: 31.33,  4.73, 23.53,  5.68,  3.49 ms.  Total throughput: 747.89 iter/sec.
[Worker #1 Feb 4 13:49] Timing 2048K FFT, 15 cores, 6 workers.  Average times:  9.01, 29.74, 24.16, 12.34, 12.34,  5.34 ms.  Total throughput: 535.40 iter/sec.
[Worker #1 Feb 4 13:49] Timing 2048K FFT, 15 cores, 7 workers.  Average times: 18.63, 18.62, 18.53, 18.54, 14.31, 14.28,  8.45 ms.  Total throughput: 473.55 iter/sec.
[Worker #1 Feb 4 13:50] Timing 2048K FFT, 15 cores, 8 workers.  Average times: 23.10, 22.67, 20.14, 22.66, 28.29, 28.28, 10.82, 10.81 ms.  Total throughput: 436.86 iter/sec.
[Worker #1 Feb 4 13:50] Timing 2048K FFT, 15 cores, 10 workers.  Average times: 38.21, 37.73, 19.11, 19.07, 19.09, 33.58, 33.58, 33.47, 16.32, 15.62 ms.  Total throughput: 424.54 iter/sec.
[Worker #1 Feb 4 13:51] Timing 2048K FFT, 15 cores, 12 workers.  Average times: 35.40, 33.78, 33.81, 44.05, 17.13, 26.88, 31.70, 35.94, 37.07, 35.77, 35.78, 17.72 ms.  Total throughput: 404.38 iter/sec.
[Worker #1 Feb 4 13:51] Timing 2048K FFT, 15 cores hyperthreaded, 1 worker.  Average times:  1.43 ms.  Total throughput: 700.01 iter/sec.
[Worker #1 Feb 4 13:51] Timing 2048K FFT, 15 cores hyperthreaded, 2 workers.  Average times:  1.64,  1.84 ms.  Total throughput: 1154.71 iter/sec.
[Worker #1 Feb 4 13:52] Timing 2048K FFT, 15 cores hyperthreaded, 3 workers.  Average times:  4.01,  3.58,  1.89 ms.  Total throughput: 1058.36 iter/sec.
[Worker #1 Feb 4 13:52] Timing 2048K FFT, 15 cores hyperthreaded, 4 workers.  Average times:  5.20,  5.14,  6.03,  3.87 ms.  Total throughput: 811.21 iter/sec.
[Worker #1 Feb 4 13:53] Timing 2048K FFT, 15 cores hyperthreaded, 5 workers.  Average times: 16.92,  5.58, 10.29, 19.56,  3.21 ms.  Total throughput: 698.12 iter/sec.
[Worker #1 Feb 4 13:53] Timing 2048K FFT, 15 cores hyperthreaded, 6 workers.  Average times: 21.66, 13.01, 13.01, 11.00,  8.17, 21.10 ms.  Total throughput: 460.52 iter/sec.
[Worker #1 Feb 4 13:53] Timing 2048K FFT, 15 cores hyperthreaded, 7 workers.  Average times: 15.15, 12.90, 12.88, 15.15, 11.52, 39.68, 24.29 ms.  Total throughput: 440.38 iter/sec.
[Worker #1 Feb 4 13:54] Timing 2048K FFT, 15 cores hyperthreaded, 8 workers.  Average times: 18.78, 18.78, 18.78, 18.78, 34.35, 16.04, 15.99, 15.96 ms.  Total throughput: 429.64 iter/sec.
[Worker #1 Feb 4 13:54] Timing 2048K FFT, 15 cores hyperthreaded, 10 workers.  Average times: 23.33, 23.28, 46.47, 15.51, 48.92, 37.00, 34.91, 37.94, 17.97, 17.62 ms.  Total throughput: 386.66 iter/sec.
[Worker #1 Feb 4 13:55] Timing 2048K FFT, 15 cores hyperthreaded, 12 workers.  Average times: 36.93, 34.34, 32.89, 36.67, 18.02, 17.76, 21.32, 51.69, 72.20, 51.94, 70.87, 28.85 ms.  Total throughput: 373.77 iter/sec.
[Worker #1 Feb 4 13:55] Timing 2048K FFT, 16 cores, 1 worker.  Average times:  1.16 ms.  Total throughput: 859.31 iter/sec.
[Worker #1 Feb 4 13:56] Timing 2048K FFT, 16 cores, 2 workers.  Average times:  1.71,  1.71 ms.  Total throughput: 1171.28 iter/sec.
[Worker #1 Feb 4 13:56] Timing 2048K FFT, 16 cores, 3 workers.  Average times:  4.98,  3.79,  1.72 ms.  Total throughput: 1045.37 iter/sec.
[Worker #1 Feb 4 13:56] Timing 2048K FFT, 16 cores, 4 workers.  Average times:  4.66,  4.65,  4.34,  4.28 ms.  Total throughput: 893.52 iter/sec.
[Worker #1 Feb 4 13:57] Timing 2048K FFT, 16 cores, 5 workers.  Average times: 11.85,  8.37,  8.77, 13.18,  3.66 ms.  Total throughput: 666.95 iter/sec.
[Worker #1 Feb 4 13:57] Timing 2048K FFT, 16 cores, 6 workers.  Average times: 20.00, 12.42, 12.41, 12.03, 14.28,  7.40 ms.  Total throughput: 499.31 iter/sec.
[Worker #1 Feb 4 13:58] Timing 2048K FFT, 16 cores, 7 workers.  Average times: 25.57, 14.70, 15.60, 27.20, 13.98, 15.68, 10.51 ms.  Total throughput: 438.47 iter/sec.
[Worker #1 Feb 4 13:58] Timing 2048K FFT, 16 cores, 8 workers.  Average times: 15.81, 17.29, 16.76, 17.28, 34.15, 43.99, 31.93, 27.06 ms.  Total throughput: 358.91 iter/sec.
[Worker #1 Feb 4 13:59] Timing 2048K FFT, 16 cores, 10 workers.  Average times: 33.92, 35.88, 19.88, 22.86, 18.16, 32.76, 39.80, 24.48, 26.53, 20.16 ms.  Total throughput: 390.26 iter/sec.
[Worker #1 Feb 4 13:59] Timing 2048K FFT, 16 cores, 12 workers.  Average times: 39.65, 39.41, 39.44, 39.16, 18.75, 19.71, 39.40, 39.27, 39.24, 39.06, 19.72, 18.74 ms.  Total throughput: 411.56 iter/sec.
[Worker #1 Feb 4 14:00] Timing 2048K FFT, 16 cores hyperthreaded, 1 worker.  Average times:  1.41 ms.  Total throughput: 707.96 iter/sec.
[Worker #1 Feb 4 14:00] Timing 2048K FFT, 16 cores hyperthreaded, 2 workers.  Average times:  1.63,  1.63 ms.  Total throughput: 1224.61 iter/sec.
[Worker #1 Feb 4 14:00] Timing 2048K FFT, 16 cores hyperthreaded, 3 workers.  Average times:  3.97,  4.50,  1.64 ms.  Total throughput: 1083.11 iter/sec.
[Worker #1 Feb 4 14:01] Timing 2048K FFT, 16 cores hyperthreaded, 4 workers.  Average times:  5.02,  4.89,  6.95,  4.14 ms.  Total throughput: 789.23 iter/sec.
[Worker #1 Feb 4 14:01] Timing 2048K FFT, 16 cores hyperthreaded, 5 workers.  Average times: 64.19, 37.43, 16.72,  6.61,  5.54 ms.  Total throughput: 433.85 iter/sec.
[Worker #1 Feb 4 14:01] Timing 2048K FFT, 16 cores hyperthreaded, 6 workers.  Average times: 18.01, 10.67, 10.67, 17.78, 10.68, 10.69 ms.  Total throughput: 486.45 iter/sec.
[Worker #1 Feb 4 14:02] Timing 2048K FFT, 16 cores hyperthreaded, 7 workers.  Average times: 21.60, 21.56, 20.46, 19.24, 13.20, 18.38, 18.38 ms.  Total throughput: 378.12 iter/sec.
[Worker #1 Feb 4 14:02] Timing 2048K FFT, 16 cores hyperthreaded, 8 workers.  Average times: 20.59, 15.78, 15.79, 15.77, 22.04, 22.11, 22.07, 22.15 ms.  Total throughput: 419.73 iter/sec.
[Worker #1 Feb 4 14:03] Timing 2048K FFT, 16 cores hyperthreaded, 10 workers.  Average times: 89.45, 96.29, 49.64, 11.53, 11.49, 41.97, 36.07, 17.72, 19.98, 20.00 ms.  Total throughput: 423.55 iter/sec.
[Worker #1 Feb 4 14:03] Timing 2048K FFT, 16 cores hyperthreaded, 12 workers.  Average times: 46.82, 46.99, 46.87, 46.82, 23.29, 23.29, 32.99, 33.05, 33.11, 33.05, 16.21, 22.27 ms.  Total throughput: 398.83 iter/sec.
[Worker #1 Feb 4 14:04] Timing 2048K FFT, 17 cores, 1 worker.  Average times:  1.27 ms.  Total throughput: 790.15 iter/sec.
[Worker #1 Feb 4 14:04] Timing 2048K FFT, 17 cores, 2 workers.  Average times:  1.55,  1.69 ms.  Total throughput: 1238.53 iter/sec.
[Worker #1 Feb 4 14:04] Timing 2048K FFT, 17 cores, 3 workers.  Average times:  4.54,  3.49,  1.55 ms.  Total throughput: 1150.98 iter/sec.
[Worker #1 Feb 4 14:05] Timing 2048K FFT, 17 cores, 4 workers.  Average times:  6.79,  4.25,  3.56, 14.24 ms.  Total throughput: 733.60 iter/sec.
[Worker #1 Feb 4 14:05] Timing 2048K FFT, 17 cores, 5 workers.  Average times:  8.21,  9.93, 10.95,  3.96,  8.64 ms.  Total throughput: 682.10 iter/sec.
[Worker #1 Feb 4 14:06] Timing 2048K FFT, 17 cores, 6 workers.  Average times: 11.90, 14.28, 12.62, 16.77,  9.94, 12.83 ms.  Total throughput: 471.44 iter/sec.
[Worker #1 Feb 4 14:06] Timing 2048K FFT, 17 cores, 7 workers.  Average times: 24.55, 19.74, 24.53, 14.47, 58.99,  8.22, 36.31 ms.  Total throughput: 367.38 iter/sec.
[Worker #1 Feb 4 14:07] Timing 2048K FFT, 17 cores, 8 workers.  Average times: 21.63, 21.60, 21.22, 13.76, 18.09, 17.97, 17.97, 17.97 ms.  Total throughput: 434.56 iter/sec.
[Worker #1 Feb 4 14:07] Timing 2048K FFT, 17 cores, 10 workers.  Average times: 42.52, 20.74, 20.97, 16.05, 16.64, 45.70, 33.14, 40.74, 61.94, 60.80 ms.  Total throughput: 351.05 iter/sec.
[Worker #1 Feb 4 14:08] Timing 2048K FFT, 17 cores, 12 workers.  Average times: 52.53, 46.39, 51.86, 22.49, 20.51, 18.48, 39.54, 42.33, 42.45, 40.19, 19.79, 19.79 ms.  Total throughput: 405.61 iter/sec.
[Worker #1 Feb 4 14:08] Timing 2048K FFT, 17 cores hyperthreaded, 1 worker.  Average times:  1.45 ms.  Total throughput: 687.84 iter/sec.
[Worker #1 Feb 4 14:08] Timing 2048K FFT, 17 cores hyperthreaded, 2 workers.  Average times:  1.52,  1.64 ms.  Total throughput: 1266.36 iter/sec.
[Worker #1 Feb 4 14:09] Timing 2048K FFT, 17 cores hyperthreaded, 3 workers.  Average times:  4.88,  3.13,  1.67 ms.  Total throughput: 1122.07 iter/sec.
[Worker #1 Feb 4 14:09] Timing 2048K FFT, 17 cores hyperthreaded, 4 workers.  Average times:  9.65,  2.99,  4.68,  4.54 ms.  Total throughput: 872.73 iter/sec.
[Worker #1 Feb 4 14:09] Timing 2048K FFT, 17 cores hyperthreaded, 5 workers.  Average times: 10.93, 13.05, 10.89,  5.48,  5.39 ms.  Total throughput: 627.90 iter/sec.
[Worker #1 Feb 4 14:10] Timing 2048K FFT, 17 cores hyperthreaded, 6 workers.  Average times: 14.90, 14.91, 14.90, 24.67,  8.13,  9.98 ms.  Total throughput: 465.01 iter/sec.
[Worker #1 Feb 4 14:10] Timing 2048K FFT, 17 cores hyperthreaded, 7 workers.  Average times: 21.39, 21.38, 21.39, 13.83, 17.80, 10.63, 10.62 ms.  Total throughput: 456.91 iter/sec.
[Worker #1 Feb 4 14:11] Timing 2048K FFT, 17 cores hyperthreaded, 8 workers.  Average times: 21.51, 21.51, 21.52, 13.68, 19.13, 18.64, 19.13, 18.66 ms.  Total throughput: 424.33 iter/sec.
[Worker #1 Feb 4 14:11] Timing 2048K FFT, 17 cores hyperthreaded, 10 workers.  Average times: 103.67, 86.11, 36.99, 77.55, 72.55, 38.87, 22.84, 15.25, 18.87, 19.29 ms.  Total throughput: 314.90 iter/sec.
[Worker #1 Feb 4 14:12] Timing 2048K FFT, 17 cores hyperthreaded, 12 workers.  Average times: 79.03, 31.56, 44.10, 31.82, 31.40, 28.93, 39.84, 39.72, 36.06, 42.91, 21.00, 20.82 ms.  Total throughput: 361.81 iter/sec.
[Worker #1 Feb 4 14:12] Timing 2048K FFT, 18 cores, 1 worker.  Average times:  1.22 ms.  Total throughput: 817.77 iter/sec.
[Worker #1 Feb 4 14:13] Timing 2048K FFT, 18 cores, 2 workers.  Average times:  1.54,  1.53 ms.  Total throughput: 1302.02 iter/sec.
[Worker #1 Feb 4 14:13] Timing 2048K FFT, 18 cores, 3 workers.  Average times:  4.35,  3.09,  1.54 ms.  Total throughput: 1203.83 iter/sec.
[Worker #1 Feb 4 14:13] Timing 2048K FFT, 18 cores, 4 workers.  Average times: 10.47,  5.78,  4.93,  4.46 ms.  Total throughput: 695.81 iter/sec.
[Worker #1 Feb 4 14:14] Timing 2048K FFT, 18 cores, 5 workers.  Average times: 13.54, 14.49, 12.42, 11.34,  5.57 ms.  Total throughput: 491.24 iter/sec.
[Worker #1 Feb 4 14:14] Timing 2048K FFT, 18 cores, 6 workers.  Average times: 15.04, 14.18, 14.08, 44.57, 19.67, 25.38 ms.  Total throughput: 320.69 iter/sec.
[Worker #1 Feb 4 14:15] Timing 2048K FFT, 18 cores, 7 workers.  Average times: 52.23, 37.59, 51.17,  7.67,  7.70, 10.44,  9.96 ms.  Total throughput: 521.72 iter/sec.
[Worker #1 Feb 4 14:15] Timing 2048K FFT, 18 cores, 8 workers.  Average times: 15.52, 14.20, 20.35, 13.63, 80.38, 44.27, 67.84, 25.22 ms.  Total throughput: 346.78 iter/sec.
[Worker #1 Feb 4 14:15] Timing 2048K FFT, 18 cores, 10 workers.  Average times: 45.46, 21.28, 22.54, 22.56, 22.53, 42.51, 21.29, 21.32, 21.30, 20.49 ms.  Total throughput: 415.21 iter/sec.
[Worker #1 Feb 4 14:16] Timing 2048K FFT, 18 cores, 12 workers.  Average times: 44.24, 42.67, 42.83, 21.80, 24.51, 22.97, 51.44, 34.99, 40.87, 17.57, 30.61, 28.23 ms.  Total throughput: 397.12 iter/sec.
[Worker #1 Feb 4 14:16] Timing 2048K FFT, 18 cores hyperthreaded, 1 worker.  Average times:  1.56 ms.  Total throughput: 640.83 iter/sec.
[Worker #1 Feb 4 14:17] Timing 2048K FFT, 18 cores hyperthreaded, 2 workers.  Average times:  1.51,  1.48 ms.  Total throughput: 1338.12 iter/sec.
[Worker #1 Feb 4 14:17] Timing 2048K FFT, 18 cores hyperthreaded, 3 workers.  Average times:  4.52,  4.23,  1.50 ms.  Total throughput: 1125.08 iter/sec.
[Worker #1 Feb 4 14:18] Timing 2048K FFT, 18 cores hyperthreaded, 4 workers.  Average times: 11.50,  2.83,  4.93,  3.48 ms.  Total throughput: 930.42 iter/sec.
[Worker #1 Feb 4 14:18] Timing 2048K FFT, 18 cores hyperthreaded, 5 workers.  Average times:  6.26, 21.62, 12.47,  6.63,  3.65 ms.  Total throughput: 711.09 iter/sec.
[Worker #1 Feb 4 14:18] Timing 2048K FFT, 18 cores hyperthreaded, 6 workers.  Average times:  6.91, 22.17, 22.19, 12.27, 12.30,  8.46 ms.  Total throughput: 515.90 iter/sec.
[Worker #1 Feb 4 14:19] Timing 2048K FFT, 18 cores hyperthreaded, 7 workers.  Average times: 15.88, 17.11, 13.13, 11.52, 29.06, 28.69,  6.10 ms.  Total throughput: 517.47 iter/sec.
[Worker #1 Feb 4 14:19] Timing 2048K FFT, 18 cores hyperthreaded, 8 workers.  Average times: 20.35, 21.00, 20.35, 13.22, 22.19, 22.18, 22.19, 14.33 ms.  Total throughput: 426.54 iter/sec.
[Worker #1 Feb 4 14:20] Timing 2048K FFT, 18 cores hyperthreaded, 10 workers.  Average times: 109.33, 77.61, 13.71, 51.96, 60.64, 30.02, 18.37, 18.04, 18.27, 13.41 ms.  Total throughput: 403.23 iter/sec.
[Worker #1 Feb 4 14:20] Timing 2048K FFT, 18 cores hyperthreaded, 12 workers.  Average times: 44.48, 45.19, 45.24, 22.26, 22.52, 22.54, 44.82, 44.08, 43.79, 22.08, 22.48, 22.23 ms.  Total throughput: 403.02 iter/sec.
[Worker #1 Feb 4 14:21] Timing 2048K FFT, 19 cores, 1 worker.  Average times:  1.20 ms.  Total throughput: 832.66 iter/sec.
[Worker #1 Feb 4 14:21] Timing 2048K FFT, 19 cores, 2 workers.  Average times:  1.43,  1.53 ms.  Total throughput: 1352.77 iter/sec.
[Worker #1 Feb 4 14:21] Timing 2048K FFT, 19 cores, 3 workers.  Average times:  5.03,  5.61,  1.53 ms.  Total throughput: 1029.58 iter/sec.
[Worker #1 Feb 4 14:22] Timing 2048K FFT, 19 cores, 4 workers.  Average times:  9.80,  8.85,  4.81,  3.34 ms.  Total throughput: 722.63 iter/sec.
[Worker #1 Feb 4 14:22] Timing 2048K FFT, 19 cores, 5 workers.  Average times: 13.10, 12.02,  4.66,  7.62,  3.68 ms.  Total throughput: 777.25 iter/sec.
[Worker #1 Feb 4 14:23] Timing 2048K FFT, 19 cores, 6 workers.  Average times:  7.15, 35.26, 27.51, 12.94, 12.94,  8.03 ms.  Total throughput: 483.52 iter/sec.
[Worker #1 Feb 4 14:23] Timing 2048K FFT, 19 cores, 7 workers.  Average times: 23.01, 17.38, 11.48, 15.75, 41.20, 37.25, 19.97 ms.  Total throughput: 352.81 iter/sec.
[Worker #1 Feb 4 14:23] Timing 2048K FFT, 19 cores, 8 workers.  Average times: 26.38, 25.83, 17.86, 17.76, 27.09, 27.09, 24.25, 37.07 ms.  Total throughput: 330.97 iter/sec.
[Worker #1 Feb 4 14:24] Timing 2048K FFT, 19 cores, 10 workers.  Average times: 29.40, 22.92, 22.92, 28.60, 24.80, 40.98, 20.71, 20.70, 24.38, 25.42 ms.  Total throughput: 397.93 iter/sec.
[Worker #1 Feb 4 14:24] Timing 2048K FFT, 19 cores, 12 workers.  Average times: 47.45, 39.30, 29.12, 28.86, 21.46, 23.76, 38.75, 43.43, 34.09, 29.85, 25.75, 32.60 ms.  Total throughput: 385.37 iter/sec.
[Worker #1 Feb 4 14:25] Timing 2048K FFT, 19 cores hyperthreaded, 1 worker.  Average times:  1.48 ms.  Total throughput: 676.17 iter/sec.
[Worker #1 Feb 4 14:25] Timing 2048K FFT, 19 cores hyperthreaded, 2 workers.  Average times:  1.45,  1.48 ms.  Total throughput: 1365.74 iter/sec.
[Worker #1 Feb 4 14:26] Timing 2048K FFT, 19 cores hyperthreaded, 3 workers.  Average times:  2.97,  4.92,  1.52 ms.  Total throughput: 1199.53 iter/sec.
[Worker #1 Feb 4 14:26] Timing 2048K FFT, 19 cores hyperthreaded, 4 workers.  Average times:  4.46,  6.27,  4.25, 18.91 ms.  Total throughput: 671.89 iter/sec.
[Worker #1 Feb 4 14:26] Timing 2048K FFT, 19 cores hyperthreaded, 5 workers.  Average times: 14.57,  8.45, 10.39, 14.85,  3.13 ms.  Total throughput: 670.16 iter/sec.
[Worker #1 Feb 4 14:27] Timing 2048K FFT, 19 cores hyperthreaded, 6 workers.  Average times: 15.67, 15.05,  9.34, 14.12, 11.17, 14.28 ms.  Total throughput: 467.67 iter/sec.
[Worker #1 Feb 4 14:27] Timing 2048K FFT, 19 cores hyperthreaded, 7 workers.  Average times: 44.03, 14.40, 31.16, 23.71, 13.58, 12.65,  6.74 ms.  Total throughput: 467.36 iter/sec.
[Worker #1 Feb 4 14:28] Timing 2048K FFT, 19 cores hyperthreaded, 8 workers.  Average times: 20.40, 25.69, 11.50, 13.96, 25.39, 25.32, 25.35, 16.29 ms.  Total throughput: 426.24 iter/sec.
[Worker #1 Feb 4 14:28] Timing 2048K FFT, 19 cores hyperthreaded, 10 workers.  Average times: 106.57, 70.28, 106.65, 98.85, 103.79, 52.11, 15.84, 25.48, 25.56, 25.48 ms.  Total throughput: 252.67 iter/sec.
[Worker #1 Feb 4 14:29] Timing 2048K FFT, 19 cores hyperthreaded, 12 workers.  Average times: 49.70, 49.95, 25.23, 25.26, 25.24, 24.84, 44.80, 44.74, 44.76, 22.26, 22.78, 22.27 ms.  Total throughput: 399.98 iter/sec.
[Worker #1 Feb 4 14:29] Timing 2048K FFT, 20 cores, 1 worker.  Average times:  1.32 ms.  Total throughput: 759.51 iter/sec.
[Worker #1 Feb 4 14:29] Timing 2048K FFT, 20 cores, 2 workers.  Average times:  1.45,  1.44 ms.  Total throughput: 1386.05 iter/sec.
[Worker #1 Feb 4 14:30] Timing 2048K FFT, 20 cores, 3 workers.  Average times:  4.85,  4.15,  1.42 ms.  Total throughput: 1152.07 iter/sec.
[Worker #1 Feb 4 14:30] Timing 2048K FFT, 20 cores, 4 workers.  Average times:  5.88,  5.40,  4.93,  5.15 ms.  Total throughput: 752.25 iter/sec.
[Worker #1 Feb 4 14:31] Timing 2048K FFT, 20 cores, 5 workers.  Average times: 12.93, 12.93,  9.49,  4.42,  4.42 ms.  Total throughput: 712.87 iter/sec.
[Worker #1 Feb 4 14:31] Timing 2048K FFT, 20 cores, 6 workers.  Average times:  7.63, 12.64,  8.91, 28.78, 42.96, 39.25 ms.  Total throughput: 405.88 iter/sec.
[Worker #1 Feb 4 14:31] Timing 2048K FFT, 20 cores, 7 workers.  Average times: 33.78, 33.15, 29.16, 11.60, 11.30, 13.43,  9.35 ms.  Total throughput: 450.13 iter/sec.
[Worker #1 Feb 4 14:32] Timing 2048K FFT, 20 cores, 8 workers.  Average times: 26.76, 26.48, 17.16, 17.11, 20.45, 20.44, 13.46, 14.91 ms.  Total throughput: 431.06 iter/sec.
[Worker #1 Feb 4 14:32] Timing 2048K FFT, 20 cores, 10 workers.  Average times: 25.81, 22.01, 21.92, 25.44, 26.08, 45.65, 35.56, 17.28, 29.00, 35.10 ms.  Total throughput: 378.33 iter/sec.
[Worker #1 Feb 4 14:33] Timing 2048K FFT, 20 cores, 12 workers.  Average times: 85.85, 62.42, 62.51, 78.16, 66.46, 66.04, 41.90, 51.75, 18.32, 19.49, 25.39, 25.93 ms.  Total throughput: 313.69 iter/sec.
[Worker #1 Feb 4 14:33] Timing 2048K FFT, 20 cores hyperthreaded, 1 worker.  Average times:  1.52 ms.  Total throughput: 656.24 iter/sec.
[Worker #1 Feb 4 14:34] Timing 2048K FFT, 20 cores hyperthreaded, 2 workers.  Average times:  1.45,  1.36 ms.  Total throughput: 1425.63 iter/sec.
[Worker #1 Feb 4 14:34] Timing 2048K FFT, 20 cores hyperthreaded, 3 workers.  Average times:  4.41,  3.22,  1.41 ms.  Total throughput: 1248.01 iter/sec.
[Worker #1 Feb 4 14:34] Timing 2048K FFT, 20 cores hyperthreaded, 4 workers.  Average times:  4.54,  5.13,  5.47,  5.00 ms.  Total throughput: 798.07 iter/sec.
[Worker #1 Feb 4 14:35] Timing 2048K FFT, 20 cores hyperthreaded, 5 workers.  Average times:  7.89, 12.13,  8.45, 13.25,  5.71 ms.  Total throughput: 578.00 iter/sec.
[Worker #1 Feb 4 14:35] Timing 2048K FFT, 20 cores hyperthreaded, 6 workers.  Average times: 10.25, 16.08, 11.84, 21.30, 15.93,  9.42 ms.  Total throughput: 460.08 iter/sec.
[Worker #1 Feb 4 14:36] Timing 2048K FFT, 20 cores hyperthreaded, 7 workers.  Average times: 24.20, 24.21, 15.40, 15.39, 14.34, 14.35,  9.74 ms.  Total throughput: 454.64 iter/sec.
[Worker #1 Feb 4 14:36] Timing 2048K FFT, 20 cores hyperthreaded, 8 workers.  Average times: 24.73, 24.73, 19.45, 16.82, 18.19, 24.53, 27.33, 13.56 ms.  Total throughput: 397.80 iter/sec.
[Worker #1 Feb 4 14:37] Timing 2048K FFT, 20 cores hyperthreaded, 10 workers.  Average times: 21.30, 21.08, 20.98, 14.93, 13.38, 75.20, 63.41, 41.90, 75.60, 29.19 ms.  Total throughput: 384.20 iter/sec.
[Worker #1 Feb 4 14:37] Timing 2048K FFT, 20 cores hyperthreaded, 12 workers.  Average times: 47.05, 52.05, 26.22, 26.22, 25.92, 25.72, 47.48, 49.10, 24.61, 24.65, 24.63, 24.28 ms.  Total throughput: 398.63 iter/sec.
[Worker #1 Feb 4 14:38] Timing 2048K FFT, 21 cores, 1 worker.  Average times:  1.35 ms.  Total throughput: 738.44 iter/sec.
[Worker #1 Feb 4 14:38] Timing 2048K FFT, 21 cores, 2 workers.  Average times:  1.34,  1.39 ms.  Total throughput: 1462.84 iter/sec.
[Worker #1 Feb 4 14:38] Timing 2048K FFT, 21 cores, 3 workers.  Average times:  5.12,  3.46,  1.45 ms.  Total throughput: 1174.20 iter/sec.
[Worker #1 Feb 4 14:39] Timing 2048K FFT, 21 cores, 4 workers.  Average times:  6.58,  4.18,  4.50,  7.44 ms.  Total throughput: 747.74 iter/sec.
[Worker #1 Feb 4 14:39] Timing 2048K FFT, 21 cores, 5 workers.  Average times: 15.02, 33.18, 25.37,  4.55,  5.24 ms.  Total throughput: 546.71 iter/sec.
[Worker #1 Feb 4 14:39] Timing 2048K FFT, 21 cores, 6 workers.  Average times: 43.51,  7.16, 38.70, 15.02, 17.01, 10.18 ms.  Total throughput: 412.17 iter/sec.
[Worker #1 Feb 4 14:40] Timing 2048K FFT, 21 cores, 7 workers.  Average times: 24.35, 20.95, 15.63, 15.46, 13.54, 15.55, 10.11 ms.  Total throughput: 454.51 iter/sec.
[Worker #1 Feb 4 14:40] Timing 2048K FFT, 21 cores, 8 workers.  Average times: 24.04, 17.16, 11.57, 17.06, 42.72, 68.76, 13.58, 42.73 ms.  Total throughput: 379.93 iter/sec.
[Worker #1 Feb 4 14:41] Timing 2048K FFT, 21 cores, 10 workers.  Average times: 27.57, 28.00, 28.00, 27.48, 18.33, 24.00, 23.99, 22.44, 24.01, 23.66 ms.  Total throughput: 410.46 iter/sec.
[Worker #1 Feb 4 14:41] Timing 2048K FFT, 21 cores, 12 workers.  Average times: 57.76, 27.99, 29.58, 29.71, 29.40, 27.21, 39.92, 39.73, 20.25, 33.63, 28.54, 26.65 ms.  Total throughput: 393.17 iter/sec.
[Worker #1 Feb 4 14:42] Timing 2048K FFT, 21 cores hyperthreaded, 1 worker.  Average times:  1.54 ms.  Total throughput: 647.70 iter/sec.
[Worker #1 Feb 4 14:42] Timing 2048K FFT, 21 cores hyperthreaded, 2 workers.  Average times:  1.39,  1.37 ms.  Total throughput: 1449.67 iter/sec.
[Worker #1 Feb 4 14:42] Timing 2048K FFT, 21 cores hyperthreaded, 3 workers.  Average times:  6.20,  3.27,  1.40 ms.  Total throughput: 1178.71 iter/sec.
[Worker #1 Feb 4 14:43] Timing 2048K FFT, 21 cores hyperthreaded, 4 workers.  Average times:  6.75,  4.18, 10.43,  3.64 ms.  Total throughput: 757.66 iter/sec.
[Worker #1 Feb 4 14:43] Timing 2048K FFT, 21 cores hyperthreaded, 5 workers.  Average times: 13.00, 15.19,  9.00,  5.63,  5.60 ms.  Total throughput: 609.97 iter/sec.
[Worker #1 Feb 4 14:44] Timing 2048K FFT, 21 cores hyperthreaded, 6 workers.  Average times: 18.89, 13.58, 11.31, 48.31, 46.97,  9.19 ms.  Total throughput: 365.86 iter/sec.
[Worker #1 Feb 4 14:44] Timing 2048K FFT, 21 cores hyperthreaded, 7 workers.  Average times: 32.64, 38.38, 54.18, 48.09, 10.44,  9.82,  8.72 ms.  Total throughput: 408.17 iter/sec.
[Worker #1 Feb 4 14:45] Timing 2048K FFT, 21 cores hyperthreaded, 8 workers.  Average times: 23.87, 15.72, 15.74, 15.70, 27.20, 26.44, 17.28, 17.29 ms.  Total throughput: 422.99 iter/sec.
[Worker #1 Feb 4 14:45] Timing 2048K FFT, 21 cores hyperthreaded, 10 workers.  Average times: 19.08, 42.93, 19.10, 19.07, 31.70, 30.36, 30.14, 30.40, 29.60, 29.26 ms.  Total throughput: 379.03 iter/sec.
[Worker #1 Feb 4 14:45] Timing 2048K FFT, 21 cores hyperthreaded, 12 workers.  Average times: 122.90, 97.11, 74.90, 67.78, 57.22, 33.67, 49.81, 28.30, 27.36, 27.66, 28.43, 27.42 ms.  Total throughput: 293.47 iter/sec.
[Worker #1 Feb 4 14:46] Timing 2048K FFT, 22 cores, 1 worker.  Average times:  1.20 ms.  Total throughput: 832.05 iter/sec.
[Worker #1 Feb 4 14:46] Timing 2048K FFT, 22 cores, 2 workers.  Average times:  1.37,  1.30 ms.  Total throughput: 1498.05 iter/sec.
[Worker #1 Feb 4 14:47] Timing 2048K FFT, 22 cores, 3 workers.  Average times:  4.62,  3.32,  1.32 ms.  Total throughput: 1276.40 iter/sec.
[Worker #1 Feb 4 14:47] Timing 2048K FFT, 22 cores, 4 workers.  Average times:  5.41,  5.97,  6.02,  3.98 ms.  Total throughput: 769.67 iter/sec.
[Worker #1 Feb 4 14:47] Timing 2048K FFT, 22 cores, 5 workers.  Average times: 15.02,  7.98,  9.91,  7.88,  4.10 ms.  Total throughput: 663.37 iter/sec.
[Worker #1 Feb 4 14:48] Timing 2048K FFT, 22 cores, 6 workers.  Average times: 17.11, 11.91, 11.75, 14.15, 11.91,  9.78 ms.  Total throughput: 484.45 iter/sec.
[Worker #1 Feb 4 14:48] Timing 2048K FFT, 22 cores, 7 workers.  Average times: 103.53, 54.06, 67.35, 67.34,  8.40,  9.65,  9.28 ms.  Total throughput: 388.31 iter/sec.
[Worker #1 Feb 4 14:49] Timing 2048K FFT, 22 cores, 8 workers.  Average times: 26.20, 17.06, 17.06, 16.74, 26.01, 16.98, 17.12, 16.58 ms.  Total throughput: 431.22 iter/sec.
[Worker #1 Feb 4 14:49] Timing 2048K FFT, 22 cores, 10 workers.  Average times: 76.66, 74.71, 19.77, 37.33, 37.04, 23.45, 21.64, 21.75, 19.56, 14.04 ms.  Total throughput: 387.94 iter/sec.
[Worker #1 Feb 4 14:50] Timing 2048K FFT, 22 cores, 12 workers.  Average times: 41.89, 37.04, 36.26, 20.46, 29.74, 37.18, 41.70, 35.18, 35.72, 21.60, 35.79, 27.89 ms.  Total throughput: 378.34 iter/sec.
[Worker #1 Feb 4 14:50] Timing 2048K FFT, 22 cores hyperthreaded, 1 worker.  Average times:  1.44 ms.  Total throughput: 696.56 iter/sec.
[Worker #1 Feb 4 14:51] Timing 2048K FFT, 22 cores hyperthreaded, 2 workers.  Average times:  1.41,  1.31 ms.  Total throughput: 1475.61 iter/sec.
[Worker #1 Feb 4 14:51] Timing 2048K FFT, 22 cores hyperthreaded, 3 workers.  Average times:  4.79,  2.99,  1.33 ms.  Total throughput: 1295.13 iter/sec.
[Worker #1 Feb 4 14:51] Timing 2048K FFT, 22 cores hyperthreaded, 4 workers.  Average times:  7.58,  2.87,  6.84,  3.74 ms.  Total throughput: 894.39 iter/sec.
[Worker #1 Feb 4 14:52] Timing 2048K FFT, 22 cores hyperthreaded, 5 workers.  Average times: 12.23, 10.13,  6.74, 14.09,  7.45 ms.  Total throughput: 533.96 iter/sec.
[Worker #1 Feb 4 14:52] Timing 2048K FFT, 22 cores hyperthreaded, 6 workers.  Average times: 19.39, 13.13, 13.45, 51.13, 41.67, 11.96 ms.  Total throughput: 329.29 iter/sec.
[Worker #1 Feb 4 14:53] Timing 2048K FFT, 22 cores hyperthreaded, 7 workers.  Average times: 37.58, 13.28, 15.42, 15.38, 19.96, 13.56, 13.56 ms.  Total throughput: 429.32 iter/sec.
[Worker #1 Feb 4 14:53] Timing 2048K FFT, 22 cores hyperthreaded, 8 workers.  Average times: 33.17, 18.85, 15.39, 12.19, 32.26, 21.02, 21.04, 20.10 ms.  Total throughput: 406.11 iter/sec.
[Worker #1 Feb 4 14:53] Timing 2048K FFT, 22 cores hyperthreaded, 10 workers.  Average times: 19.74, 46.72, 19.72, 29.95, 29.30, 33.82, 33.78, 34.02, 34.04, 21.22 ms.  Total throughput: 355.36 iter/sec.
[Worker #1 Feb 4 14:54] Timing 2048K FFT, 22 cores hyperthreaded, 12 workers.  Average times: 55.80, 17.54, 25.72, 16.13, 27.15, 25.78, 96.20, 58.59, 54.36, 60.71, 45.69, 20.66 ms.  Total throughput: 384.04 iter/sec.
[Worker #1 Feb 4 14:54] Timing 2048K FFT, 23 cores, 1 worker.  Average times:  1.21 ms.  Total throughput: 823.22 iter/sec.
[Worker #1 Feb 4 14:55] Timing 2048K FFT, 23 cores, 2 workers.  Average times:  1.32,  1.29 ms.  Total throughput: 1537.72 iter/sec.
[Worker #1 Feb 4 14:55] Timing 2048K FFT, 23 cores, 3 workers.  Average times:  3.66,  4.35,  1.33 ms.  Total throughput: 1256.63 iter/sec.
[Worker #1 Feb 4 14:56] Timing 2048K FFT, 23 cores, 4 workers.  Average times:  3.06, 11.82,  4.37,  3.02 ms.  Total throughput: 971.80 iter/sec.
[Worker #1 Feb 4 14:56] Timing 2048K FFT, 23 cores, 5 workers.  Average times: 10.79, 11.33,  5.80, 24.85, 15.28 ms.  Total throughput: 459.19 iter/sec.
[Worker #1 Feb 4 14:56] Timing 2048K FFT, 23 cores, 6 workers.  Average times: 12.43, 12.41,  6.60, 57.77, 26.62, 38.57 ms.  Total throughput: 393.37 iter/sec.
[Worker #1 Feb 4 14:57] Timing 2048K FFT, 23 cores, 7 workers.  Average times: 20.04, 19.49, 18.32, 18.36, 14.35, 23.54,  8.73 ms.  Total throughput: 437.00 iter/sec.
[Worker #1 Feb 4 14:57] Timing 2048K FFT, 23 cores, 8 workers.  Average times: 16.32, 17.18, 17.95, 17.55, 74.38, 19.29, 51.29, 48.82 ms.  Total throughput: 337.45 iter/sec.
[Worker #1 Feb 4 14:58] Timing 2048K FFT, 23 cores, 10 workers.  Average times: 31.89, 31.75, 32.35, 20.60, 16.79, 40.45, 37.90, 20.06, 42.43, 16.37 ms.  Total throughput: 387.48 iter/sec.
[Worker #1 Feb 4 14:58] Timing 2048K FFT, 23 cores, 12 workers.  Average times: 31.29, 30.44, 31.20, 30.96, 30.45, 29.99, 53.06, 26.92, 26.86, 26.68, 26.90, 26.34 ms.  Total throughput: 401.21 iter/sec.
[Worker #1 Feb 4 14:59] Timing 2048K FFT, 23 cores hyperthreaded, 1 worker.  Average times:  1.50 ms.  Total throughput: 666.59 iter/sec.
[Worker #1 Feb 4 14:59] Timing 2048K FFT, 23 cores hyperthreaded, 2 workers.  [Feb 4 14:59] Updating computer information on the server
[Worker #1] Average times:  1.37,  1.32 ms.  Total throughput: 1492.00 iter/sec.
[Worker #1 Feb 4 14:59] Timing 2048K FFT, 23 cores hyperthreaded, 3 workers.  Average times:  5.75,  4.01,  1.34 ms.  Total throughput: 1167.66 iter/sec.
[Worker #1 Feb 4 15:00] Timing 2048K FFT, 23 cores hyperthreaded, 4 workers.  Average times:  2.95,  6.98,  6.67,  3.34 ms.  Total throughput: 932.10 iter/sec.
[Worker #1 Feb 4 15:00] Timing 2048K FFT, 23 cores hyperthreaded, 5 workers.  Average times: 17.07, 12.88,  6.97,  7.07,  3.94 ms.  Total throughput: 675.07 iter/sec.
[Worker #1 Feb 4 15:01] Timing 2048K FFT, 23 cores hyperthreaded, 6 workers.  Average times: 12.00, 16.13, 11.68, 17.22, 12.88, 12.89 ms.  Total throughput: 444.19 iter/sec.
[Worker #1 Feb 4 15:01] Timing 2048K FFT, 23 cores hyperthreaded, 7 workers.  Average times: 15.62, 17.60, 15.41, 16.01, 27.21,  7.99, 21.40 ms.  Total throughput: 456.83 iter/sec.
[Worker #1 Feb 4 15:01] Timing 2048K FFT, 23 cores hyperthreaded, 8 workers.  Average times: 19.56, 19.57, 19.17, 18.70, 27.21, 17.36, 17.38, 16.79 ms.  Total throughput: 419.33 iter/sec.
[Worker #1 Feb 4 15:02] Timing 2048K FFT, 23 cores hyperthreaded, 10 workers.  Average times: 35.35, 35.08, 35.33, 22.66, 21.92, 23.98, 25.94, 24.01, 24.00, 15.27 ms.  Total throughput: 403.89 iter/sec.
[Worker #1 Feb 4 15:02] Timing 2048K FFT, 23 cores hyperthreaded, 12 workers.  Average times: 35.16, 34.96, 36.44, 35.12, 34.02, 33.60, 47.66, 28.85, 29.84, 23.57, 23.20, 22.89 ms.  Total throughput: 390.50 iter/sec.
[Worker #1 Feb 4 15:03] Timing 2048K FFT, 24 cores, 1 worker.  Average times:  1.24 ms.  Total throughput: 806.07 iter/sec.
[Worker #1 Feb 4 15:03] Timing 2048K FFT, 24 cores, 2 workers.  Average times:  1.32,  1.23 ms.  Total throughput: 1571.30 iter/sec.
[Worker #1 Feb 4 15:04] Timing 2048K FFT, 24 cores, 3 workers.  Average times:  4.06,  3.43,  1.24 ms.  Total throughput: 1341.85 iter/sec.
[Worker #1 Feb 4 15:04] Timing 2048K FFT, 24 cores, 4 workers.  Average times:  5.37,  5.36, 11.32,  3.23 ms.  Total throughput: 770.52 iter/sec.
[Worker #1 Feb 4 15:04] Timing 2048K FFT, 24 cores, 5 workers.  Average times:  9.41, 11.78, 28.25,  5.37,  5.02 ms.  Total throughput: 612.02 iter/sec.
[Worker #1 Feb 4 15:05] Timing 2048K FFT, 24 cores, 6 workers.  Average times: 15.01, 15.01, 14.82,  9.84, 14.36,  9.76 ms.  Total throughput: 474.49 iter/sec.
[Worker #1 Feb 4 15:05] Timing 2048K FFT, 24 cores, 7 workers.  Average times: 41.04, 14.50, 30.40, 42.34, 11.78,  9.30, 11.83 ms.  Total throughput: 426.80 iter/sec.
[Worker #1 Feb 4 15:06] Timing 2048K FFT, 24 cores, 8 workers.  Average times: 23.73, 23.73, 23.33, 22.55, 37.62, 11.70, 46.42, 11.27 ms.  Total throughput: 393.85 iter/sec.
[Worker #1 Feb 4 15:06] Timing 2048K FFT, 24 cores, 10 workers.  Average times: 32.62, 30.61, 32.48, 21.43, 21.26, 27.12, 38.75, 21.33, 30.67, 13.64 ms.  Total throughput: 403.33 iter/sec.
[Worker #1 Feb 4 15:07] Timing 2048K FFT, 24 cores, 12 workers.  Average times: 30.29, 32.20, 26.17, 32.14, 32.22, 30.93, 82.23, 72.70, 74.86, 31.21, 67.68, 82.19 ms.  Total throughput: 295.02 iter/sec.
[Worker #1 Feb 4 15:07] Timing 2048K FFT, 24 cores hyperthreaded, 1 worker.  Average times:  1.52 ms.  Total throughput: 657.92 iter/sec.
[Worker #1 Feb 4 15:07] Timing 2048K FFT, 24 cores hyperthreaded, 2 workers.  Average times:  1.37,  1.28 ms.  Total throughput: 1514.49 iter/sec.
[Worker #1 Feb 4 15:08] Timing 2048K FFT, 24 cores hyperthreaded, 3 workers.  Average times:  3.90,  5.73,  1.30 ms.  Total throughput: 1199.54 iter/sec.
[Worker #1 Feb 4 15:08] Timing 2048K FFT, 24 cores hyperthreaded, 4 workers.  Average times:  7.98, 13.33,  3.99,  3.56 ms.  Total throughput: 731.76 iter/sec.
[Worker #1 Feb 4 15:09] Timing 2048K FFT, 24 cores hyperthreaded, 5 workers.  Average times: 12.50,  8.26, 12.47,  6.08, 17.68 ms.  Total throughput: 502.15 iter/sec.
[Worker #1 Feb 4 15:09] Timing 2048K FFT, 24 cores hyperthreaded, 6 workers.  Average times: 12.72, 12.72, 12.17, 13.96, 13.95, 13.28 ms.  Total throughput: 458.05 iter/sec.
[Worker #1 Feb 4 15:09] Timing 2048K FFT, 24 cores hyperthreaded, 7 workers.  Average times: 19.37, 14.71, 18.24, 18.15,  9.52, 37.59, 24.75 ms.  Total throughput: 401.65 iter/sec.
[Worker #1 Feb 4 15:10] Timing 2048K FFT, 24 cores hyperthreaded, 8 workers.  Average times: 19.95, 19.23, 19.22, 18.62, 19.46, 19.50, 19.35, 18.43 ms.  Total throughput: 416.47 iter/sec.
[Worker #1 Feb 4 15:10] Timing 2048K FFT, 24 cores hyperthreaded, 10 workers.  Average times: 38.18, 80.25, 80.10, 27.98, 47.13, 23.43, 20.94, 23.42, 15.79, 15.07 ms.  Total throughput: 370.89 iter/sec.
[Worker #1 Feb 4 15:11] Timing 2048K FFT, 24 cores hyperthreaded, 12 workers.  Average times: 118.67, 118.35, 118.57, 109.60, 46.49, 114.55, 18.26, 17.36, 25.13, 25.22, 23.35, 24.52 ms.  Total throughput: 340.11 iter/sec.

Last fiddled with by Uncwilly on 2020-02-05 at 16:12 Reason: changed to code for better reading of text
jas is offline   Reply With Quote
Old 2020-02-04, 16:59   #10
sdbardwick
 
sdbardwick's Avatar
 
Aug 2002
North San Diego County

2×11×31 Posts
Default

12 cores per socket, perhaps?
Code:
[Worker #1 Feb 4 15:03] Timing 2048K FFT, 24 cores, 2 workers. Average times: 1.32, 1.23 ms. Total throughput: 1571.30 iter/sec.
sdbardwick is offline   Reply With Quote
Old 2020-02-04, 18:30   #11
jas
 
"Simon Josefsson"
Jan 2020
Stockholm

308 Posts
Default

Quote:
Originally Posted by sdbardwick View Post
12 cores per socket, perhaps?
Code:
[Worker #1 Feb 4 15:03] Timing 2048K FFT, 24 cores, 2 workers. Average times: 1.32, 1.23 ms. Total throughput: 1571.30 iter/sec.

Yes, the 2673v2 has 12 cores per CPU. It seems the same performance generalized to 20480K FFTs too, so I will use 2 workers with 24 cores for real usage on this machine. And I've learned how to pick good settings for my other machines too....


Thanks for all your help!



/Simon


Code:
[Worker #1 Feb 4 17:27] Timing 20480K FFT, 1 core, 1 worker.  Average times: 133.38 ms.  Total throughput:  7.50 iter/sec.
[Worker #1 Feb 4 17:28] Timing 20480K FFT, 4 cores, 1 worker.  Average times: 51.25 ms.  Total throughput: 19.51 iter/sec.
[Worker #1 Feb 4 17:28] Timing 20480K FFT, 4 cores, 2 workers.  Average times: 74.35, 73.54 ms.  Total throughput: 27.05 iter/sec.
[Worker #1 Feb 4 17:29] Timing 20480K FFT, 4 cores, 3 workers.  Average times: 143.43, 142.49, 73.08 ms.  Total throughput: 27.67 iter/sec.
[Worker #1 Feb 4 17:30] Timing 20480K FFT, 4 cores, 4 workers.  Average times: 143.07, 142.80, 143.83, 143.46 ms.  Total throughput: 27.92 iter/sec.
[Worker #1 Feb 4 17:30] Timing 20480K FFT, 8 cores, 1 worker.  Average times: 50.28 ms.  Total throughput: 19.89 iter/sec.
[Worker #1 Feb 4 17:31] Timing 20480K FFT, 8 cores, 2 workers.  Average times: 51.82, 51.72 ms.  Total throughput: 38.63 iter/sec.
[Worker #1 Feb 4 17:31] Timing 20480K FFT, 8 cores, 3 workers.  Average times: 104.44, 104.32, 51.68 ms.  Total throughput: 38.51 iter/sec.
[Worker #1 Feb 4 17:32] Timing 20480K FFT, 8 cores, 4 workers.  Average times: 103.54, 103.40, 103.27, 103.27 ms.  Total throughput: 38.70 iter/sec.
[Worker #1 Feb 4 17:33] Timing 20480K FFT, 8 cores, 5 workers.  Average times: 203.82, 203.40, 105.24, 103.68, 103.28 ms.  Total throughput: 38.65 iter/sec.
[Worker #1 Feb 4 17:34] Timing 20480K FFT, 8 cores, 6 workers.  Average times: 206.46, 204.71, 105.40, 202.39, 202.89, 104.75 ms.  Total throughput: 38.63 iter/sec.
[Worker #1 Feb 4 17:36] Timing 20480K FFT, 12 cores, 1 worker.  Average times: 50.64 ms.  Total throughput: 19.75 iter/sec.
[Worker #1 Feb 4 17:36] Timing 20480K FFT, 12 cores, 2 workers.  Average times: 50.92, 50.64 ms.  Total throughput: 39.39 iter/sec.
[Worker #1 Feb 4 17:37] Timing 20480K FFT, 12 cores, 3 workers.  Average times: 103.31, 102.76, 50.81 ms.  Total throughput: 39.09 iter/sec.
[Worker #1 Feb 4 17:37] Timing 20480K FFT, 12 cores, 4 workers.  Average times: 102.89, 102.78, 103.53, 103.17 ms.  Total throughput: 38.80 iter/sec.
[Worker #1 Feb 4 17:38] Timing 20480K FFT, 12 cores, 5 workers.  Average times: 155.37, 154.95, 153.66, 103.33, 103.51 ms.  Total throughput: 38.74 iter/sec.
[Worker #1 Feb 4 17:39] Timing 20480K FFT, 12 cores, 6 workers.  Average times: 156.61, 155.96, 155.28, 154.65, 155.38, 154.77 ms.  Total throughput: 38.60 iter/sec.
[Worker #1 Feb 4 17:41] Timing 20480K FFT, 12 cores, 10 workers.  Average times: 310.01, 306.04, 310.50, 310.04, 156.99, 309.24, 306.57, 311.59, 310.85, 157.25 ms.  Total throughput: 38.59 iter/sec.
[Worker #1 Feb 4 17:43] Timing 20480K FFT, 12 cores, 12 workers.  Average times: 315.71, 314.05, 314.14, 314.38, 312.10, 310.50, 312.24, 311.98, 312.24, 311.32, 307.62, 310.17 ms.  Total throughput: 38.44 iter/sec.
[Worker #1 Feb 4 17:45] Timing 20480K FFT, 14 cores, 1 worker.  Average times: 46.55 ms.  Total throughput: 21.48 iter/sec.
[Worker #1 Feb 4 17:45] Timing 20480K FFT, 14 cores, 2 workers.  Average times: 50.87, 50.61 ms.  Total throughput: 39.42 iter/sec.
[Worker #1 Feb 4 17:46] Timing 20480K FFT, 14 cores, 3 workers.  Average times: 121.05, 89.68, 50.70 ms.  Total throughput: 39.14 iter/sec.
[Worker #1 Feb 4 17:47] Timing 20480K FFT, 14 cores, 4 workers.  Average times: 121.22, 89.24, 120.35, 89.68 ms.  Total throughput: 38.92 iter/sec.
[Worker #1 Feb 4 17:48] Timing 20480K FFT, 14 cores, 5 workers.  Average times: 180.19, 180.22, 120.15, 120.66, 90.08 ms.  Total throughput: 38.81 iter/sec.
[Worker #1 Feb 4 17:49] Timing 20480K FFT, 14 cores, 6 workers.  Average times: 180.81, 182.16, 121.41, 179.78, 178.61, 119.64 ms.  Total throughput: 38.78 iter/sec.
[Worker #1 Feb 4 17:50] Timing 20480K FFT, 14 cores, 10 workers.  Average times: 363.97, 357.42, 360.06, 179.87, 181.23, 359.01, 361.16, 361.67, 182.84, 182.26 ms.  Total throughput: 38.68 iter/sec.
[Worker #1 Feb 4 17:52] Timing 20480K FFT, 14 cores, 12 workers.  Average times: 359.84, 365.63, 364.84, 355.85, 364.31, 181.51, 364.00, 357.53, 357.43, 362.51, 361.00, 180.68 ms.  Total throughput: 38.72 iter/sec.
[Worker #1 Feb 4 17:54] Timing 20480K FFT, 16 cores, 1 worker.  Average times: 47.49 ms.  Total throughput: 21.06 iter/sec.
[Worker #1 Feb 4 17:54] Timing 20480K FFT, 16 cores, 2 workers.  Average times: 51.45, 50.19 ms.  Total throughput: 39.36 iter/sec.
[Worker #1 Feb 4 17:55] Timing 20480K FFT, 16 cores, 3 workers.  Average times: 102.62, 102.42, 50.68 ms.  Total throughput: 39.24 iter/sec.
[Worker #1 Feb 4 17:56] Timing 20480K FFT, 16 cores, 4 workers.  Average times: 102.19, 101.91, 103.56, 102.53 ms.  Total throughput: 39.01 iter/sec.
[Worker #1 Feb 4 17:57] Timing 20480K FFT, 16 cores, 5 workers.  Average times: 207.36, 138.25, 134.84, 103.52, 101.59 ms.  Total throughput: 38.98 iter/sec.
[Worker #1 Feb 4 17:58] Timing 20480K FFT, 16 cores, 6 workers.  Average times: 205.89, 137.71, 138.29, 203.23, 136.28, 137.27 ms.  Total throughput: 38.89 iter/sec.
[Worker #1 Feb 4 17:59] Timing 20480K FFT, 16 cores, 10 workers.  Average times: 414.53, 412.43, 206.45, 203.56, 208.34, 403.16, 409.19, 205.46, 208.74, 208.92 ms.  Total throughput: 38.76 iter/sec.
[Worker #1 Feb 4 18:01] Timing 20480K FFT, 16 cores, 12 workers.  Average times: 407.24, 403.65, 409.84, 408.68, 208.41, 208.86, 404.26, 410.58, 411.82, 412.21, 208.05, 208.57 ms.  Total throughput: 38.77 iter/sec.
[Worker #1 Feb 4 18:03] Timing 20480K FFT, 16 cores, 16 workers.  Average times: 419.70, 406.29, 416.70, 415.07, 404.28, 415.87, 406.63, 407.10, 416.98, 405.67, 419.06, 417.15, 415.76, 406.76, 404.59, 417.94 ms.  Total throughput: 38.82 iter/sec.
[Worker #1 Feb 4 18:06] Timing 20480K FFT, 18 cores, 1 worker.  Average times: 48.01 ms.  Total throughput: 20.83 iter/sec.
[Worker #1 Feb 4 18:06] Timing 20480K FFT, 18 cores, 2 workers.  Average times: 50.68, 50.52 ms.  Total throughput: 39.53 iter/sec.
[Worker #1 Feb 4 18:07] Timing 20480K FFT, 18 cores, 3 workers.  Average times: 115.31, 92.76, 50.42 ms.  Total throughput: 39.29 iter/sec.
[Worker #1 Feb 4 18:08] Timing 20480K FFT, 18 cores, 4 workers.  Average times: 115.43, 92.16, 115.18, 92.56 ms.  Total throughput: 39.00 iter/sec.
[Worker #1 Feb 4 18:08] Timing 20480K FFT, 18 cores, 5 workers.  Average times: 155.03, 154.47, 155.86, 114.02, 92.48 ms.  Total throughput: 38.92 iter/sec.
[Worker #1 Feb 4 18:10] Timing 20480K FFT, 18 cores, 6 workers.  Average times: 154.25, 154.07, 153.05, 153.70, 153.57, 156.46 ms.  Total throughput: 38.92 iter/sec.
[Worker #1 Feb 4 18:11] Timing 20480K FFT, 18 cores, 10 workers.  Average times: 470.53, 237.78, 235.70, 235.13, 238.68, 465.49, 233.35, 231.70, 235.76, 234.75 ms.  Total throughput: 38.27 iter/sec.
[Worker #1 Feb 4 18:13] Timing 20480K FFT, 18 cores, 12 workers.  Average times: 478.08, 475.66, 476.06, 239.27, 240.27, 234.45, 470.13, 468.52, 469.86, 234.77, 235.85, 236.64 ms.  Total throughput: 38.02 iter/sec.
[Worker #1 Feb 4 18:15] Timing 20480K FFT, 18 cores, 16 workers.  Average times: 477.40, 472.88, 477.01, 470.37, 474.66, 475.93, 476.36, 238.41, 473.75, 467.90, 470.88, 467.87, 454.44, 473.70, 471.08, 237.83 ms.  Total throughput: 38.08 iter/sec.
[Worker #1 Feb 4 18:18] Timing 20480K FFT, 20 cores, 1 worker.  Average times: 50.99 ms.  Total throughput: 19.61 iter/sec.
[Worker #1 Feb 4 18:18] Timing 20480K FFT, 20 cores, 2 workers.  Average times: 53.81, 50.25 ms.  Total throughput: 38.48 iter/sec.
[Worker #1 Feb 4 18:19] Timing 20480K FFT, 20 cores, 3 workers.  Average times: 107.32, 103.28, 51.15 ms.  Total throughput: 38.55 iter/sec.
[Worker #1 Feb 4 18:19] Timing 20480K FFT, 20 cores, 4 workers.  Average times: 105.94, 105.50, 103.31, 103.58 ms.  Total throughput: 38.25 iter/sec.
[Worker #1 Feb 4 18:20] Timing 20480K FFT, 20 cores, 5 workers.  Average times: 175.39, 175.52, 130.49, 104.84, 105.13 ms.  Total throughput: 38.11 iter/sec.
[Worker #1 Feb 4 18:21] Timing 20480K FFT, 20 cores, 6 workers.  Average times: 178.53, 178.44, 132.98, 169.56, 173.49, 128.18 ms.  Total throughput: 38.19 iter/sec.
[Worker #1 Feb 4 18:23] Timing 20480K FFT, 20 cores, 10 workers.  Average times: 264.27, 262.51, 260.31, 264.56, 262.12, 260.58, 261.14, 261.46, 262.93, 255.23 ms.  Total throughput: 38.24 iter/sec.
[Worker #1 Feb 4 18:25] Timing 20480K FFT, 20 cores, 12 workers.  Average times: 526.14, 524.28, 261.96, 267.19, 264.83, 264.43, 518.12, 516.09, 261.06, 262.88, 264.66, 257.61 ms.  Total throughput: 38.09 iter/sec.
[Worker #1 Feb 4 18:27] Timing 20480K FFT, 20 cores, 16 workers.  Average times: 532.28, 526.53, 524.03, 525.00, 526.39, 532.02, 262.09, 261.25, 528.84, 524.88, 519.31, 525.42, 523.86, 533.10, 268.82, 258.25 ms.  Total throughput: 38.02 iter/sec.
[Worker #1 Feb 4 18:29] Timing 20480K FFT, 22 cores, 1 worker.  Average times: 49.25 ms.  Total throughput: 20.31 iter/sec.
[Worker #1 Feb 4 18:30] Timing 20480K FFT, 22 cores, 2 workers.  Average times: 52.20, 51.69 ms.  Total throughput: 38.50 iter/sec.
[Worker #1 Feb 4 18:30] Timing 20480K FFT, 22 cores, 3 workers.  Average times: 114.90, 97.22, 51.53 ms.  Total throughput: 38.39 iter/sec.
[Worker #1 Feb 4 18:31] Timing 20480K FFT, 22 cores, 4 workers.  Average times: 118.52, 97.77, 112.57, 94.31 ms.  Total throughput: 38.15 iter/sec.
[Worker #1 Feb 4 18:32] Timing 20480K FFT, 22 cores, 5 workers.  Average times: 195.19, 147.64, 144.81, 114.09, 96.06 ms.  Total throughput: 37.98 iter/sec.
[Worker #1 Feb 4 18:33] Timing 20480K FFT, 22 cores, 6 workers.  Average times: 192.20, 146.34, 146.76, 195.45, 145.66, 142.73 ms.  Total throughput: 37.84 iter/sec.
[Worker #1 Feb 4 18:35] Timing 20480K FFT, 22 cores, 10 workers.  Average times: 286.97, 285.71, 294.92, 293.39, 190.90, 292.63, 287.60, 295.18, 297.38, 194.37 ms.  Total throughput: 37.81 iter/sec.
[Worker #1 Feb 4 18:36] Timing 20480K FFT, 22 cores, 12 workers.  Average times: 585.98, 288.35, 287.72, 296.41, 290.74, 288.40, 576.60, 290.16, 290.02, 296.78, 285.82, 284.88 ms.  Total throughput: 37.94 iter/sec.
[Worker #1 Feb 4 18:38] Timing 20480K FFT, 22 cores, 16 workers.  Average times: 587.33, 578.66, 581.33, 576.12, 578.54, 294.85, 293.35, 291.39, 583.08, 579.94, 578.33, 575.09, 573.22, 292.12, 290.90, 287.56 ms.  Total throughput: 37.84 iter/sec.
[Worker #1 Feb 4 18:41] Timing 20480K FFT, 24 cores, 1 worker.  Average times: 48.32 ms.  Total throughput: 20.69 iter/sec.
[Worker #1 Feb 4 18:42] Timing 20480K FFT, 24 cores, 2 workers.  Average times: 52.50, 52.18 ms.  Total throughput: 38.21 iter/sec.
[Worker #1 Feb 4 18:42] Timing 20480K FFT, 24 cores, 3 workers.  Average times: 107.42, 106.41, 51.79 ms.  Total throughput: 38.01 iter/sec.
[Worker #1 Feb 4 18:43] Timing 20480K FFT, 24 cores, 4 workers.  Average times: 106.39, 107.59, 106.32, 103.62 ms.  Total throughput: 37.75 iter/sec.
[Worker #1 Feb 4 18:44] Timing 20480K FFT, 24 cores, 5 workers.  Average times: 167.93, 162.24, 155.22, 105.51, 104.49 ms.  Total throughput: 37.61 iter/sec.
[Worker #1 Feb 4 18:45] Timing 20480K FFT, 24 cores, 6 workers.  Average times: 159.38, 161.82, 158.10, 158.03, 161.49, 154.05 ms.  Total throughput: 37.79 iter/sec.
[Worker #1 Feb 4 18:46] Timing 20480K FFT, 24 cores, 10 workers.  Average times: 324.38, 318.66, 321.75, 215.18, 211.91, 320.12, 320.67, 323.91, 210.77, 206.31 ms.  Total throughput: 37.62 iter/sec.
[Worker #1 Feb 4 18:48] Timing 20480K FFT, 24 cores, 12 workers.  Average times: 321.00, 324.07, 327.11, 325.03, 319.79, 316.09, 318.55, 314.69, 324.16, 320.75, 311.18, 305.64 ms.  Total throughput: 37.63 iter/sec.
[Worker #1 Feb 4 18:50] Timing 20480K FFT, 24 cores, 16 workers.  Average times: 649.61, 641.57, 648.14, 639.73, 329.77, 323.81, 321.97, 315.73, 638.12, 627.14, 626.55, 628.26, 322.09, 321.06, 310.92, 306.49 ms.  Total throughput: 37.65 iter/sec.

Last fiddled with by Uncwilly on 2020-02-05 at 16:13 Reason: applied code tags vs quotes
jas is offline   Reply With Quote
Reply

Thread Tools


Similar Threads
Thread Thread Starter Forum Replies Last Post
access to windows machine files from ubuntu machine wildrabbitt Hardware 1 2019-02-24 18:34
prime95 settings evanh Software 3 2017-12-04 15:18
CPU settings and other newbie Qs Z3noN Information & Answers 2 2017-01-11 05:58
Memory Settings Fred Software 5 2016-05-03 00:51
settings for cudaLucas fairsky GPU Computing 11 2013-11-03 02:08

All times are UTC. The time now is 15:48.

Sun May 9 15:48:04 UTC 2021 up 31 days, 10:28, 1 user, load averages: 2.45, 2.09, 1.75

Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, Jelsoft Enterprises Ltd.

This forum has received and complied with 0 (zero) government requests for information.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation.
A copy of the license is included in the FAQ.