Well that was a bit odd

[Frailie60]

A few days ago I accidentally allocated seventy gigabytes from a process running on my 64GB 48-core Opteron machine. The process died and I felt nothing of it.

Except that I noticed that other processes which had been running at the time were now running bizarrely slowly - I was getting 300ms rather than 65ms iteration time from mprime, some gnfs-lasieve4I14e jobs were suggesting they would take a week to run whilst ones on adjacent ranges were expecting less than 24 hours.

So far so odd; muttering something about NUMA I killed the mprime process and restarted. And it continued to have 300ms iteration times.

Everything was fixed by a reboot, but I don't understand how forcing the machine into swap would have these persistent bad consequences.

[retina]

Originally Posted by Frailie60

A few days ago I accidentally allocated seventy gigabytes from a process running on my 64GB 48-core Opteron machine. The process died and I felt nothing of it.

Except that I noticed that other processes which had been running at the time were now running bizarrely slowly - I was getting 300ms rather than 65ms iteration time from mprime, some gnfs-lasieve4I14e jobs were suggesting they would take a week to run whilst ones on adjacent ranges were expecting less than 24 hours.

So far so odd; muttering something about NUMA I killed the mprime process and restarted. And it continued to have 300ms iteration times.

Everything was fixed by a reboot, but I don't understand how forcing the machine into swap would have these persistent bad consequences.

IME Linux is stupid when it comes to memory allocations for anything other than a basic system with one CPU.

[jnml]

Originally Posted by retina

IME Linux is stupid when it comes to memory allocations for anything other than a basic system with one CPU.

AFAICT it's quite the opposite. The kernel assumes the last high memory pressure is likely
to repeat soon and shrinks the swap-in-use size only after that does not happen for some
time. Meanwhile several effects kick in making things slower.

It's a tradeoff that works better in high overall resource demand scenarios compared to releasing things as soon as not needed.

[retina]

Originally Posted by jnml

... shrinks the swap-in-use size only after that does not happen for some
time.

What is the "some time"? Is it configurable?

[jnml]

Originally Posted by retina

What is the "some time"? Is it configurable?

IDK, my machine seems to take few minutes to get back to nearly normal after a big spike in
memory usage (using almost all swap space).

But the swap file does not go completely back to zero, only down to ~30%. Maybe some
time later it shrinks even more.

[Happy5214]

Based on my experience with Linux, the kernel doesn't make much of an effort to move memory back to RAM until the program swapped out actually needs to be used. On my computer (8 GB RAM, 24 GB swap), I had a huge spike in memory usage yesterday which sent my swap usage to a little over 2 GB. My RAM usage has gone back down to 4.8 GB, but it's still using 1.8 GB of swap space.

[xilman]

Originally Posted by Happy5214

Based on my experience with Linux, the kernel doesn't make much of an effort to move memory back to RAM until the program swapped out actually needs to be used.

Isn't that exactly what it should do?

Swapping in something which isn't going to be run just uses up memory which could be used by something which is running.

Anyway, swapping behavior can be configured with sysctl(8). Details left as an exercise in reading TFM. If I had to find out for myself, I don't see why you shouldn't educate yourself

[Xyzzy]

https://en.wikipedia.org/wiki/Swappiness

[chris2be8]

Originally Posted by Frailie60

A few days ago I accidentally allocated seventy gigabytes from a process running on my 64GB 48-core Opteron machine. The process died and I felt nothing of it.

Except that I noticed that other processes which had been running at the time were now running bizarrely slowly - I was getting 300ms rather than 65ms iteration time from mprime, some gnfs-lasieve4I14e jobs were suggesting they would take a week to run whilst ones on adjacent ranges were expecting less than 24 hours.

So far so odd; muttering something about NUMA I killed the mprime process and restarted. And it continued to have 300ms iteration times.

Everything was fixed by a reboot, but I don't understand how forcing the machine into swap would have these persistent bad consequences.

In a NUMA system each CPU has fast access to memory attached to it and slower access to memory attached to other CPUs. So you get better performance if a process is using memory attached to the CPU it's running on. If it gets swapped out, then brought back into memory on other CPUs it will run slower. And Linux probably isn't smart enough to migrate processes back into memory on the CPUs they run on once they have been pushed out by something else.

I don't know the details but that's probably why thing ran slower until you rebooted.

Chris

[jnml]

Originally Posted by chris2be8

And Linux probably isn't smart enough to migrate processes back into memory on the CPUs they run on once they have been pushed out by something else.
Chris

Both NUMA and HPC are a thing for long enough that, while considering Linux dominance in
supercomputing, the above quoted is probably not true.

[Mark Rose]

Originally Posted by chris2be8

In a NUMA system each CPU has fast access to memory attached to it and slower access to memory attached to other CPUs. So you get better performance if a process is using memory attached to the CPU it's running on. If it gets swapped out, then brought back into memory on other CPUs it will run slower. And Linux probably isn't smart enough to migrate processes back into memory on the CPUs they run on once they have been pushed out by something else.

I don't know the details but that's probably why thing ran slower until you rebooted.

Chris

That's probably what is happening. It's a case of the program and/or the system administrator not tuning numa behaviour.

https://sitano.github.io/2014/08/20/numa-swap/