I have 4GiB swapfiles on the cluster nodes I manage, which have 512GiB RAM.

It's hardly used, around 300MiB at present, probably things like the mail daemon. It's been useful to have a very slow node, which I can SSH into (after 10 minutes) and kill a chosen process, rather than a dead/OOMed node. But I think the difference is marginal, and perhaps 512MiB would have been a more appropriate size for the partition.

(Swappiness is set to 1.)

We run without swap, and haven't noticed an issue. Did you see any practical benefit to running swap vs not?

Other than a couple of occasions where things have run slowly, giving me time to kill the process I choose, I've not noticed any benefit.

Picking a random machine, there is 600MB of swap used. "top" shows where about 50MB is used (Hadoop daemons, systemd bits) but I don't know what the rest is. I guess it could backfire, since logind is swapped out, and I might want to log in on the serial console if the machine is very busy.