As you might know the system allocates memory on a server by contiguous memory blocks of different size. And there may be a situation where the system does not have contiguous memory blocks of certain size due to memory fragmentation and another service(i.e. VM) requires memory block of that very size. In such situation the service will try using SWAP memory instead of RAM. As a result you can witness a situation where RAM has enough ‘free’ memory but the service will use SWAP instead. And if Swap memory is insufficient the VM can be killed by OOM

For more information about this please refer here:
https://stackoverflow.com/questions/46464785/cannot-allocate-memory-despite-free-reporting-available
https://forum.proxmox.com/threads/memory-allocation-failure.41441/

Here is memory fragmentation explained itself:
https://stackoverflow.com/questions/3770457/what-is-memory-fragmentation

Please, try the below:
To ‘defrag’ the RAM run this command:
echo 1 > /proc/sys/vm/compact_memory

When 1 is written to the file, all zones are compacted such that free memory is available in contiguous blocks where possible.

Then try to tune vm.swappiness in /etc/sysctl.conf:
vm.swappiness = 0

To prevent this issue in the future:
Increase vm.min_free_kbytes in the file /etc/sysctl.d/99-vm.min_free_kbytes.conf
The value should be a multiple of 4096.
You can start at about 5% of total RAM divided by a number of CPU cores.