Error message
VixEOUTOFMEMORY: Memory allocation failed. Out of memory.
Error description
Failed VMware VM backup due to NBD limits
Beside limitations on connections count to VC/ESXi, ESXi itself also limited by a transfer buffer for all NFC connections. This limitation is enforced by the host and cannot by bypassed or to be known in advance. The sum of all NFC connection buffers to an ESXi host cannot exceed 32 MB, and by default it is configured as 16 MB.
The primary physical appliance uses NBD protocol to backup VM disks. NBD, in turn, employs the VMware network file copy (NFC) protocol and thus is a subject of aforementioned limitations.
This happened because the ESXi server could not serve the request due to lack of enough resources (NFC connection buffer).
At the time there were N failed jobs with VixEOUTOFMEMORY error you had N+1 or more simultaneously running FULL backups. All of them were backing up VMs located on a single ESXi host. We use up to 10 MB buffer to transfer data. So there is a probability of facing with NFC buffer limitation on ESXi host. Which, as you saw, happened to occur. It does not mean the probability is always 100% with parallel backup jobs. It very depends on a lot of factors.
Steps to resolve
Nothing if there are no other failed backups with VixEOUTOFMEMORY error on your appliance on consequent backups.
You can also optimize the ESXi network (NBD) performance by increasing the NFC buffer size from 16 MB to 32 MB and reducing the cache flush interval as suggested in VMware knowledge base article.
Do it on all of your ESXi hosts. You can query current values using the following commands (from ESXi host): esxcfg-advcfg -get /BufferCache/MaxCapacity and esxcfg-advcfg -get /BufferCache/FlushInterval. It will not guarantee the VixEOUTOFMEMORY error never happen again but will decrease its probability. And it seems to be a good idea in general since you perform a lot of simultaneous backups.
Consider upgrading your network to 10 GbE. That should cover every network link in the chain between the appliance and the VMware host.