We are looking for an expert to review our setup and provide recommendations to optimize and increase greater stability.
This is our setup and symptoms:
We have a Proxmox cluster consisting of three Proxmox nodes - 2 Dell R710s and 1 Dell 2950 - and a storage cluster consisting of two Gluster nodes - both Dell 2950s. We used to have the recommended 3 nodes in the Gluster array, but performance was absolutely terrible, and removing one of those nodes solved that problem.
We have a problem where, under certain circumstances if our Gluster array gets busy, all of our VMs cease to work, or at best they work extremely slowly. On our Proxmox cluster, we are hosting 1 VM each for our website, syslog server, IMAP, SMTP, MySQL, MX1 and MX2, Asterisk, and our billing database. The IMAP, SMTP, MX and VoIP servers rely on the the MySQL server, and as far as I can tell, this reliance on the MySQL server will cause all of these servers to time out.
We need our cluster to be reliable even when the shared Gluster disk gets very busy.