MySQL DNS Problem

The call came in a little after 3AM - the site is down. Stumbled down to the office and dollars to doughnuts bringing up the site in a browser resulted in a spartan yet blood pressure-raising "Server Error" message. My first inking was to suspect there was a problem connecting to the database server and that was confirmed - "Too many connections". Mysqld wouldn't politely shutdown so it had to be killed. At this point you could copy and paste this blog post here as the exact same thing was happening - mysqladmin processlist was exploding with "unauthenticated user" messages. Reassuringly the client's IP addresses were all the web servers' so I don't believe it was some sort of denial of service attack. I followed the advice in said blog and added entries in /etc/hosts for each of the web servers. Restarting mysqld after doing so brought everything back to a working state - whew! Why this happened at this particular moment though is still a mystery and a little troubling as who knows when it may happen again.

Coincidentally earlier that day CERT issued a warning that some DNS implementations are vulnerable to cache poisoning. In the back of my mind I feared a world wide DNS exploit was in effect and our poor mysql server was a victim. So far it seems that was not the case.


Further reading
MySQL DNS Details
http://hackmysql.com/dns

MySQL unauthenticated login pile-up
http://rackerhacker.com/2007/08/16/mysql-unauthenticated-login-pile-up/

Multiple DNS implementations vulnerable to cache poisoning
http://www.kb.cert.org/vuls/id/800113

Stalled MySQL Logins
http://www.paperplanes.de/archives/2008/5/20/stalled_mysql_logins/

Bug #2814 multiple connections, database locking up.
http://bugs.mysql.com/bug.php?id=2814