Blame it on sendmail

A new day, a new challenge: recently automated emails from some webservers just stopped arriving. The webservers are (almost) identical configuration, load balanced, hosting the same sites. You should also know, mail destined for other domains was getting through just fine (new user registration confirmations), but mail going to our own domain was not getting through (support form emails, internal notifications).

The first place to look was maillog:

Jul 17 10:15:35 www3 sendmail[25808]: m6HHCt7S025798: to=, ctladdr= (500/404), delay=00:02:20, xdelay=00:02:20, mailer=esmtp, pri=120342, relay=mail.example.com. [10.0.0.22], dsn=4.0.0, stat=Deferred: Connection timed out with mail.example.com.

That seemed strange because mail.example.com is in the same LAN as www3 and is pingable. I could telnet to port 25 of mail.example.com and get a message through that way. After hitting some dead ends (and wrong ends) messing with sendmail configuration files the problem presented itself to me. It was a DNS problem. Now, I explicitly defined the mail server with its internal IP address in /etc/hosts, but it seems sendmail was ignoring /etc/hosts and consulting a name server and getting the (external) IP address of said mail server. The mail server is not reachable by its external IP to hosts in the internal network, hence the time outs. A little modification of /etc/resolv.conf to add an internal name server, and I don't know if this was absolutely necessary, but a restart of the sendmail service, and messages started flowing. Inboxes started exploding with queued mail from the last five days. I'm sure there is a way to configure sendmail to use /etc/hosts but frankly editing sendmail configuration files scares me so I'm leaving well enough alone.

Websites that make you pick your country must die!

plugin geoIP or something - for fsck's sake man it's not rocket science.

Die canon.com die!

eat my bizzalls fedex.com!

ups.com you have no chance to survive!

suck it intel.com (also sites with flash intros must be destroyed!) - ditto linksys.com

MySQL DNS Problem

The call came in a little after 3AM - the site is down. Stumbled down to the office and dollars to doughnuts bringing up the site in a browser resulted in a spartan yet blood pressure-raising "Server Error" message. My first inking was to suspect there was a problem connecting to the database server and that was confirmed - "Too many connections". Mysqld wouldn't politely shutdown so it had to be killed. At this point you could copy and paste this blog post here as the exact same thing was happening - mysqladmin processlist was exploding with "unauthenticated user" messages. Reassuringly the client's IP addresses were all the web servers' so I don't believe it was some sort of denial of service attack. I followed the advice in said blog and added entries in /etc/hosts for each of the web servers. Restarting mysqld after doing so brought everything back to a working state - whew! Why this happened at this particular moment though is still a mystery and a little troubling as who knows when it may happen again.

Coincidentally earlier that day CERT issued a warning that some DNS implementations are vulnerable to cache poisoning. In the back of my mind I feared a world wide DNS exploit was in effect and our poor mysql server was a victim. So far it seems that was not the case.


Further reading
MySQL DNS Details
http://hackmysql.com/dns

MySQL unauthenticated login pile-up
http://rackerhacker.com/2007/08/16/mysql-unauthenticated-login-pile-up/

Multiple DNS implementations vulnerable to cache poisoning
http://www.kb.cert.org/vuls/id/800113

Stalled MySQL Logins
http://www.paperplanes.de/archives/2008/5/20/stalled_mysql_logins/

Bug #2814 multiple connections, database locking up.
http://bugs.mysql.com/bug.php?id=2814