MySQL problems
Since a lot of readers seem to want more war stories, here’s one that happened tonight. About three hours ago, any person that tried to go to MyListo got a MySQL database error: “SQLSTATE[08004] [1040] Too many connections”.
The first thing we tried to do was restart our production web and database servers. EC2 provides tools that allow you to do this through the command line.
After restarting, we got a different MySQL error: “SQLSTATE[HY000] [2013] Lost connection to MySQL server at ‘reading initial communication packet”. What the heck does that mean? Perhaps the MySQL isn’t running on the db machine?
So then we login to the database machine and try to restart MySQL. Sadly, another error: “/etc/init.d/mysql: ERROR: The partition with /var/lib/mysql is too full!”.
At least we’re getting somewhere now. The results of running “df -h” confirm that one of our partitions was completely full:
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 9.9G 9.4G 0 100% /
varrun 851M 164K 850M 1% /var/run
varlock 851M 0 851M 0% /var/lock
udev 851M 16K 851M 1% /dev
devshm 851M 0 851M 0% /dev/shm
/dev/sda2 147G 188M 140G 1% /mnt
overflow 1.0M 0 1.0M 0% /tmp
Everything returns to normal after deleting some unneccesary files to free up space.
You’re probably wondering why our server only had 10 GB of space. No, we’re not running some old school Pentium II. We’re using Amazon EC2 and it turns out that the small instances only have 10 GB of permanent storage.
1 year ago