smartd[18500]: Device: /dev/sdd [SAT], 1 Currently unreadable (pending) sectors

Any hard disk error is a nightmare. It’s even more interesting when you have no ideea what happened.

The drive in question (/dev/sdd) is set up with another one in RAID 1. The error appeared while mdadm was running its monthly consistency check. Basically mdadm was checking if both drives have the same data by reading and comparing bytes, and at one point it got an error from /dev/sdd but it continued along instead of kicking the drive out of the raid array.
Then smartd kept on screaming the line above in the logs for every 30 minutes.

What does it mean?
Well… the hard disk drive now has 1 unreadable (pending) sector. This means there’s probably a bad block on the hdd, but it hasn’t been yet remapped. The bad block was detected by the hdd when mdadm tried to read it, but didn’t take any further action. A bad block is remapped only when a write happens.

Is the hdd failing soon?
Maybe. Let’s just say the chances are a little higher.
You should make sure you do have a backup at this point, and to be prepared to replace the hdd at any time.
On the other hand, the hdd may function very well another couple of years. But you decide how precious is your data.

Can I get rid of the error?
Yes, by forcing the hdd to write on the bad block, it will re-assign it to a spare block, if there are any left.

There are probably a few ways to achieve this:
1) Format the whole hdd, but make you use the option to overwrite or zero the data – this means no quick format.
2) Use the tool badbocks, run it with “-n” parameter (non-destructive read-write mode), any filesystem needs to be unmounted beforehand. badblocks will read and the write the same data on the hdd, block by block.
3) If you want to do this with no downtime and the hdd is part of a raid array that can sustain one failed drive there’s another way:
- run badblocks on the hdd (read only mode) with -v (verbose, will display percent of progress) and watch the logs to see if you catch at what percent do you get any media error from the hdd
- stop badblocks once you get the error
- optional – mdadm: you can enable internal bitmaps, it will make re-adding the drive later very fast compared to complete syncing (at least on RAID 1), check the man page
- mdadm: fail the drive, then remove it from the array
- run badblocks, using non-destructive read-write mode, you can specify which block portion to go trough – this is where you can calculate an approximate position from the percent obtained above
- badblocks should finish faster this way, then running it through the whole drive
- re-add the drive to the raid array
- mdadm: you can disable internal bitmaps if you don’t want it anymore

What to check for?
Run “smartctl -a” on the drive. Check the SMART attribute table, look for the row “Current_Pending_Sector”, and the corresponding value under “RAW_VALUE”.
That’s the number of the current pending sectors, it is 1 when you first get the error. And then it goes back to 0 if you manage to remap the bad block.

There are no explicit commands given here, you should be reading the man pages of all the tols involved and make sure you understand what are you doing and why.

Why use InnoDB?
I am not going to start debating this issue, each database environment has it’s own unique needs. If you are switching from MyISAM to InnoDB make sure you know why you are doing this, and what are you hoping to gain. For example I badly needed row-level locking because most of my queries kept waiting by the table locking in MyISAM.

I recommend reading the following articles:
MySQL Performance Blog – Should you move from MyISAM to Innodb ?
MySQL Performance Blog – Moving from MyISAM to Innodb or XtraDB. Basics
MySQL Peformance Blog – Innodb Performance Optimization Basics
MySQL – Restrictions on InnoDB Tables
MySQL – InnoDB Performance Tuning Tips

Conversion
There are 2 ways to handle the conversion:
1) run “ALTER TABLE t ENGINE=InnoDB”
2) dump the database using mysqldump, edit the file and change the “engine” line to use InnoDB and re-import it back, the table will be re-created using InnoDB

I’ll go with the first method, it should theoretically be faster, but you should do you own tests.

The conversion should be very fast for small tables (less than 100.000 rows). But you need to do some planning if you need to convert larger tables.
I recommend first doing a test run of the conversion process on a server which is not in production or actively used, using a recent copy/backup of the live database. Selects are going to work doing the conversion process, but updates/inserts are going to be blocked or may time out depending on your application.

My Story
My main issue was a table with 16M rows. I started by running the conversion in a testing environment, and it took around 6 hours – a pretty long time but considered acceptable since I could run it during night and the downtime should not be noticed by a lot of users.

Before starting the conversion on the live server, I have also applied the following MySQL server parameters:
innodb_buffer_pool_size = 1536M (this should be less than 80% of the available memory)
innodb_log_file_size = 128M (beware that in order to change this parameter, you need to delete/move the log files or mysql won’t start)
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_flush_method = O_DIRECT
innodb_doublewrite = 0
And don’t forget to remove these temporary parameters after you’re finished with the table conversion.

The final table conversion, using these parameters, ran for 1 hour and 11 minutes. I have no idea if these parameters account for such a large change in time (from 6h to 1h), or is just the fact that the live server had more memory available (the old myisam data files were also prefetched into memory by running dd on the .MYD and .MYI files a couple of times).

I will not go into gettext usage details, there are enough resources available for that.

Official PHP gettext documentation:
http://www.php.net/manual/en/book.gettext.php
A nice “gettext introduction” and well-done article:
http://mel.melaxis.com/devblog/2005/08/06/localizing-php-web-sites-using-gettext/
Some benchmarks:
http://mel.melaxis.com/devblog/2006/04/10/benchmarking-php-localization-is-gettext-fast-enough/

And now let’s get back to what I actually wanted to talk about. It’s a well known problem that new or updated translated strings don’t appear immediately on your PHP page. Or to say the truth – they don’t come up at all. This happens only if you are running PHP as a module (mod_php, …) inside your webserver (Apache, etc), it does not affect php scripts run as CGI.

Why does it happen? The short answer is – because of caching.
The first time you are initializing a text domain (loading a compiled binary .MO file), the file is loaded and cached in memory. Performance-wise, this is a very good thing, the translation string lookups run very fast and not needing to check the filesystem again and again helps a lot with the speed.
The disadvantage is that if you update the .MO file or even delete it, gettext will not catch the change, it would still run with the file from the cache. Very irritating, especially in a development environment.

The recommended and most well-known solution is to restart the web server – this will restart the PHP module with its gettext extension and the cache will be cleared. Unfortunately restarting the web server is not always something that can be done easily, or we might not even have the privileges to do it if we’re on a shared hosting.

I came up with a workaround, that does not require fiddling at all with the web server, it’s all done through PHP. Basically, what we do, is make sure that we are always using the last updated .MO file by using copies of it. We check for modifications by checking the file modification time using filemtime and then we make sure that’s the one we’re using by binding to a unique text domain and a filename computed from the file modification time. If that file exists (from a previous run), then that’s all. But if it doesn’t exists, then create it by copying the original .MO file. So, what we are actually doing is activating a new unique translation domain for each and every time the .MO file is modified.

What you need to consider:

• you need write permissions in the locale folder
• if you update the .MO file many time, then a lot of garbage temporary .MO files are generated – you need to remember to clear the folder once a month or something like that
• is meant to be used in a development environment only, the filesystem checks are not benchmarked (even though PHP filestat caching may help here) and may be too expensive on a production server.

Finally, the example code is:

// settings you may want to change
$locale = "en_US";  // the locale you want
$locales_root = "locales";  // locales directory
$domain = "default"; // the domain you're using, this is the .PO/.MO file name without the extension

// activate the locale setting
setlocale(LC_ALL, $locale);
setlocale(LC_TIME, $locale);
putenv("LANG=$locale");
// path to the .MO file that we should monitor
$filename = "$locales_root/$locale/LC_MESSAGES/$domain.mo";
$mtime = filemtime($filename); // check its modification time
// our new unique .MO file
$filename_new = "$locales_root/$locale/LC_MESSAGES/{$domain}_{$mtime}.mo"; 

if (!file_exists($filename_new)) {  // check if we have created it before
      // if not, create it now, by copying the original
      copy($filename,$filename_new);
}
// compute the new domain name
$domain_new = "{$domain}_{$mtime}";
// bind it
bindtextdomain($domain_new,$locales_root);
// then activate it
textdomain($domain_new);
// all done

On the old server I was using qmail+vpopmail+courier.
qmail – smtp, pop3 server
vpopmail – provides mailbox & virtual domain management
courier – for the imap server part

I got tired of all the patching mess in qmail. The only way to continue using qmail was to copy the already patched version, because I couldn’t recall all the patches and hand changes I have done over time. Yeah, I even modified its source code myself to fix some small things, that’s the best part of it – qmail is composed of very very small programs and have nice clean source code, easy to understand and follow. But you get tired after a while and you want something that can do more and just work. So, here I am now using postfix+dovecot:
postfix – smtp server
dovecot – pop3+imap server, mailbox & virtual domain management

I recreated all the emails and domains in dovecot, copied the Maildir directories, set the old server to forward all emails to the new one just in case someone still sent email to the old MX entry, and that was all. 0 second downtime and nobody noticed a thing.

I had a great howto/guide for installing Postfix+Dovecot:
http://johnny.chadda.se/2007/04/15/mail-server-howto-postfix-and-dovecot-with-mysql-and-tlsssl-postgrey-and-dspam/

One thing to check if using that howto as a base, a friend that has an account and uses Outlook Express couldn’t send mail at all. I then figured out that OE would not want to authenticate at all with the smtp server, I have then found out that OE will not login if the only auth method advertised by the smtp server is “PLAIN”, even though everything is under TLS. After testing different auth methods, it seemed to be ok if “LOGIN” it’s on. So, just edit doveconf.conf and make sure you have both PLAIN and LOGIN activated:

doveconf.conf :

...
auth default {
  mechanisms = plain login
  ...
  }
...


That’s all for now.

Why are all the cables on left/right side, it must be really cute having cables going out of the laptop on the left on your desk, what a mess.
Why are the audio jacks in front, now really, why would you want to have cables in front of your laptop.
Who killed the pointing stick? I loved it on my old IBM ThinkPad, it was so precise. But no, no, no, the touchpad is the way to go. And if you somehow really want a pointing stick… they have it hidden away in one of the “premium” models.

Where is the world heading to…
at least my Dell Inspiron 6400 managed to get some things right

Trying to debug the problem, I added a 1px red border to all div’s to check correct positioning. So, there was supposed to be 4 red squares, the div’s were each exactly 9 x 9 px. But instead of that, Internet Explorer would display 4 red boxes having their height waaaay larger than 9px. The div border did not appear as a square even though its css specified exactly “width: 9px; height:9px“. That was pretty nice for the browser to ignore my css code.

After some researching, found about someone who stumbled over the same problem:
http://www.zucchetti.co.uk/?p=75

It seems it’s actually a IE-only problem. When rendering an empty div, Internet Explorer will use the current font size as its height. The fix is as simple as setting the font size to 0 by using “font-size:0“, getting rid of the minimum div height being the font-size.