DRBD and MySQL

| 1 Comment
Yesterday Phillip Reisner was giving an introduction to DRBD. I won't get into it here, but essentially it's a way to keep blockdevices on two different hosts in sync to get a kind of shared disk. (Go read about it on their site!)

I've used it for a while to get high-availability from two NFS servers and I always recommend it in my scalability talk. It's awesome. (Heartbeat which is the "fail-over" software in the typical configuration isn't so great, more on that another time).

Anyway, being at the MySQL conference the context of course is "how can this be used for MySQL". Usually I prefer a simpler master-master replication setup for redundancy. Later in the afternoon Mats Kindahl (one of the replication developers) pointed out how the drbd setup is great for having a highly-available master in a more typical master-slave(*N) setup.

It's funny, just as I write this Paul Tuckfield from YouTube is giving a keynote and telling about the complicated stuff they were doing to deal with failure of their master in their replication system. :-)

If they used drbd the redundant masters would "share the disks" (but you have the content duplicated so there isn't even a disk SPOF). When the primary master fails, the second one would just start up and all the replication slaves wouldn't know the difference. Magic!

1 Comment

DRBD is pretty cool, I havent used it but I did the same thing with different technology on oracle.

One problem with this scenario, is that InnoDB must do what amounts to a crash recovery when it's started up on the failover node. This works, except at the update rates we were getting, we could not get innodb to crash recover reasonably fast. I found that (similarly to oracle and other databases) Various methods of reducing this recovery time come a the tradeoff of database writeback caching performance on OLTP-ish workloads. If blocks are checkpointed lazily, often the db can write back a databases block once for several changes made over a time period, whereas with more aggressive checkpointing it would have done multiple writes in that interval. So lazy results in less IO bandwidth needed during normal operation, whereas aggressive results in less work to do (if) you crash. we took the performance tradeoff for better performance during workloads rather than performance during a crash recovery.

our failover solution was an otherwise idle replica, which (like all our replicas) uses a script to make replication get better cache hits by pre-caching entries found in the relay log on the slave. This replica always kept up. It was the replicas serving traffic that were difficult to tune keep reasonably in sync: we'd find ways to keep up but always be overwhemled by more user demand in a see-saw battle.

Leave a comment

About this Entry

This page contains a single entry by Ask Bjørn Hansen published on April 26, 2007 9:32 AM.

Find recent content on the main index or look in the archives to find all content.

Pages

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.33-en
/* bf */