Ask Bjørn Hansen

JavaScript: The Good Parts

2008-05-17T00:11:37Z

Douglas Crockford of JSON fame has written a beautiful book about JavaScript.

First of all - at only 170 pages it is short. Even though some of the key points are repeated through the book it's dense with information. You don't need any JavaScript experience, but it's not a "beginning programming" book so if you haven't been programming before this is not the right book for you.

Reading this book a couple of times will give you an appreciation for the JavaScript language that you almost certainly didn't have before. It'll give you tools to write better programs that you and others will actually be able to maintain over time.

I've learned lots of little things that I maybe knew from experience, but now I know and I know why.

This book will help you battle with JavaScript rather than against it.

(this review was also posted on amazon.com)

After reading half the book I went and bought a bunch of extra copies and had them sent to people I work with who are working with JavaScript.

Slides from "Real World Web: Performance & Scalability, MySQL Edition"

2008-04-15T19:27:13Z

The slides from my tutorial yesterday are up at develooper.com. 189 slides this year! A handful of people have told me today (and more yesterday obviously) that they learned a lot and got new ideas for how to build things - yay!

I’m considering putting up a version of the slides with sound. Would anyone want a 3 hour quicktime movie of that?

Well, maybe split up into smaller bits, but you get the idea. You wouldn’t get the hand waving, but you would get a bunch more detail, obviously.

I actually had audio recorded, but I haven’t checked how it came out, yet, and I’d have to make a synchronized movie version of the slides (and likely be tempted to just redo the audio anyway). Thoughts? Would it be a worthwhile effort?

Dash Express: Living in the future

2008-04-13T15:12:34Z

Nelson mentioned the marvel of checking traffic on the iPhone. Nelson, I'm sorry - that's so last months future. Freeway traffic flows? Boring! I've only driven about 5000 miles the last year, and almost none of it were on freeways.

A couple weeks ago I ordered a Dash Express. Let me tell you: that is the future. The essence of the Dash is that it tracks your speed as your drive and uploads it to the Dash servers (anonymously, supposedly - they don't explain what they do to keep it that way) as you drive. Of course it also downloads traffic data from other drivers and historical data when no Dash owner has driven on a particular street recently.

It is so cool to watch. Entirely accurate? Not yet. Pretty darn good already? Check. A glimmer of the future? Definitely.

The Dash has wifi, but most of the time it uses GPRS for communication via Jasper Wireless. As a user you don't know that or even care, but as a geek it was fun to find out that there's a wireless carrier that doesn't do anything else than GPRS for mobile gadgets.

Since it's always connected they have built in local search via Yahoo local (not as good as YellowBot, of course, but pretty neat all the same ;-) ). They're working on giving you more "online data". For example there are gas prices and movie showtimes available, but the UI for that isn't very useful.

Dash are suggesting that the Dash Express will be the first GPS for "daily use" rather than just to be used when you are going somewhere new. it suggests multiple routes with distance and time based on current traffic. Pretty darn neat; but as I mentioned earlier - at least for my short-ish city-only stretches it's still not entirely accurate. I'm sure they'll improve on it though. The wonder of automatic software updates and constantly improved traffic data. I've noticed a big improvement in available traffic information just over the last two weeks since they started selling units to non-beta testers.

So to get better data it is of course in my interest to tell you to go buy one. In particular if you live in Los Angeles. :-)

If you are driving a lot and a lot to new places where you don't know the regular flow of traffic then I'm sure it's worth getting the first generation unit.

However, there's a lot to improve before I'm going to recommend it to "regular drivers" and non-early-adopter types. The physical form factor is, well, not really sleek. It's GIGANTIC compared to a modern Garmin Nuvi. They say it's to have room for the wifi and GPRS antennas (huh? The iPhone fits both a fraction of the space) and for the battery (which still only lasts a couple of hours; keep that 12v outlet available). The screen is a well working touch-screen, but the "hardware buttons" on top are crazy annoying when it's not mounted in the window or on the dash board.

Speaking of the screen - My first unit had a slightly defective screen, but Dash were great about getting it fixed and I could see that they had already improved their process on the replacement unit so it won't be an issue again. My second unit has been flawless so far.

Other than the issues mentioned above, it's mostly software improvements that are missing and those will hopefully be updated over time. It's done a bad job telling us a quick sequence of directions on freeways. It doesn't zoom in to show how clearly to navigate intersections or freeway interchanges. The UI is somewhat inconsistent (for example some buttons disappear when not usable, others stay on the screen and just "don't work"). It seems very slow at updating the screen at times. As I mentioned above the UI needs some work for some of the features to work better. Etc etc.

All that being said though - if you are in the market for a new GPS, give it a serious look. It's lots of fun and I trust the software will get better month for month. It will be interesting to see if or when Garmin catches up with the traffic data system or if one of the big players just goes and buys Dash. A device built with the Dash traffic data and with Garmins long UI, navigation and hardware expertise: drool. I guess if the Dash is the future; then that device is the future of the future.

Dash has a great help section that answers lots of questions in addition to their general feature overview.

Tax season

2008-04-13T03:35:18Z

I was preparing to do our taxes last week. Every year in late March or early April I realize that there's an account or two (or three) that I never got around to reconcile or maybe even download activity for, so lots of tedious work ensues.

Vani's been scanning lots and lots of documents into PDF files; I can't recommend getting a scanner enough.
I use QuickBooks for my corporate accounting and like I only sheepishly admit that I really like the new Radiohead album I'm also reluctant to admit that QuickBooks works pretty darn well for me.
Personal finance software though. Geez. Is Moneydance really the leading Quicken alternative? I tried it briefly and it seemed even more painful to use and I use a ~3 year old Mac version of Quicken!
Some of the banks are really awesome with letting you download activity from a long time back and having the statements in easily downloaded PDF files. Bank of America (who I otherwise generally loathe) and American Express are in this category. Chase (Amazon Visa) are notoriously not. Oh man, don't get me started on Chase.

Reclaim space from a sparse OS X disk image

2008-03-26T00:42:43Z

In 10.5 and 10.5.1 there was a bug that’d make my iDisk explode to take all available space, yikes. Fortunately that has been fixed now (not that I use the iDisk much anymore - hello JungleDisk!).

It doesn’t seem like the iDisk disk images get compacted automatically to take less space if you delete files, so what I learned from the bug is generally useful.

You can compact a “sparse” disk image manually from the Terminal application.

The iDisk can’t be in use, so close any files opened on it. Then open a terminal and run:

hdiutil unmount ~/Library/FileSync/*/*.sparsebundle
hdiutil compact ~/Library/FileSync/*/*.sparsebundle

The hdiutil compact feature also works on old fashioned sparse disk images (all data in one file versus the “sparsebundles” that are really directories with lots of smaller files). I used it to reclaim a few gigabytes from some old disk images, yay.

MySQL conference tutorial almost sold out

2008-03-07T01:25:04Z

It looks like my tutorial is almost sold out (and so are several of the others already!)

So if you're planning to go to the MySQL conference, don't delay - go sign up!

Servers are too fast!

2008-03-04T23:44:29Z

We got a couple of new servers at Solfo recently which showed me one of the reasons virtualization is so popular now: Servers are too fast!

The "standard issue" CPU is now a quad-2.5GHz CPU, so in each server we have 20 GHZ CPU and 32GB ram (at less than $50 per gigabyte it's too cheap to not just fill it up and be done upgrading). Just a few years ago the CPUs we were getting were "only" dual 2GHz, for ~8GHz CPU per box. That's a big increase!

In each "tier" of the application (app servers, db servers, search servers) our main reason for having more than one or two servers is redundancy / high availability - never lack of CPU and rarely because we need more memory.

Here's from one of our webservers (virtualized with Xen with 6 of the 8 CPUs on the "real" hardware).

The big exception is the MySQL servers where we get constrained by I/O so we need a single chassis with lots of disks (the $$ version would be to get external enclosures or SAN boxes with disks) and of course in the MySQL servers we can easily use all 32GB ram.

Anyway, the conclusion: Please give us cheaper, lower power CPUs. More memory, sure - we'll figure out to use it. More disk I/O: yes, please! More more more! Faster CPUs makes sense at scale, of course, but for a smaller website with just a handful of million visitors a month we just can't make a dent in the available CPU. Maybe if we used Ruby instead of Perl. ;-) (just kidding).

(I realize that AMD and Intel makes plenty "slow" CPUs, but they don't come in server boxes with fast I/O and all that).

When you program open source, ...

2008-02-24T21:59:23Z

Flickr sure has some passionate users!

10000+ tables in one MySQL database

2008-02-15T03:57:46Z

Once in a while I hear people talking about using thousands or tens of thousands of tables in one MySQL database and often how it “doesn’t work”. There are two things to say to them:

Are you nuts?!
Sure it works

I’ll elaborate a little on both …

Why’d you do that? Are you nuts?!

In most cases when extraordinarily many tables are brought up the “correct answer” is: Fix your schema to not duplicate the same table layout for each customer/user/site/…!

Once in a while though there are good reasons to have ~~way too many~~ lots of tables. Even “takes too long to fix the application now” can be a good enough answer sometimes.

My use case was to use the many tables as an extra “index” for a situation that a regular index couldn’t cover. Tens of thousands of tables, here we come.

Sure it works, just configure it right!

… which is easy!

In your startup script (/etc/init.d/mysql with the MySQL RPMs), add “ulimit -n 30000” somewhere near the top to allow mysqld to have 30000 open files (and sockets).
In your mysql configuration, add something like the following - adjust the number as appropriate.

set-variable = table_cache=12000

This will let MySQL keep up to 12000 tables open. The default limit is much too low and the system will spend all its time closing and opening the files. The other few thousand handles are free for database connections or whatever. You surely can tune the numbers more, but I haven’t needed to be more specific yet. MySQL uses two filehandles per open table (for MyISAM, it depends on the table type…)

Flushing the tables

One curious thing you’ll run into is that MySQL can take forever (read: hours!) flushing thousands of tables that have been changed if you do it with a simple “flush tables” or when you are shutting down mysql. That’s of course not “hmn, how curious” but rather insanely frustrating if you were going for a quick restart. This occasionally seems to happen with InnoDB tables too, but in particular with our large MyISAM system (we use fulltext indexes, hence MyISAM) this is a big issue.

With MyISAM tables, you also have a lot of cleaning up to do if the system crashes one way or another with thousands of tables open.

There’s an easy-ish solution to both these problems, though! Just flush the tables individually before the shutdown command and on a regular basis to mitigate the issue if it crashes. Remember the system or MySQL can crash for all sorts of reasons. Recently we had a motherboard with a BIOS or hardware bug that made it unstable after adding more than 16GB memory. On another box, one of the memory sticks went bad so suddenly it came up with 4GB memory less than previously. With MySQL carefully tuned to use all the memory (it was an InnoDB-only installation) it’d try using all 24GB memory and get killed by the kernel when it got above 20! Yeah, that one took some head scratching before I figured it out.

Below is a small program that’ll go through the database twice flushing all tables in all databases and then end with a regular “flush all”. We go through the database twice just in case the first flush took so long that a lot of tables got opened again. Two works for me, depending on your needs you might make it an option. Or a fancy version would check how many open tables are at the end of the run and go through it again if too many are open (and abort with an error if they open faster than they can get closed!). The final “flush tables” worked for me in getting everything closed just before the script exits (and the shutdown or whatever starts).

]]>

#!/usr/bin/perl
use strict;
use warnings;
use Getopt::Long qw(GetOptions);
use DBI;

my %args = (
            verbose => 0,
            user     => $ENV{USER},
            hostname  => '127.0.0.1',
            port     => 3306
           );

GetOptions(\%args,
           'verbose!',
           'user=s',
           'password=s',
           'hostname=s',
           'port=i',
           'database=s@'
          );


my $dbh = DBI->connect("dbi:mysql:hostname=$args{hostname};port=$args{port}",
                        $args{user}, $args{password})
  or die DBI->errstr;

my @dbs = $dbh->func('_ListDBs');

for my $db (@dbs) {
  next if uc($db) eq 'INFORMATION_SCHEMA';
  $dbh->do("use $db");
  my $tables = $dbh->selectcol_arrayref("show tables");
  for (1..2) {
    for my $table (@$tables) {
      print "flushing $db.$table\n";
      $dbh->do("flush table $table");
    }
  }
}

print "flushing all\n";
$dbh->do("flush tables");

Real World Web: Performance & Scalability, MySQL Edition

2008-02-14T16:27:20Z

In exactly two months - April 14th - I'm doing a scalability tutorial at the MySQL conference.

It'll, of course, be based on the material I've been throwing around since 2001, but once again it's greatly expanded, extended and improved! It should be lots of fun.

I've yet to hear anyone tell me that they wasted their time with this particular session and I don't think people are just being polite because I'm often told that people find it tremendously useful. So go sign up already. And don't pay any attention to Brian Aker and dormando's shiny memcached tutorial going on at the same time. I promise I'll cover memcached, too. :-) Speaking of Alan Kasindorf (aka Dormando) he had a most excellent post about how to run your systems recently!

O'Reilly sent me a discount code to hand out to "friends and colleagues". I'm not sure what they mean by that. Can I post it here? If you read my drivel, surely you must be a friend. If you are interested in the session in the first place, surely you are some kind of colleague. To be safe however I won't post it here. If you want 20% off for the MySQL Conference send me an email.

What's the point of blade servers?

2008-01-21T21:44:26Z

There's a news story today about Dell, IBM and HP all working on or having released new blade servers. Huh? What's the point of those at this point?

You can easily (easily) use up all the power and air conditioning available in most facilities with "regular" servers. Why pay more and get a proprietary form factor etc? Are there enough corporate customers who have plenty AC and power available in a small computer room? Or is it something else?

YellowBot at Twiistup 3

2008-01-17T07:59:55Z

YellowBot was presenting at Twiistup 3 today. It was a-awesome. We told about two billion people about a site and gave many of them a quick tour - and gave out a lot of t-shirts and a big box of delicious cup-cakes. :-)

Now (1am): So Tired.

Avoid the "top of the minute problem" - space out your cronjobs

2007-11-19T17:10:11Z

On Unix the universal job scheduler is "cron" (the other universal but less commonly known daemon is the at daemon). On Unix systems everywhere, lots and lots of batch jobs are kicked off all day from the cron daemon. The typical configuration looks something like this:

18 2 * * * ~/bin/some_batch_job

This means at 18 minutes past the hour at 2am on every day of the month of every month of every weekday, run the following command (You can read about the configuration format by typing man 5 crontab on the command line of a nearby unix-y system).

Why 18 minutes past the hour? For an hourly job you often want to pick a "semi random" time of the hour to run the job. If you run the same batch job on several servers all accessing a shared resource for example.

Sometimes though then the configuration has to be exactly the same everywhere, but you still want to spread out the runs. Or it's a job that runs every minute or two on many systems. Or you have a system that just runs a lot of cronjobs and you can't have them all start at exactly the same second. You have to introduce second granularity to spread them out. But how? The cron format works with minutes at the lowest granularity.

You use the magic $RANDOM variable in the shell!

You can get a random number by just accessing $RANDOM, try it:

echo $RANDOM

If you want a number between 1 and 10 you do a modulo operation on the random variable. In shell syntax that'd be

echo $(expr $RANDOM % 10)

The $( ... ) syntax works like backticks, runs the command and returns the result. The expr command runs the arguments as a shell expression.

Now if you want to run a command (on average) every two minutes, but spread out the executions over 90 seconds, you can do

*/2 * * * * sleep $(expr $RANDOM \% 90); ~/bin/batch_job

This will run the cronjob every 2 minutes (120 seconds), but wait between 0 and 89 seconds before running the actual script. Magic!

Whenever you setup something to access a remote resource (RSS feed, NNTP server, ...) it's always polite to do this, otherwise the server operator will get an onslaught of requests exactly on the minute -- or even worse at the top of the hour.

Selectively silence the volume adjustment "click"

2007-11-18T00:09:14Z

By default in OS X when you hit the volume keys on the keyboard the OS X gives a little “click”. In 10.5 there’s an option to disable it (Sound Preferences, “Play feedback when volume is changed”. Is that option in 10.4 too?). However, if you usually want it on you can still selectively make the adjustment silent.

Just hold down the shift key while pressing the volume button. I think this is a new feature in 10.5, but I’m not sure - I stumbled over it accidentally (accidentally holding down random keys - amazingly I ever get the computer to do what I want!). It works on the “real” keyboard on my PowerMac, but not on the internal keyboard on the MacBook Pro - I haven’t tried with an external keyboard on the laptop yet.

Does it work for you?

Army desertion rate up 80 pct. since '03

2007-11-17T21:14:25Z

This must be the craziest spin I've seen in a while:

"We're asking a lot of soldiers these days," said Roy Wallace, director of plans and resources for Army personnel. "They're humans. They have all sorts of issues back home and other places like that. So, I'm sure it has to do with the stress of being a soldier."

So they're deserting to go home because of their "issues" there? Uh-huh, sure ...