Open Source. No longer hype, just business as usual

I am reading an article by Matt Asay about the position of "open source" in the "hype cycle". He asserts that 2010 has been the year that Open Source became "invisible", that it is no longer hyper-worthy news that some company is using or producing open source software, but that instead that open source has become the foundational default for infrastructure and tools software.

I think he is correct, and as a long term proponent of the "open source" project, dating back from when I was a young undergrad who had just read Stallman's manifesto, I am gratified that the concept is having been proven correct. But as professional open source advocate, and as an employee Eucalyptus Systems, a company that actually creates and distributes GPL-licenced open source infrastructure software, this makes my life more difficult.

In the past, one could "ride the wave" of the hype a bit, and take advantage of the opportunity to educate reporters and potential buyers to both evangulize the whole open-source project, and also the specific project one wants to present. That free ride for publicity is now over.

RFID door opener

I live in a geekhouse, which is shared housing situation that is more social than an old-school boarding house, but less "into each other's business" then being "family". This kind of shared housing is a model that is popular with the geeky set, especially in my neighborhood in Seattle.

One of the problems with this situation is physical access control, that is, locking and opening the front door. There are a number of people who live there, a number more who have "walk in at any time" social status, plus out of town guests and visitors, and so forth. Keeping track of who has a key and recovering guest keys is awkward at best. And rekeying the locks would be very annoying.

I've been reading about RFID, MIFARE, and the Maker revolution for some time, and have decided to actually Make something interesting and useful. I'm building a way to open the door with an RFID card and/or a biometric.

So many people immediately say "but, it could get hacked!". And my response is: a break-and-enter criminal type would just break the window, and any hypothetical black-bagging spooks would just pick the lock. So adding this will not lower the actual security, and would increase it by decreasing the actual physical keys "out there".

So I'm starting with this project. Then I will add a fingerprint reader, and then run a link back to a server with a database, either over ethernet, zigbee, or some other link. I will use a digital output line to drive a transistor that will drive a relay that will drive a standard off-the-shelf electromagnet door striker (the thing that goes BUZZ when you get buzzed in). All of this stuff is off-the-shelf from places like AdaFruit.

Then I can start doing cool stuff on the server side, such as do the access lookup via RADIUS or LDAP, so existing scalable and federating identity systems can be used, across multiple units. And set it up so people can choose to do a Foursquare checkin when they touch in.

Problems at the physical layer and at the political layer of cloud computing

I was at the Seattle CloudCamp last week, and ended up on the unpanel. An "unpanel" is the unconference version of a panel talk. Half a dozen self admited "experts" are picked from the body of participants, then the body of participants are polled for questions, and then members of the panel take turns picking a question and answering it.

One of the questions was somewhat defiantly called out by Steve Riley, the public face of AWS security. From memory, it was "what applications can NOT be moved to the cloud. Give reasons why, with data"

While I understood his point, which is that most of the applications that people think cannot be moved to the cloud, actually can be, I took the question, and with a mix of humor and defiance, answered it.

There are two kinds of apps that cannot move to "the public cloud", especially as it exists right now.

The first kind are the set of high frequency realtime applications that have a sense/react cycle faster than the networking latency to the cloud provider's datacenter. As I tried to tweet immediately after, "CPUs are getting faster, faster than the speed of light is getting faster".

And the the second kind are the set of applications that piss off people like Senator Joe Lieberman, who will call up your cloud provider's CEO to threaten him, resulting in your account getting killed and the cloud provider issuing a press release lying about the call from the Senator.

Fixing these bugs in the physical layer and in the political layer of the public cloud is going to be... challenging.

Unsavory Passwords

I was at the OpenStack Developer Summit in San Antonio last week. One of the sessions I participated in was a proposal for a design for authorization modules. I found the the session both enlightening and troubling, for a number of reasons. Here is just one.

The proposed reference implementation would mandate "HTTP Basic Auth" (which itself made me facepalm), and it would store the credentials in a text file at /etc/openstack/users.ini that looks like this:

  alice:c8fed00eb2e87f1cee8e90ebbe870c190ac3848c
  bob:fc683cd9ed1990ca2ea10b84e5e6fba048c24929
  charlie:c4f9375f9834b4e7f0a528cc65c055702bf5f24a
  dave:3c287b08e11e55d1e90ef4e1f152f144bdcdc393

e.g. it stores the unsalted hash of a password.

<headdesk/ >

I tried to interject just how bad an idea this was, but could tell I was not communicating my concern well enough, and so just pointed out it was a bad idea, heard but did not believe the response that "it's just the reference implementation, sites can implement better if they want to", and I let the discussion continue.

Cue next the announcement a few days ago from Amazon Web Services. There is a new AWS EC2 instance type. It provides a teraFLOPS of GPU computation. Right now, anyone can spin up a cluster of these, and run just under 30 quadrillion FLOP in an hour, for USD16.80. I have been known to spend more for a pot of tea.

The obvious next step immediately happened. Someone used off the shelf software to bruteforce break every text password of length 6 or less in less than an hour. link

I have already started whiteboarding together a space/speed AWS service that uses GPU instances to compute hashes, and then uses S3 to both store and to look up results. Submit a hashed password to the service, and it looks it up in S3 and returns the password, while at the same time independent GPU instances are grinding thru various bruteforce password templates, and populating S3 with the results. I could even easily make it a paid service.

In short, the proposed reference implementation might just as well store credentials like this:

  alice:password
  bob:secret
  charlie:123456
  dave:qwerty

This way will be much more honest and apparent about the safety of that credential file, and it could require, not HTTP Basic Auth, but HTTP Digest Auth, which is also nearly universally implemented in HTTP libraries such as cURL, and so is just as available for both command line level testing, and writing interoperable modules.

What I fear will happen now is that this will get accepted as the reference implementation, implemented, included as the default install, and we all will have just created Yet Another soft underbelly of not-really- security in our public infrastructure.

TaxiMagic, a user report

I saw TaxiMagic presented by Aimee Cardwell at Ignite Seattle last June, and set up an account soon after. But I did not try it until last week.

I normally start my business trips with a cab ride to the airport. All the rest of my travel planning and reservations are via the net, so actually making a voice call to a dispatcher feels out of place. Taxi Magic promises to take the process of using a cab into the early 21C.


Actually setting up my account, and then setting it up on my phone was needlessly complicated. I did it a few months ago, and so I hope they have streamlined the process.

They don't work in every city, or with every cab company.

They do work in Seattle, but instead of working with Yellow Cab (which is everywhere in Seattle, and often loiters in my neighborhood, so often I will have a cab at my door in under 5 minutes), they work with Orange Cab, so it took longer for the cab, once dispatched, to reach me.

Actually setting up my account, and then setting it up on my phone was needlessly complicated. I did it a few months ago, and so I hope they have streamlined the process.

The Android app needs some work. It has a "map mode", that is supposed to show where the taxi is. But it kept zooming out to also display the start and destination points, even when I didn't want it to. Minor annoyance. Also, occationally it would time out on the server side, and ask me to "try again". This something that needs fixing, both needing a better protocol, and better scaling.

After I first requested a cab, it took about 10 minutes to update the status to "taxi dispatched". That was long enough to make me start wondering if it was working at all. They need to get better at immediate user feedback, and also fix that delay in finding and dispatching a cab.

It took maybe half an hour for the cab to get to me. That isn't really their fault. I got used to Seattle Yellow Cab, which loiters in my neighborhood, so I would often get picked up in less than 5 minutes.

Here is where TaxiMagic is awesome, and why I will keep using it. Paying for a cab is a pain. They are the only handwritten credit card receipts I end up with when doing expenses, and the receipts are never complete, and are hard to read, and the process slows down getting out of the cab at the destination.

With TaxiMagic, when I got to the airport, I pressed a button on the app, and the bill was paid. Period. And a nice clear receipt got emailed to me as a PDF. That itself is worth the whole thing for me.

I hope they iron out those wrinkles, get into all the cities, get all the "computer dispatched" cab companies to get with the program.

And as it is now for me, Seattle Yellow Cab doesn't get my business any more, Seattle Orange Cab does, just because of Taxi Magic.

Tech Press, fail

For a while, I monitored the tech press about Gear6's closure, and was darkly amused by it. All of the tech press and reporters that wrote about it were obviously cribbing from each other's texts, and they talked about Gear's "vanishing" and the lack of forthcoming information.

All of Gear6's customers had been told about it, and had had their support contracts transitioned. It seems that none of the tech press ever contacted any of them.

I was a "known face" of the company to the tech press. I had been on PR conference calls, had been interviewed, and had swapped emails. And yet, none of them ever contacted me to ask what had happened.

It was apparently easier to bang out yet another "me too" story about the "lack of forthcoming information" than it was for them to google their own past articles, pick out mine or anyone else's contact information, and write an email asking for information.

Anything that is not properly masticated and then pushed into the proper input hoppers, of a press release or a controlled interview, is called a "lack of forthcoming information". There seems to be a difficulty is seeing what isn't there, and a near complete lack of energized and directed curiosity brought to bear on illuminating discovered ignorance.

If I needed any more proof (which I didn't) that the tech press does very little "journalism" and nothing at all like any sort of "investigative journalism", that was it.

The job of digesting turgid hyperbole press releases and emitting CIO-friendly 4 column inch "articles" might as well go the way of the sovietologist or soothsayer.

Instead, let people and organizations just cleanly and conversationally write about what they are doing, and interested people go read it themselves.

This is not to say there is not a job for a "press". Some press-like things that do need doing are digging up and making known the things that powerful entities are NOT saying (muckraking), organizing the sea of information into learnable chunks (education), discovering and highlighting interesting stuff (curation), and generating informed prognostication (futurism).

And those things are being done, more or less (sometimes too much less). Often by skilled amateurs.

But almost none of it is being done by the most of what is currently the "tech press".

New Job, Community Manager for Eucalyptus Systems

Six months ago, right around the O'Reilly MySQL Conference, my previous employer, Gear6, suffered from "unfortunate cash flow event". That is, they ran out of money faster than their sales grew. Which is too bad, it was a good company with good and useful products, and it was staffed with good people. I appreciate the honest and ethical dealings of the board and the executive staff, who kept the we the staff "in the light" as the situation developed, and did things like paying out the accumulated vacation time and such. No bounced paychecks, unpaid expense reports, or surprise locked doors.

I spent the time working on personal projects, preparing for and going to Burning Man, studying up more on open source community management, digging more into cloud computing, and interviewing at a number of interesting companies.

And now, as of November 1st, I have a new gig. I am the Community Manager for the open source company Eucalyptus Systems.

Eucalyptus is based in Santa Barbara. I will remain based in Seattle, and will be travel down to the offices regularly, and will be travelling for conferences.

My first conference in my new corporate livery will be the second Open Stack Design Conference, which is next week in San Antonio.

A popular NoSQL document store, and "worse is better" is worse

Over in this post, Brian Aker talks about a popular social networking application (Foursquare), which recently moved off a classical durable transactional store (Postgres) and onto a new-fangled NoSQL system (10gen MongoDB), and how MongoDB's very poor durability is negatively impacting the end user experience (e.g. Foursquare keeps losing a days activity stream history).

I posted this comment:

One of the things I took away from NoSQL Live in Boston is that the standard "MySQL master/slave with replication" configuration is even worse than most of the NoSQL solutions out there already, not even having an "eventual consistancy" guarantee, instead having, at best, "wishful consistancy".

I sometimes quip that MySQL 3.11 became popular a decade ago because it fit very well to quickly and poorly written PHP3 apps. MongoDB may be the modern version of that, fitting very well to quickly and poorly written Ruby apps.

But I fear that because that because 10gen has VC mindshare over other technologically better document stores and other NoSQL stores, it might win on the "worse is better" principle.

More Drizzle Syslog. Logging big and slow queries.

One extremely useful feature in MySQL is the "slow query log". In the stock distribution, it records every query that takes more than 1 second to execute, and every query that requires a full table scan. A useful part of the Percona patches extend this feature to measure sub-second slow queries, which became important as machines got faster.

When someone asks "Why is the database slow?", the first counter question to ask is "What's in the slow query log?"

One of the confusing parts of administering MySQL is that the "query log" and the "slow query log" are different things. They have different controlling options, and different log formats.

When I ripped out and replaced the logging system during the Drizzle fork, I took these two logging systems, which were implemented as completely separate code, and merged them together. There is now only the "query log", but it can be filtered in-process to only record queries that are "too slow", "too large", or "too expensive".

This filtering is controlled with three configuration options or server variables:

syslog-logging-threshold-slow
syslog-logging-threshold-big-resultset
syslog-logging-threshold-big-examined

These all have a default value of is zero.

The "-slow" variable is in units of milliseconds. If a query takes longer than that much time, it gets logged.

The "-big-resultset" variable means that any query that has a resultset with more rows than this value logged.

The "-big-examined" is a bit more complicated. It counts the number of rows that were retrieved from the underlying storage engines. Any query that read more than the value gets logged. This is useful for detecting table scans.

For example, if a table PARTS has 10,000 rows, and has a column SID that is not indexed, then the query

SELECT * FROM PARTS WHERE SID='0042';

than the number of rows examined will be 10,000, no matter how many rows match the query. But if that column is indexed, and that index has a high cardinality, than the number of rows examined will be much less. In the best case, it will be the same as the number of rows returned.


Another major annoyance with stock MySQL is that to enable the query log and the slow query log, you have to restart the server to enable them. This is a problem for large systems with uptime requirements.

In Drizzle, on the other hand, the variables syslog-errmsg-enable and syslog-logging-enable can be set while the server is running.

Also, the threshold variables can be changed while the server is running. So you can leave them at the default values of zero when first setting up an application, and get a full log. Then turn them up to high values, and only get logged events when something unusual happens, and then turn the thresholds back down to lower more sensitive values to investigate the problem, all without restarting the server.

A Modest Suggestion for a Truly Digital Unicode Script

I have an idea for a useful Unicode script.

It consists of a single character, but will have a number of "accents" or "combining characters".

MANUS SYMBOL
MANUS COMBINING CHIRALITY DEXTER
MANUS COMBINING CHIRALITY SINISTER
MANUS COMBINING FACING PALMAR
MANUS COMBINING FACING DORSAL
MANUS COMBINING DIGIT THUMB
MANUS COMBINING DIGIT INDEX
MANUS COMBINING DIGIT INSULT
MANUS COMBINING DIGIT RING
MANUS COMBINING DIGIT SMALL

For each of the chirality and facing types of combining character, exactly one must be specified. For the digit type of combining combining, zero to five may be specified.

It would take 128 glyphs to display this font, less if the font system can handle reflections or even better, algorithmically generate the glyphs.

What this font represents should be obvious.

Setting up Drizzle and the Syslog for each other

The UNIX syslog is well-known way for applications to emit potentially interesting messages. A message is merely a string of characters with a some associated metadata. That metadata includes an "identity", a "facility", and a "priority". A daemon process running on the host reads a configuration file, and then collects and processes the messages as they are generated. It can discard them, append them to a text file, or send them them over the network to other syslog daemons.

The "identity" is just a short string of characters, and identifies the source of the events. Traditionally, it is the name of the program. The default syslog identity for the Drizzle server is "drizzled". It is set with the "syslog-ident" option, and can be read, but not changed, via a server variable of the same name.

The syslog "facility" and "priority" values are selected from a predefined list. If you read the UNIX man page syslog(3) or examine the contents of /usr/include/syslog.h, you will find the facilities and priorities defined for your system.

The default syslog facility for the Drizzle server is "local0". It is set with the "syslog-facility" option, and can be examined but not changed, via a server variable of the same name. The Drizzle server has two priority settings. The priority of the query log messages is set with "syslog-logging-priority", and defaults to "info". The priority of the error message messages is set with "syslog-errmsg-priority", and defaults to "warning". These can be read, but not changed, via server variables.

The details of configuring syslog are very specific to your operating system distribution, local operations doctrine, and personal whims. You should carefully read the documentation your distribution's syslog daemon, and understand how it is currently configured.

Create some new empty logfiles, /var/log/drizzled/query.log and /var/log/drizzled/error.log and then configure syslog to send local0.info messages to that query.log and local0.warn messages to that error.log. You may also want to reconfigure such those events do not go to the "catch all" destinations, such as the /var/log/messages and /var/log/syslog files. Also, remember to configure logrotate to manage the the new files.

On a modern Ubuntu system, you do all that like so:

Run the following commands

mkdir -p /var/log/drizzled
touch /var/log/drizzled/query.log
touch /var/log/drizzled/error.log

Create a file named /etc/rsyslog.d/90-drizzled.conf with the following contents:

local0.=info  -/var/log/drizzle/query.log
local0.=warning  /var/log/drizzle/error.log

Find the file named /etc/syslog.d/50-default.conf and change the following lines

Change the line

*.*;auth,authpriv.none -/var/log/syslog

to

*.*;local0.none,auth,authpriv.none -/var/log/syslog

Change the lines

*.=info;*.=notice;*.=warn;\
    auth,authpriv.none;\
    cron,daemon.none;\
    mail,news.none  -/var/log/messages

to

*.=info;*.=notice;*.=warn;\
    local0.none;\
    auth,authpriv.none;\
    cron,daemon.none;\
    mail,news.none  -/var/log/messages

Start drizzle the way you are used to, but add some new options to the command line or to the configuration file.

/sbin/drizzled ${other_options} \
    --syslog-logging-enable \
    --syslog-errmsg-enable

Open up some new terminal windows and run "tail -f" against the new log files, and then connect to the drizzle server you just started, and run some queries. You will see the queries show up in the query log, and if there are any errors, error messages show up in the error log.

Drizzle plugin/syslog has been merged, exploring how to use it

My new syslog module has been merged with the Drizzle mainline.

After updating your local copy of the tree, you can see the sources in plugin/syslog, which have completely replaced plugin/logging_syslog.

To check to see if your build or distribution of Drizzle has this module available, run the the drizzled binary with the --help option, and look for lines referring to "syslog", like this:

  $PREFIX/sbin/drizzled --help | grep syslog

You should see something similar to this:

  --syslog-ident=name
  --syslog-facility=name 
  --syslog-logging-enable 
  --syslog-logging-priority=name 
  --syslog-logging-threshold-slow[=#] 
  --syslog-logging-threshold-big-resultset[=#] 
  --syslog-logging-threshold-big-examined[=#] 
  --syslog-errmsg-enable 
  --syslog-errmsg-priority=name 
  syslog-ident                      drizzled
  syslog-facility                   local0
  syslog-logging-enable             false
  syslog-logging-priority           info
  syslog-logging-threshold-slow     0
  syslog-logging-threshold-big-resultset 0
  syslog-logging-threshold-big-examined 0
  syslog-errmsg-enable              false
  syslog-errmsg-priority            warning

The lines beginning with the double dash are options, which can be specified on the drizzled command line, or in the drizze server configuration files. The lines without the double dash are server variables, which can be examined and sometimes set from a SQL prompt or other Drizzle client. Next to the server variables are their current value. Notice that both syslog-logging-enable and syslog-errmsg-enable default to false.

The next post will be a quick overview of the UNIX syslog, and how this module works with it.

New syslog module for Drizzle DB

One of the long-time continuing complaints against using MySQL in an enterprise or large scale environment is it's logging system. The query log, slow query log, and error log all get written to the server's local filesystem, which is difficult and awkward to hook up to enterprise log collection and analysis systems. A recurring feature request is to be able to use the UNIX syslog instead.

One of the first things I did to Drizzle was tear out the existing logging code. This also removed a number of mutex locks, one of which was completely unnecessary, and one that got constantly hit even if logging was turned off.

In it's place I put two of the first Drizzle plugin interfaces, one for the query logging, which combined the features of the old MySQL query log and the slow query log, and one for error message reporting. I then wrote a plugin that took the query log, and sent it to the syslog.

At the last MySQL Conference, during the Drizzle Developer Day, Monty Taylor pointed out that we were missing something. There was no "send error messages to the syslog" plugin. Oops.

Now there is.

I've completely replaced the plugin/logging_syslog module with a new plugin/syslog module that contains three plugins. In addition to a rewritten "query log to syslog", there is "error messages to syslog", and also a new SQL function SYSLOG(facility, priority, text).

I've pushed the work up to launchpad, and I'm confident that after review and possibly some tweaks, it will be integrated with Drizzle.

Documentation and examples for using this new module are in upcoming posts.

Centralized Social Networking vs One Social Web, and Cloud Computing

A few weeks ago, at the 10th Internet Identity Workshop in Mountain View, I watched an impressive demo of the reference implementation of the One Social Web design.

It's a fully decentralized full federated "Facebook". I was amazed. It can do everything that Facebook can, it has far better trust model than "Trust Mark Zucherberg's ethics", it scales without polling, it uses simple open network protocols, and the reference implementation is open source.

Since before and while the Diaspora people have been collecting money via Kickstarter and giving interviews to the public media, the OSW people have designed a good scalable network protocol based on proven components and designs, and have written running code.

Huge centralized systems like Facebook and Twitter have been driving the creation of some pretty neat technology, which they have been forced to use because of their gigantic size and load and growth. Engineering effort that use to be used to add value and features are now instead consumed to just stay alive and running. These huge systems tend to run in their own data centers because at that scale it makes sense, for the same reason that Coca-Cola runs their own water treatment plants.

But, if a system can federate into small pieces then it does not have to go to such heroic efforts as it grows. The implementations can have simpler designs, and more straightforward code that is easier to develop and maintain. They are "small pieces, loosely joined", and "Fast, Cheap, and Out of Control", to invoke some proven patterns. They can keep growing when the centralized systems fail. They can handle the load of ten billion people and a trillion devices.

I've preached all these points before. But now I come to the point that occurred to me today:

These federated systems work very well with hosting providers and with cloud computing systems. They require basic off-the-rack system administration skills, and can be designed to use basic cloud infrastructure APIs and components.

If you bring up a OWS node, you could run it on a junk machine under your desk. But when you start depending on having it be stable and managed, it makes since to let Rackspace Slicehost or Aamzon AWS EC2 run it for you.

Feh on "Expedited Border Crossing Schemes"

I just got spammed with an invitation to join "Flux", which appears to be yet another Expedited Border Crossing Scheme, this time between the US and the Netherlands.

I think that the proliferation of all these special pairwise expedited border crossing schemes is stupid. Each one has it's own special set of requirements, each one has it's own special treaty basis, each one has it's own fee, each one has its own special kiosk, each one has it's own database (of varying, unproven, and opaque security), and each one issues it own special card. And they all are exist only as workarounds to bypass the idiotic assinine border paranoia and xenophobia zeitgeist that infest the current system international border control and visas (of which my own country, the US, is one of the very worst offenders).

If we are going to have an international "trusted traveller" scheme, make it much more useful and general. That is, one that lets me use the TSA "registered traveller" inside the US, and also gives me a visa and expedited entry into all of NAFTA, the Commonwealth countries (UK,CA,AU,NZ), and the Schengen Area, and so forth. Also, it shouldn't have it's own special card, but instead can just be accessed via the barcode on my passport, or if the PTBs really insist, a biometric enhanced passport.

But for now, I will give Flux a pass, just like I've given SENTRI and NEXUS and TSART a pass.

Wishlist of features for Google Reader

I use Google Reader as my blog reader.

In the "manage subscriptions" page, where it lists all the feeds I follow and lets me rename, delete, and tag/folderize them, it would be useful for it to also display stuff like "is this feed still live", how many entries per week it generates, and how many other people read it.

It would be also useful to have a "cleanup", to remove blogs who's feed URL is broken, or haven't generated any articles for months.

About Lisp

This is something I posted to USENET in January 2002:

If you learn Lisp correctly, you can grok all programming styles with it: procedural, OO, predicate, functional, pure or full of side-effects. Recursion will be your friend, function references your allies, you will truly know what a closure is, and that an argument stack is actually a performance hack. You will see that the most elegant way to solve a problem is to create a custom language, solve the generic problem, and have your specific one fall out as a special case. You will learn to truly separate intent from the bare metal, and you will finally understand the two deepest secrets, which are really the same secret, which we tell all, but so few understand, that code and data are the same thing, but organize your data and your code will follow.

I have been walking much more and driving much less.

I don't understand how people can complain that Seattle is not a walkable city:

Capitol Hill, First Hill, the Downtown, SoDo, the International District, Lower Queen Anne, and Belltown are all less than half an hour's walk away from my home. Add another half hour and I can get to SoDo, Colombia City, Fremont, or Ballard. Half an hour is less than the average commute, not even counting the time spent parking. Plus it eliminates the stress of close quarters city driving and dealing with one-way streets.

Hopefully I will not turn into one of those people who either preach or desire to compel people out of their cars. I intensely dislike that mindset. I am just enjoying the different perspective.

The different pace and the different experience of the urban environment, it gives me a whole new way of looking at my city home. I notice things I had not noticed before. More of the greenery and odd shaped pieces of unbuildable land that are everywhere. The texture of the sidewalks and roads. The decorations on the houses and buildings and storefronts. Little hole-in-the-wall cafes and shops.

The really big win is that I am literally smarter and more intellecually creative when I am out walking. I suspect it's a combination of exercise being good for thinking, the stimulation of seeing the urban landscape in higher detail, and not having to spend large amounts of brainpower flow avoiding a traffic accident.

Turning that increased mental capacity into actually produced work is being an interesting problem. I'm considering a solid state recorder.

Thought: Microblogs and duplication

Microblog readers should detect duplicates and suppress displaying them. I have a couple of Twitter accounts, an Identi.ca account, and also my Facebook account: I see many duplicates.

It would be best if the microblogging services would generate unique message IDs on post, and then preserve them on read and syndication, but it's probably too late to demand that they do that, as each service wants to think of itself as "Special" and has little interest in "playing well with others" in a larger media stream ecology.

But even without that, a smart microblog aggrigator/reader should be able to notice things like "I've just seen this exact tweet from this poster, just read from another logged in account", and fold them together. This would be useful for the common case of reading more than one twitter account that has some of their follow list in common.

The metarule is "reduce the cognitive load on the user". Anything that makes it easier for the human in the loop to read and understand the stream faster and easier, is better.

MySQL Conference, Day 4 (being written well after the fact)

As I said at the end of the Day 3 post, I did well enough at the Ignite session that I "won" the opportunity to give it again. There was some confusion about the right time to be seated, and some of the "winners" were not actually properly informed of what they had "won". Fortunately, IM conversations sorted most of it out, and Gerry, Sarah, Gillian, and myself reprised our talks.





I got the opportunity to take notes at the MySQL Ecosystem Summit, but instead spent the time taking in a few more sessions. It was a hard choice, and from what I learned later about the cool discussions at the summit, I think I may have made the wrong choice.

There were some sessions I was glad to have gone to, however. The best was Paul Vallee's "Worst Practices of Database Outsourcing Infrastructure Support", which was an excellent "peopleware" presentation on, well, exactly what it says on the tin, how to have a good or bad relationship between a client and an IT shop.

The rest of the time was spent socializing, networking, "hallway tracking", chatting with the Amazon AWS RDS folks, and so forth. The sort of thing that conferences like this are really for.

Memcache is "Elastic", not just "Distributed"

In this article, the Forrester analyst Mike Gualtieri draws a useful distinction between "local cache", "distributed cache", and "elastic cache".

He then places Memcached firmly into Distributed Cache catagory.

No living technology holds still, and Memcached is no exception. Memcached did start out as just a distributed hash, but there has been a lot of work by many smart people to make it elastic as well.

I work for Gear6, which makes a Memcached server that has a number of neat "enterprise" features, one of which is elastic resizing, which we call "dynamic services". You can resize in a datacenter by adding more units and resizing into them, and even neater, you can resize "in the cloud", on Amazon EC2. You go to the UI, tell it to expand the cluster, and it will spin up more EC2 instances itself, and then expand into them.

MySQL Conference, Day 3

The day started out with 3 keynotes: Brian Aker, Michael Widenius, and Sheeri Cabral.

Brian gave his "The Drizzle Story" talk, which I tweeted via @drizzledb. Then Monty spoke about about Monty Program AB and MariaDB, which is his continuing work on his fork of MySQL. Following them both was an excellent talk from Sheeri about how the community interaction model will continue to work in this Oracle era.

I went to my employer's Gear6's session done with Answers.com. Answers is a top 20 web site, and is the 2nd fastest growing one, after Facebook. It was fun to see their performance and scaling numbers, and how they are using Gear6's memcached product, especially how two 1U G6 boxes replaced an entire rack of 20 2U memcached servers.

I also really enjoyed the Facebook operations session, and also Matt Yonkovit's NoSQL characterization and benchmarking talk.

I talked to a lot of people about Memcached, and about Gear6, and about NoSQL.

The big event for me was the Ignite session. I got there a touch late, Sarah Novotny was already giving her "Backups Don't Make Me Money" talk. Gillian Gunson gave a really good talk "Rules for DBAs", and then after Brian Aker gave his famous NoSQL talk, I gave my responding session as the last Ignite. I hit it really well, and Sarah and Gillian and I all "won", which means we all get to do our Ignites again the next morning as part of final day keynote.

Interviewed by Scoble about Gear6 Memcached

Yesterday, while at the MySQL Conference, I was interviewed by Robert Scoble about my employer, Gear6 and our product, an enterprise memcached distribution.

MySQL Conference, Day 2

While waiting in the line for a breakfast table, I found Reggie Burnett, who is still with MySQL now Oracle. We shared a table and talking about Android and the future of handhelds.

I missed the keynotes by Edward Screven and by Tim O'Reilly. Instead I had scheduled interviews with The 451 Group and then with Robert Scoble. Those both went really well. And I learned that the Screven speech went not so well, which would have been amusing, but not a good use of time.

The rest of the day, so far, has consisted of meeting people, spending time at the Memcached.org booth and the Gear6 booth, and doing more scheduled tech press interviews. Sarah Novotny showed up during the nosh and free beer, right before the BOF sessions.

I attended BOF session for "How To Use Memcached", which unfortunately is scheduled at the same time as the Drizzle BOF. Most memcached BOFs in past years have been very rarified hackathons and protocol design sessions. This one was a "there are no stupid questions" style Q&A for interested novices.

MySQL Conference, Day 1

I checked in in the speaker lounge and picked up my bag of shwag. This year O'Reilly is giving speakers some free books, slide:ology and Confessions of a Public Speaker . It makes sense for them to give these books to their speakers, as the better the speakers are, the better the conference is, and the more successful O'Reilly is at getting more conference business.

I spent the first part of the day fielding scheduled calls with members of the tech press about Gear6's new release of our Memcached Server.

Over lunch I picked a random table, and had a good conversation with a bunch of "ordinary attendees", people who are not "known names", but came to the conference to attend tutorials and sessions, and actually learn how to use this stuff. Most had been to Sheeri's tutorial about MySQL configuration, but the ones about Replication by Mats Kindahl & Lars Thalman, and the one about Performance Tuning by Baron Schwartz and Morgan Tocker were also popular.

The Drizzle project had a two different tutorials, merged together, about joining the project and about writing replication plugins and storage plugins, presented by Padrig, Jay, and Toru, and it inspired me more to get back into hacking on Drizzle. While there I hecked Monty Taylor (who was sitting a row behind me) via IRC into fixing the instructions for installing on Ubuntu.

Later that afternoon, by the hotel bar, I finally met again Gillian Gunison. She and I first met at this conference 4 years ago, and then have kept in touch swapping technical problems back and forth over IM since then.

The annual community dinner was huge, overfilling the space at Pedro's.

MySQL Conference, Day 0

After spending the morning looking over photographs with the amazingly talented Julian Cash, I drove through the pouring rain from San Francisco to the SJC airport, to pick up Brian Aker. From there, to the convention center, where I linked up with Monty Taylor, Jay Pipes, Stewart Smith, and a number of other people involved in Drizzle and Memcached. And from there, we all headed over to Sarah Sproehnle's place for a potluck party.

So much of the history and development of technology is based on a foundation of personal relationships. The people working on stuff get to know each other, and form friendships outside of just the work, and they introduce each other to their other friends, and from that, connections and cross-fertilization of ideas happen.

This is especially true for open source software and all the aspects of internet technology. The real but mostly unstated reason for technical conferences is exactly that. We could just learn from books, articles, and maybe classrooms. But it wouldn't be enough.

The hallway track, the conversations over beer, the friendships formed, the "networking", and the defusing of tension by seeing faces, seeing old friends, and making new ones. That is the value.

This is my fourth MySQL Conference. The first keynote is in 12 hours.

This is going to be fun!

The IETF, and membership via corporation

The IETF is the standards body that describes and defines the basic protocols that make the internet work. There is a lot of basic knowledge and community wisdom "baked in" to the IETF process, and much of what makes the IETF work is how it is very different from other standards bodies (such as IEEE, ANSI, ISO, NIST, ITU, etc etc).

One key difference is that "groups" do not join the IETF. Cisco Systems or IBM is not a "member" of the IETF. No agency of the US government, or of any other government, is a "member" of the IETF. No university, or non-profit, or even "concerned citizens group", is a "member" of the IETF.

Only individual people can be "members" of the IETF. And "membership" is mostly defined as "who shows up on the mailing list" and "who shows up at the meetings".

There have been many cases in the history of the IETF where well known members who are in the middle of writing standards or of chairing various important working groups, who have worked for well known large companies, will change employers, to other companies, to startups, switch around between industry, universities, research labs, and government, and this will not, does not, and should not, affect their position inside the IETF at all.

Sometimes people, inside and outside of the IETF, need to be reminded of this.

If you want to write standards in the IEEE and ITU style, you know where you can find it. When you are writing standards inside the IETF, that is how it works.

Just pump IP datagrams

Whenever someone designs a fancy complex network stack, the first thing any buyer/user wants to do is run the internet over it.

Said designer will then, grudgingly, write an appendix standard, usually calling it something like "legacy protocol support".

And that is the only part that anyone actually uses.

And then people paying the bill for the network notices all the processing and bits-on-the-wire overhead they are paying for that they never use, and demand that layers of complexity and cost be peeled off and thrown away, so that carrying IP datagrams be cheaper and faster.

This gets iterated down to the underlying real physical wire.

For example:

1. "IP over PPP over V.44bis over dialup POTS" and "IP over ISDN", to
   
2. "IP over SS7 T1/E1", to
   
3. "IP over ATM", to
   
4. "IP over SONET", and now
   
5. "IP over WDM" and "IP over lightwave"
   

Quit trying. Just give up, and cut to the chase. Whenever you have an idea for pumping bits over any distance, either as technology startup, an industry consortium, a standards body, or even a maker/hacker playing around with open source hardware and unlicensed spectrum, just set it up to carry IPv4 and IPv6 datagrams as close as possible to the real underlying carrier (using PPP if really necessary), and then stop.

The only possible sane exception would be adding FEC (forward error correction) and detecting and retransmitting on carrier and bit garbling (like what 802.11 does).

But, don't waste your time and anyone else's money on anything more.

Reacting to "Memcached is not a store"

I keep seeing "Memcached is not a key value store. It is a cache. Hence the name." This is strongly reinforced by statements made in the memcached mailing list itself.

This is short sighted.

Memcached is a number of things. It is an idea (fast key value store, with distributed hash function scaling), it is a network protocol (two of them, in fact), it is a selection of client libraries and APIs (most based on libmemcached), and it is a server implementation. In fact, now, is is now a number of server implementations, because now there are a number of different things that implement the memcached protocol.

Only one of which is the open source community edition of the memcached server, version 1.4, downloadable from http://memcached.org/

Despite what you may get told, especially on the memcached mailing list, you can in fact use memcached as a store, not just as a cache.

Yes,if you fill it up, it may either error or evict something, but you have to live with if you fill up a MySQL tablespace disk, it crashes or errors on you too. Yes, it's not persistent. But, frankly, there is a lot of data that doesn't HAVE to be persistent, or at least doesn't have to persist in that format. MySQL, Postgres, Oracle, and MSSQL have temp tables and memory tables, and people are ok with that.

But it's not even "not persistent". You can get various kinds of "persistent memcached" by using Gear6 which can persist to flash, or by using Tokyo Cabinet, which has a memcached interface. There are a number of other stores that are adding memcached interfaces. I have seen people working to use libmemcached to add memcache "memcapable" interfaces to NDB cluster, to "embedded" InnoDB, and even to the filesystem. And more are coming.

I've also seen work to do crazy stuff like put a full relational SQL interface on top off of memcached, or a simple "cut down" SQL that just does primary key inserts and lookups for the "e-blob pattern", or the NDB Cluster API on top of it. It's all good.

So when someone sneers "memcache is a cache, not a store", don't be discouraged. They have reasons to say that, but IMO, they are not very good reasons.

Use it as a store, just know what it does and doesn't do. Just like when you chose any other data store.

Attending MySQLcon, round 4

I will be at the MySQL Conference in Santa Clara in a few weeks in the middle of April. This will be my fourth time there. Gear6 is going to be there as a sponsor, and some of my coworkers will be presenting, and I will probably share some stage time with them as they do.

My first time, was my first time. I was still working solo, as an independent. I met many MySQL-sphere people who are now personal friends, as well as being industry peers and contacts. I gave the talk that "put me on the map", for a project that was my opportunity to learn about distributed version control, cloud computing, Amazon Web Services, MySQL internals, and MySQL storage engines. I would look at the presentation schedule, and usually not even understand the titles.

My second year, I was working as part of MySQL Professional Services. My presentation was about MySQL User Defined Functions. The major buzz was "What will Sun do?". The technology arising that I was the most aware of was Memcached. One of the first memcached hackathons happened. The business thing I became aware of was Rackspace's gearing up to enter the cloud computing space. I learned later that the seeds of Drizzle DB germinated there. I would look at the presentation titles, and wish I could attend more presentations, because I could learn from almost all of them.

My third year, last year, I was working on Sun Cloud. The major buzz was "What will Oracle do?" and about the separate Percona conference and similar political/personal stuff. The tech buzz was about Drizzle DB. I would look at the presentation titles, and realize I could teach most of the presentations myself, and so focused mainly on network and on the "hallway track".

It will be fun to see what this year brings.

So tired of MySQL's old "GPLed network protocol" FUD.

Over in the blog post High Availability MySQL: Can a Protocol be GPL? Mark Callahan found the following comment in the source file sql/net_serv.c

This file is the net layer API for the MySQL client/server protocol, which is a tightly coupled, proprietary protocol owned by MySQL AB. Any re-implementations of this protocol must also be under GPL, unless one has got an license from MySQL AB stating otherwise.

I am second to few in being a fan and a proponent of the GPL.  However, this claim in this comment is utterly bogus. It dates back to willful misunderstanding and FUD spreading and the desire to strong-arm the sale of licenses on the part some people at the old MySQL AB company.

If license of source code followed network protocol of running implementation (which stretches "derived work" to an extent even an RIAA flack dare not even dream), then it would be illegal to view a web page served from IIS from Firefox.

And besides, the MySQL client server protocol is not even "owned" by MySQL AB.  It's very directly merely extended (and not very much) from the old MSQL client server protocol.

To moot the entire issue, the new libdrizzle client protocol library not only speaks the new different and superior Drizzle client/server protocol, it also speaks the old MySQL protocol. The license for libdrizzle is BSD.

This whole thing makes me annoyed, because this kind of overreach and FUD just makes the GPL itself look bad.

Why companies need to care about "social media"

Part of my day job now is to keep up with the "social media" stuff for my employer and for the technologies we play in.  And while I know it's important, and that it's part of my job, I've often been really annoyed by the cohorts of nearly useless "social media experts" who say very little and charge huge bills for knowing how to search twitter.

But I saw today on Chris Messina's Buzz about "The Browser as Social Agent, part 3" that really states simply and directly why corporations and organizations, especially non-tech companies, should care about social media stuff:

I noticed a sign on the wall that I’d not seen before, providing links to that local Whole Foods’ Twitter and Facebook pages. It struck me as rather strange that a company like Whole Foods would promote their profiles on networks owned by other companies until I got out of my tech bubble mindset for a moment and realized how irrelevant Whole Foods’ homepage must seem to people who are now used to following friends’ and celebrities’ activities on sites like Twitter and Facebook. What are you supposed to do with a link to a homepage these days? Bookmark it? — only to lose it among the thousands of other bookmarks you already forgot about?

Snuggies, Netbooks, and NoSQL

A few days ago, a friend of mine sent me an Instructables link on modifying  a Snuggie to make it more usable with a netbook.  I don't like their approach, which consists of cutting a hole in the Snuggie so the netbook can rest on your legs and not slide down.  A better approach would be to sew or glue a grippy material on the right spot for the netbook.

And then we started rapping on how to modify an array of Snuggies and Netbooks to provide a NoSQL service:

First, get a bunch of netbooks, then glue them to the Snuggies.  Add an extra hole for each netbook so you don't have to get your hands cold while you attempt to turn them all into servers. Then you cut a hole for the distributed hash table to get through so your arms don't get cold, but conveniently, you don't have to build an extra tube for every table's relational guarantees because nosql does away with that silliness. Gone are the hours spent hand-sewing each tube on for each netbook, which makes this system way more scalable for your average snuggie, though results may vary, and you can sew the extra tubes on if you want.  Then embroider on your associative arrays or key-value pairs, whichever you want to use. Oh, and remember to sew with scant quarter-inch seams - don't waste extra fabric, quarter-inch seams scale better anyway...

Revisiting and defending my Tweets from NoSQL Live in Boston

Some people have been confused by my Twitter stream from last week's NoSQL Live in Boston conference.  I've never been very good at staying "on message", or adhering to a set of "talking points".  Some people thought I was there to "defend the status quo", or to defend and promote memcached, or memcached+mysql.

True, I was there in part to teach about and promote memcached, and especially Gear6 Memcached.  I have a great employer, but they are not sending me to conferences because they are a charity.  I taught the memcached  breakout session, and also worked the "hallway track", giving a number of people a crash course in what memcache is and what makes it useful, and handed out a pile of business cards.

As for my tweets and pithy statements... Well, some over-simplification has to happen to reduce a concept to 140 characters

My statement, "NoSQL as cheaper to deploy, manage, and maintain is a myth. it costs just as much, if not more", which I said that into the mike at the start of the scaling panel, was very popular to tweet and retweet.

It's something that is a "Jedi Truth", it's true from a "certain point of view".  When you add together the costs of the much shallower "bench" of hirable operational experience, the increased "minimum useful hardware footprint"  (which seems to be about at least 5 full storage nodes), and the evolving maturity of the client libraries, and such, NoSQL is not going to save you money.  Until an important threshold is reached.  When you scale and/or your data representation "impedance mismatch" hits that nasty inflection point, where the "buck for bang" curve suddenly starts to rise hard, and it looks like you will need to start spending infinite amounts of money to keep growing.  Then the NoSQL approach does become cheaper, because it's actually doable.  And it's probably wise to start considering, researching, and then migrating to NoSQL before you hit that wall.

My statement "people have been wanting DBA-less databases about as long as they have wanted programmer-less programming languages" was also popular.  I stand by it.  NoSQL doesn't crack this nut, nothing ever well.  Some NoSQL solutions look like they are "DBA-less", such as AWS SDB, AWS RDB, FluidDB.  Those systems are not DBA-less, they have DBAs, just that the cost of the DBA is "hidden" in the per-drink rental cost of those systems, instead of sitting on your balance sheet as a salary.

The statement "Twitter is using Cassandra because bursty writes are cheap, compared to others" is something I said not because I knew it, but because I just learned it, and I was a bit surprised by it.  I think that the original statement was by Ryan King of Twitter, who was also on the scaling panel.

My statement "Memcached should be integrated into all NoSQL stores" is something I also firmly believe.  The very-high-performance in-memory distributed key value store is a very useful building block for larger systems, and I think that whatever larger NoSQL systems we end up will use it as a component of their internal implementations.

The statement "being able to drop nodes as important as being able to add, because scalability is pointless w/o reliability" was also by Ryan King.  I tweeted it because it is very much something worth broadcasting and remembering it.  It has a little more context in his next statement "The first day we stood up our #cassandra cluster, 2 nodes had hdd die... Clients never noticed."  Machines fail.  And as they get faster and cheaper, and as clusters get bigger, machine failure must become something that must not be any sort of emergency.

My statement "open source means folks dont need a standards body" was a extreme simplication of part of Sandro Hawke's talk.  I tweeted it because it was something I've felt to be mostly true enough for a long time, and it was nice to see someone else recognize it.  As Sandro stated later in twitter, "I think I added an important "sometimes"!".  And he is correct.  It's not true as an absolute statement.



All in all, NoSQL Live was a very good conference.  I felt that the speakers taught and learned, all the other attendees taught and learned, and the networking and hallway track was first rate.  Thanks to 10gen for organizing it, and being entirely fair to their "competition" in the rapidly growing and evolving NoSQL space.

Social location services need to have an XMPP IM interface

If there is any social application that needs "the real time web", and yet, nobody yet is getting it right.


Let's look at Foursquare.

If I have it write to Twitter, then all my followers, independent of their location or interest, get spammed with my location.  (Which is why I've stopped having it do so by default, and instead just push to Twitter for specific cases).


And yet if I don't have it write to Twitter, then people have to poll the website or API, to see where I have been, but they don't get alerted to where I actually am.

Neither of these are good or useful, especially if one has a lot of microblog followers, and more than a very few followers and followees on 4sq.

Spam is bad, and polling is bad.

A better way would be for me to add "updates@foursquare.com" to my chat roster.

And then when someone I follow checks in, and they are near me, I would get an IM from that, saying "Bob Smith is at Joe's Bar".

And for me to check in, I send a message to updates@foursquare.com, saying either "checkin" or "checkin Remedy" or "checkin 32417", and it will use geo IP lookups, XEP-0080 data, FireEagle data, Google Latitude data, my past checkin history, the checkin string hint, or the actual venue id, to figure out that I am at http://foursquare.com/venue/32417

People who want to see where I have been can just poll the RSS/Atom feed of my checkin history.  If that Atom feed implements PubSubHubBub, they don't even need to poll it.

A desire for an even more open and even more public geolocation database

The world needs better well-known geolocation place database.


The idea/vision I'm having right now looks like this:

The raw data is sitting FluidDB or AWS S3 or something like it, e.g. a public readable and updatable key value store.

In it there could be stored the wardriving results of wireless network AP MAC address, tagged with lat, lon, alt, signal strength, time-seen, etc

In it there could be stored every cellphone tower everywhere, along with lat, lon, alt, tower id, carrier, technology, addressing, etc

In it there could be every possible geographic outline cluster of interest, by some sort of unique id, and tagged with its name, kind, outline, list of parent containers, etc.

Kinds would be things like "nation", "state", "provence", "region", "metro area", "city", "neighborhood", "park", etc

It would have both "official" things like nations and states, and less official things like "metro areas" and "neighborhoods".  Flickr/Yahoo has been doing some really cool stuff with doing tag+geo analysis to crowdsource compute things like "what is the area that people on the ground actually call 'Paris', which is rather different then the official border of the City Of Paris".

In addition to all this raw data, a data service that speaks REST and speaks XMPP (XEP-0080 etc etc) can sit on top of it.

A client could say "I have this raw data from my GPS, I see the following list of APs with the following strengths, I am associated with this one, I see the following list of cellphone towers at the following signal strengths, and I'm associated with this one, and the last time I saw these APs and towers, I turned out to be at one of the following location IDs".

The service would send back "You are at [place Remedy Tea; neighborhood Capitol Hill; city Seattle; county King; state WA; nation US], your exact lat/lon/alt is foo, and the id for that specific location is foo".

And then the client could say "tell me more about location id foo" and the service could send back stuff like the OSM data, FireEagle data, the Google Places info, the Yelp id, the Foursquare id, the Brightkite id, the postal address, the phone number, etc etc.  Whatever it thinks it knows about that location.

What all this needs is that it needs to be as much as possible open and public.  Which is why I suggested something like FluidDB.

Twitter geoloc ideas

The Twitter microblogging service has the ability for clients to "stamp" updates with a geolocation.  If they would extend that just a little bit, some really actually useful stuff would become possible.

Some of this would need features in the client, some would need features in the server, some could be done with work in both sides.

I would like my handheld client to give me a special alert when someone I follow does an update that is geographically near me.  As it is now, my client in my phone only alerts me when someone DMs me or mentions me, as it would be entirely too noisy if everyone I follow does anything.

It would be useful to be able to subscribe to the public timeline of some space.  For example, then a conference could put up on public monitors all the local tweets, not just the ones that match a hashtag.

Another neat thing would be if I could set some metadata in an update that means "this is probably only interesting to friends near me".  The service then could either filter it, and/or pass it out to my followers, but their client could choose to suppress it if it's not near enough.  How near is "near enough" could be set as a hint in the metadata, and/or a default my account settings, and/or by settings in my follower's clients.

Labeling a location would be neat.  For example, it would be neat if I could right now tell Twitter "loc Remedy", and instead of a generating a public update, the service says "hmm, from geoloc data and past history and past updates, I am pretty sure that he means "Remedy Tea; Capitol Hill; Seattle; WA; US".  So now my location has a name. Which can be fed back into the public latlon-to-name database as well.

Another useful inline command could be "checkin".  If I just say "checkin", then a geolocation tagged update would go out saying something like "is at $PLACENAME", where the placename is pulled from a latlon-to-name database, and/or from my recent or past "loc" history. The "loc" and "checkin" commands could even be combined, so if I send "checkin Remedy", then my local placename gets set and refined, and the announcements goes out as well.

If Twitter and/or the other microblogging services get this right, they could very well "eat" the specialized location/social services such as foursquare and brightkite.

MySQL+Memcached is still the workhorse

(originally posted at the Gear6 corporate blog: MySQL+Memcached is still the workhorse.  Please comment there.)

Because I'm becoming known as someone who knows something about "this whole NoSQL thing", people have started asking me to take a look at some of their systems or ideas, and tell them which NoSQL technology they should use.

To be fair, it is a confusing space right now, there are a LOT of NoSQL technologies showing up, and there is a lot of buzz from the tech press, and in blogs and on twitter.  Most of that buzz is, frankly, ignorant and uninformed, and is being written by people who do not have enough experience running live systems.

A couple of times already, someone has described an application or concept to me, and asked "So, should I use Cassandra, or CouchDB, or what?"

And I look at what they want to do, and say "Well, for this use case, probably the best storage pattern is normalized relational tables, with a properly tuned database server.  Put some memcached in the hot spots in your application. Maybe as you get really big, you can add in replication to some read slaves."

The relational table model is not dead, at all. It will never die, nor should we try to kill it.  We no longer have to be as religious about normal form, and we don't HAVE to fit everything everywhere into this form, but there is no reason to avoid it just because it's not "sexy and exotic".

Running a real live system is not a junior high prom.  You don't "win" by showing up with a sexy exotic date, and by wearing the prettiest outfit.

Running a real live system is running a farm, ploughing a field.  You want a workhorse that you know how to use, you know you can get gear for at the blacksmith and the tackle shop, and that you know you can hire field hands who know how to use it well and take of properly.

MySQL+Memcached is, today, still, that workhorse.

Empire Avenue verification

EAVB_MZRWCSYXDA

The death of Memcached is greatly exaggerated

There are many reactions going around to MySQL and Memcached: End of an Era, especially to the statement "it's clear the MySQL+memcached era is passing".

I think that really depends on what you mean by "era" and "passing".

The era of memcached being THE cutting edge technique for getting speed at scale may be "ending", but not because memcached is failing, but because there are additional (not replacement, additional) techniques now emerging.

When I was doing MySQL Professional Services, back when I started, most of my gigs had just barely heard of "sharding MySQL plus memcached".  By the time I left that world, everyone had heard of it, many were doing it, and some very few were doing it well. Since then, we've all reached the point where nearly everyone who NEEDS it is doing it. And still the load and pressure rises on running systems, and so we are looking for more tools.  And more tools are being developed.  That need, that development, that excitement, is what "NoSQL" is all about.

But that won't be the end of memcached.  The technique of the high-performance key-value store is just to useful of a building block, both on it's own, and as a sub-component of other technology components, to just throw out.

I'm sure that memcache will continue to evolve.  There will be more implementations, there will be limitations removed, there will be more management tools, there will be other systems that add the memcached network protocol, there will be ORMs and other frameworks that will build in the assumption that memcached is available, there will be features to the protocol and implementations for shared hosting and cloud environments.

And even ignoring all that, there are still a myriad of internet and intranet systems running that really need what memcached can give them, and bringing it to them all is going to be a long and interesting task.

Memcached is going to stick around.

(This was originally posted at the Gear6 corporate blog.  Please comment there.)

"How do I add more memcached capacity without an outage?"

Once someone starts using memcached, they tend to quickly find themselves in the state of: "my database servers overload and my site goes down if the memcached stops working". This isn't really surprising, quite often memcached was thrown into the stack because the database servers are melting under the load of the growing site.

But then they face an issue that is, as mathematicians and programmers like to call it, "interesting".

"How do I add more capacity without an outage?"

At first most people just live with having that outage. Most systems have regularly scheduled downtimes, and during that them the memcached clusters can be shut down, more storage nodes are added, and then it is all restarted, with the new distributed hash values for the new number of nodes.

Ironically, the more successful the site is, the more it grows, the more costly that outage becomes. And not linear to that growth either. The increasing cost is more on the order of the square of the growth, until they literally cannot afford it at all. As the cache gets bigger, it takes longer for it to rewarm, from minutes to hours to days. And as your userbase grows, the more people there are to suffer the poor experience of the cache warming up. The product of those two values is the "cost" of the outage. This is bad for user satisfaction, and thus is bad for retention, conversion, and thus revenue.

This can be especially frustrating in a cloud environment. In a physical datacenter, because you have to actually buy, configure, and install the hardware for a node, it somehow feels easier to justify needing an outage to add it to the cluster. But in a cloud, you can start a new node with the click of a button, without getting a purchase approval and filing a change plan. And also, in cloud environments, we all have been evangelizing "dyanamic growth", and "loosely coupled components", and "design for 100% uptime in the face of change". And yet here is this vary basic component, the HDT KVS cluster, that doesn't want to easily work that way.

There are ways to resize a DHT cluster while it is live, but doing so is an intricate and brittle operation, and requires the co-operation of all the clients, and there are no really good useful open source tools that make it easier. You have to develop your own custom bespoke operational processes and tools to do it, and they are likely to miss various surprising edge cases that get learned only by painful experience and very careful analysis. Which means that the first couple of times you try to resize your memcached cluster without a scheduled outage, you will probably have an unscheduled outage instead. Ouch.

One commonly proposed variety of solution is to make the memcached cluster nodes themselves more aware of each other and of the distributed hash table. Then you can add a new node (or remove a failed one), and the other nodes will tell each other about the change, and they all work together to recompute the new DHT, flow items back and forth to each other, put items into the new node, and try to keep this all more or less transparent to the memcached clients with some handwaving magic of proxying for each other.

And that, more or less, is what Gear6 has just done, under the name "Dynamic Services". We have released it first in our Cloud Cache distribution, initially on Amazon AWS EC2, and then on other cloud infrastructure systems. Soon next it will be in our software and appliance distributions.

This is an especially useful and neat in a cloud environment because the very act of requisitioning and starting a new node is something that the underlying infrastructure can provide. So you can go to the Gear6 Cloud Cache Web UI, and ask it to expand the the memcached cluster. That management system will interface to the EC2 API, and spin up more Gear6 memcached AMIs, and once they are running, add them to the cluster and then rehash the DHT. All while the cluster is serving live data.


(This entry was originally posted at my Gear6 Corporate Blog. Please comment there.)

Things I wish Buzz and other social networking systems did when someone starts following me

Google Buzz links up with Google Reader, more or less.  I use Google Reader twice a day. The main effect that Buzz has on my experience is that each morning there are a handful of people in the "people have started sharing with you" item.  Which I then open, and do not recognize them, and so close the tab back without following them back.

Just displaying a name or handle and maybe a cute picon is not sufficient.  Some people have this wonderful memory for names and faces, but some of us are not so much.

When J Random Person starts following me, Buzz & Reader should display to me some more context about that person.  For example, that person could have 140 characters of self description, plus 140 characters of "why I want to follow you".  With no URLs or any "rich content" allowed.  The reason to keep it short and simple is to reduce the exposure surface for spamming.

Most instant message systems have the provision of sending a "Why you should add me to your roster" when making a connection request.  Facebook does this as well.

And something that would be amazingly useful is, if I am following anyone who is following that person, show that to me.  If someone I interested enough in to follow already thinks that someone is interesting enough to follow, I am more likely to decide to as well.

Most instant message systems have the provision of sending a "Why you should add me to your roster" when making a connection request.  Facebook does this as well.

I am more likely to approve and/or follow someone back after a connection request if I am shown some additional context, especially how they already are linked into my extended social graph.  I don't want to have to take the time to do various graph network search tools to figure it out.

Just displaying a name or handle and maybe a cute picon is not sufficient.  Some people have this wonderful memory for names and faces, but some of us are not so much.

Idea: parallax, 3D streaming video compression, and webcams

If your computer has two or more webcams pointing at you from opposite corners of the display, then it can do parallax calculations, just like your eyes and brain do, and by doing so, construct a decently good depth map 3D model of what it was seeing.  Most of the time what it would see would be your  face  in the near area, and whatever wall was behind you in the far area.

For sending streaming video for video conferencing, the simple and dumb thing to do would be to send a complete video stream from each camera.  This is dumb and wasteful.  The way that video compression works is that it looks for similarities between frames over time, and then sends a stream of transformations and differences.  An obvious technique for sending parallax video streams is for the video compressor to look at the multiple simultaneous frames, as well as over time, and send transformations and differences over space as well as over time.  I do not know if any of the MPEG or other video stream compression standards can do this, but it will soon be necessary, especially now that the video entertainment industry is seriously talking again about 3D.

I'm sure there are a whole slew of obvious patents all claiming assorted variations of this simple idea

A particularly simple and stupid way to do it would be to just feed the alternating frames, left right left right, into the video compressor, and pretend that flicking back and forth in space is the same as motion through time.  While I'm sure that some useful level of compression would result, it would be extremely, shall we say, non-optimal.

Anyway, the math for doing the parallax calculation and the math for doing the video compression transformation and differences would be very similar.  A smart implementation could be spitting out a rough Z depth map as well as the compressed data stream.  In fact, it would make sense for the Z depth map to be sent as part of the compressed video data stream.

Doing all this with more than two cameras makes calculating the depth map even easier, plus each camera can have cheaper lower resolution sensors,  which can be mathematically combined into a much higher resolution view.

Another useful trick for the webcam case would be to use the depth map to distinguish between the stuff in the near field (which probably is the face of a talking head) and the random background clutter, and either send the background video data at a much lower resolution, or maybe not even send it at all.  This would save bandwidth, increase privacy, and improve the user experience.  After all, when you are on a video chat call with someone, you usually dont care at all what is behind them.  It carries no useful signal for the conversation.

This same trick could be doable even without two cameras and parallax.  Facial recognition software is good enough that it could draw a good enough bounding curve around your face on a frame by frame basis, before sending the raw frames into the compressor.  This might even be a CPU time win, because the compressor doesnt have to spend any time at looking at and compressing the background image.  (After talking to a friend, and doing some google searches, it turns this sort of thing is already an available feature in off-the-shelf consumer grade webcams.)

Mixing together the idea of the depth map and facial recognition software has many other interesting implications.

Faces all have the same basic depth map.  The webcam video stream could basically say "this is a face.  here are the transformations that turn a generic face into THIS face.  here are the stream of transformations over time that are THIS face changing over time".  Then interleaved with that, is a color image map of your "skinned" face, that the receiving side can then "wrap" over the 3D depth map.  And it can do such tricks as prioritize and do in higher resolution (in both space and time) the face's eyes, lips, and jaw, and maybe even do some work to sync the mouth motions with the audio data.

This makes other neat tricks possible as well.  By distorting the face model and the unwrapped color image in various ways, you could make yourself look thinner, fatter, a different gender, a different race, a fictional face (elf, dwarf, na'avi, etc) or even look like some other specific person.  This would be useful for amusement, enertainment, deception, privacy, and corporate branding.

Imagine calling up a corporate customer service rep on a video call, and no matter who you actually talk to, they all look like the same carefully branded innoffensive person.  Or even possibly based on your customer record and call data geolocation data, the CSR face looks like the race/ethnicity of your local area.

Another useful shortcut would be that once someone's specific face specification is generated, it can be given a specific unique id.  At some time in the future, in another video call, just the ID could be sent, and if the receiver has seen it before, the full face definition does not have to be sent.   This would work for more things than faces.  Whenever ANY definable object is seen and characterized, it gets an ID, and is opportunistically cached by the receiver.

All this also can work with the other implications of linking facial recognition with augumented reality and lifestreaming that I've mused about in a previous post.