Mass. town makes it easier for cops to hand out $20 fines for public profanity

Problem: kids are hanging out in public places and swearing causing a public nuisance. Solution: impose a$20 fine. Really? Want to try and find out why they are hanging out in the public places first? Maybe they don’t have enough to do, like homework, a job, a home to go to?

Top 10 Quotes from XSITE 2012: Bill Warner, Andy Ory, Phil Libin, & More I was only able to stay for the first three speakers, Evernote, Watson, and Birchbox. Phil Libin’s talk was awesome especially seeing the graphs showing user count and cohort analysis.

HP Reveals Post-Acquisition Plans for Autonomy

HP’s rationalization of its business strategy choices and acquisitions.

Interesting, but will my printer work any better next year? I guess we don’t know yet …

Battery Power Drives Screen Size An interesting, but when you think about it an obvious thing. You have density, size and thickness. Since we seem to want thin, and density (watts-hours per cubic inch) is growing at its own rate, you only have size to play with!

Comparing parallel databases to sharding

I just posted an article comparing parallel databases to sharding on the ParElastic blog at

It was motivated by the fact that I’ve been asked a couple of times recently how the ParElastic architecture compares with sharding and it occurred to me this past weekend that

“Parallel Database” is a database architecture but sharding is an application architecture

Read the entire blog post here:

Scaling MongoDB: A year with MongoDB (Engineering at KiiP)

Here is the synopsis:

  • A year with MongoDB in production
  • Nine months spent in moving 95% of the data off MongoDB and onto PostgreSQL

Over the past 6 months, we’ve “scaled” MongoDB by moving data off of it.

Read the complete article here:


Laptop vs Mobile (from Fred Wilson’s blog)

Interesting post. Now that I’m traveling more, I do face this dilemma.

The tablet (I’m using to post this) doesn’t cut it as a travel device. Not does the droid x2, nor my netbook. But since the option is straining my back and lugging a laptop along, I am making do with a netbook when I travel.

A universal dock in hotel rooms would be great. Better voice transcription would make devices better, even this tablet!

Old memories … a foggy day on Memorial Day Weekend ’97

Was thinking about an old picture I took in 1997.

It is on flickr …

I had copies of this picture made and I used to have it hang on my office wall just as it is above. Look carefully, it is upside down … How often have you seen a duck swimmg in the air!

SQL, NoSQL, NewSQL and now and new term …


Read all about it at C Mohan’s blog (

Mohan knows a thing or two about databases. As a matter of fact, keeping track of his database related achievements, publications and citations is in itself a big-data problem.

NonsenSQL, read all about it!


Strike 3 for #ATT … and I thought #verizon was bad ….

Take a read of what it is like to have a windows phone on AT&T …

Here I was unhappy at Verizon for not updating my two year old tablet to ice cream sandwich.

Looks like the carriers and their interference in the timely release of software is a serious problem worth considering some more during the buying decision.

Revisiting Network I/O APIs: The netmap Framework

Great article on High Scalability entitled Revisiting Network I/O APIs: The netmap Framework. Get the paper they reference here.

As network performance continues, the bottleneck will become the amount of time spent in moving packets between the wire (hardware) and the application (software) and vice versa. The netamp framework is an interesting approach to address this.

if this then that (@ifttt)

Great service called “if this then that” (ifttt). Allows you to create tasks based on specific triggers from one of about forty channels.

I’m thrilled that ‘starring an item’ in Google Reader is a channel.

When a trigger occurs, you can have ifttt generate a specified action.

I have one …

When I post this article, it will be automatically tweeted … Very cool, check them out!

My blog is all f’ed up

My blog has been all f’ed up for some time now, and I didn’t realize it. I’ve been reading stuff and tagging it on my tablet and in the past that used to make it pop up in an RSS feed that was displayed on my blog as ‘breadcrumbs’. But, somewhere along the way, all that fell apart.

  • Maybe it was because something changed in the way the links were shared.
  • Maybe it was because the the ‘unofficial’ client that I was using didn’t really work and therefore nothing made it to and therefore to the RSS feed.
  • And Gimmebar did one thing, and they did it well. But they didn’t do the next thing they promised (an android app).

So, from about November 2011 when Google went and wrecked Google Reader by eliminating the ‘share this’ functionality till today, all the stuff I’ve read and thought I shared is gone …

Time to use twitter as the sharing system. That seems to work. I don’t like it, but it will have to do for now.

Cloud CPU Cost over the years

Great article by Greg Arnette about the crashing cost of CPU Costs over the years, thanks to the introduction of the cloud.

Personally, I think the most profound one was in December 2009 with the introduction of “spot pricing”.

Effectively you have an auction for the cost of an instance at any time and so long as the prevailing price is lower than the price you are willing to pay, you get to keep your instance.

Do one thing, and do it awesomely … Gimmebar!

From time to time you see a company come along that offers a simple product or service, and when they launch it just works.

The last time (that I can recall) when this happened was when I first used Dropbox. Download this little app and you got a 2GB drive in the cloud. And it worked on my Windows PC, on my Ubuntu PC, on my Android phone.

It just worked!

That was a while ago. And since then I’ve installed tons of software (and uninstalled 99% of it because it just didn’t work).

Last week I found Gimmebar.

There was no software to install, I just created an account on their web page. And it just worked!

What is Gimmebar? They consider themselves the 5th greatest invention of all time and they call themselves “a data steward”. I don’t know what that means. They also don’t tell you what the other 4 inventions are.

Here is how I would describe Gimmebar.

Gimmebar is a web saving/sharing tool that allows you to save things that you find interesting on the web in a nicely organised personal library in the cloud, and share some of that content with others if you so desire. They have something about saving stuff to your Dropbox account but I haven’t figured all of that out yet.

It has a bookmarklet for your browser, click it and things just get bookmarked and saved into your account.

But, it just worked!

I made a couple of collections, made one of them public and one of them shared.

If you share a collection it automatically gets a URL.

And that URL automatically supports an RSS Feed!

And they also backup your tweets, (I don’t give a crap about that).

So, what’s missing?

  • Some way to import all your stuff (from Google Reader)
  • An Android application (more generally, mobile application for platform of choice …)
  • The default ‘view’ on the collections includes previews; I will have enough crap before long where the preview will be a drag. How about a way to get just a list?
  • Saving a bookmark is right now at least a three click process; once you visit the site, click the bookmarklet and you get a little banner on the bottom of the screen, you click there to indicate whether you want the page to go to your private or public area, then you click the collection you want to store it in. This is functional but not easy to use.

I had one interaction with their support (little feedback tab on their page). Very quick to respond and they answered my question immediately.

On the whole, this feels like my first experience with Dropbox. Give it a shot, I think you’ll like it.

Why? Because Gimmebar set out to do one thing and they did it awesomely. It just worked!

The MongoDB rant. Truth or hoax?

Two days ago, someone called ‘nomoremongo’ posted this on Y Combinator News.

Several people (me included) stumbled upon the article, read it, and took it at face value. It’s on the Internet, it’s got to be true, right?

No, seriously. I read it, and parts of it resonated with my understanding of how MongoDB works. I saw some of the “warnings” and they seemed real. I read this one (#7) and ironically, this was the one that convinced me that this was a true post.

**7. Things were shipped that should have never been shipped**

Things with known, embarrassing bugs that could cause data
problems were in "stable" releases--and often we weren't told
about these issues until after they bit us, and then only b/c
we had a super duper crazy platinum support contract with 10gen.

The response was to send up a hot patch and that they were
calling an RC internally, and then run that on our data.


Who but a naive engineer would feel this kind of self-righteous outrage 😉 I’ve shared this outrage at some time in my career, but then I also saw companies ship backup software (and have a party) when they knew that restore couldn’t possibly work (yes, a hot patch), software that could corrupt data in pretty main stream circumstances (yes, a hot patch before anyone installed stuff) etc.,

I spoke with a couple of people who know about MongoDB much better than I do and they all nodded about some of the things they read. The same article was also forwarded to me by someone who is clearly knowledgeable about MongoDB.

OK, truth has been established.

Then I saw this tweet.

Which was odd. Danny doesn’t usually swear (well, I’ve done things to him that have made him swear and a lot more but that was a long time ago). Right Danny?




Well, he had me at the “Start thinking for yourself”. But then he went off the meds, “MongoDB is the next MySQL”, really …


I think there’s a kernel of truth in the MongoDB rant. And it is certainly the case that a lot of startups are making dumb architectural decisions because someone told them that “MongoDB was web-scale”, or that “CAP Theorem told them that databases were dead”.

Was this a hoax? I don’t know. But it was certainly a reminder that all scams don’t originate in Nigeria, and don’t begin by telling me that I could make a couple of billion dollars if I just put up and couple of thousand.

On migrating from Microsoft SQL Server to MongoDB

Just reading this article describing one companies experiences migrating from SQL Server to MongoDB.

Having read the article, my only question to these folks is “why do it”?

Let’s begin by saying that we should discount all one time costs related to data migration. They are just that, one time migration costs. However monumental, if you believe that the final outcome is going to justify it, grin and bear the cost.

But, once you are in the (promised) MongoDB land, what then?

The things that this author believes that they will miss are:

  • maturity
  • tools
  • query expressiveness
  • transactions
  • joins
  • case insensitive indexes on text fields

Really, and you would still roll the dice in favor of a NoSQL science project. Well, then the benefits must be really really awesome! Let’s go take a look at what those are. Let’s take a look at what those are:

  • MongoDB is free
  • MongoDB is fast
  • Freedom from rigid schemas
  • ObjectID’s are expressive and handy
  • GridFS for distributed file storage
  • Developed in the open

OK, I’m scratching my head now. None of these really blows me away. Let’s look at these one at a time.

  • MongoDB is free
  • So is PostgreSQL and MySQL
  • MongoDB is fast
    • So are PostgreSQL and MySQL if you put them on the same SSD and multiple HDD’s like you claim you do with MongoDB
  • Freedom from rigid schemas
    • I’ll give you this one, relational databases are kind of “old school” in this department
  • ObjectID’s are expressive and handy
    • Elastic Transparent Sharding schemes like ParElastic overcome this with Elastic Sequences which give you the same benefits. A half-way decent developer could do this for you with a simple sharded architecture.
  • GridFS for distributed file storage
    • Replication anyone?
  • Developed in the open
    • Yes, MongoDB is free and developed in the open like a puppy is “free”. You just told us all the “costs” associated with this “free puppy”

    So really, why do people use MongoDB? I know there are good circumstances where MongoDB will whip the pants off any relational database but I submit to you that those are the 1%.

    To this day, I believe that the best description of MongoDB is this one:

    Mongo DB is web scale
    by: gar1t

    Google went and broke Google Reader!

    A very nice feature of Google Reader (my RSS reader of choice) was that there was a simple button at the bottom of each article called “Share”, and the current URL would be added to a list of shared articles and an RSS feed could be created of that list!

    The breadcrumbs feature on my web page relied on that; as I read things, if I wanted to make them show up in breadcrumbs, all I did was to hit the Share button. If I visited some random URL and wanted to share that, I used the “Note in Reader” bookmarklet. All very good. Till Google went and broke it.

    Now all I get is this:

    This sucks!

    Others seem to have noticed this as well. A collection of related news:

    Trick or treat the google reader changes are coming tonight

    Save the Google Reader petition

    Upcoming changes


    Database scalability myth (again)

    A common myth that has been perpetrated is that relational database do not scale beyond two or three nodes. That, and the CAP Theorem are considered to be the reason why relational databases are unscalable and why NoSQL is the only feasible solution!

    I ran into a very thought provoking article that makes just this case yesterday. You can read that entire post here. In this post, the author Srinath Perera provides an interesting template for choosing the data store for an application. In it, he makes the case that relational databases do not scale beyond 2 or 5 nodes. He writes,

    The low scalability class roughly denotes the limits of RDBMS where they can be scaled by adding few replicas. However, data synchronization is expensive and usually RDBMSs do not scale for more than 2-5 nodes. The “Scalable” class roughly denotes data sharded (partitioned) across many nodes, and high scalability means ultra scalable systems like Google.

    In 2002, when I started at Netezza, the first system I worked on (affectionately called Monolith) had almost 100 nodes. The first production class “Mercury” system had 108 nodes (112 nodes, 4 spares). By 2006, the systems had over 650 nodes and more recently much larger systems have been put into production. Yet, people still believe that relational databases don’t scale beyond two or three nodes!

    Systems like ParElastic (Elastic Transparent Sharding) can certainly scale to much more than two or three nodes, and I’ve run prototype systems with upto 100 nodes on Amazon EC2!

    Srinath’s post does contain an interesting perspective on unstructured and semi-structured data though, one that I think most will generally agree with.

    All you ever wanted to know about the CAP Theorem but were scared to ask!

    I just posted a longish blog post (six parts actually) about the CAP Theorem at the ParElastic blog.


    Embracing opposing points of view

    In a blog aptly called “Both Sides of the Table”, I read a post entitled “Why You Should Embrace Opposing Views at Your Startup”.

    It is a great post by Mark Suster and I highly recommend you read it.

    If your startup lives in an echo chamber, and the only voice you hear is your own, it is most certainly doomed.

    Dayton, OH embraces immigrants, cites entrepreneurship as a big draw

    Bucking a national trend, Dayton Ohio has taken the bold step to welcome immigrants. They published a comprehensive 32 page report describing the program that was approved some days ago.

    Here are some quotes that I read that I found encouraging.

    According to the city, immigrants are two times more likely than others to become entrepreneurs.


    1. Focus on East Third Street, generally between Keowee and Linden, as an initial international market place for immigrant entrepreneurship. East Third Street, in addition to being a primary thoroughfare between Downtown and Wright Patterson Air Force Base, also encompasses an area of organic immigrant growth and available space to supportcontinuing immigrant entrepreneurship.


    2. Create an inclusive community-wide campaign around immigrant entrepreneurship that facilitates startup businesses, opens global markets and restores life to Dayton neighborhoods.

    Other coverage of this and related issues can also be found here:

    Running databases in virtualized environments

    I have long believed that databases can be successfully deployed in virtual machines. Among other things, that is one of the central ideas behind ParElastic, a start-up I helped launch earlier this year. Many companies (Amazon, Rackspace, Microsoft, for example) offer you hosted databases in the cloud!

    But yesterday I read this post in RWW. This article talks about a report published by Principled Technologies in July 2011, a report commissioned by Intel, that

    tested 12 database applications simultaneously – and all delivered strong and consistent performance. How strong? Read the case study, examine the results and testing methodology, and see for yourself.

    Unfortunately, I believe that discerning readers of this report are more likely to question the conclusion(s) based on the methodology. What do you think?

    A Summary of the Principled Technologies Report

    In a nutshell, this report seeks to make the case that industry standard servers with virtualization can in fact deliver the performance required to run business critical database applications.

    It attempts to do so by running Vware vSphere 5.0 on the newest four socket Intel Xeon E7-4870 based server and hosting 12 database applications each of which has an 80GB database in its own virtual machine. The Intel Xeon E7-4870 server is a 10 core processor with two hardware threads per core. It was clocked at 2.4GHz and 1TB of RAM (64 modules each of which had 16GB). The storage in this server was 2 disks, each of which was 146GB in size (10k SAS). In addition, an EMC Clarriion Fibre Channel SAN with some disks configured in RAID0. In total they configured 6 LUN’s each of which was 1066GB (over a TB each). They VM’s ran Windows Server 2008 R2, and SQL Server 2008 R2.

    The report claims that the test that was performed was “Benchmark Factory’s TPC-H like workload”. Appendix B somewhat (IMHO) misleadingly calls this “Benchmark Factory TPC-H score”.

    The result is that these twelve VM’s running against an 80GB database were able to consistently process in excess of 10,000 queries per hour each.

    A comparison is made to the Netezza whitepaper that claims that the TwinFin data warehouse appliance running the “Nationwide Financial Services” workload was able to process around 2,500 queries per hour and a maximum of 10,000 queries per hour.

    The report leaves the reader to believe that since the 12 VM’s in the test ran consistently more than 10,000 queries per hour, business critical applications can in fact be deployed in virtualized environments and deliver good performance.

    The report concludes therefore that business critical applications can be run on virtualized platforms, deliver good performance, and reduce cost.

    My opinion

    While I entirely believe that virtualized database servers can produce very good performance, and while I entirely agree with the conclusion that was reached, I don’t believe that this whitepaper makes even a modestly credible case.

    I ask you to consider this question, “Is the comparison with Netezza running 2,500 queries per hour legitimate?”

    Without digging too far, I found that the Netezza whitepaper talks of a data warehouse with “more than 4.5TB of data”, 10 million database changes per day, 50 concurrent users at peak time and 10-15 on an average. 2,500 qph with a peak of 10k qph at month end, 99.5% completing in under one minute.

    Based on the information disclosed, this comparison does not appear to be valid. Note well that I am not saying that this comparison is invalid, rather that the case has not been made sufficiently to justify it.

    An important reason for my skepticism is that when processing database operations like joins between two tables, doubling the data volume quadruples the amount of computation that may be required. If you are performing three table joins, doubling the data increases the computation involved may be as much as eight times. This is the very essence of the scalability challenge with databases!

    I get an inkling that this may not be a valid comparison when we look at Appendix B that states that the total test time was under 750 seconds in all cases.

    This feeling is compounded when I don’t see how many concurrent queries are run against each database. Single user database performance is a whole lot better and more predictable than multi-user performance. The Netezza paper specifically talks about the multi-user concurrency performance not the single-user performance.

    Reading very carefully, I did find a mention that a single server running 12 VM’s hosted the client(s) for the benchmark. Since ~15k queries were completed in under 750s, we can say that each query lasted about 0.05s. Now, those are really really short queries. Impressive but not what I would generally consider to be in the kinds of workloads that one would expect Netezza to be deployed. The Netezza report does clearly state that 99.5% completed in under one minute, which leads me to conclude that the queries being run in the subject benchmark are at least two orders of magnitude away!


    Virtualized environments like Amazon EC2, Rackspace, Microsoft Azure, and VMWare are perfectly capable of running databases and database applications.One need only look at Amazon RDS (now with MySQL and Oracle),, SQL Azure, and offerings like that to realize that this is in fact the case!

    However, this report fails to make a compelling case for this. By making a comparison to a different whitepaper and simply relating the results to the “queries per hour” in the other paper causes me to question the  methodology. Once readers question the method(s) used to reach a conclusion, they are likely to question the conclusion itself.

    Therefore, I don’t believe that this report achieves what it set out to do.


    You can get a copy of the white paper here, a link to scribd, or here, a link to the PDF on RWW.

    This case study references a Netezza whitepaper on concurrency, which you can get here. The Netezza whitepaper is “CONCURRENCY & WORKLOAD MANAGEMENT IN NETEZZA”, and prepared by Winter Corp and sponsored by Netezza.

    I have also archived copies of the two documents here and here.

    A link to the TPC-H benchmark can be found on the TPC web site here.


    In the interest of full disclosure, in the past I was an employee of Netezza, a company that is referenced in this report.

    High Availability and Fault Tolerance

    This article on HA and FT at ReadWriteWeb caught my eye. A while ago I used to work at Stratus and it is not often that I hear their name these days. Stratus’ Fault Tolerant systems achieve their impressive uptime by hardware redundancy.

    In very simple terms, if the probability of some component or sub-system failure is p, then the probability of two failures at the same time is a much smaller p * p.

    When I was at Stratus, we used to guarantee “five nines”, or an uptime of 99.999% on systems that ran credit card networks, banking systems, air traffic control systems, and so on. Systems where the cost of downtime could be measured either in hundreds of thousands or millions of dollars an hour, or in human lives potentially lost.

    Before I worked at Stratus, I used to work for a Stratus Customer and my first experience with Fault Tolerance was when I received a box in the mail with a note that said something to the effect that a CPU board had failed in one of our systems (about a month ago), so please pop that board out and put this replacement board in its place.

    And we hadn’t realized it, the system had been chugging along just fine!

    So what does uptime % translate to in terms of hours and minutes?

    99% uptime : 3.65 days of downtime per year

    99.9% uptime: 8.76 hours of downtime per year

    99.99% uptime: 52.56 minutes of downtime per year

    99.999% uptime: 5.256 minutes of downtime per year

    Stratus claims that across its customer base of 8000 servers the uptime is 99.9998%

    99.9998% uptime: 63 seconds of downtime per year.

    Now, that’s pretty awesome!

    And when I flew into Schipol Airport, or saw containers being loaded onto ships in Singapore, or I used my American Express Credit Card, logged into AOL, or looked at my 401(k) on Fidelity, I felt pretty darn proud of it!

    Oracle’s new NoSQL announcement

    Oracle’s announcement of a NoSQL solution at Oracle Open World 2011 has produced a fair amount of discussion. Curt Monash blogged about it some days ago, and so did Dan Abadi. A great description of the new offering (Dan credits it to Margo Seltzer) can be found here or here. I think the announcement, and this whitepaper do in fact bring something new to the table that we’ve not had until now.

    1. First, the Oracle NoSQL solution extends the notion of configurable consistency in a surprising way. Solutions so far had ranged from synchronous consistency to eventual consistency. But, all solutions did speak of consistency at some point in time. Eventual consistency has been the minimum guarantee of other NoSQL solutions. The whitepaper referenced above makes this very clear and characterizes this not in terms of consistency but durability.

    Oracle NoSQL Database also provides a range of durability policies that specify what guarantees the system makes after a crash. At one extreme, applications can request that write requests block until the record has been written to stable storage on all copies. This has obvious performance and availability implications, but ensures that if the application successfully writes data, that data will persist and can be recovered even if all the copies become temporarily unavailable due to multiple simultaneous failures. At the other extreme, applications can request that write operations return as soon as the system has recorded the existence of the write, even if the data is not persistent anywhere. Such a policy provides the best write performance, but provides no durability guarantees. By specifying when the database writes records to disk and what fraction of the copies of the record must be persistent (none, all, or a simple majority), applications can enforce a wide range of durability policies.

    2. It sets forth a very specific set of use-cases for this product.There has been much written by NoSQL proponents about its applicability in all manners of data management situations. I find this section of the whitepaper to be particularly fact based.

    The Oracle NoSQL Database, with its “No Single Point of Failure” architecture is the right solution when data access is “simple” in nature and application demands exceed the volume or latency capability of traditional data management solutions. For example, click-stream data from high volume web sites, high-throughput event processing, and social networking communications all represent application domains that produce extraordinary volumes of simple keyed data. Monitoring online retail behavior, accessing customer profiles, pulling up appropriate customer ads and storing and forwarding real-time communication are examples
    of domains requiring the ultimate in low-latency access. Highly distributed applications such as real-time sensor aggregation and scalable authentication also represent domains well-suited to Oracle NoSQL Database.

    Several have also observed that this position is in stark contrast to Oracle’s previous position on NoSQL. Oracle released a whitepaper written in May 2011 entitled “Debunking the NoSQL Hype”. This document has been removed from Oracles website. You can, however, find cached copies all over the internet. Ironically, the last line in that document reads,

    Go for the tried and true path. Don’t be risking your data on NoSQL databases.

    With all that said, this certainly seems to be a solution that brings an interesting twist to the NoSQL solutions out there, if nothing else to highlight the shortcomings of existing NoSQL solutions.

    [2011-10-07] Two short updates here.

    1. There has been an interesting exchange on Dan Abadi’s blog (comments) between him and Margo Seltzer (the author of the whitepaper) on the definition of eventual consistency. I subscribe to Dan’s interpretation that says that perpetually returning to T0 state is not a valid definition (in the limit) of eventual consistency.
    2. Some kind soul has shared the Oracle “Debunking the NoSQL Hype” whitepaper here. You have to click download a couple of times and then wait 10 seconds for an ad to complete.

    A brief hiatus, and we’re back!

    Since early last year when I posted my last blog entry, I’ve been a bit “preoccupied”. Around that time, I started in earnest on getting a start-up off the ground. It was a winding road, and I did not get around to writing anything on this blog. Over the past several months, I have been resurrecting this blog.

    The old blog (there’s still a shell there) was called Hypecycles ( and try as I might, I could not get for this blog.

    What’s with “Pizza and Code”?

    The last eighteen or so months have been spent getting ParElastic off the ground. The quintessential startup is two guys working in the garage, and subsisting on Pizza! The software startup is therefore two things, Pizza and Code!

    What’s ParElastic?

    ParElastic is a startup that is building elastic database middleware for the cloud. Want to know more about ParElastic? Go to Starting ParElastic has been an incredible education, one that can only be acquired by actually starting a company.

    Over the next couple of blog posts, I will quickly cover the two or so years from mid 2009 to the present.



    It is 2010 and RAID5 still works …

    Some years ago (2007, 2008) when I cared a little more about things like RAID and RAID recovery, I read an article in ZDNET by Robin Harris that made the case for why disk capacity increases coupled with an almost invariant URE (Unrecoverable Read Error) rate meant that RAID5 was dead in 2009. A follow-on article appeared recently, also by Robin Harris that extends the same logic and claims that RAID6 would stop working in 2019.

    The crux of the argument is this. As disk drives have become larger and larger (approximately doubling in two years), the URE has not improved at the same rate. URE measures the frequency of occurrence of an Unrecoverable Read Error and is typically measured in errors per bits read. For example an URE rate of 1E-14 (10 ^ -14) implies that statistically, an unrecoverable read error would occur once in every 1E14 bits read (1E14 bits = 1.25E13 bytes or approximately 12TB).

    Further, Robin argues that a RAID array (RAID5 or RAID6) is running normally when a drive suffers a catastrophic failure that prompts a reconstruction from parity. In that scenario, it is perfectly conceivable that while reading the (N-1) data drives and the parity stripe in order to rebuild the failed data drive, a single URE may occur. That URE would render the RAID volume failed.

    The argument is that as disk capacities grow, and URE rate does not improve at the same rate, the possibility of a RAID5 rebuild failure increases over time. Statistically he shows that in 2009, disk capacities would have grown enough to make it meaningless to use RAID5 for any meaningful array.

    So, in 2007 he wrote:

    RAID 5 protects against a single disk failure. You can recover all your data if a single disk breaks. The problem: once a disk breaks, there is another increasingly common failure lurking. And in 2009 it is highly certain it will find you.

    and in 2009, he wrote:

    SATA RAID 6 will stop being reliable sooner unless drive vendors get their game on. More good news: one of them already has.

    The logic proposed is accurate but, IMHO, incomplete. One important aspect that the analysis fails to point out is something that RAID vendors have already been doing for many years now.

    Image courtesy of

    When disk drives looked like this (picture at right), the predominant failure mode was the catastrophic failure. Drives either worked or didn’t work any longer. At some level, that was a reflection of the fact that the Drive Permanent Failure (DPF) frequency was significantly higher than the URE frequency, and therefore the only observed failure mode was catastrophic failure.

    As drives got bigger, and certainly in 1988 when Patterson and others first proposed the notion of RAID, it made perfect sense to wait for a DPF and then begin drive reconstruction. The possibility of a URE was so low (given drive capacities) that all you had to worry about was the rebuild time, and the degraded performance during the rebuild (as I/O’s may have to be satisfied through block reconstruction).

    But, that isn’t how most RAID controllers today deal with drive URE’s and drive failures. On the contrary, for some time now, RAID controllers (at least the recent ones I’ve read about) have used better methods to determine when to perform the rebuild.

    A 5400 RPM SATA DriveConsider this alternative, that I know to be used by at least a couple of array vendors. When a drive in a RAID volume reports a URE, the array controller increments a count and satisfies the I/O by rebuilding the block from parity. It then performs a rewrite on the disk that reported the URE (potentially with verify) and if the sector is bad, the microcode will remap and all will be well.

    When the counter exceeds some threshold, and with the disk that reported the URE still in a usable condition, the RAID controller will begin the RAID5 recovery. Robin is correct that RAID recovery after DPF is something that will become less and less useful as drive capacities grow. But, with improvements in integration of SMART and the significant improvements in the predictability of drive failures, the frequency of RAID5 and RAID6 reconstruction failures are dramatically lower than those predicted in the referenced articles as these reconstructions occur on URE and not DPF.

    Look at the specifications for the RAID controller you use.

    When is RAID recovery initiated? Upon the occurrence of an Unrecoverable Read Error (URE) or upon the occurrence of a Drive Permanent Failure (DPF)?

    Several have proposed ZFS with multiple copies is the way to go. While it addresses the issue, I submit to you that it is at the wrong level of the stack. Mirroring at the block level, with the option to have multiple mirrors is the correct (IMHO) solution. Disk block error recovery should not be handled in the file system.

    The Application Marketplace: Android’s worst enemy?

    I recently got an Android (Motorola A855, aka droid) phone. I had been using a Windows based device (have been since about 2003). I was concerned about the bad reviews of poor battery life and the fact that Bluetooth Voice Dialing was not present. I figured that the latter was a software thing and could be added later. So, with some doubt, I started using my phone.

    On the first day, with a battery charged overnight,  I proceeded to surf the Marketplace and download a few applications. I got a Google Voice Dialer (not the one from Google), and a couple of other “marketplace” applications. I used the maps with the GPS for a short while and in about 8 hours the yellow sign of “low battery” came on. I had Google (GMAIL) synchronization set to the default (sync enabled).

    Pretty crappy, I thought. My Samsung went for two days without a problem. I had activesync with server (Exchange) or GMail refresh every 5 minutes for years!

    The Google Voice dialer I downloaded had some bugs (it messed up the call log pretty badly) and I got bored of the other applications I had downloaded.

    Time for a hard reset and restart for the phone (just to be sure I got rid of all the gremlins. After all, I was a Windows phone user, this was a weekly ritual).

    I got the update to Google Maps, set synch to continuous, downloaded the “sky map” application and charged the phone up fully. That was on Wednesday afternoon (17th). Today is the 20th and the battery is still all green on the home page.

    The robustness of downloaded Android Apps

    One of the things that makes the android phone so attractive (the application marketplace) is certainly a big problem. The robustness and stability of the downloaded applications cannot be guaranteed. We all realize that “your mileage may vary”. But, a quick look at the “Best Practices” on the android SDK site indicate that a badly written application can keep the CPU too busy and burn through your battery.

    Maybe Android phones (and the battery life in particular) is more an issue of poorly written applications.

    Apple (with the Macintosh) had a tight grip on the applications that could be released on the Mac. This helped them ensure that buggy software didn’t give the Mac a bad name. I’m sure Windows users can relate to this.

    They seem to have the same control on the iPhone App Store. Maybe that’s why I don’t hear so much about crappy applications on the iPhone that crash or suck the battery dry!

    Should Google take some control over the crap on the marketplace or will it all straighten itself out over time?

    Punishment must fit the crime

    I regularly read Dr. Dobbs Code Talk and noticed this article today. What caught my attention was not the article itself, but rather the first response to the article from Jack Woehr.

    Reproduced below is a screen shot of the page that I read and Jack’s comments. Really, I ask you, is C# all that bad?

    Microsoft patent 7617530, the flap about sudo

    The blogosphere has been buzzing with indignation about a Microsoft patent application 7617530 that apparently was granted earlier this month. You can read the application here.

    Yes, enough people have complained that this is like sudo and why did Microsoft get a patent for this. In fairness the patent does attempt to distinguish what is being claimed from sudo and provides copious references to sudo. What few have mentioned is that the thing that Microsoft patents is in fact the exact functionality that some systems like Ubuntu use to allow non-privileged users to perform privileged tasks.

    In PC Magazine, Matthew Murray writes,

    Because a graphical interface is not a part of sudo, it seems clear the patent refers to a Windows component and not a Linux one. The patent even references several different online sudo resources, further suggesting Microsoft isn’t trying to put anything over on anyone. The same section’s reference to “one, many, or all accounts having sufficient rights” suggests a list that sudo also doesn’t possess.

    IMHO, they may be missing something here.

    Let’s set that all aside. What I find interesting is this. The patent application states, and I reproduce three paragraphs of the patent application here and have highlighted three sentences (the first sentences in each paragraph).

    Standard user accounts permit some tasks but prohibit others. They permit most applications to run on the computer but often prohibit installation of an application, alteration of the computer’s system settings, and execution of certain applications. Administrator accounts, on the other hand, generally permit most if not all tasks.

    Not surprisingly, many users log on to their computers with administrator accounts so that they may, in most cases, do whatever they want. But there are significant risks involved in using administrator accounts. Malicious code may, in some cases, perform whatever tasks are permitted by the account currently in use, such as installing and deleting applications and files–potentially highly damaging tasks. This is because most malicious code performs its tasks while impersonating the current user of the computer–thus, if a user is logged on with an administrator account, the malicious code may perform dangerous tasks permitted by that account.

    To reduce these risks, a user may instead log on with a standard user account. Logging on with a standard user account may reduce these risks because the standard user account may not have the right to permit malicious code to perform many dangerous tasks. If the standard user account does not have the right to perform a task, the operating system may prohibit the malicious code from performing that task. For this reason, using a standard user account may be safer than using an administrator account.

    Absolutely! Most people don’t realize that they are logged in as users with Administrator rights and can inadvertently do damaging things.

    My question is this: why is the default user created when you install Windows on a PC an administrator user? As you go through the install process, the thing asks you questions like “what is your name” and “how would you like to login to your PC”. It uses this to setup the first user on the machine. Why is that user an administrator user?

    If you are smart (and if Microsoft really wanted to be good about this) the installation process would create two users. A day-to-day user who is non-Administrator, and an Administrator user.

    I’m a PC and if Windows 8 comes up with an installation process that creates two users, a non-administrator user and an administrator user, then it would have been my idea. But, I don’t intend to go green holding my breath for this to happen. Someone tell me if it does.

    A Report from Boston’s First “Big Data Summit”

    A short write-up about last night’s Big Data Summit appeared on xconomy today.

    My thanks to our sponsors, Foley Hoag LLP and the wonderful team at the Emerging Enterprise Center, Infobright, Expressor Software, and Kalido.

    Boston Big Data Summit Kickoff, October 22nd 2009

    BBD_logoSince the announcement of the Boston Big Data Summit on the 2nd of October, we have had a fantastic response. The event sold out two days ago. We figured that we could remove the tables from the room and accommodate more people. And, we sold out again. The response has been fantastic!

    If you have registered but you are not going to be able to attend, please contact me and we will make sure that someone on the waiting list is confirmed.

    There has been some question about what “Big Data” is. Curt Monash who will be delivering the keynote and moderating the discussion at the event next week writes:

    … where “Big Data” evidently is to be construed as anything from a few terabytes on up.  (Things are smaller in the Northeast than in California …)

    Little FishBig FishWhen you catch a fish (whether it is the little fish on the left or the bigger fish on the right), the steps to prepare it for the table are surprisingly similar. You may have more work to do with the big fish and you may use different tools to do it with; but the things are the same.

    So, while size influences the situation, it isn’t only about the size!

    In my opinion, whether data is “Big” or not is more of a threshold discussion. Data is “Big” if the tools and techniques being used to acquire, cleanse, pre-process, store, process and archive, are either unable to keep up, or are not cost effective.

    Yes, everything is bigger in California, even the size of the mess they are in. Now, that is truly a “Big Problem”!

    The 50,000 row spreadsheet, the half a terabyte of data in SQL Server, or the 1 trillion row table on a large ADBMS are all, in their own ways, “Big Data” problems.

    The user with 50k rows in Excel may not want  ( or be able to afford ) a solution with a “real database”, and may resort to splitting the spreadsheet into two sheets. The user with half a terabyte of SQL Server or MySQL data may adopt some home-grown partitioning or sharding technique instead of upgrading to a bigger platform, and the user with a trillion CDR’s may reduce the retention period; but they are all responding to the same basic challenge of “Big Data”.

    We now have three panelists:

    It promises to be a fun evening.

    I have some thoughts on subjects for the next meeting, if you have ideas please post a comment here.

    On MapReduce and Relational Databases – Part 1

    Describes MapReduce and why WOTS (Wart-On-The-Side) MapReduce is bad for databases.

    This is the first of a two-part blog post that presents a perspective on the recent trend to integrate MapReduce with Relational Databases especially Analytic Database Management Systems (ADBMS).

    The first part of this blog post provides an introduction to MapReduce, provides a short description of the history and why MapReduce was created, and describes the stated benefits of MapReduce.

    The second part of this blog post provides a short description of why I believe that integration of MapReduce with relational databases is a significant mistake. It concludes by providing some alternatives that would provide much better solutions to the problems that MapReduce is supposed to solve.
    Continue reading “On MapReduce and Relational Databases – Part 1”