To Escape the Multicore Crisis, Go Out Not Up
Posted by Bob Warfield on September 29, 2007
Of course, you should never go up in a burning building, go out instead. Amazon’s Werner Voegels sees the Multicore Crisis in much the same way:
Only focusing on 50X just gives you faster Elephants, not the revolutionary new breeds of animals that can serve us better.
Voegels is writing there about Michael Stonebreaker’s claims that he can demonstrate a database architecture that outperforms conventional databases by a factor of 50X. Stonebreaker is no one to take lightly: he’s accomplished a lot of innovation in his career so far and he isn’t nearly done. He advocates replacing the Oracle (and mySQL) style databases (which he calls legacy databases) with a collection of special purpose databases that are optimized for particular tasks such as OLTP or data warehousing. It’s not unlike the concept myself and others have talked about that suggests that the one-language-fits-all paradigm is all wrong and you’d do better to adopt polyglot programming.
I like Stonebreaker’s work. While I want the ability to scale out to any level that Voegels suggests, I will take the 50X improvement as a basic building block and then scale that out if I can. That’s a significant scaling factor even looked at in the terms of the Multicore Language Timetable. It’s nearly 8 years of Moore’s Cycles. I’m also mindful that databases are the doorway to the I/O side of the equation which is often a lot harder to scale out. Backing an engine that’s 50X faster sucking the bits off the disk with memcached ought to lead to some pretty amazing performance.
But Voegels is right, in the long term we need to see different beasts than the elephants. It was with that thought in mind that I’ve been reading with interest articles about Sequoia, an open source database clustering technology that makes a collection of database servers look like one more powerful server. It can be used to increase performance and reliablity. It’s worth noting that Sequoia can be installed for any Java app using JDBC without modifying the app. Their clever monicker for their technology is RAIDb: Redundant Array of Inexpensive Databases. There are different levels of RAIDb just as there are RAID levels that allow for partitioning, mirroring, and replication. The choice of level or combinations of levels governs whether your applications gets more performance, more reliability, or both.
Sequoia is not a panacea, but for some types of benchmarks such as TPC-W, it shows a nearly linear speedup as more cpus are added. It seems likely a combination of approaches such as Stonebreaker’s specialized databases for particular niches and clustering approaches like Sequoia all running on a utility computing fabric such as Amazon’s EC2 will finally break the multicore logjam for databases.
4 Responses to “To Escape the Multicore Crisis, Go Out Not Up”
You must log in to post a comment.
stevehpervasivecom said
It may not be for everyone, but Java developers out there that are having to build high-performance data processing or data analysis Java apps should look at our beta framework at http://www.pervasivedatarush.com
It’s a Java framework for data-intensive applications (not transactional like J2EE, but rather bulk data management) that handles horizontal, vertical and pipeline parallelism on multicore platforms. You don’t have to use any Java NIO or concurrency API’s — rather, it uses the dataflow (http://en.wikipedia.org/wiki/Dataflow) or flow-based computing (http://en.wikipedia.org/wiki/Flow-based_programming) approach.
There’s a free download of the framework at the website. I look forward to comments on this blog.
Thanks!
Christian Fauré » Blog Archive » “Web of Data” n’est pas “Data on the Web” said
[…] au SaaS est flou et risqué. En guise de réponse, et de conclusion de cette note, je paraphraserai Bob Warfield en rappellerant que : ” il ne faut jamais monter l’escalier d’un immeuble qui […]
ESB vs REST (Another Case for Multi-Language Programming) « SmoothSpan Blog said
[…] Posted by smoothspan on October 9th, 2007 Steve Vinoksy recently threw a lit match into the proverbial dynamite shack with his blog post, “The ESB Question“. Steve’s background and credentials (he was a very senior player at Iona) are impeccable, but a lot of folks had a hard time swallowing what he had to say. ESB’s, or Enterprise Service Busses are one approach to creating a Service Oriented Architecture (SOA). The idea behind wanting to use ESB’s to create SOA’s is the desire to enable software to be componentized. Interestingly, this is also wrapped up to a certain extent (but far from complete overlap!) with the question of how to break down an application into components that can operate across many cpus in a distributed network: shades of the multicore crisis again! […]
From the Dept of “That is so cool!” « SmoothSpan Blog said
[…] performance improvements for your computer may once again be possible. Here’s one the multicore crisis doesn’t seem to have screwed up. Cool […]