Grids: The Wrong Answer to the Right Problem

This morning I was out walking and listening to a podcast. Now, mostly I feel podcasts are a very low bandwidth media and not a very good way to communicate complex content. However, they're a great mode of learning for the right circumstances -- commuting, exercising, ... actually, that pretty much exhausts the right circumstances to listen to podcasts. But for them, podcasts are perfect.

On this morning's walk, I was listening to a podcast by Thomas Kurian of Oracle from Software 2006. He was talking about Oracle's architecture; one of the topics he discussed was Oracle's use of its grid architecture as the underpinning of its Fusion offering.

He identified a key problem for users: the massive increase in software processing. There's no question that most organizations are using more and more software driven by:

• Applying software to a wider range of problems. Software processing has moved way beyond the traditional back-office accounting and ERP processing. Today companies are using software to design products, mine customer data, track their supply chain, and even more.
  
• Delivering data in an ever-increasing variety of ways: Web browsers, mobile devices, home offices, and even to other computer systems via SOA and the like.
• The so-called Flat World, which requires interacting with a free-wheeling, ever-changing set of partners, customers, and outsourced staff, all of which use computer systems and data to interact with an organization.

But, are grids the way most organizations will solve the problem of exploding computer processing and data? I don't think so. I think grids are a vendor solution looking to convince customers it will solve their problems.

Now, my experience of Oracle's grid is mostly the increased difficulty it adds to Oracle's already-notoriously challenging database installation. But, he noted, it offers the ability to increase scalability of customer infrastructures. Essentially, grids offers the ability to scale applications beyond the bounds of a single machine and provide more computing power than possible from one server. Kurian maintained that grid will be the foundation of how Oracle's users will write and roll out applications in the future. Perhaps.

Despite all the hype from vendors, user acceptance of grid computing has been lukewarm at best. The biggest problem is that grids require application designers to extend their skills in order to write systems capable of executing in a grid architecture; in other words, end users have to learn new things to take advantage of the power of grids.

Generally speaking, imposing new conditions on users in order to move to a new architecture is a non-starter. 

The requirement to break the bounds of the application/single machine foundation put me in mind of virtualization. While virtualization is usually thought of as a way to consolidate multiple servers onto a single box, several virtualization vendors have extended their products to create a virtualized pool of servers running on multiple machines.

With these products, you can declare a set of servers as a resource for virtual machines to run upon. If one physical server gets maxed out, new VMs can be started on a second physical server. The new VMs can be additional instances of an existing VM; the pooling products can even move a VM from  one server to another to  reduce load on a given server. Multiple VMs can work cooperatively on a common datastore via clustering.

It strikes me that this kind of server pooling is a much more likely path for IT organizations to achieve application scalability than rewriting applications to operate in grids. Enabling virtualization server pooling puts the burden of architecture on vendors rather than imposing it on users, as with grids. With virtualization pooling, vendors can make the necessary investment to create an easily extended architecture while users can continue to operate with their existing skill sets. The most expensive technology migration possible is building a new skill set in end users -- just look at the decade-long headache that was the move to object-oriented programming to understand the issue.

Now before you get all bothered and post a comment along the lines of "some apps require grid-type processing that virtualization just can't do," it's definitely true that pooled servers won't solve every problem. Video rendering, financial derivatives, and drug discovery all require massive, coordinated processing that necessitates breaking processing up into discrete chunks spread across a grid. For the most challenging applications, grid computing is perfect.

Organizations that require that level of application will make the skill investment to take advantage of grid computing -- but for the vast majority of organizations that do not require the specialized functionality of coordinated grid computing, the solution is likely to be pooled servers via virtualization. Instead of grid computing being a high performance, difficult architecture imposed upon everyone that needs more than a server's worth of processing, it will be used only by those organizations that truly need its capabilties.

For vendors to offer grid computing as the architecture of the future is very much reminiscent of a solution looking for a problem. It's right for some organizations, but overkill for most.