By the way, if your interested in developing for Live Mesh, there are some new videos posted on Microsoft Videos that provide an impressive amount of content. The RESTful services and the Pub/Sub model are of particular interest to me, since I think it will unleash a host of service aggregation possibilities in the future. The ability to ask the cloud to give you JSON or RSS without mapping or conversion is amazing.

I’m not saying, I’m just saying.

Summer Reading While The Boss Is Away

If you are elsewhere in the country in late August, chances are you are acting like Seattle folks do the rest of the year - taking refuge inside and away from the weather outside. While you are in your air conditioned home waiting for the last and hottest part of summer to pass, why not do some good reading to prepare for the fall when everyone returns from vacation and you get back to the serious business of deadlines, programming and of course geeky arguments about the topics of the day. Here is a good reading list to bookmark.

Get up to speed of Generic Programming, or Programming In General

My first recommendation is the collected papers of Alexander Stepanov, which you can get from his website entitled… Collected Papers of Alexander Stepanov. For those who don’t know, Stepanov is the key person behind the C++ Standard Template Library, which he started to develop around 1993. He had earlier been working for Bell Labs close to Andrew Koenig and tried to convince Bjarne Stroustrup to introduce something like Ada Generics in C++. His papers are a treasure of thought on generic programming, logic, robotics and anything else that made you turn to the Computer Science page in your university’s catalog. Best of all he also provides slides for his book in progress, written with Paul McJones, called Programming Elements. This is a great book for refreshing your knowledge of abstract and concrete concepts in quick and easy powerpoint format. Just take a look at the table of contents and I dare you not to click on at least one of the Chapter links. Don’t worry, I won’t tell.

The “Core” Debate of the Community: Concurrent Programming and Software Transaction Memory

Yes, the pun was bad. It does however illustrate one of the facets of the problem that is burning up academic and commercial researchers alike, and responsible for a large amount of papers flooding the ACM portal: Software Transactional Memory (STM). Well, actually, that’s a possible answer to the problem - not the problem itself. They are often confused now. The problem is that since Intel and AMD have decided to start introducing more cores on to single chip we have to deal with the big problem that comes along with that: managing the threads of multiple cores trying to do the same work on behalf of the system it’s working for. It also scales in to bigger problems of any type of work you may want to farm off to “locales” that may need to cross boundaries and work on the same data within a transaction (for more information on some of this, see my post from the Google Scalability Conference regarding Cray’s work to replace MPI with a new concurrent language Chapel and the GIGA+ filesystem below)

I think Simon Peyton Jones from Microsoft Research in Cambridge illustrates it best in his paper Composable Memory Transactions(PPOPP’05) :

The dominant programming technique is based on locks, an approach that is simple and direct, but that simply does not scale with program size and complexity. To ensure correctness, programmers must identify which operations conflict; to ensure liveness, they must avoid introducing deadlock; to ensure good performance, they must balance the granularity at which locking is performed against the costs of fine-grain locking. Perhaps the most fundamental objection, though, is that lock-based programs do not compose: correct fragments may fail when combined. For example, consider a hash table with thread-safe insert and delete operations. Now suppose that we want to delete one item A from table t1, and insert it into table t2; but the intermediate state (in which neither table contains the item) must not be visible to other threads. Unless the implementor of the hash table anticipates this need, there is simply no way to satisfy this requirement. Even if she does, all she can do is expose methods such as LockTable and UnlockTable – but as well as breaking the hash-table abstraction, they invite lock-induced deadlock, depending on the order in which the client takes the locks, or race conditions if the client forgets. Yet more complexity is required if the client wants to await the presence of A in t1, but this blocking behaviour must not lock the table (else A cannot be inserted). In short, operations that are individually correct (insert, delete) cannot be composed in to larger correct operations.

The most that has come out of this is that we know it’s a problem and we’d love to use the keyword “atomic” to wrap our transactional code in our languages. Beyond that, it’s a lot of hand waiving and Powerpoint slides. Some people though are actually trying to work it out. The best starting point here are the papers from the before mentioned researcher Simon Peyton Jones. His collection of papers on STM offers a good starting point of the problem and what some possible solutions are. In his papers he uses Haskell, and his work has led to Concurrent Haskell. Haskell lends itself to STM for reasons I won’t go in to here, but it will be quite a bit more of a challenge to get the same functionality in Java and C#, but there is already an API for C# Software Transactional Memory from Microsoft Research you may want to explore.

If you don’t care about this, just don’t go naming classes atomic and you should be fine.

Storing The Cloud: How Do We Scale?

Solid State (read: Flash) drives aren’t the only thing showing the age of our old file system technologies. As we expose software as services and begin taking on large numbers of tenants for our software, cloud computing needs clusters with thousands of nodes that, with the multi-core technology mentioned above, will impose a challenge for storage systems. We will need the ability to scale to handle data generated by applications executing in parallel in tens of thousands of threads. There have been some solutions posed, such as IBM General Parallel File System (GPFS) and Microsoft Research’s Boxwood technology.

I was lucky enough to watch a presentation on GIGA+, another solution that is being researched by Swapnil V. Patil at Carnegie Mellon University. One of its neatest ideas is leaving the header-table behind, using a bitmap instead. I got to sit down with him afterward and talk about the challenges we face in this space. It was a great time. His primary concern about GPFS and Boxwood is the use of hashing and B-trees, which causes the possibility of bottlenecks and synchronization issues. By using a bitmap, and keeping it small so that it can be shared across nodes easily, GIGA+ eliminates a need for “metanodes” or other controllers on the HPC storage architecture.

His paper, GIGA+ : Scalable Directories for Shared File Systems, is a great read for those interested both in the problem of high-performance computing and storage. Their work seeks to maintain the UNIX file structure however, so those who care about scaling Microsoft infrastructure may find less to enjoy, but the overall architecture and problems outlined in the paper is applicable to any massively large storage cluster technology.

Enough Already

That should be enough to get you through August. When your boss comes back from his Alaskan cruise, nothing will ensure he leaves you alone more than talking about Concurrent Haskell or how much you enjoyed Chapter 9 of Programming Elements: Algorithms on increasing ranges. Enjoy the air conditioning you lucky bums.

http://discussionleader.hbsp.com/cramm/2008/06/8-things-we-hate-about-it.html

Overview:

1. IT Limits Managers’ Authority You bring in 10% of the company’s revenue but can’t authorize a $100,000 project if it requires IT. Furthermore, IT’s bureaucratic governance process rivals the tax code in complexity and inhibits rather than promotes innovation.
2. They’re Missing Adult Supervision The CIO is impressive, but totally unavailable. So the next best option is your IT “relationship manager" who’s a few clicks down the evolutionary scale and doesn’t have the breadth of expertise to truly act as a trusted IT advisor to senior business executives.
3. They’re Financial Extortionists When was the last time there wasn’t some emergency in IT (e.g. Y2K, SOX, HIPAA) that requires a zillion dollars? Compound this with the lack of visibility into how IT spends non-project dollars and it makes you want to become a technology vendor to cash in on the booty.
4. Their Projects Never End In-process projects are always 90% done. "Completed" projects don’t have agreed to functionality, and the team that promises to deliver missing functionality in future phases are always mysteriously missing-in-action.
5. The Help Desk is Helpless When glitches emerge, you are become a technology pauper, going door-to-door begging for help while functional specialists defend the reliability of their piece of the byzantine infrastructure.
6. They Let Outsourcers Run Amok You know that outsourcing wasn’t really IT’s idea, but you blame them when you’re trying to communicate with external “service” providers that lack even a basic understanding of your business. It’s like trying to teach calculus to a 4 year old.
7. IT is Stocked with Out-of-Date Geeks It’s not good when you learn about social networking from your 12-year old at home while IT is still trying to cope with email. Then, when you try to brainstorm with IT about how to apply new technology, you get paternalistic responses akin to the look that parents give their children when they play dress up.
8. IT Never Has Good News No matter how much you spend and how hard you work, you never have anything to celebrate and little to look forward to as the promise of technology seems perpetually beyond your reach.

What do you think of this list? Is it out of line or hit uncomfortably close to home?

.. and I’m only on page 33.

There are 112 pages.

It use to be that way - back from the inception of CACM on through the 1970s the magazine was a collection of computer science research for the academic professional. However, as the 1980s and 1990s moved computers in to people’s homes and the IT field changed from Phds toying with large Turing machines to undergrads who used Visual Basic and Java for basic business purposes, the magazine changed. These new practitioners didn’t come from the academic field, didn’t really understand the basic underpinnings of a computer, and usually didn’t care. The funding of the ACM dried up as well, even as the number of people in the field boomed. The CACM changed to grab these people by becoming more of a mainstream magazine geared towards those new entrants - maybe to attract these people to the ACM membership. It didn’t seem to work. The magazine lost its way.

Now we are once again approaching a change in the computer science field. Much like the way Cloud Computing is taking us back to the large machines in the back rooms and thin clients at the edge, software engineering is changing back from large numbers of engineers with basic knowledge to a smaller number with more specialized knowledge. The Googles of this world are not as worried about basic applications written across millions of detached machines - things that usually create reusable patterns and easy software construction from a weekend’s reading of O’Reilly books. Instead, they are worried about problems of concurrency, massively scalable storage systems and parallel processing while sharing the same memory space. The choice of the language has changed to an implementation detail to express these ideas and can be interchangeable. These problems require knowledge of tuples and binary trees and graph theory, to name a few.

At the same time programming jobs that boomed in the 90s and 00s are being outsourced to cheaper and cheaper labor overseas with the harder proofs being demonstrated once on the internet and then communicated across the world for others to incorporate. Pre-packaged software for businesses are becoming more configurable to existing systems and removing the need for custom software from programmers in non-software companies. This means that those who are serious about the profession are diving deeper in to the roles of architect, designer and academic - while those whom aren’t as interested are moving on to other careers. These two changes are providing an entrance for a journal like CACM to come alive again and publish the best research available needed to solve these hard problems.

The new CACM couldn’t come at a better time.

Typical Architecture Team

Enterprise Architects

Primary role is to manage large scale product and process integrations and determine which products and processes are best suited to deliver on business requirements. They control the large picture of how everything works in an organization and maintains this in a centralized location. They should be experts in software and enterprise design methodologies with experience in how large systems interact and manage data. These architects are essential to competitive and cost-effective decision making and use of technologies.

Water-Cooler Talk: The latest research in to The Staged Event-Driven Architecture for Highly Concurrent Server Applications

Integration Architects

This is an emerging role in larger companies that have large and complicated deployments, particularly around Service Oriented Architecture (SOA). They are usually the ones that have the task of managing Business Processes. Put simply, they tie the software platforms the Software Architect designs together on the environments the Enterprise Architects deliver and purchase. Although Enterprise Architects are typically restricted to existing thinking and technology products, it is the combination of Integration and Software Architects that differentiate an organization and provide maximum benefit.

Water-Cooler Talk: How to change the business workflow so that they can be quicker than their competitors. May need to talk to the Software Architects about how the platform can be changed for quicker processing too.

Software Architects

Primary role is to take architectural directions and artifacts and produce and manage a software platform that provides strategic and operational advantage to an organization. They are usually the ones who maintain the core frameworks of an organization and are considered the gurus of whatever technology they design for. They are very important as they tend to add order and discipline to projects and ensure that best practices, appropriate abstraction and code re-use occurs. These architects are essential to good outsourcing of software development, especially near-shore and off-shore.

Water-Cooler Talk: The latest research in to how Dependency Injection in Java 5 eliminates the need for the Composite Entity pattern in enterprise development.

Much of the conversation is about how to drive SOA adoption (it appears relying on developers to browse a repository is not working), how to measure cost benefits and savings to an organization (hint: measure your service reuse) and how they approach funding services which may not have an exact business advocate and therefore pocketbook to work against.

The last piece comes up all the time in organizations implementing a service oriented architecture. There are many business areas that would benefit from a high level “business service” which would result from the orchestration through externalized business logic (BPM/BPEL) of lower level “technology services”. Yet, if asked who would fund these lower level technology services so that the business services can emerge, the money dries up. Many take on the model of “first to need, first to pay” but that only works when the technology services and service orchestration aren’t that expensive. It’s hard to get project specific business users to fund enterprise wide services for the “greater good”. It’s something that has yet to be solved.

Here is the link.

Just consider it some light reading for your geek weekend.

Scalability Is Your Problem Too

The goals of the conference are laudable. Scalability is an issue that almost all practitioners of software engineering face, especially as we move towards offering services both inside and outside the enterprise. Many are taken off guard by the sudden issues that confront them after wiring up a large scale services-based environment; especially around distributing load, distributing the data, and writing the data quickly. Sadly, I didn’t see too many people from large companies there - most were software companies like Microsoft, Google, MySpace and Amazon.com. The attendance may be a consequence of the subject matter. This was some intense stuff dealing with MPI at Cray and its hopeful successor, Wikipedia redone with DHT and Erlang, a b-tree vs. Hashmap debate and scalable storage issues when dealing with billions of files. A more fun loving person would have done better going over to Adobe and hanging out at BarCampSeattle, which was going on at the same time.

Despite the intimidating material, there are real architectural and design issues that these discussions present that should be in the mind of anyone dealing with large datacenters that scale globally or even nationally. The approach of GIGA+ file storage, maidsafe’s new computer architecture, and NetWorkSpaces for the R language was uniform: off-loading responsibility for management of data (meta or otherwise) to all vertices in the deployment graph instead of a central repository. NetWorkSpaces in R and maidsafe even discussed computational scalability - while Cray’s new Chapel language and the discussion around Software Transactional Memory focused on scalability across processing cores as well as machines.

GIGA+’s approach of maintaining a small bitmap file on each node and passing that around - while anticipating and accepting stale data on a few edge nodes - was brilliant in the patterns it hinted at, including that perhaps being right all the time isn’t as important as being fast. You can be right most of the time and accept the performance hit of not being right some of the time. There are many people who would cringe at this, but at this point we’re going to have to play loose and leave a few balls up in the air as we juggle - doing the math of how often one may fall while keeping the rest going as fast as we can.

Pay No Attention To The Man Behind The Curtain

Yet if I had to sum up the content of the conference I would say it was big on strategy and architecture but short on implementation. There was a lot of things hinted at “behind the curtain” but nothing assured hand raising from the compsci geeks in the room more than hand waving when you got to the distributed piece of your solution. For instance, one of the big benefits of Chapel - the MPI successor that Bratford Chamberlain of Cray presented - was that you could have distributed arrays and graphs that would be automatically sliced up to be distributed to parallel cores or even other “locales” if desired. How the language determines where to split these large arrays and graphs and farm them out was not discussed. One of the more interesting slides was dashed lines drawn across various nodes and vertices of a graph symbolizing how it would be chopped and distributed. Someone in the audience raised their hand at this - but he moved on and the hand went back down. To be fair, Chapel was called a “multi-resolution” language where one could start fairly abstract and then add more detail and control to get the best desired result - something I assume you have to do to get good or intelligent chopping and distribution of the data. Given that one of his slides was a comparison of code lines between Fortan using MPI and Chapel: seeing a working code snippet of Chapel would have been helpful. It may turn out to be the same amount of work after you get past the “global view”.

This was the trend though, as all of the presentations had a bit of hand waving regarding performance metrics and distribution of computation. This was highlighted by the talk of Vijay Menon of Google - whose work at Intel I was familiar with - discussing Software Transactional Memory. He illustrated the challenges of implementing this in an imperative language (I’m suspicious you can even do STM well in an imperative language with state - as I discussed before) but beyond suggesting the keyword “atomic” to replace “synchronized” in the Java language there was very little real content discussed for those already familiar with the issue of locks and multiprocessors. Concurrent Haskell wasn’t even mentioned. A better introduction and discussion is to be had by watching the O’Reily’s OSCON video from Simon Peyton-Jones (the writer of GHC and now at Microsoft Research) on the subject. After that, if you’re still hungry, his collection of papers on his Microsoft Research site is a delight.

Of course the point of these conferences is the discussions that occur during the breaks and in the networking event afterwards - something that I treasure having newly moved to the Seattle area from Cincinnati. Instead of just observing and blogging from afar - I get to be at the same table as Vijay Menon, Thorsten Schuett, Swapnil Patil, Paul Watson and others.

Summary of the Architectural Patterns I Saw

If I had to summarize what I took away from the conference from a high-level architectural stand-point, here are they are:

• Every node must be aware of the state of every other node without a centralized controller.
• To do this, a mechanism should be in place to share state quickly but peer-to-peer.
• It’s ok to let some nodes go stale.
• Client/Server is now one thing. Pub/Sub with computation. Every node on the graph should do work.
• As much as possible, each node should maintain its own security and state. You should be able to have anonymous resources appear in your data center and be put to use without much configuration.
• As much as possible, abstract the distribution of processing away from programmers.
• Key,Value with Hashes are best for scalability and distribution (it seems to have won out in all the solutions presented here.) Blame MapReduce.
• Ants can be used to demonstrate anything.

I hope everyone had a good of a time as I did.

If you will be there, let me know be sure to say hi!

ABSTRACT

Recent technology trends in the Web Services (WS) domain indicate that a solution eliminating the presumed complexity of the WS-* standards may be in sight: advocates of REpresentational State Transfer (REST) have come to believe that their ideas explaining why the World Wide Web works are just as applicable to solve enterprise application integration problems and to simplify the plumbing required to build service-oriented architectures. In this paper we objectify the WS-* vs. REST debate by giving a quantitative technical comparison based on architectural principles and decisions. We show that the two approaches differ in the number of architectural decisions that must be made and in the number of available alternatives. This discrepancy between freedom-from-choice and freedom-of-choice explains the complexity difference perceived. However, we also show that there are significant differences in the consequences of certain decisions in terms of resulting development and maintenance costs. Our comparison helps technical decision makers to assess the two integration styles and technologies more objectively and select the one that best fits their needs: REST is well suited for basic, ad hoc integration scenarios, WS-* is more flexible and addresses advanced quality of service requirements commonly occurring in enterprise computing.