Brandon Werner

Over at Lambda The Ultimate, the best academic programming blog on earth, there is a large debate going on regarding what the future of languages will be for 2008. The most important thing to emerge from the discussion is the larger role functional programming will play. It seems like a safe bet. This year has seen the explosion of interest and creation of functional languages such as Apple OS X’s Nu, Java’s JVM using Scala and Microsoft Research’s .Net language F#.

I am ecstatic at this change.

The Failure Of Lisp

It’s hard to understand where it came from. Certainly one can argue the broader academic community had nothing to do with it, the old guard Common Lisp hackers are still as fickle and as judgmental to new comers as ever. Also, the old standards in Lisp languages, Franz and LispWorks have not lowered their prices to anything approachable to the casual developer. There are open source ANSI Lisp implementations without all the supporting engines and functionality, such as SBCL. In fact, my most linked thing I’ve ever written in my career is the installation walk-through I did for installing SBCL and Allegro which includes adding your repository and packages for CLOS and automatically compiling the FASL files, especially dealing with the asdf differences between the implementations. The complexity of this in itself points to problems with portability and configuration in Lisp. However, even that project that targeted Lisp’s Bread and Butter, the parsing of semantic ontologies for the Semantic Web, was met in the message boards with worries on if there would be enough developer participation using such an odd language, and recommendations on moving it to Java.

In reality, Common Lisp showed its failure as a community by sitting out this enthusiasm that has been generated around functional programming languages. It didn’t have to be that way. I recall my first awarenesses of functional programming’s growth was the awesome work of Lemonodor’s blog and Sriram Krishnan posting “Lisp Is Sin“. I was happy at the time that Lisp was getting such attention, as well as functional language architectures in general. I imagined that as OO languages had grown so verbose and feature dense that even the IDEs to develop your applications run in to the tens of gigabytes, a new evolution “Back To The Future” was inevitable. Even more, I believe long suffering Lisp deserves to be back in favor again, it’s certainly spent its time in purgetory. Yet, it didn’t happen. You can blame the old 50 year old men sitting on IRC channels for that. It was the most thorny and un-inspiring community I’ve ever participated in, despite my extreme interest in the language. It’s jaw dropping that a language with such promise has sat out the resurgence, and speaks to what an un-friendly and un-inviting community can do a technology platform. I would be the first to march it off to the grave.

The Rise Of Functional Languages

The interest in functional programming actually grew up around more academic but pure languages like Scheme and Haskell. Although these languages sit within their own island and lack many of the “dirty” aspects of Lisp’s CLOS environment that make it easy to access OS and hardware resources, they are still strikingly useful in learning things that are the staple of functional languages, such as Closures and Lambdas. Indeed, one could argue that the movement to move Closures in to OO languages (first C#, now Java) was in part due to the rise of awareness of functional languages.

Further, it seems to me that functional programming languages answered two prayers of those more ambitious engineers who don’t seem to want to stick with the script and Java worlds they were taught in college. Those two large wins, far more important than the semantic features of functional languages that have gotten all the attention, are architecture foundations of functional languages:

• Referential Transparency / Side Effects
• Concurrency

Referential Transparency

To those coming from a pure OO world, Referential Transparency and the restriction of side-effects can be something hard to get their heads around. The best way I describe this concept is by hitting at the root of their assumptions: Everything they deal with are dead. The objects are dead, the variables are dead, the entire atmosphere is dead, as if something had come along and killed everything in your stack and you have to assemble your program by only what’s been given to you, nothing more. There are no instances, objects do not “come alive” and have state; a state that you have to poke in to and a state that can change at any time. A function will always do what you expect, and nothing can come along and change that behavior.

One of the things that seems to appeal to developers most about the promise of SOA architectures happening in enterprise environments, if you’re smart enough to pry it out of them, is that they get the same referential transparency in services. No one can override a service (besides versioning, which is explicit to the developer) and a service will only return what it did earlier in your code and earlier in the year. This forces developers to design services that have the same relationship to the world as functional programmers write their functions for. This is perhaps the trickiest part of migrating enterprise teams to a services based model, their expectations of the mutableness of the services they are accessing and their inability to anticipate what working in that world will be like. Especially for those who use tools or libraries to convert service interaction in to an object, the interaction can be jarring.

However, the soon find the predictability and the safety of such an environment liberating. In much the same way OO programmers were use to making their objects or variables immutable to maintain their contracts and relationships with other objects, often sacrificing many of the benefits that OO programming promised their stack, now they have immutability and transparency in an environment where functional paradigms are key, they do not expect to be able to “embrace and extend” services. They are what they are. This tends to cascade out to the living instantiated code a developer writes as well, as there is no point in entering the world of the living if what you have to return to is a dead function.

This was hinted at in an article in the ACM Queue magazine by Terry Coatta, entitled “From Here to There, The SOA Way“. He states,

Objects are still a very good way to model systems and they function reasonably efficiently in the local context. But they don’t distribute well, particularly if one tries to use them in a naive way. A service-oriented architecture solves this problem by dealing with the latency issues up front. It does this by looking at the patterns of data access in a system and designing the service-layer interfaces to aggregate data in such a way as to optimize bandwidth, usage, and latency.

Not that SOA limitations are the only thing that is affecting the consciousness of a software engineer, the other issue is the large rise in the complexity of managing a large enterprise library written in an OO language. One of the largest pain points of any application of large size is the management of graphs and graphs of live objects and the living data within them. When software engineers experience the lack of side-effects in functional languages, it’s a breath of fresh air.

Concurrency

A funny thing happened on the way to those multi-core processors. People loaded their applications on them and noticed nothing got much faster, particularly when it came to transaction intensive tasks. Turns out Intel and AMD left out an important fact about their Moore’s Law cheating multi-core environment: you can’t ring as much performance out of it without changing the way you manage concurrency and threads. Sequential programming could always rely on going faster as the single processor speed got faster, but as multicores come in to play that isn’t always the case. You want to farm off transactions to occur on separate processors, and in the living world of mutable objects and variables, breaking out two transactions to work concurrently that operate on the same living data is a bad idea. Add structural programming’s solution to this problem, optimistic and pessimistic locking, and you have dead-locks in short order.

Functional programming has been a natural place to explore parallel processing and new ways of doing atomic transactions because of the reasons above. More important, these atomic structures can be composable which is lost when doing locks in structural programming. A lot of the buzz has been generated around the idea of software transactional memory, where execution blocks can be flagged and managed and built upon. The best introduction to this topic is the paper by Tim Harris entitled Concurrent Programming Without Locks. Although this use to be expressed only in the confines of Concurrent Haskell, others have shown how the same techniques can be used in other functional languages, such as F# using nothing more than PowerList.

This experimentation is one of the large reasons why functional languages have become more important as software engineers wrestle with the problems and promise of multi-core processors in transaction processing. Although not every engineer will be interested in the deeper details of STM or other strategies in concurrent programming, the fact that these libraries will emerge and only be available in the functional realm will force software engineers to learn the core concepts and bring even more visibility to the functional programming space.

Functional Hybrids: Functional Programming Is Now Approachable

The other driver for adoption of functional programming languages, besides the architectural benefits it has to solve current problems, is the fact that languages such as F# and Scala have adopted a more hybrid model in their language design, where a developer isn’t forced completely outside her comfort zone. Scala is a combination of functional and deeper OO methodologies (as in SmallTalk) and has access to the entire Java library, significantly reducing the learning curve. The same can be said for F# and .Net and Nu and Objective-C. This does have draw-backs however, as both F# and Scala have not been able to use more of the STM strategies that Concurrent Haskell allows because the underlying thread architecture of the VMs they run against are built for structural programming languages. It is easy to see how this can be fixed, however, and allow those using hybrid functional languages the same power as those who express their ideas in Haskell or even Lisp.

As I said, I am excited about this new resurgence in functional programming languages, and I am enthusiastic 2008 will have even more to offer those who are just getting their toes wet. I personally know some college freshman who started out using Nu as their first language, and are already contributing to the community. The future of software engineering is bright.

I seem to be visiting a lot of my prior entries from 2005, but a while back I wrote about an experience I had meeting an old IBM programmer at the local Catholic store. She was female, which seemed like a trail-blazing thing to have been working in the heavily male dominated technology industry in the 1950s. Her story, and her eagerness to share with me, led me to write my experiences with gurus in my life, entitled In The Presence of A Guru (On Catholicism, VAX/VMS and Geek Culture). I still feel profoundly stupid for not having spent more time with her, as it was obvious she wanted to share her story. I will regret it for the rest of my life, not just because of what the exchange could have done for my understanding, but more refusing my obligation, the obligation we all have, to listen to and carry on the mythology and history of our elders. I am afraid that mythology may be vanishing, and none of us are listening.

I thought about this again when I stumbled upon a new blog from Dan Weinred as linked from Lemonodor. His career in Computer Science and his contributions certainly qualifies him as a Guru.

His essays so far are a treasure trove of information and computer science archeology. One of my favorites is called The Technology and Business of ObjectStore, where he recounts taking his ObjectStore technology to Microsoft and Bill Gates as well as Steve Jobs. The reaction of both icons demonstrates humorously their approach even now:

When the new Windows technology (which was OS/2 at the time; IBM and Microsoft were still working together on it) came out, it was crucial for us that it be able to support memory mapping. Dave Stryker and Tom Atwood, flew out to meet with Bill Gates in September of 1989. Dave Stryker recalls: “We originally had a 45-minute appointment, but Gates extended the meeting to a couple of hours, and called in Dave Cutler [the architect of OS/2]. At Tom’s urging, we told Gates and Cutler everything they wanted to know about ObjectStore. Gates was complimentary of the Object Design approach, but said, in a nice enough way, that if the Microsoft Empire ever needed such a thing, they would build it themselves. Still, Gates told Cutler to make sure that the OS/2 equivalent to mmap was powerful enough to run ObjectStore, and there were some changes made to make it so.” Later, this OS/2 technology turned into Windows NT. Dave Moon adds that it turned to have a bug: it doesn’t free up disk space when it ought to. For some reason Microsoft hasn’t fixed this, even after many years. We found a way around it.)

Speaking of industry luminaries, we also met with Steve Jobs when he was at NeXT, and Jobs made a big announcement praising our technology, which resulted in a nice press release. There was some discussion that NeXT might buy Object Design, but that never went anywhere.

This type of history is fascinating and important. There have been efforts to capture this knowledge, such as Fokelore.org and ACM’s excellent ACM History Committee trying to archive CS knowledge, but their efforts so far is approaching it the wrong way. It’s getting the stories, while these people are still alive and can tell them, that is important. Much of it will not be academic or well checked, but that is missing the point. It’s the mythology as well as the history of these pioneers in the 1930s - 1970s that we should be after. We’ve already lost Jeff Raskin, Adam Osborne and others.

If we do not do this, their legacy and our history as a profession, one that has changed the world as much as any other in human history, will be lost forever.

I really want to put the Science back in to Computer Science, and am an avid member of the ACM and IEEE, although the lines between those are starting to blur. I’ve found that their magazine and website, ACMQueue, is an amazing attempt at putting academics back in to the marketing and Forrester “research” laden stuff that we are forced to contend with in this industry (their latest cover article, API Design Matters, is a good read but not yet up on the site).

A fresh new issue I got to read today, however, made me do a spit-take:

There is a “Better Software Conference and Expo” that is going to be occurring between June 18 - 21st 2007 at Las Vegas. Microsoft, a keynote speaker at the event, apparently hoping that what happens in Vegas stays in Vegas, will be presenting this topic to attendees and wondering why they can’t stop giggling in the audience (click on image to the left and look to the bottom keynotes)

I have decided, after reading Sriram Krishnan’s wonderful post on MSDN about “Lisp is sin“, that Microsoft does have incredible developers and thinkers in their crew. I want to be as smart as Sriram some day.

However, reading this entry got me thinking about why these type of developers and the type of innovation this developer is obviously capable of is not being shown to the outside world? Could it be that a lot of Sriram’s peers aren’t in the Windows world at all? That, at the end of the day, we assume Windows developers are more Morts than anything else and don’t expect much from them, regardless of the great technology Microsoft throws their way.

It might also be the case that this whole Mort, Elvis, Einstein thing is a sign of arrogance in Microsoft’s developer ranks that shouldn’t be encouraged. Read Wesner’s blog posting and see how it rubs you. It rubs me the wrong way.

Although many people have been anxious to speculate on what the Intel transformation means for Apple going forward, and what Intel machines will be released tomorrow, one issue that has not been spoken about as much is what the Intel transformation means for Java developers. It’s no secret that at any JavaOne or geek conference you attend, what would have been a sea of standard Dell laptops around 2001 are now, just five years later, a sea of glowing apple logos from Powerbooks.

The reason for this transition is simple: Apple provides developers with the best of both worlds. You can develop and set up your XML files, CLASSPATH entries and config files on Mac OS X and create a perfect build environment that can be moved over to whatever Linux or AIX/Solaris platform you are deploying on with absolutely no extra steps. Developing in Mac OSX for UNIX deployment really is WYSIWYG development on all levels. This can’t be said for Windows, which because of it’s 1980s Win32ness (which shows no signs of changing in Vista) requires a build environment totally different from where most apps will be deployed. This requires large migration efforts and usually results in the local development environment and the QA / Production environment always being out of sync.

In the other world, Apple also allows developers to plug easily in to the business side of their work life, with Microsoft Office, Adobe Photoshop and other large business applications (not to mention you can always fire up World of Warcraft in your hotel room for some down time with your favorite night elf friends). These benefits for the developer even drawf the other large strengths of Mac OSX, such as it’s state of the art Quartz rendering, open source friendliness, and gorgeous UI.

The deal seems too good to be true. For the last few years, it has been.

The dirty little secret is that Apple has been using the JDK to force upgrades to it’s latest operating system, and Java developers have had to give in to the very Microsoft tactics of Apple in order to continue to be relevant when developing applications. Further, as Dmitry from Creo (recently purchased by Kodak) recently commented to me, those companies that make Java applications that run cross-platform (including the Mac) can’t be certain if their users are using the latest JDK. Also, since they can’t legally install the new Apple JDK on their client’s computers that may not have the latest OS (think about that for a minute, this is Java we’re talking about), they can’t use JDK 5 features even if they wanted to. The write once / run anywhere benefit of Java couldn’t be stopped by Microsoft, but seems to slam in to a Sun approved brick wall with Apple and holds all Java client apps that want to be truly cross-platform without uprgrade to the JDK 1.3 level (Jaguar).

If you install the JDK 5 distribution available from Apple’s Developer Connection website, the installer will check your OS level and politely tell you that the JDK 5 update is only for Tiger users. There is no technical reason why this is the case, as many developers who choose not to upgrade their operating system simply use the excellent Pacifist application to install the JDK without going through Apple’s OS test. Apple’s JDK update simply installs the latest Java JDK in /System/Library/Frameworks/JavaVM.framework/Versions/, updates the symbolic links and goes away. The fact that there is no difference in how Java was distributed between Tiger (10.4) Panther (10.3) and even Jaguar (10.2) shows how transparent and deplorable this restriction is.

Needless to say, Java developers who moved to the Mac platform have been taken aback by the fact they don’t have a redistributable platform, the very definition of Java since it’s inception.

Luckily, Apple is slated to lose control of their PowerPC built, invitation-only JDK distribution with the release of Intel based Macs. Since OS X will run on the x86 instruction set, the JDK work that is often hardest to port (threading, “Little Endian” vs. “Big Endian”, ect.) will become mostly transparent between linux and os x compilations going forward. Either Sun will make an x86 Mac JDK with Apple’s consent, or Apache will make one without their consent. The differences at the architecture level should be minimal, although file system integration and other important things will still require at least some work.

Apple seems to know this, as they have done to Java what they do to most third party environments when they know they can’t control them anymore, they drop their own internal work on it and leave it for the vendors to do. They have abandoned the Java-Cocoa bridge, they have given the amazing WebObjects framework over to the public for free, and are minimizing Java over Objective-C in their Xcode feature development (I will be shocked if Xcode 3 even has Java in it). Once a large bullet point on the advantages of using Mac OSX, now Java is seen as something that Sun and IBM can do much better on Mac OSX than Apple can.

All of this is great for the Java developer, who will finally get the platform that OS X seemed at first to be.

If anyone needs any more examples of how behind Microsoft is in its core businesses take a look at this photo from TechEd Europe 2005. It would seem that although it’s a Microsoft keynote (TechEd), they are running the software and examples on Sun hardware. Nice photograph of the dynamics behind Web 2.0 and the new bubble that is forming. Jonathan Schwartz highlights this by stating that Sun might be the “dot in Web 2.0″ as well. The metrics he points to from James Governor’s Blog entry entitled Sun: putting the dot in web 2.0 says it all:

“I noticed an item from news.com today that made me think about Sun and its future: eBay to buy hundreds of Sun x86 servers. It turns out PayPal is also evaluating Sun’s new T1 Ultasparc line. The eBay story follows news that Google is going to buy some of the new elegant Sun kit too… You know Sun has got to be talking to MSN - and Sun is now a Windows poster child.”

I’m running Solaris x86 on all of my boxes here and at the startup. I wouldn’t run Linux again. I’ve had some of the stability and robustness sugar, and I can’t go back anytime soon to rpms, library conflicts, horrible documentation and other headaches. I think Sun Solaris 10 and OpenSolaris are the biggest unwritten stories of 2005. Then again, it’ll appear in full force in 2006 and people will wonder where it came from.