Brandon Werner

I took Franz and other Lisp companies to task a few weeks ago in a posting I wrote: The Rise Of Functional Programming: F#/Scala/Haskell and the failing of Lisp:

It’s hard to understand where it came from. Certainly one can argue the broader academic community had nothing to do with it, the old guard Common Lisp hackers are still as fickle and as judgmental to new comers as ever. Also, the old standards in Lisp languages, Franz and LispWorks have not lowered their prices to anything approachable to the casual developer.

Well, I got this email from a Franz representative in response:

Hi Brandon,

My name is Bernard… Very interesting blog, and it does look like you are still working with Lisp, and surprisingly, Semantic Web, too. Any chance you will be down in San Jose for SemTech 2008 this month?…

… I did see the post on Lisp. While we do need to run a business and stay afloat, it’s also in our best interest to have more interesting Lisp and ACL based projects out there. Give me a call if you would like to continue using ACL in your projects and we should try to work something out… I can also set up a temp license if you are interested in our RDF triple store AllegroGraph (http://agraph.franz.com/allegrograph/). The v3.0 release should be in a few weeks and will support both federation and social network analysis tools.
http://agraph.franz.com/support/documentation/3.0/reference-guide.html#header3-65

I am looking forward to talking to you.

First off, dangling the temping temp license for v3.0 of AllegroGraph is not playing fair. I have always been impressed with Allegro’s work in the semantic space set even before it was a popular buzz word. In fact, the same thing that led me to attempt to solve the Semantic problem with Lisp is what led to the same for Franz. If the semantic web and reasoning engines are to become reality, especially on parallel processing architectures, Lisp has to be at the front of the bus. Certainly, others will try to claim they do this just fine with interpreted OO languages with some runtime tweaking - but the problems facing us in the future demand we think differently about how we even construct algorithms to solve our problems. Brute force coding and heavy stacks are not going to get us there.

However, the fact that I’ve always admired it from afar is part of the discussion in my article linked above. It seems simply out of reach for mere mortals to use and incorporate in to their own development plans because of price. Certainly, Allegro deserves to be compensated for their hard work - this isn’t kids hacking PHP for the next Twitter reporting app after all - Allegro has always tackled the big problems where they can contribute value. Not the same thing can be said for many software companies out there.

Regardless, flirting with applications using this model is harkening back to the era of big client-server installations instead of quick and nimble collaborative innovation. As much as Allegro’s marketing may say that AllegroGraph is “Web 3.0″, the principles that drive it are not going to allow its success to be pinned to large engines running in a back room of a well funded company. If I get addicted to the software Allegro has - there is no remedy to bring in on board in my work.

This isn’t to say that Allegro hasn’t opened up to the community - they have opensourced good libraries - although through another license scheme, LLPGL. It also seems they are using the IBM model of “Community Driven Development” I complained about before when IBM released Project Zero to put “PHP on Rails”. They take contributions to fold back in to the Allegro products.

Although I really like working close with Allegro and writing about their accomplishments in this space, I challenge them to think if this is truly the model to gain traction in the coming Web 3.0 world. I would wager the Lisp community should still effort to create more nimble and open components for the semantic web - the internet will demand no less when picking it’s platform for Web 3.0.

Over at Lambda The Ultimate, the best academic programming blog on earth, there is a large debate going on regarding what the future of languages will be for 2008. The most important thing to emerge from the discussion is the larger role functional programming will play. It seems like a safe bet. This year has seen the explosion of interest and creation of functional languages such as Apple OS X’s Nu, Java’s JVM using Scala and Microsoft Research’s .Net language F#.

I am ecstatic at this change.

The Failure Of Lisp

It’s hard to understand where it came from. Certainly one can argue the broader academic community had nothing to do with it, the old guard Common Lisp hackers are still as fickle and as judgmental to new comers as ever. Also, the old standards in Lisp languages, Franz and LispWorks have not lowered their prices to anything approachable to the casual developer. There are open source ANSI Lisp implementations without all the supporting engines and functionality, such as SBCL. In fact, my most linked thing I’ve ever written in my career is the installation walk-through I did for installing SBCL and Allegro which includes adding your repository and packages for CLOS and automatically compiling the FASL files, especially dealing with the asdf differences between the implementations. The complexity of this in itself points to problems with portability and configuration in Lisp. However, even that project that targeted Lisp’s Bread and Butter, the parsing of semantic ontologies for the Semantic Web, was met in the message boards with worries on if there would be enough developer participation using such an odd language, and recommendations on moving it to Java.

In reality, Common Lisp showed its failure as a community by sitting out this enthusiasm that has been generated around functional programming languages. It didn’t have to be that way. I recall my first awarenesses of functional programming’s growth was the awesome work of Lemonodor’s blog and Sriram Krishnan posting “Lisp Is Sin“. I was happy at the time that Lisp was getting such attention, as well as functional language architectures in general. I imagined that as OO languages had grown so verbose and feature dense that even the IDEs to develop your applications run in to the tens of gigabytes, a new evolution “Back To The Future” was inevitable. Even more, I believe long suffering Lisp deserves to be back in favor again, it’s certainly spent its time in purgetory. Yet, it didn’t happen. You can blame the old 50 year old men sitting on IRC channels for that. It was the most thorny and un-inspiring community I’ve ever participated in, despite my extreme interest in the language. It’s jaw dropping that a language with such promise has sat out the resurgence, and speaks to what an un-friendly and un-inviting community can do a technology platform. I would be the first to march it off to the grave.

The Rise Of Functional Languages

The interest in functional programming actually grew up around more academic but pure languages like Scheme and Haskell. Although these languages sit within their own island and lack many of the “dirty” aspects of Lisp’s CLOS environment that make it easy to access OS and hardware resources, they are still strikingly useful in learning things that are the staple of functional languages, such as Closures and Lambdas. Indeed, one could argue that the movement to move Closures in to OO languages (first C#, now Java) was in part due to the rise of awareness of functional languages.

Further, it seems to me that functional programming languages answered two prayers of those more ambitious engineers who don’t seem to want to stick with the script and Java worlds they were taught in college. Those two large wins, far more important than the semantic features of functional languages that have gotten all the attention, are architecture foundations of functional languages:

• Referential Transparency / Side Effects
• Concurrency

Referential Transparency

To those coming from a pure OO world, Referential Transparency and the restriction of side-effects can be something hard to get their heads around. The best way I describe this concept is by hitting at the root of their assumptions: Everything they deal with are dead. The objects are dead, the variables are dead, the entire atmosphere is dead, as if something had come along and killed everything in your stack and you have to assemble your program by only what’s been given to you, nothing more. There are no instances, objects do not “come alive” and have state; a state that you have to poke in to and a state that can change at any time. A function will always do what you expect, and nothing can come along and change that behavior.

One of the things that seems to appeal to developers most about the promise of SOA architectures happening in enterprise environments, if you’re smart enough to pry it out of them, is that they get the same referential transparency in services. No one can override a service (besides versioning, which is explicit to the developer) and a service will only return what it did earlier in your code and earlier in the year. This forces developers to design services that have the same relationship to the world as functional programmers write their functions for. This is perhaps the trickiest part of migrating enterprise teams to a services based model, their expectations of the mutableness of the services they are accessing and their inability to anticipate what working in that world will be like. Especially for those who use tools or libraries to convert service interaction in to an object, the interaction can be jarring.

However, the soon find the predictability and the safety of such an environment liberating. In much the same way OO programmers were use to making their objects or variables immutable to maintain their contracts and relationships with other objects, often sacrificing many of the benefits that OO programming promised their stack, now they have immutability and transparency in an environment where functional paradigms are key, they do not expect to be able to “embrace and extend” services. They are what they are. This tends to cascade out to the living instantiated code a developer writes as well, as there is no point in entering the world of the living if what you have to return to is a dead function.

This was hinted at in an article in the ACM Queue magazine by Terry Coatta, entitled “From Here to There, The SOA Way“. He states,

Objects are still a very good way to model systems and they function reasonably efficiently in the local context. But they don’t distribute well, particularly if one tries to use them in a naive way. A service-oriented architecture solves this problem by dealing with the latency issues up front. It does this by looking at the patterns of data access in a system and designing the service-layer interfaces to aggregate data in such a way as to optimize bandwidth, usage, and latency.

Not that SOA limitations are the only thing that is affecting the consciousness of a software engineer, the other issue is the large rise in the complexity of managing a large enterprise library written in an OO language. One of the largest pain points of any application of large size is the management of graphs and graphs of live objects and the living data within them. When software engineers experience the lack of side-effects in functional languages, it’s a breath of fresh air.

Concurrency

A funny thing happened on the way to those multi-core processors. People loaded their applications on them and noticed nothing got much faster, particularly when it came to transaction intensive tasks. Turns out Intel and AMD left out an important fact about their Moore’s Law cheating multi-core environment: you can’t ring as much performance out of it without changing the way you manage concurrency and threads. Sequential programming could always rely on going faster as the single processor speed got faster, but as multicores come in to play that isn’t always the case. You want to farm off transactions to occur on separate processors, and in the living world of mutable objects and variables, breaking out two transactions to work concurrently that operate on the same living data is a bad idea. Add structural programming’s solution to this problem, optimistic and pessimistic locking, and you have dead-locks in short order.

Functional programming has been a natural place to explore parallel processing and new ways of doing atomic transactions because of the reasons above. More important, these atomic structures can be composable which is lost when doing locks in structural programming. A lot of the buzz has been generated around the idea of software transactional memory, where execution blocks can be flagged and managed and built upon. The best introduction to this topic is the paper by Tim Harris entitled Concurrent Programming Without Locks. Although this use to be expressed only in the confines of Concurrent Haskell, others have shown how the same techniques can be used in other functional languages, such as F# using nothing more than PowerList.

This experimentation is one of the large reasons why functional languages have become more important as software engineers wrestle with the problems and promise of multi-core processors in transaction processing. Although not every engineer will be interested in the deeper details of STM or other strategies in concurrent programming, the fact that these libraries will emerge and only be available in the functional realm will force software engineers to learn the core concepts and bring even more visibility to the functional programming space.

Functional Hybrids: Functional Programming Is Now Approachable

The other driver for adoption of functional programming languages, besides the architectural benefits it has to solve current problems, is the fact that languages such as F# and Scala have adopted a more hybrid model in their language design, where a developer isn’t forced completely outside her comfort zone. Scala is a combination of functional and deeper OO methodologies (as in SmallTalk) and has access to the entire Java library, significantly reducing the learning curve. The same can be said for F# and .Net and Nu and Objective-C. This does have draw-backs however, as both F# and Scala have not been able to use more of the STM strategies that Concurrent Haskell allows because the underlying thread architecture of the VMs they run against are built for structural programming languages. It is easy to see how this can be fixed, however, and allow those using hybrid functional languages the same power as those who express their ideas in Haskell or even Lisp.

As I said, I am excited about this new resurgence in functional programming languages, and I am enthusiastic 2008 will have even more to offer those who are just getting their toes wet. I personally know some college freshman who started out using Nu as their first language, and are already contributing to the community. The future of software engineering is bright.

I seem to be visiting a lot of my prior entries from 2005, but a while back I wrote about an experience I had meeting an old IBM programmer at the local Catholic store. She was female, which seemed like a trail-blazing thing to have been working in the heavily male dominated technology industry in the 1950s. Her story, and her eagerness to share with me, led me to write my experiences with gurus in my life, entitled In The Presence of A Guru (On Catholicism, VAX/VMS and Geek Culture). I still feel profoundly stupid for not having spent more time with her, as it was obvious she wanted to share her story. I will regret it for the rest of my life, not just because of what the exchange could have done for my understanding, but more refusing my obligation, the obligation we all have, to listen to and carry on the mythology and history of our elders. I am afraid that mythology may be vanishing, and none of us are listening.

I thought about this again when I stumbled upon a new blog from Dan Weinred as linked from Lemonodor. His career in Computer Science and his contributions certainly qualifies him as a Guru.

His essays so far are a treasure trove of information and computer science archeology. One of my favorites is called The Technology and Business of ObjectStore, where he recounts taking his ObjectStore technology to Microsoft and Bill Gates as well as Steve Jobs. The reaction of both icons demonstrates humorously their approach even now:

When the new Windows technology (which was OS/2 at the time; IBM and Microsoft were still working together on it) came out, it was crucial for us that it be able to support memory mapping. Dave Stryker and Tom Atwood, flew out to meet with Bill Gates in September of 1989. Dave Stryker recalls: “We originally had a 45-minute appointment, but Gates extended the meeting to a couple of hours, and called in Dave Cutler [the architect of OS/2]. At Tom’s urging, we told Gates and Cutler everything they wanted to know about ObjectStore. Gates was complimentary of the Object Design approach, but said, in a nice enough way, that if the Microsoft Empire ever needed such a thing, they would build it themselves. Still, Gates told Cutler to make sure that the OS/2 equivalent to mmap was powerful enough to run ObjectStore, and there were some changes made to make it so.” Later, this OS/2 technology turned into Windows NT. Dave Moon adds that it turned to have a bug: it doesn’t free up disk space when it ought to. For some reason Microsoft hasn’t fixed this, even after many years. We found a way around it.)

Speaking of industry luminaries, we also met with Steve Jobs when he was at NeXT, and Jobs made a big announcement praising our technology, which resulted in a nice press release. There was some discussion that NeXT might buy Object Design, but that never went anywhere.

This type of history is fascinating and important. There have been efforts to capture this knowledge, such as Fokelore.org and ACM’s excellent ACM History Committee trying to archive CS knowledge, but their efforts so far is approaching it the wrong way. It’s getting the stories, while these people are still alive and can tell them, that is important. Much of it will not be academic or well checked, but that is missing the point. It’s the mythology as well as the history of these pioneers in the 1930s - 1970s that we should be after. We’ve already lost Jeff Raskin, Adam Osborne and others.

If we do not do this, their legacy and our history as a profession, one that has changed the world as much as any other in human history, will be lost forever.

Cosmin Stejerean does an awesome job laying out how to get experimental x86 OSX threads support compiled and enabled for SBCL on Apple’s latest operating system. I can’t wait to try this out with my cl-semantic project I’ve been working on off/on for some time. If you’d like to help out, I wrote detailed instructions on how to get up and running if you are using SBCL.

If you’re scratching around trying to find out how to compile Haddock after getting the source, especially if you are using the awesome EclipseFP plug-in for Eclipse which has it’s use integrated inside the IDE, here are the steps:

First, ensure you have Glasgow Haskell Compiler installed, obviously. It has a OS X binary port and I like it over HUGS.

Second, you should really have Happy and Alex installed as well, as the build will look for it, and they are good to have, but will work fine without it.

from your unzipped directory:

$ ghc -o Setup Setup.lhs -package Cabal
$ ./Setup configure --ghc --prefix=${prefix} --enable-library-profiling
$ ./Setup build -v
$ ./Setup copy --copy-prefix=/usr/local/


Of course, /usr/local could be wherever you want the executable and libs to be installed. It has a /bin and /share.

After this step, you should be able to run Haddock documentation extraction from the Eclipse IDE. If you’re curious how Haskell looks in Eclipse with the EclipseFP plug-in, here it is:

With module support and outline expansion, it still takes heavily from the OO world (it doesn’t keep this updated if playing with the shell, for instance) it’s still somewhat nice to keep things organized. Of course, the IDE paradigm changes substantially when you are talking about lazy dynamically typed call-by-value languages. Still, for a round peg trying to fit in a square hole, it’s ok.

I still perfer the Haskell editor in the incredible TextMate, however.

I have been asked to present my work: “The Future of ACORD: Using Data Standards In The Semantic Web” at the Insurance Industry’s major conference ACORD-LOMA Insurance Systems Forum from May 20th - 22nd at the Dolphin Resort at Disney World. Since Microsoft and IBM are platnum sponsers, I’m expecting lots of free stuff :-) At least, more than you get speaking at the Rational Conference.

I have included the detailed snippet from the brochure below, and more information about when I’m scheduled should appear on the website after February 2nd when I deliver all my media information (my head shots from junior high apparently don’t cut it anymore)

My primary thesis is how to use the language in which we communicate, XML (here ACORD standard) and apply meta-data to enable us to gain insight and increase both human and machine understanding of the data we have, essentially helping us make better decisions. It’s easy to see how a system knowing what a policy or a named-insured is beyond just an XML tag can benefit decision making systems including competative rating and data mining for risk patterns in insurance. All of this is based off of using RDF and Ontologies (here developed for insurance) to make relationships between data. Imagine how much more valuable a data warehousing operation would be with this solution.

I think you’ll like it and I hope you can make it.

My question to you is: Do you think the industry is ready, having just now got up to speed with webservices and Service Oriented Architecture, to apply meta-data to our language? Although the benefits of making the semantic leap are real, is it realisitic in the near future?

My Presentation Brochure Detail Snippet

XML, as a describer and standardization of data and datatypes, are changing as we move in to a future where humans won’t be consuming data as much as other computers will. The Semantic web is about metadata, or “data about data” that tells computers what type of data it is, and it’s context. It is about a language for recording how the data relates to real world objects. This allows a person, or a machine, to start off in one database, and then move through an unending set of databases which are connected not by wires but by being about the same thing.

For the insurance industry, this transformation can bring incredible advances and solutions to the problems we are experiencing as an industry that XML and webservices alone can’t solve, including managing large amounts of data and customer information. In the Semantic Web, for instance, we can know what products may interest a customer through relationships built from their CLUE reports and their preferences or send self-describing products to agents without any upfront setup. Most importantly, it can be used in data warehousing applications to allow insurance companies to create their own meaning from their customer data for better rating by deploying reasoning systems to determine what risks are related to what datasets deeper than ever before.

ACORD standards will play the lead in this, and companies that are adopting ACORD standards in managing their business communications now, especially using ACORD in their webservices, will have a lead in this advancement. ACORD can become “richer” when paired with RDF documents that not only enforce the standards, but describe them as well.

In my presentation, I will demonstrate the future value of deploying these reasoning systems, demonstrate the work I’ve done in developing reasoning systems for ACORD formatted data, demonstrate a sample using Firefox and MIT’s Piggy Bank plug-in (http://simile.mit.edu/piggy-bank/ ) of a future ACORD standard based website that will show a customer, the products they may like and the best rate; all from self-describing meta-data and without any code (only the data itself)

I work with MIT’s Semantic Web SIMILE (http://simile.mit.edu/) project as well as invented and run the opensource cl-semantic project (http://common-lisp.net/project/cl-semantic/ ) that uses Lisp to discover relationships between data for automatic processing and the realization of patterns, geared specifically towards insurance and other probability based businesses. I have also been involved in research creating a RDF version of the ACORD standards (the subset of Surety) to allow ACORD standards to be used in a Semantic Web demonstration. I am also an ACORD voting member of the Casualty/Surety working group and sub-groups underneath. I am well known in the industry for my work, and you can browse my referred and non-referred research papers from my website (http://www.brandonwerner.com).

Martin Simmons of LispWorks confirmed today on the LispWorks mailing list that an Intel native version of LispWorks for the Macintosh platform will be available by June 2006.

There will be an Intel native Mac version of LispWorks 5.0, which is due in Q2
this year…We can probably fix problems with LispWorks for Macintosh 4.4 running under
Rosetta as chargeable support work. Please ask about that on lisp-support.

Many people have commented that almost all versions of Lisp (both free and commercial) have problems running on the Intel platform of Mac OSX do to signifigant problems with using Lisp through the Rosetta software program that translates PPC instructions to Intel for compatibility with applications that do not have a universal binary or Intel native product available.

Lisp is a language that ties very deeply to the kernel and the hardware of a computer, and it is not surprising that Rosetta is causing problems with Lisp developers. Among reports of problems are hanging and very slow compile and start-up times for Lisp IDEs and applications.

An un-answered question is the support for PPC and in particular Mac OSX on PPC in the future. Because Lisp is processor dependent, a universal binary (fat binary) solution is not possible, and Lisp vendors will have to actively support versions for both PPC and Intel on Mac OSX.

Given the wide use of Mac OSX in the acedemic and biomedical industries, this move was expected but previously un-released. Allegro, the other big commercial Lisp vendor, has not disclosed any plans on when it will go Intel native.

John Wiseman sadly reports that the latest Allegro version of their Common Lisp, 8.0, which has gotten written up in eWeek and made us all proud, also crashes (not really) on Mac OSX the same way that SBCL does. This is a problem that’s going to have to be fixed, because the number of CrashReporter instances it generates on a system is unacceptable when in REPL developing.

Imagine every stacktrace spawning a 30MB process on your system each time…

I also wrote something about it as a note (read:warning) on my research project’s installation instructions. It goes:

You can use SBCL on Mac OSX from both the command line and visually, but you probably don’t want to at this moment. SBCL throws a SIGSEGV at every stracktrace, and Apple’s Crash Reporter looks for these events at the kernel level and opens a Crash Reporter process. When being used visually, SBCL will give you a dialogue box displaying this issue. However, when you run SBCL from the command line or even in the REPL in interactive mode (like you do in Emacs with SLIME), every compile will trigger an Apple Crash Reporter process. This means that after one hour of development you could have 30 or 40 crash reporters launched on your system without any visual indication. Since SBCL developers are aware of the problem, but have not patched the SBCL distribution, we do not recommend or support using SBCL in development on Mac OSX. You can still do it, but keep your Activity Viewer open.

Now it would appear the same thing is true with Allegro 8.0, but something tells me this will be fixed quick. There are a lot of scientist running OSX in the biomedical and computer science field and they are Allegro’s bread and butter.

I don’t think we’ll get the same from SBCL.

Probably the most read book on learning Lisp, Paul Graham’s On Lisp was limited online to just two chapters. Well, it’s now completely free and 100% complete! Anyone that wants to learn Lisp (or where Python gets it’s inspiration) should download this book, curl up with your Powerbook and read away (with emacs & slime or a GUI editor in another window).

Download in your favorite format here

Lisp has always been well positioned do to it’s functional programming model and list/symbol processing features to make a gigantic contribution to the Semantic web, or since people like a release number, “web 3.0″. In fact, many of you will be quick to point out that Racer Systems (which needs to hire a web designer fast) has led the charge of using Lisp’s incredible AI history to make some pretty compelling advances in reasoning systems with it’s products such as RacerPro.

The semantic web idea has been around for some time, and has in the last few years began to solidify around standards such as OWL/OWL-QL and SWRL. However, although a lot of the modeling tools for modeling ontologies and exporting them to OWL are opensource and has a vibrant community of developers (of course Protege from Stanford University being the one that comes to mind fastest), the backend reasoning systems that assembles these ontologies and creates relationships are either completely closed or free in binary form but closed source. The reason for this is obvious, the first group to build a really good OWL parsing and reasoning system will be very rich.

However, I have been looking over a lot of research from the likes of Eugene Agichtein & Silviu Cucerzan of Microsoft Research regarding speeding up extraction [Predicting Accuracy of Extracting Information from Unstructured Text Collections] and I can’t help but think that the community could use a backend that competed with, if not RacerPro directly, at least it’s object code library, RacerMaster.

I’ve been working on a side project building out some simple macros on how this could work, and think that if we all got together to work on building a great language model for expressing ontologies and manipulating them at the high level, as well as defining the relationships between them in a OWL-QL like language, we could significantly contribute to moving the semantic web environment forward.

So, I discussed the idea with a few friends with more education than me, presented it to the folks over at common-lisp.net and they decided it sounded like a great project. So, we’re up and running. Now I need your help :-)

This is how I see the project progressing:

• develop parsing of OWL Lite / OWL DL documents (using Wilber).
• develop easy to use macros for T-Boxes, A-Boxes, ect.
• grab a whole bunch of PhDs and get them to implement SHIQ for us.
• implement the OWL-QL query processing system for compatibility.
• implement the high-level language model for easy manipulation of the ontologies.
• give it Java wrappers or use Allegro’s and make people pay for that much.
• connect to Protege using simple TCP/IP or some protocol we can all agree on.
• beer for all the developers on me!

OK, maybe we should just start with bullet point 1 and be thankful we did that. However, I think developing this language model to compete with RacerMaster would be a great thing for the semantic web and Lisp communities.

If you want to help, visit the project page || email me using my contact info and PGP key (for adding you to the project if you are approved.)