Brandon Werner

Some time ago I ran across one of your blogs in which you made some comparison between Solaris and Linux. I was very impressed and bookmarked it. I may have made a mistake but it seems that the article is no longer there… If it is possible, could you please direct me to where I can find it? Better yet, would you mind sharing the article (again) with the world? Thanks, Hering

Dear Hering,
With pleasure. It was on The Post Modern website and published August 4th, 2005. Here it is.

Back in 1999 when I was an 18 year old in corporate America doing my thing, you always knew who the really important people were because they had Sparc Ultra 5s on their desk in addition to their clunky Windows PCs. The rest of us were forced to use Hummingbird to emulate X and screen-pop our control-M to monitor jobs. I’m proud to say at the end of my run of working at this place, I had my own Sparc Ultra 5 with 21 inch monitor running Solaris 7 in the CDE environment doing C++ coding and building applications for servers. There was just something powerful about Sun OS that came through; it felt solid under your fingers, like a fine instrument. This was Unix Proper. But alas, it was expensive as hell, locked on expensive hardware, and seriously useful only for workstation tasks (it had a clunky Real Player and Flash, but that was it). Linux came on strong, with it’s free price and it’s compatibility as the primary target of all open source technology builds. Solaris binaries were well down the webpage, and getting lower.

This week, after years of GNU Linux, I’ve come home. I must admit when I saw CDE load on my screen again (I shun Java Desktop for server tasks) I was a little nostalgic.

The Hacker Learning Curve

Much has been made of the latest free Solaris 10 release for x86 and Sparc, and although the OpenSolaris project is gaining steam (but not up to Solaris 10 in functionality yet because of third-party issues) many hard nosed open source zealots are shunning Solaris for Linus’ and GNU’s old stand-by. This is where ideology can hurt you. Although I do understand people who grew up with Linux in their homes or small offices would never have exposure to a real Unix operating system, I think the vast majority of Linux users are staying away because the learning curve is steep, even for a seasoned Linux pro. Like I discovered when I installed Solaris 10 over the weekend, Linux has made us lazy.

Huh?

Linux comes from the same environment that scripting languages like PHP and databases like MySQL come from; they are littered with hacks and shortcuts from developers who just want to get it to work, and don’t sit down long enough to think about what a particular strategy means over the long term, or about others who may come along later. This has resulted in Linux being fast to administer and fast to get to do what you want, but as stability and universal configurations come in to play when a system grows, the need for serious structure in process execution and user management becomes clear. Solaris requires different thinking, and you graduate in the course of learning Solaris (or re-learning for me) from thinking only in the moment to really understanding the power and resources a Unix operating system provides when well configured. Many Linux users, sadly, will simply give up in frustration over this simple issue.

Here are some examples:

I pain to think of how many Linux admins are frustrated at why they can’t just log in to Solaris and do a /etc/init.d/tomcat start and then logout without Tomcat closing on them. nohup has no meaning to them. The idea that an OS garbage collects all the terminal processes a person starts after they log-off, even if they become root, is foreign to those brought up in the Linux environment. In Linux you can linger long after you’re gone. The same holds true for directory structure and security, which Solaris has not forgotten in it’s attempts to be more accessible.

Another thing that will make Linux users cry is the fact that, even as almighty root, you can’t create a directory in /home for users. Solaris smartly assumes roaming profiles where different uses will want to bring their /home directory with them to whatever box they go to, and the automount service watches over /home like a hawk. You have to create home directories in /export/home if you want to do it manually.

Init.d is so 90s

Another issue that can face a Linux switcher is Sun’s answer to the mess that is now init.d. The SMF (Service Management Framework) is a spectacular replacement for the old init.d and runlevel based process management, and provides milestones in place of run-levels and easy XML files that provide for restarting crashed services and maintenance mode of processes, something a Linux user has to deal with just by using /etc/init.d/service restart command and praying it doesn’t crash again. It also stores these files in a hierarchical fashion that works on both the intuitive and management level.

As an example, start-up XML files for DNS applications are stored in the directories:

/network/dns/server
/network/dns/client

If you wish to start these services, you can call the following:

svcadm enable /network/dns/client
svcadm enable /network/dns/server

The System Administrator themselves can write their own XML files and plug them in to the architecture, and several blogs exist that provide step by step guides for setting up MySQL, Tomcat, JBoss, and many other common services you may want to move over to the SMF to gain the benefits.

Even boot-time process starting is handled with greater elegance, since enable and disable carry over to all other boots of the OS. Want to only start a service now and have it not start at next boot? Don’t cd in to those rc3.d/ files, Just use:

enable -t /network/dns/server (for temporary)

With Solaris, you’ll get to drink a lot more coffee and have a lot less carpal tunnel.

Solaris 10: Linux Inside

Not that Solaris hasn’t tried to bridge the gap with the open source world. In fact, init.d/ does indeed work (mostly to start Linux and OSS programs like MySQL and samba) and the operating system comes loaded with all the Linux goodies such as Apache, Tomcat, MySQL, Samba, and the rest as well as a Gnome environment that actually completely works out of the box, the excellent Java Desktop System. It also comes with StarOffice pre-installed for workstation use. It keeps these, for the most part, locked away in /opt/sfw/ which is like having a complete Linux distribution mounted inside your Solaris OS. This helps ease the transition for Linux users, since ifconfig, top and the rest are there for your use.

All of Solaris’ applications are installed when you install your operating system, but none of them are turned on by default for security reasons. They are locked away by way of config files. Apache, Tomcat, Sun Studio Server 8, Samba and the rest are set up to start at boot time, but all the scripts in init.d look for a .conf file typical with the application (i.e. server.conf or httpd.conf). It provides these files after install with example implementations under the suffix of example in the various directories (For example: httpd.conf.example), but if there isn’t a .conf file for the process, it won’t start. To enable the Apache server, for instance, you would simply do the following:

cd /opt/sfw/apache/conf/
mv httpd.conf.example httpd.conf
/etc/init.d/apache start - now the server will start.

This is a much easier way of providing all the services a user might want to start without forcing them to hack the init scripts immediately, and gives them the ability to install the latest and greatest versions themselves without clashing with distro installed defaults, which often happen when you already have Apache installed on Fedora or Mandrake but want to compile your own.

Solaris 10: Documentation without the O’Reilly Tax.

What really separates Solaris 10 from the pack is the documentation. If you don’t think you’ll take advantage of documentation, it’s only because you grew up without it on Linux, and Google has been your only recourse. Solaris did not grow up in the world of HTML HOW-TOs and Wikis. Far from it, there is enough documentation online to kill a rainforest if printed. And it’s really good documentation too; documents that walk you through everything you’d ever want to do from setting up Sendmail to using NIS+ to configuring network services. It has high-level overviews with detailed walk-throughs and advances at the level of the reader by providing quick methods to get started and then drilling down in to the details as people get more comfortable with the services they are deploying.

To put it bluntly, Sun provides O’Reilly quality documentation for free, where open source technologies require you to run out and give O’Reilly your $40.00 to get the exact same thing. Although open source technologies (companies?) like MySQL, Tomcat and the rest are going the “professional open source” route and trying to offer the same level of documentation and support, the open source world will always be plagued by the fact that writing documentation, not even good documentation, is a horrible job and no programmer who just wrote a Really Awesome Thing is going to want to take the time to write a detailed technical book about it.

Beyond this, un-like the Linux admin community, the Solaris admin community is together and solid, with one place to go. It’s Sun’s BigAdmin site, and it’s loaded with everything you’d ever want to know or search for. Also, Solaris has incredible management applications that make managing users, groups, and projects a breeze (and gets rid of that pesky /home issue for you automatically). It now includes a webconsole interface as well for managing not just users and installed applications but containers (Solaris zones) as well.

There is so much more to talk about, such as the great package management, the Solaris zones which is virtualization and software partitioning that enables you to create many private execution environments within a single instance of the Solaris Operating System ( I have three set up on my home server), the DTrace debugging system and predictive self-healing. However, these technologies are so beyond the current Linux feature set that I doubt a “switcher” would even be motivated to use them immediately.

It’s Only My Opinion

My reasons for advocating a technology are plain and not tied to ideology. Apple has the best client operating system, with expose window management, innovative unix technology and GUI, while still being able to run Microsoft Office and Adobe apps that are in the customer marketplace. Therefore, I advocate them. Java is one of the most flexible and powerful programming language in the world, and in a world where international governments are passing laws requiring open source to be used exclusively for computers and servers, C# just isn’t going be portable beyond the United States in the near future (unless you want to hang your fortunes on the fever-inducing named Mono). Therefore, I advocate Java.

My reasons for advocating Solaris are the same: Now free, it is simply the most advanced and powerful server operating system available to you, with plenty of face-time with highly paid doctorates of computer science over the years to make it’s advantage lasting while also playing well and complementing the open source technologies that most people use. It runs fast from the PIII 256 MB RAM machine I have tested it on to the run of the mill Intel Pentium 4s I use as servers at home to the fastest 64-bit AMDs we have in the dev environment at my start-up.

Even if you are a Linux lover, you owe it to yourself and your intellectual curiousity (that thing that makes you format your box every 9 months with some new OS) to try out Solaris 10.

I’m twenty-six years old. It might seem odd that, given I’m so young, I would have fond memories of an OS like VMS, an architecture like VAX, seeing a sound studio hooked up to a Sparc Ultra 5, an old man who knew everything about everything and an old Catholic sacred store employee who hacked mainframes in the 50s. However, so far I’ve taken the road less traveled in my technology career, and because of that have had a great time playing with and learning things most people would never have the opportunity to see or use. I write about that here, in particular, my experience with Gurus.

Guru Is Sacred

HR recruiters use a lot of words they don’t understand. The one word, however, I must ask back is the word Guru. Turn it over now, and pretend you never heard it. Do not ask, in your very scientific rating system for skills, if a person is “Beginner, Intermediate, Expert, or Guru”. If someone was a Guru, they would not fill out this form and they certainly wouldn’t be working for you. As anyone that has ever been in presence of a real Guru understands, there are no “Java” Gurus or “Linux” Gurus or any other demarcation. Gurus transcend all of this, and yet embrace it all as well. They are at least 50 years old or more, they do not conform to any past, current or future dress trends (go ahead and tell them about your dress code standards, just make sure I’m around when you do) and they certainly don’t care about money, fame or titles. They are as close to priests as technology will get, and hearken back to an era when technology was science and imagination, not press releases and vaporware.

When true geeks encounter such a person, we instantly sit crossed-legged on the floor, turn off our cell phones (not just vibrate) and say, with deep respect “Tell us stories!”. I’ve only met two in my life, and they were in the oddest places. You won’t find them at conferences or pitching their latest book. You have to find them, they don’t find you.

Catholic Computing

The first place I met a guru was in a Catholic store. Seriously. My father is a Catholic and, as it seems with most religions, alternates between fits of devotion and fits of secularism. During one such fit of devotion, he took me to the Catholic store in the city. Cincinnati has a deep history of both German Catholic and Jewish immigrants and so finding these exclusively Catholic stores is not hard (but getting harder). After grabbing a rosary, a few holy cards and other knick-knacks, we moved to the counter where we met a very gracious old women in her 60s who you could tell was doing this very boring and leisurely job for a higher purpose than a paycheck. As she was packing up my father’s purchases I was talking about one of the various jobs I was doing and what it entailed. I as 17 at the time, and had moved to the Delta Upsilon frat house in Columbus to go to Ohio State University. However, as I needed money, I applied for a small job at Nationwide Insurance doing mundane mainframe vs. UNIX report resolution using nothing but a thick mainframe stack of printed batch jobs, a UNIX terminal, and a calculator (it was a nice desk one though). As I found problems, I went through the C code or COBOL and figured out where the problem had occurred. I had done so well that I was hired full time at Nationwide, and school was put on the backburner in a not-so-smart move. Instantly, the woman’s face lit up in surprise and delight.

“I worked on IBM Mainframes in the 50s”

My father smiled and started to walk away, not knowing that in that moment she had given herself away and that I was obligated to stay a lot longer. Instantly her face and manner changed. She moved from extruding very stoic resignation and comfortableness about her life and her old age in to a fiercely intellectual and witty archetype of a Guru. She was more alive than seemed possible, and proceeded to tell me about her punch cards, the programs she would write, the ins and outs of the machines she would use, how her hands were sometimes bloodied by the very mechanical machines that read things in and spit things out, and the things she loved most about the math behind the machines. She worked on a program that calculated something for the government, but the exact specifics had long faded from her memory. She said she retired in the late 70s and hasn’t liked or cared for the personal computer thing that happened afterwards. She said she could never understand a mouse. I eventually left after absorbing all her stories, and her pious Catholic manner we had first seen on arriving was beginning once again to poke through as she got tired. I promised I’d come by again, and she said she’d really enjoy it, but I never did.

Solaris does Apple

Years later I would run in to another Guru. I was in the unenviable position of being picked by the director of a large company in Cincinnati to travel with him and one other person to Columbus where the company had just purchased an IVR / Interactive Voice Response company to use in connection with it’s Customer Service outsourcing business. This company was built from the ground up (the CEO actually made the custom propeller blade ceiling fans that were used in the company), everyone had known each other for years, and I was one of the lucky ones that had to learn all the technology and then take the equipment, the knowledge and the best practices back to our corporate headquarters for assimilation. Of course, the employees would be left behind. This was not a fun assignment.

The technology used was mostly homegrown as well. Because of the massive amount of lines needed to run the inbound and outbound automated call centers, the business was located right next to an AT&T POP (Point of Presence) and they were a heavy user of Periphonics and Lucent IVR software, which ran on top of Solaris 7. What was amazing was that to interface with the Periphonics IVR for voice prompts you had to use a custom language. To interface with the Lucent IVR technology you wrote in.. wait for it.. COBOL. What was also amazing was that the client for all audio recording, encoding and management was on Solaris as well. I gawked at seeing an entire sound studio hooked up to a Pizza box Sparc Ulrta 5 with a wave-editor and audio tools in a CDE interface. Imagine the maroon ugliness for a moment…

What caught my eye most though was the wall long library of technical books in the center of the building. As I walked closer, I could see they were all one publisher, but various editions. As I got closer I saw the Digital logo. In my final steps I could finally make out what they were, ten years worth of VAX/VMS technical manuals.

“Don’t touch those, they’re Bills.”, said a cold and aloof employee passing by.

Apparently, Bill was an old-skool consultant who ran the whole show and was the brains behind the entire setup. Sure there were System Administrators and IVR programmers, but the rest of the company and the connections between them were invented and managed by him, and he loved VAX. Everything was VAX. These were old systems (they still had the Digital nameplate on them) but they ran the latest version of OpenVMS for the time (1999). Although most of the technology he wrote would be replaced by the integration, there was one small thing we couldn’t replace: A web-based reporting system that all clients used to get their call metrics, written by Bill, which ran (obviously) on VAX/VMS and was written in, of all things, DCL and Perl.

I would have to meet Bill.

The Functional Guru - A Digital Rogue?

Later that week a rusted, blue car of un-discernable make and model pulled in to the parking lot next to the POP. Out came a man around 60 years old, wearing a pony-tail behind his very bald head and red sweat pants that were a little too tight for the women’s stomachs. Through the days, the outfit changed very little. For our sanity’s sake we assumed he had more than one pair of red sweat pants and didn’t ask. Others scoffed, but I admired him as having a Berkley-esque manner about him. We sat down to go over the system, and after the following turn-over weeks I became a deep admirer of his. He was, certainly, another Guru.

At first I had imagined he had put a lot of traps and complicated code in to the system to ensure a paycheck. Why else would he stick with VAX/VMS and write code away from such tried technologies and frameworks? As he walked me through the system, however, the elegance of his design began to show through. He had, after all, managed to write an entire web based reporting application that supported thousands of clients and queried hundreds of thousands of records in an application that was smaller than the HTML it eventually rendered. I now know he used a lot of functional programming methodology, basically building list-like code structures that built on top of each other and essentially became a very easy macro interface. It was a feat I would have admired had it been in Lisp, but what made it even more impressive was that it was in Perl and obscure DCL. Certainly, it was not terse, but he didn’t seem that concerned. It was also extremely scalable and customizable, in a manner than I must admit I’d still want to use an entire JavaEE application stack and code to do myself. Gradually, as I would continue to use and work with the application and expand it to our needs, it was the only part of the whole system that never once crashed, balked at a complex programming task or did something unexpected. It was a bubble unto itself, with no huge frameworks, no webservices or XML config files. It was a pure, clean web application that looked great on any browser and ran as fast as the wind.

This code could only come from a guru.

Further, he talked about why he loved VAX (Virtual Address Extension) and in particular the OS OpenVMS. He use to work on the VAX development at Digital all the way back to 1975 with the PDP-11/20, and had helped develop the software for the VAX-11 architecture. He showed me file versioning (and why it was a good idea), the special commands that allowed you to tune the kernel and how to manage the network and filesystems in a distributed manner. He talked about UNIX in general, about his days of programming for the Apple IIe, about how the old Apple’s wrote to the disk drive based on processor ticks (before the Folklore website existed, by the way) and how he first wrote the drivers for some Apple hardware for a company in Columbus. He knew, at every execution, what was going on in the compiler, in the code and in the object model. I learned more from him than I could ever say.

The thing was, whenever another programmer or manager type would come up to him, or ask him a question, he would pipe up and give the very default answers. This led them to come to the conclusion he was a hack, or at best unproductive. I knew differently, and also understood that true Gurus don’t really care if you believe they know what they are doing or not. They simply assume the engineering respect that most developers don’t get from the business side these days, and it’s most likely why they aren’t seen in very many data centers or cubicles anymore.

The Need For Elder Respect In Technology

That’s a shame, because what shows through in both of the gurus I’ve met is how they desperately wanted someone to share all of their knowledge with, their successes and their hacks that they had invented over the years. Its sad to think there are people with these great stories and knowledge without an ear to talk to. You see, the only thing that Gurus can do is teach. Not that they couldn’t code C or Lisp or C++ around us youngin’s three times fast, but that would be a monumental waste of talent. In a world where Economics has Alan Greenspan and Biology has Timothy Hunt and Paul Nurse, Computer Science puts it’s best out to pasture way too fast, making them nothing but fodder for PBS documentaries.

What they teach is more than just heroic stories or a good way of thinking of a problem, but not to give up being who we are. As the personal computer has taken off and information technology has moved from science and imagination to business cases and flame-wars on Slashdot, these people remind us that the core of who we are is also alive in them, and that in many ways they found a way to express that passion that we may never get a chance to.

When we, as programmers, rally around Linux or point to the beauty of Mac OSX or talk up the idea of open standards and open source code, we do not rally around these things individually and for separate reasons, but because we all share in the idea that technology can be something amazing, liberating and powerful even if it may have lost it’s way. Geek culture is at it’s core optimistic about humanity, even in it’s weary cynicism of today, and it’s only in talking with these gurus that we can understand that this same shared tradition of optimism is also a heritage, one we must respect.

Steve Jobs stated in his commencement address at Stanford last year that after he had been removed from Apple in the 80’s he apologized to Mr. Packard and others for letting the previous generation of innovators down. While if he did in fact let these people down could be disputed, I find in his words the best illustration of someone who understands the Geek culture, someone who understands we have a heritage to respect. I wonder if it even occurred to the companies and internet start-up firms to apologize or feel guilty for letting the previous innovators down, for losing site of the goals of technology for their own personal gain. I’m certain those who invented the World Wide Web on a NeXT box back in the 1980s or worked hard on the TCP/IP protocol in the 1960s didn’t get one red cent from the boom. Then again, these people wouldn’t care, they weren’t thinking about money when they churned away for endless hours on it. They got it.

So, don’t ask for Gurus. You don’t really want a Guru. They care a lot about the bottom line, just not yours.

Whenever there is a major JDK update everyone on the Sun Microsystems side seems to love to beat the drum of upgrading. This leaves developers in a situation where they love the new features of the new JDK but quickly find out the rest of the world isn’t so quick to follow them. Recently, even James Gosling argued with developers about moving to JDK 5.0, stating that, “Don’t wait for a .1 release… We’re not going to do it because 5.0 is very stable. Don’t wait just get the .0 release.”.

Worse, the platform of choice of Java developers, Apple Mac OSX, has just released a new update to their J2SE 5.0 Developer Preview that finally changes that magic symbolic link to make JDK 5 perferred (before you had 5.0 but all apps ran 1.4.2 by default).

Well, after having just got burned again by coding to JDK 5 with the StringBuilder, the autoboxing, generics and the printf() for a friend only to find out the deploy envionment doesn’t support JDK 5 yet, and having to spend two days changing by code, (and feeling the pain I cringed at after reading Kirill Grouchnikov’s entry on Java.net in July ) I created the following list of things that must occur before people stop coding in JDK 1.4.2:

1. Sun’s own tools, such as NetBeans, Studio Creator, and Enterprise Studio must not ask for a JDK 1.4.2 or refuse to install and in the future, they should not recommend people upgrade their JDK UNTIL Sun’s tools themselves can at least run with the latest JDK, not a year after the JDK’s release.

2. All third-party vendors that have shared VMs across their Tomcat hosting systems, such as GoDaddy.com, must give us good information on what JDK is installed and when it will be upgraded.

3. Someone must write a NetBeans or Eclipse plug-in to reverse all trivial JDK 5.0 uses such as auto-boxing and StringBuilder. Don’t write, I know IntelliJ already does this and provides intentions to correct this, and that is why all my opensource and professional development uses IntelliJ and provide licenses to others who work with me. Also, I am aware of RetroWeaver, but I’ve heard it’s success is uneven.

Until these things happen, I’ll just have to write to JDK 1.4.2 and try not to think how easy .Net developers had it when upgrading to the recently released .Net 2 Framework. Yes, Java runs everywhere, but not evenly.

I just got done talking about the interview, which is happening in 5 hours, and apparently Tor Norbye is going to go to Jonathan’s office and install Skype for him on his laptop. I hope someone gets a picture of Jonathan Schwartz with his microphone and headset on! Dick and I ran in to a lot of problems with Skype doing interviews with Bruce Tate and Tor Norbye.. what is worse is that Tor and Jonathan are on the West Coast and Dick is on the East / South area.

So, if you don’t mind, could you please turn off your internet router around 6:00PM EST for them? Just an hour, all I’m asking. Why not read a book instead of playing World of Warcraft?

I am proud to see that Jonathan mentioned in his blog that he is going to do the show with the JavaPosse. I’m excited beyond words for Dick Wall, who I believe has taken over my role of getting grief from the pod-listeners regarding their show. If I were them, I’d disable comments and just let email be submitted as feedback. If there is one thing I’ve learned being an academic, it’s that the commoners are many and mean. I think there should be a question on what your favorite opera is before you can post, and if it’s not Gounod’s Faust, you don’t get to comment.

Dick and I were working on the Jonathan Schwartz interview for some time during the JavaCast, and I’m glad to see it came about finally.

Also, the JavaCast archives staring me are online! For those who are complaining that Tor and Dick just read the news and don’t comment or analyze, perhaps you should listen to my endless rants and analysis after every news item on the previous JavaCast shows; It will allow you to appriciate the more rushed news they do now. Is there a happy medium? Perhaps, but it’s not nearly as much fun.

The best and most relevant JavaCast is the Bruce Tate interview we did two months ago regarding Beyond Java, a book that I predicted in the interview with Bruce would be huge and I was right.

He is a great guy, and I think it’s making a lot of people take a new look at how much longer Java has in the world. Luckily for Sun, they have two huge things going for them now, Java and Solaris, and I think Solaris will be their new claim to fame for mass adoption.

I am a huge fan of Jonathan Schwartz, and I don’t know why. Yes I do. I relate to him. He mouths off and fights and scraps no matter who he is with or sitting in front of, and he is always excited about the new ideas, even if they get ridiculed. He might be too big for Sun to be honest, I’d love to see him have a higher profile. The tech industry desperately needs more people that can tell Tim O’Reilly to go to hell. Jonathan called almost every bluff poor Tim tried to throw at him at his Web 2.0 conference. Download the podcast and listen to Jonathan. To those people who are harsh on Jonathan & call him arrogent, I think you’re confusing confidence, which is derived from intelligence, with arrogance, derived from ignorance.

I look forward to the interview.