I was wandering around Wikipedia today when I came across this article on the Dunning-Kruger effect. According to the article, this effect is "the phenomenon wherein people who have little knowledge tend to think that they know more than they do, while others who have much more knowledge tend to think that they know less."

The article goes on to say, that Dunning and Kruger "hypothesized that with a typical skill which humans may possess in greater or lesser degree,

1. Incompetent individuals tend to overestimate their own level of skill.

2. Incompetent individuals fail to recognize genuine skill in others.

3. Incompetent individuals fail to recognize the extremity of their inadequacy.

4. If they can be trained to substantially improve their own skill level, these individuals can recognize and acknowledge their own previous lack of skill."

They tested their theories on a bunch of Cornell undergraduate students which verified their hypothesis. I wonder if the students knew what they were getting into?

Finally, the world makes sense. It feels so good to know that it has a name.

The quote below from Neal Stephenson is from the book In the beginning...was the command line. I think the quote may be something of an over-generalization, but I understand the point he is trying to convey. In the computer world, simplicity and ease of use are two phrases oft repeated by software vendors. Of course, they only say these things because it works. These phrases help sell the product.

Computers are complicated beasts. Software is complicated. Networks are complicated. Security is complicated. I think this complexity creates a deep need in people. People want an easy answer. Complexity means stress, insecurity, sleepless nights, and bad dreams. These problems are why so many people fall victim to the idea of the "perfect solution". The solution that eliminates all of the complexity, and makes the nightmares go away.

I guess it is simply human nature to want an easy way out, but it is sad to see how much money is wasted chasing a perfect solution rather than the best solution. Finding the best solution involves work. It involves an actual understanding of technology that only comes through study and experience. It also involves making trade-offs because the best solution will never be perfect, and knowing which trade-offs to make involves taking a good hard look at what your real needs actually are.

As a developer, I think that it is worthwhile to follow Einstein's maxim to "make things as simple as possible, but not simpler". The second part of that quote is the important piece. Neal's quotation is certainly true within the context of the "perfect solution". It just is not true in the general sense.

A good interface can certainly go a long way toward hiding the complexity of a system. For example, consider the transmission in your car. Have you ever seen the inside of one of those things? The complexity is amazing. However, you do not have to understand any of that to drive a car. For an automatic, you just have to know what the letters P, R, N, D, 1, and 2 mean. Now that is a nice interface.

The problem comes about if a person tries to rely on the simple interface when they really need to have a deeper understanding. For instance, I would not want to have my transmission serviced by a mechanic whose knowledge only extends to the meaning of the letters on the shifter. We expect more of mechanics. Surprisingly, the computer industry often has lower expectations.

I hear the commercials all over the place. Pay us some money, and in a few months time you will be Microsoft certified and have a good high paying job. The problem is that the commercials are probably true. The people coming out of those classes probably do get a decent job, but how much of a fundamental understanding do they really have. The company is hiring a person to fix a transmission because they know how to shift gears.

In the Rails world, I have heard people complain about Rails being too easy to use. Apparently, the simplicity of Rails makes it possible for people to sell themselves as a web developer without a real understanding of what is going on underneath. I think it may be the equivalent of not letting a kid use a calculator until they know how to do basic math by hand.

Back in the early days of web development, I learned CGI programming with PERL. Then PHP came along, and I never looked back. Until last week, when I found myself writing a CGI wrapper in C around some legacy C code to make it available as a web service. CGI at this level is really the web programming equivalent of assembly language. There are no nice abstractions to hide behind, and the only solution is to have a real understanding of what is really going on at a low level. I am certainly not recommending that we return to CGI programming, especially not in C. However, it could certainly help someone whose only experience is with Rails.

In summary, allow me to dispense some wisdom as I sit atop my high horse. To the computer users of the world, stop chasing the "perfect solution" like a drug, and stop throwing money at any company that offers it to you. You are helping a bad business to thrive. To the "computer professionals" whose only experience is a two month class and a certification, you really don't know anything. Do yourself and the industry a favor: keep learning, dig into the technology that you work with until you really do understand it, and stop giving the industry a bad name.

One of the things that I love about technology is that it does make life easier in ways that truly matter. It also brings new complexities that did not exist before, and these complexities demand intelligence and understanding. Such is life.

A week ago, I was attending an Oracle class on their App Server. At one point, the teacher mentioned that one of the scripts was written in PERL, and a loud moan came from the back of the class followed by the statement "I hate PERL". I was one of the few developers in the class. Most of the other people were sys-admins, and more than one of them admitted that they did not know any programming languages.

I was intrigued by the strong sentiment expressed against PERL, so I struck up a conversation with the person who made it during a break. It turns out that he was also not a developer, and he did not know the language. His hatred for PERL stemmed solely from a past experience with a particular PERL script that came with an ORACLE product. Apparently, it had some problems.

What I found interesting in the conversation was that his experience with a single script led to his strong dislike for the language itself. Not to a hatred of that one buggy script, or the developers of that one script, or even for the company that delivered the script, but the PERL language itself. I wonder how many times people come to a conclusion about a language based on a single experience like this?

Of course, I used this conversation as an opportunity to mention Ruby, and how much better I thought it was than PERL. When I first came to PERL, I had written a good many SED and AWK scripts. Usually, these programs consisted of multiple SED and AWK scripts tied together with a shell script which was something of a pain. When PERL came along, it was pretty exciting. PERL had all the features of AWK and the shell scripting languages combined. Plus a whole lot more.

The next thing I knew, the web caught on and I learned about CGI scripts using the same language. I never failed to be amazed by just how powerful PERL was. Until this time, scripting languages were not that impressive. You used them occasionally, but the real work was done in a compiled language. PERL really changed all of that. I learned just how useful hash tables are, how nice it is to have dynamic data structures built into a language, why strong typing is incredibly overrated, and how great life can be without the whole compilation step.

But, no matter how much I accomplished with PERL, there was always a nagging little thought in the back of my mind that would not go away. How could this ever be used in a production environment? I was not sure I could ever convince my managers to use PERL for delivered code, and I was not sure that I wanted to. PERL certainly had power and a lot of cool features, but it had one serious problem: it was too easy to write code that no one could understand six months later. Not even the original programmer could figure it out half the time.

Sure, a good programmer with some discipline could write PERL code that was maintainable. The problem is that good programmers are in the minority. It will always be the mediocre programmers that make or break a software project. Bad programmers do limited damage before they get fired or moved somewhere else. Mediocre programmers hang around for the long term. The kind of people who got a comp-sci degree just to get a good job. Someone who says "I use a computer all day at work, why would I want to use one at home?" A person who never learns a new language until they are told to.

Mediocre programmers do not get fired. The code that they write generally works. It just has more bugs throughout it's lifetime, it is harder to maintain, and rarely shows any signs of elegance. After a while, their code settles into an uncomfortable stability due to the fact that no one wants to touch it.

In the hands of a mediocre programmer, PERL is the equivalent of letting a six year old child drive your Lamborghini. I hope it works out for you, but I will stay home that night. When Ruby came along, it did not take me long to switch. Ruby had all of the power of PERL with much more elegance. Sure, a bad programmer can write Ruby code that speaks of madness and abomination, but they are quickly expunged.

With Ruby, the mediocre programmers in your organization have a much better chance of writing maintainable code. And the good programmers will be free to do something amazing.

Having been a huge movie fan for most of my life, I have always been interested in the movie industry itself. Over the years, I have often been struck by the similarities between the movie industry and the computer industry. The quote from George A. Romero given below is from the January issue of Rue Morgue magazine where he is discussing his latest film project.

It will be interesting to see if the film industry will have an open source revolution of it's own. Originally, in the computer industry, all software was written by governments and large corporations simply because they were the only ones who could afford the hardware to run it. As powerful hardware became cheap and ubiquitous, the open source revolution was born. Much to the consternation of the established software companies, many of whom are still reeling from the shock.

Initially, the open source movement was viewed as the product of a bunch of ideological "hippie" types touting software that could not even begin to compete with commercial code. Over time, open source software has risen to a level of prominence that no one could have predicted. What disturbed people the most in the beginning was the whole issue of how to make money from software that was freely available. Many people predicted that open source would never be commercially viable.

So, here we are now. Some of the companies that had the tightest control over proprietary software products have gone out of business completely: Silicon Graphics, anyone? Others, like Sun, are adopting more open source products, and adding open source licenses to existing products like Solaris and Java. Oracle sells an application server based on Apache, and now has a virtualization product based on Xen and a version of Linux based indirectly on Red Hat Linux. These companies certainly seem to be making a hefty profit out of free software.

The really cool part of this situation is that a lot of really creative ideas get implemented from outside the corporate structure, but companies are still able to make money from them. In the end, though, the greatest benefit has been to the consumer who have much more freedom to use software in a way that meets their needs rather than the needs of a large corporation.

In the film industry, making movies has largely been under the control of giant studios due the nature of the technology. Films cost a lot of money to make, edit, and distribute, but the technology is starting to change. Equipment to make and edit a movie is becoming cheaper all the time, and the advent of streaming video on the internet does away with the distribution problem. Famous actors can still demand large salaries, but many noname actors are willing to work for a lot less.

The real question now is whether the talented people with the creative ideas will find another business model that allows for a more collaborative and independent effort to succeed. Will we see talented individuals banding together to create movies that they distribute for free, and will this approach spill over into other types of artwork? One thing we can be sure of though. If such an effort occurs, the major studios will hate it. Some of them will adapt, but others will go out of business. Regardless of how it all settles out though, the greatest benefit will once again be to the consumer.

Good talk by David Pollak on "How To Design A Domain Specific Language".

I used to have the actual video embedded here, but I discovered that my tumblelog would no longer validate due to a bunch of non-standard tags for the Flash movie. Yes, I do know about the Flash Satay method. A bit of a pain if you ask me. Much easier to dump the whole thing, and give you a link to the video.

You may want to contact the maker of your browser, and give them a bit of a what for: web standards and what not, fancy a shag, interoperability, and we all just get along and such. Not like it is now. A bit of a muddle and all.

So there you have it.

On the third day of rubyconf, the morning started off with an entertaining talk by Dr. Nic on "Using Ruby to generate More Ruby - RubiGen is Everywhere". Dr. Nic presented a history of RubiGen development using an A-Team theme. RubiGen is a generalization of the generator scripts used by Rails. Currently, RubiGen has been incorporated into Merb to provide it with a generator capability similar to Rails, but it could also be used by any applications that need this kind of functionality.

The second talk of the day was by David Chelimsky and Dave Astels on "Behavior Driven Development with RSpec". RSpec provides a DSL to assist with test driven development, provide customer acceptance tests, and can also help with documentation. I liked how it allows the acceptance tests to use the customers own vocabulary to describe the testing scenarios.

The third talk of the morning was by Jay Phillips on "Next-Gen VoIP Development with Ruby and Adhearsion". This was largly the same talk that Jay gave at the Ruby Hoedown that I reviewed here. Still a great talk on an interesting subject.

The afternoon of the second day was again divided into two separate tracks. I attended Luke Kanies talk on "Essential Incompleteness in Program Modeling". This talk was on the application of Kurt Gödel's incompleteness theorems to software development. Luke argued that a model of any sufficiently complex system can never completely match reality.

The last talk that I attended was by Francis Hwang on "Conversations vs. Laws: What do we mean when we say Ruby is dynamic?" The talk was interesting and covered a lot of ground as it incorporated ideas from philosophy, politics, and biology in relation to the dynamic features of Ruby. He first posed the question "Does anything that matters have a fixed definition?" This question was then driven home using examples from modern day social controversies such as the nature of race, the definition of marriage, assisted suicide, and the efforts of an overly scrupulous town council to provide an exact definition of buttocks.

Luke's talk seemed to be a reformulation of the idea that everything is a trade off. The classic programming example being the memory vs. speed thing. Francis' talk seemed to be a post-structuralist justification of the dynamism and agility that can come from a less rigid view of the world.

With Marcel's talk on the application of the classical theory of beauty to software design, this would make three talks on philosophy at one conference. Although, I doubt that the subject is exhausted. For instance, how about the application of existentialism to traditional software development processes that use the waterfall method. I foresee endless discussions on meaninglessness and the nature of the absurd.

The second day of RubyConf began with talks on IronRuby, jRuby, and Rubinius. The day started with John Lam's talk on the "State of IronRuby". IronRuby is a .NET implementation of Ruby that can be integrated into the .NET world. It looks like they have made a fair amount of progress. Although it allows Ruby code to be integrated with the .NET stuff, the process looked almost as painful as writing JNI code in Java.

Speaking of Java, the next talk was by Charlie Nutter on "jRuby: Ruby for the JVM". It looks like the jRuby staff is also making a lot of progress. I had seen the Ruby IDE in NetBeans before the conference, but I did not know that it was powered by the jRuby parser. It is also nice that you can integrate Ruby with Java code. Although, it did not look any more fun than the .NET stuff. I think the major downside to jRuby is the inability to use Ruby libraries such as RMagick that incorporate C extensions.

In an ideal world, I would prefer to just do Ruby. Having the IronRuby and jRuby implementations are a nice way to get Ruby in the door of companies that already have a large codebase in .NET or JAVA. Basically, you bring Ruby into an existing .NET or JAVA environment, people see how great it is, the .NET or JAVA environment goes away, and everyone just does Ruby.

The final talk of the morning was Evan Phoenix's talk on "Rubinius 1.0". This talk was definitely the most exciting of the three. Evan emphasized one of his main points by asking the audience how many people would rather program in C or Java instead of Ruby. Not surprisingly at a Ruby conference, no hands were raised. One of the key features of Rubinius is that a large part of the codebase is actually written in Ruby instead of C. This allows the Rubinius programmers to take advantage of the many features offered by the Ruby language. It also allows more Ruby programmers to participate in Rubinius development without having to have a background in C programming.

Rubinius is definitely where the excitement is, and it looks like they are making good progress. It will be interesting to see where this goes in relation to YARV as Ruby 1.9 comes online.