From: miles@puzl.pixar.com (Miles Egan)
Newsgroups: comp.lang.lisp
Subject: evolving Lisp
Date: 20 May 2001 17:59:01 GMT


Like many of the readers of this group, I've spent a lot of time pondering
the popularity and evolution of Lisp.  I've also spent a lot of time
playing with the various implementations of Lisp out there and I've come to
a few tentative conclusions:

1.  Although Lisp users often criticize Scheme for its overly conservative
and incomplete definition, in many respects Lisp now suffers from the same
incompleteness for modern application development.  Many of the tools
needed to develop typical modern-day applications are non-standard among
implementations, crude and often only partially functional, or simply
unavailable.  FFI, database connectivity, network protocol bindings, gui
toolkit bindings, xml tools etc. are fragmented and unstandardized among
all CL implementations.

2.  Ironically, the diversity of CL implementations seems to exacerbate the
problem of extending CL.  The implementations I've seen of some of the
above tools clearly expend a lot of effort trying to support them all, with
code which is a mess of implementation-specific switches and parallel FFI
code.  I have to imagine that time spent dealing with cross-implementation
compatibility is time not spend adding robustness, functionality, or
documentation.  The irony here, of course, is that in some ways it is the
very standardization of Lisp that allows for this kind of forking because
there is a reference standard upon which to base new and otherwise
divergent implementations.

3.  As a result of this fragmentation, Lisp is actually a poor choice for
many important kinds of application development.  More and more it is
pushed into the role of a central application intelligence interfaced with
more mainstream applications which provide connectivity to the rest of the
world.  Only its superb core design has allowed to retain even this
position.  But for many other applications, the burden of maintaining this
kind of homogeneous system is too high and it really is easier to develop
in a more gregarious language.

To me, this is truly depressing.  The more I study Lisp the more I
appreciate the genius of its design.  As others have recently commented in
this group, mainstream computer hardware has finally evolved to the point
where all the old objections to Lisp are irrelevant and other popular
languages are much LESS efficient.  In the distributed, networked world of
modern software development, Lisp should be king, but it languishes in
niche applications.

A survey of the handful of languages that are growing reveals some obvious
commonalities.  Living languages today are either funded and marketed by
huge corporations with vast advertising and development budgets or are
open-source languages supported by volunteers.  Since a Java or C# push
isn't likely for Lisp, the only alternative seems to be a strong
open-source development effort.  My impression of the popular open-source
languages (Perl, Python, Ruby, Tcl and, to a lesser extent, Ocaml) is that
they all have the following in common:

1.  A single implementation.  This means any extensions or add-on modules
are developed only once and that their authors need only worry about
architecture-related portability issues.  All developer effort goes toward
maintaining and improving a single common implementation.  

In general, the authors of language extensions and external bindings are
less experienced than the gurus who develop the core language, so it's
crucial that the techniques for doing this are straightforward and
well-documented.  I think that, in the long run, the success of an
open-source langauge depends on these people more than on the core
designers.  Most of these people would run in terror if they saw how
difficult it is to write GTK bindings that work in CMUCL, CLISP, and ACL,
for example.

2.  A simple and straightforward means of bootstrapping the language on any
sane platform with a c compiler.  All of these languages are very portable.

3.  No langauge standardization bureaucracy.  The langauge designers are
free to evolve the langauge when necessary.  These langauges have evolved
quickly in response to user requests and technological shifts.

Although there are high-quality free implementations of Lisp, the
open-source Lisp community has so fair failed to build any of the momentum
of these lesser langauges.  I believe it's due to the failure to recognize
their strengths.  I may be heading a new and fairly large-scale development
effort soon which will involve a fair amount of the sophisticated logic and
intelligence for which Lisp is such a superb tool, but I don't think I can
choose Lisp in good conscience for two reasons.  First, because I know I
will waste a lot of time patching together code to interface to the rest of
the system and second because none of my developers are going to know Lisp.
Why?  Because the kinds of programs on which most younger programmers cut
their teeth are a PITA to write in any open-source Lisp.

What can be done about this?  I think it's imperative that the open-source
Lisp community should learn from the Pythonistas:

1.  Pick a single open-source Lisp and focus all development effort on that
implementation.  We have more than our share of brilliant implementors, but
splitting our efforts among CLISP, CMUCL and SBCL is lethal.

2.  Do what it takes to make porting this implementation as simple as
possible.  This implementation should compile and run on any new
posix-compliant platform with nothing more than a C compiler.  It should
also be trivial to make executables with this Lisp and it should function
as an executable script interpreter with no extra gymnastics.

3.  Take advantage of having a single reference implementation to define a
clean and straightforward CPAN-like system and a well-documented and
straightforward C FFI.  I know there are efforts to define something like
this now, but, to my eyes, they are all severely limited by the difficulty
of supporting so many implementations.

4.  Build full-featured and reliable interfaces to common toolkits and
protocols on this infrastructure.  Lisp should talk to databases as
fluently as Perl, parse XML as well as Python, speak SNMP, HTTP, LDAP,
IMAP, and SOAP, and couple with C as easily as Ruby.

5.  Take advantage of having a single implementation to extend and change,
when necessary, the Common Lisp standard.  Obviously this should be done
extremely carefully, but I think it's a mistake at this point to consider
the CL spec to be either complete or sacrosanct.

Speaking as an open-source hacker that would love to do what he can to help
promote and grow Lisp, I think these kinds of changes are necessary.  As it
stands now, trying to get anything done in the current Babel is even more
frustrating than watching Ruby, Perl, and Python slowly reinvent Lisp the
hard way.

-- 
miles egan