This book was made possible by a publishing system called Pollen. I created Pollen with the Racket programming language. Racket is a descendant of Scheme, which in turn is a descendant of Lisp.
So while Racket is not Lisp (in the specific Common Lisp sense), it is a Lisp (in the familial sense) which means that its core ideas—and core virtues—are shared with Lisp. So talking about Racket means talking about Lisp.
In practical programming projects, Lisps are rare, and Racket especially so. Thus, before I embarked on my Lisp adventure, I wanted to understand the costs & benefits of using a Lisp. Why do Lisps have such a great reputation, but so few users? Was I seeing something everyone else missed? Or did they know something I didn’t? To find out, I read whatever I could find about Lisps, including Paul Graham’s Hackers & Painters and Peter Seibel’s Practical Common Lisp. (OK, parts. It’s a big book.)
What I found was plenty of Lisp flattery from expert Lisp programmers. (Also plenty of Lisp kvetchery from its detractors.) What I didn’t find were simple, persuasive arguments in its favor. So here’s why Racket was the right tool for this project, and what I see as the practical virtues of Lisps in general.
I didn’t study computer science in college (though I was a math major for two years, before switching to design). I’ve never held an official job as a programmer. Rather, programming has been a secondary skill I’ve used in my work as a web designer, type designer, and writer.
These days, I spend an increasing amount of my time programming. This programming generates income. So by the simplest definition—does the skill make you money?—I suppose I qualify as a professional programmer. And since most of my programming efforts are in Racket, I qualify as a professional Racket programmer.
Mind you, I’m not claiming that I’m a good programmer. (You can decide for yourself.) Among the Racket community, which is laden with computer-science PhDs & professors, I (have no choice but to) embrace my relative ignorance. Hence the title of my talk at RacketCon 2014: Like a Blind Squirrel in a Ferrari.
Yet despite my flaws as a programmer, with Racket I’ve been able to render bigger ideas into programs more quickly, and with fewer bugs, than any language I’ve used before (and there have been many—Basic, C, C++, Perl, Java, JavaScript, Python, and others). Since I haven’t gotten a brain transplant recently, there must be something special about Racket as a language.
Lisp is a language most programmers have heard of, for two reasons. First, it’s one of the oldest computer languages, in use since 1958. Second, it’s accrued a reputation as a language for brainiacs. Originally this reputation arose from its association with the field of artificial intelligence. Since then, this reputation has been maintained by periodic endorsements from respected programmers (latterly, Eric Raymond and Paul Graham) and the enduring fame of the textbook used in introductory computer-science courses at MIT, Structure and Interpretation of Computer Programs (which uses Scheme, and that one I did read start to finish).
But as mainstream programming tools, Lisp and its descendants have been largely ignored. Popularity of programming languages is tricky to measure, but here’s a simple proxy—let’s count the number of projects currently hosted on GitHub. One could quibble about the accuracy of this method, except that the results aren’t even close:
Language | GitHub projects |
---|---|
JavaScript | 885,467 |
Java | 644,711 |
Ruby | 622,088 |
Python | 432,533 |
PHP | 420,913 |
C | 218,287 |
Clojure | 21,108 |
Common Lisp | 6883 |
Scheme | 5128 |
Racket | 2727 |
The last four languages are Lisps, and together account for only 35,846 projects. Racket itself only accounts for a small fraction of this small fraction.
Popular programming languages aren’t necessarily good—look what’s at the top of that list—but unpopular languages often have fatal flaws that prevent wider adoption. As I was considering languages, Racket had a lot to recommend it. But was there a fatal flaw I was overlooking? And by committing to a Lisp, would I be painting myself into a corner? I wanted to understand the risks and benefits.
I said above that Lisp flattery is easy to find. The problem with Lisp flattery is that it makes sense only to experienced Lisp programmers. To others—especially those who are trying to decide whether to learn and use a Lisp—it just comes across as unsubstantiated hoodoo.
For example, in his essay How to Become a Hacker, Eric Raymond says
To be fair, Raymond’s essay is not focused on Lisp. But compare Beating the Averages, by Paul Graham, which is. Graham starts off by citing Raymond’s compliment to Lisp and seems ready to make the claim concrete.
Instead, he breaks it into smaller chunks of flattery.
Graham offers one concrete example: Lisp’s macro facility, which he describes as its ability to make
I was hopeful when I opened Peter Seibel’s Practical Common Lisp and saw that the introduction was subtitled
And by the way, when do I get the speed and power you keep promising?
In short—what’s in it for me, now?
This is the fundamental question that Lisp advocates have to answer for new users. But more often, it’s sidestepped. I’m not picking on Raymond or Graham or Seibel. They’re excellent writers, and as programmers, they’re way out of my league. As I learn more about Lisps, I return to these pieces and they make more sense.
But these pieces are also emblematic of a general weakness of messaging about Lisp. I say that not as a ranking member of the Lisp community, but rather as someone who spent a lot of time seeking an answer to that fundamental question. I never got it.
Seibel is passing the buck when he says that to understand the benefits of Lisp,
That’s asking too much. If Lisp languages are so great, then it should be possible to summarize their benefits in concise, practical terms. If Lisp advocates refuse to do this, then we shouldn’t be surprised when these languages remain stuck near the bottom of the charts.
In a word, expressiveness: the measure of how easy it is to put your ideas into code. For instance, an expressive language like Racket lets you write the
"Hello world"
Whereas a less expressive language—I won’t name names—requires this:
public class HelloWorld {
public static void main(String[] args) {
System.out.println("Hello world");
}
}
Concision is valuable, but expressiveness also embodies other qualities: precision, readability, flexibility, potential for abstraction.
Compared to other languages, Lisps are tremendously expressive. Like the overpowered Japanese motorcycle I once owned, they go where you want, very quickly, with a minimum of input. If you’ve ridden a motorcycle, then you know what I mean. If you haven’t, good news—Lisps are cheaper and safer.
Here’s my ranking of the language features that have offered the most immediate value to me, a programmer new to the Lisp world. For each, I’ve noted whether it’s a feature of Racket specifically, or Lisps generally.
Everything is an expression. [Lisps] Most programming languages are a combination of two syntactically distinct ingredients: expressions (things that are evaluated to produce a value) and statements (things that express an action). For instance, in Python,
x = 1
is a statement, and(x + 1)
is an expression.Statements and expressions are distinct because while expressions can be naturally nested with each other, statements and expressions cannot. For instance, in Python, this is still a valid expression:
(x + (y + 1))
but this is not:
(x + (if is_true(): 1 else: 2))
because the if–else conditional is a statement, and can only be used in certain positions.
By making everything an expression, however, Lisps remove this limitation. Since expressions are nestable, anything in the language can be combined with nearly anything else. For instance, because an if–else conditional is an expression, you can use it in place of a value:
(x . + . (if (is_true) 1 2)
You could also use a conditional in place of an operator:
(x . (if (wants_sum) + *) . 1)
You could even nest another conditional within that:
(x . (if ((if (cond) cond_1 cond_2)) ...)
And so forth. This is a synthetic example. The point is not that you’d necessarily want to do this, but that Lisps permit it. As a programmer, this both simplifies your work (because everything snaps together easily) and expands your possibilities (because you can combine parts of the language in unusual ways if you feel like it).
It’s similar to the basic idea behind Legos. Other building sets offer specialized pieces that can only fit together certain ways. But by sharing uniform measurements, Lego bricks offer maximum possibilities for combinations. This ends up being more flexible & more fun.
So it is with an expression-based language. If you find this idea exciting, congratulations—you might be a Lisp programmer. (If you find this idea weird and scary, this is a good moment to bail out.)
Every expression is either a single value or a list. [Lisps] Single values are things like numbers and strings and hash tables. (In Lisps, they’re sometimes called atoms.) That part is no big deal.
The list part, however, is a big deal. In a language like Python, the list is one data type within the language. But in Lisps, the list is more like an organizing principle for everything that happens. So yes, you can use the list as a data type. But a function call is also a list. In fact, the source code for the function is a list. Actually, the rest of the program is too. Lists are everywhere. (The fancy CS term for this property is homoiconicity.)
The benefits of lists are similar to that of expressions. By bringing more of the language into a consistent form, more possibilities arise for how pieces can be combined and manipulated.
Seibel describes Lisp as a tool for getting
“more done, faster”. Here, you can start to see why this is so. Lisp languages are immensely flexible and permissive in how their pieces can be connected. This means that the way you think about a programming problem can be quite close to the way you actually program it. (This is also why Lisps have traditionally excelled for prototypes and exploratory work.)To be fair, getting the most out of a Lisp means learning to think more in the Lisp idiom of lists and expressions. So I can see why Seibel says that trying it yourself is the best way to be convinced of the benefits. As you get a feel for lists and expressions, it does pay increasing dividends throughout the language. You see how tiny lines of code can produce epic amounts of work. You also start to appreciate that even in a well-designed language like Python, you’re spending a lot of time shaping your ideas to fit its limitations, like shaving an invisible yak.
Functional programming. [Lisps] Yes, I know that other languages offer functional-programming features, and that Lisps aren’t considered pure functional languages. But many programmers haven’t been exposed to this idiom, and thus tend to underrate its benefits. I know I was in that category.
Functional programming doesn’t mean programming with functions. Everybody does that. Functional programming refers to a stricter style where functions receive certain data as input, process only that data, and return a result. In functional programming, functions avoid two habits common in other languages: data mutation (= changing data in-place rather than returning a value) and relying on state (= extra context that’s not provided as input, for instance global variables).
“Wait—I love state and data mutation. Why would you take them away?” Because they’re false friends. They contradict the essential concept of a function, which is to encapsulate data and algorithms. When a function relies on state or mutation, it’s operating outside those boundaries. You either take on an increasing housekeeping burden to keep track of how functions affect each other, or watch the program sink into a swamp of mysterious, complicated bugs.Programming in a functional style takes more effort at the outset. But it encourages you to structure the program in a clean, compartmentalized way. This pays off immediately in programs that are easier to test and debug. It’s also more likely to lead to reusable components, since functions are truly independent.
This bite-the-bullet aspect of functional programming is another reason why you can get
“more done, faster” with a Lisp. The difference between prototype and production code often ends up being small, because you don’t take as many shortcuts at the start. The program grows and evolves more smoothly because it’s easy to change one part without causing ripple effects elsewhere.Libraries & documentation. [Racket] This might not look like a competitive differentiator—doesn’t every programming language have libraries & documentation?
Yes, but probably not like this. As a consequence of being used in research settings for many years—Racket’s core development team is mostly CS professors—Racket’s libraries & docs are more like a transmission from a highly evolved alien intelligence.
You get the essentials, of course: web server, JSON, XML, drawing, foreign-function interface, and so on. Then you notice packages you maybe didn’t expect: GUI application framework, math plotting, package-distribution system, unit tester. Beyond that, your face starts to melt a little bit: semantics engineering? Futures visualizer?
I won’t pretend to know what all this shit does. A lot of it is over my head. But I like that. Each week I use Racket, I end up exploring a new part of the library, and learning something new. As opposed to other languages that seem to kill brain cells on contact (= pretty much anything named *Script, I find).
This learning is only possible because of Racket’s truly outstanding documentation. It’s vast, thorough, precise, and approachable. See for yourself.
DrRacket. [Racket] Yes, I know how to use a command line. But Racket includes a cross-platform graphical IDE called DrRacket that’s pretty great. DrRacket lets you edit, run, and debug Racket source files (or any other language based on Racket—see item #9 on this list.)
No, it doesn’t have the Ginsu-level search-and-replace facilities of something like Sublime Text. But it does have helpful editing features optimized for Racket code (for instance, you can right-click on a symbol name and rename it throughout the file, or jump from a function to its documentation).
Moreover, the command line within DrRacket doesn’t just show plain text—it can show stacked fractions, drawings, math plots, and other unexpected guests. If your command line does all that, by all means keep using it.
X-expressions. [Lisps] This choice is somewhat biased by my work with Racket, which mostly involves document processing and typesetting. But related topics arise in most web programming. An X-expression is a special native data structure that Lisps use to represent HTML and other XML-ish data.
Well, not
“special” in a Lispy sense—keeping with the usual policy, an X-expression is just another list—but special in the sense that other programming languages don’t have it. Usually your choice is to represent HTML either as a string or as a full XML tree. A string is wrong because it doesn’t capture the structure of the HTML, as defined by its tags and attributes. An XML tree shows this structure, but conceals the sequential nature of the data elements, and is unwieldy to work with.An X-expression ends up being an ideal hybrid between a string and a tree. Moreover, because it’s just another list-based expression in the language, you have a lot of options for processing it. Translating an X-expression to or from a text representation using angle brackets is trivial and fast. (Details.)
Given the close kinship between XML-ish data structures and Lisp languages, I have no explanation why, during the internet era, they’ve not been paired more often. They’re like peanut butter and jelly.
Scribble. [Racket] Pollen wouldn’t have been possible without Scribble, so for me, this has been the stone-cold killer feature of Racket. But that won’t be true for everyone, so I’m moving it down on the list.
Scribble is a dialect of Racket that inverts the ordinary relationship of plain text and code: rather than embedding text strings within source, a Scribble document consists of code expressions embedded within plain text.
“So it’s like an HTML template language.” Yes, in the sense that a template language allows code to be embedded in text. But also no, because a template language is usually a pidgin version of a real programming language. Scribble, by contrast, lets you invoke any Racket code simply by adding a command character to the front. In keeping with the theme already established, this approach is both simpler (because there’s almost nothing new to learn) and more powerful (because you can invoke anything in Racket).In its combination of text and code, Scribble has more kinship with LaTeX. While it doesn’t have the typesetting facilities of LaTeX, the programming facilities are much better.
Macros. [Racket] Also known in Racket as syntax transformations. Serious Racketeers sometimes prefer that term because a Racket macro can be far more sophisticated than the usual Common Lisp macro.
A macro in Common Lisp is a function that runs at compile-time, accepting symbols as input and injecting them into a template to produce new code. Syntax transformations in Racket, on the other hand, rely on the concept of hygiene and can offer Common Lisp-style macros, but also more elaborate syntax rearrangements.
But forget that—what’s in it for you? As a programmer, you end up getting two bites at the apple every time you run a file: Racket runs the syntax transformations (which alter the source code), and then the source code itself.
Unlike something like the C preprocessor, which is basically a separate mini-language, Racket syntax transformations are themselves Racket functions that give you access to everything in Racket. Like lists and expressions, syntax transformations add another layer of expressive possibilities.
Create new programming languages. [Racket] When I first read that Racket could be used to create new languages, I had two thoughts—are they serious? and would I really want to do that? The answers were yes and oh hell yes.
Between expressions, lists, and syntax transformations, Racket gives you a huge amount of semantic flexibility. But on top of that, it also adds syntactic flexiblity, in that you can define a reader that converts surface syntax into standard Racket.
You can use this facility to make specialized dialects of Racket. Or implement earlier languages. Or create entirely new languages with their own rules. You can use any of these languages within DrRacket to code new projects. (These special languages are sometimes called domain-specific languages, or DSLs.) Scribble is a DSL based on Racket; Pollen is a set of DSLs based on Scribble.
If you’re like most programmers, you’ve never had a tool for making a new language, so you’ve not considered it a realistic approach to a problem. And maybe you’ll never use it. But when you do, it is awesome, in both the new and old senses of that word.
Opportunities to participate. [Racket] In theory, open-source software projects create the opportunity for groups of developers to join together and make better things in collaboration than they could separately.
In practice, I’ve found that open-source projects sort into a bimodal distribution: over here, the underdocumented solo projects that sputter along fitfully (if at all); over there, the mature, popular projects that can be intimidating for new contributors.
As an open-source project, Racket is positioned at a happy medium. The core development team has been working together for years, and the commits remain fast & furious. But they’re friendly scientists, not Shire-dwelling egotists, and remain receptive to improvements across the whole system. If you have a better idea, they’ll listen; if you code it up to their standards and make a pull request, they’ll take it.
[2021 update: I no longer contribute to Racket due to abuse & bullying by the project leadership. Everyone in the broader Racket community, however, has always been helpful and kind.]
The point of this list has been to tell you about the positives. That doesn’t mean there aren’t negatives. The small pool of Racket programmers means that when you hit a pothole, it’s possible no one’s ever seen your problem (= the inverse of Linus’s Law). If I wanted to hire a Racket programmer, the options would be few.
Still, why shouldn’t I be enthusiastic? What I’ve been able to accomplish so far with Racket has been tremendously useful, educational, and fun—the most fun I’ve had in 25+ years of programming.
If you think I sound like a fanboy or cult member, I can live with that. But those are people whose enthusiasm is disproportionate to reality. Here, I’ve tried to stay out of the clouds (and the weeds) and explain the concrete, practical features that have made Racket such a pleasure in my own work.
As always, your mileage may vary. But if I persuade a few people to download Racket and try it, I’ll be happy. In fact, if you try it and don’t like it, I invite you to contact me, because I’m always curious to hear dissenting opinions.
I will end by taking on the big kahuna—
I won’t claim I’ve reached the top of the mountain. But I can tell you what the view looks like so far.
There’s a sense in which Lisp and its descendants are more than programming languages. They’re tools in the broader intellectual inquiry into the theory of computation. Lisp’s inventor, John McCarthy, originally considered Lisp a
The theory of computation is just one of many great scientific discoveries in the last 100 years. But I don’t get to use quantum mechanics or relativity or DNA sequencing in my daily work. When I’m programming, however, I’m using computation.
Racket, as a Lisp dialect, has many practical benefits. But it also opens a window onto this vast theoretical world that underlies everything we can do with programs. I’m not a brainiac computer scientist. But some days, through that window, I can start to see a bit of what they see—some math, some science, a lot of truth, and more than a little beauty and mystery.
Paul Graham calls Lisp a
If that’s not a step toward enlightenment, I don’t know what is.
The aforementioned Structure and Interpretation of Computer Programs deserves every compliment it gets. It’s mind-bendingly great, and accessible to anyone with a mild curiosity about software. By using Scheme to illustrate meta-topics about programs, it impliedly makes a great case for Lisps.
Likewise, Peter Norvig’s Paradigms of Artificial Intelligence Programming is not intended as a general introduction to Lisp, but it ends up doing that job well, by showing how sophisticated programs can arise from small constructions of Lisp code.
As I’ve already mentioned, Racket’s documentation is consistently excellent. Simply reading through The Racket Guide and trying some examples is rewarding.
I explain more about why Racket was essential for Pollen in the Pollen documentation.
If you want to try Racket, you can download the current stable version (easy), a nightly build (easy), or compile it from source (slightly daunting the first time but then easy).
20 Aug 2014
At RacketCon in September 2014, I gave a short talk about Pollen, typography, and digital bookmaking. Here’s the video.