Economy Size Geek

This month's column diverges from the normal pattern of covering my struggles with technology. When I started this column, I was really looking forward to the Software Development issue, because I was hoping to spotlight a language I am learning—Clojure (pronounced like “closure”). But, I found myself still in the process of bending my brain around it as my deadline loomed, so instead of trying to teach you while I am still learning, I decided it would be more interesting to hear from the man who created the language instead, Rich Hickey.

Rich Hickey, Creator of Clojure

DE: What did you do before you started the Clojure project?

RH: I'm a consultant, so I work on various things. I think the big thing I've done recently is I worked on the national exit poll.

DE: What other languages did you use before inventing your own?

RH: I was a C++ developer for a long time. I taught it at NYU for a little while. I worked in scheduling systems and broadcast automation all in C++. Then I moved from that to Java and C#. At the same time, I also started doing some Common Lisp.

DE: Most people who create their own language start from scratch—what was it that drew you to do a Lisp?

RH: I discovered Lisp after ten years of C++ and said to myself, “What have I been doing with my life?” I realized it was just much more productive for me. I fell in love with it right away. Basically, I said to myself, “At some point in my life, this is really what I want to be doing.” I spent several years subsequent to that continuing to do my commercial work in Java and C#, but all the while hoping at some point to be able to get Lisp into the mix. That's what drove me to doing Clojure, because I saw that either .NET or the JVM were requirements for all the customers I was encountering. Lisp really couldn't reach those platforms, so I decided to make a Lisp that could.

DE: Why put it on the JVM?

RH: It's designed to be useful for the work I was doing, where you have customers that have requirements that things run on one of these standard platforms—platform not meaning the OS but these infrastructures, like the JVM or .NET.

DE: And putting it on the JVM, did that get you out of doing a lot of the low-level stuff?

RH: Absolutely! I mean as a separate concern from the practicality of being able to access all the libraries, being on the JVM gives you a really nice separation of concerns between the high-level language, which is what I got to focus on, and the runtime, which is something that the JVM does an excellent job with. It's got a great garbage collector and a very sophisticated runtime compilation infrastructure. So as a language developer, you don't have to worry about emitting machine code or anything that low level. But, I'm not on the JVM just to get a free ride in terms of making it easier to implement. It was very much a part of the design of the language that you be able to touch and reach the underlying host platform, because there's a lot of value there.

DE: Libraries are part of that value, right?

RH: The amount of libraries available are really fantastic. That meant as a new language, Clojure had a ton of libraries right out of the gate. People didn't have to wait for a Clojure library for sockets or for talking to the database or for doing a GUI. They had libraries for those things right away. So they were productive with Clojure right away.

DE: You seem very pragmatic. You talked about productivity and libraries and how you get to use this on a daily basis. That seems like a different view of what a language is and how it fits.

RH: It's a little bit different. There's no taking away from Common Lisp being very powerful. I just think that it was standardized back before the Internet was commonplace. Then it was hard to move because there was a change in the marketplace in the 1990s that shut down artificial intelligence and a bunch of other things that were where Lisp lived. It wasn't really about the language, but it did cause some stagnation. So coming out fresh, not being compatible, really gave me a blank slate. Being a commercial developer, I knew what I needed to have be present in order for it to be practical.

DE: Clojure is not object-oriented, why not?

RH: Well it's not object-oriented the way Java, C# or C++ is. That's not really the way you would structure things. It is in some ways a rejection of my heritage, as an object-oriented programmer. I think after having done it for two decades, I don't believe in it anymore. I just don't think it's the right way to start. It can help you organize your code, but it brings along with it some complexity that I have found in real systems always ends up biting you—and it is related to mutability. By default, an object-oriented program is a graph of mutable objects. That's a very, very difficult thing to think about, debug and keep running. I've worked on very big object-oriented systems, and you always essentially run into the problems related to that architecture. I think that even before you get to concurrency, there are complexity problems with mutable objects that basically affect every large object-oriented application. When you add in concurrency, those problems become much clearer.

So a functional approach was something that I had already started doing, even in programs I was writing in C#. For instance, there were parts of the national exit poll system that were very functional, even though it's a C# system, because the way to defend yourself against this complexity is to write in a more functional style with a lot more immutability. The problem is that it's not very idiomatic in C# or Java to do so. I wanted to make a language where it was—where the default was to do the right thing. When you needed mutability, there would be a good story about how to do that compatibly with concurrency.

DE: So is that why you have the transaction system built in?

RH: Once you say by default everything should be immutable, real systems can't work that way everywhere. There have to be places where people can see the effect as if something were changing. So how do you mimic that? The Clojure system says, “Well, we're gonna take something immutable and put it into a cell or a reference, and we'll swap out what's in that reference from one immutable thing to another.” And, that will look like change to the program. It's a little bit different from a variable in that you're promising yourself the only thing you'll ever put in there is an immutable thing. But, that gives you a great separation of concerns. Because then you have these immutable values that most of your program is manipulating. You can say about those references, “This is what happens in the concurrent program.” And, one of the things could be transactions, and that's the STM (Software Transactional Memory).

So it's a nice story. You can switch back and forth between the different constructs depending on how much sharing there will be in your application. But, yes, that's why it's built in. Because without it, you just leave people wondering, “Okay, well, when I need to change, what do I do? Do I go back to locks?”, and that kind of thing.

DE: Clojure introduces a lot of different ideas all at once. How do you get started?

RH: Well, you start small. I think the way in for most people is to see the way Clojure uses associative data structures (maps in Clojure). They're called dictionaries or hashes in other languages. So people that have used Python and Ruby have gotten a good feel for working with objects as if they were just generic dictionaries.

You will be facing challenges because some of the things that you're used to aren't there. On the other hand, you also are handed a lot of things that are much easier and clearer and fit together better. So you get a little bit of a bump to try to change your habits, but you're getting immediate payback in a program that's easier to understand, easier to test and much closer to the way you're thinking about the problem.

DE: Is that the power you see—that it makes it easier for you to code the problem you're trying to tackle clearly?

RH: Absolutely. I mean, that's really the great thing about a Lisp or one of the highest-level functional languages like a Haskell, the program pretty much consists of stuff that matters. Anything that doesn't matter, you can make go away. Whereas, if you look at a Java, C# or C++ program, it's full of things that don't matter—things that are there because of the way you have to do things or the syntax and the language.

DE: Well, I noticed that in May 2009 you guys announced Clojure as 1.0. It seems like open-source projects often linger below 1.0. How did you decide this is 1.0?

RH: I think one of the nice things about being a Lisp is that most of the additions you make to the language aren't really in the language, they're just libraries. It is very easy to add and not break anything. Clojure has grown really rapidly and yet in a non-disruptive way, because new things are just new things. If you don't use them, they don't impact you.

DE: How does your process for evolving the language work?

RH: It's not very formal. I'm trying to stay in touch with what people are doing and what's working and what needs refinement. Sometimes there are performance things I want to tackle. Other times there are new things in the libraries people need that I'll tackle. It's not like there's a big road map, because I don't really believe in getting too far ahead of yourself. Things are done incrementally.

DE: How does code get into Clojure? Is it just you, or is there a team?

RH: On the core stuff it's me. There's one other person who can apply patches that I've approved. Then contrib is much wider. In contrib, I think there are about a dozen people who can commit and there are 100 registered contributors.

DE: And how do you register to become a contributor?

RH: You sign a contributor's agreement. It's a lot like the one that Sun used, and then you're a contributor.

DE: If I want to make a difference in the language where do I get started?

RH: Well you should become familiar with what Clojure has got already. Usually, you'll get good at the core of Clojure, and then you'll start building apps and using parts of contrib. Then perhaps you could find something that contrib isn't covering and contribute a whole new library there. Or, you'll find something in contrib that could be enhanced. You can talk to the person who owns that part and say, “Hey, I've got ideas”, and work together. It's not difficult. There's not a lot of hurdles. The core language being much more focused I think is critical, because I don't think languages really are built by teams of people usually. You have to have a vision, and you really don't want to be going in more than one direction at a time. In particular for Clojure, it's essential to me that the core stays very small. Being simple at the very core of things is part of what makes it good.

DE: Are you surprised by the growth?

RH: It's out of control. When I released it, I had the realistic expectation that if 10–100 people used it, that would be amazing. Because that's all you can expect. But for some reason, it took off. It was not something I anticipated. While we're talking, we're at 2,999 members of the Google group. Who could know that was going to happen? It's been crazy. Very crazy.

DE: Is it exciting for you to kind of get some validation that you weren't the only person who wanted a Lisp you could use?

RH: Yeah, I think there's the Lisp aspect, and then there are plenty of people using Clojure that Lisp is a kind of hurdle for initially. They're not coming looking for the Lisp part. They want the dynamic development. They want the immutability. They want a good concurrency story. They want functional programming. The Lisp part is not the appeal.

DE: I noticed in one of your posts talking about Clojure in Clojure. What does that mean?

RH: Well, Clojure was written from scratch. So I started writing it in Java. A lot of the bootstrap, the underpinnings of Clojure, were written in Java. The basic data structures were all written in Java, and the first compiler was written in Java. Then, once you had a compiler and the data structures, you could write the rest of the language in Clojure. So what I want to do is go back to those parts that are written in Java and rewrite them in Clojure now that Clojure exists. Recently, I've been doing a lot of work so it has the features and performance needed to do even the lowest-level parts of implementing itself. We can go back and re-implement Clojure in Clojure.

DE: So is the goal to make it so that you have more kind of idiomatic flexibility? Or just for completeness?

RH: Well, the goal is to get rid of the Java code. It demonstrates that Clojure has sufficient performance and expressiveness to do everything the Java part did. The other part is it will make moving forward a lot easier. David Miller has been porting all the Java to C# to port it to .NET. So when there's a lot of Java, there's a lot of C#. When most of Clojure is written in Clojure, only a very little bit will be Java-specific. And, only that little bit would need to be ported to .NET. In addition, I think a really important target for Clojure will be JavaScript. So moving Clojure mostly into Clojure means reducing the footprint of a port to a very small amount, and I think that's a valuable goal.

DE: When you move to these other kinds of things, do you worry about the low-level stuff you have counted on the JVM for?

RH: The whole thing is not to try to make all these platforms the same. It's to say, “Well, I know how to accomplish things quickly in Clojure.” And, when my target platform is the browser, I can leverage my knowledge about Clojure to do that as well. You'll still want the same kind of ability to access the hosts that you have for Java when you're in JavaScript. You end up with a body of expertise. You will have libraries that will work in all those places that are not host-specific. So, more and easier portability is an objective of Clojure in Clojure. And, it'll just be more fun for people to hack on it.

DE: What is your typical Clojure user like?

RH: There isn't a typical user. It's very much split among people with very different backgrounds. You have people who are solid Lispers who are looking forward to being able to reach the rest of the world easily, who actually know very little Java or none. Then you have Java people who are looking to get that expressiveness and agility, but they want to know it will be solid enough to do everything they were doing in Java in terms of performance and threading. We get people from Ruby and Python who know they love dynamic languages already, but have found performance or other issues there. Or they're just trying to up their game and learn more about functional programming. And then we'll get people from the functional camp (Haskell or ML) who are looking for the practicality of being on a platform like the JVM. What's great is they can all help each other.

DE: They all have a different piece. Your platform provides something that they can each bring and they each need.

RH: That's a great way to put it. It's made for a fantastic community where somebody who's struggling to learn the functional programming side can turn right around and be the expert about CLASSPATH for somebody else because he knows Java. So that's been great. I just can't say enough about the community—it's fantastic! I think there's just a tremendous amount going on. A lot of libraries. The numbers and the growth are a big part of the appeal of Clojure. There are books. One book is out, and more books are on the way. So I think it's very approachable now in terms of getting a lot of help.

DE: What do you see that's exciting for you about the future?

RH: I think there are lots of challenges still to come, for people who are looking to try to take advantage of all these CPUs they are going to have. I'm excited about Clojure in Clojure and growing that. In general, I'm trying to work on the general problem of dealing with time in programs. I think that's part of what Clojure is about. I still have a lot of ideas around that that I haven't implemented yet.

DE: When you say time in programs, what do you mean?

RH: Well, when we talk about things changing, and concurrency and mutability and things like that, those are all about time. I don't think the languages that we've had up till now have been explicit enough about time. So we've run into a lot of problems, because we're not thinking about time explicitly. Clojure makes time explicit. I want to do more work on that, because I think that will be a really important thing as we move to these multicore platforms.

DE: So are you trying to create something that is missing from other languages?

RH: Well, I mean I'm borrowing as much as I can from wherever I can for sure. I definitely stand on the shoulders of giants. But in this area, I never claim any novelty for Clojure. Clojure is mostly about trying to take existing good ideas that may not have been put together and put them together. It's certainly missing from the more traditional languages like Java, C# and C++. But, there are other ways to address it that are present in Erlang and Haskell, for instance.

DE: Any final thoughts?

RH: I would encourage everybody to try it out. It's a welcoming community, and beginners are always welcome and treated nicely. And, we're happy to have more users.