Language-oriented Programming and Language Workbenches: Shifting Paradigms

Neal Ford: So, good morning everybody. I am Neal Ford. This is Martin Fowler. We are going to be fighting our slide show all morning this morning. So, that should be interesting. We are here to talk about language-oriented programming and here are some questions that we are going to answer. Why is there so much XML mixed in with my Java code? Why would not everybody shut up already about Ruby on Rails? It seems like any time you talk to someone about Ruby on Rails they are irrationally exuberant about it and there has got to be some reason for that and why do things like aspects exist? Why do we actually need something like aspects if Java is enough and is there an evolutionary step beyond object-oriented programming? Is the abstraction layer we have been using so far, is that sufficient for the problems that we are trying to solve today and what the heck is language-oriented programming anyway? That is what Martin and I am here to talk about is this idea of language-oriented programming. For the past 20 years or so, we have been trying to model the world with trees. That is the way we model the world in object-oriented languages is build hierarchies, trees, trees of how we model things in code and it turns out that that works out pretty well because most of the world is tree shaped. Most of the world is hierarchical. You can fit things into trees pretty easily but that abstraction breaks down sometimes because a lot of times, we try to model the real world. We model it like this with a nice tree abstract picture of trees, and of course this is idealized. This is not the way the world really looks. The world really looks like this, tangled branches and interconnections and all sorts of other things that are really, really hard to model in these kind of idealized pictures. So what we have been trying to do is model the world with hierarchies but hierarchies fall down at some point and so we invent things like aspects. The red line that you see here represents aspects, which cut through the tree-shaped hierarchies that we built to try to model things in the world. But that just adds complexity to the problem we are trying to solve, and one of the things that we are trying to kind of kill off when we do abstractions is to try kill off complexity. Well, how have we done this in the past? If you look, for example, at assembly language, how many people here still write assembly language for their day job? Yeah, that is what I thought. Nobody writes in that any more because it is too low a level of abstraction. So, what we have always done in the past is take our abstractions and raise them a few levels. We do not write in assembly language any more because it just takes too long to get anything done. It is way too low a level of abstraction and so we build abstractions on top of that. In fact, if you think about your hard drive, it is really just a spinning platter with 1s and 0s on it. We never think of it that way either. We have all these nice metaphors and abstractions on top of it. So, what we are suggesting is that maybe it is time to upgrade our abstraction layer one more step toward language rather than just hierarchies to be able to represent stuff. Martin Fowler: So, one, one…sorry. Neal Ford: Go ahead. Go ahead. Martin Fowler: One way of thinking about this is that the object-oriented stuff and the kinds of abstractions that we build with our object-oriented thinking are abstractions of really allowing us to build up a vocabulary. We are able to create our own words that talk about the problem space that we are working with. When we think about our languages in a way we talk to each other, it is not just about vocabulary, it is also about how we put the words together, the grammar of how we speak, and so one of the things we are beginning to think about is okay, we know how to now build up these vocabularies, how can we better do the combination of these things? How can we start thinking about the grammar side as well as the vocabulary side? Neal Ford: What we are talking about here is changing abstraction mechanisms modeling the world with the language instead of hierarchies, because in some cases language actually makes a better modeling mechanism than hierarchies do. Well, what we are talking about here is using abstractions from the past, objects and generics and aspects and all those things to build better abstractions. We are not talking about throwing away object-oriented programming. Clearly, object-oriented programming has done a lot of good stuff for us, but for a lot of the problems we are trying to solve it still resides at too low a level of abstraction and that is what we are trying to do is raise our abstraction level one more step higher, which is what we have always done to solve problems in the computer science world. So, why language? Why choose this as our new favorite form of abstraction mechanism? Well, it turns out that the human brain is really, really good at supplying context and context is a really important concept when you are talking about using languages in abstraction mechanism. Here is a classic example of a DSL: an Iced Decaf Triple Vanilla Skim with whip latte. Of course, this is the Starbucks DSL. This is how you order coffee at Starbucks. In fact, when new employees come to work at Starbucks, the first thing they have to do is learn the Starbucks DSL and if you say it to them incorrectly they repeat it back to you in the correct format. There is a very strict structure to the way people at Starbucks talk and there are all these rules, there are over a million combinations of possible ways you can order coffee at Starbucks and have a very exacting way of talking about that and I will let Martin talk about this one because I have no idea what this means. Martin Fowler: Exactly, but the point is not so much the content of the example as the notion that it is again a combination of words for a specific contextual area arranged in a particular way but how that carries that meaning and you have a sense also with this but there is a sense of flow with this, but as you use and combine things together, you are able to express yourself much more clearly and you need both the vocabulary and the way of combining things to make that work. Neal Ford: This is really just a shorthand mechanism for human communication. When you think about this, this cricket example, if you were talking with your friends about cricket, think how cumbersome it would be to start at first principles all the time. There is a religion called sport, which is where you gather groups of people together as a team, and they play each other on something called a pitch and there is a ball with a bat. I mean it would take so long to have any kind of conversation that it would be useless to have the conversation and yet, that is exactly what we do when we talk to APIs and frameworks. We start at the lowest level possible of understanding and have to explain every single detail in code to our framework because it does not have any sort of context built into it, including your own business. If you are a Java ace and you go to work for a new business, day one, the hardest job that you face is learning the DSL for the business that you are going to work for. Every business has their own domain-specific language and it is very tightly keyed to the kind of problems they solve in their business and the kind of work that you do. Martin Fowler: And this is where the tying between this kind of thinking comes with the classic object-oriented ideas of domain modeling or indeed a lot of what database people do with data modeling. In all of these cases, you are trying to understand some particular business domain and build up that vocabulary and talk about how the various ideas fit together. But what these techniques have not typically done is talked about how do we express these combinations. Again, it is the vocabulary is what we focussed on and what we have yet to focus on is the grammar side. Neal Ford: So, all complex human endeavors have their own DSL. It is all about this implicit context that the brain is really, really good at supplying when you have a conversation. If someone not in your line of work has a discussion with you, they do not have to start over from first principles every time because you share that same context and that is what we are trying to convey to our APIs and frameworks in the computer world because people are really, really good at recognizing implicit content. Our brains are wired up specifically to be able to do that. So what we are talking about here, you may say, well, that is just another kind of API. What is the distinguishing factor between a DSL and a API? I am going to show you a couple of examples of this. Here is a good example of an API. This is what it would look like if you have to go order coffee at Starbucks using a Java framework for ordering coffee, and the interesting thing about this example you will notice is Coffee latte=new Coffee(Size,VENTI), latte.setFatContent, latte notice how much repetition there is there. We repeat the object over and over and over again because it is like Java is a completely context-free language. You have to tell it over and over again what the intent is for the code that we’re executing here. It is almost like Java severely retarded and we have to repeat ourselves over and over again to say this is what I want you to do. Remember this is what we are talking about still. It is still the same context… versus the DSL which has an implicit context, notice that the word coffee never actually shows up here and yet most people when they start reading this, when they get two or three words into it understand there is an implicit context here that we are actually talking about coffee. DSLs always have an implicit context that shows up either not at all or shows up in a very, very light way and usually at the most one time, so that you do not have to supply that context over and over again. Martin Fowler: And what is happening here is further development in something that has been a constant part of the software development space for quite a while, which is what we have been doing is concentrating on how do we make the code we write, more readable and more expressive? I remember 10, 15 years ago, talking about people, arguing with some people about whether it is worth putting a lot of effort into good naming of objects and methods. How important is it where you have methods that really convey what you do? And increasingly over the years people have realized that it is important to have very clear method names, very clear class names. But by thinking about how you name things well, you can reduce the need for other forms of documentation, and that clarity of code becomes very important. Now, with the interest in DSLs people are beginning to say, “Okay, we have got that part of things sorted out”, but if we look at the example on the previous slide, it is still hard to read, because of all of this repetition, because of this lack of context. So, how can we take a step forward to again make things a lot more readable and clear? Because the real art of programming in the computer is not the communication with the computer, it is the communication with other human beings who’re going to have to read that code now and in the future. I have often liked to be quoted on saying that any damn fool can write a program that a computer can understand, but good programmers write code that humans understand and this is really part of this, “how do we communicate with humans” drive? Neal Ford: And in fact if you look at this previous example that we are talking about, this is one of the reasons you business people do not like to try to read source code, because it is obfuscated because there is so much repeated context. The meaning here gets lost in noise. All end users who do not understand Java see as noise here drowning out the actual content of what you are actually talking about whereas this is very boiled down, this is the way that people actually talk, not the way that people usually write code. So, let us talk about some nomenclature, let us create some definitions here, and this is actually Martin’s definition of a domain-specific language, a limited form of computer language designed for a specific class of problem. Martin Fowler: I would not call it my definition. This has been around in the software world for quite a while. One of the things that we will see, is that domain-specific languages, as a term has very fuzzy boundaries. There are things that are very clearly a domain-specific language, and things that are very clearly not a domain-specific language, but there is a very large overlap area. It is like classifying something as blue or green where things are clearly blue, things are clearly green, but there are some colors where I can argue with my wife endlessly about whether it is blue or green, and that is definitely the case with domain-specific languages. But, one key property of domain-specific languages is that they have a narrow focus. You could not write a programming system entirely using a single domain-specific language, because the range is just too small. The idea is that you combine one or multiple domain-specific languages with other domain-specific languages and usually with general-purpose programming language as well, in order to actually to get stuff done. Neal Ford: And this is actually one of the places where Martin and I disagree very slightly because Martin I do not believe thinks that this Starbuck’s DSL is actually falls in the definition of domain-specific language. Martin Fowler: Yeah, I tend to use domain-specific language to specifically mean a software language. So, it is only something that we can actually execute on the computer, and when we talk about things that humans use, that we have not necessarily formalized to that degree, I tend to use a term like domain language or something a little bit broader, so as to make a distinction between the specific software construct and the more fuzzy real world construct. Neal Ford: I am much more liberal with my definition. I believe that any language that describes a problem domain is a domain-specific language and we just agree to disagree about that. Martin Fowler: Yeah. You are just wrong. Neal Ford: Yes. The other term and, I think this is actually your term. Martin Fowler: No, this is not my term either. Neal Ford: So, you’ve stolen this one as well. Martin Fowler: I try my best not to invent new terms, but you cannot help it from time to time. So, language-oriented programming, I first came across this term from an article by Sergey Dmitriev, he is one of the founders of JetBrains, who came up with the IntelliJ tool. As we will see shortly they are working on some very interesting stuff in this space and he got the term from some obscure academic paper. So, I do not know where it came from, but I like the term very much because it talks about this shift of moving from thinking about vocabulary, which is objects, to the notion of a language that combines vocabulary and grammar, and so I felt language-oriented programming was a good term. Also, I was very looking very much for a generic term that would stretch across a whole range of different styles that was not owned by a company. And so language-oriented programming seemed to make a good fit for that because it is not something that is tied to a particular product. Neal Ford: And this also kind of ties in the idea that what we are talking about here is an evolutionary step beyond just object-oriented programming, actually using objects as building blocks is the next abstraction layer, which is to use languages as an abstraction mechanism. There are a couple of different fundamental types of DSL, as you can pretty much separate DSLs into two broad categories, one of which is internal DSL built using an underlying syntax of base language. Martin Fowler: Right. So, it is easy to quickly define both of them first I think. So, I use the terms internal and external and these are terms I decided to use, where external is minilanguages in the UNIX tradition and internal are languages that are really expressions within a programming language, but done in a sort of language feel to the way. So, with an internal DSL you are completely operating within your host language. If you are programming in Java, your DSL is Java. If you are programming in Ruby your DSL is Ruby. One of the strongest traditions of this style of programming is in the Lisp world. You talk to Lisp people about how they construct Lisp programs, and they often think in terms of building up languages and this is one of the many reasons why this kind of stuff is seen as very, very old school, because Lisp people were doing this 30, 40 years ago. So, it is nothing new in that sense, but the new thing is perhaps putting more attention into it, particularly in places where people have not put much attention to it in the past. Neal Ford: In fact, I think this is so prevalent in Lisp, because Lisp as a language is so horrific to actually code in, the first thing you want to do is hide that under some extra layers of abstraction, so you can hopefully get away from all the parentheses and other sort of stuff. So, they almost had to invent this style of programming to get away from the core syntax, because it is so confusing, it is so daunting. Martin Fowler: All the Lisp people out there, he said that, not me. Neal Ford: An external DSL is the opposite kind of DSL, which is built using your own grammar and a lexer and a parser generated code of some kind. Martin Fowler: And this is the traditional UNIX style little language, where you have to say how something operates and you configure a little language to work things through. Very, very common in the UNIX world, and very often UNIX people will talk about how they will put together some little language in order to drive a particular programming or will configure a particular programming environment and I think of nothing to pulling out lex and yacc and twisting them together and producing the stuff that you have. Neal Ford: So, let us look at some examples of internal DSLs. Is anybody already doing this in the mainstream world? If you discount the very interesting stuff that has been there in the Lisp community around this area, is anybody really doing this for real today right now? Well, a good example of this is Ruby on Rails. If you look at Ruby on Rails code, it is very, very declarative. In fact, most of the code that you see here is not technically Ruby code at all. It is this DSL they have written in Rails. Ruby is actually a very popular target for building internal DSLs because the language has very, very loose syntax rules and you can get by with a lot of stuff that you cannot get up by with in a more strongly typed, statically-typed language that has much stricter rules about its syntax and the way that its code looks. Martin Fowler: The important thing here is that, I do not know how familiar you are with Ruby, this is actually old valid Ruby code, but it feels like a different kind of language. It is like you have invented whole new key words and ways of putting them together. So, as a result, you feel like you are in a different language to the actual Ruby language itself. And at one level, yes it is all Ruby. But it does not feel quite like that, and here again, we are talking about this very fuzzy thing about what is the difference with an internal DSL and what is the difference between that and an API. The fuzzy boundary for internal DSLs is between APIs and the language, but the essence of it is, I think, a sense of you just do not feel as if you are in regular Ruby. You feel like you have extended the language in this case, and to do something slightly different. And it is partly about the syntax as Neal said, but it is also partly about the features of the language, certain programming constructs. In particular, the ability to have closures are very, very useful for doing this kind of thing. And one of the things that makes Ruby very interesting and particularly in comparison with something like Lisp, is that Lisp gives you a limited set of mechanisms to work with but those mechanisms work really, really well. Ruby gives you a very wide range of mechanisms. Some of the work that I have been doing, experimenting for a book, I am working on this topic, I took a very simple DSL and ended up implementing it 20 odd different ways in Ruby, using different combinations of language constructs. Some of them worked well, some of them worked less well, but the really interesting thing is, how many different ways you can take a different DSL and work it out with Ruby, because of the fact that Ruby gives you so many options to work with. If you are working in Lisp, you have equivalent power, but less range of options and if you are working with Java, you also have less range of options, but also less power, because it does not give you some of these alternatives that you might need. Neal Ford: In fact, as Martin said one of the things that makes Ruby so effective, and Rails uses it very effectively is this idea of a closure, the last line of code you see here before destroy actually takes in a block of code but it is nicely contextual, so it is very easy to read right in the context of where it is being defined rather than having to create a new class, an anonymous inner class, and attach a lot of handlers and that sort of stuff. A lot of behavior can be defined just in line using closures the way that Ruby supports this idea of closures. And I think that the DSL portion of Rails is one of the reason that people are so irrationally exuberant about it. How many times had someone come up to you and just almost phoned with the mouth that how much they loved using a Java Web framework of some kind, and yet you see these Rails’ guys do this all the time, and part of the reason for that is that the tool they are using - Rails - is perfectly suited for the problem they are trying to solve. This is a domain-specific language for building web applications that offer persistence. It is very, very highly tuned to do that and so the tool fits into your hand really well as you write code in it, there is very little friction between what you want to accomplish and what the tool allows you to do. You do not have to do a lot work around and lots of other stuff, it just, you can express the intent of what you want to do very clearly and very succinctly using the DSL portion of Rails, and that movement should directly toward intent is really important because the more friction you can remove between your intent and the way that you realize that intent is really, really important. Another good example of where this is being used in the world right now are the expectations in Mock Object Libraries. Virtually every Mock Object Library that you see and I have got a couple of snippets here of JMock, EasyMock does the same thing and even in the .NET world RhinoMock does the same thing. So, does Mocha in the Ruby world. The reason that you see expectations written this way goes exactly to what we were talking about for DSLs. They are a limited problem domain, which we are trying to solve which is to set expectations from Mock Object. Think about how many lines of code this would take if this were written in a more traditional sort of Java API style. You would have at least five different lines of set code here, set this, set this, set this, set this and what that actually does is obscure the intent of what you are trying to do here, which is set an expectation to say that this things expects, this thing wants, with this method, etc., notice how much context has been drained away from setting this expectation in JMock and what you are left with is just the intent of what you are trying to accomplish. Martin Fowler: A few interesting things here with this. Again notice this, this is using Java. We are not using some weird language like Ruby or Lisp to do this kind of thing. So, again it brings out the point that this kind of internal DSL work can be done in a relatively straightforward language. It is also important to notice that the language kind of looks different, even in terms of formatting, in fact most obviously in terms of formatting. Because now we have got these cascades of methods with dots on each line which is formatted very ugly. I remember the first time I looked at code that had been written this way, it kinda looked weird, but you get used to it fairly quickly, and then you begin to appreciate how useful it can be. It is also worth mentioning at this point as well that I use this term here internal DSL, you will also hear some people and particularly the Mock people also use this, they use the term embedded DSL. In fact, an embedded DSL has a longer usage than internal DSL. I avoid it because it gets confused with embedded languages in the sense that say VBAs is an embedded language in Microsoft word, and so, because embedded has these two meanings, I decided to use internal DSL to focus on these things. It is also important to realize here that this is very much again the way of manipulating an object model. In the end, these are just objects that are being wired together in a particular way. There is no reason you could not use a regular API to do this. Indeed, what is going on under the covers in the JMock library is a regular API, and they originally built it with a traditional API. What they did with this expression syntax that they put over it is they have added a layer over that API that allows you to correct these expectations in a more friendly and readable format. Neal Ford: If you think about how this is implemented it is not like this is some sort of mysterious rocket science implementation, especially the way this is chained together because in Java for all of your traditional set methods instead of returning void, which is kind of a waste of a perfectly good return value, why not just return this? That allows you to chain together a series of method calls like this and achieve this kind of what we are calling a fluent interface which we will talk about in just a second. In fact, we will talk about it now, fluent interfaces where you treat lines of code as sentences because in English and in most spoken languages a sentence is a complete unit of thought and this idea of a fluent interface comes from readability and I am going to give Martin credit for this term too but he will probably deny this one as well, this fluent interface. Martin Fowler: I get half credit for this. Neal Ford: Okay. Martin Fowler: The origin of this came from, I was at a workshop with Eric Evans, you might have heard of Eric Evans. He wrote the excellent book ‘Domain-Driven Design’, and one of the things about myself and Eric is that we are both ex-Smalltalkers, and we did a lot of Smalltalk programming in the mid 90s, in fact that is where we first met, working together on a Smalltalk project. And one of the things that we talked about during the course of the workshop that Eric particularly lamented was the fact that when he worked with APIs in Java, they did not seem to have that same flow that a lot of better Smalltalk APIs seem to have perhaps and we would try and explain what we meant and in that discussion when we came up with the term fluent interface, and we liked the term because it brought up that notion of a language. If you read a regular API, it seems to have that kind of stuttery quality of somebody that does not really speak your language properly the way…probably more lack of the other languages.. being British, I cannot say anything, but you know what I mean in the sense somebody is speaking not a language that they are really comfortable with, and what we wanted to see was much more of this sense of fluency in a flow. And Smalltalk people unlike Lisp people never really talked about defining languages in Smalltalk. They talked about building domain models and putting them together, but they still had, a good number of them still had this very strong notion of trying to make that model really work in this flowing way. I remember sitting down and doing some programming once with Ward Cunningham and was really quite taken aback at the way he would rearrange what I was doing with an API just to make this read and things that fit together much better. I did not really understand what he was doing at the time, it’s something I only really appreciated it later on, and again, it is this push towards readability that does it. And so we felt that fluent interface was a good term for thinking of that. You are taking an API and then making it flow a bit more, and tactics, particular mechanisms like having set methods that return themselves, return the object you have just changed, cascading the method things, those are a common thing to do, and in fact that was default practice in Smalltalk, but that is a mechanism. The real aim at what we are trying to do is to have something that has this readability, this sense of flow, which is a very hard to define thing, it is not a precise thing, but it is what we are trying to achieve. Now, fluent interface and an internal DSL are really two ways of looking at the same thing depending on where you come from, whether you think of yourself as “I am trying to create a language here” or “I just have an API and I am just trying to make it more readable and more useful.” Neal Ford: And readability is critical. Everybody is well familiar with the statistic that lines of code are read two-and-a-half times as often as they are written. So, even if it takes a little bit longer to write a line of code to make it more readable there is a big payoff at the end because anyone who reads it can actually understand what you are talking about in a much clearer way. So, let us look at a concrete example of building a fluent interface, like I said this does not have to be any sort of rocket science or any kind of brand new technique or anything, this is the kind of code most of us deal with all day every day. This is a traditional sort of API kind of code in Java where you are creating a Car object and you are associating a MarketingDescription with it. So, you create a Car object and you create a MarketingDescription and you setType and setSubType and set all these attributes, and finally, you set the description to that Car object. You can very easily convert this into a fluent API that looks like this, and all this really is Car.describedAs that is instantiating any Car object and each one of the dots here are really just what used to be, the setters that you had in the API style of code, but what you are doing here is actually creating set methods that are aware of the context in which you are operating that allows you to create a sentence out of this Java code and what is nice about this is that your business analyst or whoever is consuming this code now actually has a fighting chance to be able to read this code because it is much, much closer to the way they talk about Cars and MarketingDescriptions rather than being in this kind of stilted, almost very formal old English style of writing code which we are accustomed to in the Java World with all this repeated context and all this extra noise in terms of sets and properties and that sort of stuff. Martin Fowler: This writer on graphic displays called Edward Tufte who is highly regarded as one of the best people to read when it comes to presenting visual information, and he says that when you are doing this in a chart or diagram, it is very important to remove the noise dots on the diagram. Everything on the diagram should convey some meaning. There should not be any extraneous stuff, and that is really what we are trying to do with the code, remove all the extraneous stuff. The second thing that I tend to do a lot when I am doing this which is not at all obvious from something like in the final picture that we have here, is that I typically create an additional object as a layer over the basic API and I refer to this as an expression builder object. The problem is that if you put all these methods on the Car class and you looked at the API of the Car class, it would look very odd because the methods do not make much sense sitting there in the javadoc on their own. The methods look meaningless because they are robbed of their context. So, what I would like to do is to have an expression builder object that I put over my regular API, which is purely designed to support the fluent interface. And then I have a way of getting that, the Car object back out when I am done and that is the kind of mechanism I use to implement it. In that way, the regular object can have a normal API with nice-looking javadoc and the fluent stuff is nicely contained in this builder. And I can play all sorts of tricks inside the builder in order to make the language flow because I will focus on language. The jMock libraries are an excellent example of that structure. Again, they have their regular API and then they have the fluent stuff that they place on top of it through just a couple of extra classes that they add to the structure. They do quite a sophisticated approach to this that uses quite a number of tricks to makes things flow well. One in particular that is very nice is that depending on where you are in your expression certain terms are legal or not and they do this by use of interface, multiple interfaces on the same object. And the nice consequence of this is that if you are in a good editor, the IntelliSense that you get leads you through building the expression, and so, it has that as a server usability thing when you are writing it. Neal Ford: Absolutely, if you are writing jMock code it is very well designed because every time you hit dot the things that you see there are the things that you are interested in doing next with that expectation that you are setting. And I want to re-emphasize the point here that this is not some sort of magical thing that we are creating here, all the building blocks are already there in the Java language and doing this, it is more an attitude shift of creating these fluent APIs more than anything else because there is no rocket science technology at all here, it is really just the intent of let us see how readable we can make code rather than how obfuscated we can make it which seems to be the default in the Java world for some reason. There is another good example of this style of coding that exists out in a world. There is an open source library you can download from Google called Hamcrest, which is really just fluent interface wrappers for JUnit matchers. So, this is an add-on to JUnit that allows you to say things like, I guess at Google when they do good work they give them biscuits because all their examples are in terms of biscuits. So, I guess that is what they are doing with their millions of dollars is buying biscuits, but you can say for example assertThat(theBiscuit is(equalTo(myBiscuit))) and that is much more readable line of code than the kind of a formal assert equals this, that syntax that you normally see. And all this is very light patina over the existing JUnit match or classes that are already there. In fact, if we look at the latest versions of jMock, they incorporate the Hamcrest library to make their expectations even more expressive. So there are lots of people out in the world that are kind of embracing this style of coding. There are some building blocks for internal DSL. As we emphasized you do not have to do anything special to create these kind of fluent interfaces, but there are some building blocks that you can use. Languages with looser rules tend to make better DSL bases, simply because the looser rules allow you to get closer to English and closer to this goal of fluency because Java has some pretty strict rules about where its punctuation goes, and every line has to have a semicolon and all those rules that make up Java. And so, you see a lot of people pursuing this style of coding in these more dynamic languages like Groovy or Ruby and of course JRuby, which allows you to write Ruby code that runs on the JVM, and here is a good example of this. You can add a support for time intervals in Groovy because let us face it, the java.util.Calendar class is pretty broken in Java. It is different from all the other classes. The sets work completely differently from every other class in Java. They cannot seem to get date stuff exactly right in Java because the java.util.Date and then they said no let us take a mulligan on that. Let us throw it away and redo it. So they duplicated that whole thing and said no, use Calendar instead. What you can use in Groovy is this thing called a Category, which allows you to essentially add new methods to built-in classes like Integer. So, I have got a category here IntegerWithTimeSupport that lets you add method calls to Integer and the goal here is to actually create this line of code that you see in red there 2.days.fromToday.at(4.pm) that returns a java.util.Calendar class instantiated at, as you probably guessed, two days from today with the time set at 4 pm. This is a perfect example of what we were talking about before is using the building blocks that we already have java.util.Calendar and the APIs that exist in Java and building a fluent interface on top of that and the nice thing that Groovy allows us to do here, Ruby does as well, is to do things like add methods to numbers so that you can create a time-interval support for core objects like Integers in Java. Martin Fowler: Yeah, there are a number of things that I think help make a language more DSL friendly. Looser rules is important. A lot of this is syntactic stuff like dropping things are necessary. It is amazing how much difference readability you get if you do not have to put parentheses, particularly empty parentheses pairs around things. Little things like that often make a lot of difference. Being able to add methods to existing classes within a context allows you to reorganize things and gives you more flexibility. Closures are really very, very important for more sophisticated cases. Another thing that is very important is the ability to have a literal collection structures particularly lists and HashMaps and be able to easily put those in as literals. In fact, that is the coolest structure of lists. It is the easability to write lists easily, and similarly you need that, I think in order to be able to do DSLs well. It is also useful to be able to easily construct symbolic types as opposed to using strings everywhere or enums because again it will allow a shorter, more compact way of representing things. Neal Ford: And this is an example of open classes which is what you add behavior to core types like Integers and the stuff that you normally cannot touch in Java, but it is very handy to be able to add behavior to them, for example, add this idea of time intervals to the Integer class so that you can actually say 2. something and have that instantiate the time interval for you based on that method on Integer. Martin Fowler: Lots of little things, but each one like the parentheses, does not seem that important when you mention it on its own and yet it makes a difference in getting something that reads clearly. Neal Ford: So, when you think about how much it would obfuscate this example if we had to put parentheses at the end of each one of these things that would just be noise that would help destroy the fluency of the line of code that we were trying to create here. So, dynamic languages tend to make better building blocks just because they have support for some of these building block elements that we are talking about. Of course, it is not required that you use these, but they make generally better base for internal DSLs. External DSLs are the next thing, the other category of DSLs that we have here which are written in a different language than the main host language of the application and transform using some form of compiler or interpreter into some executable code of some kind. And these can be plain text files like configuration files in the Apache world, in UNIX world, XML documents, I think that is one of the reasons why Java is overrun with XML right now is we really want some sort of external syntax and we have kind of settled upon XML because it is so easy to parse, and of course now we are overrun with XML. Every framework has at least one XML configuration document and you end up with a non-trivial application and you have four or five different dialects of XML document, each of which is essentially their own language because when you create a grammar for an XML file you are simply defining the grammar for an external language that is embodied in that XML document. Martin Fowler: This is a very important point because people will often say, well, do we actually use DSLs, and I would argue that most Java projects use a lot of DSLs and they are all embedded in these endless XML files that float around, and they are an external DSL, you have to parse them and bring them in, and usually, the framework will do that for you. We typically, we kind of gravitated to XML because there are very easy tools available for the syntactic analysis of the XML files, but the resulting XML is not very clear, and you do not really get great readability because there is a lot of noise words involved. And again you have that lack of flow. But I think the fact that the XML files are so pervasive in the Java world, in the .NET world is a testament of the fact that we need and we wanted to express things in domain-specific languages, but the mechanism we have used actually is not terribly good. Neal Ford: And so what we have done is sacrificed readability for parsability in the XML world. You would never show an XML document to an end user to try to explain anything to them because it would frighten them to death. All those pointy places on it looks like a porcupine, looks like if you touched it you can cut yourself on it because it looks hazardous to consume, and one of my favorite quotes about XML is actually a Dave Thomas quote, he said that XML is really just data dressed up like a hooker. So, building blocks for external DSLs, you have parser generators and we have a wealth of these in the Java world right now, we have Antlr, which the first time I saw it I thought it had something to do with ant which has nothing whatsoever to do with ant, JavaCC and Yacc and SableCC including some really sophisticated tools in the Antlr world now are called AntlrWorks which makes it very easy to work with these grammars that you create. Martin Fowler: Yeah, the interesting thing about these tools, however, is that there is not very much out there that is written about how to use them well. Like most people I did a little compiler class when I was at college for a few weeks and promptly forgot most of what I was taught there, and as a result I never got terribly comfortable with using these tools. Over the last few months as part of the research for the book I am working on, I had been very much burying my head in these kinds of things, looking at example, little languages in the UNIX world, in the Java world, and seeing how people use these kinds of tools. And one thing that is very, very obvious to me is (a) the tools are actually not that hard to use once you get the hang of them, but (b) there is very, very little stuff out there to help you get the hang of them. So, it is a real struggle to learn how to use these. Now, the situation is improving on some fronts, in particular with the Antlr toolset. There is a book just appeared in ‘The Pragmatic Programmer’ series, that talks about how to use Antlr. There is a very nice IDE called AntlrWorks that gives you things like syntax highlighting, refactoring of grammars, visual…visualizations of what gets recognized by the grammar files, specialized debugger and that will certainly help to make it a lot easier to use this kind of stuff. But there is still a big gap there between the actual tools and the knowledge to be able to exploit them effectively. I am hoping that some of the work I am doing will help fill that gap, but at the moment I do have to warn you on that. If you want to play around with these kinds of external tools, for the moment I would definitely recommend Antlr because of the book and because of the IDE. It helps you get into it and use it a lot more easily. Unfortunately, what is not in the book is really very much a device about how to choose and design a DSL very appropriately. It tells you a lot about how Antlr works, but not a huge amount on how to use it for DSL work. Neal Ford: Internal DSLs have some advantages because you have the full power of the underlying language and you have full access to sophisticated tools like IDEs. When you are defining a fluent interface in Java, you can use IntelliJ or Eclipse and you get code insight and all the other things that we are accustomed to in the Java world because we have all these sophisticated tools for working with Java code and of course an internal DSL if it is written in Java is fundamentally a Java program just expressed in a different way. Martin Fowler: And that is both the plus and the minus because in many ways what you want to do with the DSL is kind of restrict your level of expression because that way you make less mistakes. And so, sometimes having that full blown-ness can be confusing, but of course if you are very familiar with that then it becomes much less of an issue. Neal Ford: The disadvantage of internal DSLs is it is hard to write these in modern “curly brace” languages because you just cannot boil away a lot of the context. Java is very strict about parentheses and periods and semicolons and that sort of punctuation that is required by the Java language. You are limited by the syntax and the semantics of the language, so for example if you are doing this purely in Java, you cannot add methods to the Integer class to allow you to express time ranges and using Integers as the ultimate object that you are calling your method on. And you have to understand the base language or you are in syntax trouble. This is a classic blunder by people from the PHP world that look at Rails and go, oh Rails, this is a much more sophisticated form of PHP. I will start doing that and they read some recipes from the Rails book somewhere, but they do not understand Ruby syntax and they very quickly get in trouble because ultimately you are writing a Ruby code even though it does not look much like Ruby code, that is what you are writing because it is an internal DSL sitting on top of this base language of Ruby. External DSLs had the advantage that you are free to use any form you like, just let your imagination be your guide. That is actually an advantage and a disadvantage because you have infinite possibilities for what this thing can look like. You are limited really only by your ability to parse the language that you create into something that you can produce code from. Like Martin said, he has created 20 different versions of the same DSL experimenting with exactly how does this look and how does it work. Martin Fowler: And then the parse point is important because there are certain things that you can do to make parsing easier and things you can do to make it harder. And again, this is part of the stuff that is not often very clearly indicated out there. Neal Ford: The disadvantages of external DSLs is you have to build the translator and that is no small feat because you have to understand grammars and parsing and even though the tools make that easier, it is still a daunting task. You lack the support of your base language which means that you finally create this beautiful external DSL that you have created, but now you have got to use just a regular text editor to be able to edit it. You do not get any of the nice symbolic integration that you get with tools like Eclipse and IntelliJ because you are ultimately just writing out a plain text file that gets translated into something else. Of course, you could write your own IntelliJ for your own language, but now you are talking about a really daunting task of building not only a language, but also the tools that understands the language. Martin Fowler: This was of course less of an issue in UNIX days because hey! You just make your own emacs major mode and that will be straightforward, but the bar has gone up now. With more sophisticated IDEs you expect a lot more in the programming experience. So, a plain text file becomes much more awkward to use. Neal Ford: That is what you want to tell your users is, oh well, here is emacs, just edit it in emacs, that is perfectly fine, which brings us around to Language Workbenches. Well, Martin and I are not the only people in the world who are interested in this whole language-oriented programming idea. In fact, there is three, at least, major vendors, actually more than this, but these are three representatives ones who are actively pursuing this as a style of coding. There is Intentional Software developed by Charles Simonyi. A lot of you probably recognize his name. He was the guy at Microsoft who created Word and Excel and at some point, I think he was trying to get Microsoft to kind of pursue his vision for this Language Workbench kind of tool and they were not that interested in pursuing it. So, he did what any self-respecting Microsoft employee would do. He took his billions of dollars and he stomped out the door and created his own company called Intentional Software, and they have been working for years now on this very sophisticated tool that allows you to do external DSLs. Microsoft has since started embracing this idea and they are creating what they are calling Software Factories, which is sort of this Language Workbench idea but it is a little more tilted toward the modeling world in creating executable models and those sort of things and finally, Meta Programming System which is developed by JetBrains, the makers of IntelliJ, next the one we are going to talk a little bit about here is MPS because it is the only one that you can really actually touch right now. Martin Fowler: Yeah, just before we dive into it though, I will mention briefly, partly to do with the Microsoft stuff. We focused on textual languages so far in this talk with all sorts of reasons, I am not going to go delving into but there is also a school that is very interested in graphical languages and the Microsoft, the main specific language stuff is very much focused on that. A lot of people in that world come out of the CASE tool model-driven development kind of background and there is a lot of questionable stuff in that area but there is also some very good stuff as well and there is some definite interlinks with some of the things that we are talking about and I tend to see that as within the broad category of the main specific languages but it is just outside the bounds of what we have got time to talk about today. Neal Ford: So let us do a little bit of background and talk about Language Workbenches for a second. This is pretty much what compilation has looked like since the time you wrote your very first computer program where you have some sort of external representation, a text file, you hand it to a compiler, which parses it and compiles it into a syntax tree and then use something some sort of linker or some other mechanism and create an executable representation of that code and that is the way compilers have worked for a very, very long time now since the 1950s at least. Martin Fowler: You think about this as a sequence of transformations. You begin with source code, which is one representation. We transform this into a different representation that sits inside the compiler’s memory which is the parse tree or the abstract syntax tree and then we use a code generator to transform it into a third representation which is the executable representation that we actually run. In Java’s case, that executable representation is Java bytecode which is itself then goes through another transformation step to actually get into a machine code that runs for a particular system. So in many ways, what we are thinking here is a kind of a pipeline of multiple representations and transformation steps between them, and when I say that you parse a source file into an abstract syntax tree, if you look inside that process, again you find a sequence of transformations inside there. Neal Ford: But all of this have changed a little bit when IntelliJ first came out and this is a phrase that Martin coined and I believe this is yours, wholly yours, post-IntelliJ IDEs because IntelliJ was really the first tool that let you edit directly against the abstract syntax tree instead of just the text. That is actually how refactoring works. They do not do a massive search or replace operation in your source file. What they do is modify the abstract syntax tree and then reflect that back in the actual source code for your application and you will notice that we are getting better and better at this because the early refactoring tools were very intolerant of compilation errors. You had to be able to compile the whole thing. Now, they’ve gotten much better and you could actually edit parts of an abstract syntax tree even if another part of it does not actually work because there are syntax errors and other source of ambiguities in it. That leads us around to this idea of a Language Workbench, which rotates around this relationship with your custom tool from the traditional world because now you are operating mostly in some sort of projection from the abstract syntax tree and that could be a plain text file but it could just as easily be some sort of graphical designer tool or it could be database schema designer or something like that. You still have some sort of versioning storage that let you version this abstract syntax tree and you have some way to transform it into an executable representation of some kind, but the transformation process is actually different here. The general focus of your work in traditional compilation is the beginning of that sequence, which flows through to the end. This is more concerned with the middle part, which is really the important part, which is the abstract syntax tree there in the middle. Martin Fowler: In traditional compilation, the key artifact is the source code which has to contain everything about the program and the abstract syntax tree is something which is really very transient because it occurs during compilation although it also occurs during editing in a post-IntelliJ IDE. With the Language Workbench, the central thing is the abstract representation and when you edit it, you edit through this projection what it looks to you and you do not have to have everything in the projection. You can just have different projections with different subsets of information and then the abstract representation ties everything together. So, it twists around the traditional way of looking at the relationship between source code and the other representations. Neal Ford: The editable representation is a projection of the abstract representation and the abstract representation has to be comfortable with errors and ambiguities just the way refactoring tools are now. Think how annoying a tool would be that said you have to be able to compile it before we can actually save your work. We would never want to use tool like this. So, one of the challenges of building a tool like this is to build it in such a way that it is actually comfortable and lets you save that internal representation even if it is an error or if it is incomplete somehow which is a nifty trick. So the examples we are going to show here just very briefly are screenshots and these are actually fairly old screenshots of MPS. The idea behind MPS is that instead of building objects and APIs and hierarchies like you normally do, you build concepts and a concept is something like a date, a very, very small thing but along with that concept, you also build an editor that is concept aware. So, this is an example that Martin actually created in conjunction with the guys at JetBrains to create a DSL that describes rate plans for electricity usage, I believe it is a… Martin Fowler: Hmm…mm. Neal Ford: …gas utility usage and what you do is build up a date concept and a numeric financial concept and you build all these concepts as individual entities and then wire them together to make bigger and bigger things. So, this is an example and notice you get syntax highlighting because now the language you have created is aware of what the pieces are. So, you can do intelligent syntax highlighting. You can also do intelligent editing so that when you enter a cell, it knows what format it is expecting. You can also do things like code insight because the concept is defined as some sort of entity that you are interested in. You can define in what ways it can be edited legally. So, one of the things that is created along with this rate plan DSL is this concept of financial number, which can encompass dollar figures but it can also encompass Excel-like formulas and other sort of functionality, and the interesting thing about a Language Workbench like this is, when you tab into this field, just like in an IntelliJ, the options that you have there are the options that are in context over this particular concept that you created and those are the only things that show up there are the things that are pertinent to that context. So, one of the common complaints against the style of coding is doesn’t this lead to language cacophony? Well, if I have developers out here creating their own languages, is not it going to be a nightmare because some guy who develops some sort of language and he will wander off and now I have got some sort of mess of a whole bunch of different languages and if you end up in this case, that means your language is very, very poorly designed. This is exactly the same problem you run into when you have your your developers internally building APIs. If they do really a terrible job of building the API, it is going to be hard to maintain too. Exactly, the same with language, because in some ways, what we are trying to do is represent the same kind of concept just in a slightly different way. Martin Fowler: In many ways, really what is going on with domain-specific languages is very much the thing that is doing with frameworks. In order to take on a project, you have to not just use your base language. We have to also introduce multiple APIs, multiple libraries, multiple frameworks as part of this. If you are going to build a web app these days, you have to decide oh, I need some kind of something like Spring MVC. I need to use Spring. I need to use Hibernate or use a whole bunch of these frameworks and that becomes part of your development environment and it is a whole bunch of things you have to learn. I do not think actually that domain-specific languages add very much to that. They are not any more hard to learn than the fundamental frameworks that they cover because all they are fundamentally is a fairly slim veneer over the actual frameworks themselves. It is these abstractions and frameworks, which are the hard things. The domain-specific languages are just merely small things that make it easier to use them. So I do not think it actually makes a big difference. We are not, in particular, it is important to stress we are not talking about multiple general-purpose languages. Each domain-specific language is usually something very small and limited. Again, the comparison is to think of something like a Hibernate configuration file or a Struts configuration file. The only difference is it is written in something a bit more readable than the XML. Neal Ford: And in fact, if it is hard to read your DSL, then you have done a very poor job of creating it, because that was one of the goals, to create more readable code. Martin Fowler: Well, keeping an eye on our time, we should probably think about wrapping up. I know we have got a few more slides but, so we can… Neal Ford: Yep. We will just go over these very quickly. Why are not we already doing this? External DSLs give you the most potential but up into at a much higher cost because you have to build your own parser generator, etc. There is this COBOL inference which says, well, once we are able to write our own languages, then we will not need developers any more and this is another Martin Fowlerism, that most technologies that are supposed to eliminate professional programmers do nothing of the sort. Your end users are not interested in writing code. They are interested in the code doing stuff and they still need programmers to do that. What we are talking about is creating code that a business analyst can read, not write, and speaking the same language as the business people. If you can create really fluent interfaces, it allows you to communicate with them better which is what I said. Martin Fowler: And that is a point worth emphasizing. That is something we have begun to see in some of the projects that we have done at ThoughtWorks where we have taken a deliberate attempt to use DSLs and make them readable for business people and the important thing here is readable and reviewable, not that they are necessarily writable. Neal Ford: So, the boundary between external DSLs and general-purpose languages, Martin has already talked about, you do not want to create a general-purpose language. You want to create a very, very tightly focused domain-specific language that covers just one domain. You are better off having a lot of small domain-specific languages and trying to create a new general-purpose language, which is... Martin Fowler: In general, domain-specific languages are not too incomplete and in general, they do not provide abstraction mechanisms such as subroutines or object or things of that kind. It is not 100% true but it is true most of the time. Neal Ford: They tend to be very declarative rather than imperative in nature. Martin Fowler: Yeah. Neal Ford: So, there is a token type not recognized…. Martin Fowler: Oh! Cool. Neal Ford: That is interesting. When you are building DSLs, it is a good idea to start with the end in mind. There are two different ways to do this. You can take an API and morph it into a DSL, but the other way to do this, this is a classic picture. This is the Rake napkin. Jim Weirich when he created the Rake, which is make for Ruby, was frustrated with the make utility that he was using and he was at lunch one day and talking to a colleague and he said, “I am so frustrated with make. I think I am just going to rewrite it in Ruby”. So they sat down and sketched down on a napkin, what it would look like if it were in Ruby and this is an actual scan of the Rake napkin. He took it back to his desk and two hours later, he had the beginnings of the Rake project. So he started with what he wanted to see and worked toward that. So, internal DSLs and dynamic languages, Language Workbenches for static languages, we believe that this is a huge competetive advantage because you have got better abstraction mechanisms, slightly harder to write but easier to maintain. This may well be the next big paradigm. This may be the next evolutionary step beyond object-oriented programming. Martin Fowler: Yeah. I do not tend to look at it as the next big thing. I do not necessarily know what the next big thing is but I do think it is something that is interesting, and the fact that I am putting the time in…to make this almost certainly my next book is a sign of how interesting I think something like this might be. But I do not want to oversell how important it is. It is a useful, another tool to consider adding to the toolbox but it is also worth when you have some bits of it you can use today, particularly the internal DSL stuff. Some bits of it are definitely much more on the horizon. The Language Workbenches, I think are really, really interesting but it is going to be a few years before most people can think about using them for real projects. Neal Ford: So they are running us away from here, so thank you very much and I hope you enjoyed the rest of the conference.