Enter Xtext

Xtext is an Eclipse framework for implementing programming languages and DSLs. It lets you implement languages quickly, and most of all, it covers all aspects of a complete language infrastructure, starting from the parser, code generator, or interpreter, up to a complete Eclipse IDE integration with all the typical IDE features we discussed previously.

The really amazing thing about Xtext is that, to start a DSL implementation, it only needs a grammar specification similar to ANTLR. You do not have to annotate the rules with actions to build the AST, since the creation of the AST (and the Java classes to store the AST) is handled automatically by Xtext itself. Starting from this specification, Xtext will automatically generate all the mechanisms sketched previously. It will generate the lexer, the parser, the AST model, the construction of the AST to represent the parsed program, and the Eclipse editor with all the IDE features!

Xtext comes with good and smart default implementations for all these aspects, and indeed most of these defaults will surely fit your needs. However, every single aspect can be customized by the language designer.

With all these features, Xtext is easy to use. It produces a professional result quickly, and it is even fun to use.

Since version 2.9.0, Xtext allows you to seamlessly port your DSL implementation and IDE tooling to IntelliJ and also to embed your DSL editor in a web application.

Installing Xtext

We will use the latest version of Eclipse. At the time of writing this book, the latest version of Eclipse is 4.6, named Neon. This version requires Java 8, so you will also have to make sure that you have Java 8 installed (see https://www.eclipse.org/downloads).

Xtext is an Eclipse framework; thus, it can be installed into your Eclipse installation using the update site as follows:

http://download.eclipse.org/modeling/tmf/xtext/updates/composite/releases

Just copy this URL into the dialog you get when you navigate to Help | Install New Software… in the textbox Work with and press Enter; after some time (required to contact the update site), you will be presented with lots of possible features to install, grouped by categories. Navigate to the Xtext category and select the Xtext Complete SDK feature.

Alternatively, an Eclipse distribution for DSL developers based on Xtext is also available from the main Eclipse downloads page, http://www.eclipse.org/downloads, called Eclipse IDE for Java and DSL Developers.

Using Xtext with IntelliJ will be discussed later in Chapter 11, Continuous Integration.

Note

At the time of writing this book, the current version of Xtext was 2.10, and this is the version used in this book.

Let's try Xtext

Hopefully, by now you should be eager to see for yourself what Xtext can do! In this section, we will briefly present the steps to write your first Xtext project and see what you get. Do not worry if you have no clue about most of the things you will see in this demo; they will be explained in the coming chapters.

Perform the following steps:

1. Start Eclipse and navigate to File | New | Project…; in the dialog, navigate to the Xtext category and select Xtext Project. Refer to the following screenshot:
   
2. In the next dialog, you can leave all the defaults and press Finish:
   
3. The wizard will create several Eclipse projects and will open the file MyDsl.xtext, which is the grammar definition of the new DSL we are about to implement. You do not need to understand all the details of this file's contents for the moment. But if you understood how the grammar definitions work from the examples in the previous sections, you might have an idea of what this DSL does. It accepts lines starting with the keyword Hello followed by an identifier, then followed by !. Refer to the following screenshot:
   
4. Now, it is time to start the first Xtext generation, so navigate to the file MyDsl.xtext in the org.xtext.example.mydsl project, right-click on it, and navigate to Run As | Generate Xtext Artifacts. The output of the generation will be shown in the Console view. You will note that a file will be downloaded from the Internet (thus, an Internet connection is required at this stage): ''downloading file from http://download.itemis.com/antlr-generator-3.2.0-patch.jar …''. This JAR will be downloaded and stored in your project once and for all as .antlr-generator-3.2.0-patch.jar (this file cannot be delivered together with Xtext installation: its license, BSD, is not compatible with the Eclipse Public License. Note however that such a jar is not needed at runtime). Note that since this file starts with a dot, it is automatically hidden from the Eclipse Package Explorer, just like the files .classpath and .project. You can make them visible by removing the filter .* resources in the Package Explorer. Wait for that file to be downloaded, and once you read Done in the console, the code generation phase is finished, and you will note that all the projects now contain much more code. Of course, you will have to wait for Eclipse to build the projects and you need to make sure that Project | Build Automatically is enabled.

   Tip

   If you want to avoid downloading this additional JAR, you need to install an additional feature into your Eclipse. Use this update site:

   http://download.itemis.com/updates/releases/

   Navigate to the Xtext Antlr category and install the feature Xtext Antlr SDK Feature. When you restarted Eclipse, if you create a new Xtext project from scratch and you run Generate Xtext Artifacts, then the additional JAR is not needed anymore.

5. Your DSL implementation is now ready to be tested! Since what the wizard created for you are Eclipse plug-in projects, you need to start a new Eclipse instance to see your implementation in action. To do so, right-click on the org.xtext.example.mydsl project and navigate to Run As | Run Configurations…; in the dialog, select Eclipse Application, and then the New button to create a new launch configuration. Select it and click on Run. Refer to the following screenshot:
   

   Note

   If for any reason, you are using an earlier version than Java 8, before you start the new Eclipse instance, you must make sure that the launch configuration has enough PermGen size; otherwise, you will experience out of memory errors. You need to specify this VM argument in your launch configuration's Arguments tab, in the box VM arguments: -XX:MaxPermSize=256m.

6. A new Eclipse instance will be run and a new workbench will appear (you may have to close the Welcome View). In this instance, your DSL implementation is available. Make sure you are using the Plug-in Development perspective or the Java perspective (you select the perspective with Window | Perspective | Open Perspective). Let's create a new General project (File | New | Project… | General | Project) and call it, for instance, sample. Inside this project, create a new file (File | New | File); the name of the file is not important, but the file extension must be mydsl (remember that this was the extension we chose in the Xtext new project wizard). As soon as the file is created, it will also be opened in a text editor, and you will be asked to convert the project to an Xtext project. You should accept that to make your DSL editor work correctly in Eclipse.
7. Now, try all the things that Xtext created for you! The editor features syntax highlighting. You can see that by default Xtext DSLs are already set up to deal with Java-like comments such as // and /* */. You also get immediate error feedback with error markers in the relevant parts of the file, even if you have not saved the file yet. The error markers will also appear in the Problems view and in the corresponding file in the Package Explorer as you soon as you save the file. The outline view is automatically synchronized with the elements in the text editor. The editor also features code completion. All of these features have been automatically generated by Xtext starting from a grammar specification file:
   

This short demo should have convinced you about the powerful features of Xtext. Implementing the same features manually would require a huge amount of work. The result of the code generated by Xtext is so close to what Eclipse provides you for Java that your DSLs implemented in Xtext will be of high-quality and will provide the users with all the IDE tooling benefits.