interview with Eric Ludlam (CEDET)

One of the notable new features of emacs 23.2 was the inclusion of CEDET. CEDET adds IDE-type functionality to emacs, and one only needs to read the responses to the 100th post to see the great interest in that.

The man behind CEDET is Eric Ludlam (EML), a long-time Emacs user and developer. He kindly answered the many questions we had for him. Thanks, Eric!

djcb: Eric, could you tell us a little bit about yourself? And about that fantastic hobby of yours, building siege engines?

EML: I manage a software engineering team at the MathWorks. The MathWorks are the creators of Matlab. In my spare time I maintain CEDET, the Matlab support for Emacs, and build replicas of ancient siege engines.

I've been building catapults of various types for use at the World Championship Punkin' Chunk since 1998. My team started with a traction trebuchet, then a HCW Trebuchet, then moved to Human Powered Centripetal, and finally to the torsion division with MistaBallista. As far as I know, Mista Ballista is the worlds largest farthest throwing currently functional ballista.

Most recently, I got to help demolish a building where I work by shooting stuff at it, which was a lot of fun.

djcb: How did you get involved with Emacs? Do you spend a lot of time with it?

EML: I've been hacking Emacs since version 18.54 on Ultrix because the alternative was vi. My first post (and thus contribution to Emacs) was to gnu.emacs.sources on June 6, 1992. I've used Emacs to work on C/C++ code for many years which inspired all the various parts of CEDET, but these days I don't get much coding time.

My work with Emacs and CEDET is seasonal currently. The summers are usually busy with family activities, and my falls are busy getting the catapult ready, so CEDET work usually happens in the winter.

djcb: Can you tell us a bit about the background of CEDET?

EML: I started with some really basic editors and eventually with LSE (Language Sensitive Editor) on VMS. LSE was pretty cool. When I switched OSes to Ultrix and was forced to use Emacs, and discovered you could script it, I was hooked. My first big Emacs hack was etalk, an implementation of Unix talk in Emacs.

In one of the Emacs related newsgroups, a spoof story floated around about how Emacs became sentient, and started writing code for Richard [Stallman] instead of the other way around. Since then I've always wanted to get Emacs to start writing my code for me. I wrote cparse.el, a regexp nightmare that could parse almost any C file into a sequence of highly detailed tags. The tags could be used to then generate code, and I had a cool comment writing utility, automatic prototype generation, and some navigation features. On the whole it failed for many reasons.

The concept, however, continued. CParse became the basic idea behind what is now the Semantic package in CEDET, though none of the old stuff remains beyond some echos in some SRecode utilities.

My end goal is to provide round-trip code management in Emacs. Emacs needs to be able to read your code, allow you to perform transformations in the code in an abstract way, and then write new code for you. Everything in CEDET is building toward that goal. It is taking a long time, and you can see it starting to take shape. Semantic can parse the code, COGRE can manipulate UML diagrams representing your code, and you can export diagrams into code through SRecode. Unfortunately this isn't a round-trip solution as the generated code currently should not replace existing hand-written code.

Can you tell us a bit about the CEDET-project?

EML: I maintain CEDET and field questions on the mailing list. There are several contributors who either own some tool in the "contrib" area, fix bugs, answer questions or apply patches for new features. There has always been someone to pick up the slack when I'm busy elsewhere, and I've greatly appreciated that.

The easiest (and most common) way to contribute code to CEDET is to provide support for some new language, or to create a language agnostic tool using the CEDET APIs. As with Emacs, you need to be able to assign copyright to the FSF to contribute to CEDET's core, but new language support or utilities can have a home in the contrib area as well.

djcb: CEDET is part of Emacs since 23.2. Can you tell us a bit about how that happened?

EML: I've been providing assignments for the parts of CEDET since 1996 or so. For Emacs 23.1, CEDET was at a stable and useful enough state that Chong Yidong was willing to do the work to integrate it into Emacs. My ability to get a legal release for CEDET in time for Emacs 23.1 failed, so it went into 23.2 instead. It is my understanding that CEDET's smart completion engine reached a performance and reliability point that made it desirable to have in Emacs.

Emacs currently includes the parts of CEDET needed to make EDE, the project management system, Semantic and it's user tools, and SRecode all work from a user perspective. Emacs lacks the Semantic development environment, needed for writing new language support, and COGRE, the UML diagram editing tool.

djcb: Are there plans to include those parts as well?

Yes. It is important for users with a stock Emacs to develop new languages using the missing Semantic development tools. I don't know if COGRE will ever go in, though I expect that this will be done via the new package management system [elpa].

djcb: Did you ever try IDEs such as Eclipse? How do you think CEDET stacks up against such programs?

EML: I have used Visual Studio more than Eclipse, and certainly watched demos on how these and other tools work.

These tools win over CEDET's features in that they usually integrate directly with the compiler, VM or whatever, and have very good completion engines and UIs that you can interact with.

CEDET wins in that you don't need a compiler, or even code that can compile for it to work and start providing useful completions. It can infer an awful lot from a project.

CEDET's language support structure is thus simple in comparison, and CEDET supports many more languages as a side effect. It also wins because it is in Emacs, and I've done my best to try and match the "Emacs Way". For an Emacs user, this is intuitive, but can appear quirky to outsiders.

djcb: What about other development tools for in emacs?

EML: There are a lot of language specific solutions like GCCSense, slime, JDEE, and others out there. These projects are great in that they take a language, and push it to the edge of what you can do, and the users generally love that stuff. When someone wants to do the same thing, like when CSDE started for C#, and it tried to copy JDEE, it was just that, a copy that didn't quite make it.

It is my hope that CEDET will become the target of choice for users who want to make their language support in Emacs the best. They will save a lot of time doing so as well. The same infrastructure for integrating in external tools can be used to get improved results out of CEDET, but CEDET will handle converting that into a representation that would then allow any tool built on CEDET to work. This is the same model that worked well for comint.el, and gud.el.

Yasnippet is a bit different in the way it "competes" with SRecode. My first implementation of SRecode tried recycling tempo templates, but I couldn't get it to work. In fact all the template systems I investigated fell short of what I was trying to do, so I had to roll my own, and ended up using a variant of the Google template format.

The key difference is that the goal was for SRecode to provide a series of base templates for code generation. A tool writer would then write some tool to generate code. A user would then use the tool, and say "This is great, how do I change where the { goes?"

The answer is to override the template with that detail in it without interfering with the system templates. That only works if the templates are sufficiently granular that the changed template can be simple. Template reuse needs to be high so that if your company formats your methods as:

int mymethod() { implementme() };

instead of

int mymethod() {
}

that one template change will allow this to happen for all code generated from every tool.

Thus, the audience for yasnippet and SRecode is quite different. Of course you could do yasnippet like things with SRecode if someone took the time to improve the field-editing feature in SRecode, but that has not been my focus.

djcb: Do you follow the overall Emacs development process? Are there things you would like to see changed/improved?

EML: I watch the emacs-devel mailing list, mainly to see if anything related to CEDET goes by that I can help with, or what the latest cool feature might be. My first contribution of Speedbar to Emacs in Emacs 20 helped drive some features related to overlines, boxes, and other face attributes that appeared in Emacs 21.

In the future, once CEDET and Emacs cross merge techniques are hashed out, I would expect some key parts that need performance improvements might move into C. Another part of CEDET that I'd love to see become bound closer to Emacs is mode-local.el. It tries to simulate mode-local variables and mode-local functions. This is critical in a complex system like CEDET that needs to provide a language support author with fine grained control. There are good reasons not to make what is currently implemented "the Emacs way", but it would be nice to resolve those and enable mode authors a with a more powerful way to customize the user experience.

djcb: Do you have specific plans for CEDET in the future?

EML: Here is the short list:

• Improve Emacs/CEDET cross merging
• Offload tag storage to an external process - needed for scalability
• Design/define a "CEDET mode" as a way of simplifying the confusing array of distinct tools and modes that make up CEDET now.
• Finish the smart-context menu project
• Resolve the "code replacement" problem of parsing a block of code, transforming it, and re-creating that code in place reliably.

djcb: In addition to all these 'generic' improvements, are there specific areas where new contributors could make a difference?

EML: Supporting new "stuff" in CEDET is one big win. Stuff can be:

• New project types under EDE to ease transition from something like Visual Studio to Emacs.
• Language support, such as parsers, or mode overrides. Finding ways to take existing cool tools, like JDE's beanshell, or slime's inferior lisp process and allowing it to do work for CEDET is another big win.
• Templates for code generation in new languages.

Tool writing would also be good. JDEE's author Paul Kinnucan converted many bits of JDEE to CEDET and co-designed many of CEDET's parts along the way, such as semanticdb, and that was a huge help. ECB's current maintainer Klaus has also had a big impact on the way concepts are abstracted to a tool that depends on CEDET.

Naturally, joining the mailing list and fixing reported bugs and improving the doc is also a huge help, but not as exciting as writing new code. :)

djcb: Finally, many people want to start using CEDET, but it seems they have a bit of trouble to get started. Do you have any recommendations for them?

EML: Start with the cedet.info file which has many of the common setup configurations in it. If you run into something not explained well or at all, be sure to join the mailing list and be specific about how that doc failed. Very few questions on the mailing list refer to the doc, so the doc is rarely improved in a way that can help others.

Another good starting point is Alex Ott's article A Gentle Introduction to CEDET, which is very helpful.

Thanks a lot for your time, Eric! Wishing you a lot of success with CEDET and all your other projects!

interview with Chong Yidong and Stefan Monnier

Earlier this year, Chong Yidong and Stefan Monnier took over Emacs maintainership from Richard Stallman, and they successfully completed the Emacs 23.1 release. I asked them a couple of questions about the process, Emacs-development and some of the plans for the future.

I'd like to thank Chong (CYD) and Stefan (SM) to take the time to answer my (djcb) questions and even more so for doing an excellent job bringing us Emacs 23!

---

djcb: First of all, could you tell us a bit about yourself? For example, what you do when not hacking on Emacs?

CYD: I'm a postdoc in theoretical physics, specializing in photonic crystals and other optical phenomena.

SM: I'm a professor at the University of Montréal, teaching and researching theory of computer languages. More specifically, I design new type systems and try and abuse existing type systems for "type based formal methods" purposes.

djcb: Earlier this year, the two of you took over the maintainership of Emacs from Richard Stallman. How did you get involved in hacking on Emacs? How has the transition gone?

CYD: My first involvement in Emacs-related development was around 2004 or 2005—very recent by Emacs hacker standards—when I found myself with some free time on my hands after college. At that time, I wrote wikipedia-mode, a major mode for editing Wikipedia articles, plus some word-wrapping code that eventually became longlines-mode, and patches to emacs-devel fixing a few minor bugs. My level of involvement gradually grew, until eventually I was helping Richard to roll the pretest tarballs for the Emacs 22 release.

Because I was quite active in the Emacs 22 release process, I've been pretty comfortable with my role in Emacs 23. It helps, of course, that many parts of Emacs have their own dedicated and experienced maintainers, e.g. the major Lisp packages such as CC-mode, Gnus, and Org-mode.

SM: I started hacking on Emacs a fairly long time ago when I was waiting to start my PhD, but it only got more serious during my PhD when I decided that PCL-CVS was a neat idea but unusable as it stood (for lack of maintainership). It all went downhill from there.

The transition to maintainership happened very smoothly. I had already considered maintaining Emacs when Gerd [ Gerd Moellmann ] left (i.e. when 21.1 was released; at which point Richard ended up regaining maintainership for lack of any other volunteer), but it was a pretty busy time for me, so I decided not to. This time Richard kept a very active role, which coupled with the help of Chong made it very pleasant.

There's a fair bit of pressure, of course, because it's a very old package, so people have a lot invested in it, making some changes terribly delicate. As a maintainer, I did get to steer the direction of Emacs development, tho mostly by my own contributions and by imposing some contentious new defaults. The role of a maintainer as I see it is mostly to make sure the package keeps its integrity.

But I have to say, that while Chong started maybe a bit more of a "rookie maintainer" than I, he quickly took over and he deserves much of the credit for 23.1, while I was too busy with my work to do much good.

djcb: Talking about Emacs development: there are of course many people involved. Can you give a estimate of how many?

CYD: There are about 120 people who have commit access to the code repository; of these, I think around 20 contribute regularly. This does not count the packages that are maintained separately from Emacs.

Additionally, we do of course receive a steady stream of small patches from various users.

Emacs 23 has just been released (on July 29 2009), congratulations, a great accomplishment indeed! From your perspective, what are the most important improvements in Emacs 23 for end-users? And what about the internals? Are there any big changes in the way Emacs operates?

CYD: I'd describe the Emacs 23 release cycle as dominated by internals changes, in contrast with Emacs 22, where most of the major improvements occurred at the Lisp level. There are two fundamental changes. First, the internal character representation is now Unicode-based, which simplifies various aspects of multilingual editing. Second, the font engine has been revamped, and, among other things, we now support anti-aliasing on X. Both these changes are due largely to Kenichi Handa, who deserves a huge amount of credit for patiently developing the code a period of years.

One other major internals change is a restructuring of the terminal interaction code, by Károly Lőrentey, which allows a single Emacs process to display on X and text terminals simultaneously. Building on this "multi-tty" code, Dan Nicolescu implemented a small but clever hack, allowing Emacs to run as a daemon serving emacsclient connections.

There are several Lisp-level changes, large and small. For instance, Stefan revamped the minibuffer completion code, which is now more sophisticated about generating completions. And there are, as usual, new modes and packages: Doc-view mode, Ruby mode, nXml mode, etc.

SM: Better support for Unicode, and better support for fonts, multi-tty support, plus lots of new modes as always. Of course, I'm very happy with my new completion code, which makes partial-completion-mode obsolete (and enabled by default).

The new support for Unicode and for fonts required significant changes. Big thanks to Kenichi Handa for most of that.

djcb: Are there any features that you would have liked to add, but that were somehow not yet ready?

CYD: One feature that I'd have liked to include into 23.1 is CEDET, a set of packages by Eric Ludlum (the author of Speedbar), which turns Emacs into an IDE. There was no time to merge it for 23.1, but hopefully it will be included in 23.2.

SM: Several packages were planned for inclusion, but didn't make it. Support for GNUstep was planned (and is actually in there) but doesn't work. Also I hoped the new VC code would be developed further, but it sadly stayed at the stage where it mostly provides the same features as the old one (with all kinds of improvements in the way it supports them, tho).

djcb: There is always a bit of tension in Emacs between keeping things as they are, and changing things to be more like other programs - for example when thinking about key bindings and various defaults. What is your take on this? Should Emacs try to accommodate new users, or instead try to keep things as they are?

CYD: My impression is that I'm a little more conservative than Stefan with regards to changes, though I'm not sure what he thinks ;-) That said, we seem to arrive at the same conclusions with surprising frequency.

SM: Emacs standard key bindings (like C-x and C-c prefixes) clash badly with "standard" key bindings of other apps, so I don't think there's much hope to make Emacs like other applications. But yes, I generally believe that, all things being equal, it's better to be like others than to be different in this respect. But since changing bindings (or behaviors) is disruptive, I only consider it worthwhile if I believe the new default is really superior (not just for new users).

djcb: For example, in Emacs 23, transient-mark-mode is the default, but delete-selection-mode is not. How do you decide such things?

CYD: Typically, we try not to make flashy changes. The transient mark mode change is the exception that proves the rule: transient-mark-mode is so useful, and is so widely used (even Richard uses it), that it doesn't make sense to leave it off by default. But the rule of thumb is to improve Emacs on Emacs' own terms; for instance, CUA mode will not become the default anytime soon, I think.

SM: transient-mark-mode is an enabler: it allows some commands to behave differently depending on the activation state of the region. So it's a clear improvement. delete-selection-mode is not as important in this regard. We may see something along the lines of delete-selection-mode at some point, tho probably something more minor that only caters to the few cases where delete-selection-mode is more than just a way to avoid hitting C-w.

djcb: How do you see the competition with other text editors? Do you look for ideas elsewhere? Is there any other editor you would be using if Emacs did not exist?

CYD: I'm afraid I don't pay much attention to other editors.

SM: I used Zmacs, XEmacs, and Epoch at some point. That's about it. I do like structured editors, and I think Emacs should and will move in this direction (with more parsing going on).

djcb: It's a bit premature of course, but it's always interesting to speculate a bit about the future. Do you have any particular post-Emacs-23 plans? Obviously, this all depends on what people come up with, but are there any directions you would like Emacs to go?

CYD: The present plan is for Emacs 23.2 to contain a small number of new features, in addition to bugfixes. As mentioned above, I'd like to try to include CEDET. In general, I hope to move to shorter, more disciplined release cycles. Emacs 23 was a good step in that direction, as it was shorter than the previous cycle.

SM: My main goal for Emacs-23 was to shorten the release cycle. Hopefully, the quality has not been reduced accordingly. For 23.N there are several improvements planned (or even done), mostly about inclusion of packages like js2-mode and CEDET. In the longer term, the main goals for me are the integration of the lexical-scoped branch, the support for bidirectional display, and adding more parsing technology (basically replace syntax-tables with something like lex & yacc, maybe).

Thanks a lot Stefan and Chong!