introducing mu4e, an e-mail client for emacs

background

I've haven't written too much here lately. This is because I have spent most of my emacs hacking time on mu4e, an emacs e-mail client. It's only slightly over six months old since its first release, but it has progressed rather quickly! So, I think it is time to give it an introduction to a wider audience. In future posts, I'll go into more detail.

E-mail is a very important means of communication for me; and for purposes of integration, it prefer to do my e-mail with emacs. For a long time, I used mutt with emacs as the editor; a few years back I switched to Wanderlust, and wrote a few articles about it.

There's a lot to like about Wanderlust: it's very featureful, and allows for a lot of customization. Still, there were a few things I was not fully happy with; the most important one was the speed of some operations; another annoyance was the fact that I got rather frequent cache-corruptions when using newer emacs versions.

So, what was I to do? Well, a few years back, I had written an e-mail indexer/searcher, called mu. It takes the messages in a maildir-folder, and allows you to query them. I wondered if I could use mu as the backend for an emacs-based e-mail client. So, I fired up emacs, started writing some code and some later I had a first version of… mu4e.

how does it work?

mu4e is an emacs front-end for mu; when you start it, mu4e connects to a mu server process (started on-demand). It accepts simple commands, and responds with emacs s-expressions, in asynchronous fashion. In practice, all the heavy work is done in this server process, and emacs shows the results when it is ready (usually almost instantly). Emacs does not need to wait for the backend, and things stay snappy, even with tens of thousands (or more) e-mail messages. The only operation that may still require some waiting is sending mail, since that uses the emacs built-in smtpmail, which blocks during its operation.

mu expects your e-mail messages to be stored in a maildir (a one-file-per-message system); you can get these mails there through tools like offlineimap or fetchmail. mu periodically indexes the messages, and stores the results in a Xapian database. A lot of what mu4e does is querying this database – so for example, what you see as 'my inbox folder' in the mu4e-frontend, is in fact just the result of a query of all message that happen to live in that folder. You can query for many things – message sender, subject, date, words in the body and more.

mu4e does not just query messages; it can also move messages between folders, delete them, modify their flags and so on (and afterwards update the database). All 'state' is in the file system, and the database is just a quick way to get to it; if you delete the database and then re-index, nothing is lost, and changes made with other tools (e.g. mutt) are picked up by mu as well.

trying mu4e

Some Linux distributions ship mu4e, but not necessarily a very recent version; and because mu4e is developing quickly, you may want to use the latest official release, or (if you are more adventurous) check out the git repository. You'll need GNU/Emacs 23 or 24, GLib, GMime >= 2.4 and Xapian.

After installing mu4e, you can check the mu4e manual, which should be fairly complete. In fact, one of the goals of mu4e is to make it easy to set up; for that reason, the documentation includes complete examples of how to set things up. The Getting started chapter should get you up to speed quickly.

So, give it a try if you're interested; the mu=/=mu4e project is very open to suggestions (and even bugs reports!), so feel free to put them here or subscribe to the mu mailing list.

a small tour

I'll end this with a number of screenshots, to give you a rough idea of how things look.

The main screen

When you start mu4e (M-x mu4e), you're taken to the main screen, which list the things you can do. The entries should speak for themselves. Note that you can define you're own bookmarks. In general, mu4e tries to make things as easy as possible, and provide auto-completions where possible.

The headers / view split-screen

The results of your queries are shown as a list of headers; if you click one of the headers, mu4e splits the screen horizontally (optionally, you can split the screen vertically, or only show the message view). You can customize the headers shown, re-order them and so on.

The message view supports html message, inline images, and has a mechanism to associate user-functions with messages and attachments (such as 'view in browser' or 'pipe through command'); see the manual for some examples.

The message composition view

Obviously, you can reply to messages, forward them, or compose a new message. For writing and sending messages, mu4e reuses emacs' built-in message-mode. There's support for auto-completing addresses (based on the e-mails you've sent/received before).

There's even (experimental) support for writing you're messages using org-mode, and then sending them as rich-text (html) messages.

parting thoughts

So, this was a short introduction to mu4e, barely scratching the surface, but hopefully conveying the main ideas. The manual discusses things in much more detail, and I'll write more about it in the future.

mu4e is still a young project, but, for what's it worth, I've been using it full-time for over six months, and a growing number of people are doing the same. So, if you're interested, give it try and let us know you think!

Have fun!

searching e-mails with wanderlust and mu

I have discussed the Wanderlust e-mail client a couple of times already. I'm still using it, so I keep on learning new tricks. Even though there has been quite a bit of action in the competing gnus e-mail client, for my particular use-case, Wanderlust is still the best option.

'My particular use-case' consists of storing my mail in Maildirs, which I fill with either offlineimap (which has fortunately found a new maintainer) or fetchmail.

mu

When dealing with e-mail, one particularly important feature for me is the ability to search my messages. In fact, it's so important for me that I wrote some software to do this for me; the software is called mu; it indexes the messages in my Maildirs, and then allows for searching them using queries, based on message contents, headers, or other message properties.

mu works through a command-line interface, although there is an experimental GUI available as well. The command-line interface makes it possible to hook mu up with various mail-clients, such as mutt, or Wanderlust. Some Linux distributions ship mu, but since the versions they ship are often a bit outdated, I recommend building it yourself from the sources linked on the mu website. The process is fairly straightforward; and there is plenty of documentation in the form of man pages.

mu and wanderlust

I've been combining mu and wanderlust for a while (see mu and wanderlust - the old way, below), but this week Sam B. on the mu mailing list showed a way to do so in a much more elegant way - using virtual or query folders.

How does this work? Well, after installing mu, add the following to your Wanderlust setup file (~/.wl or it's moral equivalent – see the older Wanderlust posts for the details):

(require 'elmo-search)
(elmo-search-register-engine
    'mu 'local-file
    :prog "/usr/local/bin/mu" ;; or wherever you've installed it
    :args '("find" pattern "--fields" "l") :charset 'utf-8)

(setq elmo-search-default-engine 'mu)
;; for when you type "g" in folder or summary.
(setq wl-default-spec "[")

So, to start with the last part, whenever you type g in folder or summary, in the mode-line you will get something like Folder name (.inbox): [. Now simply type your mu search expression and press Enter, and wanderlust opens a (temporary) folder with the search results. Brilliant!

Next, to add virtual folders for searches you do often, simply add some folder specifications like the following to your .folders file (again, check the older Wanderlust posts if you're not familiar with folders-file):

VFolders {
# message I received today
  [date:today..now]!mu  "Today"

# messages bigger than 1Mb  
  [size:1m..100m]!mu    "Big"

# signed messages i got in 2010 related to emacs
  [date:2010..2011 flag:signed emacs]!mu "Signed-Emacs2010"

# unread messages
  [not flag:seen]!mu    "Unread"
# or (for mu  >= 0.9.4):
# [flag:unread]! mu      "Unread"
}

After this, restart Wanderlust, and there you go! Wanderlust will display your brand new virtual folders with an icon that looks like a little whale.

You can put arbitrary mu search expressions between the [], matching whatever is useful in a certain case. Check the mu documentation to see how to do this.

Note, the messages you get in these virtual folders are links to the original messages. In practice, this means that changes you make to the links do no affect the originals – if you delete a link you're not deleting the message.

mu and wanderlust - the old way

This discussion would not complete without a description of the old way I used search. This method may still be useful for integrating mu with other clients such as mutt.

What I've been using for a while is a (in retrospect) rather clumsy way to integrate message searches with Wanderlust: based on the results of a query, I would create some special Maildir and fill it with symbolic links to the matched messages, and the visit this special Maildir with Wanderlust. I'll include the code here to contrast it with the more elegant solution that we saw before, but also because the approach taken might be easily adapted for other mail-clients.

;; search using mutt
(defvar mu-wl-mu-program     "/usr/local/bin/mu")
(defvar mu-wl-search-folder  "search")

(defun mu-wl-search ()
  "search for messages with `mu', and jump to the results"
   (let* ((muexpr (read-string "Find messages matching: "))
          (sfldr  (concat elmo-maildir-folder-path "/"
                    mu-wl-search-folder))
          (cmdline (concat mu-wl-mu-program " find "
                      "--clearlinks --format=links --linksdir='" sfldr "' "
                     muexpr))    
          (rv (shell-command cmdline)))
    (cond
      ((= rv 0)  (message "Query succeeded"))
      ((= rv 2)  (message "No matches found"))
      (t (message "Error running query")))
  (= rv 0)))

(defun mu-wl-search-and-goto ()
  "search and jump to the folder with the results"
  (interactive)
  (when (mu-wl-search)
    (wl-summary-goto-folder-subr
      (concat "." mu-wl-search-folder)
      'force-update nil nil t)
    (wl-summary-sort-by-date)))

;; search by pressing 'Q'
(define-key wl-summary-mode-map (kbd "Q") ;; => query
  '(lambda()(interactive)(mu-wl-search-and-goto))) 
(define-key wl-folder-mode-map (kbd "Q") ;; => query
  '(lambda()(interactive)(mu-wl-search-and-goto))) 

After installing mu and putting the above in your wanderlust startup file, you should be able to search by pressing Q. The mu documentation has an example for mutt as well.

conclusion

It's straightforward to integrate advanced searching capabilities to Wanderlust using mu, and thanks to Sam B., it's gotten a lot easier! The second (old) approach may be useful as 'inspiration' for use in other e-mail clients as well, if they do not provide the kind of hooks that the first solution needs.

wanderlust iii

Wanderlust

I have been using the Wanderlust E-mail client for almost a year now, and I am very happy with it. In previous postings I already discussed setup and some tips and tricks; and I also had somewhat related posts on bbdb (the address book) and warning you about new mail. But I think it's time for an update.

maintenance

One question that I have received quite a few times was about the maintenance of Wanderlust; it's may be a nice e-mail client, but little seems to have happened in the last few years. It's reasonable concern.

Fortunately, it seems the situation has improved significantly. There's more traffic on the mailing lists, both from old and new users. Various improvements are circulating; the place to get the latest stuff is in David Abrahams' git-repository: semi, flim, apel and wanderlust. The plan is to gradually apply the changes to the upstream (CVS) repository.

bugs

Although I have been quite happy with Wanderlust, there was one bug #5534 that has bitten me a few times, causing occasional hangs when saving (draft) messages with non-ascii characters. It seemed finally to be a bug in emacs itself, which is triggered by something in Wanderlust. A fix will probably be available for the next emacs version; until then, you can work around this by using an external MIME-encoder. To do this, add the following to your WL initialization code (thanks to various helpful people on the WL mailing list):

(setq mel-b-ccl-module nil)
(setq mel-q-ccl-module nil)
(setq base64-external-encoder '("mimencode"))
(setq base64-external-decoder '("mimencode" "-u"))
(setq base64-external-decoder-option-to-specify-file '("-o"))
(setq quoted-printable-external-encoder '("mimencode" "-q"))
(setq quoted-printable-external-decoder '("mimencode" "-q" "-u"))
(setq quoted-printable-external-decoder-option-to-specify-file '("-o"))
(setq base64-internal-decoding-limit 0)
(setq base64-internal-encoding-limit 0)
(setq quoted-printable-internal-decoding-limit 0)
(setq quoted-printable-internal-encoding-limit 0)

(setq-default mime-transfer-level 8)
(setq mime-header-accept-quoted-encoded-words t)

This requires the mimencode-program, which is part of the metamail-package. For the time being, this seems to be the best solution when using Wanderlust with Emacs 23.

one more trick: reformatting

Let's finish with one more trick. Sometimes, incoming mail is formatted quite badly; in particular, the new lines are too long for comfortable reading. From the WL-mailing list, here's a trick to deal with that:

(require 'filladapt)

;; from a WL mailing list post by Per b. Sederber
;; Re-fill messages that arrive poorly formatted
(defun wl-summary-refill-message (all)
  (interactive "P")
  (if (and wl-message-buffer (get-buffer-window wl-message-buffer))
      (progn
        (wl-summary-toggle-disp-msg 'on)
        (save-excursion
          (set-buffer wl-message-buffer)
          (goto-char (point-min))
          (re-search-forward "^$")
          (while (or (looking-at "^\\[[1-9]") (looking-at "^$"))
            (forward-line 1))
          (let* ((buffer-read-only nil)
                 (find (lambda (regexp)
                         (save-excursion
                           (if (re-search-forward regexp nil t)
                               (match-beginning 0)
                             (point-max)))))
                 (start (point))
                 (end (if all
                          (point-max)
                        (min (funcall find "^[^>\n]* wrote:[ \n]+")
                             (funcall find "^>>>>>")
                             (funcall find "^ *>.*\n *>")
                             (funcall find "^-----Original Message-----")))))
            (save-restriction
              (narrow-to-region start end)
              (filladapt-mode 1)
              (fill-region (point-min) (point-max)))))
        (message "Message re-filled"))
    (message "No message to re-fill")))

(define-key wl-summary-mode-map "\M-q" 'wl-summary-refill-message)

Now, you can refill your messages with M-q when in the Summary.

If you have any other nifty WL-tricks that could be useful for others, please share them in the comments, thanks!

managing e-mail addresses with bbdb

BBDB, the Insidious Big Brother Database is an Emacs addressbook application that works particularly well with e-mail. It's one of the classic emacs packages, written in 1991 by Jamie Zawinski.

Personally, I do not use BBDB as a general address book. Instead, I use it to harvest the e-mail addresses of the people that send me mail, so that the next time I'd like to send e-mail, I can auto-complete the names / e-mail addresses. If you're not using some emacs-based email client, BBDB might not be that useful in practice.

Note that other people are using BBDB for much more, see the Emacswiki-entry for some examples of that. Here, I just give some basics to get you started; please refer to the fine manual for all the details.

configuration

So, after you've installed bbdb (just follow the instructions; alternatively Ubuntu/Debian users can simply install the bbdb package), we can add the following to .emacs to set it up:

(setq bbdb-file "~/.emacs.d/bbdb")           ;; keep ~/ clean; set before loading
(require 'bbdb) 
(bbdb-initialize)
(setq 
    bbdb-offer-save 1                        ;; 1 means save-without-asking

    
    bbdb-use-pop-up t                        ;; allow popups for addresses
    bbdb-electric-p t                        ;; be disposable with SPC
    bbdb-popup-target-lines  1               ;; very small
    
    bbdb-dwim-net-address-allow-redundancy t ;; always use full name
    bbdb-quiet-about-name-mismatches 2       ;; show name-mismatches 2 secs

    bbdb-always-add-address t                ;; add new addresses to existing...
                                             ;; ...contacts automatically
    bbdb-canonicalize-redundant-nets-p t     ;; x@foo.bar.cx => x@bar.cx

    bbdb-completion-type nil                 ;; complete on anything

    bbdb-complete-name-allow-cycling t       ;; cycle through matches
                                             ;; this only works partially

    bbbd-message-caching-enabled t           ;; be fast
    bbdb-use-alternate-names t               ;; use AKA


    bbdb-elided-display t                    ;; single-line addresses

    ;; auto-create addresses from mail
    bbdb/mail-auto-create-p 'bbdb-ignore-some-messages-hook   
    bbdb-ignore-some-messages-alist ;; don't ask about fake addresses
    ;; NOTE: there can be only one entry per header (such as To, From)
    ;; http://flex.ee.uec.ac.jp/texi/bbdb/bbdb_11.html

    '(( "From" . "no.?reply\\|DAEMON\\|daemon\\|facebookmail\\|twitter")))
)

This will set up BBDB for you. I have commented the various settings; you can of course get more information for each of them by putting your cursor on them (in emacs) and issuing C-h v.

integration with e-mail clients

Another important part is the integration with e-mail - which is why I am using BBDB in the first place.

As I mentioned before, I am using the wonderful Wanderlust e-mail client for emacs, and you can easily integrate it with BBDB by putting the following in your .wl-file:

(require 'bbdb-wl)
(bbdb-wl-setup)

;; i don't want to store addresses from my mailing folders
(setq 
  bbdb-wl-folder-regexp    ;; get addresses only from these folders
  "^\.inbox$\\|^.sent")    ;; 


(define-key wl-draft-mode-map (kbd "<C-tab>") 'bbdb-complete-name)

For gnus, you'd use something like:

(add-hook 'gnus-startup-hook 'bbdb-insinuate-gnus)

(I am not using gnus myself, feel free to add you setup in the comments)

The BBDB-manual has the details for some other mail clients as well.

Use

So, we have everything set up now – but how does it work? Well, whenever you read e-mails, BBDB stores the various e-mail addresses. Whenever you write an e-mail, you can complete the names with M-x bbdb-complete-name, by default bound to M-TAB. This keybinding conflicts with many window managers, which already use M-TAB (Alt-TAB) for window switching. So, it's useful to rebind it to something else, for example C-TAB (I already did that for Wanderlust, in the example above).

I don't use BBDB too much directly, but you can manipulate the address; first display one with M-x bbdb-display-address, they you can edit a field by pressing 'e', 'd' for deleting the field or record, C-o for adding a new field and so on – see the whole list.

mutt inside emacs

There are some emacs-native programs for e-mail, such as gnus, vm and a couple of others. I never really got into them for some reason; I found them rather hard to set up and a bit counter-intuitive. Maybe I should give them another try, but for the last decade, I've been using the mutt console e-mail client - even with graphical alternatives like Evolution and Thunderbird available.

mutt is not perfect -- you cannot read older e-mails when composing a new one (without starting a new instance) and the macro language is a joke compared to elisp. Still, because it's totally keyboard-driven, I'm really efficient with mutt. Side note: if you need quick searching capability which can be integrated with mutt, see my mu search tool.

Mutt does not have a built-in editor -- instead, it relies on an external program. Obviously, I use emacs for that. Now, I don't want to start a new emacs for each new message that I write; instead I make use of the emacs server functionality. Practically, we can start an emacs server by adding to .emacs:

(server-start)

Now, we can open files in the existing emacs by calling emacsclient instead of emacs; and tell mutt to use that editor. Put in your .muttrc:

set editor="emacsclient +8 %s -a emacs"

This opens new messages in an existing emacs, and put the cursor on line 8 (after the mail headers). If no emacs is running yet, a new one will be started, due to the '-a'-argument.

This solution will work nicely, but requires you to start mutt in a console, and start emacs in a separate window. I prefer, however, to run everything inside emacs. This can easily be done - please look at the 'running console programs inside emacs'-entry again, to create a short-cut.

Now, when editing e-mail messages, it's nice to use one the special modes for that; they provide some interesting 'syntax highlighting' to your mail. You can use mail-mode, or my favorite, post-mode. To automatically use it, download and install it, and add some code to your .emacs:

(autoload 'post-mode "post" "mode for e-mail" t)
(add-to-list 'auto-mode-alist 
             '("\\.*mutt-*\\|.article\\|\\.followup" 
                . post-mode))

This will automatically load post-mode when editing a message from mutt (and for some other programs). You might also want to set up some other things when editing mails, using a hook:

(add-hook 'post-mode-hook 
  (lambda()
    (auto-fill-mode t)    
    (setq fill-column 72)    ; rfc 1855 for usenet messages
    (require 'footnote-mode) 
    (footmode-mode t)
    (require 'boxquote)))

Obviously, you need to install the boxquote and footnode modes, if you want to use those.

Final note: this same setup works nicely for the slrn news reader as well; just use the set editor_command= in your .slrnrc.