2011-03-31

searching e-mails with wanderlust and mu

I have discussed the Wanderlust e-mail client a couple of times already. I'm still using it, so I keep on learning new tricks. Even though there has been quite a bit of action in the competing gnus e-mail client, for my particular use-case, Wanderlust is still the best option.

'My particular use-case' consists of storing my mail in Maildirs, which I fill with either offlineimap (which has fortunately found a new maintainer) or fetchmail.

mu

When dealing with e-mail, one particularly important feature for me is the ability to search my messages. In fact, it's so important for me that I wrote some software to do this for me; the software is called mu; it indexes the messages in my Maildirs, and then allows for searching them using queries, based on message contents, headers, or other message properties.

mu works through a command-line interface, although there is an experimental GUI available as well. The command-line interface makes it possible to hook mu up with various mail-clients, such as mutt, or Wanderlust. Some Linux distributions ship mu, but since the versions they ship are often a bit outdated, I recommend building it yourself from the sources linked on the mu website. The process is fairly straightforward; and there is plenty of documentation in the form of man pages.

mu and wanderlust

I've been combining mu and wanderlust for a while (see mu and wanderlust - the old way, below), but this week Sam B. on the mu mailing list showed a way to do so in a much more elegant way - using virtual or query folders.

How does this work? Well, after installing mu, add the following to your Wanderlust setup file (~/.wl or it's moral equivalent – see the older Wanderlust posts for the details):

(require 'elmo-search)
(elmo-search-register-engine
    'mu 'local-file
    :prog "/usr/local/bin/mu" ;; or wherever you've installed it
    :args '("find" pattern "--fields" "l") :charset 'utf-8)

(setq elmo-search-default-engine 'mu)
;; for when you type "g" in folder or summary.
(setq wl-default-spec "[")

So, to start with the last part, whenever you type g in folder or summary, in the mode-line you will get something like Folder name (.inbox): [. Now simply type your mu search expression and press Enter, and wanderlust opens a (temporary) folder with the search results. Brilliant!

Next, to add virtual folders for searches you do often, simply add some folder specifications like the following to your .folders file (again, check the older Wanderlust posts if you're not familiar with folders-file):

VFolders {
# message I received today
  [date:today..now]!mu  "Today"

# messages bigger than 1Mb  
  [size:1m..100m]!mu    "Big"

# signed messages i got in 2010 related to emacs
  [date:2010..2011 flag:signed emacs]!mu "Signed-Emacs2010"

# unread messages
  [not flag:seen]!mu    "Unread"
# or (for mu  >= 0.9.4):
# [flag:unread]! mu      "Unread"
}

After this, restart Wanderlust, and there you go! Wanderlust will display your brand new virtual folders with an icon that looks like a little whale.

You can put arbitrary mu search expressions between the [], matching whatever is useful in a certain case. Check the mu documentation to see how to do this.

Note, the messages you get in these virtual folders are links to the original messages. In practice, this means that changes you make to the links do no affect the originals – if you delete a link you're not deleting the message.

mu and wanderlust - the old way

This discussion would not complete without a description of the old way I used search. This method may still be useful for integrating mu with other clients such as mutt.

What I've been using for a while is a (in retrospect) rather clumsy way to integrate message searches with Wanderlust: based on the results of a query, I would create some special Maildir and fill it with symbolic links to the matched messages, and the visit this special Maildir with Wanderlust. I'll include the code here to contrast it with the more elegant solution that we saw before, but also because the approach taken might be easily adapted for other mail-clients.

;; search using mutt
(defvar mu-wl-mu-program     "/usr/local/bin/mu")
(defvar mu-wl-search-folder  "search")

(defun mu-wl-search ()
  "search for messages with `mu', and jump to the results"
   (let* ((muexpr (read-string "Find messages matching: "))
          (sfldr  (concat elmo-maildir-folder-path "/"
                    mu-wl-search-folder))
          (cmdline (concat mu-wl-mu-program " find "
                      "--clearlinks --format=links --linksdir='" sfldr "' "
                     muexpr))    
          (rv (shell-command cmdline)))
    (cond
      ((= rv 0)  (message "Query succeeded"))
      ((= rv 2)  (message "No matches found"))
      (t (message "Error running query")))
  (= rv 0)))

(defun mu-wl-search-and-goto ()
  "search and jump to the folder with the results"
  (interactive)
  (when (mu-wl-search)
    (wl-summary-goto-folder-subr
      (concat "." mu-wl-search-folder)
      'force-update nil nil t)
    (wl-summary-sort-by-date)))

;; search by pressing 'Q'
(define-key wl-summary-mode-map (kbd "Q") ;; => query
  '(lambda()(interactive)(mu-wl-search-and-goto))) 
(define-key wl-folder-mode-map (kbd "Q") ;; => query
  '(lambda()(interactive)(mu-wl-search-and-goto))) 

After installing mu and putting the above in your wanderlust startup file, you should be able to search by pressing Q. The mu documentation has an example for mutt as well.

conclusion

It's straightforward to integrate advanced searching capabilities to Wanderlust using mu, and thanks to Sam B., it's gotten a lot easier! The second (old) approach may be useful as 'inspiration' for use in other e-mail clients as well, if they do not provide the kind of hooks that the first solution needs.

16 comments:

situmam1 said...
This comment has been removed by the author.
situmam1 said...

Thank you very much. Now I have no more excuses to switching to WL. One thing I wonder though is how different mu is from mairix.

p.s. first post was premature :)

slb said...

Thanks for the props. Again, the following replaces "namazu search (default)" completely.

(setq elmo-search-default-engine 'mu)

;; for when you type "g" in folder or summary.
(setq wl-default-spec "[")


Thanks for all the work on mu. I've re-discovered a lot of email.

djcb said...

@slb: thank you! i've added the missing search thing to the post.

Anonymous said...

(setq wl-default-spec "[")

this fucks up file moving ("o"), since it prepends [ to the modeline

ramon said...

Really neat! One question and a comment, though.

Comment: for those who do not read the man page, you need to build the database first (mu index)

Question: is it possible to know where (i.e., what maildir folder) the found messages are coming from?

djcb said...

@ramon: i don't think you can see the maildir it's coming from *directly* from within wl (should not be too hard to figure out, but will require some elisp); from the command-line you can get the information easily using the "l" format specifier.

ramon said...

Great, thanks! (Sure enough, there was an example in the cheatsheet that I did not read before posting ;-)

Anonymous said...

You should take a look at notmuch. It's a command-line based email indexer that is very fast and has a very nice emacs interface.

ramestica said...

Thanks for the mu+wl recipe. I have installed mu and I have indexed my maildir.

However, before installing mu I noticed that 'make check' did not succeed. mu-util-dir-expand-01 failed. Skipped.

Then in wl I made a search for 'index' in Body, this is a folder with +7000 messages. It took a couple of minutes to complete. I ran it again and it took the same long time to complete. Is this normal? I was expecting this second time to complete in no time.

thanks

Marc said...

Hoping that this blog area can help me as the mailing lists seem to have disappeared!
I've recently converted to Wanderlust (2.12.2) and am running in with Emac 23.3 on an Ubuntu 11.10 system.

When I get an email with file attachments (say a .doc files) that has spaces in the name, and I select 'v', I get a message, from LibreOffice 3.4 that "/home/marc//tmp/EMIzzzz/XXXX/ YYYYYY.doc" does not exist.
When I try to 'v' on a docx file, I get a similar message but this time the file name "/tmp/EMI.../ZZZZ\ YYY\ ZZZZ.doc" does not exist.

Notice that the programs are treating 'space' in file name differently and also note that the tmp directory is different (one is my home and one is the system).

I have made changes to the ~/.mailcap file to use "%s" instead of %s but I don't think that the applications there are being called.

Any help would be appreciated
Thank you
Marc

Marc said...

Have installed mu linkages in my .wl file. It works great. WL gets better and better.
Have one question, however.
The command line for searches, with the "[" prompt does not allow any spaces after it. So... If I were looking to search for something like:
subject:options* date:1w..2w --fields "d s", I would not be able to put it is. Is there a way around the 'spaces' issue.
Thanks in advance.

djcb said...

@Marc: you can add spaces to your queries with
C-q SPC

Anonymous said...

How do you do the initial mu index command?

Unknown said...

I was seeing different results from the mu command.
Then I realized I wasn't putting !mu after [term] so it must not have been using the mu engine by default. It did on a restart of emacs though. Now it is working great. Thanks!

Unknown said...

The elmo-search-defautl-engine setting didn't work until a restart of emacs for me. Now it is working great. Thanks!