I Transitioned Local E-mail Management to Gnus

I have been mostly using E-mail on Gmail interface, as I use Gmail and Google Apps (or, I guess they changed their enterprise offering to Google Apps for Work…) I have been using Mozilla Thunderbird once in a while. In normal circumstance, it would have been OK, but here are some issue of bit slow for the load.

Basically my requirements were:

  1. I literally receive a few hundred of E-mail a day and it needs to be able to handle that. Not that I have to read over every single E-mail with details, but there are a handful of mailing list articles I’d like to triage.
  2. Being able to author and send text E-mail (more on it later)
  3. I use Mac and Linux, and cross platform is a must. (This is one of reasons, in addition to bitter experience in the past that I would never use Microsoft Outlook — in case someone would suggest this — and it’s not great at handling #1 point above.)
  4. Support OpenPGP
  5. It must be able to support multiple account (Home and Work)

So this left me with few choices. Pretty much even 3rd point would disqualify a lot of choices to begin with. Then I just thought, Emacs already has built-in mailer, and considering every platform I ever have to interact would have it already installed, mainly because I use it daily.

So this is how I landed on Gnus.

In the case of myself, the following was an advantage I have found of this set up:

  • It pretty much fulfills my requirements.
  • As far as authoring text E-mail, it’s one of the most powerful softwares you can find out there. After all, it’s a text editor.
  • Gives me full control, won’t try to hide things from me.
  • It’s relatively easy for me to service it myself to modify or extend its parameter and functionality, knowing some Emacs Lisp.
  • Works with Org-mode. (See here.)

Things I purposely didn’t consider are:

  • Search facility — I felt it’s better to leave it up to Gmail interface, considering how fast it can search. Beside some automatic tagging, pretty much my E-mail is managed chronologically, and searching for E-mail more than I can glance over would require some serious deep searching. I have determined that after E-mail gets old enough to past certain point, it would be too old to be useful, and never get touched. Although, I still keep a copy of it, they are pretty much dormant, so I tend to leave the structure flat.
  • Notification — The age of smartphone (and I even have smartwatch) made it quite unnecessary for me to receive notification from my mailer. Although I can have Gnus pull E-mail and notify of new E-mail arrival, I purposely didn’t configure it.
  • Richtext authoring — I do not need fancy HTML/Rich text authoring. If you know me, I don’t purposely send out E-mail with mark ups. I’d rather stick with plain text and occasional use of simplified markdown. (Only time you’d see any formatted E-mail is coming from me is when I send the message out from my phone — because Gmail for Android sends the message out that way. I don’t know the reasoning behind the use of formatted E-mail as opposed to plain text considering you can’t really take advantage of rich formatting on that app.) Occasionally some ill-behaving (and possibly misconfigured) mailer sends me E-mail without alternative MIME part for text. In this case, I’d just read that portion of E-mail on the browser. (K-H command works very well to make this happen.) By the way, if you send mail this way, your E-mail will be ignored and/or will be sent to the very end of my queue.

So if you care about those points, my experience wouldn’t be too useful to begin with, so you can stop reading here.

Considering I have multiple Gmail/Google Apps account, I had to have something that manages multiple SMTP address.
I used using ‘gnus’ to read mail article from the emacs 30 Day Challenge useful to configure this. What I did not do was fully automated the process of choosing SMTP server, as there are fairly complex set of E-mail aliasing I have to take care, so I left some manual control to configuring this information by myself when sending E-mail, specifically by configuring X-Message-SMTP-Method and From fields. The code from the site is still incorporated, to validate that only valid E-mail addresses are ever specified in such configuration. This is primarily done by the code like below:

(defun gnus-set-mail ()
  (interactive)
  (message-goto-from)
  (kill-whole-line)
  (insert "X-Message-SMTP-Method: smtp smtp.googlemail.com 587 example@gmail.com.com\nFrom: Example \n\n")
  (message-goto-body)
  )

Further, I’ve replaced the default key map for toggling thread view (gnus-summary-toggle-threads initially defined as C-M-t to C-c C-t simply because I use C-M-t for opening a console on my Linux machines. (I currently use KDE, but I find this shortcut like Unity useful.) Configuring this is actually easy, by the way:

(define-key gnus-summary-mode-map "\C-c\C-t" 'gnus-summary-toggle-threads)

Gnus contains powerful E-mail authoring feature called MML, which basically allows me to mark up MIME structure. This would allow me, for example, to insert arbitrary MIME parts into E-mail. While I don’t do so much of this, nonetheless, this would allow me to fine tune the way attachments are configured, for example.

Defining MIME parts, each with different character set.
Defining MIME parts, each with different character set.

Smooth sailing so far, but I like to improve few of process to fit my needs, which I will be exploring as they becomes necessary!

Why I Love LaTeX

My History with LaTeX

I have been using LaTeX and its variant (mostly pLaTeX variant, for doing Japanese documents) for about 10 years now. Back when I was in school, I have used to take a note in real time. (Back around that time, Org-mode was still about to be released. If I’d do anything differently today, I’d take a note in Org-mode, and perhaps export it to LaTeX or other formats.) I even negotiated, for some class, that I can submit assignments in PDF output from \LaTeX, which I did get favorable responses from all I asked. As a premise, most others submitted non-coding assignments on paper — some submitted electronically, presumably with Microsoft Word format. (How they can live with formula editor on Microsoft Word, is outside of my comprehension…)

Why You Should Use LaTeX?

To me, LaTeX is a superior document creation tool compared to other forms of document creation.

Flexibility in Toolchain

Essentially, working on a document using an office suite is pretty much take-all-or-nothing. If you are completely sold to an application software you are using, that’s great, but if not, you either have to continue working in an environment you are not satisfied with, or should you choose to integrate process using another method, such as alternative softwares — that’s when things get very awkward. While most of office suite applications provide some support for interfacing with other applications, which could be through set of APIs, or scripting, but because of the complexity of the software, customizing those environments could well be a full time job of its own.

LaTeX file is essentially a plain text format with markup. Any text editors can handle LaTeX files. If the editing LaTeX file, learning LaTeX commands are not your cup of tea, then there are many “IDE for LaTeX” systems you can use instead. One such application I’ve used in the past is called TeXnicCenter which provided menus and toolbar access to many of commonly (and some obscure) commands available in LaTeX. If you are being intimidated by the markup nature of LaTeX, you may want to try it, or many other editors around.

Because of a text based workflow, it integrates very well among other tools. For instance, I use Org-mode to create a document. Org-mode is a very versatile tool that can be used for many forms of data management and authoring. Perhaps because Carsten Dominik, the original author of Org-mode is an astronomy professor, Org-mode is designed so it works very well with LaTeX system.

This provides multiple export path as well. While I prepare a document as a document, the same document also can be used to output as an HTML page. (It is also worthwhile to mention that the HTML output from Org-mode is clean enough that it validates as XHTML strict.)

Clear Separation of Style and Contents

You may have come across the documents that are horribly formatted. I have seen a document that the heading was merely done by change of font size, spacing between paragraph being adjusted by an empty line of different font size; the list goes on and on. This comes from overemphasis of so called WYSIWYG (What You See Is What You Get) concept, where what you see on the screen is what you get as a result. For instance, when you print out the document, the result will be what you see on the screen. (While some may disagree with this notion, but it was premise as marketed back then.)

Even after sociological emphasis is shifted from circulation of document as a paper, and electronic transmission, collaboration, and management, mindset of users were never shifted, and as a result, poorly formatted documents are being produced, perhaps in a degree of hundreds of Terabytes per seconds world wide.

Bad formatting was, perhaps, acceptable when paper trails were the center of focus; after all, noone really cared how data that resulting print was originated from, as long as it is legible.

LaTeX in nature, often enforces document markups, which makes it much harder to do it wrong way. It is still possible to format document in LaTeX wrong way, but either makes getting what you want done more challenging, or will cause errors.

Better Version Control

LaTeX document is a text file, so the document can be version controlled as easily as computer program codes. Difference between version can be inspected easily. This is fundamentally different from how many of Office Suite applications record changes; essentially they are keylogger system where user actions are recorded into the document. If this feature is not enabled when the document is being written, or information is lost for some reason, recovering difference information between two document often provide inaccurate information. Moreover, merging difference from multiple sources are often very difficult because of the nature that style information often interleaved within the document.

Want Try It Out?

Installing LaTeX to your computer is easier than ever, as there’s installation package called TeX Live which includes most of programs you will need to get started. Depending on your preference, you may want also install aforementioned editing software, but your favorite text editors should do a trick.

My Random Emacs LISP Scripts

I’m not particularly, LISPer, but I do code some Emacs LISP for my convinience. Just to pick a few…

  • Create Random Buffer is a script to create some random buffer. I’m actually wondering why something like this doesn’t exist as default. (well, there’s scratch buffer, but…) Useful for experimenting.
  • Anti-Space Invader is a simple script to make two spaces after period, because you should never do that.

Whitepaper: Org-mode at Sakura-Con

This article explains use of Org-mode in a context of convention management.

Who am I?

I am Japanese Guest Manager appointed by relations department of Sakura-Con 2012. My duty at Sakura-Con involves management of Japanese guests of honor for Sakura-Con 2012; everything related to Japanese guests, from schedule managements, human resources, to the field management of the team consisting of a little less than 30 managing 17 guests of honor and their associates — about 60 in total. The team consists of Japanese guest liaisons assigned to each guest as well as 3 of event interpreters assigned to events such as panels.

The team managed more than 80 hours of programming spanning three days as well as times outside of convention hours.

Objective

Managing Japanese guest relations at Sakura-Con involved multiple considerations. Here are some aspects of such managements:

Sakura-Con used its own scheduling system, which is mainly used to schedule events on the global calendar, thus making it possible to coordinate scheduling between multiple departments. You can imagine this as a big conference room scheduling system. Unfortunately, the way the schedule is presented and solely chronological and was very hard to glance. (It was better designed to see what time tables are open but not designed to follow what each resource are assigned for.) Japanese guest relations required customized solution as we had to need of further digesting this information.

Japanese guest relations issues schedule sheet which consists of formatted schedule for each guest of honor as well as staff. This process was assigned to one of my staff, however this had to be handled somewhat concurrently while schedules were being finalized.

Aside from liaisons, Japanese guest interpreters had their own assignment. It is because they were assigned to events rather than guests. This presents interpreters different set of requirements in their scheduling. As the schedule called for many events concurrently occurring but greatest number of such concurrency were constrained by available resources. This also had to be updated on the fly when pertinent changes occur in the scheduling process.

Sakura-Con 2012 particularly hired many new staff due to its number of guests. Each of member had to be accounted for their availability for as well as number of “checklist” for signing them up.

Each staff is assigned to guest to manage. Assignments are done taking their ability in account as well as their preferences.

Management Using Org-mode

During the preparation of Sakura-Con 2012, there were so many type of information to process. Because Org-mode allows variety of output from its org file, it helped to produce output necessary for consumption by our staff.

Schedules had to be constructed so it is easy to glance as well as to have detailed information within. To carry out this, under heading for each guest, information contained table of events, which contained name of the event as well as time (with inactive timestamps) and venue. I can then sort this table based on the time, which generates chronological time table to each guest.

Furthermore, the list required some more information, therefore, it would contain subheading which lists each event by its type, but alphabetically for easy lookup.
This heading had active timestamp. This allowed each of those entry to be exported as iCal calendar file; this file can then be used by interpreters to schedule their assignment.

The following is the sample segment from the real schedule: (Due to confidentiality issues, some of information from the example are eradicated and does not show real schedule of the convention.)

Org-mode sample used at Sakura-Con

Yes, this is probably redundant having two representation of same event, but given calendar output from table seems to be not possible, this was a middle ground I had to reach.
Also having these sectional event subheading means they could be marked with various TODO field. I used this feature to label status of each event to signifying events still requiring changes.

From this file, I could derive HTML output for printed schedule as well as calendar file for interpreter schedules.

Other than some errors originating from input mistakes, the process supported the preparation effort very well.

When the convention added a guest, it involved multitude of process, such as guest invitations. These were managed using mainly by checkbox on Org-mode, among with relevant information linked to. Again, all those information could be converted to HTML and other format for sharing.

Staffs are managed same way as guests, except they had different type of checklist. Their contact information as well as interview processes were logged and amended as the process continued. Staff directory was also published utilizing same data compiled in this process.

The Team Website

Utilizing same data set above, the team website containing guest schedules, staff information as well other miscellaneous information could be produced and maintained very rapidly. Utilizing smartphones browser, everything from guest schedule to assignment information could be referenced everywhere there are coverages. While being static site, onsite update to the contents were very simple. Backend featured Git and new revision could be pushed to update the site from local machine. This was mainly done by a laptop with Clear hotspot. I have also designed emergency USB thumbdrive which contains all the necessary tools (including complete version of Emacs, SSH, Git, etc.) which could be used to continue updating the site in the event that laptop would be out of order. Fortunately, I did not have to resort to use it.

The team website was not only utilized by the team itself, but its reference was expanded to entire department as well as relevant personnel outside of the department, logging more than 3500 requests on the convention week alone, which would have been additional inquiry to the department otherwise.

Future Considerations

Org-mode was an integral part of Japanese Guest Relations at Sakura-Con 2012. While model based on static webpage has worked very well for the purpose, perhaps I can perhaps develop or utilize frameworks that can parse org file input to generate format better fit for many of use cases.

Also mobile integration utilizing tools such as MobileOrg was not within the scope of the project, this is certainly something I will want to explore in the future.

Org-mode and Why You Should Consider Using It

Org-mode is certainly best one of many things I’ve discovered in first 2 months in 2012. Now I use it for anything from note taking, convention management, and even writing full report.

This was great discovery to me, that recently, I have been started a lot of things on the web as well as on more specialized platforms such as Chromebook rather than on traditional PC. Sure I will have to use some type of computer to convert document prepared in Org-mode, more about it later.

Binary Blobs Suck

I have been complaining about various binary blobs in the past. I expressed my hate anything from proprietary format, as well as winmail.dat attached to E-mails. (I’m still seeing that, albeit less often. Hey Microsoft, if you are seeing it push some Windows Update to everyone and get rid of that thing altogether.)

Classic examples are documents generated by any office softwares traditionally had this problem. It started to improve when standardization efforts such as OpenDocument and some sense Office Open XML are introduced in the market. This may have changed significantly in standardized to approach to otherwise proprietary formats that was (and still is) dominating the market, was was very important movement.

However, what these standardization efforts did not solve was accessibility problem. No matter how open those formats are, they would still require use of some extra piece of software, that requires degree of complexity than what you could attain using plain text. TeX (and LaTeX) has been around for years allowing plain text edit, but it was far harder to use than, although, I consider TeX/LaTeX being 100 times better way of creating documents than on Microsoft Word.

Org-mode being plain text, I can be editing the document everywhere: I can do it on PC, I can do it on Mac, I can do it on Chromebook (considering ChromeOS does not have any software packages it is bit cranky to do it with) and I can do it on my phone. Eventually, I will be relying on actual PC to convert, but being able to write documents is very useful aspect of Org-mode and mark up languages in general.

Structured Documents Made Easy

One thing I like about Org-mode (and also TeX/LaTeX) is that it enforces structured documents. I often across many documents E-mail to me that are horribly created document – heading being mere font size change, table of contents written as text, rather than auto generated based on heading information, because it lacks document structure information because of the above.

Sure some people are apparently smart enough to do this properly even when they are using Microsoft Word, but the point is that Org-mode enforces this with positive motivation. By making sure the document has proper format on Org-mode, it makes Org-mode much more useful, as it is outline processor, you will be able to organize contents more efficiently when it is structured.

Project Management Made Easy

Most of things I do at my work is project management. I used to use Microsoft Project, which after I’ve used it for while, I found it wasn’t necessary solution to the problem. Yes, it is perhaps good at making graph that wows management in some companies, but constraints it assume is not always in-line with real life situation. Unless you are working for perfect company, schedule slips, requirements change, which causes heck of changes to the beautiful chart you’ve made using those project management softwares.

Past few days, I have been working on adjusting events which involves around 80 hours of sessions – Org-mode helped this process such a breeze, because getting the big picture of dependencies was very easy with Org-mode.

And what best, is that reports can be generated to number of format for others to review, which is my next point:

Document Creation Made Easy

Once authoring of the document is completed on Org-mode, it is very easy to generate reports for others to consume. Be it HTML, OpenDocument, LaTeX, etc., all it involve is to export as those format.

It is even possible to create table contains dynamically generated field, much like you can do in Microsoft Excel. It’s like text based spreadsheet.

This creates seamless pipeline of information management to document creation based on it.

Shortcomings?

It is fair to say there are some shortcoming to Org-mode. For example, it is harder to embed diagrams on the document. (It’s not impossible, but the diagrams need to be created externally.) Documents involving heavy use of diagram and drawing are much better done in applications like Microsoft Powerpoint, Microsoft Visio, or LibreOffice Draw. (And please don’t do it on Microsoft Excel!) Other than that, Org-mode is very good all-round information management and document authoring tool.