Emacs The Ultimate: Mail reader - Setting up notmuch with emacs

I’ve been using the Emacs editor for the last six years and haven’t regreted the choice for a moment. Emacs is truly the programmers editor since it allows you to easily extend it and mold it to your own liking. It is also very versatile editor with modes and programs for most anything that you would want to do, this is powerful since it allows one to not switch to another program for most tasks.

For example there is no need to switch between emacs and the shell since I can just have a shell open in an emacs buffer!

One of the last holdouts of things I hadn’t integrated in my emacs workflow was email. I started looking at Gnus but that didn’t really appeal to me so I left it there for a while and continued using Thunderbird as my email client. One of the main issues that I have been having with thunderbird (besides it not being emacs..) is that working with mailing-lists have been a pain.

So last week I put a side a couple of hours and got my email into emacs! There are three parts to this.

  - Getting the emails from my server
  - Reading/Writing emails
  - Sending emails

Something like thunderbird takes care of all these points and Gnus does to. But I went with using notmuch (https://notmuchmail.org/) and that pretty much only does efficient search and tagging, but there is an emacs package for notmuch that allows you to read and interact with your mail archive. It also means we have to do point 1 and 3 some other way!

Getting email

To fetch my emails from my server I went with the old standard fetchmail program. This is super easy to install and setup. I just added a .fetchmailrc in my home directory with the server and login information and it was ready to fetch some mail. One issue with fetchmail is that it can’t keep the emails on the server for n days and then delete them and since I for now also get emails directly to my phone this is somewhat of a problem.

The fetchmail configuration file looks like this:

poll <host>
protocol pop3
timeout 600
port <port>
username <username> password <password>
keep
ssl
mda "/usr/bin/procmail - '<location of procmail configuration>'"

Procmail is the mail delivery agent which puts the fetched emails in the correct directory.

My procmail configuration looks like

LOGFILE=$HOME/.procmail.log
MAILDIR=$HOME
VERBOSE=on

:0
Maildir/

We then need to get the emails into notmuch. This can just be done with notmuch new. I then use a program called afew (https://github.com/teythoon/afew) that does initial tagging of emails. For example it tags each different mailing-list with the correct name of the list. My afew config looks like:

[SpamFilter]
[KillThreadsFilter]
[ListMailsFilter]
[SentMailsFilter]
sent_tag = sent
[ArchiveSentMailsFilter]

[Filter.1]
message = "Get mailing lists out"
query = 'tag:lists'
tags = +unread;-new;

[InboxFilter]

And I run afew with afew -tn.

I then run notmuch tag -inbox tag:inbox AND tag:lists. This removes the inbox tag from mails that come from a mailing-list.

I’ve put all of these commands in a cronjob that runs once every hour. My file looks like:

fetchmail
notmuch new
afew -tn
notmuch tag -inbox tag:inbox AND tag:lists
exit 0

Reading/Writing Emails

As I mentioned above there is the notmuch package for emacs which allows you to interact with notmuch. To retrieve emails from notmuch one searches for the desired emails/tags and notmuch will show the matched emails to you. You can also set default searches with keyboard shortcuts to get to common searches fast! For example hitting j i in the notmuch buffer will get all the emails with the inbox tag.

Sending emails

Emacs has the built in message mode that can be reached by hitting m in the notmuch buffer or r on an email to reply to that specific email. When you have finished writing your email you need to send it to the server that can relay the message to the recipient. Emacs as usual has some builtin functionality to do this with sendmail.el but for flexibility I wanted to use an external program. I chose to use msmtp for its simplicity. Again you just need to install msmtp, add the correct server configurations and tell emacs to use msmtp to send email with.

The configuration of msmtp is similar to fetchmail.

# Set default values for all following accounts
defaults
port <port>

# TLS should always be used
tls on
account example
host smtp.example.org

from tester@example.org
auth on
user tester
password best-password

account default : example

We then need to tell emacs about msmtp and configure it to set the correct headers. In my emacs configuration file I set these variables:

(setq mail-host-address "example.org")
(setq user-full-name "Tester Testington")
(setq user-mail-adress "tester@example.org")
(setq mail-user-agent 'message-user-agent)
(setq message-send-mail-function 'message-send-mail-with-sendmail)
(setq message-kill-buffer-on-exit t)
(setq mail-specify-envelope-from t)
(setq sendmail-program "/usr/bin/msmtp"
	  mail-specify-envelope-from t
	  mail-envelope-from 'header
	  message-sendmail-envelope-from 'header)

It is important to set the mail-host-address otherwise emacs will set the Message-Id header to something like “Message-ID: 8737m2q8na.fsf@localhost.i-did-not-set--mail-host-address--so-tickle-me”

And that is it, I just put all of the shell commands in a file and use cron to get and ingest mail every hour!

I’m now be comforted by the fact that I don’t have to leave emacs to read and write emails ever again!

Declarative OpenGL (sort of) in D

Normally when writing OpenGL code, it can get very cumbersome/error-prone when defining how to pass vertex data to the GPU. As a given example, heres a toy vertex structure we want to tell OpenGL how to handle when it gets passed in as an array of data.

struct Vertex3f2f {
    Vec3f pos;
    Vec2f uv;
}

With this, the following needs to be done to upload the data to the GPU with OpenGL (and it needs to change if the vertex structure changes, where making these changes by hand every time can easily lead to mysterious buggery)

//some excellent array of vertices
Vertex3f2f vertices = [...];

//storage for vao and vbo
GLuint vao, vbo;

glGenVertexArrays(1, &vao);
glBindVertexArray(vao);

//create 1 buffer
glGenBuffers(1, &vbo);
glBindBuffer(GL_ARRAY_BUFFER, vbo);

// upload to GPU, send size in bytes and pointer to array, 
//  also tell GPU it will never be modified.
glBufferData(
  GL_ARRAY_BUFFER, 
  vertices.length * vertices[0].sizeof, 
  vertices.ptr, 
  GL_STATIC_DRAW
);

glEnableVertexAttribArray(0);
glVertexAttribPointer(
  0, // attrib index
  3, // number of elements (3 floats, Vec3f)
  GL_FLOAT, // element type
  GL_FALSE, // can normalize elements to range [-1, 1] for signed, [0, 1] for unsigned
  vertices[0].sizeof, // stride for given attrib
  cast(const(void)*)null // offset of the first attribute
);

glEnableVertexAttribArray(1);
glVertexAttribPointer(
  1, 
  2, // Vec2f.. 
  GL_FLOAT,
  GL_FALSE, 
  vertices[0].sizeof, 
  cast(const(void)*)vertices[0].pos.sizeof // initial offset is size of pos
);

//... potentially some other code

In the process of enduring this boilerplate I set out to create a way to generate OpenGL code from a “vertex specification”. … Which is just a convoluted way of describing exactly that Vertex3f2f struct thing up there.

Now that we’re past hinting why this might be a desirable thing, lets put it into action and describe the rest of the process!

Prelude to Code Generation

Given our type definition and D’s metaprogramming capabilities, we can actually iterate over all members in the struct we pass at compile-time and filter so we only get POD members.

template PODMembers(T) {

  import std.meta : Filter;
  import std.traits : FieldNameTuple;

  template isFieldPOD(string field) {
    enum isFieldPOD = __traits(isPOD,
      typeof(__traits(getMember, T, field)));
  }

  alias PODMembers = Filter!(isFieldPOD, FieldNameTuple!T);

} //PODMembers

//can be used like
import std.meta : AliasSeq;
alias Types = AliasSeq!(int, float, double, Object);
alias PodTypes = PODMembers!Types;
//should give us (int, float, double)

Now, except for those mysterious __traits__ instrinsics (explained here), it should be fairly straightforward for anyone familiar with filter and map primitives in most FP languages.

FieldNameTuple gives us a tuple of all the field names (as strings) in a given type, which we us as an input to our Filter computation. Filter here expects a predicate of the form:

template SomePredicate(Thing) {
  enum SomePredicate = SomethingWhichReturnsABoolean;
}

We also need to define a function to convert the given primitive member of each vector in the vertex structs to the given OpenGL equivalent, int -> GL_INT for example.

Lets define a template which basically is just a big switch on the primitive types.

template TypeToGLenum(T) {

  import std.format : format;

  static if (is (T == float)) {
      enum TypeToGLenum = GL_FLOAT;
  } else static if (is (T == double)) {
      enum TypeToGLenum = GL_DOUBLE;
  } else static if (is (T == int)) {
      enum TypeToGLenum = GL_INT;
  } else static if (is (T == uint)) {
      enum TypeToGLenum = GL_UNSIGNED_INT;
  } else static if (is (T == short)) {
      enum TypeToGLenum = GL_SHORT;
  } else static if (is (T == ushort)) {
      enum TypeToGLenum = GL_UNSIGNED_SHORT;
  } else static if (is (T == byte)) {
  	enum TypeToGLenum = GL_BYTE;
  } else static if (is (T == ubyte) || is(T == void)) {
        enum TypeToGLenum = GL_UNSIGNED_BYTE;
  } else {
      static assert (0, 
          format("No type conversion found for: %s to GL equivalent", 
                  T.stringof));
  }

} //TypeToGLenum

Actual Code Generation

Given the previous definitions, lets make a simple function which will accept a vao and vbo, as well as an array of vertices and in the process, generate the code to upload the data for the given vertex type to the GPU.

/* templated on VType
  would look something like this in c++:

    template <typename VType>
    void uploadVertices(...etc)

*/
void uploadVertices(VType)(ref GLuint vao, ref GLuint vbo, VType[] vertices) {

  glGenVertexArrays(1, &vao);
  glBindVertexArray(vao);

  glGenBuffers(1, &vbo);
  glBindBuffer(GL_ARRAY_BUFFER, vbo); 

  //assumes GL_STATIC_DRAW here, simplistic example
  glBufferData(GL_ARRAY_BUFFER, 
    vertices.length * vertices[0].sizeof, vertices.ptr, GL_STATIC_DRAW);

  //static foreach, is executed *at compile time and produces 
  // the block of code inside for each iteration*
  foreach (i, m; PODMembers!VType) {

    alias MemberType = typeof(__traits(getMember, VType, m));
    enum MemberOffset = __traits(getMember, VType, m).offsetof;
    //note that types need to provide a _T to access type of members 
    // for example with Vec3f, T_ = float
    alias ElementType = MemberType._T;

    glEnableVertexAttribArray(i);
    glVertexAttribPointer(i
      //computes num of elems by sizeof(Vec3f) / sizeof(Vec3f._T)
      MemberType.sizeof / ElementType.sizeof,
      TypeToGLenum!ElementType, //GL_FLOAT if ElementType is float
      GL_FALSE, //TODO: handle normalization
      vertices[0].sizeof, //stride as mentioned earlier, is size of struct
      cast(const(void)*)MemberOffset //offset for given member in bytes
    );

  }

}

// so if you wanted to upload data for a rectangle
float w = 1.0f;
float h = 1.0f;

Vertex3f2f[6] vertices = [
    Vertex3f2f(Vec3f(0.0f, 0.0f, 0.0f), Vec2f(0.0f, 0.0f)), // top left
    Vertex3f2f(Vec3f(w, 0.0f, 0.0f), Vec2f(1.0f, 0.0f)), // top right
    Vertex3f2f(Vec3f(w, h, 0.0f), Vec2f(1.0f, 1.0f)), // bottom right

    Vertex3f2f(Vec3f(0.0f, 0.0f, 0.0f), Vec2f(0.0f, 0.0f)), // top left
    Vertex3f2f(Vec3f(0.0f, h, 0.0f), Vec2f(0.0f, 1.0f)), // bottom left
    Vertex3f2f(Vec3f(w, h, 0.0f), Vec2f(1.0f, 1.0f)) // bottom right
];

GLuint vao, vbo; //wee
uploadVertices(vao, vbo, vertices[]);

The most mystifying thing up there might be the foreach, when a foreach is operating on a tuple of types it executes at compile time and essentially acts as a sort of rudimentary code generator, producing the block inside the foreach for each iteration. (an example from p0nce’s excellent d-idioms)

So given the uploadVertices definition, it should be 100 % equivalent to what we wrote at the very beginning of the post, except now it works with an arbitrary vertex definition, assuming it defines a _T to get the element type.

Given each different instantiation of the function, another version of this function will be generated, so you might have one for the Vertex3f2f and another for a variant which simply needs to upload a Vertex3f vertex buffer, or… you get the idea.

Now this is a rather simplistic example, which doesn’t handle normalization of attributes, or any kind of instancing related attributes, but already here we have something we can build on! Integrating index buffers might be an interesting idea, but it is out of scope for this quick introduction.

I hope this showed you a little bit of just how useful D’s metaprogramming capabilities can be, and that it wasn’t too big of a mess to read through.

Today I Learned About Locality Sensitive Hashing

I’ve been planning on writing a longer series of post about hash functions to learn more about how they work and what goes into designing good hash functions.

During my research I stumbled upon something called Locality Sensitive Hashing. If you’ve used a hash function before or know something about them, you probably know that changing the input to a hash function by just a little bit produces a wildly different result. Take for example the sha256 hash function with the string: “Hash functions”.

The output of this is: 5a2dba974930153b12b69e46023e73bd09dcdc8216fe2960a991791389f4519c

But if we just change one letter so that we have the string “Hash functjons” we get the output: 457321e76142b2fcb0147ceb8233f4efc7f1d0154c1b1aa48778f905f7de7bc0

The idea behind a locality sensitive hash function is that data that are very similar would hash to similar outputs. This would mean that the example above would with high probability hash to two close hash digests. There are several applications where using hash functions with this property are useful such as malware fingerprinting and similarity between documents.

To read more about this quite interesting subject I would recommend both the wikipedia page: https://en.wikipedia.org/wiki/Locality-sensitive_hashing and this section from Mining of Massive Datasets: http://infolab.stanford.edu/~ullman/mmds/ch3a.pdf

Post 0

Since my esteemed friend profan posted a small intro post I thought that I would do the same.

Welcome to our new shiny blog!

I’m Eric and together with profan (and maybe more people to come in the future) will write in this space. I’m a former computer science science student now working programmer. I’ll be mostly writing about computer science and programming topics interspersed by occasional other topics such as chess and art.

I have some ideas on what I want to do with this blog such as small Today I Learned posts and longer series which I’m thinking about calling This Year I Learned which will be long running posts about a single topic that I want to learn deeper about.

Hopefully it will be useful for some people, at least I will probably be more motivated to learn and share!

Hello, World!

Welcome!

This newly instated programming and general computer science related blog will our little corner of the blogosphere where we post our assorted ramblings and adventures in convoluted solutions.

We all share an interest in computer science and will hopefully produce something interesting along that note.

Subscribe to our atom feed and get all the nonsense right when it comes out!