Hook to process message before display/reply?

[larskotthoff]

One of my email providers inserts a warning into the body of the email if certain conditions are met. This is annoying and completely useless, and worse shows up in the quoted text when I reply.

It would be great to have a hook before astroid reads a mail that allows to pipe it through a script. I could remove the warning as if it never existed this way, without having to change mails on the server.

[gauteh]

You can use plugins to pre-process for viewing. See the syntax-highlight plugin for an example. This does not change it when including it in the reply though, but we could allow pre-processing here as well.

…

On Wed, Oct 10, 2018 at 11:15 PM Lars Kotthoff ***@***.***> wrote: One of my email providers inserts a warning into the body of the email if certain conditions are met. This is annoying and completely useless, and worse shows up in the quoted text when I reply. It would be great to have a hook before astroid reads a mail that allows to pipe it through a script. I could remove the warning as if it never existed this way, without having to change mails on the server. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#571>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AADd-6M_SqqKc0Ai35PHvwl1Vxabvt3uks5ujmN1gaJpZM4XWWSx> .

[larskotthoff]

Thanks, I'll have a look at that. It would be great if you could include something that allows this to hook in before a reply.

[larskotthoff]

It looks like if I return the plain/text part of a mail from a plugin all the newlines are gone. This happens even if I return the text part verbatim without changes:

  def do_filter_part(self, text, html, mime_type, is_patch):
    if mime_type == 'text/plain':
      return text

Is this a bug or am I missing something?

[larskotthoff]

Also, is there a way to modify the headers?

[gauteh]

Lars Kotthoff writes on October 12, 2018 0:37:

It looks like if I return the plain/text part of a mail from a plugin all the newlines are gone. This happens even if I return the text part verbatim without changes: def do_filter_part(self, text, html, mime_type, is_patch): if mime_type == 'text/plain': return text Is this a bug or am I missing something?

You have to return HTML. The html argument is the GMime filtered and escaped HTML (so it is pretty safe), this is what is usually displayed, use this if possible. The html vs text is related line by line (as far as I can remember). You can probably also access GMimes filters via python if you need to re-encode later. The documentation (in the wiki) is probably a bit sparse, feel free to elaborate as you go along :) It might be useful to provide a `filter_message` or another `filter_part` with perhaps a context argument. This could be used directly in message reading (chunk.cc / messagethread.cc) to pre-process the mail.

[larskotthoff]

Thanks, I was looking through the code and came to the same conclusion. I'm starting to think that I really want a plugin interface that allows to process the raw message.

[gauteh]

Lars Kotthoff writes on October 12, 2018 18:08:

Thanks, I was looking through the code and came to the same conclusion. I'm starting to think that I really want a plugin interface that allows to process the raw message.

Yes, the current interface is very ad-hoc for what I need. Could definetely be made much better organized and comprehensive! Contributions very welcome. - gaute

[larskotthoff]

Ok, I've had a go at implementing a plugin that returns a modified stream of the raw message (i.e. instead of astroid reading the message directly, it gets a stream from the plugin which is passed the file name). I can't get the plugin function to be called though -- the check whether there's a pointer to the function fails (pointer is NULL, https://github.com/larskotthoff/astroid/blob/master/src/plugin/astroid_activatable.c#L175). I did add it to the interface (https://github.com/larskotthoff/astroid/blob/master/src/plugin/astroid_activatable.h#L31), so I'm not sure what's going on.

It would be great if you could have a look please -- full code at https://github.com/larskotthoff/astroid, diff at larskotthoff@501384f. The python plugin has a function "do_process(self, fname)".

[gauteh]

Lars Kotthoff writes on October 14, 2018 4:44:

Ok, I've had a go at implementing a plugin that returns a modified stream of the raw message (i.e. instead of astroid reading the message directly, it gets a stream from the plugin which is passed the file name). I can't get the plugin function to be called though -- the check whether there's a pointer to the function fails (pointer is NULL, https://github.com/larskotthoff/astroid/blob/master/src/plugin/astroid_activatable.c#L175). I did add it to the interface (https://github.com/larskotthoff/astroid/blob/master/src/plugin/astroid_activatable.h#L31), so I'm not sure what's going on. It would be great if you could have a look please -- full code at https://github.com/larskotthoff/astroid, diff at larskotthoff@501384f. The python plugin has a function "do_process(self, fname)".

I'll see if I get the chance ASAP. Looked briefly through, a few things: The comment describing the function has meaning to gobject-introspection. So makes sure the GMimeStream* object is passed along as it should. Maybe a hint can be found in the the GMime introspection? Not sure, but I think this could result in mismatching function declarations. Also, make sure you set the GIR_TYPELIB path etc correctly so that the new gir libraries are found when you run astroid and plugins. See example/astroid. Also make sure that your plugin is found! This should be visible in the debug output. You can just put some print statements in the main scope of the python file and it should be printed regardless of whether the function/class is matched to the interface. Do you have a minimal test plugin I could test?

[gauteh]

https://wiki.gnome.org/Projects/GObjectIntrospection/Annotations

[larskotthoff]

Thanks, I've verified that the plugin is loaded by putting in print statements like in the example plugins. The problem is almost certainly caused by the introspection, but I can't figure out where. Below is a small example plugin to demonstrate.

import gi
gi.require_version ('Astroid', '0.2')
gi.require_version ('Gtk', '3.0')
gi.require_version ('GMime', '2.6')

from gi.repository import GObject
from gi.repository import Gtk
from gi.repository import Astroid
from gi.repository import GMime

import email
import os
import mimetypes

from email.policy import default

class FooPlugin(GObject.Object, Astroid.Activatable):
  def do_activate(self):
    print('Foo: activated', __file__)

  def do_deactivate(self):
    print('Foo: deactivated')

  def do_process(self, fname):
    print("Processing " + fname)
    with open(fname, 'rb') as fp:
      msg = email.message_from_binary_file(fp, policy = default)
      for pt in msg.walk():
        if pt.get_content_type() == 'text/plain':
            pt.set_content("Lorem ipsum")

      return GMime.StreamMem.new_with_buffer(msg.as_bytes())

[larskotthoff]

And of course I discover the problem as soon as I've posted this -- the transfer annotation needs to be "full"...

[gauteh]

Lars Kotthoff writes on October 17, 2018 4:44:

And of course I discover the problem as soon as I've posted this -- the transfer annotation needs to be "full"...

Nice, I'll take a look soon. I can imagine this will be very useful in converting HTML to text when replying or forwarding e-mail, but that would be dependent on context. Maybe a context flag should be added. The Message::viewable_text should probably be moved to ::quote or something, I think the only place it is used is in reply/forward. So then the Message-object will know at process time. Related to #532 and #545. Does the memory mangement work out correctly (ref count)? Is the GMimeStream cleared up? I assume this happens in Message::?

[larskotthoff]

Ah, good question -- it looks like there's lots of stray WebKit processes now. I'll have a look at that.

Regarding HTML to text; my plugin is adding an alternative text/plain part to the email if it's HTML-only, which I believe should work fine in all cases without a context flag. That said, there seems to be a bug either in the Python email processing library or astroid that doesn't delineate the new part correctly -- when I open an email, it shows the text and that there's an HTML part, but when I open the HTML part it is merged with the text part into one email that doesn't show separate parts anymore.