Still struggling with how I want to archive/save emails

Cool Workflows
1

This has been discussed here a number of times previously, including by me, and this is not the first time I’ve brought this up. Still, I am still struggling with this question and so I am soliciting thoughts and advice once again. I’m not sure why I am having such a hard time figuring this out…

I want to be able to essentially archive my emails, in the sense of having pretty much anything that I might need to find again someday available. This is not necessarily the same thing has having an email related to a specific task or project that I choose to store in a file in a folder of other documents related to this project, but rather a simple/quick/low effort way to archive everything before I delete it.

I don’t want to simply rely on the archive function in Apple Mail or something similar, because my goal is to create this email store outside of any current email provider I am using. The reason is that as things change and evolve, some email providers also change and I may no longer be using a particular email system a year from now. I also have multiple email addresses and providers including gmail, my own email server (hosted by an outside ISP), several accounts in outlook.com related to different businesses, etc, and I want the email to all be stored in one archive.

My top three contenders right now:

  1. Use a workflow of printing emails to PDF and saving attachments in separate files. I wrote a KM workflow that does this using Mail’s Save of PDF menu item, and also exports the entire raw email message and then parses out attachments to save in separate files (I don’t do this part in AppleScript because there are bugs in AppleScript’s ability to save mail attachments that Apple has not fixed in multiple OS versions). The files are named with a standard filename format that includes a SHA255 hash of the message to ensure uniqueness, and tagged as ‘email’, ‘has attachments’, ‘mail attachment’, etc, and Hazel can sort the emails based on which account they came from.

Advantages include local files that can be backed up, sync’d, etc, easy searching with Spotlight or HoudahSpot, easy viewing as PDFs or other documents. However, the fact that a dialog pops up for each save makes it harder and glitch-prone to save multiple messages, and so right now the KM macro works on only one selected email at a time, and of course UI scripting (even with KM’s ability to select menu items) is always glitch prone. There is no way to script having Mail write an email to a PDF.

  1. Archive to DevonThink: I keep going back and forth on whether I want to use DevonThink or not. I can import multiple emails at once and DT does a reasonable job of it, and displays and searches them reliable - and can index them in Spotlight as well so HoudahSpot will work if DT’s own search does not. On the other hand, DT is a very heavy-weight app that may be overkill, and I keep going back and forth on just keeping things in a plain file system vs an app. In addition, I have written rules in DT to have imported emails automatically sorted to the correct database (work related, personal, etc) but have found DT’s rule execution unreliable, although I have not been able to troubleshoot why.

  2. Synology mail server: I created a mail server setup on my Synology, which allows me to just drag messages in AppleMail to this account and they get stored there. This is very quick and easy, but: sorting emails into individual folders once they get there is not very good; I created Mail rules but they don’t work unless specifically executed, which is problematic, searching is now only available in Mail.app, and I need to backup the Synology data since it is not subject to the standard BackBlaze and cloning on my Macs.

  3. I looked at EagleFiler, but it’s interface just didn’t work out well for me, plus if you want to actually work with any of the emails you pretty much need to reimport them into Mail.

I would be very interested to hear what other people have done or could suggest.

3 Likes
2

If you had not already evaluated EagleFiler, it would have been my first recommendation. Being able to quickly find an email in EF, then double-click and open it in Apple Mail to forward, reply, etc. is all I need. Thanks to Apple’s increased “security” in Catalina, HoudahSpot can no longer search Apple Mail, so you are limited to Spotlight for searching. And, IMO, PDFs are useful only as reference material

I think the simplest solution would be to keep your messages in an email client. You could create a local folder on your Mac in Apple Mail, then use a rule to copy all incoming mail to the local Archive folder.

Or you could use a separate email client as your archive. Before my company started using a dedicated email archiver I did this for one of our senior executives using the Thunderbird email client. He worked remotely 99% of the time, so I set up TB on one of my utility macs to archive all his messages to the local HD.

Thunderbird has the added advantage of storing messages in standard MBOX files which can be reimported into many/most email including Outlook, Apple Mail, etc.

Either Apple Mail or Thunderbird can store a copy of your messages locally and easily copy/move them back to your preferred email provider using standard IMAP.

1 Like
3

Eagle filer sounds like something I’m looking for but when I went to the site it struck me as old. They’re still showing the old Mac OS icon.
Is the sw updated often? Anybody work with it that can share some experiences?
I searched for reviews but they’re all really old.
Thanks

4

I discovered this program called E-mail Archiver years ago. I use that and have been very happy with it. Once a month, I have it archive all my emails. It directs them to certain folders that I selected. It saves the emails in folders by year and month and it extracts all the attachments. I used the E-mail Archiver Business version which had some features that I needed over the basic version. I’ve been very happy with it.

5 Likes
5

I have been using EagleFiler since at least 2007. And, coincidentally, it received an update last week. It’s my email archiver as well as the archive for everything I want to keep permanently. It uses standard files and folders but maintains an index which allows me to find things quickly.

My iPad is my primary device and I recently switched from keeping a copy of my more recent documents in Evernote to Apple Notes (yet again). I imported the Evernote .enex files into EagleFiler, where any duplicates were removed, then copied the newer items into Apple Notes. Now everything I want to keep goes first to EF then to Notes thanks to Hazel.


Update (2020-11-05): For those using/considering eaglefiler the latest update includes a new script to fully download Apple Mail messages.
2 Likes
6

The biggest problem I’ve had with this is searching. When email is in your email application, you can search by the various fields in the database. Once you flatten that structure to a PDF or Evernote then you’ve lost your ability to easily find things.

I’d be interested to know what you decide on.

1 Like
7

PDFs seems to be a cumbersome format to deal with. It’s essentially a one-way street that’s very inconvenient to convert back - or convert to anything, really. Setting up a mail server on a NAS seems very complicated.

I’m not sure if I’m missing something or if I’m thinking too simple here but…

Why not store them as simple .EML files?
You can literally just drag mails out of Apple Mail into a folder in Finder to do this - and I’m sure that can be scripted or automated as well.

  • No proprietary database, gives you one mail, one file. And the file format is pretty much a facto standard (see link above), and basically a text file at that
  • As such, the .eml message can be searched in spotlight
  • You can use Finder tags on the individual files
  • .eml messages can be previewed with Quick Look
  • Attachments are saved inline, can be opened up from the mail file’s Quick Look preview
  • .eml can be viewed in Apple Mail. Or other Mail applications, even cross-platform
  • and they can be re-imported as mails back into Apple Mail if need be
2 Likes
8

PS: This might be due to having MailMate installed in my particular setup, which comes with some sort of bundled Spotlight Importer as I just found out. Not sure how it looks on a Mac without spotlight.

9

That’s been my choice for over 20 years of emails that I had to clean out of my Apple Mail system as it was getting impossible to use. Rather than try to sort by work or personal for my archive I just selected to put everything in year groups. And I archive nearly all messages, orders for things bought, sent mail incoming etc. I routinely go back 5-10-15 or my all time high 30 years to get old messages that I need to review or use again so the power of DT is worth it for me.

3 Likes
10

Thanks everyone. Advice much appreciated.

I certainly do like DevonThink and saving to DT is reasonably easy since they provide a mail plugin. However, as such as I want to like DT, I tend to find it more cumbersome than helpful compared to just keeping all of my “stuff” in various files accessed via Finder and searched via HoudahSpot. I find some cognitive dissonance with having some stuff in DT and some in files organized in Finder, and I just want to have one place to look for everything. (And yes, I considered indexing my whole file store in DT, but the problem with indexing is the problems with syncing across multiple computers, AND the fact that DT does not automatically reindex when new files are added AND creating a file or document in DT does not cause it to be stored in the indexed folder, so things get out of sync).

I don’t know if I want to keep using DT just for email storage…although I guess since I own a license I could.

I hear you, @AppliedMicro, regarding the portability and long term durability of various file formats. I would argue that PDF is not likely to disappear in my lifespan, although I agree some search ability is lost with PDFs vs EML files.

I have a KM macro that addresses this in the following way. When run, that KM macro saves the (single) selected message in Mail.app as an eml file. It using KM’s ability to control menu items to have mail export the message as a PDF. It runs a python script that parses the email message in the eml file and writes all attachments to individual files. The files are renamed using a prefix that contains the mail’s date and a SHA256 hash (which ensures that all of these related files have the same filename prefix, and that names are unique between different emails). The pdf is tagged as “email”, the EML file as “raw email”, the attachments as “mail attachment:” and all get a tag based on the email account the mail was received by (which Hazel can use to sort the emails to a folder system).

The problems here are many: It’s not actually that easy to create a KM macro that reliably handles multiple selected emails at once; there is the user interface popping up dialogs in the middle of everything to do the save to PDF; Scripting the user interface is often prone to breakage with new OS versions.

Maybe I should just use DT for email archiving after all…

11

Very insightful thread!

Just FYI: I’ve been using Email Archiver Pro for years and am happy with it. However, I just saw that their new version 4 is subscription. While I have nothing against subscriptions, I back up 10 email boxes for the family. Under the new subscription plan, it would be $15/month, which I find extremely steep. So I’ll be trying EagleFiler soon.

12

Seems reasonable to try EagleFiler, although it did not click for me. Possibly because as far as I could tell you needed to send the email back into a mail app to actually view it. Maybe I was doing something wrong.

I have been using DevinThink and it’s ok for my needs so far. I did get my Keyboard Maestro workflow working very well, but then decided to go with DT primarily because DT is in many ways a “better Finder” if part of your goal is to view file contents in the same window as the file listing.

I should look into some of the finder alternatives like Forklift to see if they have better inline viewing then Finder without the DT overhead.

I don’t need to sync capability of DT (although I am using it and it is excellent) as I can store everything in a SynologyDrive synced folder.

13

You can view your messages in EagleFiler. There should be a viewer window directly below the list of messages. If you open a message it will launch your email client so you can forward or reply, etc.

2 Likes
14

Thanks! I had missed that.

1 Like
16

Wow, this is such a great thread!

@nlippman What did you decide to do?

What is “KM”?

For email archive location, are people preferring storing the them in a local folder in an email client, or in a program database like DevonThink or EagleFiler, or in the Finder? Trying to sort out the pros and cons of each.

For email archive format, is there much of a difference between .eml and .txt? It looks like .eml has the advantages of being able to be viewed in mail application, reimported to mail application, and then also maybe handles attachments better. PDF seems to be a more permanent archive format since you can search pdfs but not reimport into mail program.

Thanks again to everyone for such great thoughts!

18

In this context, I think KM = Keyboard Maestro.

1 Like
19

@Qwerty:

In this context, KM does indeed refer to Keyboard Maestro, my go to tool for this kind of automation.

.eml files are .txt files, in which there is a structure to the content that allows appropriate software to interpret them as email files, similar to the way that markdown files are just text files, but they have a structure that allows them to be interpreted by appropriate software. In the case of .eml files, that structure includes headers (metadata) that address the email delivery protocol and content that is the email itself and associated attachments. Programs that can handle email will recognize the .eml extension just like programs that can interpret markdown will recognize a .md extension.

There are a lot of issues with deciding how to archive email, and my system remains in flux. For some time I have been importing email to DEVONThink and having DT use the email recipient field as the basis for organization into smart groups. As I have not been using DT much beyond that recently, I have been looking once again to make a change.

Saving email as a .eml file has the advantage of retaining all of the original information, mail headers, routing information, mail content, attachments, etc, all in one file. It is not hard to pull the attachments out of a .eml file; I have a python script that will do so and save each attachment in a separate file, for example. You can also double click an eml file and it will open in Mail.app, or drag it into Mail, etc.

One problem with saving as .eml however is that the only way to view the email in a nicely formatted manner is to open it in Mail (part of the reason to archive outside of mail is to NOT have it in the Mail program, however) or find a third party app that can display email properly. QuickView can do a reasonable job of showing an email message, but is not the easiest way to accomplish this.

Another problem with saving email as eml is that if it is a HTML-based email with active links, those links may not be available the next time you go to view the message and hence cannot be properly displayed. Printing an email to PDF will at least include an image of the link content so that will be available to you at a later date even if the web content has gone away. However, I have found that outputting an email message from Mail to PDF is very difficult and unreliable. It can only be done as far as I can tell via UI scripting (or similarly using KM to select menu items) and I have found that to be a very finicky process that never worked reliably for me, not to mention the garbage popping up on the screen while this is done - not a very nice “background” automation process. I do not have a the time to write my own program to render an email accurately and print it to a PDF. That is not a trivial process due to the complexity of email message, handling attachments, etc, and not something I could knock off in a day or two, or anything close to that.

I am now playing a bit with using a email server on my Synology and just moving emails into that server. The advantage is that Mail.app will still handle viewing, etc, but I will have archived the emails out of my “regular” email servers where I send and receive email into an area just designed as archival and which is fully under my control.

One thing thatI had wanted to do was to output eml files from Mail and have them indexed by Spotlight with very email-specific attributes, eg which of my mail accounts the email had been sent to, are there attachments, etc, and for the attachment files, pull them out of the .eml file as separate files, tagged as attachments, etc. I could accomplish a lot of this in Hazel and use Finder tags. Since I have only a couple of email accounts I won’t be creating a huge number of Finder tags making it hard to search for anything. I may wind up moving to this vs DEVONThink. I was considering using Spotlight with custom attributes, but to do so requires writing an imported to the Spotlight database. Sadly, the only developer information I can find assumes you are coding in Objective C, and I wanted to do it in Swift, so I got sidetracked trying to figure out how to make that work and never got around to writing the Spotlight importer itself…

I have played with other programs like EagleFiler, and it is a great piece of software, but it does not do exactly what I am looking for with my email, so I didn’t adopt it.

TL;DR: It remains a work in program, and I would love to hear what others are doing.

20

Thank you for the update @nlippman. Mine is still a work in progress too, although I am many steps behind you. Thanks again!

21

Am in the same boat. I actually have been using spark mail which does a decent job of moving email to pdf and saving to my synology drive. The files are searchable and this works well enough, but I’m looking for a way to get my notes and documents all together under one roof.

I’m considering using devonthink for all documents and then obsidian (with Devon think indexing this file) so I can search for anything in finder, iOS, or devonthink. With this change I’m not sure if I should keep sending pdf from spark to devonthink or switch back to Apple mail (with updates coming with ios16 it’s a little better) and using EML files then.

22

If spark mail can be scripted to save to pdf I might want to try it out. It’s probably worth waiting to see how Mail looks in Ventura though.