Automation request - Is this even possible?

I’m playing around with some ideas and not even sure if it’s going to be possible so wanted to run it by folks here for a sanity check… (please, no remarks about my sanity :wink: )

I process email on a single machine and use POP accounts. There are MANY of them neatly organized by organization or role etc. I generally get between 200-300 email messages a day. Of those roughly 100 are immediately useful and of that bunch maybe 20 are relevant to projects I am working on and thus need to be able to be found in my project support material. That is all now in Obsidian. Each project has a top note that acts as the TOC for stuff or in the case of most project is all I need to manage the tasks etc for it.

I can drag a message to the appropriate part of my top project note in Obsidian and I get a link. I can easily click on that and see the email as long as I am at the computer with my email. But obviously with POP mail that will not work when I look at that project on a different device or on mobile.

What I want is some way to use a combination of shortcuts, keyboard maestro, hazel and who knows what else to do the following for a message I need in my Obsidian vault as the complete message.

As I am reading a message in Apple mail I want a simple key combination or hot key that saves the message with a filename that includes the date sent, an underscore, the time sent, underscore and the subject using the plain text save as out of mail. Then I want it converted to a .md file by adding a new email note template, then the message text as the body of the note and moves it to my Obsidian inbox folder. Extra points if I can get the from, to and subject into Obsidian properyies by filling in my email note from from the text and also add in the link to the message in my mail system as well as part of the new Obsidian note.

I understand that from there in Obsidian I will have to manually link it appropriately within the correct project note but at least I’ll get the data in there.

Is this even going to be possible?

Right now the only way I can do it is to save the message as plain text, open it in text editor, copy the contents. Then in Obsidian crate a new note, insert my email template file, paste the text from the message in the place for it in my template. Manually copy and move the appropriate data into the YAML of the new note. Go back to email and then drag the message to the link area of the Obsidian note.

This works but is painfully slow.

I want a one button hotkey that does all that.

Any suggestions?

The short answer, I think, is that it’d be possible via AppleScript — though perhaps tedious!

Proof of concept:

tell application "Mail"
	set theMessage to the first item of (get selection)
	set theSender to theMessage's sender
	set theSubject to theMessage's subject
	set messageBody to theMessage's content
	set messageURL to "message://%3c" & theMessage's message id & "%3e"
	
	set noteBody to "---" & return & "sender: " & theSender & return & "subject: " & theSubject & return & "link: " & messageURL & return & "---" & return & return & messageBody
	
	display dialog noteBody
end tell

Invoking that looks like this …

Adding template text to noteBody and saving it into a .md file in a specific folder shouldn’t be too hard.

(I don’t usually work with Apple Mail so this is a rough, untested work, heh.)

3 Likes

First of all, 200-300 emails a day!? :astonished:

Secondly, how comes with these amounts of e-mail and this workflow that you’re still using POP?

And finally, this sounds like a problem that can be easily fixed by using IMAP. You’re now looking for a clumsy way to solve a potentially simple problem.

Which doesn’t help you much of course.

Yep on average about 300 messages a day. I am an officer in 2 organizations, a BOD member in another one, am running a farm, doing a major SW development project with 2 other out of town collaborators and handling all customer support for our deployed systems. About 100/day are spam. Easily sorted out by Spam Sieve and I can review the spam every couple of days before I delete it. Reason is that I average 2-3 messages a week that are really impoprtant but get flagged as spam and I can’t whitelist the senders because they are all new people I don’t know and haven’t corresponded with.

One reason for POP is that I keep and archive important messages. I have archives going back over 20 years and yes I DO reference and use those archives on a regular basis. I’ve already hit limits and had to dump out tens of thousands of saved messages off of Apple mail into other options because Apple mail can’t handle that level of messages in an archive and I’d have run off even the most generous reasonably priced IMAP email service years ago.

Another reason for sticking with POP is that it forces me to not be continually chasing email and responding to latest and loudest. By relegating it to a single machine I am not tempted to do it all the time and that helps a lot. As I’ve said before, I don’t handle switching between apps at all well. So I much rather stay in one app and switch projects when I am working. I am far more productive batching my emails. I also know myself, and if I had email on a mobile device I’d use it and that way is the path to frustration because not everything I need to do with an email that comes in is available on my mobile devices. I don’t want to process emails twice, once on a mobile device and then again when I finally get back to my main machine where the rest of my tools are.

I don’t see how switching to IMAP solves the need to keep the emails in with my project support material. All it does is make the links to the entire message more easily read. But I don’t just link to entire messages. I need and want to link to paragraphs or sections within a message. A single important email may have pieces that need to get referred to several different projects. I loathe subprojects and so I have a wider fatter project set-up than a lot of people. I don’t want to have the entire message in each separate project page (which is what a link to the message in an IMAP account would in effect provide me) because for some of them trying to read down to the snippet that matters for this particular project is a trial. I want to link direct to that tiny piece in my note.

1 Like

I do the same with IMAP. It depends on your e-mail provider and the plan you have how many messages they can store for you. With these amounts it all adds up very quickly. To be honest, although these aren’t by far enterprise level amounts, one can hardly call these numbers regular. Let’s assume that on average you get 250 emails per day. And only during working days. That ends up to 65,000 (!) emails a year. It’s quite an achievement that you still have time to do anything else but processing email. :wink:

The reason I’m referring to IMAP is that it has an UID that stays the same everywhere. That 's the beauty of server based (or web based) computing. To be honest, I feel like you try to stick to old technology while this problem can more easily be fixed with current (well, IMAP, cloud ,etc. aren’t really that new…) technology.

It’s this part (emphasis mine)

that triggered my IMAP response.

1 Like

It is easy when subscribed to multiple professional/technical mailing lists.

Manage my 200-300 using filtering rules in Apple Mail as most of those techie mail messages are for reference rather than action.

Also possible to get 200–300 Discourse posts a day for a similar reason being signed into multiple installs for professional/technical purposes.

1 Like

For me I have to have all my achive emails in one place. I already have a set of them that is about 90K messages going back over 30 years. Plus, Apple Mail dies handling anything over about 10K messages. So again, it doesn’t fit the system requirements I outlined.

First off I don’t have non-working days. And it’s easy to getthat with a bunch that are ephemeral. Mailing lists I only occasionally need the info from etc. I also ahve over 100 mail rules to properly file those most of the time irrelevant messages.

But that also presumes that when you need the info you will have access to the internet. That is not at all possible for me. I can go 5 minutes away and be in a place with no internet access at all, not even cell phone coverage. Yet I can be doing work there that needs access to the info in my emails. Working on a ranch with some AnimalTracker stuff. I can go 30 minutes away and be in a place with no GPS coverage. It’s actually where a working corral is so also, I need my emails there and it has to be on my device NOT on some server elsewhere.

That’s why I need to get them into Obsidian intact, as I have that on all devices locally.

:rant on: I am REALLY tired of solutions that assume that internet is both ubiquitious and fast! A huge portion of the US has zero cell phone or internet covereage and even poor Starlink coverage. Most rural areas have slow internet if at all. I am lucky in that at the main house and farm HQ we have fiber to the house (and barns but that is another story) That is VERY unusual for most of the western US. So when I am opposed to cloud computing for critical and basic functions and why I stick to old technologies and seek solutions that fit the required parameters it’s not because I don’t know and understand the benefits of newer options it’s because they… DO… NOT… WORK… for me! :rant off:

1 Like

As a programmer, I had a hard time remembering that most of our users did not have a computer that worked as fast or as well as the developer-class device on my desk. :upside_down_face:

1 Like

Rant received and understood.

But as long as the US government defines “High Speed Internet” as at least 25 Mbps and actual upload speeds of at least 3 Mbps and allows big ISPs to maintain legal monopolies there is very little incentive for anything to change. And that’s a shame because most non power users are not able or willing to keep their devices and data secure.

I doubt if anyone has the exact figures but most sources claim that at least 90% of businesses use some cloud services and at least 66% of all business data now resides in the cloud. That’s one reason companies are continuing to move to web based solutions rather than native apps. Another is because it avoids having to modify their programs every time Apple or Microsoft, etc. changes their OS.

I congratulate you for coming up with solutions on your side of the “digital divide”. I hope one of the satellite broadband companies comes up with a fix because I don’t see Comcast, etc. lifting a finger to help.

That amazes me! What do you use to archive your email? A special archiving app, Obsidian, or…?

Right now most of them are stored in my one remaining DEVONTHink database. I also have them out with each year as a Thunderbird mbox File but that’s not as easy to search. Some are in text files that are harder to parse out (Old Compuserve ones for example) but there aren’t too many of those. I would MUCH prefer they are all in Obsidian, probably in a separate vault. But I need to get all the metadata included as properties. First task is dealing with the ones I need for sure in Obsidian as project support material.

As I said I’m lucky in that our local internet provider is our rural electric co-op. They needed to run fiber to meters to do real time load balancing. So they formed a wholy owned subsidiary that provides internet on a fiber backbone to over 95% of the electric customers in the area. The remaing 5% are being installed as I type. It’s been a multi year rollout. I just ran speed test and right now I’m getting 365Mbps download and 362Mpbs upload speeds. It’s Saturday, lots of folks are streaming stuff right now. I pay $79.99/month for “ONT UN-BRIDGED DATA SERVICE (Qty: 1) RESIDENTIAL INTERNET 1 GIG (1000 MBPS) WI-FI BASIC (Qty: 1)” So internet access and speed is not an issue when I am at home in the house or out in the barn on the main property. But go anywhere “off grid” as in not within a wifi connection of a local electric meter I can pretty much count on no cell phone or internet. Our cell phone tower is the last one in a string up the valley. Forget using 2 or more to triangulate position or improve signal here. It’s actually LOS from our house. It’s a T-Mobile (formerly Sprint) service, ATT also has some equip on the same tower. The nearest Verizon tower is about 10 miles away around the edge of a hill. Go to our other property, Juniper Gulch Ranch and there is no cell phone service in most places. That’s about 5 minutes away from our house.

1 Like

Depends from which server I pick the messages up from. Could be inbox format (i.e. plain text) or left on the server using IMAP4 with folders to keep the archived messages in.

I wish mine was. I pay Comcast $92/month for 400Mbps down, 12 Mbps up. If I lived in Chattanooga, TN I could get 1000 Mbps down & up for $68/month. (Their power company also has 10,000 Mbps down & up for $300/month)

I was in your part of the country many years ago and the installation of a new cell tower was apparently big news. Too bad cell service is still a problem 20+ years later. Being able to connect when away from home is the only way my setup will work. When I’m not connected to the cloud I’m backing up to SSDs.

vco1 wasn’t saying to stay connected to the IMAP server; he was saying that your offline copy of each email on each device can be consistently referenced.

Anyway, I like Ryan’s AppleScript approach, but if I wanted to handle all pop3 email in Obsidian, I might try a little Python pop3 server and Obsidian instance (or a sync folder) on a headless Linux server. poplib and email to download and process the email, markdownify to generate the markdown content, and then save each file in the right locations. Wouldn’t be more than 50 lines.

You’d have to do some processing in Obsidian instead of your usual Mail client, but you could add filters over time for certain senders, keywords, etc.

I’m still hoping someone will make a dead simple cross-platform, multi-protocol email client that’s just a sqlite database…

1 Like

That’s not what I read at all. But let’s say that you are correct. There is still the problem that Apple Mail does not handle the number of messages I keep. I would also need to have all messages on my devices not just the only ones I need in my project support material. IMAP is NOT a solution to the complete problem. It only solves a small portion of it and only until I have to remove emails from my server and out of Apple Mail because I’ve yet again exceeded the numbers that the mail app can handle. Not to mention that the problem is not just refering to a single entire message, but to referring to paragraphs within that message. That’s why the messages need to end up in Obsidian.

But I don’t want to handle all POP mail in Obsidian. Just the ones I need as project support material. Now for the long term archive of messages going back decades that is a separate problem. Since most of them are already downloaded I have to deal with that outside the scope of this question.

fix most of the typos I think

Hey, fair enough. As for archiving, you could use that pop3 client approach to only fetch, archive and delete certain emails (from certain senders, subject lines, etc.) You could list and retrieve them and manually flag which ones to delete (a pop3 server deletes everything flagged at the end of the connection and leaves the rest…) Over time you’d build up which project-related emails it automatically moves into Obsidian for you, rather than pare back its scope.