Which Do You Consider “Safer” for Research Files: DEVONthink, Finder, or Obsidian?

Software
1

Let me set the context:

  1. I perform two separate weekly backups of all of my data (all iCloud files are downloaded to the Mac so I can back everything up). One external drive is at home, the other is at the office.
  2. I recognize that anything in Obsidian is in the Finder.
  3. I use, and will continue to use, DT as a powerful utility, e.g., file conversion, OCR, etc.
  4. IF I continue to use DT for my research files, they will continue to be imported, not indexed.

Perhaps it is unwarranted, but I have a low grade nagging worry that databases are more prone to corruption than having individual files on my hard drive because if a database is corrupted, I could potentially lose access to all of my files wheras files on the Mac can only become corrupted one at a time, assuming there is no castrophic harddrive failure.

This worry is related to two apps I use: Ulysses and DEVONthink. I’m less worred about Ulysses because it has its own robust backup system, to wit:

How often is my library backed up?
On macOS, Ulysses keeps hourly backups for the last 12 hours, daily backups for the last seven days and weekly backups for the last six months of work.
On iOS, Ulysses keeps hourly backups for the last 6 hours, daily backups for the last 5 days and weekly backups for the last two months of work.

Does Ulysses create backups of all sections?
Backups are complete, stand-alone copies of your library — except for the External Folders section. We strongly advise you to create backups yourself if you work with External Folders in Ulysses. To ensure the safety of your work, you could create a Time Machine backup or use another 3rd party tool to back up your work.

Based on the above, what is your expert advice? Is it safer to store research files as individual files on the Mac or in a database like DT?

2

Well, with DEVONthink the documents are in individual files. DEVONthink does not use something like Oracle, SQL Server, SQL light where your files are stored in that technology. All your documents are recoverable even with no DEVONthink. And as you know there are numerous checks that DEVONthink does to check on and actively protect the integrity of its databases automatically along with built in archiving tools in addition to system backups. And I believe importing is “safer” than indexing with the risks of importing explained in the DEVONthink Manual.

Whatever you pick, control these risks you now worry about with a robust and tested backup process.

Pick what supports your work best and allows you to produce outstanding work products most effectively.

[I think you are over-thinking it]

5 Likes
3

I did not realize there was a difference between how DT handles files and software like Oracle, etc. I also did not realize I could access my files without DT. That relieves a lot of my concern, thanks!

Ok, no more thinking …. :grin:

4

If the worst happens, try yourself. do not rely on a random like me. Look inside the DEVONthink macOS package for your files. they will be in folders with “funny” names which DEVONthink manages, but your files are there none the less.

My best suggestion is you focus on picking the tool which works best for you.

2 Likes
5

I agree not to worry. Ulysses has the right idea by giving regular backups far back. If what you’re storing is important, keep monthly backups of the critical data for the year and yearly backups going back forever.

That’s my recommendation. What follows is just additional discussion:

You can run misconfigured smart rules and transforms in DT and you only have a few weeks to realize what you’ve done or automated if you don’t have backups going back aways.

In this respect, Finder is safer because it doesn’t give you much power. So besides the backups, you just have to ask if it’s more important to preserve what you have, or to improve the state of your data by working on it and incurring some risk by touching it. In practice, you’re getting both benefits, of course.

1 Like
6

Thanks for the advice; I like the backup suggestion; I’ll modify mine to mirror my current weekly backup regimen. Thanks again!

7

DEVONthink 3 stores the files individually for want of a better word. Other posters have explained it better than me.
I have never lost a file using DEVONthink 3 and in fact have never needed to use my backups, touch wood as we say. I back up the DEVONthink 3 databases as well as my 3-2-1 Mac general back up system.
I have, in 15 years using Macs ever gone back to a ‘previous state’ for any file system I ever used. I do have some old DEVONthink 3 databases on a thumb drive somewhere.

4 Likes
8

Thank you, this is helpful and encouraging.

1 Like
9

It’s important to note that a DT “database” is exactly that: a bunch of regular files along with DT managed metadata (basically indexes, aliases, replicants, tags, group information). That comprises the database.

Yes, your files are there. But if you run “Show package contents” on a DT database file, you will find that the folder structure under which those files are organized does not resemble at all your folder structure. So in the case of a catastrophic crash, certainly your files will be there but your folder structure would not necessarily survive because now it’s part of DT’s “metadata” indexes (the .dtMeta files insides the DT database).

Where was the WINVER._ file located?

Screenshot 2023-05-31 at 08.41.42
Screenshot 2023-05-31 at 08.41.42770×220 13.6 KB

Answer: it was at /cdrom/diskettes/WIN31_5/WINVER._. (This is using DT as a digital vault, which is not exactly a recommended use case, but it works as an example, by the way)

The only way to recover your original folder structure, the one that shows inside DT, would be opening the database with DT to drag and drop the files & folders to your desktop or another Finder window. Granted, I’ve never seen DEVONthink crash in such a way that a database (its metadata index, that is) becomes corrupted and cannot be opened again, but it’s something to consider. Another remote concern would be DEVON Technologies going out of business so in, say, 25 years the app would not be there for macOS Sant Gorgonio.

As a somewhat pedantic note, the approach DT follows to organize files in this way is a very common traditional system used to optimize access to a huge amount of files in a way that is friendly to the underlying disk filesystem but I believe that with newer Apple filesystems the benefits are marginal at best. For example, Obsidian does not need to use anything else than your folder structure in order to have quick access to everything (and of course Obsidian also has its vault indexing database, but it is kept separate and the files and the folder structure are untouched). It could also possibly be that some of DT’s extra magical capabilities like replicants depends on this folder structure, but I’m not sure here.

My final piece of advice: if DEVONthink works for you today, do not switch because of this particular quirk, just adapt your backup strategy accordingly. We are not banks, we do not need to base our decisions on the risk of a sequence of very remote bad events happening.

7 Likes
10

That is also helpful. I’m learning a lot about DT databases! Much appreciated!

11

Thanks, everyone, for the helpful insights and suggestions. I realize that I tend to be paranoid when protecting my data, but I have several decades of articles and research files that are important, and I never want to lose them. Stuff happens so I want to do all I can to protect my data.

After reading through the responses to my post, here is what I’ve decided, which maintains my workflow and app choices (no changes) while adding a layer of additional protection beyond my weekly backups to external drives.

  1. Each December I will export my Ulysses files to an Obsidian vault using the external folder feature in Ulysses. I will also export these duplicate files to my DEVONthink archive database, which also has my Apple Note exports.

  2. Every six months, I will upload my DEVONthink research and archive database to my Google Drive and export the duplicate research and archive files in the DT database to my Obsidian vault.

This approach will give me the benefit of using the great features of Ulysses and DEVONthink while ensuring that my data and work is backed up and safely warehoused in other locations should I experience a catastrophic computer failure or file corruption.

Thanks again. I deeply appreciate the expert and thoughtful advice. I only wish I had the same expertise to share with others!

1 Like
12

If you have access to Keyboard Maestro or similar, you could automate these app-to-app backups so that they don’t have to have such a long cadence.

Pseudocode would be something like:

Every day at 4am

  • Activate Ulysses
  • Click the “Export” option in the menubar (or invoke the “Export” keyboard shortcut
  • Export the files to your backup folder in a folder named [today’s date]
  • Delete the old backup folder

Exact implementation depends on a bunch of little things of course.

1 Like
13

Thanks for the idea, I like it. I have Keyboard Maestro. Would that process be difficult to create? As you know, I’m no programmer, I can’t even make a decent shortcut! :slightly_smiling_face:

14

I am a computer programmer. I cannot do a decent Shortcut either.

4 Likes
15

That made me smile and made me feel better! :grin:

2 Likes
16

No mention of routine and automatic full system backups using the 3-2-1 “standard”. Pity. That’s the most important thing to do, IMHO. “Weekly” backups sufficient? Not for me.

For DEVONthink, read “A Word about Backups”, page 19, of the DEVONthink Manual (version 3.9) and best to conform to their recommendations. I don’t use Obsidian so have no comment.

What you are planning to do is too sporadic and manual for my taste.

Nor would I count on any archives/backups that go into “synced” services (e.g. Google Drive) as being safe. Flaw in one place immediately gets synched to all sync locations.

1 Like
17

Will you be uploading the .dtBase2 file or an export generated by “File > Export > Files and folders” ? The second option would re-create your folder structure outside DEVONthink just like you see inside the app so I would recommend that to be totally futureproof. Perhaps there are other DEVON tricks to do this in a better way.

Also, check your backups and beware of indexed files, perhaps they will not be part of the export!

1 Like
18

It depends on a variety of things. I’d block off a half-hour and fiddle to see if you can do it easily. If it doesn’t work out, then maybe it isn’t easy.

It’s not the kind of thing that someone can easily set up for you because there are at least a few decisions to make along the way. And after all …

Give a person an automation and they will automate for a day. Teach a person to make their own automations and they will procrastinate for a lifetime.

1 Like
19

Wait until I tell you that my backup strategy is to rsync my Finder folders every year to an external SSD drive and be done with it. :smiley:

And no, I’m not flexing my brains or mocking you or would recommend this strategy to anyone who deeply cares about not losing anything. It’s just that, having everything on iCloud Drive, I feel I don’t need the overhead of a full backup strategy myself and my only concern are my own fuckups --for example, when trying shiny new apps–, so I am living on the edge of losing some of my stuff. Yes, being locked out of my Apple account would probably mean disaster.

20

Noting that @OogieM had some significant problems with DEVONthink and data loss. I’ll let her pop in and explain if she so desires, but the short version is that ANY APP that’s syncing your data carries at least some risk of data corruption/file loss.

If you’re just popping data into DT/Obsidian/whatever and there’s no sync involved (either in the app or via iCloud Drive/Dropbox), the risk of data loss due to the app itself is vanishingly small, and is probably equal across applications.

1 Like