DEVONthink ‚ should I buy it?

[nationalinterest]

I use EagleFiler for document storage, but acknowledge it has a subset of DT functionality. Plenty for me, and the search is powerful enough. There’s no mobile app, but files can be freely viewed/amended on iOS via iCloud/Dropbox/OneDrive. However, you may find functionality you consider essential I DT. As others suggest, the DT manuals are superb and the trial long.

I did use DT but had various database issues. I never found out why, but life’s too short . I do worry about an app where the advice is to check weekly that your database aren’t corrupted. In 2023?

[rms]

FYI the process checks for more than “corruption”. For example will look to see if user or other deleted indexed files. And system file corruption not unheard of.

[OogieM]

@Ulli

Do you figured out the technical reason behind your data loss?

not my job to debug closed source software

[macsorcery]

@OogieM

not my job to debug closed source software

my thoughts on iOS and Ventura

[OogieM]

@Rkrug

As always with software, one should not trust it completely to keep data safe - there is nothing like a solid backup strategy:

1. keep backups onsite and offsite
2. Keep old backups following a multiple generation model (e.g. keep yearly backups forever, the monthly of the last year, weekly of the last month, daily of the last week)
3. verify your backups and data for corruption

My backup strin was time machine, daily backups to attached hard drive, daily backups to an in hiuse server, monthly backups and a yearly backup. Save the monthly backups for a year. Full sets in 2 different cities, one set in a media rated fire safe in a different building at my main site.

And as for the rest, I did not and have never used iCloud. I performed the recommended verify and repair of all databases weekly, I also did the recommended rebuild on a monthly basis. The data I lost are data that I only rarely refer to. It was information that I typically need about once every 3-5 years. I didn’t realize the system was broken until even my extensive backup option was filled with bad empty files.

@Rkrug

It should be easily possible to implement such a rule and execute it either manually or on quit.
Or - could one leave “automatic sync” on, and only sync if these checks pass?

It was only AFTER those of us affect got DT to acknowledge the issue that another user implemented looking for zero length files as a way to catch it and yes, I found it happening afterwards, provided logs and other information and got fed up with the abusive nature of the support reponse. I gave them ideas of where I thought it might be going wrong as well as some sugestions for what to ipmplement in their system to identify it sooner. That did eventually result in the verify file integrity option but it’s still not recommended as a normal part of DT database maintenance.

@macsorcery

I think it is too powerful and versatile to ever be replaced.

If you had asked me this question before my problems I would have agreed with you. As I said I’d had DT for over 10 years. I do not agree with that now.

A tool like DT has one majot funciton to be a safe and trusted repository of data that I want toaccess later. DT failed in that massively for me and I can’t in good conscience let others risk it without knowledge of the issue.

[rkaplan]

@rms

FYI the process checks for more than “corruption”. For example will look to see if user or other deleted indexed files. And system file corruption not unheard of.

I suspect if you ask around - the number of people who have actually lost data in DT3 due to a software issue (i.e. not a storage device crash outside of Devonthink’s control) is on the same order of magnitude as those who lose data in an enterprise-grade SQL or MongoDB database.

[rms]

@rkaplan

s on the same order of magnitude a those who lose data in an enterprise-grade SQL or MongoDB database.

And those software systems also have verify and repair components … all good.

[Ulli]

@OogieM

not my job to debug closed source software

Of course not!
But there could be a lot of reason why a file lost his content, beside that it has to do anything with DT3.
I had never an issue with this since I use DT3, but I also had lost the content of several old files before I startet using DT3, so if you do not know the reason for the problem, it might be possible, that it was something other, than DT3, who caused the problem.
As you develop Apps yourself, I thought that it could be possible, that you had looked deeper into that, to see what happened with the files.

[TudorEynon]

I have to say that is at odds with my own experience. In the early days especially a great number of user errors on my part and a great deal of patience on the support community side. I would say beyond the norm even for the high standard that most of the Mac Power Appls have.
Sooner or later an outsider situation and crossed wires will occur with anything of course, I can’t judge from the outside on that. I know in other situations I am familiar with it happens: it is very hard to convey technical information in fact.
Bear in mind that data loss and complications were routine in most computer systems and have radically decreased over time.

I have heard the complaint similar to yours during discussions I was actually part of and I have to say I couldn’t see what the ‘insults’ were. There can be cultural factors I guess? Then again as an European I can be taken as a bit ‘abrupt’ in the USA.

[Jezmund_Berserker]

Thanks for sharing your experience. I’m glad it was a good one.

[Bmosbacker]

@rms

I make automated archives of all datatabases once a week,

@rms may I ask how you automate this? If this requires a lot of effort to explain, please feel free to point me to a resource. My workflow has completely settled down. The only thing I find myself wrestling with is whether or not to use DT or just keep files in Finder. I would not hesitate to use DT except for a few of the bad experiences that have been shared in this forum. I like DT and could make good use its many features but the bad experiences shared make me a bit nervous. An automated archiving procedure would make me feel more confident in using DT over the long-haul.

[rkaplan]

@Bmosbacker

An automated archiving procedure would make me feel more confident in using DT over the long-haul.

There are probably many ways

I use CCC to backup my DT3 directory to Synology on a schedule

In turn Synology syncs to Dropbox to give me one off-site backup
The Synology also syncs to a 2nd Synology in a differerent location
And Arq works automaticlaly as well to backup everything to cloud on a schedule

[rms]

Normal system backups of DEVONthink databases (TimeMachine, ChronoSync, Arc, CCC, Backblaze, whatever) should be good enough and work just fine. Not complicated. And can and should be automated. Just what you want. Not sure why you are hesitant of doing this normal, automated, standard way of backing up DEVONthink databases as part of normal system backups.

Archive zip files great for moving a database somewhere else, or simply taking a (as the name suggests) an “archive” copy for whatever reason. Or use this as yet another backup.

I’m not convinced I NEED these database archives, but as it “costs” me basically nothing in money or system performance I do it. And creating one for a database is a standard part of DEVONthink, and can be automated.

To make scheduled archive backups, see the whole thread, but I copied and tweaked this one. It zips all the currently open databases to archive. Read the whole thread, but this one sums it up and works.

Backup all open databases

I use a macOS cron job to launch this apple script once a week. DEVONthink needs to be open when this goes, so it’s scheduled in the morning when I can expect to be at my desk with the Mac not sleeping.

I have a Hazel rule setup to delete any zip files older than the three most recent.

Works great. And I’ve tested that the databases can be restored from these archives. Not the indexed files, of course. These zip archives (and indexed files) backed up by the normal system methods.

This another reason to rely on the DEVONthink forum to learn good stuff about DEVONthink.

@Bmosbacker

he only thing I find myself wrestling with is whether or not to use DT or just keep files in Finder.

that would not be useful, IMHO. That implies you are indexing stuff. Frankly, don’t bother. DEVONthink not a Finder replacement. It’s more than that. And these indexed files NOT in the database archive anyway, so back to the top paragraph here on this post.

[OogieM]

@DEVONtech_Jim

we suggest a weekly or bi-weekly check with File > Verify & Repair on each database

File verify and repair does not catch the zero file length bug. You have to also recommend that users do a full file integrity check at least weekly.

[rms]

@OogieM

and repair does not catch the zero file length bug.

That bug fixed a very long time ago …

[Rkrug]

I am using DT build in scheduling (hourly, daily and weekly smart Rule) at the moment: I have set up rsnapshot to do snapshots of the Databases folder whenever triggered. OK - I know not ideal (database open message when openint the snapshot) but I think a uaeful approach.

If you want to know more, let me know, and I can post some more info.

[rms]

@Rkrug

to do snapshots of the Databases folder

Interesting idea … but how is this better than relying on a system backup that is running all the time, e.g. how TimeMachine works? Beyond that, I just feels better packaging up the entire database (excluding the indexed files, of course) into the standard DEVONthink archive. This comment not to dissuade you, I’m just curious.

[Rkrug]

Time Machine: Independent backup of critical data

DEVONthink archive: I am new to DT and just put the Snapshot script together - wasn’t aware of the DT archive. But nothing against that one - but maybe smaller, as rsnapshot is using hard links for identical files.

[rms]

TimeMachine backs up everything unless you exclude.

Read the section about backups in the DEVONthink Manual. You want to be sure to do it the way that works. I know that normal system backups work. I know that database archives work (but do not include any indexed files outside the DEVONthink database).

[95omega]

Do you work for Devon Technologies? This snarky comment is your tell. This type of comment is pretty typical in the forums, and why I moved on.

@rms

And can and should be automated. Just what you want. Not sure why you are hesitant of doing this normal, automated, standard way of backing up DEVONthink databases as part of normal system backups.