Devonthink data loss anybody?

Don’t shoot me. I have had a DEVONthink license since 2007. I know this is heresy. But I have to ask.
Has anyone ever experienced data loss with DEVONthink?
My use case has mainly been as my paperless repository. Invoices, documents, manuals.All my documentation collected for tax purposes and to keep track of home purchases.

On a number of occasions I have suspected that DT may be shedding data. I have gone back some years to find maybe a third of the receipts I was expecting in a particular time period. I have just gone in search of a receipt from 2011 and have found a 4 month gap in that year with no receipts, utility bills, nothing.

For the majority of time the database was on my desktop machine. Currently it is synced so I can also see the data on DTtoGo

Anyone else noticed issues?

I don’t know if you’ve been aware but there has been problems of data loss following the release of DTTG3. The latest updates have fixed this, but there has been two weeks when « ghosts » (corrupted files of zero size) could appear and the files were forever lost.

It has been solved with the latest versions and it was a rare occurrence, but scary for sure. On my end, I have never seen DT lose data beyond that specific issue (and only three unimportant files were affected so it was not even worth fishing for them in backups).

Overall a good habit to have is to regularly verify and optimize databases and check cloud locations from the settings. I have a monthly recurring OF task for that.

2 Likes

I’ve been using DT for about 10 years now, and as far as I’ve been aware have never lost data.

Have you contacted DT support?

3 Likes

I had a database that Devonthink said needed repairing this week.

When I checked, there were also a lot of missing files. I also checked my receipts, and I add them every day. There were two periods of about three weeks where all the receipts had disappeared.

I ended up pulling the files out of Devonthink as when I contacted support they could only tell me to press repair on the database (which resulted in an error message saying the database cannot be repaired).

1 Like

I had problems just recently with « damaged » databases which looks like yours and I received much more helpful support.
Turns out I had one of those zero sized files in the database (seen in the log). Trashing them and emptying trash fixed everything.

1 Like

I managed to get the files back from a backup, so all is good.

I’ve decided to use the Finder to store all my files instead, as this has seriously dented the confidence I had in Devonthink.

3 Likes

I didn’t say you lost indexed data; didn’t reply to you at all. I said when I “lost” data that’s what happened.

1 Like

I had considered using DevonThink some time back but opted against it due to the learning commitment required and it didn’t offer me that much in benefit for my use case. Just stored everything in Finder using a folder hierarchy similar to the paper files I had before. Has worked well for 8 years. Easy to backup.

2 Likes

Thanks. I just typed “data loss” into the DEVONthink discourse forum and received over 20 hits, all separate threads. So if it has never happened to you, fantastic, but now that DT seems to be becoming the centrepiece of many people’s digital life maybe a word of warning.

For me, I am wondering whether indexing in DT is a better strategy long term.

I dagger trouble seeing what DEVONThink would do to permanently lose a file that it wouldn’t also do to an indexed file. Can’t it delete the file in either case? And wouldn’t a corrupting save work similarly since the underlying file inside a DT archive is the same format as the indexed one would be?

I had that too. I index instead of import my most important docs to minimize risks.

This is a question most probably better discussed in the DT Discourse rather than here. That said, I’m a long-time user and haven’t had any problems. When I first read about the zero-length file issue, I set up a smart group to always be on the lookout for zero-length files. So far, it hasn’t found any. I sync between two Macs and two iOS devices, so perhaps syncing has prevented data loss, but that’s just a guess.

3 Likes

To resurrect an old dead horse.

I just did a file verify on the Mac as I have been doing every week now on my remaining DT databases. It’s an indexed database so none of the files are actually in DT. I have 15 files that failed the new file verify test. Went to look at them on the hard drive and they are all zero length text files, zero length .png files or corrupted PDF files. I checked an older backup and they were fine on November 13.

Even though I don’t use the DB much I still open it up during my normal BU session, run the file verify process and then do a backup. Except that now it won’t backup with the DT tools because the database is corrupted. Just for grins I tried the verify and repair procedure in DT. Nope, irrecoverable error.

Now I haven’t lost anything, first off I have my recent backup and second it happens to be my dowloaded copy of a forked GitHub repository. I could even go back to Git Hub and get it if I had to. But I wanted everyone here to know that the DT file bug still exists and that I can now confirm that it occurs in indexed databases not just imported ones.

As I have said I no longer trust DT. I wanted to point out that the problem still exists, This was a database I created in October so the problem is not the age of my database. I am continuing to remove all critical data from DT. I also plan to not open or use that database at all any more for any reason.

Some folks had indicated that indexed files would not be affected and since this test proved that is not the case I see DT as a loose cannon that could attack my data at any moment. I’m backing out as fast as I can.

5 Likes

In fairness to DT (because you have posted repeatedly here about this) it does seem to me that you are in a most unfortunate minority. Of course I sympathise with your difficulties and hope you’ll soon be able to relax with some alternative software that betters suits you and that you find more reliable—and about which you can post some really positive experiences. However, I don’t think your posts ought to detract from the fact that a number of us (here and on the DT forums) trust DT implicitly and do not share your experiences of it.

I merely wish to add a little balance to the discussion. :smiley:

Stephen

3 Likes

In DEVONThink with the database open select File , Check File Integrity per the latest release notes where DT indicated that this would at least find the files that have been nuked.

1 Like

This is exactly what happened to me, except it was 3 months of data. I went looking for some important receipts that I’d saved in DT and lots of files had been converted into zero-byte files. These were indexed, so the originals on iCloud had also been corrupted. I managed to restore from Backblaze, so I was lucky. Since then, I’ve removed everything from DT, and I’ve even deleted the app now as I have zero trust in it.

2 Likes

That’s really concerning. Indexing files on iCloud is exactly my setup. It’s strange that DT would alter old files. There’s at least one report of the same issue on the DT forum: Files lost again - #25 by OogieM - DEVONthink - DEVONtechnologies Community. Perhaps DT doesn’t play nice with iCloud indexing?

What worries me is that DT has recently implemented various scripts to check for corruption and fix corruption. I’d rather an application with such a long history didn’t corrupt in the first place!

Useful if you look at cgrunenberg’s (from DEVONtechnologies) post in reply to this same issue Files lost again - #30 by OogieM - DEVONthink - DEVONtechnologies Community for some perspective.

2 Likes

Thanks for highlighting that new post. It is possible that iCloud is the issue, rather than DT. Which is good, as I like DT functionality, but I might need to rethink how I sync.

That said, I set up a new, fresh database just a few days ago, indexing some folders. No system crashes or failures. I just did the DT verify after reading this: ‘Database XXXX is damaged.’ No clue how it has become damaged or how to fix it. It seems to be working OK.

1 Like

I don’t use iCloud (but for the native Apple apps, of course) for synching. Probably anecdotal and coincidental … or maybe a reason, but I’ve detected no data loss ever in more than a decade of using DEVONthink.