Do you store proprietary files in DEVONthink?

This was an imported file. I am not sure if it’s happened to indexed files in my case but others have reported that with indexed files. I know it can affect ANY file type, text, rich text, JPG, PDF markdown or email for sure as that is what I’ve personally lost. I suspect any file is susceptible.

Again, not to diminish any data loss, but…

this is not a widespread issue

This certainly could be due to specific environmental conditions in @OogieM’s setup - hardware or software but it’s unclear as we don’t have other reports coming in and we can’t reproduce the behavior.

Note: We aren’t working in a laboratory setup with specific conditions being maintained. We are in varied environments, geographically distant, with varying databases, hardware, etc. We have databases well over a decade old to brand new ones, with imported files, indexed files, and a mix of the two. So while we can’t reproduce the exact conditions and environmen of a specific user, we are definitely covering a broad range of those variables.

We are also working on some extra internal mechanisms to further improve file integrity checks of files in DEVONthink, whether imported or indexed. These are in internal testing right now. And while we can’t say, “Yes, this will fix the issue” authoritatively (again, as we can’t reproduce the issue), consistency checks may certainly help avoid such issues.

5 Likes

Have you looked at having a detailed byte/bit level investigation of the details on the hard drive of an instance where this loss occurs to see if you are hitting some issue with the index of sectors/bytes that contain a file that is not contiguous being inadvertantly set to a zero byte length? (See my CPM discussion over breakfast this am on a nearly exactly the same type of problem that my husband dealt with years ago that I posted in the DEVONThink forum.)

Of all the potential reason this one actually sounds closer to accurate compared to other issues.

I cannot speak to all the tools and techniques Development employs.

However, in some cases of hardware failure, we have sometimes seen file and database corruption. That would potentially corrupt any file on the drive.

1 Like

Correct.

However the number of files affected, the fact that they were all only in DEVONThink databases, the fact that tools provided by DT to ensure data integrity failed to alert me to the problem and the fact that NO other similar things have happened toany of my other files stored in other places leads me to the conclusion that DEVONThink is the source of the problem.

the fact that NO other similar things have happened toany of my other files stored in other places

What tools did you employ to assess and conclude this?

2 Likes

I have manually looked at every folder on my internal hard drive in my user space including the hidden files and folders looking for any other files with a zero byte length. Then I did the same for a recent clone backup of my hard drive. After verifying I had no zero length files I did a full disk copy using Carbon Copy Cloner with find and replace corrupted files which does a full checksum verification on the file compared to a backup. No errors were found. I do that sort of backup at least once a month. I also went toa selection of my oldest archive files and opend them all up. No problems.

There are folders with no contents, for example .conda .gconfd and others but no files that are empty.

The only other software package I use that pulls some actual files into a database structure or hides them in a package is Scrivener and I’ve never had any data loss in any Scrivener document. I do use Scrivener for some of my workflow documents including one on how to clean, scan and catalog glass plate negatives for the historical society, a document that has been in constant revision and updates since 2017. That is my oldest Scrivener document and the closest thing I can compare to an old DT database.

When I looked at the package contents of the DT database to pull in the backup of the most recent file lost the backup had a file with data in it, i.e. not zero byte length, while the missing one was a zero byte length file.

1 Like