Do you store proprietary files in DEVONthink?

Having my paperless system in Devonthink is what made me realize it had lost data. I was searching for reciepts when I found the data had been lost. When I checked three months of scans and other files had become 0 byte files. Thankfully, it was covered by my Time Machine backups.

Now, I keep all my paperless files in Finder and use Abbyy FineScanner for scanning and OCR. Alfred is my preferred method for seaching in and for files, and was before anyway.

2 Likes

I like to have a button in my Finder to create an empty file like I could in DevonThink. I can even create an empty Docx/ppt-file without having to open Word or Powerpoint first. This also put the file in the right folder just by creating it. How to do this without DevonThink?

There is a BetterTouchTool action that creates a file in the current Finder window, although I have this associated with a keyboard shortcut, not a button:

When I run the action with a Finder window open, the following occurs…

Likewise but I’m moving on. Fortunately, I had always indexed my files in DT so I was at minimum risk of losing files but I’ve found the hassle of keeping DT on my MBP and my iPad synced properly to be counter productive. And, if I can’t with confidence store file directly in DT, it has marginal utility given that I can use Alfred to find what I need. At this point I’m relegating DT to the occasional file conversion and OCR tasks.

2 Likes

I never used DTs search and see also features much at all. For my use case I am leaving my external files that were indexed in DT where they are in Finder. I don’t use Alfred either as I’ve never seen a need for it. For all the shorter notes that I had been keeping in DT I am converting them to markdown and moving them into an Obsidian vault. I am replacing the groups that I had in DT with a note in Obsidian that contains wiki links to all the new notes I have converted. In DT where I had folders that contained groups I am creating a note in Obsidian that is a link to the notes that have links to the new Obsidian notes. So I’m re-creating the folder and group structure in linked notes.

I do like DTs webclipper and have found that clipping web pages in RTF gets the pictures and text in a fairly uncluttered format. Those things will be moved into Obsidian with the pictures as attachments and the text in as a source or reference but also including the URL. However, since stuff on the web dissappears often I never count on being able to find it again if I need it so I do web clip a lot.

I was using DT to do the OCR of PDF files. For things I scan I use ScanSnap for OCR which I believe uses Abbyy. I don’t have to do too much of that so I’ll see how it all goes.

Like @Bmosbacker I’ll probably relegate DT to use as a file converter and OCR engine and themmove the resulting stuff into Obsidian or the Finder as appropriate.

et al:

No, this is not a widespread issue and reports are not only few but we haven’t been receiving any in quite some time.

Could this be some just walking away without opening a support ticket? Sure.
That would be unfortunate but out of our control.
On the other hand, it’s highly unlikely that everyone would have just bailed.

The issue is actually something that could have occurred in the past, back in DEVONthink To Go 2. With the upgraded functions and new maintenance routines in DEVONthink To Go 3, these sporadic issues came to light finally. Fortunately, occurrences were also infrequent and almost exclusively few in number, i.e., a handful of old files, not entire databases.

Anyone frequenting on our forums would know we quickly investigated the issue and implemented some fixes and checks to locate and corral these files, if they were found.

As the culprit is from some time ago, we can apologize and sympathize with any data loss, but we have no reports of such behavior stemming from DEVONthink To Go 3.

And as always, thanks for your patience and understanding. :slight_smile:

12 Likes

I do understand that others may have had unfortunate experiences at some time with DEVONThink when synchronising with DEVONThink To Go. This post is not intended to detract from that. However, I have to say that (as a user of DEVONThink 3 since it was launched—but, admittedly, not using DTTG) DEVONThink has for me been rock solid and I have found it to be an indispensable application.

Further, the support provided on the DEVONThink forum is, in my experience, unsurpassed by any other application support forum. The contributors are good-humoured and really helpful and I have learned so much simply by reading posts on the forum.

Speaking for myself, this is an application that will always be required on my computer.

I do appreciate there are two sides to every story—I merely wished to provide the other one! :grin:

Stephen

9 Likes

I can’t do more than I do really. It is mostly papers from journals, articles, bookmarks and stuff I have written that at a push I could find elswhere or re write. My way of working means very little is critical. I have documents which I have hard copies of anyway which I could re copy I suppose. I don’t really understand what happened to you and why your old copies were corrupted. I thought back ups, sort of went back and not just the latest iteration. MIne do but I admit I don’t keep them for years and years and really consider a back up a short term thing. Maybe I should keep older ones then.
Something can always happen I guess.

I guess it was a handful of old files when you consider that the total population of things I have in DEVONThink database is approching 30K items in one database and over 90K items in another one but I lost over 400 files. The thing is, unlike a lot of folks I do consider my old files critically important. I go back and use files that I’ve carried forward from computer to computer since the mainframe I used in 1978! Yep in some cases I had to retype but every time I make a major system change or change formats or major sofgtware applicatiosn or versions I go and bring all the old archive data forward into the new format.

A lot of my stuff is not available on the net. For some of it I may at once time have had copies, usualy on some old format like bernoulli disks, or mag tape reels or even punch cards. Those all got converted to computer files when I moved the data forward system by system over the years.

Not corrupted in the normal sense wher I might have been able to recover some of the data but erased except for the file name and reference to it. The contents were destroyed. A number of different file types too.

2 Likes

FWIW, I had the same problem where, on multiple occasions, files went missing in a folder indexed by DEVONthink. I’m guessing the bug is related to DT’s sync engine. Fortunately, I was able to recover my files using Time Machine.

I’ve all but lost faith in DT.

3 Likes

These things are hard to diagnose I found too. I had several periods of cursing DEVONthink 3 and earlier versions when the problems were nothing to do with the app but showed up in it as it were. I am not an IT person or a programmer. I have some math background. I am, I suppose a power user, people do ask me stuff and often what they decide the issue is isn’t what it is, they are quite adamant often and jab at the keyboard rather than systematically try to resolve it too. I threw away a back up disk once in temper… making my issue unsolvable.

1 Like

Latest update the same DEVONThink bug just happened.

No warning, database passes all verifications but suddenly there is a zero byte length file in my system. This time at least it was a file I was using regularly and I found it out quickly so I got it back from a backup.

The item in question was created on 03 May 2021 and went AWOL between Aug 7 and today.

But this is the same ghost file bug that got me with over 500 files lost.

As I said, I am moving out of DT as fast as I can and in fact the loss of this file just now made that project move way upon the stack of things to do.

2 Likes

More details on the issue over on the DEVONTHink forum posting I did

DT Forum post re file lost today

1 Like

Very good to know. Thank you for keeping us up-to-date!

Are these indexed files ? Do you know if this has been reported to happen to indexed files or just imported files? I only index, but this scares me !!

This was an imported file. I am not sure if it’s happened to indexed files in my case but others have reported that with indexed files. I know it can affect ANY file type, text, rich text, JPG, PDF markdown or email for sure as that is what I’ve personally lost. I suspect any file is susceptible.

Again, not to diminish any data loss, but…

this is not a widespread issue

This certainly could be due to specific environmental conditions in @OogieM’s setup - hardware or software but it’s unclear as we don’t have other reports coming in and we can’t reproduce the behavior.

Note: We aren’t working in a laboratory setup with specific conditions being maintained. We are in varied environments, geographically distant, with varying databases, hardware, etc. We have databases well over a decade old to brand new ones, with imported files, indexed files, and a mix of the two. So while we can’t reproduce the exact conditions and environmen of a specific user, we are definitely covering a broad range of those variables.

We are also working on some extra internal mechanisms to further improve file integrity checks of files in DEVONthink, whether imported or indexed. These are in internal testing right now. And while we can’t say, “Yes, this will fix the issue” authoritatively (again, as we can’t reproduce the issue), consistency checks may certainly help avoid such issues.

5 Likes

Have you looked at having a detailed byte/bit level investigation of the details on the hard drive of an instance where this loss occurs to see if you are hitting some issue with the index of sectors/bytes that contain a file that is not contiguous being inadvertantly set to a zero byte length? (See my CPM discussion over breakfast this am on a nearly exactly the same type of problem that my husband dealt with years ago that I posted in the DEVONThink forum.)

Of all the potential reason this one actually sounds closer to accurate compared to other issues.

I cannot speak to all the tools and techniques Development employs.

However, in some cases of hardware failure, we have sometimes seen file and database corruption. That would potentially corrupt any file on the drive.

1 Like

Correct.

However the number of files affected, the fact that they were all only in DEVONThink databases, the fact that tools provided by DT to ensure data integrity failed to alert me to the problem and the fact that NO other similar things have happened toany of my other files stored in other places leads me to the conclusion that DEVONThink is the source of the problem.