Au revoir Bookends, bonjour Zotero

@toffy Note that coming from Citavi sets a very high bar w.r.t. software capabilities and quality of support (I could experience this first hand when I worked for them to bring Citavi to the Mac – the project was eventually abandoned in favour of the Citavi web version). However, I’ve also worked for Papers (pre ReadCube) and (as a contractor) for Bookends, and I can attest that the people behind these tools were/are very much dedicated to develop sound reference management solutions.

Personally, I wouldn’t dismiss an entire app based on some public support threads. For a developer, it’s often very tricky to balance individual user requirements/wishes with the overall goals of the app and its target user group. Also, feature requests that may seem straightforward to implement from the outside often aren’t easy at all, and may technically be very challenging to implement. As an example, adding fully featured AppleScript support to Bookends (which I helped to implement/design) wasn’t easy. This is since Bookends is a Xojo app which, by default, only supports raw AppleEvents but not AppleScript. So there’s often technical challenges that the user isn’t (and shouldn’t need to) be aware of.

Overall, and ignoring their individual idiosyncrasies, I think that both Bookends and Zotero are robust apps with sound support. And both apps have been (and likely will be) supported long-term (Bookends even since 1984!). Will these apps leave things to be desired? For sure! But both can be scripted to fine tune to your needs (though I think, on a Mac, AppleScript automation is easier and also works between different apps).

W.r.t. the issues you’ve mentioned:

  • Bookends offers a “Find A Group…” command (available from the source list’s contextual & cog wheel menus) which will select the first matching group.
  • To batch delete multiple user-created groups in Bookends, select them all (via shift-click) in the source list, then hit the Delete key and press “Delete” in the displayed dialog. Alternatively, you could also batch-delete multiple groups via scripting.
10 Likes

Given your in depth knowledge of both Citavi and Bookends, how would you go about importing a citavi database into bookends? I’d like to preserve as much information as possible (even the “date created” and “date modified” are valuable to me), but especially important would be the attached pdfs and the notes/quotes. As you know, notes in Citavi don’t have to be associated to a reference and I can see how those might pose a problem for bookends, so we could drop those, if necessary.

I‘d try out each of the export formats supported by Citavi and see which is the most lossless. RIS doesn’t appear to be the best choice. IIRC, the EndNote XML format should be supported well by Bookends. I’m not sure but if you copy your PDFs into the default Bookends attachments folder, Bookends may even be able to auto-match them. Otherwise, it should be possible to open the Citavi export file in a text editor and change all attachment paths to the path of the Bookends attachments folder.

I remember that Citavi also offered its own (Citavi XML) exchange format. This likely would be the most data-rich format. But one would need to write a script to import this into Bookends. Maybe you could try one of the standard exchange formats (like EndNote XML) first, and then only use a script to pick missing data from the Citavi XML. This, of course, requires some coding. I could help you with this but my time is unfortunately pretty limited right now. Alternatively, it‘s definitively worth asking in the Citavi and/or Bookends forums about your specific issue.

1 Like

Wow, I really appreciate your tipps. My impression is that either endnote XML or ENL are the only real options here because the others don’t support quotes/comments. In fact, I don’t even know whether the endnote formats do. Perhaps not. There is probably a good reason why the guy who wrote the Citavi to Zotero export translator is not using any of the official export formats but rather Citavi’s backup file (which is a zipped XML file. Not sure whether it’s the same as the Citavi XML that you’re referring to but I suppose so.

Actually, bibtex might also be an option because bibtex export is heavily customizable in Citavi, but even if it goes as far as letting me include my quotes, it would then still require a custom import on the other side, so I’m not sure how much work can be saved going down that path.

I haven’t gotten down to testing bookends import yet (currently testing Zotero import) but I suspect that I would end up importing that Citavi XML to bookends and if I do, it’s good to know I can ask you stuff (really appreciate it) but to be honest: I just noticed (by looking at your homepage) that you’re the developer of keypoints and I’d much rather have you work on that… When I discovered it a couple of weeks ago, I was thrilled and knew immediately that this is what I want. But looking at the progress history, I wasn’t too optimistic about when it will be released. No wonder when the developer offers his time to strangers trying to import stuff into bookends :wink: (No offence! I really appreciate it, but speaking from experience, I see some procrastination going on here…)

That said: could you point me to some existing import file (that would be an apple script, from what I’ve understood, right?) to give me an idea of what kind of work would need to be done?

Another thought: Since it looks like I can get my stuff into Zotero by editing the existing import script, maybe the easiest way to get it into bookends would be by exporting it from Zotero?

Oh, and another quick question: when I try in Bookends to import from Endnote, it askes me where the Endnote app is installed. Do I have to install endnote just to point bookends to my ENL file?

Question for everyone in here, who clearly knows reference managers. I’m an attorney. Could I get any use of out of these things? What makes them useful to you in your professions? I assume some of you are in academia or any hard science that makes regular use of journals and “publishing a paper” is how you advanced the field.

While lawyers LOVE their citations (have you seen The Bluebook?), I don’t get the impression that very many use any of these apps, and I don’t know why…

Thanks!

3 Likes

Citation managers in the physical sciences (and the medical studies and the liberal arts fields and …) consistent of some or all of an ensemble of tools required to manage …

  • the source (a journal, report, dissertation, Webpage …)
  • methods to annotate or markup the source
  • methods to search, assemble, and brainstorm through the information highlighted by annotations and markup
  • a citation link (a citation key) for binding to the storage point of the source
  • meta-information documenting where to locate the source in the public domain
  • summary information to oversee the source
  • methods to search on meta-information or summary information
  • methods to search in the public domain (the Web) for citation links to comparable references to the source itself using any number of low to high power search approaches
  • methods to export one or all of the above information in formats that can be read off-line
  • methods to export one or all of the above information in formats that can be (automatically) incorporated by a “mail-merge” type of operation when writing a journal article, proposal, or other publishable work
  • … other types of bells and whistles

Distinctions between the various citation managers (macOS related … Bookends to Zotero) include what sub-sets of these tools they each offer, how easy the tool is to use in the given app, and how the tools are arranged in the UI for the app. Jumping to Windows systems expands the offerings.

The battles here are not that the big name or indie-developer citation managers do not do citation management. The battle is whether you want a top-of-the-line Ford F350 (and are willing to forego high fuel economy and ease of parking) or a Honda Civic LX (and are willing to forego hauling off to routine backwoods camping trips with your buddies and the chance to be on the front cover of the next issue of Road & Track magazine).

I can see the Blue Book but I cannot see how it fits to this. So, as a lawyer, where do the needs of lawyers to manage “citations” fit in the above?


JJW

3 Likes

In the world of BibTeX at least, there’s not a lot of support for legal citation - see this 8-year long thread for a discussion:

I’ve been doing a lot of work in South African legal history, but I kind of dread the day that I actually submit something to a SA law journal.

2 Likes

Thanks for the detailed reply @toffy. I‘m on the road today, so just quickly: Citavi’s backup file should indeed be the same as the Citavi XML file I mentioned. Given your requirements, trying to parse & import that file strikes me as the most promising approach. However, as mentioned, this would require some scripting.

The Zotero page that you linked mentions that, before exporting from Citavi, you could bake your PDF-specific notes into the PDF. That approach will likely also work for Bookends.

Also, since there’s already a supported import process for Zotero, it‘s worth trying that route first.
Additionally, if you send me a representative Citavi backup file (containing a subset of your refs & notes), I could have a look. I‘ll promise not to procrastinate! :-]

Thank you for that! Honestly, I’m not sure lawyers have those needs…or at least we aren’t set up for that type of functionality. We primarily cite to case law, statutes, and various secondary sources (mostly law review articles and treatises). There is one official public statute source per State. Most courts self-publish cases and then you have the century-plus Westlaw and LexisNexis duopoly on “case reporters.” Law reviews are inherently limited by the number of law schools in the US.

And that’s the other thing…American lawyers basically cite only to American law, so there’s no complication of foreign sources.

BUT that being said, at least the first 3 bullet points apply equally to lawyers. I think where you begin to lose me is the 4th bullet point and below. There simply isn’t “meta information” available for cases. And even worse, every court has slight variances in the so-called local rules for citations, so there’s not even a single standard to rely upon for citation style! If you are outside my home state of Washington, for example, you cite to our official Supreme Court reporter as follows: XX Wash. 2d YYY (Year). But if you are in Washington courts, that Wash. 2d converts to Wn.2d because…some judge said so at some point.

So maybe this is why lawyers don’t use citation managers. They don’t really help us!

1 Like

Currently, it’s definitely me who’s procrastinating… But at least it’s what they call “productive procrastination” as I need to get this reference management thing solved before I can properly transition to Mac.

Citavi → Zotero

Anyway, I have experimented with the Zotero route now and I’m no longer so optimistic about that route. There is quite some fixing that needs to be done, the biggest one being that relations between titles are currently and everything I find about that is that people say that this is how it is. So I assume there is a reason why apparently no Zotero translator can import relations between references. UPDATE: it turns out relations are supposed to be preserved but currently broken. See here.

I also found at least one item that was imported incorrectly and as a result lost quite some information. It is a book published online and I’m pretty sure I stored it as a Book in Citavi but Zotero imported it as a Webpage, which, in turn, entailed (at least that’s my guess) that the attached pdf was not imported. While this individual case can probably be solved, the problem is that there are probably quite some more similar edge cases that I may or may not discover before the final import, which means I may loose valuable data (I have been building my Citavi database over the past 15 years).

Zotero → Bookends

I did go ahead and exported a couple of references from Zotero to RIS (including notes) and imported these into Bookends. (I’m not sure why the bookends website recommends RIS for Zotero → Bookends, but I just stuck to those instructions.) Now, the first thing I noticed was that there seems to be a bug in the import script that cuts off the last character from each note/annotation. This turns out to be the closing > for the last HTML tag.

image

I suspect that it is due to these incomplete tags that the notes themselves are not rendered properly (they show as raw html), but when I fix the tags, it still doesn’t work, so the conversion, if any, must be done by the import script.

Also, I noticed that notes are imported into one single note and I suspect (hope) that this is also due to the faulty html tags, because there is no need for this, given that bookends also supports individual notes.

Citavi → Bookends?

Anyway, long story short: after having played a bit with bookends, I believe not only that it is far superior to Zotero in terms of features but also that importing to it directly from Citavi is probably the way to go.

One clear advantage of this route is that both Citavi and Bookends support notes on relationships but Zotero doesn’t, so this information would be lost on the way. Zotero also loses the direction of the relationship (which title is referring to which) and since there are no comments on relations, I see no way of fixing that in Zotero.

In Bookends, I haven’t been able to ascertain whether the links are symmetrical or directional but even if they are symmetrical, the comments filed could be used to indicate the direction.

There is one slight downside with Bookends compared to Zotero, though, and that is the absence of standalone notes. But that could possible be fixed by using one or more dummy references that can hold those notes.

One problem with both Bookends and Zotero is that annotations cannot be members of categories (Zotero) or groups (Bookends). This is a problem, because all my most important annotations are linked to one or more categories, indicating a specific paper or section thereof where I intend to use them. So I’ll probably have to write those categories as tags into the note itself.

So I’d be curious to learn more about what kind of work would be necessary to import from a citavi XML file. To get an idea of the data structure to be imported, you can look at this publicly available sample file at https://gist.github.com/zuphilip/76ce89ebbdac0386507b36cff3fd499a

Since you have so much invested in your knowledge in Citavi, something you might consider is using VMware to run Windows in Unity mode, then Citavi would be available to you while running Mac software. Whether this is useful probably depends on how integrated you want the reference manager to be with other Mac software.

From what I understood, Windows virtualisation doesn’t work with the new M1 processor… But I did start to contemplate using Citavi web, which is currently in public beta. Two problems with that, however:

  1. Citavi web has been in development for about 10 years now and while it is good to see it reaching Beta, something is telling me that we won’t see a final release any time soon.

  2. Even if it is released soon, I’m having difficulties seeing how exactly I would be using it on Mac. Obviously through the browser, but having to go through the browser will severely limit what I can do with it, right? They have a Word plugin that can talk to their API, but that may already be it. If the API will be public, things would look quite differently, though… But also in that scenario, time will be a heavy factor. I need something now…

Wow, I didn’t think I would end up right there before I even started using Bookends. I can’t believe just how non-caring Sonny Software actually is.

Just to quickly summarize my conversation with the developer, because it was via email, not on the “support”-forum. (Sorry, it became less of a summary than an annotated version of the exchange…)

I reported a bug related to importing pdfs from the watched folder:

I just tried the "watch folder” feature and suddenly the three references that I had manually added to my empty bookends database were done and it looked like this:

This looks like a bug to me but maybe I’m doing something wrong?

He replies:

I’m sorry, it’s not clear what you did. You had 3 references in the library and then did what? And can you reproduce whatever it was that resulted in this?

I confirm that I can reproduce the issue:

Yes, I can reproduce this:

  1. I’m using the demo version of bookends.
  2. I have three references in my database
  3. I add one or more files into the Watched folder
  4. I select “import from watch folder → start watching” (if it wasn’t already running)
  5. Once the import starts, an error message appears telling me that I cannot import more references.(one error message per imported file)
  6. The list of references looks like in the screenshot in my first email.

Does that help?

Te developer asked a few more questions:

Interesting. There’s no reason for that error message if you have only 3 messages.

Can you send me the following, please?

  1. A screen snap of the error message.
  2. The library file.
  3. One of the PDFs.

Also, where is your watch folder located? The Desktop? I wonder if this could be due to some permissions problem that could be fixed by providing Full Disk Access in System Preferences Privacy.

He also noted that

According to the screen shot you sent, you have 2628 references in your library, not 3. What is going on?

At the time, I read this as he scratching his head asking himself “What is going on” but in retrospect, he probably wanted me to explain what was going on. But instead, I tried to clarify the issue. (I have a lot of experience with support, so I know you can never explain your issue too much):

I gave you all I know. It’s the demo version, I added three refs manually and then I tried to add some more via the watched folder. It is very weird indeed:

  • Why is bookends adding all those messages as references?
  • How van it add more than a thousand references when there is a limit of 50?
  • Why does it suddenly start complaining at 2628?

Hope you can figure it out.

Will send you the requested files as soon as I can. But, yes, my watched folder is a folder on the desktop.

Yes, I also know that you always have to send whatever support requests, but I was away from the Mac, so I had to briefly postpone that. I am also used to support not really doing anything with the stuff you sent, but usually they somehow try to pretend (“I’ve forwarded your files to our developers”). Not so in this case:

Sorry, I have no idea. Since I can’t reproduce this, and no one else is reporting a similar problem, it’s a local issue and it’s unlikely I’ll be able to help. In this case, I suggest that you explore the other reference managers for macOS.

In fact, within less than 24 hours, he didn’t even seem much interested any more in the files he had requested. I sent them anyway, assuming that they would allow him to reproduce the issue or at least trace how it happened:

attached are the requested files:

  • the database
  • one of the files I was trying to import
  • and here is a screenshot of the error message:
    PastedGraphic-1

So the moment he has everything he wanted, he switches into PEBKAC-mode. Of course, the user is too stupid to realize that he exceeded the number of references allowed in the demo version:

this is simple. Your database has 2433 references!

PastedGraphic-1

You cannot add any more without a license.

He added the following:

P.S. I suggest that you trash this library file and start again. Create a fresh new library and try importing 3 references.

But in order to bring us back on track (i.e. bug-tracking), I chose to further clarify the problem:

Obviously, the number of references is the reason for the error message. But the actual bug is that

  1. I didn’t add those references
  2. They are not references but some kind of error messages or log-entries.

But to him the P.S. was very important:

You got my second email?

So I acknowledge that I received it while getting right back to the actual issue, this time breaking it down into smaller pieces and mentioning only one of them (first rule when communicating with evasive others: one - and only one - question at time! Don’t let them pick and choose which question to reply to!):

You mean the one where you suggest I should trash this library file and start again? Yes, this seems obvious to me since I cannot do anything with the current one except for remove 50 “references” at a time.

So, where you able to figure out what caused the library to be filled up with hundreds of strange items?

And indeed: I managed to make him answer the question, but he also pulled out a good defense: answer the question only in it’s literal sense and ignore any implicit meaning that would be obvious in a normal conversation:

I have no idea. No one has ever reported that before.

Well, strictly speaking, he wasn’t even answering the question in its literal sense, probably because that would have required him to say “No, I wasn’t able to figure it out, In fact, I haven’t even tried.” But even so, his “couldn’t care less” attitude is becoming quite apparent.

So I’m giving it one last try, by adding some humour and focusing on an a cue that he himself previously brought up (that this might be related to the location of the watched folder):

I guess if someone had reported it before, you’d already have fixed it and we wouldn’t be mailing about it… But hey, someone’s got to start…

Were you able to confirm whether the issue has something to do with the location of the watched folder as you mentioned earlier?

Now the blame is on “the system” (and ignorance):

The “imported” items are all code modules in the Bookends.app bundle. Why the system notified Bookends they were files in the watch folder I don’t know.

I advised you before to look at other reference managers, and I reiterate it now. We’re not going to figure out what went wrong in your case and unless it happens again there’s nothing to pursue. Please don’t invest any more time in Bookends.

Note the paternalistic tone and note the “we”. I would say I have done my part of the “we” figuring stuff out. If this is artisanal software development, it’s the kind that combines the bad sides of “artisanal” with the bad sides of “software development” instead of the other way around. The artist claiming unlimited authority over the art, not willing to treat it as a product that he actually has to accept responsibility for.

While writing this post, I realized that my bug is quite similar to the one @JohnAtl linked to above. So it is not even true that “No one has ever reported that before.” But that clearly is irrelevant to him.

4 Likes

Not to wag my finger as personal admonishment but … please appreciate that some will see airing of extensive complaints about software as being disrespectful.


JJW

1 Like

I’m not sure I appreciate it, but I’m aware of the phenomenon. I still haven’t quite understood how people can be offended by criticism not against themselves but against the software they use, not against the software they made, but the software they use.

But given that this phenomenon exists, I’m glad my post above is not about software but about so called support, and that it is not so much complaining but describing and analysing. If anyone feels I failed to mention one nevertheless important aspect of my exchange with the Bookends developer, I’m sorry about that. I should of course have said that the developer was extremely fast to reply.

3 Likes

I see it more as reporting than extensive complaints.
I understand @DrJJWMac 's reasoning below. Just an overview would be better.

Just saw this post - and thanks to you- now I will have to start reading axle.design daily- I blame you fully :slight_smile: Quite an interesting blog

Ha. Good thing there is nowhere near a daily frequency of posts. :sweat_smile:

Glad to be of service!

I’ll amend this way:

Have you posted your detailed description and analysis on the SonnySoftware site to see if any of the users can reproduce your problem and perhaps offer a solution?

In doing so, would you post verbatim what you posted here as describing and analyzing? Or would you amend it? If the latter case, why? How does any step to amend it make what you posted here really "not so much complaining?

The shorter form of your report needed only to say that you reported a bug, the bug appeared to be the same as the one from @JohnAtl , your report was essentially ignored, and you too are frustrated. The offense that I take is to the longer report with side comments on the details that you give behind the developer’s back (so to speak).

In a nut shell: I will defend your right to speak against software. I will however question anyone seeming to demean a developer, even tangentially, in a forum where the developer may not correspondingly be made aware in order to exercise their right to reply openly, even though I may feel the same way as you about that developer.


JJW

5 Likes

I usually prefer public support tickets/ forums but in this case, I followed the path of least resistance from within Bookends, which led me to write an email. Regarding reproducing the problem, the developer was only interested in reproduction until I said that I was able to reproduce it and provided the steps:

If the developer had suggested that I should check the forum to see if anyone else can reproduce it, I might have done so. But why invest time when the developer is not interested?

I’m not sure I understand you here, but since I didn’t post to the forum, I suppose it is irrelevant.

I’m glad you raise that point so explicitly because it allows me to explain why I disagree that such a shorter form of my report would have been better or more appropriate.

First off: I agree that shorter (as in more concise) texts have advantages. If you can convey something with X words, don’t use X + Y words. I take this principle quite serious. But there is another principle that is equally if not more important:

Let others speak for themselves as much as possible, especially when you disagree with them.

If I had written the kind of summary you are suggesting, I would have provided a summary of what happened based on my own relevance criteria, in my own words, and with an emphasis on my feelings. This is the kind of reports you find all over the internet and they are pretty much useless, IMHO, because they are more about the author than about the object they are reporting about. Such reports only become somewhat useful in larger quantities, like star ratings on certain online stores. If may people give a certain rating, that gives you some indication about the product. But one x-star rating?

My intention with my report above was not to add yet another complaint (1-star rating) to the pile of complaints about Bookends already out there. It wasn’t even primarily about the Bookends app itself. It was about the company behind it (which I suppose is the developer himself, but if it had been a huge software company, it would have been about the what kind of support the company provides and how it interacts with its users).

If my interaction with Bookends support had been publicly accessible on the Bookends forum, I would have done the same as @JohnAtl above and simply linked to it. Since it wasn’t, I had to provide it, not only in order to let people judge for themselves but also to be fair to the developer.

When I started the post, I intended to provide more of a summary, but as I went through the interaction, I actually found very little that I could cut out or rephrase without generating a bias. So I opted to provide the original text so that whoever cared to read my post, would be able to check my commentary against what was actually said.

So when you say:

I reply: my comments about the developers behaviour could perhaps be seen as demeaning if I hadn’t provided the evidence in which those comments are grounded, i.e. if I had provided a shorter version of my report. To state something like “I reported a bug but the developer ignored it” can be quite offensive against the developer as he may well respond “What do you mean I ignored it? I looked into it and said X. How dare you distort our interaction like that to make me look bad!” So I agree that, it in that case, would be appropriate to give the developer the opportunity to provide their view of what happened. But since I chose not to provide such simplifications, there is not much to dispute.

Of course, my interpretations of the quoted text can be disputed, but anyone can do that and I’m more that happy to listen to alternative interpretations of what happened. Indeed, I consciously formulated some of the comments somewhat pointedly in order to provoke contradiction. It would be incredibly enlightening to be told that I completely misunderstood this or that or that someone else might have reacted differently to when the developer said X and thus the exchange might have taken a different direction.

2 Likes