I have a number of textbooks, most in PDF format and some epub. Any recommendations on apps that would enable to me to highlight the books and then export the highlights into a common format, e.g., rtf, markdown, etc., preferably with the page numbers and metadata for the highlights?
I use DEVONthink to do this for articles, etc., but I have heard that you shouldn’t import entire books into a DEVONthink database.
EPUBs can’t be annotated in DEVONthink. That would be the jurisdiction of Apple Books or a bespoke EPUB app.
but I have heard that you shouldn’t import entire books into a DEVONthink database.
Not sure where you heard that but, PDFs are certainly welcome 
You can read epubs and PDFs in Zotero and highlight them. If you create a highlight note (a simple right-click on the document), it can be imported into Obsidian with page numbers, citations, and a link back to the document. The document imported is a markdown file.
I know for EPUBs this app may do what you want. It’s not a reader though but a separate app with multiple export options like you mentioned. Is also a great option for physical books as well.
EPUBS can be annotated in many apps.
I’m using Readwise reader and it exports the highlights into Obsidian as markdown files.
I have also used Zotero plus the Highlights app to get multi colored highlights for different types of notes.
I put all my books into Calibre to get them to a common format I can use everywhere and no, I do not sell or trade or share the files.
I have a big Finder folder of research books in PDF format that I’ve indexed in DEVONthink so that I can use DT’s search tools to help me work with them. I’ve never encountered a problem doing so.
Like @OogieM I use Readwise Reader to annotate and highlight PDFs and ePUBs and automatically port my highlights into Obsidian. Reader gathers the exported material into a single note for each book or article. accompanied by some basic metadata (author, title, summary if there is one, tags, etc.) Each highlight has a link to its place in the original text so you can refer back to it. It’s a pricey app, but I haven’t found anything else that allows highlighting of web articles, PDFs, and ePubs.
I put all my books that are pdf and epubs imported into DEVONthink which helps me to not lose track of them. And if DEVONthink cannot do with the files as I want I simply use the “open with …” feature to open with another app that does it. That does not happen often except for epubs.
This can be easily done with Skim.
There’s no issue with storing books in DevonThink though, I keep a lot of books (and other very large PDF documents, some with over 2.000 pages) in DevonThink.
Zotero 7 can do that for webpages, PDF and epubs - and using the Zotero plugin the highlights can be imported in Obsidian. Readwise is almost certainly more polished and intuitive - but it can be done in Zotero.
Good to know! I don’t find Readwise Reader particularly intuitive, and it’s more than a bit fiddly. I will definitely give Zotero a test drive as a reader / highlighter. So far I’ve mostly used it to grab citation information.
If paid cloud apps are on the table, BookFusion might be an option. It’s free or about $25/year, has a nicer two-page view than Readwise Reader, supports highlights that export to Readwise or download as HTML, markdown, PDF. Again, it’s cloud and paid if you have more than a few books. The citations aren’t as good as Zotero’s. The highlighting experience isn’t quite as good as Readwise Reader’s.
Be aware that if you simply open an ePub file using Books, highlight something there and then copy this file to another Mac using regular “copy-and-paste” way, the file will be copied without highlights. At least, this is how it worked two years ago.
Another issue with Books, is that on a Mac, Books is really just Preview for PDFs. Also, the iOS versions are not feature identical, with iOS lacking a lot of what Preview on a Mac can do. Books is another one of those Apple apps that I use and then realize all the things that are missing or could use an update.
I read PDFs using the Preview app. I don’t like Books, and Music. I would say, both feel more like App Store add-ons than real apps. At least on Mac.
Preview is great for PDFs, I just wish there was a Preview for iPadOS so I could use them interchangeably. I mean I can using Books, but when on iPad, I miss the features of Mac Preview.
Books needs some Apple love, but I think it is another forgotten app. Maybe they will find a way to add AI to it and make it worse? 
I’ve used Zotero as a Bibliography manager. Can you sync the attachments between devices using iCloud?
Zotero recommends you use their own sync service or webDAV
I have read elsewhere that using iCloud, Google drive, One Drive etc will likely end in data corruption. I use the Zotero servers without any problem. I have also used pCloud in the past as a webDAV server. A bit fiddly but it worked
I’ll say that I’ve had a ton of problems using Zotero with WebDAV, and just this month switched everything over to DevonThink.
This also raised another issue that isn’t immediately obvious: getting your materials out of Zotero (if you’ve made any kind of structure of groups within it) is surprisingly difficult, in a way that getting your materials out of DevonThink isn’t. (By materials here, I just mean the PDF attachments, I’m not even talking about annotations, which one might reasonably expect would be app-specific.) Zotero makes it easy to export your attachments, but not in any kind of structure-preserving way without resorting to plugins etc.
Some fair points
I agree with your thoughts on WebDAV. I did get get it to work, but as I said it was fiddly. I eventually went back to their own sync system, with which I have had no problem.
I take your point about getting materials out of Zotero, but I am happy to use Zotero for structure. If I do want a PDF (or whatever filetype) out I can just right click and find in finder, or just save it with the annotation intact.
I have always thought Devon Think looks like a powerful app. I tried version 2 for a long time, but could never click with it - however seeing how others use the latest versions I am impressed. It feels like the type of app I should take advantage of, but I also think it would be overkill.
Nevertheless Zotero has been perfect for my use case. I am studying at the moment and I need somewhere to store, annotate and note a lot of papers (mostly PDFs). I just needed it to work.
My work flow is simple.
Zotero ingests PDFs either from an extension in the browser, where it is send directly into the correct part of my library, or from a download.
I then read or scan the pdf, highlight and note. (which I can also do on my iPad)
The notes are then exported into obsidian via a plugin (I run Obsidian with very few plugins). I do further note taking in Obsidian if required.
I have well over 500 papers in my library so I am sure there is a lot more I could do with it even in Zotero. But I just don’t have time to fiddle with it all.
I know apps like Zotero are not for everyone, but it has been an absolute stalwart for me.
I absolutely think that if you never have to leave the ecosystem, Zotero is mostly great. The WebDAV thing annoys me because it’s what basically forced me to leave the ecosystem…at which point I discovered how hard exfiltrating my data in bulk really is. I accomplished it, but at great manual effort. More or less their WebDAV support feels (to me) like a feature they offered 10 years ago and then immediately regretted.
I guess I feel this is sort of a “Your backup is only as good as your ability to restore” type situation", if you see the analogy I’m making.
But, I’m glad it is working for your use case!