I have a number of textbooks, most in PDF format and some epub. Any recommendations on apps that would enable to me to highlight the books and then export the highlights into a common format, e.g., rtf, markdown, etc., preferably with the page numbers and metadata for the highlights?
I use DEVONthink to do this for articles, etc., but I have heard that you shouldn’t import entire books into a DEVONthink database.
EPUBs can’t be annotated in DEVONthink. That would be the jurisdiction of Apple Books or a bespoke EPUB app.
but I have heard that you shouldn’t import entire books into a DEVONthink database.
Not sure where you heard that but, PDFs are certainly welcome 
You can read epubs and PDFs in Zotero and highlight them. If you create a highlight note (a simple right-click on the document), it can be imported into Obsidian with page numbers, citations, and a link back to the document. The document imported is a markdown file.
I know for EPUBs this app may do what you want. It’s not a reader though but a separate app with multiple export options like you mentioned. Is also a great option for physical books as well.
EPUBS can be annotated in many apps.
I’m using Readwise reader and it exports the highlights into Obsidian as markdown files.
I have also used Zotero plus the Highlights app to get multi colored highlights for different types of notes.
I put all my books into Calibre to get them to a common format I can use everywhere and no, I do not sell or trade or share the files.
I have a big Finder folder of research books in PDF format that I’ve indexed in DEVONthink so that I can use DT’s search tools to help me work with them. I’ve never encountered a problem doing so.
Like @OogieM I use Readwise Reader to annotate and highlight PDFs and ePUBs and automatically port my highlights into Obsidian. Reader gathers the exported material into a single note for each book or article. accompanied by some basic metadata (author, title, summary if there is one, tags, etc.) Each highlight has a link to its place in the original text so you can refer back to it. It’s a pricey app, but I haven’t found anything else that allows highlighting of web articles, PDFs, and ePubs.
I put all my books that are pdf and epubs imported into DEVONthink which helps me to not lose track of them. And if DEVONthink cannot do with the files as I want I simply use the “open with …” feature to open with another app that does it. That does not happen often except for epubs.
This can be easily done with Skim.
There’s no issue with storing books in DevonThink though, I keep a lot of books (and other very large PDF documents, some with over 2.000 pages) in DevonThink.
Zotero 7 can do that for webpages, PDF and epubs - and using the Zotero plugin the highlights can be imported in Obsidian. Readwise is almost certainly more polished and intuitive - but it can be done in Zotero.
Good to know! I don’t find Readwise Reader particularly intuitive, and it’s more than a bit fiddly. I will definitely give Zotero a test drive as a reader / highlighter. So far I’ve mostly used it to grab citation information.
If paid cloud apps are on the table, BookFusion might be an option. It’s free or about $25/year, has a nicer two-page view than Readwise Reader, supports highlights that export to Readwise or download as HTML, markdown, PDF. Again, it’s cloud and paid if you have more than a few books. The citations aren’t as good as Zotero’s. The highlighting experience isn’t quite as good as Readwise Reader’s.
Be aware that if you simply open an ePub file using Books, highlight something there and then copy this file to another Mac using regular “copy-and-paste” way, the file will be copied without highlights. At least, this is how it worked two years ago.
Another issue with Books, is that on a Mac, Books is really just Preview for PDFs. Also, the iOS versions are not feature identical, with iOS lacking a lot of what Preview on a Mac can do. Books is another one of those Apple apps that I use and then realize all the things that are missing or could use an update.
I read PDFs using the Preview app. I don’t like Books, and Music. I would say, both feel more like App Store add-ons than real apps. At least on Mac.
Preview is great for PDFs, I just wish there was a Preview for iPadOS so I could use them interchangeably. I mean I can using Books, but when on iPad, I miss the features of Mac Preview.
Books needs some Apple love, but I think it is another forgotten app. Maybe they will find a way to add AI to it and make it worse? 
I’ve used Zotero as a Bibliography manager. Can you sync the attachments between devices using iCloud?
Zotero recommends you use their own sync service or webDAV
I have read elsewhere that using iCloud, Google drive, One Drive etc will likely end in data corruption. I use the Zotero servers without any problem. I have also used pCloud in the past as a webDAV server. A bit fiddly but it worked