Probably far from the ideal solution, and I’m not sure how effective it is, but you can block some of those LLM scrapers while letting standard spiders through.
There’s more info at https://darkvisitors.com/ where you can find a robots.txt builder.
Fixed that for you 
1 Like
They put out a 15 minute video about their desktop version of this with the usual madcap energy. I wonder if you find this more encouraging @mlevison . It seems like their ultimate destination is less “here’s a summary of Mark’s website” and more “here are some agile coaches you should contact, including Mark, and we’ve prefilled his signup form.”
Optimizing to be recommended by an LLM isn’t too different from optimizing to be found in a SERP.
1 Like
And with the update today it now supports Kagi - so I’m giving it a spin
3 Likes
They just put out an update supporting Kagi as default
1 Like
Thanks for the recommendation, I’m also trying it out. Love the AI features and skipping search engines. I was pleasantly surprised that SuperHuman and all my plugins worked out of the box.
I’m giving it a try as my full-time browser.
1 Like
Say more? What plugins are you using with it? Or you probably just mean on macOS?
I mean Chrome extensions. When I did the profile import, it automatically installed Superhuman, Grammarly, 1Password, etc…
What @ryanjamurphy was asking is if this is on MacOS or iOS? This thread is about the ArcSearch app on iOS, which I don’t think has any way to use plugins…
1 Like
Ah, yes. I meant on MacOS.
2 Likes
I love this paragraph from the announcement on Reddit:
" we started by asking members what they wanted and here’s what we heard:
- ability to set as Default browser
- block GDPR popups
- block Newsletter popups
- block Ads
- block Trackers
done, done, and done.
Arc Search cleans up the internet for you."
I mean, what’s not to love?
3 Likes
Just block 99% of the net and tell me what I need to know, really.
And yeah this will change the net - but that happens every few years anyway.
Actually, this isn’t far from the much dreamed of semantic web…
2 Likes
Everyone who likes (or especially almost likes) Arc Search’s Browse For Me feature should try Perplexity, which is the same idea but gives straightforward answers. I’m generally using Perplexity when I want answers to questions and Arc search when I want an overview of a topic. Conveniently Perplexity is a default search engine option in Arc Search
1 Like
So much this! Remember the old days of the Internet, when Google triumphed and that discipline called “SEO” came out. Creators could conceivably think that the fight for SEO relevance was forcing them to publish content that otherwise would have been behind a paywall, so that Google could index it (tell the online newspapers!). And it was true, but content creators adapted and both sides of the business (Google and the content creators) thrived for a couple of decades I would say.
Now it’s the end of life for SEO.
For one, it has become less important as Google has pushed for paid placements and programmatic advertising, but this made the search result pages a confusing mess that do not return relevant info to the user.
But, more importantly, the Search Engine concept has become disrupted by the LLMs. There’s no way I would run a Google Search to get an answer an LLM would give me, and there are quite a few cases where Bard or ChatGPT would work so most users will tend to reach these tools instead of a search engine. But Google is the main incumbent here and it will figure out a way to monetise Bard otherwise they are dead in the water. You can see now Bard returning URLs, and things like OpenAI’s “custom assistants” will also direct customers to you. It’s just that they will not be using a search box, but a chat session.
It’s now the era of “Chatbot Engine Optimisation”.
3 Likes
I continue to be mystified why a purpose-built and optimized search engine like Google’s would not stand head and shoulders above an LLM which knows what words, phrases, and sentences correlate but not what correlates with the real world. 
I would say that if given a prompt the LLM output successfully correlates with phrases and sentences, then it is effectively correlating with the real world answer to the question because sentences built by humans refer to the real world, we humans are not stochastic monkeys. Of course it is not a perfect answer because it is not driven by any reasoning logic but LLMs capture some aspect of our thinking. But not being perfect, it can be good enough. Mixing this (potentially hallucinated) output with indexed real world facts is what Google or Microsoft are hard at work.
You can use Kagi now.
Meh. SEO is basically marketing. They’ll relabel it to AIO or something, bump the price by a laughable factor and just switch out some of their automatic tools to get the same effect.
It doesn’t matter what the truth about products is, the biggest spenders are still going to be the biggest sellers.
2 Likes
+1
You have correctly identified the reason I’ll not have much to do with an LLM.
The infinite monkey theorem states that if you let a monkey hit the keys of a typewriter at random an infinite amount of times, eventually the monkey will type out the entire works of Shakespeare.
An LLM shortcuts this by simply ingesting the complete works of Shakespeare and then regurgitating them. I get a chuckle out of all the ways people try to justify their excitement over, and use of, LLMs.