Has anyone figured out a prompt for on-page SEO audits that use live-crawling?
I can occasionally get it to work, but it’s not exact:
prompt:
[visit mauipestcontrol.com and put together an on page seo audit]
There’s literally hundreds of very successful tools out there, predating ChatGPT, that are purpose built to perform various SEO lookups and reports and audits. The most valuable and reputable use up to date information on the latest updates, changes, practices, etc. Meanwhile, ChatGPT is built entirely on a corpus of documents as the basis for all of its ‘knowledge’ that was last updated in 2021, and much of it is older.
Use the right tools for the right jobs. AI can be incredibly powerful in the right uses, and can be worse than useless (as in actually makes things worse) when applied to the wrong kinds of tasks.
3 Likes
I understand all of that, but that’s not what I asked.
The advantage of having AI print these audits is a huge timesaver and can create a digestible document for non-tech-savvy business owners. I have tons of SEO tools, but none can do that.
I am sure it will be possible soon.
Appreciate the input though!
1 Like
Right… But now think that thought through to the obvious conclusion. Every single idiot in the entire world will have the exact same access to the exact same AI you do, and can input the exact same prompts. What will they need you for?
You have a very short window of opportunity, timewise, so build a reputation for a level of human analysis that no AI can rival. To set a bar that AI alone won’t reach for a few more years. Because when AI alone is competent to do a really useful audit, it will do it automatically before the content is created and will be a plugin on WordPress, eliminating the need for SEO input at all since it will all be done in the planning stages, or at the click of a button.
Once that small window of opportunity closes, and it will close quickly and without warning, its too late. The AI will have the market and the press and the buzz such that they won’t even try other services. If you want even the chance to be compared, to be in their consideration, you have to be the first-mover. You need to be what they benchmark AI against.
I’ve been in SEO a long, long time now. What I offer isn’t what is in the tools, but rather, the understanding and insights built on decades of experience. Not just the understanding of the metrics, of SEO practices, of the engines and the engineers, but far more importantly, of my clients. I know what works for others that won’t work for them, and conversely, what will work for them that wouldn’t work for others with a different situation or workflow or process or brand or… You get the idea.
My clients use a huge variety of different tools, especially the larger brands and the multi-nationals. They all have their own balance of metrics they use, and with different priorities too. And that is precisely where automation, of all kinds, falls over. Clients are every bit as unique, as individual, and as changeable, as we individuals are. Play to that if you want to survive and not be replaced with AI.
3 Likes
Yes, it is possible to perform live crawling for an SEO audit. Live crawling involves analyzing a website in real-time to gather data about its structure, content, and other relevant factors that impact its search engine optimization (SEO) performance.
To conduct a live crawl for an SEO audit, you can use various tools and software available in the market. These tools simulate search engine bots and visit your website to collect data on different SEO aspects. They analyze factors such as website architecture, URL structure, internal linking, meta tags, header tags, keyword usage, page load speed, mobile-friendliness, and more.
Live crawling tools typically generate comprehensive reports that highlight areas for improvement, potential issues, and optimization opportunities. These reports can help you identify technical SEO issues, content gaps, broken links, duplicate content, and other factors that may affect your website’s visibility in search engine results.
By conducting a live crawl as part of your SEO audit, you can gain valuable insights into your website’s current SEO performance and discover areas where you can make improvements to enhance its visibility and ranking in search engines.
If that’s what you wish to believe, go right ahead.
It won’t be right, or true.
When real SEOs talk about a ‘live crawl’, this is not fetching one page, but rather, crawling methodically through the entire site, finding all of the pages, images, scripts and more, by analysis of the links on each page, in the sitemap, etc, and adding them to the task. It takes a significant amount of memory. ChatGPT does not have that memory, by design.
Just fetching one single URL sometimes exceeds the entire working memory of ChatGPT, especially when using GPT3.5 which only allows for 4,000 tokens, (roughly 3,000 words), and won’t include the CSS file, any JS files, Image files, etc, etc. That’s not really a problem because there are hundreds of dedicated SEO tools available built specifically for the task, that vastly outperform the limitations (and accuracy) of ChatGPT.
For a full crawl, you might go with ScreamingFrog (one of the most highly recommended and regarded), or go as low-tech as using the old but trusty Xenu’s Link Sleuth (which can crawl to any specified depth slowly but surely and is decades old), or any of hundreds of tools in between at various prices (including other free ones).
For just crawling links and analysis of site structure you might look at Majestic, or Ahrefs, or Semrush, or others, all of which have a specific SEO focus to them, built by SEOs that at least know enough to know that ChatGPT cannot do this.
3 Likes
May be you’re right at this.
I respect everyone suggestion.
whilst i understand your sentiment entirely, when a guy wants to create his own seo tool and perhaps be the first to create it in a way that works for him you sort of shoot him down? and say use existing tools 
I like to see folk eperimenting, whilst it can actually waste a lot of time I concede, we dont fully understand AI so cant really measure every aspect of it against what has happened previously. He may be able to train his model to use that seo reporting on a template that work particularly well for something. I mean maybe not, Im thinking out aloud as much as anything but i can imagine training a model to check certain things i may or may not have done to my website and make a general comparison of speciic things that i want comparing that may not be in the semrush/ahref models . rheatorical
Oh you could build an AI that was trained extensively in/on SEO, coding practices, etc, and give it the memory necessary to compare maybe 20 or so different pages, while also remembering the whole site layout, general ontology and information architecture, and to look up and understand link data in still other tools. Cost you around 10 million dollars or so, but it is doable.
What you can’t do is take an AI that wasn’t specifically trained in the right way on the right data (GPT itself), then mounted into a specific chat application (ChatGPT) that actually limits what GPT as a whole can do, and expect it to work. It doesn’t have the capabilities needed.
By licensing GPT4 itself, the raw engine and AI outside of the chat-limited application of ChatGPT, and using it as part of a broader program that could grab data and pass it to GPT in manageable chunks, you can get something that will do… Just as the latest versions of the tools already did, sharing the thousands of hours necessary for that work among their whole dev team.
For anyone trying to bootstrap it alone, well, if you have a passive income so that you can dedicate every working minute to the project, and still afford the licensing of GPT, you could probably get something of passing quality in a year. If only able to spend non-working hours on it though, you’d be extending that a long, long time.
2 Likes
All that is obvious sure, but we arent talking about a guy wanting to create an seo tool empre, just someone who wants to check certain things on his website, it isnt always about taking over the world. Theres always another perspective. My point really about people trying to get it to do what they want by trying it. It could actually be something specific , not scraping an entire source code
On a grander scale, of course I agree, hard not to agree, you know your stuff., just voice my own thoughts here and there bcause theres always an outlier user. On that big scale Google maybe wouldnt have started because we already had Yahoo as everything seems to cost 200 times more than it did 30 years ago in the spaces, so much for the price of everything coming down with progress eh? 
Far removed from a few geeks typing code for hours on end and blue/black screens
1 Like
Great info and answered a ton of my questions with that one sentence. Also been doing SEO for 2 decades and I do exactly what you describe and its works great. This is another tool in my toolbox. It has quickly become one of my favs but it is only one of them. Getting close but If I had to choose only one screaming frog answers too many question in a great interface to give up. I have barley scratched the surface and AIPRM and it surprise’s me everyday with what I can do and streamline my process shaving hours off some of the things I hated to do. Bonus is there is so much to learn and play with.
3 Likes
If youre like me, youre wearing 6 different hats, it will be nice to only need 3 eh
1 Like