On August 20, 2024, Google launched the Vertex AI Crawler for commercial users. With the help of this new AI crawler, webmasters can identify website crawler traffic. Vertex AI uses the Google-CloudVertexBot and Googlebot user agents but will only work when site admins make a request.
If you don't know what Vertex AI is, now is your chance to learn everything about it.
Google Vertex AI Crawler:
Vertex AI is a full-fledged platform for building and testing generative AIs. It uses AI Studio, foundation models, Gemini Pro, and Gemini Flash. Google is offering $300 worth of free credits to new customers.
Google has also launched a new crawler named Google-CloudVertexBot.
What Does It Do?
Google-CloudVertexBo t crawls websites for AI clients. In other words, it operates differently from traditional crawlers.
While other Googlebot crawlers automatically crawl websites, Google-CloudVertexBot only works when an AI user makes a request. It has multiple data stores, but each data store can save only one type of data.
There are two major types of website crawling and indexing:
According to Google's documentation, Vertex AI data stores access data from indexed websites. Commercial users can set up domains and search for crawlers.
Basic Website Crawling and Indexing:
Vertex AI uses a fraction of indexed content to crawl.
Advanced Website Crawling and Indexing:
Google-CloudVertexBot uses advanced crawling and indexing techniques. It is necessary to verify domain ownership before making a crawl request. There is also a crawl and indexing budget. The AI crawler will only crawl the webmasters' websites on request and will not access other websites that are not verified by the user.
Conclusion:
The Vertex AI crawler is different from the Googlebot crawler that automatically crawls and indexes websites. It is available only for commercial users and will only crawl websites verified by the account holder.
Share it with your friends and family.
You May Also Like These: