OpenAI has launched the web crawler GPTBot to improve its artificial intelligence models.
“Webpages crawled with the GPTBot user agent may be used to improve future models, sources that require access to paywalls, collect personally identifiable information (PII) It is filtered to remove sources that are known to do so or that contain text that violates our policies,” the company said. in a post on the website.
“Allowing GPTBot to access your site may improve the accuracy of your AI model and improve general functionality and safety,” OpenAI wrote.
A web crawler is a type of bot.
What is AI?
The OpenAI ChatGPT logo will be displayed on your mobile phone. (Jaap Arriens/NurPhoto via Getty Images)
Internet company Cloudflare said they are typically run by search engines, which index the content of websites so that sites appear in search results.
Crawling means using software to automatically access websites and retrieve data, hence the name “web crawler”.
OpenAI also provided instructions to partially or fully ban GPTBot from accessing its website.
OpenAI Inc. CEO Sam Altman speaks with media representatives at the Allen & Co. Media and Technology Conference in Sun Valley, Idaho, Wednesday, July 12, 2023. (David Paul Morris/Bloomberg via Getty Images)
What is Chat PT?
Websites can block crawler IP addresses or add GPTBot to their site’s robots.txt file. This file basically tells web crawlers about what they can access from your site.
It says, “To allow GPTBot to access only parts of your site, you can add a GPTBot token to your site’s robots.txt.”
A man is seen using the OpenAI ChatGPT artificial intelligence chat website. (Jaap Arriens/NurPhoto via Getty Images)
CLICK HERE TO GET THE FOX NEWS APP
“For OpenAI’s crawlers, calls to the website are made from the IP address blocks listed on the OpenAI website,” OpenAI concluded.
Notably, AI companies, including OpenAI, have signed an agreement with the White House to develop a watermarking system to tell internet users whether something was generated by AI. However, the organizations have not committed to stop using Internet data for training.
