ChatGPT helps people write codes but OpenAI has introduced CriticGPT, a new AI model based on GPT-4 designed to identify mistakes in the codes generated by its AI chatbot. The tool aims to improve the alignment process in AI systems using a technique known as Reinforcement Learning from Human Feedback (RLHF) which will eventually improve the accuracy of large-scale language model outputs.
The company discovered that when users obtain help from CriticGPT to examine ChatGPT code, they outperform those without assistance 60 percent of the time.
“We are beginning the work to integrate CriticGPT-like models into our RLHF labeling pipeline, providing our trainers with explicit AI assistance,” the company wrote on its blogspot.
Through RLHF, ChatGPT's GPT-4 models are intended to be informative and engaging. AI trainers compare and rate the quality of various responses as part of this procedure. As ChatGPT's reasoning gets better, its errors get more subtle, making it more difficult for trainers to spot the errors.
“This is a fundamental limitation of RLHF, and it may make it increasingly difficult to align models as they gradually become more knowledgeable than any person that could provide feedback,” OpenAI wrote.
However, just like human suggestions, the CriticGPT’s suggestions are also not always correct but they can help trainers to catch more problems with model-written answers than they would without AI-help. In trials, teams using CriticGPT produced more detailed critiques and identified fewer false positives than individuals working alone. “A second random trainer preferred critiques from the Human+CriticGPT team over those from an unassisted person more than 60% of the time,” wrote OpenAI.
According to OpenAI, CriticGPT showed a 63 percent improvement over ChatGPT in detecting code mistakes. However, the model has certain limitations. It was trained on short ChatGPT answers and requires additional refinement to handle longer and more complex tasks. Furthermore, although models continue to hallucinate and trainers occasionally make labelling mistakes, the focus on single-point errors must be expanded to address errors spread across various portions of an answer.
The new AI model CriticGPT will assist human trainers in producing better RLHF data for GPT-4. Also, the company intends to grow this work further.
Comments
0 comment