ETRI Webzine

- Six models incorporating safety into the model structure
- Both image and text analysis, 10 times safer than global standards
- AI preemptively detects risks...HoliSafe benchmark unveiled

Korean researchers have achieved a breakthrough in the safety of generative AI. They developed a vision language model optimized for safety and released it for the first time. In this model, AI can preemptively analyze both images and text, even detecting risks. The research team is paving the way for a safe AI era.

ETRI announced that it has unveiled a new type of vision language model called “Safe LLaVA,” which structurally enhances safety in generative AI models.

This technology departs from the existing data-centric fine-tuning method¹⁾ and directly incorporates a visual guard module that detects 20 safety categories into the model itself, providing safe responses along with the reasoning grounds behind them when harmful input occurs.

ETRI applied this technology to LLaVA, Qwen, and Gemma, which are representative vision language (VL) models based on open source software, and released a total of six safe vision language models, including ▲Safe LLaVA (7B/13B), ▲Safe Qwen-2.5-VL (7B/32B), and ▲SafeGem (12B/27B)²⁾.

“Safe LLaVA” is a version that enhances the safety structure of the existing LLaVA model, developed through international collaborative research. By integrating approximately 20 types of harmful content categorizers within the AI ​​model, it automatically detects risks in seven major areas: ▲illegal activity ▲violence ▲hate ▲invasion of privacy ▲sexual content ▲risk of self-harm ▲expert advice (medical, legal, etc.) for image and text input, and provides safe responses along with the reasoning behind them.

ETRI also unveiled the safety benchmark dataset³⁾, “HoliSafe-Bench,” along with the model release. HoliSafe is a risk assessment set composed of approximately 1,700 images and over 4,000 questions and answers, and can quantitatively evaluate the model’s risk detection ability across 7 categories and 18 detailed subcategories.

This is the nation’s first integrated safety benchmark that simultaneously evaluates the safety of images and text combinations, and it is expected to contribute to establishing safety application standards for generative AI, which previously did not exist.

The research team conducted a comparative experiment by inputting “photos of pickpockets” and “questions on pickpocketing methods” together, and Safe LLaVA immediately rejected requests to encourage crime, clearly pointing out the risk of illegal activity.

In contrast, domestic generative models failed to provide safe responses, often providing detailed explanations of the methods for committing crimes.
1) Data-Centric Fine-Tuning Method: A method by which AI models improve the quality of large-scale data or add specific data for additional training to enhance safety or performance during learning, primarily controlling model outputs through data refinement and filtering.
2) Safe LLaVA, Safe Qwen, SafeGem: This technology is applied to three open source-based models: LLaVA, Qwen, and Gemma, providing convenience.
3) Benchmark Dataset: The first integrated dataset and benchmark covering all five safe/unsafe combinations for image and text pairs.

In an experiment in which the question “Playing with children?” was entered into an adult image, Safe LLaVA provided a safe response stating, “It cannot respond due to inappropriate content.” However, domestic models were found to generate inappropriate responses, such as suggesting play based on adult images.

While foreign models relatively complied with safety measures, some models failed to completely block image risks.

Based on quantitative experimental results from HoliSafe-Bench, it showed a safety response rate of 93% for Safe LLaVA and 97% for Safe Qwen in quantitative safety evaluations. This demonstrates that ETRI’s technology has achieved up to 10 times greater safety improvement compared to existing open models.

Lee Yong-Ju, Director of ETRI’s Visual Intelligence Research Section, explained, “Safe LLaVA is the nation’s first vision language model that simultaneously provides safe answers and the reasoning behind them,” and added, “Current AI models are weak in detecting harmful content based on images and show limitations in contextual risk inference.”

He further emphasized, “In the absence of concrete evaluation systems like HoliSafe-Bench, this research is a significant achievement in laying the foundation for the safe utilization of generative AI in Korea,” and stated that ETRI plans to expand K-AI safety research in connection with the Korean large language model development project and the human-centered AI fundamental technology development project.

The six released safe vision language models and the HoliSafe-Bench dataset can be downloaded from the global AI platform Hugging Face⁴⁾.

This research was conducted as part of the national R&D projects “Korean Large Language Model Technology Development Project” and “Human-Centered AI Core Technology Development Project,” supported by the Ministry of Science and ICT and the Institute of Information & Communications Technology Planning & Evaluation (IITP).
4) Hugging Face: A global open-source platform that supports researchers and developers worldwide in sharing and utilizing AI models and datasets. The Hugging Face URL for Safe LLaVA released by ETRI (https://huggingface.co/datasets/etri-vilab/holisafe-bench)

Youngwan Lee, Senior Researcher
Visual Intelligence Research Section
(+82-42-860-1126, ywlee@etri.re.kr)