South Korea is in the midst of a rapid AI race, marked by a surge of new large language models (LLMs) released by leading technology firms including Kakao, NAVER, LG, SK Telecom, Korea Telecom, and the rising startup Upstage. Spurred by robust government investment and a national digital strategy, these companies are competing to push the boundaries of sovereign AI innovation at an unprecedented pace.
With this wave of new LLMs entering the market, questions around AI safety, security, and ethical resilience are more important than ever. To address these critical concerns, Vulcan presents the latest results from its X-Bench red teaming assessment, focused exclusively on the Korean foundation models.
X-Bench, Vulcan’s exploit benchmark that leverages advanced adversarial testing framework, probes LLMs with over 1,000 adversarial prompts in Korean, spanning 19 threat categories and more than 30 attack techniques. Vulcan X-Bench exposes vulnerabilities related to biased responses, sensitive data leakage, and inappropriate content generation.
This assessment covers the newest models from Kakao (Kanana 1.5), LG (EXAONE 3.5 and 4.0), NAVER (HyperCLOVA X SEED), SK Telecom (A.X 3.1 Light and A.X 4.0), Korea Telecom (Mi:dm 2.0), and Upstage (Solar Pro 2). These evaluations measure the Attack Success Rate (ASR), quantifying the percentage of adversarial prompts that models failed to neutralize.
Key findings:
- Bias Risks: Occupational bias emerged as a significant challenge, with some models demonstrating ASRs exceeding 70%, indicating substantial vulnerability. Bias based on age, gender, geographic origin, socioeconomic status, and racial characteristics also displayed concerning rates above 50% in specific models.
- Sensitive Data Exposure: Several models showed substantial vulnerabilities, particularly concerning financial data exposure, where ASRs ranged between 20% to over 60%. Select models are more prone to personally identifiable information (PII) data leakage risks.
- Harmful Content Generation: Some models remain susceptible to generating outputs involving violent crimes, suicide, specialized advice, and related harms, with ASRs nearing or surpassing 45%.
As this competitive landscape evolves, Vulcan remains committed to ongoing, transparent security evaluations and actionable guidance for enterprises, developers, and policymakers—helping Korea realize its vision as a global leader in secure, responsible AI.
“Our mission is clear—supporting Korean AI leadership means prioritizing robust security and resilience standards,” stated Alex Leung, co-founder of AIFT. “X-Bench raises awareness and empowers organizations to proactively identify and mitigate risks, fostering safer, more trustworthy AI deployments across industries.”
