News

2026-07-17 14:42:05 +0000

Our paper, “Decoding the Trade-off: A Large-Scale Analysis of Latency and Stability in LLM-based Speech Translation Cascades,” has been accepted at INTERSPEECH 2026.

The paper investigates a practical but often overlooked challenge in real-time speech translation: reducing latency does not always make a system faster. In cloud-based cascaded systems, aggressively shortening speech segments increases the number of ASR and machine translation requests. Once the request rate exceeds the system’s processing capacity, a backlog accumulates and the apparent fast configuration becomes slower over time.

To analyze this behavior, we developed RST, a reproducible Korean-to-English real-time subtitle pipeline that separately measures time-to-first-text and time-to-stable-text. This distinction makes it possible to evaluate both how quickly users first see a subtitle and how long it takes for the final translation to stabilize.

Our large-scale evaluation on the full Korean-to-English FLEURS corpus identifies a clear stability boundary around a median real-time factor of 1. Below this threshold, the system can maintain stable processing. Above it, queueing delay continues to grow and eventually dominates the total latency.

The study also shows that Whisper prompt conditioning can be unreliable under short, frequently segmented audio. Compared with prompt-free decoding, prompt conditioning increased Korean character error rate by approximately 14 to 17 percentage points and, in some cases, leaked prompt instructions into the transcription.

The results suggest that throughput stability must be treated as a prerequisite for low-latency speech translation. Rather than selecting the shortest possible segment, a practical system should use the most aggressive endpointing configuration that remains within the throughput-stable regime.

Congratulations to Shinyoung Sun, a Ph.D. student in our lab and the first author of this work, on this achievement. The paper will be presented in the “Robust and Real-World ASR Systems” oral session at INTERSPEECH 2026.

KOCCA Culture, Sports and Tourism R&D Grant

2026-06-15 00:00:00 +0000

We have been selected for the 2026 Culture, Sports and Tourism R&D Program supported by the Korea Creative Content Agency, a national research initiative aimed at advancing next-generation technologies for Korea’s cultural and creative industries.

This project focuses on developing a Physics-Informed AI platform for the real-time simulation, production, and validation of mega-scale content, including K-POP concerts, festivals, exhibitions, and large-scale media events. The research aims to address the limitations of existing pre-visualization tools, which mainly support visual planning but provide limited capabilities for physical safety assessment, automated production assistance, and real–virtual synchronization.

By integrating Physics-Informed AI, real-time physics engines, digital twins, and Hybrid Neural Rendering, the project explores a new production paradigm in which stage structures, lighting systems, cameras, LED displays, and moving equipment can be simulated, optimized, and validated before and during actual productions.

The project will develop a physics-aware content asset database containing more than 500 stage structures and production devices, with properties such as mass, center of gravity, load capacity, friction, collision range, and motion limits. These assets will be used to train PI-AI models capable of estimating and correcting discrepancies between simulated behavior and real-world measurements.

The research also includes real-time safety validation technologies for detecting excessive loads, structural instability, collisions, and equipment operating-limit violations. The system targets safety parameter inference within 100 milliseconds, enabling immediate risk assessment during simulation and production.

In addition, AI-assisted production tools will be developed to recommend camera positions, lighting configurations, equipment layouts, and directing sequences while considering visibility, illumination uniformity, physical constraints, and operational safety.

A Hybrid Neural Rendering Digital Twin will combine 3D Gaussian Splatting for photorealistic venue reconstruction with mesh-based dynamic simulation and AI-based relighting. This approach is designed to preserve both the visual realism of physical venues and the physical consistency required for interactive simulation.

The platform will support bidirectional synchronization between Unity-based virtual environments and real stage equipment through protocols such as sACN, Art-Net, FreeD, Modbus, MQTT, OSC, and SMPTE timecode. It will also include automatic calibration methods for aligning physical and virtual stages and reducing discrepancies between simulated and real-world environments.

The proposed system will be validated through real-world K-POP concerts and large-scale event productions. These demonstrations will assess the platform’s ability to reduce production risks, improve pre-visualization workflows, support real-time safety monitoring, and enable synchronized offline, online, and metaverse-based performances.

This work is expected to contribute not only to improving the safety and efficiency of mega-scale content production, but also to establishing a foundational platform for the digital transformation of K-content production and the global expansion of AI-driven performance technologies.

NRF Early Career Research Grant (National Research Foundation)

2026-03-24 00:00:00 +0000

We have been selected for the National Research Foundation Early Career Research Program (우수신진연구), a highly competitive national research grant supporting emerging researchers in Korea.

This project focuses on developing a hallucination-free on-device egocentric multimodal AI agent equipped with self-correction mechanisms. This research aims to address one of the most critical challenges in modern AI systems: ensuring reliability and trustworthiness in real-world, context-aware environments.

By integrating actor-validator architectures, RLAIF-based alignment, and dynamic precision optimization for on-device deployment, the project explores a new paradigm where AI systems can autonomously evaluate and refine their own outputs in real time.

This work is expected to contribute not only to advancing multimodal AI and on-device intelligence, but also to establishing a foundational framework for trustworthy AI systems that can safely operate in human-centered environments.

WCCA-AK (ICCV Workshop 2025)

2025-10-20 00:00:00 +0000

WCCA-AK is a large-scale dataset of 3D scans and multi-view images capturing 100 haute couture garments by André Kim (1962–2010), one of Korea’s most iconic fashion designers. This dataset bridges computer vision research and cultural heritage preservation, enabling both faithful documentation of historical artifacts and generative exploration of artistic vision.

NVIDIA Academic Grant for Researchers (NVIDIA)

2025-09-24 00:00:00 +0000

The NVIDIA Academic Grant Program supports our research on building a Validator LLM designed to ensure logical consistency in generative AI explanations. This project focuses on moving beyond surface-level fluency in large language models by introducing a structured validation mechanism that can assess and refine the reasoning process itself.

Our approach adopts a dual-model architecture, where an actor LLM generates responses and explanations, and a validator LLM evaluates their logical coherence. By incorporating reinforcement learning from AI feedback (RLAIF), the system iteratively improves its ability to detect inconsistencies and hallucinations without relying solely on human supervision. In addition, we explore multi-pass reasoning and cross-model verification strategies to enhance robustness and reliability.

On the systems side, this research leverages NVIDIA’s optimized AI stack, including NeMo Framework and TensorRT-LLM, to ensure that the proposed validation pipeline is not only theoretically sound but also deployable in real-world environments with practical latency and scalability constraints.

Ultimately, this work aims to establish a foundation for trustworthy generative AI systems that can self-evaluate and provide explanations users can rely on, addressing one of the most critical challenges in the deployment of large-scale AI models.

Project Aria (Meta Reality Lab)

2025-07-16 00:00:00 +0000

Project Aria is a research initiative led by Meta Reality Labs aiming to develop next-generation augmented reality technologies. Through our academic partnership with Meta Reality Labs, we are currently developing an Egocentric Multimodal AI Agent leveraging Project Aria’s advanced wearable device, Aria Glass. By integrating real-time visual streams from cameras, Visual SLAM for precise spatial understanding, and sophisticated eye-tracking data, this collaboration seeks to enable context-aware, personalized AI interactions. Our goal is to explore how multimodal AI can interpret and seamlessly respond to users’ real-world environments, providing foundational insights for future AR and wearable computing applications.

AgentVox

2025-06-28 00:00:00 +0000

AgentVox — Edge-based voice assistant using Gemma LLM with Speech-to-Text and Text-to-Speech capabilities — officially Released!

Features:

Speech Recognition (STT): High-speed speech recognition using RealtimeSTT
Conversational AI (LLM): Local LLM based on Llama.cpp (Gemma 3 12B)
Speech Synthesis (TTS): Fast response with RealtimeTTS
Complete Offline Operation: All processing is done locally, ensuring privacy

DocsRay

2025-06-07 00:00:00 +0000

DocsRay — Lightweight PDF Q&A tool powered by RAG (Retrieval-Augmented Generation) with MCP (Model Context Protocol) Support — officially Released!

Features seamless MCP (Model Context Protocol) integration with Claude Desktop, comprehensive directory management capabilities, visual content analysis, and intelligent hybrid OCR system.

Now you can upload any type of documents including HWP!

Try out our online demo at docsray.com!

WCCA@ICCV2025

2025-05-23 00:00:00 +0000

We are delighted to announce that our workshop proposal to ICCV 2025 got approved!

The Workshop on Cultural Continuity of Artists (WCCA) brings together researchers, creators, and cultural institutions to explore how computer vision, multimodal AI, and XR technologies can safeguard and reinterpret artistic legacies. Our inaugural edition, co‑located with ICCV 2025, highlights the visionary South Korean fashion designer André Kim and introduces a rich, newly curated dataset from his archives.

Find out more at WCCA2025 Website!

QueryDoc

2025-05-05 00:00:00 +0000

QueryDoc — Lightweight PDF Q&A tool powered by RAG (Retrieval-Augmented Generation) — officially Released!

Just upload a PDF and start asking questions.

Evangel : Catholic Priest AI powered by QueryDoc
VerNova: Presbyterian Pastor AI powered by QueryDoc

Welcome

2024-08-09 00:00:00 +0000

Welcome to MIMIC’s offical website.