Duke Study: GPT-5 Ranks AR Scene Quality Close to Humans

A Duke University team has put a HARD number on a question every augmented reality developer eventually FACES: can an AI model tell when a virtual object looks WRONG in a real room? In a preprint posted to arXiv on September 30 and slated for ACM VRST 2026 in Sendai, Japan (November 16–18), researchers tested ELEVEN commercial vision-language models against HUMAN raters. The BEST result, from GPT-5 on VIDEO, reached a Spearman rank correlation of 0.8695 with the human panel. MIXED reported the study on October 4; the FULL paper fills in the DETAILS its abstract leaves out.

Meta Quest 3 headset on a store shelf, the device used to record the RateAR video set
Every RateAR video and the closed-loop user study ran on Meta Quest 3. Photo: Kyu3a via Wikimedia Commons, CC BY-SA 4.0.

What RateAR Actually Measures

The paper, by Elias Rotondo, Lin Duan, Yanming Xiu, Sangjun Eom, Conrad Li and Maria Gorlatova, introduces RateAR, a PUBLIC benchmark of 321 AR images and 112 AR videos. The stills come from the earlier DiverseAR+ set and were captured on Android phones (250), Apple Vision Pro (40) and HoloLens 2 (31). The videos were all recorded on Meta QUEST 3. Five annotators scored each sample on three factors that BREAK immersion: PLACEMENT plausibility, SIZE appropriateness and SHADOW realism. That produced 6,495 RATINGS, with inter-rater reliability of .95, .94 and .90 respectively.

The models tested were Gemini 2.5 Flash, 2.5 Pro, 3.5 Flash and 3.1 Pro; OpenAI o3, GPT-5, GPT-5.4-Mini and GPT-5.4; Grok 4.3 and 4.5; and Meta’s Muse Spark 1.1. All ran ZERO-SHOT through PUBLIC APIs. Thirteen image and video quality models fine-tuned on part of RateAR served as BASELINES.

The Numbers Behind the Headline

  • Video BEATS stills. GPT-5 scored 0.8458 on video placement, 0.8206 on size and 0.8695 on shadows. The authors say temporal cues and multiple viewpoints give models MORE evidence than a single frame.
  • Shadows remain the HARDEST still-image task. The best image shadow score was 0.6075 (GPT-5.4). The strong video shadow result comes with a CAVEAT the authors state themselves: the video set mostly showed only two states, a correct shadow or none.
  • Results VARY widely by model. GPT-5.4-Mini fell to 0.0436 on image size, while Muse Spark 1.1 posted 0.8015 on video placement. A model that is strong on one factor can still MISS on another.
  • Baselines TRAIL. Fine-tuned video quality models scored near ZERO on shadow realism. The authors attribute the VLM advantage to CONTEXT. GPT models, for example, judged a floating balloon or a toy-sized car in an office as PLAUSIBLE.

Rank correlation measures whether a model ORDERS scenes from worst to best the way PEOPLE do. It does NOT show that the model matched anyone’s exact SCORE. That is the correct way to read the 0.87 figure.

From Grader to Fixer

The second half of the work CLOSES the loop. A Unity app on Quest 3 captures the scene and sends it over 5 GHz Wi-Fi to a MacBook Air edge server. The server queries OpenAI’s o3 model, which returns NUMERIC offsets and scale factors that MOVE or RESIZE the virtual object automatically. Examples include lowering a floating microwave to the floor and shrinking a car to fit a toy city.

Apple Vision Pro on a retail display stand with its battery pack
Apple Vision Pro captures make up 40 of RateAR’s 321 images. Photo: Seasider53 via Wikimedia Commons, CC BY 4.0.

Across 84 trials with 21 participants, 45 trials IMPROVED (53.6%), 38 showed no change and just ONE got WORSE. Many unchanged trials started at the maximum score already. In the survey, 19 of 21 participants (90.5%) agreed the system improved placement or size. LATENCY is the weak point. Image-based fixes took 4.43 to 5.84 seconds and video-based fixes 7.63 to 9.62 seconds. ONLY 33.3% of participants were satisfied with video-mode speed, compared with 61.9% for image mode. The system also STRUGGLED when the virtual object was partly occluded. Shadow correction was LEFT OUT of the deployed prototype entirely.

Why It Matters for XR Developers

Platform owners are PUSHING mixed reality as a core use case, from Quest passthrough to Vision Pro and Android XR. That makes object anchoring a QUALITY problem at scale, and human test panels do NOT scale with every BUILD. A model that ranks scenes roughly the way a panel would could FLAG bad placement or scale during development. That is a REALISTIC near-term role.

The LIMITS are significant. The benchmark is the authors’ own, the user study is SMALL and used a fixed before-and-after order, and nine-second cloud round trips cannot correct content in REAL TIME. The authors name fine-tuned ON-DEVICE models as the next step. Disclosure: the work was funded in part by NSF, DARPA, Cisco and a Meta Research Award. For now, RateAR is a useful BENCHMARK for automated AR quality checks, not yet a PRODUCTION feature. Read the full paper on arXiv.

1 thought on “Duke Study: GPT-5 Ranks AR Scene Quality Close to Humans”

Leave a Comment