Unified Interaction-Ready 3D Scene Generation via LLM–RL Optimization

1Department of Computer Engineering, Sejong University, Seoul, Republic of Korea,

Overview

Interpolate start reference image.

Abstract

Language-driven 3D scene generation typically produces static layouts that neglect interaction requirements, often omit- ting task-implied objects and leaving user placement unde- fined. We introduce a unified framework for interaction-ready 3D environment generation that connects language under- standing, spatial construction, and immersive virtual reality (VR) human–robot interaction (HRI). Our pipeline integrates three core modules: a Language-Driven Scene Representation (LDSR) that builds a scene graph of explicit and task-implied objects; a VR-Aware Object Role Classification (VR-ORC) module that classifies object roles, including the anchor; and Plan2Place, a reinforcement-learning placement policy that arranges objects under relational, geometric, navigability, and affordance constraints. The user’s viewpoint is then initialized near the anchor, and the scene is deployed in VR with visual and haptic feedback. On ALFRED-based tasks, VR- ORC improves key- and inference-role F1 by 34–51 points over a rule-based reading of the LDSR scene graph, but its gain on the anchor role, which determines the viewpoint, is small (1.9–2.9 F1 points). The generated scenes contain every required object for 72%–96% of instructions per task category, against 4%–20% for Holodeck. On constraint sets from two held-out LLMs, Plan2Place places and keeps reachable more key objects than DFS, MILP, and Z3, at the cost of lower scene-level completeness than DFS or MILP on some sets. A user study of an earlier version of Plan2Place and a VR check of viewpoint initialization complement these results. These results suggest that interaction requirements can be built into scene generation rather than appended post hoc

RESULTS

A comparison of Our method and HOLODECK.

Room Type: a living room.

Instruction: Move a credit card from the coffee table to the striped armchair.

sample 2

1

Room Type: a kitchen.

Instruction: Put chilled bread on the counter.

sample 1

1



Haptic-based Human-Robot Interaction

Room Type: a living room

Instruction: Place a laptop on the dresser.

Top-Down

1

Robot

1

Human


Room Type: a living room

Instruction: Examine a credit card by the light of a floor lamp and then turn it off.

Top-Down

1

Robot

1

Human


Room Type: a kitchen

Instruction: Put a chilled bread on the counter.

Top-Down

1

Robot

1

Human


Room Type: a kitchen

Instruction: Put a heat tomato on the table.

Top-Down

1

Robot

1

Human


BibTeX

@article{vo2026,
  author    = {Anh, H. Vo and Sungyo, Lee, and Phil-Joong, Kim and Soo-Mi, Choi and Yong-Guk, Kim},
  title     = {Unified Interaction-Ready 3D Scene Generation via LLM–RL Optimization},
  journal   = {},
  year      = {2026},
}