Head-Mounted Display with Track and Project/动态追踪与投影增强的头戴显示器

Background/背景

CT图像是利用X射线对人体某一平面进行扫描得到的图像。在一组相互平行的平面扫描可得到一组CT图像，其表达的三维信息可以帮助人们了解人体内部状态。但要理解一组CT图像，要求医生受到专业的训练，未受专业训练的人（如，患者及其家属）很难理解其所表达的三维信息。三维重建技术能够帮助更好的分析和存储三维信息，但并不能帮助人们理解CT图像，因为我们往往关心的不是从外部看到人体三维模型，而是其包含的人体内部信息。虚拟现实/增强现实（VR/AR）技术提供了具有潜力的解决方案，但现有的头戴式显示器受到显示信号线或无线传输延迟的限制，影响用户体验。

Purpose/目的

我们的研究工作旨在设计一种交互式可视化方案，帮助人们理解CT图像。同时，不受传统VR/AR的头戴显示器的显示信号线限制，同时具有用户端重量轻、无能耗、低延迟等优势。

Method/方法

我们的解决方案ARSlice，将空间增强现实（SAR）技术与头戴显示器相结合，分为投影端和用户端两部分。投影端使用动态追踪技术，定位用户头部位置和姿态，根据姿态生成投影图像，并根据位置高速调整焦距，使得投影图像能够清晰地投影到用户端的投影板上；用户端是纯光学设备，负责让用户在三维空间中看到投影板上的图像。用户端通过光学投影获取信息，无需显示信号线或无线传输；也无需关心数据生成问题，因此用户端不产生能耗；另外，通过高效的动态追踪算法，使得系统整体延迟符合人眼要求。

Fig 1 Overview of the proposed ARSlice prototype. The pro- jector end generates the virtual content and augments the user end, and the user end shows the virtual content in the 3D space. The user-end equipment is a pure optical device, and thus is light weight and untethered, and there is no need to worry about en- ergy consumption.

Result/结果

我们设计制作了设备原型，用户可以在三维空间中看到CT图像切面，图像内容随着用户头部运动而变化，如同用户眼前有一个平面对人体进行切割显示。设备提供了近似的6自由度，用户端可以在距离投影端0.5-2米范围内移动；用户端可以在上下左右各方向约45°的幅度内转动；用户端的视角范围约75°；整体刷新率在50Hz以上。通过用户体验调研，大多数用户认为ARSlice系统可以帮助理解CT图像。

Fig 2 Experiment with varying focal length. (a) Original position. (b) Distant position. (c) Near position.

Fig 3 Experiment with users’ in-place movement. The yellow dotted lines indicate the projected region. (a) Original position. (b) Leaning to the left. (c) Leaning to the right. (d) Turning to the left. (e) Turning to the right. (f) Looking up. (g) Looking down.

Fig 4 (a)(b) Experiment of visualizing CT images with the AR experience. When (b) the user moves forward or backward, the user can see (a) different CT images on their corresponding position in the 3D space. At the same time, real-world scenes can also be seen. Note that the CT images seem to be out of focus, because the user’s perspective camera is auto-focusing on real-world scenes. (c)(d) Experiment of visualizing CT images with the VR experience. When (d) the user moves forward or backward, the user can see (c) different CT images on their corresponding position in the 3D space. Without seeing real-world scenes, CT images can be seen more clearly and correctly focused.

Conclusions/结论

我们设计了一种帮助理解CT图像的交互式系统方案ARSlice。能够在用户端无显示信号线、无能耗的条件下实现低延迟的虚拟现实/增强现实体验，可有效帮助用户理解CT图像。 论文对系统的一些技术指标的改进方案及可能的其他应用场景进行了讨论。后续工作将进一步提供给专业医生使用。

Video/视频

实验视频（视频时长91秒）

Reference/参考

1. Yuping Wang, Senwei Xie, Lihui Wang*, Hongjin Xu, Satoshi Tabata, Masatoshi Ishikawa. ARSlice: Head-Mounted Display Augmented with Dynamic Tracking and Projection, Journal of Computer Science and Technology 37(3), 666-679(2022) [DOI: 10.1007/s11390-022-2173-y]
2. Yuping Wang, Senwei Xie, Lihui Wang*, Hongjin Xu, Satoshi Tabata, Masatoshi Ishikawa. ARSlice: Head-Mounted Display Augmented with Dynamic Tracking and Projection, The 10th international conference on Computational Visual Media (CVM 2022) (Beijing , 2022.4.8) (link)
   selected published in Journal of Computer Science and Technology.

项目单位：广东省科学院半导体研究所、清华大学、东京大学、东京工业大学

———————————————————————————————
Abstract: Computed tomography (CT) generates cross-sectional images of the body. Visualizing CT images has been a challenging problem. The emergence of the augmented and virtual reality technology has provided promising solutions. However, existing solutions suffer from tethered display or wireless transmission latency. In this paper, we present ARSlice, a proof-of-concept prototype that can visualize CT images in an untethered manner without wireless transmission latency. Our ARSlice prototype consists of two parts, the user end and the projector end. By employing dynamic tracking and projection, the projector end can track the user-end equipment and project CT images onto it in real time. The user-end equipment is responsible for displaying these CT images into the 3D space. Its main feature is that the user-end equipment is a pure optical device with light weight, low cost, and no energy consumption. Our experiments demonstrate that our ARSlice prototype provides part of six degrees of freedom for the user, and a high frame rate. By interactively visualizing CT images into the 3D space, our ARSlice prototype can help untrained users better understand that CT images are slices of a body.

---