Hand-tracked 3D Data Selection of Point Clouds in XR
Selecting subsets of 3D data is essential for exploratory analysis, volume measurement and annotation, but it remains hard. Traditional methods stay in 2D interfaces and force you to translate between 2D projections and 3D relationships. Existing 3D approaches are mostly limited to simple primitives like spheres and boxes, and often don't scale to large datasets in real time. Point clouds, millions of 3D points from scans, astronomical surveys or medical imaging, are the typical case.
My master's thesis combines hand tracking in XR with GPU-accelerated selection: you reach into the data and select with your hands instead of a cursor. A shorter version was presented as a poster at EuroXR 2025.
Selection via Signed Distance Fields
Selection runs on the GPU using Signed Distance Fields (SDFs). A selection volume is stored as a compact 3D texture holding a distance function instead of a bounding box, which means:
- Selection volumes can have any shape, not just boxes and spheres
- Compute shaders do the point-in-volume test for all points in parallel, targeting the headset's 90Hz refresh rate The final system without any optimization ran at 70-80 FPS with up to 5 million points. Rendering, not selection, was the main bottleneck
Point positions, colors and selection states all stay in GPU memory to avoid CPU-GPU transfers, and Unity's indirect instanced rendering draws the points. Built in Unity with OpenXR, with custom compute shaders for the distance field math. It runs on standalone headsets (Meta Quest Pro and Quest 3) without controllers.
User Study
To compare the four selection techniques below, I ran a study with 28 participants, measuring completion time, accuracy and error rate, plus usability questionnaires.
Findings:
- Brushing (painting over points with your hands) was faster than shape-based selection and about 23% more accurate on average
- Participants preferred brushing's continuous feedback over discrete object manipulation
- Using only the hands, without virtual tools, scored highest on usability
Four Selection Techniques
All four techniques work with hand tracking only and share the same SDF framework. Two are brushing-based (hands, sphere) and two are geometry-based (convex hull, shapes). They differ in how direct they are, how precise, and how much thinking they take.
Painting over points directly with your hands
1. Direct Hand Brushing
You move your hand through the point cloud and every point inside the hand model gets selected. To avoid accidental selections (the "Midas touch" problem), it's only active while the other hand pinches. It scored high on usability, and participants described it as "painting in 3D space". They liked that points change color the moment they're selected. Under the hood this technique recalculates the SDF for the hand-model every frame
A sphere attached to the hand as a larger, more visible brush
2. Sphere-Based Brushing
The same idea with a sphere attached to your hand, which is easier to see. A pinch gesture changes the radius, so it works for both coarse and fine selections. It was the quickest way to select large, spread-out regions. Any mesh would work as a brush, I used a sphere because it's simple.
Placed points become the vertices, and the convex hull around them updates in real time
3. Convex Hull Selection
You place points with a pinch-and-draw gesture. The points are smoothed with a Laplacian filter, and the convex hull around them is converted to an SDF and becomes the selection volume, updated live as you add points. It sits between rigid primitives and freeform selection, but is limited to convex shapes.
Spheres, boxes and cylinders, spawned with a pinch and moved, rotated and scaled with both hands
4. Shape-Based Selection
You spawn a sphere, box or cylinder with a "pinch and pull" gesture: the distance between your hands sets the scale and the shape is centered between them. Afterwards you move, rotate and scale it with both hands. It gives precise control over the boundary, but in the study it took longer and more effort than brushing, since you're handling a tool instead of the data itself.
Manipulating a Selection
Grabbing a selection with both hands to move, rotate and scale it
A "double-fist" gesture grabs the whole point cloud and the current selection and implements the handlebar metaphor: the midpoint between your fists translates, the distance between them scales, and their orientation rotates.
Switching Techniques
The palm menu for switching techniques, changing settings and managing point clouds
A menu is attached to your hand and operated by poking, so switching between the four techniques or adjusting settings doesn't need a controller or a separate UI panel.
Limitations and Future Work
The convex hull technique only produces convex shapes, performance degrades beyond about 10 million points, and results depend on how well the headset tracks your hands. Future work includes hierarchical data structures for larger datasets, concave selection volumes, Boolean operations between selections, AI-assisted selection, and frustum culling and out-of-core streaming.
Tech Stack
- XR & Hand Tracking: Unity, OpenXR, Unity XR Interaction Toolkit
- Rendering & Compute: GPU-based compute shaders, instanced rendering
- Selection Processing: Signed Distance Fields (SDFs), real-time selection volume generation
- User Study & Evaluation: System Usability Scale (SUS), task performance metrics, data visualization
Publications
Fluri, L., & Cöltekin, A. (2025). 3D Selection through Hand Tracking in XR for Point Clouds Using Signed Distance Fields. In K. Helin, B. Schiavi, & E. Tsaknaki (Eds.), EuroXR 2025: Proceedings of the Application, Poster, and Demo Tracks of the 22nd EuroXR International Conference (VTT Technology No. 440). VTT Technical Research Centre of Finland.