Hand-tracked 3D Data Selection of Point Clouds in XR
Point clouds, millions of 3D coordinates from scans, astronomical surveys, or medical imaging, are hard to select into with a mouse. You're picking regions out of a 3D structure through a 2D interface, which is slow and imprecise for anything but the simplest shapes.
My master's thesis looks at whether hand tracking in XR fixes this: reaching directly into the data and selecting with your hands instead of a cursor.
Selection via Signed Distance Fields
The core of the system is GPU-based selection using Signed Distance Fields (SDFs). Instead of bounding boxes, a selection volume is a mathematical function evaluated entirely on the GPU, which means:
- Selection shapes aren't limited to boxes and spheres, they can be arbitrary and organic
- Compute shaders process millions of points per frame, targeting the headset's 90Hz refresh rate The final system without any optimization ran at 70-80 FPS with up to 5 million points
- Selection stays accurate even at millions of points, since the distance calculations are cheap enough to run every frame
Built in Unity with OpenXR for cross-platform VR, with custom compute shaders doing the distance field math. Runs on standalone headsets like the Meta Quest Pro and Quest 3, controller-free.
User Study
To compare the four selection techniques below, I ran a study with 28 participants, measuring completion time, accuracy, error rate, and usability questionnaires.
Findings:
- Brushing (painting over points with hand movements) beat shape-based selection on both speed and accuracy
- Participants preferred brushing's continuous feedback over discrete object manipulation
- Hand-only interaction, no virtual tools, scored highest on usability: less abstraction between intent and action seems to help
- Brush-based selection was about 23% more accurate on average than the 3D widget approaches
Four Selection Techniques
The system implements four distinct selection techniques, each exploring different trade-offs between directness, precision, and cognitive load. All techniques leverage hand tracking to enable controller-free interaction, reducing the barrier between user intent and system response.
Users paint over points directly with their hands, creating a natural, tactile selection experience with immediate visual feedback
1. Direct Hand Brushing
The most intuitive technique allows users to simply move their hands through the point cloud, selecting all points within the virtual hand-model directly. To avoid false-positives, the selection is only active if the opposite hand is doing a pinch gesture. This approach scored high in usability tests, with participants describing it as "painting in 3D space" and appreciating the immediate, continuous feedback as points changed color in real-time. Under the hood this technique recalculates the SDF for the hand-model every frame
A virtual sphere controlled by hand movements provides a larger, more visible selection tool while maintaining the natural brushing interaction paradigm
2. Sphere-Based Brushing
A virtual sphere attached to the user's hand provides a more visible selection tool while maintaining the natural brushing paradigm. Users can adjust the sphere's radius through pinch gestures, allowing for both coarse and fine-grained selection. This technique proved particularly effective for selecting large, distributed regions quickly. Any mesh shape would work as a selection tool, but a sphere was chosen for its simplicity and visibility.
Users draw individual points to define vertices of a selection region. The system computes and visualizes the convex hull in real-time, providing clear feedback on the selected volume
3. Convex Hull Selection
Users define selection boundaries by drawing individual points that serve as vertices using a pinch & draw gesture. The system computes the convex hull of these points in real-time, creating a selection volume that adapts dynamically as new vertices are added. This technique excels when selecting well-defined structural features with clear geometric boundaries.
Users spawn primitive shapes (spheres, boxes, cylinders) through pinch gestures and manipulate them using bimanual interaction for position, rotation, and scale
4. Shape-Based Selection
Users spawn primitive geometric shapes (spheres, boxes, cylinders) and manipulate them using bimanual hand gestures. While this approach offers precise control over selection boundaries, user studies revealed it requires more cognitive effort and time compared to brushing techniques, as it introduces an additional layer of abstraction between intent and action.
Manipulating a Selection
Selected point cloud regions can be transformed using a handlebar metaphor: grasping virtual handles with both hands enables intuitive translation, rotation, and scaling
Once selected, a subset of points can be grabbed with both hands and translated, rotated and scaled through a handlebar metaphor, the same motions you'd use to move a physical object.
Switching Techniques
A hand-attached UI provides quick access to selection techniques, settings, and point cloud management without requiring physical controllers or breaking immersion
A menu appears when you look at your palm, so switching between the four techniques or adjusting settings doesn't need a controller or a separate UI panel.
Tech Stack
- XR & Hand Tracking: Unity, OpenXR, Unity XR Interaction Toolkit
- Rendering & Compute: GPU-based compute shaders, instanced rendering
- Selection Processing: Signed Distance Fields (SDFs), real-time selection volume generation
- User Study & Evaluation: System Usability Scale (SUS), task performance metrics, data visualization
Links
GitHub Repository: Will be made public soon.
Publications
Fluri, L., & Cöltekin, A. (2025). 3D Selection through Hand Tracking in XR for Point Clouds Using Signed Distance Fields. In K. Helin, B. Schiavi, & E. Tsaknaki (Eds.), EuroXR 2025: Proceedings of the Application, Poster, and Demo Tracks of the 22nd EuroXR International Conference (VTT Technology No. 440). VTT Technical Research Centre of Finland.