Hand-tracked 3D Data Selection of Point Clouds in XR

Point cloud datasets—collections of millions of 3D coordinates representing scanned environments, astronomical observations, or medical imaging data—present unique challenges for interactive exploration and analysis. Traditional workflows rely on 2D interfaces with mouse-based selection tools, creating a fundamental mismatch between the three-dimensional nature of the data and the two-dimensional interaction paradigm. This disconnect limits precision, slows analytical workflows, and constrains scientists' ability to intuitively explore complex spatial structures.

My Master's thesis investigates how immersive XR environments combined with natural hand tracking can fundamentally transform point cloud selection. By allowing users to reach directly into 3D data spaces and select regions using intuitive hand gestures, we eliminate the dimensional mismatch that has constrained point cloud analysis for decades.

GPU-Accelerated Selection Through Signed Distance Fields

The technical core of this research is a GPU-based selection system that leverages Signed Distance Fields (SDFs) to enable real-time selection of arbitrary three-dimensional regions. Unlike traditional bounding-box approaches, SDFs represent selection volumes as mathematical functions computed entirely on the GPU, allowing for:

  • Arbitrary Selection Shapes: Beyond simple boxes and spheres, users can define complex, organic selection regions
  • Real-Time Performance: GPU compute shaders process millions of points per frame, maintaining up to 90Hz VR performance The final system without any optimization ran at 70-80 FPS with up to 5 million points
  • Precision at Scale: Efficient distance calculations enable accurate selection even in datasets containing millions of points

This approach was implemented in Unity using OpenXR for cross-platform VR compatibility, with custom compute shaders handling the computationally intensive distance field calculations. The system runs smoothly on standalone headsets like the Meta Quest Pro and Quest 3, demonstrating the viability of controller-free interaction for complex analytical tasks.

Evidence-Based Design Through User Research

To validate the effectiveness of different interaction approaches, I conducted a controlled user study with 28 participants comparing four distinct selection techniques. The study measured task completion time, selection accuracy, error rates, and subjective usability through standardized questionnaires.

Key Findings:

  • Brushing-based techniques (painting over points with hand movements) significantly outperformed direct shape-based selection in both speed and accuracy
  • Users strongly preferred the natural, continuous feedback of brushing interactions over discrete object manipulation
  • Hand-only interaction (without virtual tools) achieved the highest usability scores, suggesting minimal abstraction leads to better spatial understanding
  • Selection accuracy improved by 23% on average when using brush-based methods compared to traditional 3D widget approaches

These results provide empirical evidence that reducing interaction friction through natural gestures enhances both objective performance and subjective experience in spatial analytical tasks.

Four Interaction Paradigms for Spatial Selection

The system implements four distinct selection techniques, each exploring different trade-offs between directness, precision, and cognitive load. All techniques leverage hand tracking to enable controller-free interaction, reducing the barrier between user intent and system response.

Users paint over points directly with their hands, creating a natural, tactile selection experience with immediate visual feedback

1. Direct Hand Brushing
The most intuitive technique allows users to simply move their hands through the point cloud, selecting all points within the virtual hand-model directly. To avoid false-positives, the selection is only active if the opposite hand is doing a pinch gesture. This approach scored high in usability tests, with participants describing it as "painting in 3D space" and appreciating the immediate, continuous feedback as points changed color in real-time. Under the hood this technique recalculates the SDF for the hand-model every frame

A virtual sphere controlled by hand movements provides a larger, more visible selection tool while maintaining the natural brushing interaction paradigm

2. Sphere-Based Brushing
A virtual sphere attached to the user's hand provides a more visible selection tool while maintaining the natural brushing paradigm. Users can adjust the sphere's radius through pinch gestures, allowing for both coarse and fine-grained selection. This technique proved particularly effective for selecting large, distributed regions quickly. Any mesh shape would work as a selection tool, but a sphere was chosen for its simplicity and visibility.

Users draw individual points to define vertices of a selection region. The system computes and visualizes the convex hull in real-time, providing clear feedback on the selected volume

3. Convex Hull Selection
Users define selection boundaries by drawing individual points that serve as vertices using a pinch & draw gesture. The system computes the convex hull of these points in real-time, creating a selection volume that adapts dynamically as new vertices are added. This technique excels when selecting well-defined structural features with clear geometric boundaries.

Users spawn primitive shapes (spheres, boxes, cylinders) through pinch gestures and manipulate them using bimanual interaction for position, rotation, and scale

4. Shape-Based Selection
Users spawn primitive geometric shapes (spheres, boxes, cylinders) and manipulate them using bimanual hand gestures. While this approach offers precise control over selection boundaries, user studies revealed it requires more cognitive effort and time compared to brushing techniques, as it introduces an additional layer of abstraction between intent and action.

Beyond Selection: Point Cloud Manipulation

Selected point cloud regions can be transformed using a handlebar metaphor—grasping virtual handles with both hands enables intuitive translation, rotation, and scaling

Once selected, point cloud subsets can be transformed using bimanual interaction. Users grab virtual handles with both hands, enabling natural translation, rotation, and uniform scaling through familiar real-world metaphors. This handlebar technique leverages users' existing mental models of physical object manipulation, reducing learning time and improving spatial accuracy.

Hand-Based UI for Technique Switching

A hand-attached UI provides quick access to selection techniques, settings, and point cloud management without requiring physical controllers or breaking immersion

Technique selection is managed through a hand-attached menu system that appears when users look at their palm. This approach keeps UI elements accessible without cluttering the visual field, and allows for rapid switching between interaction modes during analytical workflows.

Key Features

  • Real-Time Hand-Tracked Selection

    • Users can interact directly with 3D point clouds using natural hand gestures
    • Implements four selection techniques, including shape-based, convex hull, and brush-based methods
    • Brush-based selection methods were found to be the most efficient in user tests

  • GPU-Accelerated Selection Using Signed Distance Fields (SDFs)

    • Leverages GPU compute shaders for high-speed selection processing
    • Allows arbitrary selection volumes, overcoming traditional bounding-box limitations
    • Efficiently handles large-scale point clouds with millions of points

  • XR-Based Implementation in Unity

    • Developed for VR headsets (Meta Quest Pro, Quest 3) using OpenXR
    • Fully controller-free interaction with hand tracking for a more natural experience
    • Multi-pass rendering and optimized shaders ensure smooth real-time performance

  • User Study & Validation

    • 28 participants tested different selection techniques in controlled experiments
    • Measured selection accuracy, speed, usability, and user satisfaction
    • Findings indicate higher efficiency and preference for brush-based selection methods

Technology Stack

  • XR & Hand Tracking: Unity, OpenXR, Unity XR Interaction Toolkit
  • Rendering & Compute: GPU-based compute shaders, instanced rendering
  • Selection Processing: Signed Distance Fields (SDFs), real-time selection volume generation
  • User Study & Evaluation: System Usability Scale (SUS), task performance metrics, data visualization

Publication & Download

This research was published as:

Fluri, L., Coltekin, A. (2025). 3D Selection through Hand Tracking in XR for PointClouds Using Signed Distance Fields. In K. Helin, B. Schiavi, & E. Tsaknaki (Eds.), EuroXR 2025: Proceedings of the Application, Poster, and Demo Tracks of the 22nd EuroXR International Conference (VTT Technology No. 440). VTT Technical Research Centre of Finland. https://doi.org/10.32040/2242-122X.2025.T440

Poster (EuroXR 2025): Poster
Abstract (EuroXR 2025): Abstract
📲 GitHub Repository: [Will be made public soon]