RGBD Sensor Tests

This page contains supplemental information about our research paper „An Empirical Evaluation of Ten Depth Cameras: Bias, Precision, Lateral Noise, Different Lighting Conditions and Materials, and Multiple Sensor Setups in Indoor Environments“. In this work we evaluated ten different RGBD and depth sensors covering the main three sensor technologies: Structured Light, Active Stereo, and Time of Flight. We evaluated influence of different target materials, different lighting conditions, and interference from other sensors in a multi-sensor setup. We collected 510 data points using ten different sensors in a robot setup to perform in total four experiments per sensor. We then evaluated the sensors by comparing five different metrics: bias, precision, lateral noise, behavior under different lighting conditions and materials, as well as the applicability for multiple sensor setups.

Figure 1. Depth sensor test equipment, sample output, and schematic for experiment setup.

Experiments

  • Experiment 1,2 – Bias, Precision & Lateral Noise
  • Experiment 3    – Lighting & Materials
  • Experiment 4    – Multiple Sensors

Download

Numerical data of the experiments and quadratic error models for the evaluated sensors can be downloaded here RGBD Sensor Tests Data.

The data collection contains the following  matlab files and variables:

  • bias\bias_data.mat: Bias measurements for Experiment 1
    • acc_<sensor_name>_dist: groundtruth distance in meters
    • acc_<sensor_name>: bias in meters
  • precision\precision_data.mat: Precision measurements for Experiment 1
    • prec_<sensor_name>_dist: measured distance in meters from sensor
    • prec_<sensor_name>: precision in meters
  • lateral_noise\lateral_noise_data.mat: Lateral noise level from Experiment 2
    • lateral_noise_data: Col 1: sensor name, Cols 2-4: lateral noise in pixels per distance interval
  • materials\: Precision measurements for Experiment 3
    • <sensor_name>: Col (light conditions): 4.2 Lux, 36.3 Lux, 277.4 Lux, 535.75 Lux
      Row (materials): textiles, aluminium, plastic-black, foam, paper,plastic-blue
      Entries: precision at 0.7m sensor distance
    • <sensor_name>_mat_1m: same as <sensor_name> at 1.0m sensor distance
    • <sensor_name>_mat_15m: same as <sensor_name> at 1.5m sensor distance
  • additional_sensors\: Precision/Nan ratio measurements for Experiment 4
    • all files: Row (sensor count): test sensor, + additional sensor 1, + additional sensor 2
    • <sensor_name>_nan_07: nan ratio at 0.7m sensor distance
    • <sensor_name>_nan_1: nan ratio at 1.0m sensor distance
    • <sensor_name>_nan_15: nan ratio at 1.5m sensor distance
    • <sensor_name>_prec_07: precision at 0.7m sensor distance
    • <sensor_name>_prec_1: precision at 1.0m sensor distance
    • <sensor_name>_prec_15: precision at 1.5m sensor distance
  • error_model_bias\: Fitted quadratic model for distance bias and precision
    • bias_fit_<sensor_name>.mat
    • prec_fit_<sensor_name>.mat

Research paper

A detailed explanation describing experiments and metrics is available in the research paper:

@ARTICLE{8436009, 
title={An Empirical Evaluation of Ten Depth Cameras: Bias, Precision, Lateral Noise, Different Lighting Conditions and Materials, and Multiple Sensor Setups in Indoor Environments}, 
author={G. Halmetschlager-Funek and M. Suchi and M. Kampel and M. Vincze}, 
journal={IEEE Robotics Automation Magazine}, 
year={2018}, 
volume={}, 
number={}, 
pages={1-1}, 
keywords={Measurement;Universal Serial Bus;Lighting;Cameras;Simultaneous localization and mapping}, 
doi={10.1109/MRA.2018.2852795}, 
ISSN={1070-9932}, 
month={},}

(full citation note will be available after publication)

The open-access version of the paper is available following this link.

Contact & credits

Authors: Georg Halmetschlager-Funek, Markus Suchi, Martin Kampel, Markus Vincze.

For any questions about our sensor evaluation feel free to contact:

  • Georg Halmetschlager-Funek – email: gh@acin.tuwien.ac.at
  • Markus Suchi – email: suchi@acin.tuwien.ac.at