Skip to content

Commit

Permalink
add apc evaluation code
Browse files Browse the repository at this point in the history
  • Loading branch information
andyzeng committed Dec 10, 2016
1 parent 644df6c commit 3106334
Show file tree
Hide file tree
Showing 27 changed files with 10,009 additions and 39 deletions.
32 changes: 21 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,15 +38,15 @@ Our reference implementation of 3DMatch, as well as other components in this too

1. [CUDA 7.5](https://developer.nvidia.com/cuda-downloads) and [cuDNN 5](https://developer.nvidia.com/cudnn). You may need to register with NVIDIA. Below are some additional steps to set up cuDNN 5. **NOTE** We highly recommend that you install different versions of cuDNN to different directories (e.g., ```/usr/local/cudnn/vXX```) because different software packages may require different versions.

```shell
LIB_DIR=lib$([[ $(uname) == "Linux" ]] && echo 64)
CUDNN_LIB_DIR=/usr/local/cudnn/v5/$LIB_DIR
echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDNN_LIB_DIR >> ~/.profile && ~/.profile
```shell
LIB_DIR=lib$([[ $(uname) == "Linux" ]] && echo 64)
CUDNN_LIB_DIR=/usr/local/cudnn/v5/$LIB_DIR
echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$CUDNN_LIB_DIR >> ~/.profile && ~/.profile

tar zxvf cudnn*.tgz
sudo cp cuda/$LIB_DIR/* $CUDNN_LIB_DIR/
sudo cp cuda/include/* /usr/local/cudnn/v5/include/
```
tar zxvf cudnn*.tgz
sudo cp cuda/$LIB_DIR/* $CUDNN_LIB_DIR/
sudo cp cuda/include/* /usr/local/cudnn/v5/include/
```

2. OpenCV (tested with OpenCV 2.4.11)
* Used for reading image files
Expand Down Expand Up @@ -129,18 +129,28 @@ Reference implementation for the experiments in our paper.

See `evaluation/geometric-registration`

Includes Matlab/C++/CUDA code to run evaluation on the geometric registration benchmarks described [here](). To generate intermediate data (TDF voxel volumes, keypoints, and 3DMatch descriptors) for the scene fragments, see `getKeyptsAndDesc.m` or download our pre-computed data [here](). To use intermediate data and run RANSAC-based registration for every pair of fragments, see `runFragmentRegistration.m`. To compute precision and recall from .log files for evaluation, see the example script `evaluate.m`.
Includes Matlab code to run evaluation on the geometric registration benchmarks described [here](). Overview:
* `getKeyptsAndDesc.m` - generates intermediate data (TDF voxel volumes, keypoints, and 3DMatch descriptors) for the scene fragments. You can also download our pre-computed data [here]().
* `runFragmentRegistration.m` - read intermediate data and run RANSAC-based registration for every pair of fragments.
* `writeLog` - read registration results from every pair of fragments and create a .log file
* `evaluate.m` - compute precision and recall from .log files for evaluation

Quick start: start Matlab and run script `evaluation/geometric-registration/evaluate.m`

Note: the TDF voxel grids of the scene fragments from the synthetic benchmark were computed using the deprecated code for accurate TDF. See `deprecated/pointCloud2AccTDF.m`

### Model Fitting for 6D Object Pose Estimation
### Model Fitting for 6D Object Pose Estimation in the Amazon Picking Challenge

See `evaluation/model-fitting-apc`

Includes code, pre-trained model, and pre-computed data to evaluate 3DMatch for model fitting on the Shelf & Tote dataset (Amazon Picking Challenge setting).
Includes code and pre-trained models to evaluate 3DMatch for model fitting on the Shelf & Tote dataset.

Quick start: start Matlab and run script `evaluation/model-fitting-apc/getError.m`

### Mesh Correspondence in Shape2Pose

Includes code to generate mesh correspondence visualizations on the meshes from Shape2Pose.

Quick start:


3 changes: 2 additions & 1 deletion core/demo.m
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
clear all;
% Demo script to run RANSAC-based rigid transform estimation between two
% sets of 3D keypoints and their 3DMatch descriptors

% Add dependencies (for RANSAC-based rigid transform estimation)
addpath(genpath('external'));
Expand Down
11 changes: 10 additions & 1 deletion evaluation/geometric-registration/clusterCallback.m
Original file line number Diff line number Diff line change
@@ -1,6 +1,15 @@
% ---------------------------------------------------------
% Copyright (c) 2016, Andy Zeng
%
% This file is part of the 3DMatch Toolbox and is available
% under the terms of the Simplified BSD License provided in
% LICENSE. Please retain this notice and LICENSE if you use
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

function clusterCallback(jobID)

% Parameters
% Configuration options (change me)
descriptorName = '3dmatch';
scenePath = '../../data/fragments/7-scenes-redkitchen'; % Location of scene fragment point clouds
intermPath = '../../data/fragments/intermediate-files/7-scenes-redkitchen'; % Location of intermediate files holding keypoints and descriptor vectors
Expand Down
13 changes: 2 additions & 11 deletions evaluation/geometric-registration/evaluate.m
Original file line number Diff line number Diff line change
@@ -1,17 +1,8 @@
% Script to evaluate .log files for the real-world geometric registration
% benchmark, in the same spirit as Choi et al 2015. Please see:
% Script to evaluate .log files for the geometric registration benchmarks,
% in the same spirit as Choi et al 2015. Please see:
%
% http://redwood-data.org/indoor/regbasic.html
% https://github.com/qianyizh/ElasticReconstruction/tree/master/Matlab_Toolbox
%
% ---------------------------------------------------------
% Copyright (c) 2016, Andy Zeng
%
% This file is part of the 3DMatch Toolbox and is available
% under the terms of the Simplified BSD License provided in
% LICENSE. Please retain this notice and LICENSE if you use
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

% Locations of evaluation files
sceneList = {'../../data/fragments/iclnuim-livingroom1-evaluation' ...
Expand Down
5 changes: 3 additions & 2 deletions evaluation/geometric-registration/getGtInfoLog.m
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,12 @@
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

% Configuration options (change me)
sceneList = {'7-scenes-redkitchen', ...
'sun3d-hotel_umd-maryland_hotel3', ...
'sun3d-mit_76_studyroom-76-1studyroom2', ...
'sun3d-mit_lab_hj-lab_hj_tea_nov_2_2012_scan1_erika'};
sceneList = {'sun3d-home_at-home_at_scan1_2013_jan_1', ...
'sun3d-mit_lab_hj-lab_hj_tea_nov_2_2012_scan1_erika', ...
'sun3d-home_at-home_at_scan1_2013_jan_1', ...
'sun3d-home_md-home_md_scan9_2012_sep_30', ...
'sun3d-hotel_uc-scan3', ...
'sun3d-hotel_umd-maryland_hotel1'};
Expand Down
31 changes: 18 additions & 13 deletions evaluation/geometric-registration/getKeyptsAndDesc.m
Original file line number Diff line number Diff line change
@@ -1,10 +1,20 @@
% Script to load scene fragment point clouds and generate TDF voxel
% volumes, keypoints, and 3DMatch descriptors (intermediate data).

% NOTE: uncomment the code at the end of core/demo.cu to save tdf.bin
%
% ---------------------------------------------------------
% Copyright (c) 2016, Andy Zeng
%
% This file is part of the 3DMatch Toolbox and is available
% under the terms of the Simplified BSD License provided in
% LICENSE. Please retain this notice and LICENSE if you use
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

% Configuration options (change me)
cudaBinPath = '/usr/local/cuda/bin';
cudaLibPath = '/usr/local/cuda/lib64';
cudnnLibPath = '/usr/local/cudnn/v5.1/lib64';

sceneList = {'7-scenes-redkitchen', ...
'sun3d-hotel_umd-maryland_hotel3', ...
'sun3d-mit_76_studyroom-76-1studyroom2', ...
Expand All @@ -13,40 +23,35 @@
'sun3d-home_md-home_md_scan9_2012_sep_30', ...
'sun3d-hotel_uc-scan3', ...
'sun3d-hotel_umd-maryland_hotel1'};
dataPath = '/n/fs/sun3d/andyz/3dmatch/data/fragments';
savePath = '/n/fs/sun3d/andyz/3dmatch/data/fragments/intermediate-files';

% Loop through each fragment for all scenes
for sceneIdx = 1:length(sceneList)

sceneName = sceneList{sceneIdx};

dataPath = '/n/fs/sun3d/andyz/3dmatch/data/fragments';
savePath = '/n/fs/sun3d/andyz/3dmatch/data/fragments/intermediate-files';

cloudFiles = dir(fullfile(dataPath,sceneName,'cloud_bin*.ply'));

for cloudIdx = 1:length(cloudFiles)

cloudPrefix = cloudFiles(cloudIdx).name(1:(end-4));
fprintf('%s\n',fullfile(dataPath,sceneName,cloudPrefix));

cloudName = fullfile(dataPath,sceneName,cloudFiles(cloudIdx).name);

% System call to C++/CUDA code to compute intermediate data
returnPath = pwd;
cd ../../core;
system(sprintf('export PATH=$PATH:%s',cudaBinPath));
system(sprintf('unset LD_LIBRARY_PATH; export LD_LIBRARY_PATH=LD_LIBRARY_PATH:%s:%s; ./demo %s %s',cudaLibPath,cudnnLibPath,cloudName,'test'));

% Copy files
fileTDF = fullfile(savePath,sceneName,sprintf('%s.tdf.bin',cloudPrefix));
fileKeypts = fullfile(savePath,sceneName,sprintf('%s.keypts.bin',cloudPrefix));
fileDesc = fullfile(savePath,sceneName,sprintf('%s.desc.3dmatch.bin',cloudPrefix));

copyfile('tdf.bin',fileTDF);
copyfile('test.keypts.bin',fileKeypts);
copyfile('test.desc.3dmatch.bin',fileDesc);

cd(returnPath);

end

end


Expand Down
11 changes: 11 additions & 0 deletions evaluation/geometric-registration/register2Fragments.m
Original file line number Diff line number Diff line change
@@ -1,4 +1,15 @@
function [estimateRt,numInliers,inlierRatio,ratioAligned] = register2Fragments(scenePath,intermPath,fragment1Name,fragment2Name,descriptorName)
% Run RANSAC-based pose estimation between two sets of 3D keypoints and
% their 3DMatch descriptors (of scene fragments)
%
% ---------------------------------------------------------
% Copyright (c) 2016, Andy Zeng
%
% This file is part of the 3DMatch Toolbox and is available
% under the terms of the Simplified BSD License provided in
% LICENSE. Please retain this notice and LICENSE if you use
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

% Load fragment point clouds
fragment1PointCloud = pcread(fullfile(scenePath,sprintf('%s.ply',fragment1Name)));
Expand Down
1 change: 1 addition & 0 deletions evaluation/geometric-registration/writeLog.m
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

% Configuration options (change me)
descriptorName = '3dmatch';
dataPath = '../../data/fragments'; % Location of scene files
intermPath = '../../data/fragments/intermediate-files'; % Location of intermediate files holding keypoints and descriptor vectors
Expand Down
Binary file not shown.
137 changes: 137 additions & 0 deletions evaluation/model-fitting-apc/clusterCallback.m
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
function clusterCallback(jobID)
% Predict object pose between object model and segmentation point cloud by
% running RANSAC-based geometric registration between two sets of 3D
% keypoints and their descriptors
%
% ---------------------------------------------------------
% Copyright (c) 2016, Andy Zeng
%
% This file is part of the 3DMatch Toolbox and is available
% under the terms of the Simplified BSD License provided in
% LICENSE. Please retain this notice and LICENSE if you use
% this file (or any portion of it) in your project.
% ---------------------------------------------------------

% Configuration options (change me)
descriptorName = '3dmatch';
dataPath = '../../data/apc';
addpath(genpath('../../core/external'));

% List scenarios
scenarioIdx = jobID - 1;
scenarioList = dir(fullfile(dataPath,'scenarios/00*'));
if scenarioIdx >= length(scenarioList)
return;
end
scenarioPath = fullfile(dataPath,'scenarios',sprintf('%06d',scenarioIdx));
fprintf('%s\n',scenarioPath);

% Check if pose prediction file already exists
predictedRtFile = fullfile(scenarioPath,sprintf('ransac.%s.model2segm.txt',descriptorName));
if exist(predictedRtFile,'file')
return;
end

fileID = fopen(fullfile(scenarioPath,'info.txt'),'r');
sceneName = fscanf(fileID,'scene: %s\n');
objectName = fscanf(fileID,'object: %s\n');
fclose(fileID);

% Load segmentation from scene point cloud and its TDF voxel volume
segmPointCloud = pcread(fullfile(scenarioPath,'segmentation.ply'));
matchSegmVoxFile = fullfile(scenarioPath,'segmentation.TDF.bin');
fileID = fopen(matchSegmVoxFile,'rb');
matchSegmVoxSize = fread(fileID,3,'single');
matchSegmOrigin = fread(fileID,3,'single');
segm2model = reshape(fread(fileID,16,'single'),4,4)';
matchSegmVox = fread(fileID,'single');
fclose(fileID);
matchSegmVox = reshape(matchSegmVox,matchSegmVoxSize');

% Load object model point cloud and its TDF voxel volume
matchModelVoxFile = fullfile(dataPath,'objects',sprintf('%s.TDF.bin',objectName));
modelPointCloud = pcread(fullfile(dataPath,'objects',sprintf('%s.ply',objectName)));
fileID = fopen(matchModelVoxFile,'rb');
matchModelVoxSize = fread(fileID,3,'single');
matchModelOrigin = fread(fileID,3,'single');
matchModelVox = fread(fileID,'single');
fclose(fileID);
matchModelVox = reshape(matchModelVox,matchModelVoxSize');

% Load segmentation keypoints
segmKeypointsFile = fullfile(scenarioPath,'segmentation.keypoints.bin');
fileID = fopen(segmKeypointsFile,'rb');
segmNumKeypoints = fread(fileID,1,'single');
segmKeypointsCamData = fread(fileID,'single');
fclose(fileID);
segmKeypointsCam = reshape(segmKeypointsCamData,3,segmNumKeypoints)';

% Load model keypoints
modelKeypointsFile = fullfile(dataPath,'objects',sprintf('%s.keypoints.bin',objectName));
fileID = fopen(modelKeypointsFile,'rb');
modelNumKeypoints = fread(fileID,1,'single');
modelKeypointsCamData = fread(fileID,'single');
fclose(fileID);
modelKeypointsCam = reshape(modelKeypointsCamData,3,modelNumKeypoints)';

% Load segmentation keypoint descriptors
fileID = fopen(fullfile(scenarioPath,sprintf('segmentation.keypoints.%s.descriptors.bin',descriptorName)),'rb');
segmKeypointDescriptorsData = fread(fileID,'single');
segmNumKeypoints = segmKeypointDescriptorsData(1);
segmDescriptorSize = segmKeypointDescriptorsData(2);
fclose(fileID);
segmKeypointDescriptors = reshape(segmKeypointDescriptorsData(3:end),segmDescriptorSize,segmNumKeypoints)';

% Load model keypoint descriptors
fileID = fopen(fullfile(dataPath,'objects',sprintf('%s.keypoints.%s.descriptors.bin',objectName,descriptorName)),'rb');
modelKeypointDescriptorsData = fread(fileID,'single');
modelNumKeypoints = modelKeypointDescriptorsData(1);
modelDescriptorSize = modelKeypointDescriptorsData(2);
fclose(fileID);
modelKeypointDescriptors = reshape(modelKeypointDescriptorsData(3:end),modelDescriptorSize,modelNumKeypoints)';

% Find mutually closest keypoints in descriptor space
modelKDT = KDTreeSearcher(modelKeypointDescriptors);
segmKDT = KDTreeSearcher(segmKeypointDescriptors);
nnSegmIdx = knnsearch(modelKDT,segmKeypointDescriptors);
nnModelIdx = knnsearch(segmKDT,modelKeypointDescriptors);
matchModelIdx = find((1:size(nnModelIdx,1))' == nnSegmIdx(nnModelIdx));
modelKeypointsCam = modelKeypointsCam(matchModelIdx,:);
segmKeypointsCam = segmKeypointsCam(nnModelIdx(matchModelIdx),:);

% Use RANSAC to estimate rigid transformation from model to segmentation
try
[estModel2SegmRt, inliers] = ransacfitRt([segmKeypointsCam';modelKeypointsCam'], 0.05, 0);
catch
estModel2SegmRt = [1,0,0,0;0,1,0,0;0,0,1,0];
fprintf(' Broken\n');
end
estModel2SegmRt = [estModel2SegmRt;[0,0,0,1]];

% % Visualize results
% checkObjModelPts = estModel2SegmRt(1:3,1:3) * modelPointCloud.Location' + repmat(estModel2SegmRt(1:3,4),1,size(modelPointCloud.Location',2));
% checkObjSegmPts = segmPointCloud.Location';
% pcwrite(pointCloud([checkObjModelPts';checkObjSegmPts'],'Color',[repmat(uint8([0,0,255]),size(checkObjModelPts,2),1);repmat(uint8([255,0,0]),size(checkObjSegmPts,2),1)]),'check','PLYformat','binary');

% Save results to pose prediction file
fileID = fopen(predictedRtFile,'w');
for i = 1:4
fprintf(fileID,'%15.8e\t',estModel2SegmRt(i,:));
fprintf(fileID,'\n');
end
fclose(fileID);

end













28 changes: 28 additions & 0 deletions evaluation/model-fitting-apc/compile.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
#!/bin/bash

export PATH=$PATH:/usr/local/cuda/bin

if uname | grep -q Darwin; then
CUDA_LIB_DIR=/usr/local/cuda/lib
CUDNN_LIB_DIR=/usr/local/cudnn/v5.1/lib
elif uname | grep -q Linux; then
CUDA_LIB_DIR=/usr/local/cuda/lib64
CUDNN_LIB_DIR=/usr/local/cudnn/v5.1/lib64
fi

CUDNN_INC_DIR=/usr/local/cudnn/v5.1/include


# if use opencv, add this into the command line
# `pkg-config --cflags --libs opencv`

nvcc -std=c++11 -O3 -o get3DMatchDescriptors get3DMatchDescriptors.cu -I/usr/local/cuda/include -I$CUDNN_INC_DIR -L$CUDA_LIB_DIR -L$CUDNN_LIB_DIR -lcudart -lcublas -lcudnn -lcurand -D_MWAITXINTRIN_H_INCLUDED `pkg-config --cflags --libs opencv`









Loading

0 comments on commit 3106334

Please sign in to comment.