COMP562 Project Report: An Evaluation of the Pyramid Stereo MatchingNetwork (PSMNet)

Helen Qin ZIHE@CS.UNC.COMSiqing Xu XSQ4525@LIVE.UNC.EDUHuihao Peng HUIHAOP@LIVE.UNC.EDUYijie Sun YIJIE@LIVE.UNC.EDU

AbstractRecent work proposed by Jia-Ren Chang andYong-Shen Chen has suggested that PyramidStereo Matching (PSM) Network, can potentiallybe an efficient solution to robust and accurate3D reconstructions. The purpose of our projectis to evaluate the performance of the PSMNeton different combinations of datasets and thento seek potential improvements for the network.To achieve the goals, we will test the PSMNetfirst by validating pre-trained network on publicdataset such as KITTI 2012 and KITTI 2015and then training and testing on two syntheticdatasets generated by ourselves. During theevaluation, we would like to see the robustnessand accuracy of PSMNet in various scenes. Wealso propose to add additional training layers toenhance network performance.

1. Introduction3D reconstruction pipeline with stereo vision has beenwidely employed in real-world applications such asAugmented Reality surgeries. However, most currentapproaches on this technique show a series of shortcomingssignificantly impacting the robustness and accuracy of 3Dreconstruction. First, the basic block matching algorithmon CPU is computationally expensive and its performancecan vary with the cost function. Secondly, the applicabilityof block matching algorithm on depth camera is restrictedby the capturing device and its physical size. Therefore,recent work has been focusing on improving the traditionalapproaches with machine learning and neural networks(Florence et al., 2018). In this paper, we plan to evaluatePyramid Stereo Matching Network (PSMNet) proposed inChang’s paper (Chang, 2018) which is a deep learning

University of Pennsylvania, CIS 419/519 Course Project.Copyright 2017 by the author(s).

approach to this problem.

2. Methodology2.1. Pyramid Stereo Matching Network

As explained in Chang’s paper, our model PyramidStereo Matching Network (PSMNet) consists of twomain modules: a spatial pyramid pooling and a stackedhourglass 3D CNN. The network will first take a pair ofraw stereo Left-and-Right images as input. The spatialpyramid pooling module takes advantage of the capacityof global context information by aggregating context indifferent scales and locations to form a cost volume.The 3D CNN learns to regularize cost volume usingstacked multiple hourglass networks in conjunction withintermediate supervision. The final output is a left disparitymap of corresponding image pair. The paper uses threedataset to evaluate its performance: KITTI 2012, KITTI2015 and SceneFlow (Chang, 2018).

2.2. Retraining the Whole Network

We retrain the whole network with 500 epochs on boththe datasets given in Chang’s paper (Chang, 2018) andour customized synthetic dataset. The first purpose ofretraining the network is to evaluate its robustness ondifferent, unseen datasets. Second, we also use theretrained network as the initialization of the training /tuning in other steps.

2.3. Fine-Tuning the Whole Network

The second attempt of improving this network result is tofine-tune the pre-trained network on more datasets. Ourexpectation is that by tuning the network on a greaternumber of diverse data, the robustness of this model willimprove. We start with 300 epoch at training to ensureconvergence.

Siqing Xu, Helen Qin, Huihao Peng, Yijie Sun

2.4. Transfer Learning

During transfer learning, we freeze the parameters of alllayers other than the last three parallel convolutional layers.Thus, this will be less computationally expensive comparedto the previous fine-tuning step. We start with 300 epoch attraining to ensure convergence.

Figure 1. Network structure: the unfrozen layers during transferlearning are enscribed in the red rectangle at the bottom right ofthe graph.

2.5. Evaluation

During evaluation, we will use and modify the errorformula in Chang’s paper, which is called the three pixelerror. This error counts the percentage of output disparitieswhose differences from their ground-truth disparities aregreater than three pixels.

Mathematically, the three pixel error percentage in oneimage of provided dataset is defined as below:

New Three P ixel Eror =pe∑Ni=1 pi

× 100%

Notation Descriptionpe the number of pixels in one predicted

disparity map whose disparity value hasdifference greater than three from thecorresponding pixels on the ground-truthdisparity map

pi the ith pixel in the image

For a averaged three pixel error percentage in a set ofdisparity maps, it is written as below:

Averaged ThreeP ixel Eror =

∑Ni=1 Three P ixel Erori

N

Notation DescriptionN total number of image of test

datasetThree P ixel Erori the three pixel error percentage

of ith image set

Based on the given evaluation method, we have made twomodifications. First, we derive a three pixel error withoutbackground by modifying its denominator to excludecounting black pixels. Given that most training and testingdata contain a significant amount of black background,this will help the error evaluation to focus on regionsof interested objects. The new three pixel error withoutbackground is defined as follows:

Three P ixel Eror Without Background =pepi

× 100%

Notation Descriptionpe the number of pixels in one predicted

disparity map whose disparity value hasdifference greater than three from thecorresponding pixels on the ground-truthdisparity map

pi the pixels on the ground-truth disparity mapwhich has disparity value greater than 0

Second, we loosen the error strictness by adding a tenpixel error evaluation. Our synthetic datasets generatedcontain objects that are significantly closer (600X) to thecamera. This will result in larger disparity values and thusthe estimation error produced by the network is expected tobe greater as well. Hence the ten pixel error evaluation isintroduced to accommodate this disparity change.

3. ExperimentsWe evaluated different combinations of our methods inSection 4 on four stereo datasets: KITTI 2012, KITTI2015, Synthetic Dataset 1 and Synthetic Dataset 2. Wethen further divide our experiments into three categoriesaccording to their foci: the network’s performances on thesynthetic dataset 1, performances on transfer learning, androbustness, as described starting from Section 3.2.

3.1. Dataset Details

We used four datasets for our training and evaluation:

1. Synthetic Dataset 1: A software-generated dataset thatcontains 180 image pairs of pegboard objects. Each imagehas dense ground-truth disparities. The objects are onlyaround 20 centimeters away from the stereo cameras.

Figure 2. Example image pair in the synthetic dataset 1.

2. Synthetic Dataset 2: A software-generated dataset thatcontains 180 image pairs of pegboard objects. Each image

Siqing Xu, Helen Qin, Huihao Peng, Yijie Sun

has dense ground-truth disparities. The objects are onlyaround 20 centimeters away from the stereo cameras.

Figure 3. Example image pair in the synthetic dataset 2.

3. KITTI 2012 and KITTI 2015: Both KITTI 2012 andKITTI 2015 are real-world datasets of street views. Eachdataset contains around 200 training image pairs and 200testing image pairs with sparse ground-truth disparities.Chang’s paper used both datasets for training its model.In our experiments, we first tested the PSMNet with pre-trained networks provided by the paper’s source code onthe KITTI datasets.

3.2. Performances on the Synthetic Dataset 1

For the first set of experiments, we performed threepreviously described methods on Synthetic Dataset 1 tocompare these methods. From the error table (Figure 3),we can see that the finetune on all layers have the leasterror rate in all categories. The comparison between thefinetune and retrain results demonstrates that the legacyprovided by the pretrained neural network, even thoughit is trained on a dataset which has significant differencefrom the current ones, could still help in disparity detectionin current testing datasets. The comparison between thefinetune on all layers and transfer learning only on last threeconvolutional components shows that the previous twomodules, feature extraction module and stack-hourglassmodule, have great dependence on the datasets. If the twomodules are pretrained on one dataset, and tested on otherdatasets which have huge difference from the pretrainedone, the two modules cannot extract enough informationfor accurate disparity generation. Only transfer learning onlast three components cannot make error quickly converge(We used 300 epochs but the error still could not converge).

3.3. Performances on Transfer Learning

For the second set of experiments, we took a deeper look attransfer learning on last three components method. Fromthe previous experiment, we observed that the loss ofthis method is hard to converge if the method is transferlearning between two datasets with huge difference. Inthis experiment, we tested with two datasets with smalldifference (objects are simple, detailed and in similardistance). Starting from the saved network which isalready finetuned on Synthetic Dataset 1, we tested testingset of Synthetic Dataset 2 without transfer learning asthe baseline and with transfer learning to compare the

(a) (b)

(c) (d)

Figure 4. Results of disparity estimation for the synthetic dataset1 with different methods. (a) fine-tuning, (b) retraining, (c)transfer learning, and (d) ground-truth.

results. (Figure 5) From the baseline result, we can seethat the result is visually good, the shape and contour ofthe objects are distinguishable and clear. This baselineresults show that the PSMNet can generalize some of taskssuch as segmenting the objects with depth from the blackbackground. However, the error rate of the baseline resultshows that the disparity value of the region with depthare almost all incorrect. (Figure 3) By comparison, thetransfer learning results have shown huge improvementsfrom the baseline one. For example, the contour ofthe toy car is distinguishable compared with the baselineresult. The error rate has even larger improvement. Allcategories of error rate converge to almost similar to thoseof the finetuning on all layers method from the previousexperiment. This result demonstrates that the transferlearning on last three components can be effective if thedatasets are similar. The previous two modules can berobust to similar datasets.

(a) (b)

(c)

Figure 5. Results of disparity estimation for the synthetic dataset2 focusing on transfer learning method comparison. (a) rawdisparity map from pretrained network on KITTI 2015, (b)disparity map from the transfer-learning method, and (c) ground-truth disparity map.

3.4. Network Robustness

Lastly, we simply want to see if the PSMNet can begeneralized on two datasets with huge difference. For the

Siqing Xu, Helen Qin, Huihao Peng, Yijie Sun

first trial, we tested the test set of Synthetic Dataset 1 on thenetwork which was pretrained on the KITTI 2015 dataset.For the second trial, after finetuning all layers on theSynthetic Dataset 1 with the pretrained network of KITTI2015 for 300 epochs, we tested finetuned network with testset of KITTI 2015 to see if the network can still performwell after the finetuning procedure. The results for bothtrials are bad and the disparity map is indistinguishable toeven see the objects’ shapes. This experiment shows thatthe PSMNet can only get reasonable results on the datasetswhich are similar to the one that is most recently trained orfinetuned on.

(a) (b)

Figure 6. Results of disparity estimation focusing on robustnesstesting. (a) KITTI 2015 network being tested on synthetic 1 ,and (b) network fine-tuned with first KITTI 2015 and then thesynthetic dataset 1 being on KITTI 2015.

4. ConclusionsIn this project, we evaluated the performance of thePSMNet and designed different improvements on it. Basedon our experiments, we made three conclusions. Thefirst conclusion is that the PSMNet is data dependableand cannot generalized well on a combination of diversedatasets, despite efforts through fine-tuning, transferlearning, as well as retraining. Second, in termsof the effectiveness of improving the network, thefeature extraction and stack-hourglass module needs to beretrained or finetuned for the network to produce betterresults. Lastly, the PSMNet is, after all, robust enough toperform good disparity estimation if the objects are similarin terms of camera distances and backgrounds.

Figure 7. Three & ten pixel error evaluation table.

5. Future WorkThere are still more potentials with transfer learning thatwe could explore in this project. We expect that trainingon various dataset while choosing to freeze some differentlayers might improve the performance of the network. We

can try so and see which layers/module choices can leadto a quicker and better loss convergence. Additionally, wespecifically want to unfreeze some parameters at featurelearning layers which we expect to add accuracy androbustness to the network.

ReferencesChang, J. Pyramid stereo matching network. CVPR 2018,

2018.

Florence, P., Manuelli, L., and Tedrake, R. Dense objectnets: Learning dense visual object descriptors by and forrobotic manipulation. arXiv 2018, 2018.