phenotype estimation with deep learning
TRANSCRIPT
Phenotype Estimation with Deep Learning
Hanqi Sun, Aug 5th
Outline
● Pipeline● Current Result● Future Plan
Pipeline: Part I
3D Point Cloud Reconstruction
from images
Segmented Point Cloud
Phytomer(Single leaf)Point Cloud
SegmentationAlgorithm
ClusteringAlgorithm
Pipeline: Part II (my work)
CenteredPhytomer
Point Cloud
2d GrayscaleProjection Images
Phenotype Estimations
SimulatedPhytomer
Point Cloud
2d GrayscaleProjection Images
Deep neural networks
Centered Phytomer PointCloud
● After we find the stem, we could position the stem along the z axis and the leaf along the y axis
Generate 2d projection images● 128 * 128 greyscale images● One “forward” projection (ignore x coordinate), One “downward” projection
(ignore z coordinate)
Deep neural network
● Alexnet (5 convolutional layers with 3 fully connected layers at the end.)● Built using tensorflow, trained on GPU● About 100 million parameters per network (600 MB+ space)● ~ 2.5 hour training time for one phenotype
Phenotype estimation
● In the end, the pipeline will give a txt file back: (each row is one phytomer)
Result with simulated data (No rotation angle)Name Range RMSE error
Leaf angle* 75.94 3.30 (4.35%)
Leaf radiation angle* 120.66 4.34 (3.60%)
Leaf length* 35.00 0.87 (2.49%)
Leaf width [max] 3.61 0.27 (7.48%)
Leaf width [average] 2.99 0.21 (7.o2%)
Leaf area* 133.45 8.11 (6.08%)
*: projection onto y-z plane
Three rotation angles
X Y Z
With three rotation angles (prediction only)Name Range RMSE error
Leaf angle* 75.94 26.00 (34.2%)
Leaf radiation angle* 120.66 27.75 (22.8%)
Leaf length* 35.00 8.79 (25.1%)
Leaf width [max] 3.61 1.09 (30.2%)
Leaf width [average] 2.99 0.76 (25.4%)
Leaf area* 133.45 27.19 (20.3%)
Error heat map of Leaf angle (34.2%)
X-axis: Z rotation angle
Y-axis: Y rotation angle
Color: error (degrees)
Range: 75.94
Error heat map of Leaf angle (34.2%)
X-axis: Z rotation angle
Y-axis: Y rotation angle
Color: error (degrees)
Range: 75.94
Error heat map of Leaf Length (25.1%)
X-axis: Z rotation angle
Y-axis: Y rotation angle
Color: error (unit)
Range: 35.00
With three rotation angles (With training)Name Range RMSE error
Leaf angle* 75.94 16.53 (21.77%)
Leaf radiation angle* 120.66 18.24 (15.12%)
Leaf length* 35.00 4.55 (13.00%)
Leaf width [max] 3.61 0.46 (12.74%)
Leaf width [average] 2.99 0.36 (12.04%)
Leaf area* 133.45 15.73 (11.78%)
Result with simulated data (No rotation angle)Name Range RMSE error
Leaf angle* 75.94 3.30 (4.35%)
Leaf radiation angle* 120.66 4.34 (3.60%)
Leaf length* 35.00 0.87 (2.49%)
Leaf width [max] 3.61 0.27 (7.48%)
Leaf width [average] 2.99 0.21 (7.o2%)
Leaf area* 133.45 8.11 (6.08%)
*: projection onto y-z plane
Result with Green House Data
● The image of that of green house plants is different from the ones I used to train the network○ The leaves are longer, and the point cloud is complete (no noise/lost points at all)○ So for a little phytomer (leaf), the value does not seem correct (e.g. the leaf angle)
● So for field data, I will look at the phytomer and images, generate data that are similar what we get, and retrain the network.○ It takes 12 hours to generate 60,000 phytomers or 20,000 full plants.○ It takes 2.5 hour to train one phenotype○ Phenotype prediction with pre-trained network is done instantly.○ In total, it takes about 4 days to give the result back.
Good Result 1
Good Result 2
Good Result 3
Bad Result 1 (leaf angle, leaf area too small)
Bad Result 2 (leaf angle too small)
Result for error 1 (double leaves, but prediction OK)
Result for error 2 (double leaves, leaf angle too small)
Result for error 3 (leaf/stem not connected, but prediction OK)
Result for error 4 (no leaf, leaf angle makes no sense, others OK)
Future Effort (minor)
● Retrain the network according to the green house data, see if the result gets better
● Retrain the network with full plant projection. See if other phenotypes (stem height, average leaf angle) can be captured in 2d image as well.
Future Effort (major)
● Try to simplify the pipeline. See if we could get phenotype values directly from 3d plant (without segmentation and projection) or sensor images.
● For this fall, I think we should focus on what we have so far.
Directly train on point cloud (simulated)
Original Point Cloud
Phenotype estimations
Simulated Point Cloud
Name Range RMSE error
Average leaf angle 56.47 17.29 (30.62%)
Stem height 214.89 10.27 (4.78%)
Convex hull volume 66046 4513 (6.83%)
Average leaf radiation angle 41.33 6.86 (16.60%)
Average leaf area 112.75 20.68 (18.34%)
Average max leaf width 3.39 0.87 (25.66%)
Average interligule length 12.67 1.37 (10.83%)
Leaf number [6, 15] 4.40
Thank you!
● [email protected] any questions! (12 hour time zone difference,)● Simulation code available (MATLAB, 4000+ lines from ground up)
○ https://bitbucket.org/cmu_terra/sorghum_simulation (with permission)
● Deep Learning code to be released.. (python, 1000+ lines)