Contents lists available at ScienceDirect

Sensors and Actuators B: Chemical

journal homepage: www.elsevier.com/locate/snb

Classification of hybrid seeds using near-infrared hyperspectral imagingtechnology combined with deep learning

Pengcheng Niea, Jinnuo Zhanga, Xuping Fenga, Chenliang Yub, Yong Hea,⁎

a College of Biosystems Engineering and Food Science, Zhejiang University, Hangzhou, Chinab Vegetable Research Institute, Zhejiang Academy of Agricultural Sciences, Hangzhou 310021, China

A R T I C L E I N F O

Keywords:Near-infrared hyperspectral imagingtechnologyDeep convolutional neural networkHybrid seedst-Distribution stochastic neighbor embedding

A B S T R A C T

The rapid and efficient selection of eligible hybrid progeny is an important step in cross breeding. However,selecting hybrid offspring that meets specific requirements can be time consuming and expensive. Here, near-infrared hyperspectral imaging technology combined with deep learning was applied to classifying hybrid seeds.The hyperspectral images in the range of 975–1648 nm of a total of 6136 hybrid okra seeds and 4128 hybridloofah seeds, which both contained six varieties, were collected. A partial least squares discriminant analysis,support vector machine and deep convolutional neural network (DCNN) were used to establish discriminantanalysis models, and their performances were compared among the different hybrid seed varieties. The dis-criminant analysis model based on the DCNN was the most stable and had the highest classification accuracy,greater than 95%. The values of features in the last layer of the DCNN were visualized using t-distributionstochastic neighbor embedding. The discriminant analysis model based on the DCNN had the advantages ofreducing the labor burden and time required in cross breeding-based progeny selection, which will accelerate theprogress of related research.

1. Introduction

Hybrid breeding is an effective method of combining desirable traitsfrom two or more varieties by mating, selecting, and cultivating toobtain new crop varieties. In China, hybrid rice plays a key role in thefood supply, occupying approximately 60% of the total planted ricearea [1]. Hybridization allows the genes of the parents to be re-combined to form different varieties, providing a rich assortment ofmaterials. The obtained hybrid seeds are genetically diverse, and theproductivity and quality of hybrid plants can be increased [2,3].

Okra [Abelmoschus esculentus (L.) Moench] is highly nutritious be-cause of its fibrous content and other medicinal benefits [4]. Okra seedsare also rich in antioxidative compounds that have health benefits [5].Luffa [Luffa cylindrica (L.) Roem] is eaten as a vegetable and can beused in traditional medicines that treat fever, bronchitis, jaundice, andskin diseases [6]. Additionally, luffa has interesting functional foodproperties and can exhibit antioxidant, antimicrobial, and anti-in-flammatory activities [7]. Hybridization helps increase yields and im-prove the quality of these crops. However, the offspring of mutationscreated by the hybridization need further cultivation to breed newvarieties that meet breeding objectives. The characteristics of hybrid

plants, like plant height and leaf width, can be applied to selectinghybrid offspring that meet specific requirements [8]. However, thisprocess is time consuming and laborious. Consequently, it is necessaryto develop a method for quickly and non-destructively identifying thevarieties of seeds after hybrid breeding.

Molecular overtone transitions and combined vibrations of thetested samples are the basis of near-infrared (NIR) spectral analysis. Thedetailed information of the samples’ composition and characteristics atthe molecular level can be obtained using hyperspectral imagingtechnology [9]. Using NIR hyperspectral imaging technology is a non-destructive and fast way to acquire both spatial and spectral informa-tion on test samples in the range of 780–2500 nm. The spatial andspectral information of the samples, presented in the form of hy-percube, are analyzed to extract internal and external information onthe tested samples [10,11]. The various chemical constituents of thesamples to be detected can be determined by NIR hyperspectral ima-ging system through the measurement of the unique absorption pat-terns of organic molecules in the NIR region [12]. Hyperspectral ima-ging technology has been widely used in seed species [9–11,13,14],seed quality [15–18], and food [19–22] identification. Because diversetypes of hybrid seeds contain different internal information, the

https://doi.org/10.1016/j.snb.2019.126630Received 25 January 2019; Received in revised form 10 May 2019; Accepted 29 May 2019

⁎ Corresponding author at: College of Biosystems Engineering and Food Science, Zhejiang University, 866 Yuhangtang Road, Hangzhou 310058, China.E-mail addresses: npc2012@zju.edu.cn (P. Nie), jnzhang@zju.edu.cn (J. Zhang), pimmmx@163.com (X. Feng), 21007030@zju.edu.cn (C. Yu),

yhe@zju.edu.cn (Y. He).

Sensors & Actuators: B. Chemical 296 (2019) 126630

Available online 31 May 20190925-4005/ © 2019 Elsevier B.V. All rights reserved.

T

expression of the spectral information also differs. Seeds can be quicklyand non-destructively identified using this spectral information incombination with a modeling algorithm. Silvia et al. applied NIR hy-perspectral imaging technology to classify oat and groat kernels, andthe prediction accuracy was nearly 100% [13]. Three different grapevarieties were classified using their NIR spectra combined with a partialleast squares regression (PLSR) and a principle component analysis(PCA) [14]. Additionally, the varieties of cotton seeds were identified,and a discriminant model based on a partial least squares discriminantanalysis (PLS-DA) was established using the spectra of 807 seeds of fourdifferent varieties, which had correct classification rates of over 89.7%[23]. A support vector machine (SVM), neural network with radial basisfunction, and extreme learning machine (ELM) were used to builddiscriminant models to identify four different varieties of Chinesewolfberry combined with NIR hyperspectral imaging technology. Theaccuracy of the prediction based on the ELM was 91.25% [9]. Quan-titative predictions were performed using NIR hyperspectral imaging ofcoffee beans, and only 260 samples were applied to building the PLSRsof sucrose, caffeine, and trigonelline [20]. In these studies, there werefew classifications of the more than five varieties, and the sample sizeused to establish the discriminant analysis models was relatively small.Traditional machine-learning algorithms are widely used, but the cor-rect recognition rate for different classification objects is neither highnor stable enough. Jacopo et al. showed the effectiveness of convolu-tional neural networks (CNNs) to classify spectral data [24]. The ap-plication of deep learning algorithms has great potential. NIR hyper-spectral imaging technology combined with deep learning is still astate-of-the-art method in hybrid seed classification.

Deep learning is a branch in the field of machine learning. In recentyears, deep learning has been applied to making breakthroughs invarious applications, such as computer vision [25] and speech re-cognition [26], and has made significant contributions to the realiza-tion of artificial intelligence. Deep learning algorithms can processinput data layer by layer, transform the features of the original spaceinto the new features’ representative space, and automatically learn thehierarchical features to effectively classify and characterize the data.Compared with traditional neural networks, the prominent feature isthat the number of hidden layers increases, which makes the corre-sponding optimization algorithm more efficient. Chen et al. used a deeplearning algorithm to process hyperspectral data [27]. To solve thehigh-dimensional problem associated with hyperspectral data, theyused a multi-layer stacking autoencoder combined with logistic re-gression to successfully extract hyperspectral data features and classifyland cover. A deep CNN (DCNN) was first introduced by Hu et al. toclassify hyperspectral images directly in the spectral domain. TheDCNN was superior to traditional methods like SVM. Wu et al. classifiedchrysanthemum varieties using hyperspectral imaging combined with aDCNN, and they found that a DCNN based on full wavelengths wasnearly 100% accurate in classifying both training and test sets [28].Deep learning algorithms have become the choice for building morereliable models for nondestructive testing. In this paper, we collectedhybrid okra and hybrid loofah seeds using NIR hyperspectral tech-nology. The effects of traditional and deep learning algorithms on theclassification of different hybrid seeds were compared.

The specific objectives of this study were as follows: (1) to examinethe ability to identify different hybrid seeds based on NIR hyperspectralimaging technique combined with a DCNN; (2) to compare the per-formance of discriminant models based on traditional modeling anddeep learning algorithms and select the best algorithm to build a model;(3) to examine the changes in classification results for different varietiesof hybrid seeds; and, (4) to visualize the distribution characteristics ofthe raw high-dimensional hyperspectral data and data processed by adeep learning algorithm, and visually demonstrate the effectiveness ofdeep learning algorithms.

2. Material and methods

2.1. Sample preparation

A total of 6136 hybrid okra seeds and 4128 hybrid loofah seedswere provided by the Zhejiang Academy of Agricultural Sciences,Zhejiang, China. The hybrid okra seeds included six different varieties:2014HK2, 2014HK4, 2014HK16, baiguo, cuizhi, and danzhi, and thehybrid loofah seeds also included six different varieties: zhesi35,1.35M, guangF, 25 F, guangM, and guangl. All of the hybrid okra andloofah seeds were collected and sealed in plastic bags in 2017, and themoisture absorption of the seeds was controlled to minimize environ-mental impact. The six different varieties of each plant were coded as 1,2, 3, 4, 5, and 6 on the basis of their names in the order of those pre-sented in this paragraph for later data processing. The data set of hybridseeds was randomly divided into training and testing sets at a ratio of2:1. No additional processing of the hybrid seeds was performed.

An untargeted metabolomics approach was applied to examiningthe metabolic variations between those hybrid seeds. GenedenovoBiotechnology Company in china helped to finish the metabolomicsexperiments. LC–MS/MS analyses were performed using an UHPLCsystem (1290, Agilent Technologies) with a UPLC HSS T3 column(2.1mm×100mm, 1.8 μm) coupled to Q Exactive (Orbitrap MS,Thermo). Metabolite heatmaps were drawn using twenty metaboliteswith significant differences to represent differences within the classifiedhybrid seeds.

2.2. NIR hyperspectral imaging system

A line-scan hyperspectral imaging system was used to collect thehyperspectral images of hybrid seeds. The laboratory-built hyperspec-tral imaging system consisted of an imaging spectrograph (ImSpectorN17E; Spectral Imaging Ltd., Oulu, Finland) with a spectral range of874–1734 nm and spectral resolution of 5 nm, a high-performance CCDcamera (C8484-05; Hamamatsu, Hamamatsu City, Japan) which has aresolution of 320× 256 pixels coupled with a camera lens (OLES22;Specim, Spectral Imaging Ltd., Oulu, Finland) and so on. The devicemodel and composition in detail of the whole system can be found inthe paper of Feng et al. [29]. The process of hyperspectral image ac-quisition was performed in a dark room. Initially, the hybrid seeds wereplaced evenly on a black plate, and then, the black plate was placed ona conveyor belt. The height between the camera lens and hybrid seedswas set to 15 cm. Clear and undistorted hyperspectral images werecollected while the conveyor belt moved at 15mm/s, and the exposuretime of the camera was set to 3ms. To obtain hyperspectral imageswithout dark-current effects, the raw hyperspectral images were cor-rected using the following Eq. (1)

=−

I I - II IC

raw dark

white dark (1)

where Iraw represents the raw hyperspectral image; Idark represents thedark reference image, and Iwhite represents the white reference image.The dark reference image was obtained by covering the lens completelywith its opaque cap, and the white reference image was acquired byusing a white Teflon tile with nearly 100% reflectance.

2.3. Spectral collection

The mean spectra of the whole seeds were chosen as the region ofinterest (ROI) to represent the spectras of the samples. So the back-ground spectral information needed to be removed. Threshold seg-mentation is simple and efficient, and it is widely used in the processingof hyperspectral images [9–11,13,14]. A threshold segmentation bandof 1109 nm was selected, and the difference between the hybrid seedreflectance and the background reflectance reached the maximum atthis band. To eliminate the influence of the external environment and

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

2

camera performance, spectral information in the range of 975–1645 nmwas obtained by removing the front and back bands having obviousnoise. The raw spectral information was first implemented using wa-velet transform pre-processing technology to eliminate the noise. Then,Daubechies 8, with a decomposition scale of 3, was applied. Finally,owing to the instability of the pixel-based spectral information, all thespectral information in pixels of the same seed were averaged to obtainan average spectra, which was then applied to the discriminationanalysis.

2.4. Multivariate data analysis

Three different methods, PLS-DA, SVM, and DCNN, were used toclassify the hybrid seeds of the two species. An excellent high-dimen-sional data visualization method, t-distribution stochastic neighborembedding (t-SNE), was used to explore the feature extraction cap-ability of the DCNN. A flow chart of the article is shown in Fig. 1.

2.4.1. Deep convolutional neural networkA DCNN consists of various combinations of convolutional layers,

max-pooling layers, and fully connected layers. The architecture of theDCNN in this paper is shown in Fig. 2. There were two convolutionalmodules that included a convolutional layer, a max pooling layer, andfive fully connected layers. The number of filters for the first and secondconvolutional layers were set to 32 and 64. The average spectrum of thehybrid seeds was normalized before being sent to the DCNN. One-di-mensional 1× 3 convolution kernels were used to process the averagespectrum of the hybrid seeds. The 1×3 convolution kernels, which hadtheir stride and padding set to 1, could quickly learn the local featuresof the spectral information, reduce the dimension of the spectral data,and increase the nonlinearity of the data processing. Max-poolinglayers, which had their kernel size set to 1× 3, were used to selectfeatures, and the spectral information and the stride of the max-poolinglayers were set to 1. As shown in Fig. 2, the hybrid seed spectra wereprocessed by two convolution modules to flatten all the neurons into a

one-dimensional structure, and then, they were fed into the five fullyconnected layers. The number of neurons was successively decreasedfrom 512.

To reduce the influence of the gradient disappearance problem andincrease the nonlinearity of the deep learning structure, RectifiedLinear Units (ReLUs) were used as the activation functional units. ACNN with ReLUs could train the model faster than networks with olderunits and prevent the over-fitting of the model. This is shown in thefollowing Eq. (2):

= ⎧⎨⎩

>≤ReLU(x) x if x 0

0 if x 0 (2)

where x represents the feature values of the neurons. The distribution ofthe inputs in each layer changes during the training of a DCNN. Thus,the whole process of training the model becomes very complicated andslow, and lower learning rates and careful parameter initialization arenecessary, leading to internal covariate shifts [30]. Sergey et al. usedbatch normalization to normalize layer inputs. In this paper, batchnormalization was also used before every fully connected layer andafter every convolutional module to produce a higher learning rate andfocus less on initialization [31]. The softmax function was introduced asthe objective function. It can normalize the data passed by the fullyconnected layer, highlight the greatest value, and suppress other com-ponents below the maximum. The categorical cross entropy loss func-tion was used to measures the distance between the probability dis-tribution of the DCNN output value and the probability distribution ofthe real value. The parameters of the whole structure were changed bythe gradient descent method. An adaptive moment estimation algo-rithm, with a learning rate of 0.001, beta_1 of 0.9, and beta_2 of 0.99,was used to minimize the loss of function. The method is straightfor-ward to implement, is computationally efficient, has limited memoryrequirements, is invariant to the diagonal rescaling of the gradients, andis well suited to address problems that are great in terms of data and/orparameters [32,33]. To monitor and prevent the over-fitting problemthat occurs during the training process, 30% of the training set was

Fig. 1. Experimental flow chart.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

3

selected as the validation set when choosing the best model, and drop-out methods were applied after the batch normalization of each fullyconnected layer. Mini-batch training methods, with sizes set to 35, werechosen to achieve a greater accuracy and a more rapid model con-vergence. Finally, the number of iterations of the network was de-termined according to the performance of the model.

2.4.2. Partial least squares discriminant analysisA PLS-DA is a typical method for classification, and it is considered

as a supervised method of maximum discrimination between the sam-ples [34]. Leave-one-out cross validation was applied to building thePLS-DA model, and the absolute value of the difference between thereal classification number and predicted value of the model was used tocalculate the discrimination accuracy of the training and test sets.

2.4.3. Support vector machineThe Vapnik-Chervonenkis dimensional theory and structural risk

minimization principle make SVM a very effective method in the fieldof pattern recognition [9,35]. SVM is radically different from neuralnetworks in that SVM training always finds a global minimum andfewer training samples are required. A radial basis function (RBF) waschosen as the kernel function. The penalty parameters (c) and kernelfunction parameters (g) were determined using a grid-search procedurein the range of 2−8–28.

2.5. Visualization method

Because DCNN has a black box phenomenon during data processing,the t-SNE method was used to visualize the features of the hidden layerand the raw spectral data. The t-SNE, which is capable of retaining thelocal structure of the data, can visualize high-dimensional data bygiving each data point a location on a two or three-dimensional map[36]. Yang et al. applied t-SNE to visualize their learning spectral-spatial features and achieved outstanding results [37]. The t-SNE al-gorithm models the distribution of the neighbors of each data point.Additionally, data in the high-dimensional space are modeled as aGaussian distribution, while data in the two-dimensional output spaceare modeled as a t-distribution.

2.6. Software tools

MATLAB R2013b (The Math-Works, Natick, MA, USA) was used toextract average spectra of hybrid seeds from the raw hyperspectralimages. Unscrambler X 10.1 (CAMO AS, Oslo, Norway) was used toestablish and test the PLS-DA models. SVM models were implementedusing MATLAB R2013b, and Python 3.6 (https://www.python.org/)with Jupyter Notebook was used to build the DCNN model. The famousdeep learning framework Keras (https://keras.io/zh/) was used toprogram the architecture of the DCNN model, which ran in centralprocessing unit (CPU). Origin Pro 9.0 (Origin Lab Corporation,Northampton, MA, USA) and EDraw Max (Shenzhen EDraw SoftwareCO, Shenzhen, China) were used to construct the graphs. The Win1064-bit operating system, with Inter (R) Core(TM) i5-7500 CPU,3.40 GHz, and 8GB RAM, as the software platform, carried out allsoftware operations.

3. Results and discussion

3.1. Spectroscopic analysis

The raw spectra of all varieties of hybrid loofah seeds and hybridokra seeds are presented in Fig. 3A and B. The average spectrum of eachvariety is shown in Fig. 3C and D. The spectral curves of hybrid okraseeds and hybrid loofah seeds were quite different, owing to the dif-ferences in the composition of the seed epidermises. The spectral curvesof hybrid loofah seeds were similar to those of lotus in the NIR region of1000–1700 nm [38]. Feng et al. obtained the spectral information oftransgenic maize kernels using NIR hyperspectral imaging technologyand found curve changes similar to those of our hybrid okra seeds in thesame band [29]. For hybrid loofah seeds, the spectral curves of thedifferent varieties showed the same trend, but their reflectance valueswere different (Fig. 3C). This might result from differences in the ge-netic information caused by cross breeding, which manifested in dif-ferences in the gene expression levels. The untargeted metabolomicsdetection results also proved that there were obvious differences be-tween those hybrid seeds. As shown in Fig. 4A and B, there were ob-servable differences in the metabolite heat maps of different varieties ofhybrid loofah seeds. Twenty different metabolites were shown in the

Fig. 2. The architecture of the proposed DCNN.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

4

figure which included ester metabolites (brassinolide, fenvalerate,erioflorin methacrylate), alcohol metabolites (campestanol, octade-canol), glucose, amino acid (glycine, Z-Tyr-Leu-NHOH) and so on. Sameresults were also gotten from the metabolite heatmaps of hybrid okraseeds in Fig. 5A and B. It was exactly because of the differences in themetabolites whining those hybrid loofah and okra seeds that NIR hy-perspectral technology could capture enough information and use it to

build discriminant model. As seen in Fig. 3C, the average spectrum ofhybrid loofah seeds had a valley in which the most obvious absorptionbands were located. The valley at around 1450 nm was the result of thefirst overtone of OeH and NeH stretching vibrations [13,28]. The firstovertone of the NeH stretching vibration in the range of 1410–1502 nmwas correlated with the protein content, which could be used to classifydifferent hybrid loofah seeds [39]. Hybrid okra seeds had different

Fig. 3. The spectra of hybrid seeds of loofah and okra. A, B: the profiles of raw spectra of all hybrid loofah seeds and all hybrid okra seeds; C, D: average spectralreflectance of hybrid loofah seeds of six varieties and hybrid okra seeds of six varieties.

Fig. 4. The metabolite heatmaps of hybrid loofah seeds. A: extracted from the positive ion mode; B: extracted from the negative ion mode.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

5

Fig. 5. The metabolite heatmaps of hybrid okra seeds. A: extracted from the positive ion mode; B: extracted from the negative ion mode.

Table 1Discrimination results of hybrid loofah seeds varieties by different models.

The Number of Varieties The Number of Seeds Discriminant Model Parameters1 Training Set Testing Set

2 783 PLS-DA 10 99.80% 99.60%715 SVM (147.0334,0.5743) 99.80% 99.60%

DCNN (2,32,300) 99.47% 100.00%3 783 PLS-DA 10 97.90% 97.80%

715 SVM (147.0334,0.5743) 99.80% 98.91%712 DCNN (2,32,200) 99.58% 99.64%

4 783 700 PLS-DA 10 71.40% 73.10%715 SVM (256,0.5743) 99.38% 98.86%712 DCNN (2,32,200) 99.45% 99.31%

5 783 700 PLS-DA 12 46.60% 49.60%715 616 SVM (256,1.7411) 98.68% 97.28%712 DCNN (2,32,400) 99.02% 99.09%

6 783 700 PLS-DA 10 43.50% 44.10%715 616 SVM (256,0.5743) 93.02% 90.48%712 602 DCNN (2,32,900) 98.06% 95.93%

Note: PLS-DA model’s parameter means the optimal number of LVs; SVM model’s parameters mean different penalty parameters (c) and kernel function parameters(g), shown as (c,g); DCNN model’s parameters mean the number of convolution layers, the number of the first kernels and the epoch.

Table 2Discrimination results of hybrid okra seeds varieties by different models.

The Number of Varieties The Number of Seeds Discriminant Model Parameters1 Training Set Testing Set

2 966 PLS-DA 9 99.85% 99.07%1010 SVM (256,1) 100.00% 99.09%

DCNN (2,32,120) 99.87% 100.00%3 966 PLS-DA 12 93.10% 93.37%

1010 SVM (256,0.1895) 98.98% 98.18%828 DCNN (2,32,480) 99.24% 99.71%

4 966 1136 PLS-DA 9 87.55% 86.53%1010 SVM (256,0.1859) 99.28% 98.78%828 DCNN (2,32,200) 99.53% 99.59%

5 966 1136 PLS-DA 10 50.76% 50.65%1010 1099 SVM (256,1) 99.52% 97.92%828 DCNN (2,32,300) 99.47% 98.89 %

6 966 1136 PLS-DA 10 47.48% 48.58%1010 1099 SVM (256,1) 98.70% 96.14%828 1097 DCNN (2,32,500) 98.94% 98.24%

Note: PLS-DA model’s parameter means the optimal number of LVs; SVM model’s parameters mean different penalty parameters (c) and kernel function parameters(g), shown as (c,g); DCNN model’s parameters mean the number of convolution layers, the depth of the first convolution layer and the epoch.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

6

spectral curves compared with hybrid loofah seeds. The same absorp-tion band at 1450 nm was also observed in the average spectral curvesof hybrid okra seeds (Fig. 3D). The second overtone of CeH stretchingvibrations in the range of 1410–1502 nm could be found in the averagespectral curves of hybrid okra seeds. At the same time, the values of the

spectral reflectance between different varieties of hybrid okra seedswere also different, as seen in Fig. 3D. However, distinguishing dif-ferent varieties of hybrid seeds based on the difference in reflectancevalues of spectral curves was unreliable because there was the problemof overlapping in the spectral curves. Therefore, it was necessary toestablish discriminant analysis models to effectively extract and use thefeatures in the spectra to classify hybrid seeds.

3.2. Classification results and analysis

In this step, discriminant analysis models based on 200 bands wereimplement using PLS-DA, SVM, and DCNN. The discriminant results ofthe different models of hybrid loofah seeds and hybrid okra seeds areshown in Tables 1 and 2, respectively. There were clearly differences inthe results of the various models for different varieties of hybrid loofah

Fig. 6. The loss and accuracy curves of the discriminant model of hybrid loofah seeds based on the DCNN. Classifications of two (A), three (B), four (C), five (D), andsix (E) varieties of hybrid loofah seeds.

Table 3Discrimination results of hybrid loofah and okra seeds varieties by DCNN(without batch normalization and dropout).

The Number of Varieties Species Parameters1 Training Set Testing Set

6 Loofah (2,32,900) 96.44% 93.87%6 Okra (2,32,500) 98.06% 96.12%

Note: DCNN model’s parameters mean the number of convolution layers, thedepth of the first convolution layer and the epoch.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

7

seeds (Table 1). The performance of the PLS-DA model decreased as thenumber of hybrid loofah seed varieties increased. When the varietynumber of input seeds in the model increased from two to six, thetesting set accuracy of the model decreased from 99.6% to 44.10%,respectively. PLS-DA is a linear classification method that combinesPLSR and classification techniques [40]. As a nonlinear machinelearning algorithm, SVM uses nonlinear hyperplanes to classify complexdata [35]. With the increase in the number of hybrid loofah seedvarieties, the discriminant results of the SVM decreased slightly, buttraining and test sets recognition accuracies of 93.02% and 90.48%,respectively, were finally realized. Zhao et al. applied a radial basisfunction neural network to classify maize seeds, and the best calibrationand prediction accuracies of the model based on full bands were98.03% and 93.26%, respectively [10]. Neural network models and anonlinear models, such as SVM, were better than linear models in

classifying seeds [11,41–43]. A discriminant model based on the DCNNachieved training and test set recognition accuracies of 98.06% and95.93%, respectively, which were greater than the recognition ac-curacies obtained by SVM in the identification of six different varietiesof hybrid loofah seeds. As shown in Table 1, as the number of varietiesof hybrid loofah increased, the discriminant results of DCNN remainedstable, and the accuracies of the training and test sets were alwaysgreater than 98% and 95%, respectively. The spectral data contained alot of deep features, and it was more efficient to extract features usingthe DCNN. Deep learning models usually involve a deep architecture,and the deep architecture could extract more abstract and unchangingdata features, and resulted in a better performance level, than the tra-ditional shallow classifiers [27]. The loss and accuracy curves of thediscriminant model of hybrid loofah seeds based on the DCNN areshown in Fig. 6A–E. As the number of epochs increased, the loss value

Fig. 7. The loss and accuracy curves of the discriminant model of hybrid okra seeds based on the DCNN. Classifications of two (A), three (B), four (C), five (D), and six(E) varieties of hybrid okra seeds.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

8

of the discriminant analysis model decreased continuously, but theaccuracy rate increased continuously and eventually stabilized, whichindicated that the discriminant model based on the DCNN had con-verged.

The discriminant results of the classification of hybrid okra seedsare shown in Table 2. In the identification of six varieties of hybrid okraseeds, the training and test sets’ accuracies using a discriminant modelbased on PLS-DA dropped to 47.87% and 48.58%, respectively, whichwas similar to the classification of hybrid loofah seeds. As shown inTable 2, the discriminant analysis model established by SVM and DCNNcould maintain a stable relatively high-performance level, but the SVM-

based discriminant analysis model had a problem of over-fitting. In fact,discriminant models have been shown to undergo over-fitting in seedclassification using NIR hyperspectral imaging technology [9,10,43].However, as shown in Table 2, the training and test sets’ accuracieswhen using the discriminant model based on the DCNN could reach98.94% and 98.24%, respectively, when classifying six varieties ofhybrid okra seeds. As shown in Table 3, methods such as batch nor-malization and dropout could be used to alleviate over-fitting and in-crease the stability of the discriminant model compared with the resultsgotten from Table 1 and Table 2. As seen in Fig. 7A–E, as the number ofepochs increased, the accuracy levels of the training and test set

Fig. 8. The visualization maps of hybrid loofah seeds using a PCA and t-SNE.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

9

predictions using the model based on the DCNN tended to be stable andover-fitting was not obvious.

Wu et al. achieved similar results when using a discriminant ana-lysis model based on the DCNN to classify Chrysanthemum species [28].Their DCNN model classified seven varieties of dried Chrysanthemumand produced a testing set accuracy of 99.98%. Qiu et al. used a CNNmodel to classify single rice seeds, and the effectiveness of the CNN wasproven [43]. Four varieties of rice seeds were collected to build theCNN model. They compared the classification results based on differentdiscriminant models between two spectral ranges and the CNNachieved the best test set accuracy of 87% in the range of 975–1646 nm.

The discriminant analysis models established by deep learning algo-rithms are superior [28,43,44]. With an increase in the numbers ofvarieties of hybrid okra and loofah seeds, the accuracy of the classifi-cation model based on the DCNN significantly improved compared withthose of models based on SVM and PLS-DA. Additionally, the dis-criminant model based on the DCNN used optimization algorithms toalleviate the model’s over-fitting problem. Thus, the DCNN has poten-tial applications in seed classification based on hyperspectral imagingtechnology. It was very accurate in identifying hybrid seeds.

Fig. 9. The visualization maps of hybrid okra seeds using a PCA and t-SNE.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

10

3.3. Visualizing data using t-SNE

Traditional dimensionality reduction techniques, such as PCA, arelinear techniques that focus on maintaining the low-dimensional re-presentations of faraway dissimilar data points [36]. Figs. 6 and 7present PCA results for hybrid loofah and okra seeds based on full-bandclassifications. As shown in both Figs. 6 and 7, with an increase in thenumbers of hybrid seed varieties, the problem of overlap among dif-ferent varieties in the PCA became more obvious. Additionally, it wasdifficult for the PCA to provide a visual classification of hybrid seeds.

To intuitively demonstrate the effectiveness of the DCNN for hybridseed classification based on NIR hyperspectral technology, a t-SNE wasused to visualize spectral data. It can represent high-dimensional datain a low-dimensional non-linear manifold by setting similar data pointsvery close [36]. Using a t-SNE to visually compare the raw spectral datawith the feature values of the last layer of the DCNN on a two-dimen-sional plane, the validity of the DCNN could be visually observed inFigs. 8 and 9. The raw high-dimensional hyperspectral data of differentvarieties of hybrid seeds was mixed and overlapped (Figs. 6 and 7).After the raw hyperspectral data was sent to the discriminant modelbased on the DCNN, the feature values of different varieties of hybridseeds represented by different colors were collected from the last layerof DCNN, and the feature values were clearly separated. As the numbersof varieties of hybrid seeds increased, the visualization results becamemore apparent. There were clearly boundaries between the visualiza-tion data obtained from the DCNN of the different varieties of hybridseeds in all the visualization maps. Therefore, this method had a verysignificant effect on the classification of hybrid seeds of different spe-cies based on NIR hyperspectral imaging technology. This approach hadthe potential to explain the effectiveness of the DCNN in processinghyperspectral data for hybrid seeds and to establish rapid visual clas-sification methods.

4. Conclusion

In crop cross breeding, it is essential to identify and screen hybridseeds that meet specific requirements. Based on the method of NIRhyperspectral imaging technology combined with deep learning, thisstudy identified the hybrid seeds of okra and loofah. Six differentvarieties each of hybrid loofah and hybrid okra seeds were collected toobtain the spectral information. A deep learning algorithm establishedthe discriminant model used to classify the hybrid seeds that belongedto the two species, and the results of the DCNN were compared with theresults of discriminant models based on a PLS-DA and SVM. As thenumber of hybrid seed varieties increased, the performance of theDCNN remained superior. Compared with the PLS-DA model, theclassification results of the DCNN model remained at 98% as thenumber of hybrid seed varieties increased from two to six. Comparedwith the SVMmodel, the performance of the DCNN remained stable andalleviated the problem of overfitting. At the same time, we used a t-SNEto study the results of deep learning on hybrid seed classificationcombined with NIR hyperspectral technology. This method demon-strated that it was feasible to classify hybrid seeds using the DCNNcombined with NIR hyperspectral imaging technology. The visualiza-tion method t-SNE was used to explain the effectiveness of deeplearning in hybrid seed classification. In the future, we could add morehybrid seeds to build a hybrid seed database based on spectral finger-prints. As the amount of seed’ spectral data increases, discriminantmodels based on deep learning algorithms could be applied to buildingvarietal libraries that would allow the identification of more hybridseeds of different species. This method can be used to analyze hybridsresulting from the cross breeding of crops and speed up related pro-cesses.

Conflicts of interest

The authors declare no conflict of interest. The funders had no rolein the design of the study; in the collection, analyses, or interpretationof data; in the writing of the manuscript, or in the decision to publishthe results.

Funding

This research was funded by the National Natural ScienceFoundation of China (Grant No. 31801257) and the National Key R&Dprogram of China (2016YFD0300606).

References

[1] L.P. Yuan, Development of hybrid rice to ensure food security, Rice Sci. 21(2014) 1–2.

[2] G.M. Maciel, J.M. Luz, S.F. Campos, R.R. Finzi, B.N. Azevedo, Heterosis in okrahybrids obtained by hybridization of two methods: traditional and experimental,Hortic. Bras. 35 (2017) 119–123.

[3] L. Yuan, Progress in super-hybrid rice breeding, Crop J. 5 (2017) 100–102.[4] P.S. Kumar, S. Sreeparvathy, Studies on heterosis in okra (abelmoschus esculentus (l.)

moench), Electron. J. Plant Breed. 1 (2010) 1431–1433.[5] O.E. Adelakun, O.J. Oyelade, B.I.O. Ade-Omowaye, I.A. Adeyemi, M.V.D. Venter,

Chemical composition and the antioxidative properties of nigerian okra seed(abelmoschus esculentus moench) flour, Food Chem. Toxicol. 47 (2009) 1123–1126.

[6] T.B. Hlel, F. Belhadj, F. Gül, M. Altun, A.Ş. Yağlıoğlu, I. Demirtaş, M.M. Nejib,Variations in the bioactive compounds composition and biological activities ofloofah (luffa cylindrica) fruits in relation to maturation stages, Chem. Biodivers. 14(2017) e1700178.

[7] T.H. Kao, C.W. Huang, B.H. Chen, Functional components in luffa cylindrica andtheir effects on anti-inflammation of macrophage cells, Food Chem. 135 (2012)386–395.

[8] T. Seth, A. Chattopadhyay, S. Chatterjee, S. Dutta, B. Singh, Selecting parental linesamong cultivated and wild species of okra for hybridization aiming at yvmv diseaseresistance, J. Agric. Sci. Technol. 18 (2016) 751–762.

[9] W. Yin, C. Zhang, H. Zhu, Y. Zhao, Y. He, Application of near-infrared hyperspectralimaging to discriminate different geographical origins of chinese wolfberries, PLoSOne 12 (2017) e0180534.

[10] Y. Zhao, S. Zhu, Z. Chu, X. Feng, F. Lei, H. Yong, Application of hyperspectralimaging and chemometrics for variety classification of maize seeds, RSC Adv. 8(2018) 1337–1345.

[11] X. Yang, H. Hong, Z. You, F. Cheng, Spectral and image integrated analysis of hy-perspectral data for waxy corn seed variety classification, Sensors 15 (2015)15578–15594.

[12] P. Williams, K. Norris, Near-infrared technology in the agricultural and food in-dustries, Am. Assoc. Cereal Chem. (1987) 293–313.

[13] S. Serranti, D. Cesare, F. Marini, G. Bonifazi, Classification of oat and groat kernelsusing NIR hyperspectral imaging, Talanta 103 (2013) 276–284.

[14] F.J. Rodríguez-Pulido, D.F. Barbin, D.W. Sunb, M.L. González-Miret, F.J. Heredia,Grape seed characterization by NIR hyperspectral imaging, Postharvest Biol.Technol. 76 (2013) 74–82.

[15] A. Ambrose, L.M. Kandpal, M.S. Kim, W.H. Lee, B.K. Cho, High speed measurementof corn seed viability using hyperspectral imaging, Infrared Phys. Technol. 75(2016) 173–179.

[16] L. Feng, S. Zhu, C. Zhang, Y. Bao, X. Feng, Y. He, Identification of maize kernel vigorunder different accelerated aging times using hyperspectral imaging, Molecules 23(2018) 3078.

[17] C. Nansen, G. Zhao, N. Dakin, C. Zhao, S.R. Turner, Using hyperspectral imaging todetermine germination of native australian plant seeds, J. Photochem. Photobiol. B145 (2015) 19–24.

[18] S. Shrestha, M. Knapič, U. Žibrat, L.C. Deleuran, R. Gislum, Single seed near-in-frared hyperspectral imaging in determining tomato (solanum lycopersicum l.) seedquality in association with multivariate data analysis, Sens. Actuators B Chem. 237(2016) 1027–1034.

[19] A.A. Gowen, C.P. O’Donnell, P.J. Cullen, G. Downey, J.M. Frias, Hyperspectralimaging – an emerging process analytical tool for food quality and safety control,Trends Food Sci. Technol. 18 (2007) 590–598.

[20] N. Caporaso, M.B. Whitworth, S. Grebby, I.D. Fisk, Non-destructive analysis of su-crose, caffeine and trigonelline on single green coffee beans by hyperspectralimaging, Food Res. Int. 106 (2017) 193–203.

[21] D. Barbin, G. Elmasry, D.W. Sun, P. Allen, Near-infrared hyperspectral imaging forgrading and classification of pork, Meat Sci. 90 (2012) 259–268.

[22] D.F. Barbin, G. Elmasry, D.W. Sun, P. Allen, Non-destructive determination ofchemical composition in intact and minced pork using near-infrared hyperspectralimaging, Food Chem. 138 (2013) 1162–1171.

[23] S.F.C. Soares, E.P. Medeiros, C. Pasquini, C.D.L. Morello, R.K.H. Galvão,M.C.U. Araújo, Classification of individual cotton seeds with respect to variety usingnear-infrared hyperspectral imaging, Anal. Methods 8 (2016) 8498–8505.

[24] J. Acquarelli, T.V. Laarhoven, J. Gerretzen, T.N. Tran, L.M.C. Buydens,E. Marchiori, Convolutional neural networks for vibrational spectroscopic data

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

11

analysis, Anal. Chim. Acta 954 (2017) 22–31.[25] J. Yu, C. Hong, Y. Rui, D. Tao, Multi-task autoencoder model for recovering human

poses, IEEE Trans. Ind. Electron. 65 (2018) 5060–5068.[26] H. Ze, A. Senior, M. Schuster, Statistical parametric speech synthesis using deep

neural networks, IEEE International Conference on Acoustics, Speech and SignalProcessing, (2013), pp. 7962–7966.

[27] Y. Chen, Z. Lin, X. Zhao, G. Wang, Y. Gu, Deep learning-based classification ofhyperspectral data, IEEE J. Sel. Top. Appl. Earth Obs. Remote. Sens. 7 (2014)2094–2107.

[28] N. Wu, C. Zhang, X. Bai, X. Du, Y. He, Discrimination of chrysanthemum varietiesusing hyperspectral imaging combined with a deep convolutional neural network,Molecules 23 (2018) 2831.

[29] X. Feng, Y. Zhao, C. Zhang, P. Cheng, Y. He, Discrimination of transgenic maizekernel using NIR hyperspectral imaging and multivariate data analysis, Sensors 17(2017) 1894.

[30] L. Wei, G. Wu, Z. Fan, D. Qian, Hyperspectral image classification using deep pixel-pair features, IEEE Trans. Geosci. Remote. Sens. 55 (2016) 844–853.

[31] S. Ioffe, C. Szegedy, Batch normalization: accelerating deep network training byreducing internal covariate shift, International Conference on InternationalConference on Machine Learning JMLR.org 37 (2015) 448–456.

[32] A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep con-volutional neural networks, International Conference on Neural InformationProcessing Systems 1 (2012) 1097–1105.

[33] D. Kingma, J. Ba, Adam: a method for stochastic optimization, Comput. Sci. (2014).[34] M.R. Almeida, C.H. Fidelis, L.E. Barata, R.J. Poppi, Classification of amazonian

rosewood essential oil by raman spectroscopy and pls-da with reliability estimation,Talanta 117 (2013) 305–311.

[35] C.J.C. Burges, A tutorial on support vector machines for pattern recognition, DataMin. Knowl. Discov. 2 (1998) 121–167.

[36] L. Van der Maaten, G. Hinton, Viualizing data using t-SNE, J. Mach. Learn. Res. 1(2008) 1–48.

[37] J. Yang, Y.Q. Zhao, C.W. Chan, Learning and transferring deep joint spectral-spatialfeatures for hyperspectral classification, IEEE Trans. Geosci. Remote. Sens. 55(2017) 4729–4742.

[38] G. Ying, X. Ding, Y. Ni, The combination of NIR spectroscopy and HPLC chroma-tography for differentiating lotus seed cultivars and quantitative predicting of fourmain constituents in lotus with the aid of chemometrics, Anal. Methods 9 (2017)6420–6429.

[39] M. Daszykowski, M.S. Wrobel, H. Czarnik-Matusewicz, B. Walczak, Near-infraredreflectance spectroscopy and multivariate calibration techniques applied to mod-elling the crude protein, fibre and fat content in rapeseed meal, Analyst 133 (2008)1523–1531.

[40] D. Ballabio, V. Consonni, Classification tools in chemistry. Part 1: linear models.PLS-DA, Anal. Methods 5 (2013) 3790–3798.

[41] W. Kong, C. Zhang, F. Liu, P. Nie, Y. He, Rice seed cultivar identification using near-infrared hyperspectral imaging and multivariate data analysis, Sensors 13 (2013)8916–8927.

[42] C. Wakholi, L.M. Kandpal, H. Lee, H. Bae, E. Park, M.S. Kim, C. Mo, W.H. Lee,B.K. Cho, Rapid assessment of corn seed viability using short wave infrared line-scan hyperspectral imaging and chemometrics, Sens. Actuators B Chem. 255 (2018)498–507.

[43] Z. Qiu, C. Jian, Y. Zhao, S. Zhu, H. Yong, Z. Chu, Variety identification of single riceseed using hyperspectral imaging combined with convolutional neural network,Appl. Sci. 8 (2018) 212.

[44] X. Kang, C. Li, S. Li, L. Hui, Classification of hyperspectral images by gabor filteringbased deep network, Ieee J. Sel. Top. Appl. Earth Obs. Remote. Sens. 11 (2017)1166–1178.

P. Nie, et al. Sensors & Actuators: B. Chemical 296 (2019) 126630

12