prototype report - software engineering ii - spring 2018 · web viewintroduction 1.1 purpose of the...

Prototype Report (PRO) Version 3.0

Image Processing Platform

Team 04

12/04/2016

PRO_TRR_F16a_T04_V3.0.doc Version Date: 12/03/2016

Name First Role Second RoleHao Wu Requirements Engineer Software Architect

Junran Liu Operational Concept Engineer Software ArchitectMeiyi Yang Project Manager Life Cycle Plan

Vinny DeGenova IIV & V Quality Focal PointXiangchen Zhao Life Cycle Plan Prototyper

Xinhui Liu Feasibility Analyst Operational Concept EngineerYifan Liu Prototyper Requirements Engineer


Version HistoryDate Author Version Changes made Rationale

10/05/16 Yifan Liu, Xiangchen Zhao 1.0 Created a draft version Initial draft

10/15/16 Yifan Liu 2.0 Added images Improve the first draft

12/03/16 Yifan Liu 3.0 Final modification Final version



Table of ContentsPrototype Report............................................................................................................................................................iVersion History.............................................................................................................................................................iiTable of Contents.........................................................................................................................................................iiiTable of Tables.............................................................................................................................................................ivTable of Figures.............................................................................................................................................................v1. Introduction...............................................................................................................................................................1

1.1 Purpose of the prototype.................................................................................................................................1

1.2 Status of the prototype..............................................................................................................................2

2. Navigation Flow........................................................................................................................................................33. Prototype....................................................................................................................................................................4

3.1 Prototype 1: Algorithm Implementation.......................................................................................................4

3.2 Prototype 2: Image Recognition..............................................................................................................6

3.3 Prototype 3: Model Retraining..............................................................................................................10

3.4 Prototype 4: Dataset...............................................................................................................................13

3.5 Prototype 5: UI........................................................................................................................................14

3.6 Prototype 6: Asynchronous Task..........................................................................................................17



Table of TablesTable 1: Risks and their Prototypes...............................................................................................................................1

Table 2: TensorFlow......................................................................................................................................................4

Table 3: Algorithms Comparison...................................................................................................................................5

Table 4: Image Recognition...........................................................................................................................................6

Table 5: Model Retraining...........................................................................................................................................10

Table 6: Dataset...........................................................................................................................................................13

Table 7：UI..................................................................................................................................................................15

Table 8：Asynchronous Task.......................................................................................................................................18



Table of FiguresFigure 1: Navigation Flow.............................................................................................................................................3

Figure 2: Dataset01.......................................................................................................................................................7

Figure 3: Dataset02.......................................................................................................................................................8

Figure 4: Retrain............................................................................................................................................................9

Figure 5: Tulips Sample.................................................................................................................................................9

Figure 6: Classify Result................................................................................................................................................9

Figure 7: Classify Result Details...................................................................................................................................9

Figure 8: Accuracy Small.............................................................................................................................................11

Figure 9: Accuracy Large............................................................................................................................................11

Figure 10: Distortion...................................................................................................................................................12

Figure 11: Dataset.......................................................................................................................................................14

Figure 12: Home Page.................................................................................................................................................16

Figure 13: Image Recognition......................................................................................................................................16

Figure 14: Model Retraining........................................................................................................................................17

Figure 15: Progress Bar..............................................................................................................................................19


1


1. Introduction1.1 Purpose of the prototype

This prototype report introduces top risks of our project and approaches to mitigate the risks. Prototyping is an efficient way for us to demonstrate our understanding about the project and deliver our concepts visually to the client. So that the client could have an idea about how the product will look like and give us useful feedback to improve it. In this document, risks about implementing machine learning algorithms, image recognition, model retraining, dataset, UI, and asynchronous task are described. Besides, it also introduces prototypes to mitigate these risks.

Table 1: Risks and their Prototypes

No. Risk Prototype1 Machine learning algorithms

implementationAlgorithm prototypeProviding a clear idea of how the algorithm will operate and advantages of the algorithm compared with others.

2 Model testing Testing prototypeProviding a clear idea about the time and accuracy of the algorithm on pre-trained model, our own dataset, and retrained model.

3 Train and retrain the model Training prototypeProviding a clear idea about the performance of the model when our training dataset is small.

4 Dataset Dataset prototypeA prototype that provides a feasible evidence that we can find enough image libraries to train our model.

5 UI UI prototypeProviding a clear image about the user interface to let the user know where they can start image recognition function and where they can retrain the model.

PRO_TRR_F16a_T04_V3.0.doc 1 Version Date:12/04/2016

2


6 Asynchronous task Asynchronous task prototypeA prototype that provides a feasible evidence that our system can deal with the model retraining task as an asynchronous task.

1.2 Status of the prototype1) Algorithm prototype revision2) Image recognition prototype modification.3) Risk avoidance of the API integration4) Model retraining prototype modification5) Dataset prototype modification6) UI prototype creation7) Asynchronous task prototype creation


3


2. Navigation Flow

Figure 1: Navigation Flow


4


3. Prototype3.1 Prototype 1: Algorithm Implementation

3.1.1 Purpose of this prototypeOne of core capabilities of our project is to recognize images and classify images into different classes by implementing machine learning algorithms. Once the user upload a set of images to our system, the system will preprocess the images, then output images with the class it belongs to.

Since there are many different machine learning algorithms that can accomplish image recognition, we have to ensure that the algorithm we select is easy to use with applicable performance. In addition, we have to make sure that the algorithm is not hard for all team members to learn and use.

3.1.2 ResultAfter prototyping, we found out that TensorFlow is applicable for our system compared with other machine learning algorithms.

At first, we found 4 different deep learning algorithms: TensorFlow, Deeplearning4J, Caffe, and Torch. After consulting our client, they suggested to use either TensorFlow or Deeplearning4J. By the comparison amount algorithms showed in Figure 2, Deeplearning4J does not provide pre-trained models, does not have visualization, and does not run fast enough, so we decided to use TensorFlow.

Table 1: TensorFlow

Description This screenshot is the algorithm prototype that compares TensorFlow with other machine learning algorithms.

Related Capability WC_4107: The pipeline shall use deep learning algorithm.


5


Table 1: Algorithms Comparison

Tensorflow deeplearning4J Caffe TorchCreator Google Brain Team Adam Gibson BVLC Ronan Collobert

Commercial License

Yes Yes Yes Yes

Platform Linux / OSX Linux / OSX / Windows

Linux / OSX / Windows

Linux / OSX / Windows

Interface C++ / Python Java C++ / Python Lua, LuaJITCUDA support Yes Yes Yes Yes

Pre-trained models (image)

Yes No Yes Yes

Visualization Yes No No NoRunning speed Fast Slow Fast FastWork for big

networkGood Good Cumbersome Cumbersome

Document easy to read

Yes Yes Yes No


6


3.2 Prototype 2: Image Recognition

3.2.1 Purpose of this prototypeTesting speed and accuracy of TensorFlow when dealing with pre-trained classes is great. In this scenario, classifying an image with a pre-trained class only takes 2 second with GTX970M GPU and the accuracy rate is 90%.

However, there is no feasible evidence to prove that this algorithm will work as efficient as it is when dealing with topics trained with our own dataset. We have to ensure that the performance of the algorithm is acceptable while testing images with new topics.

3.2.2 ResultAfter prototyping, we have a better understanding about the testing speed and accuracy of TensorFlow when dealing with topics trained with our own dataset.

We built our own dataset with almost 4000 flower images. These images can be separated into 5 different categories as Figure 2. Figure 3 shows the actual images. Then we retrain the model by using this dataset. As you can see the Figure 4, training accuracy is about 90%, which is really close to the performance of the pre-trained model. Also, the whole training process takes 15 minutes. In order to ensure that this model is capable of classifying flowers, we input an image of tulips as Figure 5 into our system. The model recognizes this flower as tulips with 92% possibility. The total time cost is 2 seconds. In that case, this risk of model testing is migrated. Of course, we will keep testing the model once we add a new topic and a new set of images to train our model. To make sure that the accuracy of the model is acceptable for all topics.

Table 1: Image Recognition

Description This screenshot is the testing prototype that verifies the running speed and accuracy of the algorithm with our own dataset.

Related Capability

WC_4101: The pipeline shall use detector model to detect the images uploaded by the users.

Pre-condition A user uploads a set of images.

Post condition The user can view images with their topics.


7


Figure 1: Dataset01

Figure 1: Dataset02


8


Figure 1: Retrain


9


Figure 1: Tulips Sample

Figure 1: Classify Result

Figure 1: Classify Result Details


1


3.3 Prototype 3: Model Retraining

3.3.1 Purpose of this prototypeIn order to classify topics other than pre-trained classes, we have to train or retrain the model to scale up to more topics. There is a risk that training and retraining processes may take a long period of time. As a result, the topic scope may be restricted to certain topics by the delivery time. Also, there is a risk that the testing speed and accuracy may be less than our expectation if the training or retraining dataset is small.

We have to make sure that the algorithm will work well on both small and large dataset, and the training process won’t take too much of time.

3.3.2 ResultAfter prototyping, we know that the number of training images is not the key element for testing speed, because the training time of small dataset with 500 images and large dataset with 4000 images are very close. Both of small and large datasets take around 15 minutes to complete the training process. In that case, we searched the principle of TensorFlow to train the model. It says that the steps of iteration will affect the running speed and accuracy primarily. About 10 images are randomly selected to train the model each iteration. We can adjust how many images for each iteration. And we can adjust how many iterations it takes for one training process. In this case, we selected 4000 iterations. These 2 factors contribute the total running time of the training.

Figure 8 shows the accuracy of testing 68 different images with small dataset of 500 images. The overall accuracy is 82%. Figure 9 shows the accuracy of testing 68 different image with the previous model that we trained with a dataset of 4000 flowers. As you can see, the overall accuracy is 98%. Even though the accuracy of small dataset is smaller than the large dataset, 80% accuracy is acceptable. Besides, there is another way—distortion—to increase the accuracy of the model with small dataset. In Figure 10, distortion increases the accuracy of a small dataset with poor performance into 80%.

Table 1: Model Retraining

DescriptionThis screenshot is the training prototype that figures out the difference of testing performance between model with a small dataset and model with a large dataset.

Related Capability

WC_4148: As a client I can re train the pipeline by giving a new topic and a new set of images

Pre-condition A user uploads a set of images, and add a new topic or select a topic.

Post condition The pipeline will be able to classify a new topic.


1


Figure 1: Accuracy Small

Figure 1: Accuracy Large


1


Figure 1: Distortion


1


3.4 Prototype 4: Dataset

3.4.1 Purpose of this prototypeDataset determines how many topics our pipeline can classify. There is a risk that we can’t find enough images to train our model. In that case, our pipeline can only classify a few topics. We have to make sure that there will be enough image libraries to train the model. Besides, our client is currently working on public safety area, there is a risk that we can’t find enough large dataset with related topics.

3.4.2 ResultWe have to keep looking for different image libraries. Also, we could ask the client for advice. In Figure 11, the listed topics are image libraries that we currently found. With them, we can train the model to classify the most of animals and plants. In order to make our model be able to classify topics with public safety area, we are currently negotiating with our client to see if they can provide us a useful dataset.

Table 1: Dataset

Description This screenshot is the current dataset we have.

Related Capability


Pre-condition A trainer uploads a set of images, and add a new topic or select a topic.

Post condition The dataset increases.


1


Figure 1: Dataset

3.5 Prototype 5: UI

3.5.1 Purpose of this prototypeIn order to give users an easy-to-use product, a user friendly interface could help a lot. But there is a risk that even though the UI is friendly, poor integration between the front-end and the TensorFlow may cause a bad user experience. If there is an error happened when connecting the front-end and the TensorFlow, the whole program will crash. If there is a function that costs a long period of time, the user experience will be bad.

We have to test all connections between the front-end and the Tensorflow carefully to make sure that there is no error. And we need to run and test all functions to find out a proper boundary. So that each function won’t cost too much of time. In that case, user experience will be friendly.


1


3.5.2 ResultAfter prototyping, we discovered and repaired many errors that could cause TensorFlow to crash. Since TensorFlow only deal with JPEG and JPG images, we limited the image formats that the system accepts to them. If there is an image other than these two formats, it will be filtered out automatically. As a result, the risk of system crashing with invalid input format is migrated.

TensorFlow takes 2-4 seconds to recognize an image, and if the size of the image is large, time cost is higher. In order to reduce the risk of bad user experience, we limited the accepting image size to 20M and accepting upper bound to 5 for image recognition part. Therefore, the user don’t have wait too long to get result of image recognition. Besides, for the model retraining part, if the uploaded image number is too small, the result may not be accurate. So we limited the lower bound to 30 images. If the uploaded image number is too large, the uploading process may take too long, which could cause a bad user experience. So we limited the upper bound to 1000. Also, the number of iteration steps affects running time and accuracy. If the number is too small, the accuracy is bad. And if the number is too large, the running time is high. So we made several sections for the user to choose. For instance, 1 iterations: fastest but lowest accuracy; 300 iterations: fast but low accuracy; 600 iterations: slow but high accuracy; 1200 iterations: slowest but highest accuracy.

Table 2：UI

Description This screenshot shows the current user interface we have.

Related Capability

WC_4109: As a user, I can upload multiple images(each with a label to choose) at the same time

Pre-condition Celery and server are running and a user open the home page.

Post condition Home page of our system shows up.


1


Figure 1: Home Page

Figure 1: Image Recognition


1


Figure 1: Model Retraining

3.6 Prototype 6: Asynchronous Task

3.6.1 Purpose of this prototypeIn order to run model retraining as an asynchronous task, we need to use external Celery to process TensorFlow work. But there is a risk that TensorFlow job cannot be handled by Celery. Besides, there is also a risk that the front-end cannot get and show real-time status from an asynchronous TensorFlow task.

We have to make sure that Celery in Django framework is able to handle Tensorflow tasks. To make sure that there is no error to run TensorFlow task as an asynchronous job. And to make sure that the front-end can show the real-time status of the TensorFlow task correctly.


1


3.6.2 ResultAfter prototyping, we discovered that Celery in Django framework can handle TensorFlow tasks. But there is a memory leak problem. Thus, we have to manually clean memory to make sure that Celery has enough space to run additional tasks. Besides, by prototyping, we know that storing the status of the TensorFlow tasks into database and let the font-end read database to get the status in real-time is a feasible way of forming progress bar. Since we only allow one task at a time, there is no read-and-write conflict.

Table 3：Asynchronous Task

Description This screenshot shows the progress bar of an asynchronous Task

Related Capability


Pre-condition Celery and server are running and a user upload images to run model retraining function.

Post condition Progress bar wage shows up and task is running in the back end.


1


Figure 1: Progress Bar


prototype report - software engineering ii - spring 2018 · web viewintroduction 1.1 purpose of the...

Documents