Skip to content

Project Documentation

Jump to bottom
James Abbott edited this page Apr 23, 2019 · 16 revisions

MissingMaps - Software Documentation

This page details setting up the project, and the documentation required to continue working on it.

Introduction and Advantages

The IBM Missing Maps project aims to use IBM Cloud & Watson APIs to create a web applicationthat takes aerial photographs as input, and then identiies signs of habitation. To achieve this, we chose to usethree tools: IBM Cloud for image recognition, Vue.js for front-end demonstration and JavaFX for an auxiliary tool that helps when training the classifier on IBM Cloud. The project is stable, accurate and easy to extend for further research which may be helpful to official Missing Map’s work.

We were required to use IBM cloud as part of the project, however there are several good reasons for us to use these tools regardless:

  • IBM Cloud

    • Quickly and accurately tag, classify and train visual content using machine learning.

    • For research use, we can create a Lite Plan (no charge) instance of the Visual Recognition service.

    • Put the training work on the cloud so that we don't require extensive local storage or processing power.

  • Vue

    • Vue takes concepts from Angular and React and combines them into a lightweight framework.

    • Vue offers a lot of things that make programming with JavaScript easier, such as the code being easy to understand, flexible, and having components that are very powerful features.

    • Some of our group members have prior experience with it.

  • JavaFX

    • The manual classifier is only used by developers, so we tried to minimise our workload and choose the tool that we are most familiar with.

    • Java Virtual Machine brings incredible compatibility across platforms so our group members can use it in their own PC.

Deployment

To make our app work, there are 2 things required: an IBM account with a visual recognition service and the front-end services.

Getting and configuring IBM account

Create your IBM account

If you already have had one, please skip to next section.

  1. Go to https://console.bluemix.net/ and choose to create a free account.

  2. Fill in the required information.

  3. Check your email to complete your registration.

Create a Visual Recognition service

  1. Log in to BlueMix console https://console.bluemix.net/.

  2. Accept the Account Privacy Terms upon first sign in.

  3. Click the catalog button catalogue on the toolbar on top.

  4. Search for visual recognition.

  5. Choose the result and create the service with default settings (Lite Plan).

  6. In the page of the Visual Recognition service you just created, click the "Create a Custom Model" button on “Manage” tag.

  7. Choose to Create Model.

  8. Create a new project, providing a name and description.

  9. Train a classifier (more details can be found in the Training Classifier part of this document). Click "Train Model" and wait for training.

  10. From this section you will recieve two things that are needed in the following document:

    1. Your Visual Recognition service API key and URL (can be found in the "manage" tag of your service page)

    2. The model ID of your trained classifier (can be found after clicking "Click here to view and test your model")

Deploying the front-end

Although we aim to make this document understandable with as little prior knowledge as possible, some background reading about Vue, Node.js, Linux web server may be helpful when working through the following instructions.

Esentially, the front-end is written in Vue and can be fully built into static web pages. There are several ways to run it, which are discussed below. However, before deploying in any way, some configuration is necessary.

Prepare the code

  1. Ensure that you have Node.js and npm installed on your environment.

If upon running the following commands version numbers are returned, then they are installed:

node -v npm -v

See the official documentation for more details.

  1. Clone our git repository by running the following command and move to the the web app folder:
git clone https://github.com/Carbsta/NottsMissingMaps.git cd ./NottsMissingMaps/front-end/nott-missing-map/

  1. Now you get access to all of our files that relate to the front-end Vue application.

  2. Configure the code: find the file src/config.json and change the backEnd to your own back-end server URL (This will be available after making a simple back-end).

Run the app locally

See our README in the repository.

  1. Set up the project:
npm install
  1. Compiles and hot-reloads for development in order to serve the app locally:
npm run serve
  1. Now you should see something similar to the text below in your CLI, Meaning you can view the app in your browser:
DONE Compiled successfully in 9201ms 16:18:07

App running at:

Note that the development build is not optimized. To create a production build, run npm run build.

Build the app locally

See our README in the repository.

  1. Set up the project:
npm install
  1. Compiles and minifies for production (generated files found in directory "dist")
npm run build
  1. Now you should see something similar to the text below in your CLI. The built files are ready in "dist".
DONE Build complete. The dist directory is ready to be deployed. INFO Check out deployment instructions at https://cli.vuejs.org/ guide/deployment.html

Host it on IBM Cloud

All we need is a server with a public folder to put things in dist (Build the app locally). In our case, we're using IBM’s SDK for Node.js™.

  1. Create an "SDK for Node.js™" service (or you can just create an Node-RED app according to the first part of this tutorial, which will contain an “SDK for Node.js™” service).

  2. Enable Continuous Delivery so we can easily control the content.

  3. Use the default setting in "Git Repos and Issue Tracking" and “Eclipse Orion Web IDE” and create API Key in “Delivery Pipeline”. Then create the toolchain.

  4. Access the Git Repo just created.

  5. Replace files in "public" with files in “dist”. You may create multiple commits for every single file or clone to local and make a bigger commit.

  6. Wait for several minutes to let it deploy automatically. You can check the progress in "Delivery Pipeline"

  7. After the stage passes, you can find the app on the internet. In our case, the URL is https://missingmaptestnodeapp.eu-gb.mybluemix.net/. You should be able to get something similar.

Make a simple back-end

We used a simple Node-Red back-end to provide IBM service API key as well as other querying arguments for the front-end. The steps of building such a simple back-end are shown below:

  1. Create you Node-RED server on IBM according to the first part of this tutorial.

  2. Create 2 http input tags. One for get and one for post so we can get feedback with bothe method.

node-red workflow

  1. Link both of them with a function node. Then set msg.payload to whatever http response data you want. In our project, it should has the structure shown below. Don't forget to return the message at last. You should have required information if you completed previous steps.

linking nodes

  1. Finish all of above by linking them to a http response node.

linking nodes

  1. Deploy your nodes and you can now access "<your_node_red_web_url>/api" to get all of these data you just set.

deploying nodes

Training Classifier

Preparing Data

When looking for a dataset that can be used to train an AI, you aren't going to find the perfect dataset to suit your needs unless you create your own. Although, you will find many useful ones. Looking online for datasets seems like the easiest option, as you are using someone else's time and effort. But you are going to want to find multiple sets to make sure that not only you cover all basis, but to make sure that the AI doesn’t become the basis. Through my research, I found some that were pre-classified and some that were just a folder of images. Pre-classified images were really useful as they were segmented as well, so it was just the area you would want and there was little work needed from me. Trying to find a dataset with a lower spatial resolution will make the images better quality e.g a 0.30m spatial resolution means that each pixel is equivalent to 30 cm on the ground.

Creating your own dataset make sense as you get to choose what is part of the set. So this allows you to get the images that will make the perfect dataset. However, this is not the case, unless you spend the hundreds of man-hours individually screenshotting the areas of google earth you want, as you are going to want a few thousand to just train the AI let alone validate it. There are programs out there which will do the hard work for you, but these are not perfect. The main problem is that they capture random part from all over the globe, meaning that as the world is 70% water 7/10 images are going to be useless(after the first few oceans to train the AI on). Also, once you have the satellite captures, you will then have to manually sort each image into potentially different categories.

The file format of the data set will also matter. For this project, as we used IBM's Watson, it needed to be in the png or jpg image format. Most of the datasets I came across were either tif or binary. Tif is useful as it is a lossless file format so it will have the best quality. But it will need to be converted before it can be run through a classifier. The other format was binary, so it was a file with an array of 1’s and 0’s. This was done so that programs like Matlab could rebuild them into images and in doing so it saves a lot of space. But unfortunately, Watson cannot handle data in that format, so I had to find datasets that were in an image format.

Trying to find a data set isn’t too difficult, but when I started out looking for one our original intention was to find one broken down into inhabited and uninhabited images. But I quickly found out that this was too much to as for. This is because people would pay to have specialised datasets. This was one of the problems I occurred, many potential datasets were behind paywalls, mainly the ones on AWS servers. There were a few websites like Open Street Maps that will allow you to download the map data. But obviously, it is not in image format. However, I found that on kaggle.com companies offer cash rewards to people who can train an AI to a certain accuracy using their datasets. This meant that they were free to use and download.

We used aerial images data from various sources, including:

Winsdaton was pre segmented but unclassified. It contains mainly forest areas, Greenland and a few fields.

UCMerced_LandUse was immensely helpful to us. It has 21 different categories with 100 images in each that are pre-segmented. These included rivers, roads, industrial buildings, residential etc.

Patternnet was also really helpful as it had lots of categories but had more images in each classification.

We also considered these sources, but they weren't used to train the classifier:

  • Some static map images dataset. We try some static map images, but these images are much specific and much similar to each other. They are not suitable for training the classifier.

Some of the reason we wouldn’t use a data set is that it might be from a different angle. All the previous datasets use were from a clear top-down view. With the images being from a different angle it did mean that they were closer to the ground so if it was more aerial photography rather than satellite imagery.

Another one was cloud coverage. If we had used images that contained cloud coverage to training the AI it could completely change the outcome. The user is unlikely to feed an image with clouds in.

Also, some data sets that weren’t pre segmented we not used as they would have required a lot more work on our end and with the chance of miss matched results with other data sets, leading to a confliction when training.

Processing With a Manual Classifier

To mark every slice of our source images data, we built a cheap Google Captcha style classifier. Here are the instructions for it. You can also find similar information in our README for this part in GitHub

To build and run

We are using:

  • IntelliJ IDEA (2018.3.5 Ultimate Edition)

  • Java 10

However, only source files are kept in this repository. In this way, all IDE should be able to run these code as long as using Java 10.

Here we show how to make it work in IntelliJ:

  • Open project in IntelliJ welcome window

  • Choose the "manual-classifier" directory

  • File in the top toolbar

  • Choose Project Structure

  • Set SDK and language level to Java10

  • Create an output target

  • Mark "src" as Sources

  • Mark "out" as Excluded

  • Click OK

  • Run

  • Enjoy :)

  • Personally, the writer of this doc met a problem where IntelliJ shows syntax errors while the program could still run. That problem was solved by setting the "language level" to 9 and then setting it back.

import a projectset up the project structureconfigure these settingschoose a project compiler outputadd a content root

run itmain should be in the run list

You should see something like this

Use the classifier

  1. We recommend using a window size of more than 720p. Otherwise, you may meet unexpected problems

  2. Choose the source folder, which contains pictures to classify. Currently, only png files are tested to be supported, but in theory, all image format supported by Java should be fine.

  3. Choose the target folder. You can set it to anywhere, even the same as source folder.

  4. Configure the "X slices" and "Y slices". For example, if X is set to 4 and Y is set to 6, your images will be cut into 4 * 6 slices.

  5. Flatten source functionality is not currently implemented. Noting will happens if you change it. The program will just read image files in the source folder, ignoring all other things including subfolders, if there is any.

  6. Toggle the "Put result from different images in different dir" button if you need.

  7. if you just want to segment images without tagging:

    1. Click the "Just Segment" button to perform the task. The program will automatically exit after finished.

  8. Else:

    1. Click "start" / "start with all hab" to start classifying.

    2. When classifying, all patches are originally set to NON_HABITABLE (or HABITABLE if you clicked "start with all hab"), which are marked with BLUE border. You can click them to change them to GREEN, which stands for HABITABLE. Also, you can click the "set all hab" and "set all non-hab" to set all of the tails in the current image to be habitable/non-habitable.

    3. After finished, click the "End Classification" button and the program will generate the output and then exit.

  9. The result will be put in "output" dir below the source folder specified. If there is a dir called "output", the result will be put into "output_1" or "output_2" and so on.

Training the Classifier

Images were selected from large datasets of satellite images. If using a binary classifier, which simply determines if an image is inhabited or uninhabited, the manual captcha style tool can be used to segment images into these two categories, to be fed into the classifier. However, when using a more complex classifier that tags different categories of land (e.g. desert, forest, etc), pre-sorted datasets should be used.

There are two platform on which the classifier can be trained: Watson Studio and Node-RED:

Watson Studio

Creating a classifier with Watson Studio

Advantages and Disadvantages of Training with Watson Studio

The benefits of Watson Studio are that the user can work with a simply drag-and-drop interface and view the classifier graphically. Unfortunately, the images must be uploaded from the user’s computer which can be inconvenient considering the file sizes of large datasets.

How to train with Watson Studio

  1. Upload zip files of different land categories to a new project.

  2. Create classes to represent each category.

  3. Drag the corresponding zip files to each category. Unless creating a binary classifier, ignore the negative class.

  4. Wait for all of the images to load and press ‘Train’.

  5. Once the classifier has been trained it can be tested in Watson Studio or through the API.

Node-RED

The Node-RED alternative

Advantages and Disadvantages of Training with Node-RED

The benefit of using Node-RED to train data is that the datasets can be retrieved directly from DropBox or IBM Cloud Object Storage, meaning the files do not need to be stored locally. However, the GUI is less user-friendly than Watson Studio so it is easier to make mistakes.

How to train with Node-RED

  1. Store zip files of categories in DropBox or IBM Cloud Object Storage.

  2. Create a new classifier by arranging nodes as shown above, providing the visual recognition node with the relevant zip-file names.

  3. Inject the node to start training.

  4. Inject into a visual recognition list node to check when the classifier has finished training (status will be ‘ready’).

  5. Once the classifier has been trained it can be tested on Node-RED or through the Watson API.

Scripts

Detailed comments and descriptions about our code.

Coding Conventions

To help future developers, who want to take our project forward, get a deeper understanding of our work, we’d like to introduce some code conventions we used here:

  • For JavaScript, generally, we are following the style guide from Airbnb, which is ensured by the ESLint configure.

  • For Java:

    • Variable names are camelCased, with a lowercase letter to start, i.e. "targetPosition".

    • Classes are PascalCased,

    • Method names are camelCased, with a lowercase letter to start.

    • Scripts are indented with 4 spaces (soft-indent).

    • Member variables should be either private or protected wherever possible.

    • Follow the suggestion given by IntelliJ (default setting).

Code Comments (by Scripts)

Node-Red Related code

Base directory: /nodeRed

Overview

This subdirectory is cloned from node-red-bluemix-starter can be deployed to IBM cloud. Some configure files are listed below. For those files not mentioned, they should probably not be changed and there is no reason to do so.

To deployed it to IBM cloud’s SDK for Node.js™ service, it should have been put in the repository’s root. We put these files in nodeRed subdirectory for clearness. In this way, there are some extra things required to make it work:

  1. Visit the Toolchain’s Delivery Pipeline, configure the build stage.

Setting up continuous delivery

  1. In Jobs tag, select the Builder type to Shell Script and add the following build script:
#!/bin/bash # your script here

mv ./nodeRed/* . rm -r ./nodeRed

For the front-end code

cd ./front-end/nott-missing-map npm install && npm run build cd ../.. cp -a ./front-end/nott-missing- map/dist/* ./public/ rm -r ./front-end

show the output

ls ${ARCHIVE_DIR}

manifest.yml

  • Purpose:

Some sittings about IBM cloud services

  • Some Notes:

  • memory: the run time memory of the Node.js instance, 512M should be enough

  • service: the name of Cloudent service that gets created

bluemix-settings.js

  • Purpose:

The setting of Node-Red app, playing the same role as the settings.js file in normal Node-Red app. Here is the link for Node-RED configuration.

  • Some Notes:

    • Function Nodes - a collection of objects to attach to the global function context. For example,
functionGlobalContext: { osModule:require('os') }

can be accessed in a function node as:

var myos = global.get('osModule'); 
* Cross-origin resource sharing for the nodes that provide HTTP endpoints. Written:
httpNodeCors: { origin: "*", methods: ['GET','PUT','POST','DELETE'] } 
* The name of Cloudent service can be manually changed: (near the end of file)
var storageServiceName = ...

package.json

  • Purpose:

Add the packaged used in the Node-RED app.

  • Some Notes:

Most of these packages are kept there for historical reasons -- they are mostly not used now! Their details can be found on their npm page. You can freely add new packages if needed.

Packages currently added:

node-red-contrib-file-buffer: Node-RED node which can change the file to buffer;

node-red-contrib-http-multipart: Node-RED node which can accept HTTP multipart request;

node-red-contrib-pythonshell: Node-RED node which can run Python script (but we haven't found a way to use pip, so it is not so useful);

node-red-node-dropbox: Enable Dropbox node, which was used for classifier training;

node-red-contrib-cos: Add Node-RED node of IBM Cloud Object Storage;

image-to-slices: Slice images to patches. Need to be involved in the setting file.

canvas: The canvas package used in image-to-slices.Need to be involved in the setting file.

datauri: Generate Data URI from local files. Need to be involved in the setting file.

image-size: Get image file dimensions. Need to be involved in the setting file.

data-uri-to-buffer: Convert Data URI to buffer. Need to be involved in the setting file.

Also, see this link for the reason why we need to include there node-red-contrib-xxx packages rather than directory manager them with Node-RED Palette Manager.

Front-end related code

Base directory: /front-end/nott-missing-map

Overview

Our Vue app was created by Vue CLI, according to these instructions. In the process of continuous delivering (deploy to IBM cloud), npm install && npm run build will be run and the built file will be ready in ./dist. To develop and test it locally, make sure npm is available (mine npm is "5.5.1") and run npm install && npm run serve. Check comments for each file in the following, while some auto-generated files which are never changed may not be listed.

vue.config.js

  • About:

This config file is created by Vue CLI. You can find full documentation of it here. We added an entry called configureWebpack to add aliases for paths when resolving modules (importing). You can find documentation of webpack about this feature here.

package.json

  • About:

A common file you can find in Node.js app. Here is documentation from npm about what you can set in package.json. There is another file called package-lock.json which is auto-generated and officially suggested to keep in our repository. Some important entries are list below:

* scripts: commands available for npm run xxx. We added && rm dist/js/*.map to remove the unnecessary .map files after building.

* dependencies and devDependencies: You can add dependencies with running command npm i <package-name> at the root directory. And -D for devDependencies.

* eslintConfig: the ESLint configuration. See the official document [here](https://eslint.org/docs/user-guide/configuring#using-configuration-files-1). We add the [Airbnb code conventions](https://github.com/airbnb/javascript) constraints as well as something working with webpack module alias resolution feature (match with things in [vue.config.js](#heading=h.vqar2quz8a26))

public/index.html

  • About:

This is another file generated by Vue CLI. It is the entrance of the whole app. The only change we have made is adding the line linking to google fonts and material icons.

public/favicon.ico

  • About:

This is the icon of the website, which will be shown on the top left corner of the page tab.

src/main.js

  • About:

This is the entrance of the whole web’s Javascript (including Vue) code, which imports our App.vue. We register Vuetify here with the help of vuetify-loader to reduce our building output file size. It does works (reduced one-third of the size). We also import AsyncCompute which is used in src/components/ReportCard.vue.

src/config.json

  • About:

This file is the configuration of our app which created by our own.

  • Implementation Notes:

    • Currently,5 things are recorded:

      • The URL of our back-end server which can provide our API key together with other API querying arguments

      • The number of segments the original input image will be slices into.

      • The opacity of the mask when confidence level is 1 (100%).

      • The estimated ratio of consuming time of each work, which is used to calculate progress shown by the progress circular.

      • The classifier situation which is used in data explaining and display.

    • Problems: It is a bad idea to just give our private API key by our back-end server. We used to send the image to a Node-RED based back-end server and let it communicate with the Image Recognition service for us. However, due to the limitation of IBM cloud service, the back-end service worked significantly slowly and crashes frequently. So we finally have to choose to put everything in front-end. Although we have avoided out our API key directly in our public GitHub repo by putting in our simple back-end server, this solution is still not good.

src/App.vue

  • About:

This is the main part of our app. Every .vue file is a single file Vue component, while this one is the component for our whole web app. A.vue file consists of three parts: HTML style template, Javascript code as well as components scoped (if specified) style sheet.

  • Implementation Notes:

    • The HTML template part:

    • The Javascript part:

      • Data fields used:

        • uploadingPage: a binary value indicating the current page showing -- true for uploading page and false for the report page

        • imgs: an array of data used in preview cards and report cards.

        • alert: if there is an alert message

        • alertMsg: the alert message content

        • uploading: if the app is querying the classifier (trying to change from the uploading page to the report page)

        • zipping: if the app is zipping the result images

        • previewImg: control the information for ImgPreview used in the report page

        • progress: the progress of generating the result. progress.data is the actual progress. progress.max is the maximum value can be achieved, which is set when starting generating the result.

      • There is an customized option called sliceNum which have x and y property representing the number of slices in the corresponding axes. See why we did that.

      • The methods should all be with comments in the code

    • The style sheet part:

      • Those classes with "xs" in class name are used in extra small conditions.

      • We set the progress circular’s transition to none as we only have a flash to update the screen when doing image processing (slicing). There are more details in src/functions/upload.js.

src/components/DragDropBox.vue

This is a .vue file of our project. You may find some general descriptions about .vue fiel in the src/App.vue part of this document.

  • About:

This is a drag-drop box component of our app used for image uploading. References used when we built this component: 1, 2.

  • Implementation Notes:

    • The component takes two props: files and alert.

      • files: the array used to store all images files. It will be modified when the component getting new files

      • alert: a call back function passed in which will be called with error messages as it’s first arguments when the drag-drop box component want to raise an error message.

    • The methods should all be with comments in the code

src/components/ImgPreview.vue

This is a .vue file of our project. You may find some general descriptions about .vue fiel in the src/App.vue part of this document.

  • About:

This file is for the image preview component on the report page. It is based on the image-comparison package. Actually, it just renders the image comparison part, while the outer popup window is rendered and controlled by the VDialog in App.vue.

  • Implementation Notes:

Its prop reportCard takes an Vue component instance of ReportCard.

Most of the things are done in the mounted lifecycle hook. Briefly speaking, a temporary img DOM element is created and draw to the canvas on the screen, following by the instantiation of ImageComparison.

src/components/PreviewCard.vue

This is a .vue file of our project. You may find some general descriptions about .vue fiel in the src/App.vue part of this document.

  • About: The preview cards appear on uploading page, showing the pictures in memory which are ready to upload and classify.

  • Implementation Notes:

    • A prop called uploading will be passed in so that the card can disable the dismiss button when uploading.

    • The whole image array is passed in so that it can delete itself from that array.

src/components/ReportCard.vue

This is a .vue file of our project. You may find some general descriptions about .vue fiel in the src/App.vue part of this document.

  • About: The report cards appear on the result/report page, showing the pictures in comparison with them with masks of confidence levels. In addition, it has a expand button to show the detailed report of the image.

  • Implementation Notes:

    • It has the same props as that of Preview Card together with an additional one called previewImg, which is just the one with the same name in App.vue.

    • There is a method called reportInfo which specifies how the result data from classifier will be interpreted.

    • The asyncComputed property resultBlob gives the Blob data of the current image with confidence level mask. It will be used for the app to perform bulk downloading.

src/functions/upload.js

  • About: The uploading function querying the image classifier.

  • Implementation Notes:

    • It does two things in order with the help of Promise API.

  1. Use the jimp to slice the image to pieces. As this take is CPU bound and browsers freeze the when doing this, we used a weird way(the solution we use is at the end of this linked page) to walk around. The tradeoff is that we have to disable the fantastic transition of our progress circular, which is mentioned in app.vue doc.

  2. Then we used jszip to generate zip files of chunks of images (size 20 max) to reduce future API call 20 times.

  3. Then we query our back-end server to get IBM’s image recognition API and then query IBM service with that, which is a bad practice but we just have to do this. This is mentioned in config.json doc.

    • The function exported by this file will return an array of slightly processed (.classifiers[0]) results from IBM image recognition service asynchronously. (using Promise API)

src/functions/drawCanvas.js

  • About: a function which draws an image with a mask of given confidence to given canvas

  • Implementation Notes:

    • Its params:

      • canvas: the canvas DOM element to draw on

      • img: the img DOM element which is the base of drawing

      • sliceNum: define how many slices will be divides to in x / y axis

      • confidence: a function return confidence between 0 to 1

    • Canvas drawing reference

Manual-classifier related code

Base directory: /manual-classifier/src/grp25/captcha

Overview

It is a simple JavaFX application for preprocessing image databases before them being passed to the IBM classifier for training. Find more details about this app in Processing With a Manual Classifier part of this document.

Descriptions of each single files are provided by Javadoc.

(Last Modification: April 22)

Clone this wiki locally