Skip to content

Project Documentation

Jump to bottom
James Abbott edited this page Apr 24, 2019 · 16 revisions

MissingMaps - Software Documentation

This page details setting up the project, and the documentation required to continue working on it.

Introduction and Advantages

The IBM Missing Maps project aims to use IBM Cloud & Watson APIs to create a web applicationthat takes aerial photographs as input, and then identiies signs of habitation. To achieve this, we chose to usethree tools: IBM Cloud for image recognition, Vue.js for front-end demonstration and JavaFX for an auxiliary tool that helps when training the classifier on IBM Cloud. The project is stable, accurate and easy to extend for further research which may be helpful to official Missing Map’s work.

We were required to use IBM cloud as part of the project, however there are several good reasons for us to use these tools regardless:

  • IBM Cloud

    • Quickly and accurately tag, classify and train visual content using machine learning.

    • For research use, we can create a Lite Plan (no charge) instance of the Visual Recognition service.

    • Put the training work on the cloud so that we don't require extensive local storage or processing power.

  • Vue

    • Vue takes concepts from Angular and React and combines them into a lightweight framework.

    • Vue offers a lot of things that make programming with JavaScript easier, such as the code being easy to understand, flexible, and having components that are very powerful features.

    • Some of our group members have prior experience with it.

  • JavaFX

    • The manual classifier is only used by developers, so we tried to minimise our workload and choose the tool that we are most familiar with.

    • Java Virtual Machine brings incredible compatibility across platforms so our group members can use it in their own PC.

Deployment

To make our app work, there are 2 things required: an IBM account with a visual recognition service and the front-end services.

Getting and configuring IBM account

Create your IBM account

If you already have had one, please skip to next section.

  1. Go to https://console.bluemix.net/ and choose to create a free account.

  2. Fill in the required information.

  3. Check your email to complete your registration.

Create a Visual Recognition service

  1. Log in to BlueMix console https://console.bluemix.net/.

  2. Accept the Account Privacy Terms upon first sign in.

  3. Click the catalog button catalogue on the toolbar on top.

  4. Search for visual recognition.

  5. Choose the result and create the service with default settings (Lite Plan).

  6. In the page of the Visual Recognition service you just created, click the "Create a Custom Model" button on “Manage” tag.

  7. Choose to Create Model.

  8. Create a new project, providing a name and description.

  9. Train a classifier (more details can be found in the Training Classifier part of this document). Click "Train Model" and wait for training.

  10. From this section you will recieve two things that are needed in the following document:

    1. Your Visual Recognition service API key and URL (can be found in the "manage" tag of your service page)

    2. The model ID of your trained classifier (can be found after clicking "Click here to view and test your model")

Deploying the front-end

Although we aim to make this document understandable with as little prior knowledge as possible, some background reading about Vue, Node.js, Linux web server may be helpful when working through the following instructions.

Esentially, the front-end is written in Vue and can be fully built into static web pages. There are several ways to run it, which are discussed below. However, before deploying in any way, some configuration is necessary.

Prepare the code

  1. Ensure that you have Node.js and npm installed on your environment.

If upon running the following commands version numbers are returned, then they are installed:

node -v npm -v

See the official documentation for more details.

  1. Clone our git repository by running the following command and move to the the web app folder:
git clone https://github.com/Carbsta/NottsMissingMaps.git cd ./NottsMissingMaps/front-end/nott-missing-map/

  1. Now you get access to all of our files that relate to the front-end Vue application.

  2. Configure the code: find the file src/config.json and change the backEnd to your own back-end server URL (This will be available after making a simple back-end).

Run the app locally

See our README in the repository.

  1. Set up the project:
npm install
  1. Compiles and hot-reloads for development in order to serve the app locally:
npm run serve
  1. Now you should see something similar to the text below in your CLI, Meaning you can view the app in your browser:
DONE Compiled successfully in 9201ms 16:18:07

App running at:

Note that the development build is not optimised. To create a production build, run npm run build.

Build the app locally

See our README in the repository.

  1. Set up the project:
npm install
  1. Compiles and minifies for production (generated files found in directory "dist")
npm run build
  1. Now you should see something similar to the text below in your CLI. The built files are ready in "dist".
DONE Build complete. The dist directory is ready to be deployed. INFO Check out deployment instructions at https://cli.vuejs.org/ guide/deployment.html

Host it on IBM Cloud

All we need is a server with a public folder to put things in dist (Build the app locally). In our case, we're using IBM’s SDK for Node.js™.

  1. Create an "SDK for Node.js™" service (or you can just create an Node-RED app according to the first part of this tutorial, which will contain an “SDK for Node.js™” service).

  2. Enable Continuous Delivery so we can easily control the content.

  3. Use the default setting in "Git Repos and Issue Tracking" and “Eclipse Orion Web IDE” and create API Key in “Delivery Pipeline”. Then create the toolchain.

  4. Access the Git Repo just created.

  5. Replace files in "public" with files in “dist”. You may create multiple commits for every single file or clone to local and make a bigger commit.

  6. Wait for several minutes to let it deploy automatically. You can check the progress in "Delivery Pipeline"

  7. After the stage passes, you can find the app on the internet. In our case, the URL is https://missingmaptestnodeapp.eu-gb.mybluemix.net/. You should be able to get something similar.

Make a simple back-end

We used a simple Node-Red back-end to provide the IBM service API key to the website, as well as other querying arguments for the front-end. The steps of building this back-end are shown below:

  1. Create you Node-RED server on IBM according to the first part of this tutorial.

  2. Add 2 http input tags. One for get and one for post so we can obtain feedback with both methods.

node-red workflow

  1. Link both of them with a function node. Then set msg.payload to whatever http response data you want. In our project, it should has the structure shown below. Don't forget to return the message at the end. The required information can be obtained from the previous steps.

linking nodes

  1. Finish all of the above by linking the function node to a new http response node.

linking nodes

  1. Deploy your nodes and you can now access "<your_node_red_web_url>/api" to view all of the data you just set.

deploying nodes

Training Classifier

Preparing Data

When looking for a dataset that can be used to train an AI, it is difficult to find the perfect dataset to suit your needs unless you create your own, although there are still many useful ones available. Looking online for datasets seems like the easiest option, as many organisations have already done the work. Ideally, search for multiple sets to make sure that you cover all basis. Whilst some datasets come pre-classified, others are just a folder of images. Pre-classified datasets were very useful as they were segmented already so they focused on the specific area we wanted. Finding a dataset with a lower spatial resolution will make the images better quality e.g a 0.30m spatial resolution means that each pixel is equivalent to 30 cm on the ground.

Creating your own dataset is also a good idea as you get to curate the data exactly to your needs. However, his requires you to spend many hours individually screenshotting sutible areas of Google Eath, as a minimum of 2000 images are required to train the AI, let alone validate it. There are programs out there which will take random satellite screenshots for you, but these are also not ideal. The main problem is that they capture random images from all over the globe, meaning that as the world is 70% water 7/10 images are going to be useless. Also, you will then have to manually sort each image into different categories.

The file format of the data set will also matter. For this project, as we used IBM Watson's visual recognition service, training images must be png or jpg image formats. Many datasets are stored as Tif or binary. Tif is useful as it is a lossless file format so it will have the best quality, but must be converted before it can be run through a classifier. The other format was binary, so it was a file with an array of 1s and 0s. This was done so that programs like Matlab could rebuild them into image. Again, Watson cannot handle this format.

Trying to find a data set is not too difficult, but upon starting out search, our original intention was to find one broken down into inhabited and uninhabited images. Unfortunately, we did not manage to find such a dataset, likely because because people will pay to have these specialised datasets made. Many potential datasets were behind paywalls, particularly the ones on AWS servers. There were a few websites like Open Street Maps that will allow you to download the map data but this is not in an image format. However, I found that on kaggle.com companies offer cash rewards to people who can train an AI to a certain accuracy using their datasets. This meant that they were free to use and download.

We used aerial images data from various sources, including:

Winsdaton was pre segmented but unclassified. It contains mainly forest areas, Greenland and a few fields.

UCMerced_LandUse was immensely helpful to us. It has 21 different categories with 100 images in each that are pre-segmented. These included rivers, roads, industrial buildings, residential etc.

Patternnet was also really helpful as it had lots of categories but had more images in each classification.

We also considered these sources, but they weren't used to train the classifier:

  • Some static map image datasets are highly specific and contain images that are incredibly similar to each other, meaning they are not suitable for training the classifier.

  • Some datasets contained images that were not from a top-down angle, meaning they were more like aerial photography than satellite imagery.

  • Some datasets contained images with cloud coverage. If we had used images these images to train the classifier it could completely skew the outcome. The user is unlikely to input an image with clouds in.

  • Some data sets were not pre-segmented so would have required a lot more work on our end. There was also the chance of miss matched results with other data sets, leading to conflicts when training.

Processing With a Manual Classifier

To mark every slice of our source images data, we built a simple Google Captcha style classifier. Here are the instructions for it. You can also find similar information in our README.

To build and run

We are using:

  • IntelliJ IDEA (2018.3.5 Ultimate Edition)

  • Java 10

Only source files are kept in this repository, meaning that all IDEs should be able to run this code as long as they are using Java 10.

Here we show how to make it work in IntelliJ:

  1. Open project in IntelliJ welcome window.

  2. Choose the "manual-classifier" directory.

  3. Select "File" in the top toolbar.

  4. Choose "Project Structure".

  5. Set SDK and language level to Java10.

  6. Create an output target location.

  7. Mark "src" as Sources.

  8. Mark "out" as Excluded.

  9. Click OK.

  10. Run.

Personally, the writer of this doc met a problem where IntelliJ shows syntax errors while the program could still run. That problem was solved by setting the "language level" to 9 and then setting it back.

import a projectset up the project structureconfigure these settingschoose a project compiler outputadd a content root

run itmain should be in the run list

You should see something like this

Use the classifier

  1. We recommend using a window size of more than 720p. Otherwise, you may run into unexpected problems

  2. Choose a source folder that contains pictures to classify. Currently, only png files are tested to be supported, but in theory, all image format supported by Java should be fine.

  3. Choose the target folder. You can set it to any location - even the source folder.

  4. Configure the "X slices" and "Y slices". For example, if X is set to 4 and Y is set to 6, images will be cut into 4 * 6 slices.

  5. Flatten source functionality is not currently implemented. Nothing will happens if you change it. The program will just read image files in the source folder, ignoring all other things including subfolders, if there are any.

  6. Toggle the "Put result from different images in different dir" button if needed.

  7. If you just want to segment images without tagging:

    1. Click the "Just Segment" button to perform the task. The program will automatically exit after finished.

  8. Otherwise:

    1. Click "start" / "start with all hab" to start classifying.

    2. When classifying, all patches are originally set to UNINHABITED (or HABITABLE if you clicked "start with all hab") and marked with a BLUE border. You can click on segements them to to label them as INHABITED, making the border GREEN. Also, you can click the "set all hab" and "set all non-hab" to set all of the segments in the current image to be habitable/non-habitable.

    3. Once finished, click the "End Classification" button and the program will generate the output and exit.

  9. The result will be put into the "output" directory specified. If there is a directory called "output", the result will be put into "output_1" or "output_2" and so on.

Training the Classifier

Images were selected from large datasets of satellite images. If using a binary classifier, which simply determines if an image is inhabited or uninhabited, the manual captcha style tool can be used to segment images into these two categories to be fed into the classifier. However, when using a more complex classifier that tags different categories of land (e.g. desert, forest, etc), pre-sorted datasets should be used.

There are two platform on which the classifier can be trained: Watson Studio and Node-RED:

Watson Studio

Creating a classifier with Watson Studio

Advantages and Disadvantages of Training with Watson Studio

The benefits of Watson Studio are that the user can work with a simple drag-and-drop interface and view the classifier graphically. Unfortunately, the images must be uploaded from the user’s computer which can be inconvenient considering the file sizes of large datasets.

How to train with Watson Studio

  1. Upload zip files of different land categories to a new project.

  2. Create classes to represent each category.

  3. Drag the corresponding zip files to each category. Unless creating a binary classifier, ignore the negative class.

  4. Wait for all of the images to load and press ‘Train’.

  5. Once the classifier has been trained it can be tested in Watson Studio or through the API.

Node-RED

The Node-RED alternative

Advantages and Disadvantages of Training with Node-RED

The benefit of using Node-RED to train data is that the datasets can be retrieved directly from DropBox or IBM Cloud Object Storage, meaning the files do not need to be stored locally. However, the GUI is less user-friendly than Watson Studio so it is easier to make mistakes.

How to train with Node-RED

  1. Store zip files of categories in DropBox or IBM Cloud Object Storage.

  2. Create a new classifier by arranging nodes as shown above, providing the visual recognition node with the relevant zip-file names.

  3. Inject the node to start training.

  4. Inject into a visual recognition list node to check when the classifier has finished training (status will be ‘ready’).

  5. Once the classifier has been trained it can be tested on Node-RED or through the Watson API.

Scripts

Detailed comments and descriptions about our code.

Coding Conventions

To help future developers who want to take our project forward to get a deeper understanding of our work, we would like to introduce some code conventions we used here:

  • For JavaScript, generally, we are following the style guide from Airbnb, which complies with the ESLint rules.

  • For Java:

    • Variable names are camelCased, with a lowercase letter to start, i.e. "targetPosition".

    • Classes are PascalCased,

    • Method names are camelCased, with a lowercase letter to start.

    • Scripts are indented with 4 spaces (soft-indent).

    • Member variables should be either private or protected wherever possible.

    • Follow the suggestion given by IntelliJ (default setting).

Code Comments (by Scripts)

Node-Red Related code

Base directory: /nodeRed

Overview

This subdirectory is cloned from node-red-bluemix-starter and can be deployed to IBM cloud. Some configuration files are listed below. Any files not mentioned should probably not be changed and there is no reason to do so.

To deploy it to IBM cloud’s SDK for Node.js™ service, it should have been put in the repository’s root. We put these files in nodeRed subdirectory for clarity. Some extra steps are required to make it work:

  1. Visit the Toolchain’s Delivery Pipeline and configure the build stage.

Setting up continuous delivery

  1. In Jobs tag, select the Builder type as Shell Script and add the following build script:
#!/bin/bash # your script here

mv ./nodeRed/* . rm -r ./nodeRed

For the front-end code

cd ./front-end/nott-missing-map npm install && npm run build cd ../.. cp -a ./front-end/nott-missing- map/dist/* ./public/ rm -r ./front-end

Show the output

ls ${ARCHIVE_DIR}

manifest.yml

  • Purpose: Some settings about IBM cloud services.

  • Some Notes:

    • Memory: The run time memory of the Node.js instance (512Mb) should be enough.

    • Service: The name of Cloudent service that gets created.

bluemix-settings.js

  • Purpose: The settings of the Node-Red app, which plays the same role as the settings.js file in the normal Node-Red app. Here is the link for Node-RED configuration.

  • Some Notes:

    • Function Nodes: A collection of objects to attach to the global function context. For example,
functionGlobalContext: { osModule:require('os') }

can be accessed in a function node as:

var myos = global.get('osModule'); 
* Cross-origin resource sharing for the nodes that provide HTTP endpoints. Written:
httpNodeCors: { origin: "*", methods: ['GET','PUT','POST','DELETE'] } 
* The name of Cloudent service can be manually changed: (near the end of file)
var storageServiceName = ...

package.json

  • Purpose: Add the packages used in the Node-RED app.

  • Some Notes:

    • Most of these packages are kept there for historical reasons and are mostly not used now! Their details can be found on their respective npm pages. You can freely add new packages if needed.

    • Packages currently added:

      • node-red-contrib-file-buffer: Node-RED node which can change a file to a buffer;

      • node-red-contrib-http-multipart: Node-RED node which can accept HTTP multipart requests;

      • node-red-contrib-pythonshell: Node-RED node which can run Python script (but we haven't found a way to use pip, so it is not so useful);

      • node-red-node-dropbox: Enable Dropbox node, which can be used for classifier training;

      • node-red-contrib-cos: Add Node-RED node of IBM Cloud Object Storage, which can be used for classifier training;

      • image-to-slices: Slice images into segments. Needs to be involved in the setting file.

      • canvas: The canvas package used in image-to-slices. Needs to be involved in the setting file.

      • datauri: Generate Data URI from local files. Needs to be involved in the setting file.

      • image-size: Get image file dimensions. Needs to be involved in the setting file.

      • data-uri-to-buffer: Convert Data URI to buffer. Needs to be involved in the setting file.

    • Also, see this link for the reason why we need to include their node-red-contrib-xxx packages rather than directly manage them with Node-RED Palette Manager.

Front-end related code

Base directory: /front-end/nott-missing-map

Overview

Our Vue app was created by Vue CLI according to these instructions. In the process of continuous delivery (deploy to IBM cloud), 'npm install' and 'npm run build' will be run and the built file will be ready in ./dist. To develop and test it locally, make sure npm is available (our npm version is "5.5.1") and run 'npm install' and 'npm run serve'. Check comments for each file in the following. Some auto-generated files that are never changed may not be listed.

vue.config.js

  • About: This config file is created by Vue CLI. You can find full documentation of it here. We added an entry called configureWebpack to add aliases for paths when resolving modules (importing). You can find documentation about this feature here.

package.json

  • About: A common file you can find in Node.js app. Here is documentation from npm about what you can set in package.json. There is another file called package-lock.json which is auto-generated and officially suggested to be kept in the repository. Some important entries are listed below:

    • scripts: Commands available for npm run xxx. We added && rm dist/js/*.map to remove the unnecessary .map files after building.

    • dependencies and devDependencies: You can add dependencies with running command npm i at the root directory. And -D for devDependencies.

    • eslintConfig: The ESLint configuration. See the official document here. We add the Airbnb code conventions constraints as well as something working with webpack module alias resolution feature (match with things in vue.config.js)

public/index.html

  • About: This is another file generated by Vue CLI. It is the entrance point of the whole app. The only change we have made is adding a line that links to Google Fonts and material icons.

public/favicon.ico

  • About: This is the icon of the website, which will be shown in the top-left corner of the page tab.

src/main.js

  • About: This is the entrance of the whole website's Javascript (including Vue) code, which imports our App.vue. We register Vuetify here with the help of vuetify-loader to reduce our building output file size. (This reduced one-third of the size for us.) We also import AsyncCompute which is used in src/components/ReportCard.vue.

src/config.json

  • About: This file is the configuration of our app which we created.

  • Implementation Notes:

    • Currently, five things are recorded:

      • The URL of our back-end server, which can provide our API key together with other API querying arguments.

      • The number of segments the original input image will be sliced into.

      • The opacity of the mask when confidence level is 1 (100%).

      • The estimated ratio of consuming time of each work, which is used to calculate progress shown by the progress circular.

      • The classifier information, which is used in displaying and explaining data.

    • Problems: It is a bad idea to just give our private API key via our back-end server. We used to send the image to a Node-RED based back-end server and let it communicate with the Image Recognition service for us. However, due to the limitation of IBM cloud service, the back-end service worked significantly slowly and crashed frequently. We eventually decided to put everything in the front-end. Although we have avoided giving our API key directly in our public GitHub repo by putting in our simple back-end server instead, this solution is still not ideal.

src/App.vue

  • About: This is the main part of the app. Every .vue file is a single file Vue component, while this one is the component for our whole web app. A .vue file consists of three parts: HTML style template, Javascript code and scoped (if specified) style sheet.

  • Implementation Notes:

    • The HTML template part:

    • The Javascript part:

      • Data fields used:

        • uploadingPage: A binary value indicating the current page showing 'true' for uploading page and 'false' for the report page.

        • imgs: An array of data used in preview cards and report cards.

        • alert: Shows if there is an alert message.

        • alertMsg: Contains the content of the alert message.

        • uploading: Shows if the app is querying the classifier (trying to change from the uploading page to the report page).

        • zipping: Shows if the app is zipping the result images.

        • previewImg: Controls the information for ImgPreview used in the report page.

        • progress: The progress of the result generation process. progress.data is the actual progress. progress.max is the maximum value can be achieved, which is set upon starting generation of the result.

        • helpDialog: A boolean value indicating if the help dialog window is activated.

      • There is an customised option called sliceNum which has an x and y property, representing the number of slices in the corresponding axis. See why we did that.

      • The methods should all be commented in the code.

    • The style sheet part:

      • Classes with "xs" in their name are used in extra small conditions.

      • We set the progress circule’s transition to none as we only have a short time to update the screen when doing image processing (slicing). There are more details in src/functions/upload.js.

src/components/DragDropBox.vue

This is a .vue file of our project. You may find some general descriptions about .vue file in the src/App.vue part of this document.

  • About: This is a drag-drop box component of our app used for image uploading. References used when we built this component: 1, 2.

  • Implementation Notes:

    • The component takes two props: files and alert.

      • files: The array used to store all image files. It will be modified when the component gets new files.

      • alert: A call back function passed in which will be called with error messages as its first arguments, when the drag-drop box component wants to raise an error message.

    • The methods should all be commented in the code.

src/components/ImgPreview.vue

This is a .vue file of our project. You may find some general descriptions about .vue file in the src/App.vue part of this document.

  • About: This file is for the image preview component on the report page. It is based on the image-comparison package. It provides the comparison view for both the thumbnail in report card and the larger image preview window. It renders the image comparison part, as well as managing the resizing behaviour when the size of browser window is changed. The outer popup window is rendered and controlled by the VDialog in App.vue.

  • Implementation Notes:

    • Its prop reportCard takes an Vue component instance of ReportCard.

    • Most of the things are done in the mounted lifecycle hook. Briefly speaking, a temporary 'img' DOM element is created and drawn on the canvas on the screen, followed by the instantiation of ImageComparison.

src/components/PreviewCard.vue

This is a .vue file of our project. You may find some general descriptions about .vue file in the src/App.vue part of this document.

  • About: The preview cards appear on the uploading page, showing the images that are stored in memory and ready to be uploaded and classified.

  • Implementation Notes:

    • A prop called uploading will be passed in so that the card can disable the dismiss button when uploading.

    • The whole image array is passed in so that it can delete itself from that array.

src/components/ReportCard.vue

This is a .vue file of our project. You may find some general descriptions about .vue file in the src/App.vue part of this document.

  • About: The report cards appear on the results page, showing the images in comparison with a masked version that displays habitation confidence levels. In addition, it has a expand button to show a more detailed report of the image.

  • Implementation Notes:

    • It has the same props as that of Preview Card together with an additional one called previewImg, which is just the one with the same name in App.vue.

    • There is a method called reportInfo which specifies how the result data from classifier will be interpreted.

    • The asyncComputed property resultBlob gives the Blob data of the current image with confidence level mask. It will be used by the app to perform bulk downloading.

src/functions/upload.js

  • About: The uploading function querying the image classifier.

  • Implementation Notes:

    • It does two things with the help of Promise API.

      1. Use jimp to slice the image to pieces. As this take is CPU bound and browsers freeze when doing this, we used a strange method (the solution we use is at the end of this linked page) to work around the issue. The tradeoff is that we have to disable the fantastic transition of our circular progress bar, which is mentioned in app.vue doc.

      2. Then we used jszip to generate zip files of chunks of images (with a max size of 20) to reduce future API calls.

      3. Then we query our back-end server to access IBM’s visual recognition API and then query IBM service with the zip file. This is mentioned in config.json doc.

    • The function exported by this file will return an array of slightly processed (.classifiers[0]) results from IBM image recognition service asynchronously. (using Promise API)

src/functions/drawCanvas.js

  • About: A function which draws an image with a mask of given confidence to given canvas.

  • Implementation Notes:

    • Its params:

      • canvas: The canvas DOM element to draw on.

      • img: The img DOM element which is the base of the drawing.

      • sliceNum: Defines how many slices will be divided for x / y axes.

      • confidence: A function that return a confidence value between 0 and 1.

    • Canvas drawing reference

Manual-classifier related code

Base directory: /manual-classifier/src/grp25/captcha

Overview

This is a simple JavaFX application for pre-processing image databases before they are passed to the IBM classifier for training. Find more details about this app in Processing With a Manual Classifier part of this document.

Descriptions of each single files are provided by Javadoc.

(Last Modification: 24th April 2019)

Clone this wiki locally