LearnYourEnvironment

Introduction

Learn Your Environment – Animals is developed for children who are old enough to identify elements of their environment by name. This app aims to teach children about animals domestic or wild. Children can also practice pronouncing animal names and see an estimated score based on their pronuncitaion.

About HUAWEI ML Kit – Automatic Speech Recognition

Automatic speech recognition (ASR) can recognize speech no longer than 60s and convert the input speech into text in real time. This service uses industry-leading deep learning technology to achieve a recognition accuracy of over 95%.

For more information -> HUAWEI ML Kit – Automatic Speech Recognition Guide

About HUAWEI ML Kit – Text to Speech

Text to speech (TTS) can convert text information into audio output in real time. Rich timbres are provided and the volume and speed can be adjusted, thereby natural voices can be produced.

For more information -> HUAWEI ML Kit – Text to Speech

About Huawei ML Kit – Custom Model Text Classification

It allows us to extract values ​​by grouping texts, all the while using certain classifications. The ML kit allows us to create our own custom model with AI Create.

Before the start

1- Developer Account

You should have an AppGallery Console developer account. If you do not have an account, you should create an account.

2- HMS Toolkit

To use AI Create, the HMS Toolkit must be installed. Android Studio: File → Settings → Plugins → HMS Toolkit ! Your IDE must be restarted after installation

Important: HMS Toolkit is needed to support the Android Studio Bumblebee version from June 2022. It is currently available in the Android Arctic Fox version.

3- Python for ML Kit

AI Creator requires Python v3.7.5 to use the ML Kit. Complete your installation by downloading Python 3.7.5 from here.

Notes:

• Choose to add the path during installation.
• Make sure to install with the pip selected on the wizard during installation.
• The path may conflict with a possible other python version already installed. If so, you will have to uninstall the old version to install the new one.
• If Android Studio is open when installing Python, it must be restarted.

4- Data Preparation

You can use two types of text data in HMS ML Kit Text Classification.

• Training text in JSON format: The JSON text format is [{“label”: “ “, “text”: “ “},{“label”: “ “, “text”: “ “}]
  [{“label”: “category1”, “text”: “sample text”}, {“label”: “category2”, “text”: “sample text”}, {“label”: “category3”, “text”: “sample text”}]
• Training text in TXT format: Classify the TXT files by category.
  
• You should separate dataset as % 80 train data and % 20 test data

Project Steps

1- Create an Android Project

Let’s start with creating an empty android project.
Notes: The min. sdk for text classification should be 22.

2- App Gallery Connection:

2.1- Create your project by going to My Projects -> Add new project by going to the App Gallery Connect Console.
2.2- Enable data processing location

2.2- Add the App to the Project
2.3- Integrate the AppGallery Connect SDK in Android Studio:
SDKs are added to the gradle file before the agconnect-services.json file (because we haven’t activated the ML kit service yet).
a) Project-level build.gradle
allprojects {
repositories {
// Add the Maven address. maven {url ‘https://developer.huawei.com/repo/'} } }
…
buildscript{
repositories {
// Add the Maven address. maven { url ‘https://developer.huawei.com/repo/' } }
dependencies {
// Add dependencies. classpath ‘com.huawei.agconnect:agcp:1.5.2.300’ } }

b) App-level build.gradle

dependencies { // Add dependencies.
implementation ‘com.huawei.agconnect:agconnect-core:1.5.2.300’ }
…
// Add the information to the bottom of the file.
apply plugin: ‘com.huawei.agconnect’

2.4- Enable ML Kit Service
My projects → Project → Build → ML Kit (in left menu)

2.5- Add agconnect-services.json Top left menu – project settings → app information → download agconnect-services.json file Then add it to your project in the app directory in project view Then click the "Enable now" button

3 Model Creation

Requirements:

• The number of text categories is at least 2.
• Each category contains at least 20 files. More files will guarantee a higher model precision.
• The text size does not exceed 10 MB.
• The file encoding format is UTF-8.

3.1- Coding Assistant

If you have installed Android Studio you should see a HMS plugin.
HMS → Coding Assistant → AI → AI Create → Text Classification

Note: You may see a dialog box, as seen below.

After clicking "Ok" here, you will be asked to choose a path and save a mindspore file (.whl) to the location you selected.

• Open the CMD from your computer and access the path where the mindspore file "cd Mindspore_Path" is located.
  
• Install it with the "pip install Mindspore_filename" pip.
  

3.2- Creating Model

Here we add the dataset we have created.

• Only Chinese and English are currently supported.
• Obtain a sample dataset by selecting the "Case 1" and "Case 2" options.
• Case 1: IMDB dataset (json format), Case 2: News Categories (TXT format)
• If you are experienced in this field, you can change the iteration count and learning rate values within the Advanced Settings.

3.3- Testing Your Model

Find the accuracy rate in the logs when "train" and "validate" are at 100%.

We will upload the test model we have prepared to test the model we have created.

• Obtain a sample dataset by selecting the "Case 1" and "Case 2" options.
  End the test process, measure the average accuracy of the test.
  

4 Add the Model Created to the Application

4.1- Add a Dependency

implementation 'com.huawei.hms:ml-computer-model-executor:2.1.0.300'
implementation 'mindspore:mindspore-lite:5.0.5.300'
implementation 'com.huawei.hms:base:6.1.0.300'

4.1- Add a Model File

Add .aar file in libs

implementation files('libs/ms-textclassifier-2.0.0.aar')

4.2- Add a .ms Extension

The following section must be added under the app-level android section for our model file with a .ms extension to work.
android {
...
aaptOptions {
noCompress "ms", "tflite"
}
}

4.3- Add the .ms Extension to your Asset Folder

4.3- Add Permissions to the Manifest Folder

• <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
  
• <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
  
• <uses-permission android:name="android.permission.INTERNET" />
  

What You Will Need

Hardware Requirements

• A computer that can run Android Studio.
• A Huawei mobile device with a USB data cable to run developed apps.

Software Requirements

• Java JDK 1.8 or later
• Android Studio 2021.2.x
• Android SDK package
• Android API Level 23 or higher
• HMS Core (APK) 5.0.0.300 or later
• EMUI 5.0 or later

Getting Started

Learn Your Environment uses Huawei services. To use them, you must create an app first. Before getting started, please sign-up for a Huawei developer account.

After creating the application, you must generate a signing certificate fingerprint. You must then set this fingerprint to the application you created in AppGallery Connect.

• Go to "My Projects" in AppGallery Connect.
• Find your project from the project list and click the app on the project card.
• On the Project Settings page, set the "SHA-256 certificate fingerprint" to the SHA-256 fingerprint you've generated.
• Change the agconnect-services.json file to yours.
• To use the ML Kit in your app, you will also need to enable the service in AppGallery Connect. Then obtain the API key from AppGallery Connect. For details, please refer to Preparations for ML Kit Service.
• Replace the key with the one in your API key.

Using the Application

Upon completing the essential parts of the code, connect your mobile device to your PC and enable USB debugging mode. In the Android Studio window, click the icon to run the project you have created in Android Studio to generate an APK. Then install the APK on the mobile device.

1. Before you run the app, make sure that you have a working internet connection.
   
2. Open the app upon installing it onto your device.
   
3. The application will ask you for permission to record audio. Please allow it to use the app with voice commands.
   
4. You will see a welcome screen that contains the "Play Game" button.
   
5. You will see an Animal List Screen that contains "Animals". Choose one of them.
   
6. You will see an Animal Detail Screen with the words "listen dialog", "Mic Button" and "Sound Button"
   
7. You will see a result dialog box displaying "Result", "Go To Home" and "Go to previous Page" buttons.
   

Screenshots

Project Structure

Learn Your Environment is designed from the MVVM design pattern.

Libraries

• Huawei ML Kit – Automatic Speech Recognition
• Huawei ML Kit – Text to Speech
• Huawei ML Kit – Custom Model Text Classification
• ViewModel
• Navigation
• ViewBinding
• Dagger Hilt
• Lottie

Contributors

Ertug Sagman
Ataberk Celiktas