Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Fixes #2069: Allow to scan products offline #2102

Closed
wants to merge 14 commits into from

Conversation

prashantkh19
Copy link
Contributor

@prashantkh19 prashantkh19 commented Jan 15, 2019

Description

A prompt to download the CSV file at the start and saving it in batches of 15k products to the local database, in order to allow offline scanning.

Related issues and discussion

#fixes #2069

@PrajwalM2212
Copy link
Member

@prashantkh19 How long does it take now ?

@prashantkh19
Copy link
Contributor Author

@prashantkh19 How long does it take now?

Apart from downloading time, approximately 4 minutes.

@PrajwalM2212
Copy link
Member

PrajwalM2212 commented Jan 16, 2019

@teolemon That would keep people waiting for around 8 to 10 minutes before starting the app if we consider all delays.

Do you want to go ahead with this ?

@teolemon
Copy link
Member

The pr is to try to find a solution ( I asked for feedback from a senior Android developer)

@teolemon teolemon changed the title Fixes #2069: Allow to scan offline (WIP) Fixes #2069: Allow to scan offline Jan 16, 2019
@PrajwalM2212 PrajwalM2212 added the offline Offline mode label Jan 18, 2019
@prashantkh19
Copy link
Contributor Author

@teolemon should I make these changes to work in the background just like #2122 ?

@teolemon
Copy link
Member

teolemon commented Jan 19, 2019

That sounds like a reasonable solution, since the download is not an issue. On first install, we could display a progress report on the home. The refresh option would be in the settings.

@prashantkh19
Copy link
Contributor Author

That sounds like a reasonable solution, since the download is not an issue. On first install, we could display a progress report on the home. The refresh option would be in the settings.

I was thinking to make a new tab in the nav menu - Offline Products. Navigating to that will let the user to initiate download for the offline package of products and storing to database as well. (both will be done in the background, with progress in the notification bar)
We can also show different packages of products based on country/language, user will then have the option to select the version of products he wants for the offline compatibility as well.
Also, we can show a label on the home page if no offline product packet is downloaded yet.
@teolemon What do you think about it? Should I proceed with this idea?

@teolemon
Copy link
Member

teolemon commented Feb 1, 2019

That sounds good to me.
I would do the background thing first, and then refine your proposed UX.

@prashantkh19 prashantkh19 force-pushed the 2069 branch 2 times, most recently from da6172e to cb9ddb7 Compare February 2, 2019 22:23
@prashantkh19
Copy link
Contributor Author

@teolemon please review the latest commit.

@teolemon
Copy link
Member

teolemon commented Feb 2, 2019

I've built the branch and started the download, and now it's extracting data. There should probably be a visual feedback in the navbar (by updating the color of the icon and or the content of the text)

@teolemon
Copy link
Member

teolemon commented Feb 2, 2019

2019-02-02 23:43:52.299 8913-9924/openfoodfacts.github.scrachx.openfood E/AndroidRuntime: FATAL EXCEPTION: AsyncTask #1
    Process: openfoodfacts.github.scrachx.openfood, PID: 8913
    java.lang.RuntimeException: An error occurred while executing doInBackground()
        at android.os.AsyncTask$3.done(AsyncTask.java:354)
        at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
        at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
        at java.util.concurrent.FutureTask.run(FutureTask.java:271)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
        at java.lang.Thread.run(Thread.java:764)
     Caused by: android.database.sqlite.SQLiteException: no such table: OFFLINE_PRODUCT (code 1 SQLITE_ERROR): , while compiling: INSERT OR REPLACE INTO "OFFLINE_PRODUCT" ("_id","TITLE","BRANDS","BARCODE","QUANTITY","NUTRITION_GRADE") VALUES (?,?,?,?,?,?)
        at android.database.sqlite.SQLiteConnection.nativePrepareStatement(Native Method)
        at android.database.sqlite.SQLiteConnection.acquirePreparedStatement(SQLiteConnection.java:903)
        at android.database.sqlite.SQLiteConnection.prepare(SQLiteConnection.java:514)
        at android.database.sqlite.SQLiteSession.prepare(SQLiteSession.java:588)
        at android.database.sqlite.SQLiteProgram.<init>(SQLiteProgram.java:58)
        at android.database.sqlite.SQLiteStatement.<init>(SQLiteStatement.java:31)
        at android.database.sqlite.SQLiteDatabase.compileStatement(SQLiteDatabase.java:1086)
        at org.greenrobot.greendao.database.StandardDatabase.compileStatement(StandardDatabase.java:67)
        at org.greenrobot.greendao.internal.TableStatements.getInsertOrReplaceStatement(TableStatements.java:68)
        at org.greenrobot.greendao.AbstractDao.insertOrReplaceInTx(AbstractDao.java:249)
        at org.greenrobot.greendao.AbstractDao.insertOrReplaceInTx(AbstractDao.java:259)
        at openfoodfacts.github.scrachx.openfood.jobs.ExtractOfflineProductService$saveCSVToDb.doInBackground(ExtractOfflineProductService.java:216)
        at openfoodfacts.github.scrachx.openfood.jobs.ExtractOfflineProductService$saveCSVToDb.doInBackground(ExtractOfflineProductService.java:138)
        at android.os.AsyncTask$2.call(AsyncTask.java:333)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245) 
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
        at java.lang.Thread.run(Thread.java:764) 

@teolemon
Copy link
Member

teolemon commented Feb 2, 2019

@prashantkh19 perhaps a missing GreenDAO migration ?

@teolemon
Copy link
Member

teolemon commented Feb 2, 2019

I cleaned the cache, and got to "No remaining download", so it seems to work.

@prashantkh19
Copy link
Contributor Author

@prashantkh19 perhaps a missing GreenDAO migration ?

yeah, a simple reinstall should work.

@prashantkh19
Copy link
Contributor Author

prashantkh19 commented Feb 3, 2019

2019-02-02 23:43:52.299 8913-9924/openfoodfacts.github.scrachx.openfood E/AndroidRuntime: FATAL EXCEPTION: AsyncTask #1
Process: openfoodfacts.github.scrachx.openfood, PID: 8913
java.lang.RuntimeException: An error occurred while executing doInBackground()
at android.os.AsyncTask$3.done(AsyncTask.java:354)
at java.util.concurrent.FutureTask.finishCompletion(FutureTask.java:383)
at java.util.concurrent.FutureTask.setException(FutureTask.java:252)
at java.util.concurrent.FutureTask.run(FutureTask.java:271)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:764)
Caused by: android.database.sqlite.SQLiteException: no such table: OFFLINE_PRODUCT (code 1 SQLITE_ERROR): , while compiling: INSERT OR REPLACE INTO "OFFLINE_PRODUCT" ("_id","TITLE","BRANDS","BARCODE","QUANTITY","NUTRITION_GRADE") VALUES (?,?,?,?,?,?)
at android.database.sqlite.SQLiteConnection.nativePrepareStatement(Native Method)
at android.database.sqlite.SQLiteConnection.acquirePreparedStatement(SQLiteConnection.java:903)
at android.database.sqlite.SQLiteConnection.prepare(SQLiteConnection.java:514)
at android.database.sqlite.SQLiteSession.prepare(SQLiteSession.java:588)
at android.database.sqlite.SQLiteProgram.(SQLiteProgram.java:58)
at android.database.sqlite.SQLiteStatement.(SQLiteStatement.java:31)
at android.database.sqlite.SQLiteDatabase.compileStatement(SQLiteDatabase.java:1086)
at org.greenrobot.greendao.database.StandardDatabase.compileStatement(StandardDatabase.java:67)
at org.greenrobot.greendao.internal.TableStatements.getInsertOrReplaceStatement(TableStatements.java:68)
at org.greenrobot.greendao.AbstractDao.insertOrReplaceInTx(AbstractDao.java:249)
at org.greenrobot.greendao.AbstractDao.insertOrReplaceInTx(AbstractDao.java:259)
at openfoodfacts.github.scrachx.openfood.jobs.ExtractOfflineProductService$saveCSVToDb.doInBackground(ExtractOfflineProductService.java:216)
at openfoodfacts.github.scrachx.openfood.jobs.ExtractOfflineProductService$saveCSVToDb.doInBackground(ExtractOfflineProductService.java:138)
at android.os.AsyncTask$2.call(AsyncTask.java:333)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:245) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) 
at java.lang.Thread.run(Thread.java:764) 

I would see some way to handle this exception so that app won't crash if this happens.

@prashantkh19
Copy link
Contributor Author

I've built the branch and started the download, and now it's extracting data. There should probably be a visual feedback in the navbar (by updating the color of the icon and or the content of the text)

Just for the information, aren't you getting these type of notifications during the process?

  1. 1
  2. Here user needs to select Extract and Save option to proceed.
    2
  3. 3
  4. 4
  5. 5

@teolemon
Copy link
Member

teolemon commented Feb 3, 2019

I didn't spot any. I'm on Android Pie, where notification behaviour might have changed. Going to check again.
The exception will likely happen for anyone upgrading from a previous version of Open Food Facts.

@teolemon
Copy link
Member

teolemon commented Feb 3, 2019

I just checked again. I see progress in LogCat, but nothing in notifications.

@prashantkh19
Copy link
Contributor Author

I just checked again. I see progress in LogCat, but nothing in notifications.

Okay then I will modify notifications for android pie.

@prashantkh19
Copy link
Contributor Author

Apart from this, do you have any other suggestion?

@teolemon
Copy link
Member

teolemon commented Feb 3, 2019

Probably already in your plans

  • I would probably create a real activity showing progress report rather than a menu item that you have to click twice and that doesn't change to indicate progress

Longer term:

  • ability to delete the offline db and re-download it (or even a scheduled task every month)
  • have a small version (the current one) and full versions (without images) for a whole country

@prashantkh19
Copy link
Contributor Author

@teolemon review please..
Screenshot:
screenshot_2019-02-13-12-35-10-178_openfoodfacts github scrachx openfood

@teolemon
Copy link
Member

Download https://dl.google.com/dl/android/maven2/androidx/databinding/databinding-compiler/3.2.0/databinding-compiler-3.2.0.jar
:app:compileOpfDebugJavaWithJavac/home/travis/build/openfoodfacts/openfoodfacts-androidapp/app/src/main/java/openfoodfacts/github/scrachx/openfood/fragments/ProductsDownloadFragment.java:79: error: cannot find symbol
                        dDownloadButton.setImageResource(R.drawable.baseline_autorenew_24);
                                                                   ^
  symbol:   variable baseline_autorenew_24
  location: class drawable
/home/travis/build/openfoodfacts/openfoodfacts-androidapp/app/src/main/java/openfoodfacts/github/scrachx/openfood/fragments/ProductsDownloadFragment.java:108: error: cannot find symbol
                        eExtractButton.setImageResource(R.drawable.baseline_autorenew_24);
                                                                  ^
  symbol:   variable baseline_autorenew_24
  location: class drawable
/home/travis/build/openfoodfacts/openfoodfacts-androidapp/app/src/main/java/openfoodfacts/github/scrachx/openfood/fragments/ProductsDownloadFragment.java:118: error: cannot find symbol
                        eExtractButton.setImageResource(R.drawable.baseline_autorenew_24);
                                                                  ^
  symbol:   variable baseline_autorenew_24
  location: class drawable
/home/travis/build/openfoodfacts/openfoodfacts-androidapp/app/src/main/java/openfoodfacts/github/scrachx/openfood/fragments/ProductsDownloadFragment.java:153: error: cannot find symbol
            dDownloadButton.setImageResource(R.drawable.baseline_autorenew_24);
                                                       ^
  symbol:   variable baseline_autorenew_24
  location: class drawable
/home/travis/build/openfoodfacts/openfoodfacts-androidapp/app/src/main/java/openfoodfacts/github/scrachx/openfood/fragments/ProductsDownloadFragment.java:159: error: cannot find symbol
                eExtractButton.setImageResource(R.drawable.baseline_autorenew_24);
                                                          ^
  symbol:   variable baseline_autorenew_24
  location: class drawable
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
5 errors
 FAILED

@prashantkh19

app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
app/src/main/res/values/strings.xml Show resolved Hide resolved
@teolemon
Copy link
Member

teolemon commented Mar 2, 2019

We're releasing the next version next week. Any chance the feature could make it ?

@prashantkh19
Copy link
Contributor Author

We're releasing the next version next week. Any chance the feature could make it?

I am sorry, I have my exams starting next week. I would be able to work on it after March 16 only.

@teolemon
Copy link
Member

teolemon commented Mar 4, 2019

good luck 🤞
I have fixed the CSV export for NOVA which wasn't exported correctly:
https://world.openfoodfacts.org/data/offline/en.openfoodfacts.org.products.small.csv.zip

@teolemon
Copy link
Member

teolemon commented Mar 4, 2019

In terms of chronology, we should try to load from the local db first, then if the network query is successful, replace the offline result by the online result

@teolemon
Copy link
Member

Here's what we're doing on iOS to let the user know about the Offline mode
image
@prashantkh19

@elouataoui
Copy link
Contributor

I'd like to work on this if it's okay with everyone. My understanding is that the PR currently deals with downloading the data and storing it locally. My goal will be to:

  • Implement offline scanning based on the locally stored data
  • Try fetching the data online in parallel and refresh the view if necessary
  • Align the data download UX with what has been done in iOS

@teolemon
Copy link
Member

@elouataoui sound good to me.
Note that on iOS, we download the whole world, like here, and refresh automatically on a monthly basis. We added that message while loading in home, so that people are aware that there is an offline mode, and try it :-)

@deniger
Copy link
Contributor

deniger commented Sep 18, 2019

@teolemon
Copy link
Member

=======
<RelativeLayout xmlns:android="http://schemas.android.com/apk/res/android"
        xmlns:app="http://schemas.android.com/apk/res-auto"
        xmlns:tools="http://schemas.android.com/tools"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:orientation="vertical">

    <androidx.constraintlayout.widget.ConstraintLayout
            android:layout_width="match_parent"
            android:layout_height="match_parent"
            android:layout_marginBottom="50dp">

        <androidx.cardview.widget.CardView
                android:id="@+id/message_container_card_view"
                android:layout_width="0dp"
                android:layout_height="wrap_content"
                android:layout_marginLeft="@dimen/spacing_small"
                android:layout_marginRight="@dimen/spacing_small"
                android:layout_marginBottom="8dp"
                android:elevation="1dp"
                android:padding="@dimen/padding_large"
                app:layout_constraintBottom_toTopOf="@+id/buttonSendAll"
                app:layout_constraintHorizontal_bias="0.0"
                app:layout_constraintLeft_toLeftOf="parent"
                app:layout_constraintRight_toRightOf="parent"
                app:layout_constraintTop_toTopOf="parent">

            <RelativeLayout
                    android:layout_width="match_parent"
                    android:layout_height="wrap_content"
                    android:background="@color/white">

                <ImageButton
                        android:id="@+id/message_dismiss_icon"
                        android:layout_width="@dimen/button_height_too_short"
                        android:layout_height="@dimen/button_height_too_short"
                        android:layout_alignParentTop="true"
                        android:layout_alignParentEnd="true"
                        android:layout_alignParentRight="true"
                        android:layout_margin="@dimen/padding_short"
                        android:background="@color/white"
                        app:srcCompat="@drawable/ic_action_cross" />


                <LinearLayout
                        android:layout_width="wrap_content"
                        android:layout_height="wrap_content"
                        android:orientation="vertical">

                    <TextView
                            android:layout_width="match_parent"
                            android:layout_height="wrap_content"
                            android:gravity="center"
                            android:padding="@dimen/padding_short"
                            android:text="@string/title_info_dialog"
                            android:textAlignment="center"
                            android:textSize="@dimen/font_large"
                            android:textStyle="bold" />

                    <TextView
                            android:layout_width="match_parent"
                            android:layout_height="wrap_content"
                            android:gravity="center"
                            android:padding="@dimen/padding_normal"
                            android:text="@string/text_offline_info_dialog"
                            android:textAlignment="center"
                            android:textSize="@dimen/font_normal" />
                </LinearLayout>
            </RelativeLayout>
        </androidx.cardview.widget.CardView>

        <RelativeLayout
                android:layout_width="match_parent"
                android:layout_height="match_parent"
                android:orientation="vertical">

            <ImageView
                    android:id="@+id/noDataImg"
                    android:layout_width="@dimen/img_width_height"
                    android:layout_height="@dimen/img_width_height"
                    android:layout_below="@id/noDataText"
                    android:layout_centerHorizontal="true"
                    android:layout_margin="@dimen/padding_normal"
                    android:paddingTop="@dimen/padding_large"
                    android:visibility="visible"
                    app:srcCompat="@drawable/ic_cloud_done" />

            <TextView
                    android:id="@+id/noDataText"
                    android:layout_width="wrap_content"
                    android:layout_height="wrap_content"
                    android:layout_centerHorizontal="true"
                    android:layout_margin="@dimen/padding_normal"
                    android:gravity="center"
                    android:text="@string/no_offline_data"
                    android:textColor="@color/grey_500"
                    android:textSize="35sp"
                    android:visibility="visible" />

        </RelativeLayout>

        <LinearLayout
                android:id="@+id/linearLayout2"
                android:layout_width="0dp"
                android:layout_height="0dp"
                android:layout_marginBottom="8dp"
                android:orientation="vertical"
                app:layout_constraintBottom_toTopOf="@+id/buttonSendAll"
                app:layout_constraintEnd_toEndOf="parent"
                app:layout_constraintStart_toStartOf="parent"
                app:layout_constraintTop_toTopOf="parent">

            <androidx.recyclerview.widget.RecyclerView
                    android:id="@+id/listOfflineSave"
                    android:layout_width="match_parent"
                    android:layout_height="match_parent"
                    android:layout_margin="@dimen/spacing_normal"
                    tools:listitem="@layout/save_list_item" />
        </LinearLayout>


        <openfoodfacts.github.scrachx.openfood.utils.CustomButtonView
                android:id="@+id/buttonSendAll"
                style="@style/DefaultButtonText"
                android:layout_width="match_parent"
                android:layout_height="70dp"
                android:layout_gravity="center"
                android:layout_margin="@dimen/spacing_normal"
                android:layout_marginStart="14dp"
                android:layout_marginEnd="14dp"
                android:background="@drawable/rounded_button"
                android:text="@string/txtSendAll"
                android:textSize="@dimen/font_normal"
                app:layout_constraintBottom_toBottomOf="parent"
                app:layout_constraintEnd_toEndOf="parent"
                app:layout_constraintStart_toStartOf="parent" />


    </androidx.constraintlayout.widget.ConstraintLayout>

    <include layout="@layout/navigation_bottom" />
</RelativeLayout>
>>>>>>> master

@teolemon
Copy link
Member

@prashantkh19 @deniger I just rebased. I hope I didn't break anything. 🤞

@teolemon
Copy link
Member

teolemon commented Oct 20, 2019

/Users/pierre/Desktop/openfoodfacts-androidapp-2069/app/src/main/res/layout/offline_list_item.xml:10: AAPT: error: resource color/green_700 (aka openfoodfacts.github.scrachx.openfood:color/green_700) not found.

@teolemon
Copy link
Member

teolemon commented Oct 20, 2019

/Users/pierre/Desktop/openfoodfacts-androidapp-2069  
app/src/main/java  
openfoodfacts/github/scrachx/openfood/models/OfflineProduct.java  
error: package com.opencsv.bean does not exist  
error: cannot find symbol class CsvBindByName  
error: cannot find symbol class CsvBindByName  
error: cannot find symbol class CsvBindByName  
error: cannot find symbol class CsvBindByName  
error: cannot find symbol class CsvBindByName  
openfoodfacts/github/scrachx/openfood/jobs/DownloadOfflineProductService.java  
error: package android.support.annotation does not exist  
error: cannot find symbol class NotificationCompat  
error: package com.opencsv does not exist  
error: package NotificationCompat does not exist  
error: cannot find symbol class Nullable  
openfoodfacts/github/scrachx/openfood/fragments/OfflineEditFragment.java  
error: cannot find symbol class OfflineListAdapter  
openfoodfacts/github/scrachx/openfood/views/adapters/OfflineListAdapter.java  
error: package android.support.annotation does not exist  
error: package android.support.v7.widget does not exist  
error: package android.support.v7.widget does not exist  
error: package RecyclerView does not exist  
error: package RecyclerView does not exist  
error: cannot find symbol class NonNull  
error: cannot find symbol class NonNull  
error: cannot find symbol class NonNull  

@teolemon teolemon changed the title (WIP) Fixes #2069: Allow to scan offline (WIP) Fixes #2069: Allow to scan products offline Oct 24, 2019
@teolemon teolemon changed the title (WIP) Fixes #2069: Allow to scan products offline [WIP] Fixes #2069: Allow to scan products offline Oct 27, 2019
@teolemon
Copy link
Member

teolemon commented Nov 1, 2019

Some of it seems to be related to the move to AndroidX

@teolemon
Copy link
Member

teolemon commented Nov 1, 2019

Screenshot_20191101-194529
Screenshot_20191101-194522
Screenshot_20191101-194517
Screenshot_20191101-194723
Screenshot_20191101-194907

@teolemon
Copy link
Member

teolemon commented Nov 1, 2019

For some reason, I managed to build it and it doesn't crash. The Product view has beed ****, and it seems to be the old one.

@teolemon
Copy link
Member

@teolemon teolemon added the P1 label May 26, 2020
@teolemon
Copy link
Member

teolemon commented Jun 14, 2020

New conflicts. The SplashActivity one is tricky for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Up for Adoption] Allow to scan barcodes offline
6 participants