Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dev/prod option in make flow for prebuilt images #97

Closed
johnhbenetech opened this issue Aug 27, 2020 · 3 comments · Fixed by #180 or #210
Closed

Add dev/prod option in make flow for prebuilt images #97

johnhbenetech opened this issue Aug 27, 2020 · 3 comments · Fixed by #180 or #210

Comments

@johnhbenetech
Copy link
Member

During make flow, if user selects prebuilt option, ask if they want to pull the prod or dev images i.e. dockerhub tags of gpu vs. gpu-dev

stepan-anokhin added a commit to stepan-anokhin/VideoDeduplication that referenced this issue Nov 6, 2020
@stepan-anokhin stepan-anokhin linked a pull request Nov 6, 2020 that will close this issue
johnhbenetech pushed a commit that referenced this issue Nov 6, 2020
* Ensure shell is bash

* Refactor setup script

* Select image version during setup (#97)

* Add make-goal for forced setup update

* Improve scripts output

* Add docker update and purge goals (#114)

* Use arrow-select for pre-built option
johnhbenetech added a commit that referenced this issue Nov 20, 2020
* File cluster proof of concept (#162)

* Implement files data-access object

* Migrate to files-dao

* Update server tests

* Fix regression

* Make module naming more ideomatic

* Resolve linting issues

* Support expunging by session scope

* Draft matches extraction

* Filter matches by distance

* Test query matches with cycles

* Test match loading

* Emulate matches pagination

* Update MatchesDAO tests

* Migrate server.api to MatchesDAO

* Update api/matches tests

* Test multiple hops with cycles

* Update file matches page

* Update cluster page to consume new matches format

* Improve graph container responsiveness

* Fix linting issues

* Improve graph responsiveness

* Implement generic loading trigger

* Support matches filtering

* Implement dynamic matches loading

* Fix loading trigger message

* Fix match reducers

* Improve dynamic match loading

* Implement neighbor loading

* Adjust graph style

* Enable zooming

* Enable cluster navigation

* Fix tooltips

* Make edge opacity dynamic

* Make color scheme static

* Handle playback issues (#165)

* Handle missing file in thumbnail endpoint

* Handle missing file in watch endpoint

* Display playback/loading errors

* Handle file missing in database case

* Use bundled flv.js

By default react-player tries to lazy-load playback SDK from CDN.
But the application must be able play video files when Internet
connection is not available. To solve that we bundle flv.js and
initialize global variable consumed by react-player's FilePlayer.

See https://www.npmjs.com/package/react-player#sdk-overrides
See cookpete/react-player#605 (comment)

* Suppress excessive flv.js error logs (#149)

* coding style improvements and small refactors

* Fix matches loading (#166) (#168)

* Support config tags (#167)

* Support tags by PathReprStorage

* Support tags by SQLiteReprStorage

* Update extract_features.py to support tags

* Add missing dependency

* Make config tag opaque

* Update extract_features.py sript

* Delete obsolete code

* Update repr storage tests

* Update generate_matches.py script

* Update template_matching.py script

* Update general_tests

* Add missing unit-test dependency

* Optimize module dependencies

* Implement side-by-side match comparison view (#155) (#169)

* Refactor file cluster page

* Refacto VideoInformationPane

* Make video details elements collapsible

* Move distance element to common package

* Add comparable file component

* Implement reusable file summary

* Implement match file list

* Implement mother file view

* Fetch matched files scenes

* Setup comparison page routing

* Reset video player on file change

* Fix matched file header

* Improve distance style

* Hook up compare button

* Fix match duplication

* Fix linting issues

* Make compare button primary-colored

* added missing dependency

* removed unnecessary files

* additional refactor and linting fixes

* Improve cluster view (#171)

* Refactor FileSummaryHeader

* Refactor linear list item

* Extract match file id from URL

* Navigate to comparison page on edge click

* Navigate to compare from single match preview

* Improve cluster tooltips

* Implement node highlighting

* Improve link popover

* Fix linting issues

* Add navigation from comparison page (#172) (#177)

* Handle partial processing results (#178)

* Handle missing data in backend (#135)

* Test missing data processing

* Fix go back navigation (#179)

* Fix go-back navigation

* Preserve filters and search results

Previous file filters and fetched files should not be updated when
we navigate from video-details page to the collection page.

* Handle missing video length

* Fix match category selector

* Ensure initial file loading

* Add match file enumeration & sorting (#181)

* Add match enumeration (#175)

* Sort files by match score

* Handle uninitialized ref

* Add image version option (#180)

* Ensure shell is bash

* Refactor setup script

* Select image version during setup (#97)

* Add make-goal for forced setup update

* Improve scripts output

* Add docker update and purge goals (#114)

* Use arrow-select for pre-built option

* Add various UI improvements (#173, #174, #176) (#182)

* Handle dense layout

* Add active filters indication (#176)

* Preserve file view type during session (#174)

* Enlarge links hitbox (#173)

* Fix link click handler

* augmented dataset evaluation pipeline

* Modifications to support benchmarking script

* Add cluster API endpoint (#183) (#184)

* Separate cluster and matches enpoints (#183)

* Update server tests

* Test matches endpoint

* Update REST client

* Refacto API client

* Creat generic hook for entity loading

* Consume matches and cluster API separately

* Manage fields inclusion

* Update immediate match loading

* Update comparison page

* Remove trash

* Remove trash

* Refactor ui state management (#185, #189) (#190)

* Move helpers to separate modules

* Move file cache to separate package

* Move file matches state to separate package

* Move cluster state to separate package

* Move file-list state to separate package

* Collect reusable prop-types in a public package

* Extract entity fetching logic

* Refactor file-cluster state

Use generic approach to manage file-cluster state.

* Refactor file-matches state

Use generic approach to manage file-matches state.

* Update server client

* Update matches params (#189)

* Fix file cluster update

* Disable false-positive linting issue

* Create post_push

* Simplify docker-compose workflow (#187, #188) (#191)

* Simplify docker-compose workflow

Use `build` and `image` simultaneously in compose-file. As a result
no shared configuration is needed and all the configuration is
managed by environment variables (located in the .env file).

To use prebuild images user will need to run `make pull`
To use local images (if production mode is disabled) user will
need to run `make build`. That's it.

* Commit missing script

* Add revision information to docker images (#188)

* Benchmarking improvements

* Update and rename post_push to build

* Delete build

* Parse exif general encoded date (#137) (#193)

* Update db schema

* Fix media-info output parsing

* Parse exif date-time

* Use date-time on backend

* Update tests

* Use numeric timestamps on frontend

* Handle NaN-representation of missing values

The exif metadata is pre-processed by the pandas.json_normalize
method which replaces any missing value with NaNs. As a result
the float value may appear in place of string.

* Handle pandas missing data representation

pandas.Series heuristically determines type of the underlying data
and tries to represent a missing values according to that data
time. In case of datetime the missing values are represented by
pandas.NaT which is not compatible with SQLAlchemy framework. To
fix that we have to explicitly transform NaTs to Nones.
See https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#datetimes

* Fix deprecated import

* Fix database URI

* Handle more date columns

* Update video metadata model (#139) (#196)

* Update db schema

* Update db access logic

* Update frontend

* Update server tests

* Update storage tests

* Additional error handling for dealing with missing files

* Added Missing frame level features check

* Quick merge fix

Co-authored-by: Stepan Anokhin <38860530+stepan-anokhin@users.noreply.github.com>
Co-authored-by: Felipe Batista <fsbatista1@gmail.com>
Co-authored-by: Stepan Anokhin <stepan.anokhin@gmail.com>
@johnhbenetech johnhbenetech reopened this Nov 24, 2020
@johnhbenetech
Copy link
Member Author

Reopening for a few things:
@stepan-anokhin can you update the ReadMe docs to reflects any changes to the make flow

On my ubuntu machine when I'm presented with make option selectors (e.g. GPU vs CPU) i can use my arrow keys to move between options, but my enter key doesnt make the selection and move to the next step. The only way im able to make a selection oddly is by using the Esc key

@johnhbenetech
Copy link
Member Author

Just a followup - Esc doesnt actually select - it just leaves those .env values blank

@stepan-anokhin
Copy link
Collaborator

@johnhbenetech could your please execute this script, press Enter, Space, arrows Up and Down and tell what the output was for each key?

#!/usr/bin/env bash

echo "Press a button:"

while read -sN1 key; do
    read -sN1 -t 0.0001 k1
    read -sN1 -t 0.0001 k2
    read -sN1 -t 0.0001 k3
    key+="${k1}${k2}${k3}"

    # Enter or Space was pressed
    if [ -z "$key" ]; then
      echo "Enter or Space detected!"
    fi

    echo "Sequence: '$(xxd -pu <<< "${key}")'";
    echo
    echo "Press a button:"
done

@stepan-anokhin stepan-anokhin linked a pull request Nov 27, 2020 that will close this issue
fsbatista added a commit that referenced this issue Nov 27, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants