-
Notifications
You must be signed in to change notification settings - Fork 3
/
README.md
179 lines (135 loc) · 8.33 KB
/
README.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
# BetterDay opensource 
By:
- Tu Nguyen (Nguyen Hoang Anh Tu)
- Huy Dang (Dang Nhat Huy)
- Van Bui (Bui Thi Cam Van)
- Quan Nguyen (Tran Nguyen Trung Quan)
- Dat Pham (Pham Tien Dat)
**The latest version of the BetterDay tool is now open-source and available [here](https://github.com/Nhathuy1305/BetterDay-Tool) under the Apache-2.0 license.**
This is an open-source version of the BetterDay video search and retrieval
engine, slightly simplified from the prototype that was featured at [AI Challenge 2023](https://aichallenge.hochiminhcity.gov.vn) in Ho Chi Minh City, Vietnam.
Main features:
- a very basic demonstration dataset is included in the repository and is accessible for download using the Weaviate Cloud dataset.
- similarity-based search, keyword search
- several methods for perusing and re-scoring the outcomes
- Submission and tracking API clients compatible with VBS
BetterDay is licensed as free software based on SOMHunter opensource, under the GPL-2.0 license. This grants
you freedom to use the software for many research purposes and publish the
results. For exploring and referencing the original work, you may find some of
the following articles helpful:
- Kratochvíl, M., Veselý, P., Mejzlík, F., & Lokoč, J.
(2020, January).
[SOM-Hunter: Video Browsing with Relevance-to-SOM Feedback Loop](https://link.springer.com/chapter/10.1007/978-3-030-37734-2_71).
In *International Conference on Multimedia Modeling* (pp. 790-795). Springer, Cham.
- Mejzlík, F., Veselý, P., Kratochvíl, M., Souček, T., & Lokoč, J.
(2020, June).
[SOMHunter for Lifelog Search](https://dl.acm.org/doi/abs/10.1145/3379172.3391727).
In *Proceedings of the Third Annual Workshop on Lifelog Search Challenge* (pp. 73-75).
- Kratochvil, M., Mejzlík, F., Veselý, P., Soućek, T., & Lokoć, J.
(2020, October).
[SOMHunter: Lightweight Video Search System with SOM-Guided Relevance Feedback](https://dl.acm.org/doi/10.1145/3394171.3414542).
In *Proceedings of the 28th ACM International Conference on Multimedia* (pp. 4481-4484).
- Veselý, P., Mejzlík, F., & Lokoč, J.
(2021, June).
[SOMHunter V2 at Video Browser Showdown 2021](https://link.springer.com/chapter/10.1007/978-3-030-67835-7_45).
In *International Conference on Multimedia Modeling* (pp. 461-466). Springer, Cham.
## Try BetterDay from Docker
<!--   -->
<!--  -->
You can get a working BetterDay copy from Docker:
```sh
docker pull Nhathuy1305/BetterdayTool:v0.1
docker run -ti --rm -p8080:8080 Nhathuy1305/BetterdayTool:v0.1
```
After that, open your browser at http://localhost:8080, and use login `tu` and password `cute`.
<!--  -->
## Installation from source
Prerequisites:
- a working installation of Node.js with some variant of package manager
(either `npm` or `yarn`)
- Python 3
- Java compiler
- `libcurl` (see below for installation on various platforms)
After cloning this repository, change to the repository directory and run
```
npm install
npm run start
```
(Optionally replace `npm` with `yarn`.)
If everything goes all right, you can start browsing at http://localhost:8080/
. The site is password-protected by default, you can use the default login
`tu` and password `cute`, or set a different login in `config/user.js`.
### Getting the dependencies on UNIX systems
You should be able to install all dependencies from the package management. On
Debian-based systems (including Ubuntu and Mint) the following should work:
```
apt-get install build-essential libcurl4-openssl-dev nodejs yarnpkg
```
The build system uses `pkg-config` to find libCURL -- if that fails, either
install the CURL pkgconfig file manually, or customize the build configuration
in `core/binding.gyp` to fit your setup.
Similar (similarly named) packages should be available on most other distributions.
### Getting the dependencies on Windows
The build systems expects libCURL to reside in `c:\Program Files\curl\`. You
may want to install it using
[vcpkg](https://docs.microsoft.com/en-us/cpp/build/vcpkg?view=vs-2019) as
follows:
- download and install `vcpkg`
- install and export libCURL:
```
vcpkg install curl:x64-windows
vcpkg export --raw curl:x64-windows
```
- copy the directory with the exported libCURL to `c:\Program Files\`.
Alternatively, you can use any working development installation of libCURL by
filling the appropriate paths in `core/binding.gyp`.
### Build problems
We have tested BetterDay on Windows and several different Linux distributions,
which should cover a majority of target environments. Please report any errors
you encounter using the GitHub issue tracker, so that we can fix them (and
improve the portability of BetterDay).
### Building the Docker image
The installation is described in `Dockerfile`; you should be able to get a
working, correctly tagged (and possibly customized) image by running this in
your directory:
```sh
docker build -t betterday:$(git describe --always --tags --dirty=-$USER-$(date +%Y%m%d-%H%M%S)) .
```
## Customizing BetterDay
The program is structured as follows:
- The frontend requests are routed in `app.js` to views and actions in `routes/betterday.js`, display-specific routes are present in `routes/endpoints.js`
- The views (for the browser) are rendered in `views/batterday.ejs`
- Node.js "frontend" communicates with Java "backend" that handles the main data operations; the backend source code is in `core/`; the main API is in `core/BetterDayNapi.h` (and `.cpp`)
- The backend implementation is present in `core/src/` which contains the following modules (`.cpp` and `.h`):
- `BetterDay` -- main data-holding structure with the Java version of the wrapper API
- `Submitter` -- VBS API client for submitting search results for the competition, also contains the logging functionality
- `DatasetFrames` -- loading of the dataset description (frame IDs, shot IDs, video IDs, ...)
- `DatasetFeatures` -- loading of the dataset feature matrix
- `KeywordRanker` -- loading and application of W2VV keywords (see Li, X., Xu, C., Yang, G., Chen, Z., & Dong, J. (2019, October). [W2VV++ Fully Deep Learning for Ad-hoc Video Search](https://dl.acm.org/doi/pdf/10.1145/3343031.3350906). In *Proceedings of the 27th ACM International Conference on Multimedia* (pp. 1786-1794).)
- `RelevanceScores` -- maintenance of the per-frame scores and feedback-based re-ranking
- `SOM` and `AsyncSom` -- SOM implementation, background worker that computes the SOM
Additional minor utilities include:
- `config.h` that contains various `#define`d constants, including file paths
- `log.h` which defines a relatively user-friendly logging with debug levels
- `use_intrins.h` and `distfs.h` define fast SSE-accelerated computation of vector-vector operations (provides around 4x speedup for almost all computation-heavy operations)
- `Application.java`, which is __not__ compiled-in by default, but demonstrates how to run the BetterDay core as a standalone Java application.
### HOW-TOs
- [Adding a new display type](HOWTO-display.md)
- [Modifying the re-scoring functionality](HOWTO-scores.md)
## Datasets
The repository contains a (very small) pre-extracted indexed dataset (see
https://doi.org/10.1109/ICMEW.2015.7169765 for dataset details). That should be
ready to use.
We can provide a larger pre-indexed dataset based on the [V3C1 video
collection](https://link.springer.com/chapter/10.1007/978-3-030-05710-7_29),
but do not provide a direct download due to licensing issues. Please contact us
to get a downloadable link. You will need to have the TRECVID data use
agreement signed; see
https://www-nlpir.nist.gov/projects/tv2019/data.html#licenses for details.
### Using custom video data
You may set up the locations of the dataset files in `config.json`. The
thumbnails of the extracted video frames must be placed in directory
`public/thumbs/`, so that they are accessible from the browser. (You may want
to use a symbolic link that points to the thumbnails elsewhere, in order to
save disk space and IO bandwidth.)
Description of extracting data from custom dataset is available in directory `extractor/` with a [separate README](extractor/README.md).