-
Notifications
You must be signed in to change notification settings - Fork 3
/
how_did_you_build_your_detector.html
961 lines (926 loc) · 46.1 KB
/
how_did_you_build_your_detector.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<title>How Did You Build Object Detector? - Vladimir Iashin</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Open+Sans&display=swap" rel="stylesheet">
<link rel="stylesheet" href="css/style_phd_times.css">
<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.1.0/styles/monokai-sublime.min.css">
<link rel="icon" href="./favicon.ico" type="image/x-icon">
</head>
<style>
#toc_container {
padding: 1em;
}
li {
display: list-item;
}
a.intext {
word-wrap: break-word;
}
.toc_section {
padding-top: 2em;
}
.toc_subsection {
padding-top: 0.7em;
font-size: 0.8em;
padding-bottom: 0em;
}
#toc_container li, #toc_container ul, #toc_container ul li {
list-style: outside none none !important;
}
code {
border-radius: 0.7ex;
font-size: 1em;
}
.inline {
display: inline;
padding: 0.1em 0.4em 0.1em 0.4em;
background-color: #f0f0f0;
color: brown;
border-radius: 0.7ex;
word-wrap: break-word;
}
.shadow {
box-shadow: rgba(0, 0, 0, 0.24) 0px 3px 8px;
}
img.left {
margin: 0em 1em 0em 0em;
}
img.right {
margin: 0em 0em 0em 1em;
}
p.no_bottom_pad {
padding-bottom: 0em;
}
@media screen and (max-width: 45em) {
.to_col_if_narrow {
flex-direction: column;
}
img.left, img.right {
margin: 0em 0em 1em 0em;
}
}
table {
border-spacing: 0 5px;
margin-left: auto;
margin-right: auto;
font-size: 120%;
/* line-height: 120%; */
border-top: 3px solid black;
border-bottom: 3px solid black;
/* padding-bottom: 1em; */
margin-bottom: 2em;
}
thead, th {
border-bottom: 2px solid black;
border-spacing: 5px 5px;
}
th, td {
padding: 0.4em 1em 0.4em 1em;
text-align: right;
}
.date {
padding-bottom: 1em;
text-align: right;
}
@media screen and (max-width: 50em) {
table {
font-size: 2.8vw;
line-height: 4.0vw;
border-top: 0.3vw solid black;
border-bottom: 0.3vw solid black;
}
thead, th {
border-bottom: 0.2vw solid black;
}
th, td {
padding: 0.4vh 2vw 0.4vh 2vw;
}
pre {
font-size: 0.8em;
line-height: 1.5em;
}
body {
font-size: 0.8em;
}
.date {
padding-bottom: 1em;
text-align: center;
}
}
</style>
<body>
<ul class="breadcrumb">
<a class="bread_crumb" href="index.html">Vladimir Iashin</a> / Website with an Object Detector: How to Build It?
</ul>
<div class="img background_white_squared" style="border: none;padding: 0em;">
<a href="./detector.html">
<img src="./images/detector/detector_in_action.jpg" alt="Result of the Detector">
</a>
</div>
<h1 style="font-size: 2.8em;"> Website with an Object Detector: <br> How to Build It? </h1>
<p class="p_on_project_pages date">
<i>June, 2021. The project was built in 2019.</i>
</p>
<div class="our_framework">
<div class="section_content">
<p class="p_on_project_pages">
Recently I started to get requests through multiple means asking about the implementation details
of my <a href="./detector.html" class="intext"> Online Object Detector</a>.
This post is composed to address these questions.
</p>
<p class="p_on_project_pages">
Setting up the whole engineering pipeline includes many steps full of caveats that took me some time
to work around.
However, I am not going to cover how detection or
<a href="https://pjreddie.com/darknet/yolo/" class="intext">YOLO</a>
works, how to implement YOLO in <a href="https://pytorch.org/" class="intext">PyTorch</a>,
or how to train a deep learning model.
Nowadays, there are plenty of high-quality resources online which you can refer to.
</p>
<p class="p_on_project_pages">
A disclosure: I never had proper training in web dev, not even an online course.
I had an overall idea of how this project should work eventually
and just did some googling here and there.
</p>
<p class="p_on_project_pages">
Despite this post describing a pipeline for an object detector, the approach can be adapted for other
applications where you need to send user data from, e.g.,
<a href="https://pages.github.com/" class="intext">
GitHub Pages</a>
(front-end), to the server, process it there (back-end), and send a result back to the user.
</p>
<p class="p_on_project_pages">
I developed this project in 2019, ever since I saw a couple of other wonderful solutions with similar
functionality.
Namely,
<a href="https://modeldepot.github.io/tfjs-yolo-tiny-demo/" class="intext">
in-browser detection with YOLO tiny</a> written in
<a href="https://www.tensorflow.org/js" class="intext"> TF.js </a>
that runs without a back-end.
Also, there is a cool
<a href="https://apps.apple.com/us/app/idetection/id1452689527" class="intext"> iPhone app, iDetection</a>
from well-known
<a href="https://ultralytics.com/" class="intext">Ultralytics</a>.
</p>
<p class="p_on_project_pages">
<i>Why didn't you go with front-end only (TF.js)?</i>
Well, mostly because either a user would need to download weights (~250MB) before
uploading an image or you have to reduce model capacity.
However, downloading ~250MB could be prohibitively slow in some regions on Earth while using a smaller model
also worsens the performance.
Therefore, I think running a model on a back-end delivers a better user experience.
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
Here is a plan for this post as well as the outline of the whole project pipeline:
</p>
<div id="toc_container">
<ul>
<li class="toc_section" style="padding-top: 0em;"><a href="#1-front-end" class="intext">1. Front-end</a></li>
<li class="toc_section"><a href="#2-back-end" class="intext">2. Back-end</a>
<ul>
<li class="toc_subsection"><a href="#flask-app-code" class="intext">Flask App Code</a></li>
<li class="toc_subsection"><a href="#instance-configuration" class="intext">Instance Configuration</a>
</li>
<li class="toc_subsection"><a href="#instance-setup-instructions" class="intext">Instance Setup
Instructions</a></li>
</ul>
</li>
<li class="toc_section"><a href="#3-assigning-a-domain-name-to-the-instance" class="intext">3. Assigning a
Domain Name to the Instance</a>
<ul>
<li class="toc_subsection"><a href="#getting-a-domain-name" class="intext">Getting a Domain Name</a></li>
<li class="toc_subsection"><a href="#registering-the-domain-name-in-a-dns" class="intext">Registering the
Domain Name in a DNS</a></li>
<li class="toc_subsection"><a href="#securing-our-connection-with-an-ssl-certificate"
class="intext">Securing our Connection with an SSL
Certificate</a></li>
</ul>
</li>
<li class="toc_section"><a href="#4-running-the-detector" class="intext">4. Running the Detector</a></li>
</ul>
</div>
</div>
</div>
<div class="our_framework">
<div id="1-front-end" class="section_name">
1. Front-end
</div>
<div class="section_content">
<p class="p_on_project_pages">
This part answers the question: how to send an uploaded image to a cloud server and process the
server response.
</p>
<p class="p_on_project_pages">
The Object Detector project is a part of my webpage
(<a href="https://v-iashin.github.io/" class="intext">v-iashin.github.io</a>)
which is hosted on GitHub Pages and maintained in
<a href="https://github.com/v-iashin/v-iashin.github.io" class="intext">v-iashin/v-iashin.github.io</a>
repository.
The way how everything looks is defined in
<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/detector.html" class="intext">
v-iashin/v-iashin.github.io/detector.html</a>
and this is how it looks on my screen:
</p>
<div class="img background_white_squared" style="border: none; padding: 0em;">
<img src="./images/detector/detector_waiting.png" alt="Detector Start Screen">
</div>
<p class="p_on_project_pages no_bottom_pad">
The process starts with
<span style="border: 1.5px solid lightgray; font-size: 0.7em; padding: 0.3em;">Upload Your Image</span>
button that allows a user to upload an image which is defined in these lines
(<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/detector.html#L67-L68"
class="intext">v-iashin.github.io/detector.html#L67-L68</a>):
</p>
<pre><code class="language-html"><label id="upload" for="file-input" class="btn upload"> Upload Your Image </label>
<input id="file-input" type="file" accept="image/*"></code></pre>
<p class="p_on_project_pages">
Next, we need to read the input and prepare it for sending.
This is done with <code class="javascript inline">javascript</code> and the code is located in
<a href="https://github.com/v-iashin/v-iashin.github.io/tree/af15b1421/js" class="intext">
v-iashin/v-iashin.github.io/js</a>.
</p>
<p class="p_on_project_pages">
Note, <code class="javascript inline">javascript</code> does not wait until the previous line finishes
executing before starting executing a new line as it is in e.g.
<code class="javascript inline">Python</code>.
So, get ready for some nesting and triggering.
</p>
<p class="p_on_project_pages no_bottom_pad">
When a user uploads an image, a selector <code class="javascript inline">file-input</code> is assigned with the
user image.
We, then, can select this object and assign it to the variable <code class="javascript inline">upload</code>
which can be used in <code class="javascript inline">javascript</code>
(<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/upload_handler.js#L7"
class="intext">v-iashin.github.io/js/upload_handler.js#7</a>):
</p>
<pre><code class="javascript">upload = document.querySelector('#file-input');</code></pre>
<p class="p_on_project_pages no_bottom_pad">
Once, the <code class="javascript inline">upload</code> variable is changed, it triggers the following function
(<a
href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/upload_handler.js#L77-L94"
class="intext">v-iashin.github.io/js/upload_handler.js#77-L94</a>):
</p>
<pre><code class="javascript">upload.addEventListener('change', function() {
event.preventDefault();
// clean the previous result
preview.innerHTML = '';
// start file reader
var reader = new FileReader();
reader.onload = function(event) {
if(event.target.result) {
// resize the image, send to the server, and show to a user
img.onload = onload_func;
// evokes the function above ('onload to src attr')
img.src = event.target.result;
};
};
reader.readAsDataURL(event.target.files[0]);
});</code></pre>
<p class="p_on_project_pages">
Inside, it creates a file variable <code class="javascript inline">reader</code> and defines a function
(<code class="javascript inline">reader.onload = function(event) {</code>) which is going to be triggered when
<code class="javascript inline">reader</code> variable is changed by <code
class="javascript inline">reader.readAsDataURL(event.target.files[0])</code>.
The <code class="javascript inline">event.target.files[0]</code> variable holds the user image.
Similarly, inside of <code class="javascript inline">reader.onload</code> function, we define a function to be
triggered (<code class="javascript inline">img.onload = onload_func</code>) when <code
class="javascript inline">img</code> gets input
(<code class="javascript inline">img.src = event.target.result</code>).
</p>
<p class="p_on_project_pages no_bottom_pad">
The <code class="javascript inline">onload_func</code> is defined as follows in
<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/upload_handler.js#L21-L40"
class="intext">
v-iashin.github.io/js/upload_handler.js#L21-L40</a>:
</p>
<pre><code class="javascript">function onload_func() {
// extracting the orientation info from EXIF which will be sent to the server
EXIF.getData(img, function () {
orientation = EXIF.getTag(this, 'Orientation');
});
// resize the sides of the canvas and draw the resized image
// `var MAX_SIDE_LEN = 1280` – defined above
[canvas.width, canvas.height] = reduceSize(img.width, img.height, MAX_SIDE_LEN);
context.drawImage(img, 0, 0, canvas.width, canvas.height);
// adds the image that the canvas holds to the source
resized_img.src = canvas.toDataURL('image/jpeg');
// clean the result before doing anything
preview.innerHTML = '';
// append new image
preview.appendChild(resized_img);
// hides the text with examples
examples_text.classList.remove('examples_text');
examples_text.classList.add('hide');
// send the user image on server and wait for response, and, then, shows the result
send_detect_show();
}</code></pre>
<p class="p_on_project_pages">
First, it tries to guess the image orientation from EXIF which prevents processing rotated images when
uploaded from a phone camera.
Also, the user images can be very large. So, to avoid errors due to lack of RAM, the front-end resizes
(<code class="javascript inline">reduceSize</code>) each image
such that <code class="javascript inline">max(H, W) = MAX_SIDE_LEN</code> before sending to the server.
The uploaded image is also drawn for a user to see (<code class="javascript inline">preview</code>).
We also hide the text with examples (<code class="javascript inline">examples_text</code>) and the upload
button at this step.
Finally, the <code class="javascript inline">send_detect_show()</code> function is called that sends the image
for detection to the server.
</p>
<p class="p_on_project_pages no_bottom_pad">
The <code class="javascript inline">send_detect_show()</code> function is defined as follows
(<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/upload_handler.js#L97-L135"
class="intext">v-iashin.github.io/js/upload_handler.js#L97-L135</a>):
</p>
<pre><code class="javascript">// `var SERVER_URL = 'https://iashin.ml:5000/'` – defined above
function send_detect_show() {
// remove the upload button
var element = document.getElementById('upload');
element.parentNode.removeChild(element);
// show the detect (progress) button
detect.classList.remove('hide');
// make the button unresponsive
detect.classList.add('progress');
// shows the status notification
detect.innerHTML = 'Processing...';
// form a blob from data uri
var blob = dataURItoBlob(preview.firstElementChild.src);
// form a POST request to the server
var form_data = new FormData();
form_data.append('file', blob);
form_data.append('orientation', orientation);
$.ajax({
type: 'POST',
url: SERVER_URL,
data: form_data,
timeout: 1000 * 25, // ms, to wait until .fail function is called
contentType: false,
processData: false,
dataType: 'json',
}).done(function (data, textStatus, jqXHR) {
// replace the current image with an image with detected objects
preview.firstElementChild.src = data['image'];
// remove the detect button
detect.parentNode.removeChild(detect);
// and show the reload button
rld.classList.remove('hide');
}).fail(function (data) {
alert("Wow! That's weird. It seems it didn't work for you, but it had to. Please let me know about this odd situation on vdyashin@gmail.com or in Issues on GitHub. Or reload the page and try again.");
// remove the detect button
detect.parentNode.removeChild(detect);
// and show the reload button
rld.classList.remove('hide');
});</code></pre>
<p class="p_on_project_pages">
It removes the <code class="javascript inline">upload</code> button which might confuse the user and shows the
text field saying
"Processing..." (the variable <code class="javascript inline">detect</code>).
<i>
Previously, this button was clickable such that the user could inspect what they uploaded before
sending it to the server (by clicking <span style="border: 1.5px solid lightgray; font-size: 0.7em; padding: 0.3em;">Detect</span>).
Hence the name of the variable.
However, I decided to upload the image once the user selects it on their machine providing a more snappy
experience.
</i>
</p>
<p class="p_on_project_pages">
We are going to use the <code class="javascript inline">FormData()</code> structure to form a <code
class="javascript inline">POST</code> request with user input.
The user image is encoded in <code class="javascript inline">base64</code> but <code
class="javascript inline">FormData</code> expects the data to be a <code
class="javascript inline">blob</code>.
Therefore, convert it to <code class="javascript inline">blob</code> in <code
class="javascript inline">dataURItoBlob()</code>.
Both the <code class="javascript inline">blob</code> and the orientation info are appended to the <code
class="javascript inline">form_data</code> variable.
</p>
<p class="p_on_project_pages">
The <code class="javascript inline">POST</code> request is sent using the <code
class="javascript inline">ajax</code> technique.
The syntax is fairly simple.
First, it forms the request and sends it to the server <code class="javascript inline">URL</code>, and waits for
<code class="javascript inline">timeout</code> (in ms) for
a response.
On success, it runs a function in <code class="javascript inline">.done</code> while on failing it runs the
function under <code class="javascript inline">.fail</code>.
In my case, the server sends back the results which are assigned to the <code
class="javascript inline">data</code> variable.
We extract the image with predictions and assign it to the <code class="javascript inline">preview</code>
variable which replaces the uploaded image by the image with detection results.
Finally, the button prompts the user to reload the page is displayed (<code
class="javascript inline">rld</code>).
On fail, the user gets a pop-up asking to fill an issue or to contact me.
That is all.
</p>
<p class="p_on_project_pages">
I also decided to add an indicator that checks if the app is responsive at a glance without submitting
an image for detection.
This is what you see at the bottom of the page:
</p>
<div class="to_col_if_narrow" style="display: flex; text-align: center;">
<div class="img background_white_squared" style="border: none; padding: 0.5em; flex: 1; margin-bottom: 0em;">
<img class="shadow if_narrow_add_vertical_margin" src="./images/detector/status_bar_on.png" alt="Status Bar: on">
</div>
<div class="img background_white_squared" style="border: none; padding: 0.5em; flex: 1;">
<img class="shadow if_narrow_add_vertical_margin" src="./images/detector/status_bar_off.png" alt="Status Bar: off">
</div>
</div>
<p class="p_on_project_pages no_bottom_pad">
In <code class="javascript inline">HTML</code> it is defined as a footer and located in
(<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/detector.html#L78-L80"
class="intext">v-iashin.github.io/detector.html</a>):
</p>
<pre><code class="html"><footer class="version">
Detection Project, 2019 (<span id="status"></span>)
</footer></code></pre>
<p class="p_on_project_pages no_bottom_pad">
The code that handles the check is in
(<a href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/status_checker.js"
class="intext">v-iashin.github.io/js/status_checker.js</a>)
and the <code class="javascript inline">url</code> is specified with the port that the <code
class="javascript inline">Flask</code> app is using:
</p>
<pre><code class="javascript">// url to server with flask running
var STATUS_CHECK_URL = 'https://iashin.ml:5000/status_check';
// by default it is down
document.getElementById('status').innerHTML = "<span id='status_down' alt='down'>offline</span>";
$.ajax({
//your server url
url: STATUS_CHECK_URL,
type: 'GET',
success: function() {
document.getElementById('status').innerHTML = "<span id="status_ok">online</span>";
},
error: function() {
document.getElementById('status').innerHTML = "<span id="status_down" alt="down">offline</span>";
}
});</code></pre>
Similar to sending <code class="javascript inline">POST</code> requests, we form a <code
class="javascript inline">GET</code> request and use
<code class="javascript inline">ajax</code> to send it and get the feedback.
On success, it will change the footer to <code class="javascript inline">online</code> and to <code
class="javascript inline">offline</code> on a failure.
</div>
</div>
<div class="our_framework">
<div id="2-back-end" class="section_name">
2. Back-end
</div>
<div class="section_content">
<p class="p_on_project_pages" style="padding-bottom: 0em;">
The part answers the question: how to process the <code class="javascript inline">POST</code> request received
from the
front-end on a Linux machine (server) and send back the results.
We are going to use the
<a class="intext" href="https://flask.palletsprojects.com/en/2.0.x/">Flask framework</a>
to achieve this.
</p>
<div id="flask-app-code" class="subsection_name">
Flask App Code
</div>
<p class="p_on_project_pages">
Essentially, you need to tell the <code class="javascript inline">Flask</code> app at which endpoint (<code
class="javascript inline">your.domain/something</code>) and what
kind of requests
(<code class="javascript inline">GET</code>, <code class="javascript inline">POST</code>) you want to handle.
I use <code class="javascript inline">/</code> (root) for the incoming user input (<code
class="javascript inline">POST</code>) and <code class="javascript inline">/status_check</code> to quickly
check if the app is responsive without sending anything (<code class="javascript inline">GET</code>).
In this project, the code for the <code class="javascript inline">Flask</code> app is located in
<a class="intext"
href="https://github.com/v-iashin/WebsiteYOLO/blob/master/main.py"> v-iashin/WebsiteYOLO/main.py</a>.
</p>
<p class="p_on_project_pages no_bottom_pad">
Let's start with <code class="javascript inline">/status_check</code>:
</p>
<pre><code class="python">@app.route('/status_check', methods=['GET'])
def status_check():
if request.method == 'GET':
return 'GET request received'</code></pre>
<p class="p_on_project_pages">
As you see, this one is pretty simple.
All we need is to use a decorator specifying the endpoint <code
class="javascript inline">[your.domain]/status_check</code> and methods it will be
handling at this endpoint – only <code class="javascript inline">GET</code>.
Also note, we access the content of the request in <code class="javascript inline">request</code> variable
assigned globally on import above.
</p>
<p class="p_on_project_pages no_bottom_pad">
Now, let's consider the root <code class="javascript inline">[your.domain]/</code> endpoint function which
handles user inputs
(<code class="javascript inline">POST</code> requests):
</p>
<pre><code class="python">@app.route('/', methods=['POST'])
def upload_file():
# access files in the request. See the line: 'form_data.append('file', blob);'
files = request.files['file']
# save the image ('file') to the disk
files.save(INPUT_PATH)
############# RUNNING DETECTOR ###############
try:
orientation = request.form['orientation']
print(f'Submitted orientation: {orientation}')
except:
orientation = 'undefined'
print(vars(request))
# run the predictions on the saved image
show_image_w_bboxes_for_server(
INPUT_PATH, OUTPUT_PATH, ARCHIVE_PATH, LABELS_PATH, FONT_PATH, MODEL, DEVICE, orientation
)
################################################
# 'show_image_w_bboxes_for_server' saved the output image to the OUTPUT_PATH
# now we would like to make a byte-file from the save image and sent
# it back to the user
with open(OUTPUT_PATH, 'rb') as in_f:
# so we read an image and decode it into utf-8 string and append it
# to data:image/jpeg;base64 and then return it.
img_b64 = b64encode(in_f.read()).decode('utf-8')
img_b64 = 'data:image/jpeg;base64, ' + img_b64
return jsonify(name='input.jpg', image=str(img_b64))</code></pre>
<p class="p_on_project_pages">
It starts by accessing the user data (an image) from the <code class="javascript inline">request</code> variable
The content of the <code class="javascript inline">request</code> variable is formed by the
<a class="intext"
href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/upload_handler.js#L110-L112">
front-end</a>.
We save the image in the <code class="javascript inline">.jpg</code> format at <code
class="javascript inline">INPUT_PATH</code> for processing in our detector.
Next, we run the detection function (<code class="javascript inline">show_image_w_bboxes_for_server()</code>)
which reads the saved input image,
detects objects, and saves a new image with drawn bounding boxes (<code
class="javascript inline">OUTPUT_PATH</code>).
Finally, we read the image with results, represent it in <code class="javascript inline">base64</code> format (a
string), and return it in a
<code class="javascript inline">JSON</code>.
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
This <code class="javascript inline">JSON</code> is, then, read by the
<a class="intext"
href="https://github.com/v-iashin/v-iashin.github.io/blob/af15b1421/js/upload_handler.js#L121-L123">front-end</a>
by accessing the <code class="javascript inline">image</code> field.
</p>
<div id="instance-configuration" class="subsection_name">
Instance Configuration
</div>
<p class="p_on_project_pages">
To rent a server, you can use any instance provider out there, e.g. <a class="intext" href="https://cloud.google.com/">Google
Cloud</a>, <a class="intext" href="https://www.heroku.com/">Heroku</a>, or <a class="intext" href="https://aws.amazon.com/">AWS</a>.
</p>
<p class="p_on_project_pages">
<em>Hint: Google Cloud will give you free credits when you register with a bank card (repeat with another card
when expires <span style="font-style: normal">😉</span>) and AWS credits can be obtained with
<a class="intext" href="https://education.github.com/pack">the GitHub student pack</a>
(one for each degree <span style="font-style: normal">😉</span>).
</em>
</p>
<p class="p_on_project_pages">
There are plenty of resources online on how to rent and set up an instance.
I recommend you to try setting it up if you have never done it before as it is a good exercise.
For example, try to rent one and run a Jupyter Notebook such that you could access it, e.g. from your cell phone
using
the IP of an instance.
</p>
<p class="p_on_project_pages">
My instance has 4 vCPUs, 5GB RAM, 40 GB of disk, and running Ubuntu 20.04.
I found this configuration to be a good money-performance trade-off.
You will need at least 4GB for both the OS and the detector while the CPU count and disk space can be reduced.
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
When renting an instance make sure to allow ports you are going to use by e.g. <code
class="javascript inline">Jupyter</code> (8080),
<code class="javascript inline">Flask</code> (5000), and, of course,
<code class="javascript inline">ssh</code> (22), and <code class="javascript inline">HTTPS</code> (443).
Also, reserve an IP and make it stable such that it will not be changed after an instance reboot since by
default they
are ephemeral.
</p>
<div id="instance-setup-instructions" class="subsection_name">
Instance Setup Instructions
</div>
<p class="p_on_project_pages no_bottom_pad">
Necessary OS libraries:
</p>
<pre><code class="bash">sudo apt update
sudo apt -y upgrade
sudo apt install -y git tmux wget curl
# stuff for OpenCV
sudo apt install -y libsm6 libxext6 libxrender-dev</code></pre>
<p class="p_on_project_pages no_bottom_pad">
A <code class="javascript inline">Python</code> environment with <code class="javascript inline">conda</code>:
</p>
<pre><code class="bash">wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O ~/miniconda.sh
bash ~/miniconda.sh -b -p $HOME/miniconda3
source $HOME/miniconda3/bin/activate
conda deactivate
conda init
# prevents conda from activating `base` environment when shell starts
conda config --set auto_activate_base false</code></pre>
<p class="p_on_project_pages no_bottom_pad">
Now when <code class="javascript inline">conda</code> is installed, we can build an environment for the project
and download
the detector weights
</p>
<pre><code class="bash">git clone https://github.com/v-iashin/WebsiteYOLO.git
conda env create -f $HOME/WebsiteYOLO/conda_env.yml
cd WebsiteYOLO
bash ./weights/download_weights_yolov3.sh
cd ../</code></pre>
<p class="p_on_project_pages" style="padding-top: 2em; text-align: center; color: brown;">
<b>DO NOT FORGET TO STOP YOUR INSTANCE IF YOU ARE NOT RUNNING ANYTHING.</b>
</p>
</div>
</div>
<div class="our_framework">
<div id="3-assigning-a-domain-name-to-the-instance" class="section_name">
3. Assigning a Domain Name to the Instance
</div>
<div class="section_content">
<p class="p_on_project_pages" style="padding-bottom: 0em;">
When renting an instance, we got an IP address.
We cannot, however, use the bare IP to send user requests from GitHub Pages.
The reason behind it is that GitHub bans uncertified
<a class="intext" href="https://en.wikipedia.org/wiki/Cross-origin_resource_sharing"><em>Cross-Origin Resource Sharing</em></a>
(CORS)
which happens when we try to send some possibly sensitive user info uploaded on a website with the GitHub domain
to
somewhere else.
Therefore, the connection to that domain must be secured via <code class="javascript inline">HTTPS</code>.
For this reason, we will need a domain name and a DNS provider to map the IP to the domain name.
</p>
<div id="getting-a-domain-name" class="subsection_name">
Getting a Domain Name
</div>
<p class="p_on_project_pages">
To own a domain, you need to pay $$ yearly to a registrar (e.g. <a class="intext" href="https://www.godaddy.com/">godaddy</a>
or <a class="intext" href="https://www.namecheap.com/">namecheap</a>).
For the sake of this tutorial, we will rent a "free" domain from <a class="intext" href="https://www.freenom.com/">Freenom</a>.
</p>
<p class="p_on_project_pages">
<em>However, I advise you not to rely on Freenom by any means, especially your trust and money. You can read
reviews on
HackerNews or Reddit. E.g. <a class="intext" href="https://daniel.is-a.dev/blog/freenom-the-free-domains-website-is-a-scam-3">this
one</a>. TLDR: they will remove the domain from your account and put it on sale once it will get some
traffic.</em>
</p>
<p class="p_on_project_pages">
To register a domain on Freenom, just open the website, register, and select a domain name.
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
Freenom provides domains for a few months, then you have to renew it for up to 12 months during the last
2 weeks before the expiry date.
Freenom should send you an email closer to that period.
The renewal is also free.
</p>
<div id="registering-the-domain-name-in-a-dns" class="subsection_name">
Registering the Domain Name in a DNS
</div>
<p class="p_on_project_pages">
Next, we need to associate the instance IP address with the domain name.
This is the task of a <a class="intext" href="https://en.wikipedia.org/wiki/Domain_Name_System">Domain Name System</a> (DNS).
Again, you could rent it from AWS or Google Cloud for a small amount of money but for now, I will show you how to
do it
using Freenom for free.
</p>
<p class="p_on_project_pages">
<em>However, again, I advise you not to rely on Freenom by any means, especially your trust and money. You can
read
reviews online.</em>
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
In Freenom, go to <code class="javascript inline">My Domains</code> > <code
class="javascript inline">Domain</code> > <code class="javascript inline">Freenom DNS</code>.
There you can add DNS records:
</p>
<p class="p_on_project_pages">
<table>
<thead>
<tr>
<th style="text-align:right">Name</th>
<th style="text-align:right">Type</th>
<th style="text-align:right">TTL</th>
<th style="text-align:right">Target</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right">WWW</td>
<td style="text-align:right">A</td>
<td style="text-align:right">300</td>
<td style="text-align:right">your.instance.ip.addr</td>
</tr>
<tr>
<td style="text-align:right"></td>
<td style="text-align:right">A</td>
<td style="text-align:right">300</td>
<td style="text-align:right">your.instance.ip.addr</td>
</tr>
</tbody>
</table>
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
Save everything and wait for a bit as DNS needs some time to register the mapping.
Freenom page should look similar to this:
<div class="img background_white_squared" style="border: none; padding: 0em;">
<img src="./images/detector/freenom_dns.png" alt="Freenom DNS Page Example">
</div>
</p>
<p class="p_on_project_pages">
Here I usually go to a website that checks DNS entries, e.g. <a class="intext"
href="https://dnschecker.org/all-dns-records-of-domain.php">dnschecker.org</a>, and wait until it appears
there.
</p>
<details style="border: 2px solid black; padding: 0.5em;">
<summary><i>I Want to Use Google Cloud DNS with a Freenom Domain</i></summary>
<p class="p_on_project_pages" style="padding: 2em 0em 0em 2em; font-size: 80%;">
1. Add a zone in your: <code class="javascript inline">Google Cloud Console</code> > <code
class="javascript inline">Cloud DNS</code> and register a new entry. You should be able
to add records there and see the nameservers like so:
<div class="img background_white_squared" style="border: none; padding: 0em 0em 0em 2em; margin-bottom: 0em;">
<img src="./images/detector/gcloud_dns.png" alt="Google DNS Page Example">
</div>
</p>
<p class="p_on_project_pages" style="padding: 0em 0em 0em 2em; font-size: 80%;">
2. Go to Freenom:
<code class="javascript inline">My domains</code> > <code class="javascript inline">Manage Domain</code> > <code
class="javascript inline">Management Tools</code> > <code class="javascript inline">Nameservers</code>
> <code class="javascript inline"> Custom nameservers</code> add
these nameservers to the corresponding rows.
</p>
<p class="p_on_project_pages" style="padding: 2em 0em 2em 2em; font-size: 80%;">
3. Similarly, you can check the presence of DNS entries on <a class="intext"
href="https://dnschecker.org/all-dns-records-of-domain.php">dnschecker.org</a>.
</p>
</details>
<div id="securing-our-connection-with-an-ssl-certificate" class="subsection_name">
Securing our Connection with an SSL Certificate
</div>
<p class="p_on_project_pages">
The final step here is to obtain an SSL certificate for HTTPS. Essentially, we just need the 🔒 near the URL
field in a browser.
</p>
<p class="p_on_project_pages no_bottom_pad">
Install <code class="javascript inline">certbot</code> on your instance:
</p>
<pre><code class="bash">sudo apt install certbot</code></pre>
<p class="p_on_project_pages no_bottom_pad">
and initiate a ACME challenge that will allow the certifying third-party to proof that you own
both the domain and the IP address:
</p>
<pre><code class="bash">sudo certbot certonly --manual --preferred-challenges dns</code></pre>
<p class="p_on_project_pages">
It will ask you some questions (email etc).
Enter your domain name, e.g. <code class="javascript inline">john_smith.tk</code>, and answer
<code class="javascript inline">Yes</code> to the public logging question.
Next, it will ask you to add an ACME challenge token to your DNS:
<code class="javascript inline">_acme-challenge.your.domain 6wMpfi5ZXG0rbJt7_H2qFtT9_YUVJY_5VzEtbsJnD8</code>.
Don't press <code class="javascript inline">Enter</code> yet, go to DNS provider page and add a <code
class="javascript inline">TXT</code> entry:
<table>
<thead>
<tr>
<th style="text-align:right">Name</th>
<th style="text-align:right">Type</th>
<th style="text-align:right">TTL</th>
<th style="text-align:right">Target</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:right">_acme-challenge</td>
<td style="text-align:right">TXT</td>
<td style="text-align:right">300</td>
<td style="text-align:right">6wMpfi5Z...tbsJnD8</td>
</tr>
</tbody>
</table>
</p>
<p class="p_on_project_pages" style="padding-bottom: 0;">
See screenshots <a class="intext" href="#registering-the-domain-name-in-a-dns">above</a> as your guidance.
Similarly, you can check the presence of the DNS entry in <a class="intext"
href="https://dnschecker.org/all-dns-records-of-domain.php">dnschecker.org</a> (use
<code class="javascript inline">_acme-challenge.your.domain</code> instead of just <code
class="javascript inline">your.domain</code> there).
Once, it appears there, press <code class="javascript inline">Enter</code> in <code
class="javascript inline">certbot</code>.
It should congratulate you and say where the certificates are saved.
Here is how it all looks in my case:
</p>
<div class="img background_white_squared" style="border: none; padding: 0em; margin-bottom: 0em;">
<img src="./images/detector/certbot_challenge.png" alt="The screenshot of a successful acme challenge">
</div>
<p class="p_on_project_pages no_bottom_pad">
By default, the keys are restricted to a root user only.
You can loosen the permissions with
</p>
<pre><code class="bash">sudo chmod -R 755 /etc/letsencrypt</code></pre>
<p class="p_on_project_pages">
to allow programs (Jupyter, Flask) run by a user to have an access to keys, otherwise you will have to run
these libraries under <code class="javascript inline">root</code>.
</p>
<p class="p_on_project_pages">
The certificate is valid for 3 months, then you need to <strong>renew</strong> it.
<a class="intext" href="https://letsencrypt.org/">Letsencrypt</a> (<code class="javascript inline">certbot</code>) will send an
email to the address you specified when registering your first certificate.
The renewal procedure is the same: running the <code class="javascript inline">certbot</code> command and
passing the
<code class="javascript inline">ACME</code> challenge (see above).
</p>
</div>
</div>
<div class="our_framework">
<div id="4-running-the-detector" class="section_name">
4. Running the Detector
</div>
<div class="section_content">
<p class="p_on_project_pages no_bottom_pad">
At this point, I hope, you have an instance running with the environment as well as a registered domain name
that is
mapped to the IP of the instance by a DNS.
If so, the next step is pretty simple: instantiate a <code class="javascript inline">tmux</code> session and
start the <code class="javascript inline">Flask</code> app from
there as such
</p>
<pre><code class="bash">conda activate detector
export FLASK_APP=./WebsiteYOLO/main.py
export FLASK_RUN_CERT=/etc/letsencrypt/live/your.domain/fullchain.pem
export FLASK_RUN_KEY=/etc/letsencrypt/live/your.domain/privkey.pem
flask run --host=0.0.0.0</code></pre>
<p class="p_on_project_pages">
If <code class="javascript inline">flask run</code> fails and tells you that you don't have <code
class="javascript inline">fullchain.pem</code> or
<code class="javascript inline">privkey.pem</code>, most likely you need to change the ownership of these files
to your user.
However, I also found that different versions of the environment packages
may cause this error but I didn't have an opportunity to inspect which ones and why.
Alternatively, you can <code class="javascript inline">run flask</code> under <code
class="javascript inline">root</code>.
</p>
<p class="p_on_project_pages">
If no errors occur, go to <code class="javascript inline">your.instance.ip.address:5000/status_check</code>
– your browser should print <code class="javascript inline">GET request received</code> and <code
class="javascript inline">flask</code> will print this event to the
terminal.
</p>
<p class="p_on_project_pages">
If you cannot see <code class="javascript inline">GET request received</code>, you need to check if port <code
class="javascript inline">5000</code> is opened in the
network settings of your instance.
</p>
<p class="p_on_project_pages">
Repeat the same with your domain name instead of the IP:
<code class="javascript inline">https://your.domain:5000/status_check</code>
</p>
<p class="p_on_project_pages">
Next, if we go back to front-end and try to upload the image and send a <code
class="javascript inline">POST</code> request.
</p>
<p class="p_on_project_pages" style="padding-bottom: 0em;">
On success, your instance terminal should output something like this for both
<code class="javascript inline">GET</code> and <code class="javascript inline">POST</code>
requests:
</p>
<div class="img background_white_squared" style="border: none; padding: 0em; margin-bottom: 0em;">
<img src="./images/detector/flask_terminal.png" alt="Running Flask App in the Terminal">
</div>
<p class="p_on_project_pages">
If you are seeing this, I think, you succeeded. Well done!
</p>
</div>
</div>
<div class="our_framework">
<div id="final-remarks" class="section_name">
Final Remarks
</div>
<div class="section_content">
<p class="p_on_project_pages">
This guide is a bit sparse and might not contain every small detail.
If something is not clear, feel free to let me know by <a class="intext" href="mailto:vladimir.d.iashin@gmail.com">email</a> or
form an
issue in the <a class="intext" href="https://github.com/v-iashin/v-iashin.github.io/issues">v-iashin/v-iashin.github.io</a> repository.
Also, if you found this useful and managed to build your project on top of it, send me a link I would be happy
to check
out.
</p>
<hr>
I thank Anna Iashina (<a class="intext" href="https://www.linkedin.com/in/anna-iashina/">LinkedIn</a>)
for proofreading the final draft.
</div>
</div>
</body>
<script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.1.0/highlight.min.js"></script>
<script>hljs.highlightAll();</script>
<script>
document.querySelectorAll("code").forEach(function (element) {
element.innerHTML = element.innerHTML.replace(/&/g, "&").replace(/</g, "<").replace(/>/g, ">").replace(/"/g, """).replace(/'/g, "'");
});
$(document).ready(function () {
$('code').each(function (i, block) {
hljs.highlightBlock(block);
});
});
</script>
</html>