Skip to content

NCNU-OpenSource/LSA_tomato

BranchesTags

Repository files navigation

LSA Final Project － 「一天一番茄，考試100分」

Concept Development

  • Do you have these kinds of trouble? When the exam is coming, but still can not concentrate on your studies? Wasting your time on looking at your phone？
  • We hope by develope「一天一番茄，考試100分」to supervise the status of studies and improve efficiency (reduce interference and distraction).

Function

  • 「一天一番茄，考試100分」monitors whether the user is using the mobile phone by identifying the mobile phone, and if so, system will alarm the user.

  • The system will detect gesture or motion than respond accordingly.

    • At system start, the user can select one of three timekeeping modes. Use gestures 1, 2 or 3 to select.
    • If user leave the camera (camera cannot detect the user), a warning message will appear.
    • If the sysytem detects the gesture of 『ok』, it will play relax music.

    • If you want to stop the music by detects the gesture of 『5』.

    • If the sysytem detects the gesture of 『2』, it will take a foto.

  • When the system ends, it will send a mail to the mailbox showing the usage status

  • tab q to ending the function

Implementation Resources

  • Software:
    • raspberry pi OS
    • Unbutu
  • Hardware：
序號 設備名稱 數量 來源
1 raspberry pi 3 1 MOLi
2 pi camera 1 MOLi
3 海螺揚聲器（任何揚聲器都可 1 助教
4 Screen（monitor） 1
5 adhesive tape 1
6 Laptop 1

Implementation Process

樹莓派環境

  1. 將 SD 卡格式化，與 raspberry pi 連接

  2. 架設 Raspberry pi + pi camera + 揚聲器 模組

    • pi cermera

    • raspberry pi 3

    • 將pi camera 連接 raspberry pi 3

    • 揚聲器

    • 將轉接頭插入揚聲器

  3. 建構環境：

  • Problem：
    1. 一開始希望使用 Tensorflow 訓練偵測手機模型，但後來因能力不足，加上有找到可以替代的模組，改而使用模組
    2. 如果使用 pi camera 的話樹莓派只能使用 32 bits
    3. 一開始我們原本安裝 conda 以便快速安裝 opencv (只需 20 分鐘)，但後來發現環境無法通用且我們需要的模組（dnn.detection model）只有 4.1 以上的 opencv 才可以支援，但 conda 中的 opencv 只有 3.6.7
      • 嘗試升級 conda 中的 opencv 版本但以失敗告終。
    4. 官方下載 mediapipe 的封包無法支援 32 bits，後來經由非官方資源才成功裝載支援 32 bits 的 mediapipe
    5. 將SD卡格式化，重新下載環境，直接載完整的 opencv，但途中也遇到問題：樹莓派耗損跑不動（措施：換新的樹莓派），且最終也仍然無法成功安裝。
    6. 嘗試依照學長建議直接安裝 cv2，有安裝成功但發現裡面沒有我們所需的模組，因此也失敗。
  1. 樹莓派
    • 下載 mediapipe：請參考以下網址
      • 因為我們需要使用pi camera，所以只能將raspberry pi 使用 32 bit，但目前大多數 mediapipe 都只能支援 64 bit，後來找到以下網頁內的資訊，才有辦法安裝 mediapipe。
      • 下載 mediapipe 網頁
    • 下載 cv2：請參考以下網址

樹莓派最終結果：

  • 因在樹莓派上無法下載 opencv 因此無法將環境建構完成，所以改用虛擬機建構及執行，方法請參考下方步驟

Linux

  1. Install Ubuntu：Please refer to the following website

  2. Setting the environment：

    • install cmake: sudo apt install cmake
    • install git：sudo apt install git
    • install pip:sudo apt install pip
    • install mediapipe: Please refer to the following website
    • install opencv：Please refer to the following website
      • The website of install opencv :::info
        1. git clone https://github.com/opencv/opencv.git
        2. git clone https://github.com/opencv/opencv_contrib.git
        3. cd ~/opencv
        4. mkdir build
        5. cd build
        6. cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local ..
        7. make -j3 # 看你開幾顆CPU
        8. sudo make install
        9. git clone https://github.com/opencv/opencv_extra.git :::
    • install pygame：sudo apt install python-pygame
    • pip install pygame
    • pip install opencv-python
    • install ubuntu package（for camera）：
    • check your camera

Usage

  1. Clone the project on GitHub：git clone https://github.com/wyping314/LSA_project.git
  2. cd LSA_project
  3. vim tomato.py
    1. Modify the path to store pictures:save_path
    2. Modify the recipient msg['To'] which in def mail()
    3. Modify two parameters smtp.login() which in def mail()
    4. Adjust the parameters of the lens VideoCapture() (total of three places) according to your equipment
  4. execute program python3 mainproject.py
  5. The system will pop up the camera window
  6. Read how to use & Choose mode
  7. Pomodoro Technique start
  8. System start detecting the user

Code

番茄鐘回傳剩餘時間

  • 選擇模式
start = False
if start == False:
    if text == '1':
        # test mode
        working_time = 0.1
        break_time = 0.1
        start = True
        a = tomato(working_time, 'Go working')
    elif text == '2':
        working_time = 10       # 10 minute
        break_time = 5          # 5 minute
        start = True
        a = tomato(working_time, 'Go working')
    elif text == 'ok':
        working_time = 25       # 25 minute
        break_time = 5          # 5 minute
        start = True
        a = tomato(working_time, 'Go working')
  • 計算剩餘時間
def tomato(minutes):
    start_time = time.perf_counter()
    diff_seconds = int(round(time.perf_counter() - start_time))
    left_seconds = minutes * 60 - diff_seconds
    return left_seconds
  • 在畫面上顯示讀書和休息的剩餘時間
a = a - 0.09
minute = str(int(a / 60)).zfill(2)
sec = str(int(a % 60)).zfill(2)
countdown = "remaining time = " + minute + ":" + sec
cv2.putText(frame, countdown, (10,110), fontFace, 1, (0, 0, 0), 3, lineType) # 印出文字
    if int(minute) <= 0 and int(sec) <= 0:
        count += 1
        time.sleep(1)
        # if count == 4:
            #     break
        if count%2 != 0:
            a = tomato(break_time, 'Take a break')
        elif count%2 == 0:
            a = tomato(working_time, 'Go working')

辨識手勢

  • 音效
pygame.mixer.init()
warning1 = pygame.mixer.Sound("phonealert.wav")
warning2= pygame.mixer.Sound("personalert.wav")
music= pygame.mixer.Sound("relax.wav")
shot= pygame.mixer.Sound("shot.wav")

def hand(finger_angle,frame):
    
# 根據手指角度的串列內容，返回對應的手勢名稱
    f1 = finger_angle[0]   # 大拇指角度
    f2 = finger_angle[1]   # 食指角度
    f3 = finger_angle[2]   # 中指角度
    f4 = finger_angle[3]   # 無名指角度
    f5 = finger_angle[4]   # 小拇指角度

    # 小於 50 表示手指伸直，大於等於 50 表示手指捲縮
    if f1<50 and f2>=50 and f3>=50 and f4>=50 and f5>=50:
        return 'START' #good
    
    elif f1>=50 and f2>=50 and f3>=50 and f4>=50 and f5>=50:
        return 'CLOSE' #0
    
    elif f1>=50 and f2<50 and f3>=50 and f4>=50 and f5>=50:
        return '1'
    
    elif f1>=50 and f2<50 and f3<50 and f4>=50 and f5>=50:
        shot.play()
        cv2.imwrite(save_path + file_name + '.jpg',frame)
        return '2'
    
    elif f1>=50 and f2>=50 and f3<50 and f4<50 and f5<50:
        music.play()
        time.sleep(1)
        return 'ok'

    elif f1<50 and f2<50 and f3<50 and f4<50 and f5<50:
        music.stop()
        return 'STOP' #5
    else:
        return
  • 在每一幀畫面中辨識手勢
results = hands.process(img2)                # 偵測手勢
if results.multi_hand_landmarks:
    for hand_landmarks in results.multi_hand_landmarks:
        finger_points = []                   # 記錄手指節點座標的串列
        for i in hand_landmarks.landmark:
            # 將 21 個節點換算成座標，記錄到 finger_points
            x = i.x*w
            y = i.y*h
            finger_points.append((x,y))
        if finger_points:
            finger_angle = hand_angle(finger_points) # 計算手指角度，回傳長度為 5 的串列
            text = hand(finger_angle,frame)            # 取得手勢所回傳的內容

偵測手機、人臉

classNames= []
classFile = r'Object-Detector-main/coco.names'
with open(classFile,'rt') as f:
    classNames = f.read().rstrip('\n').split('\n')

configPath = r'Object-Detector-main/ssd_mobilenet_v3_large_coco_2020_01_14.pbtxt'
weightsPath = r'Object-Detector-main/frozen_inference_graph.pb'

net = cv2.dnn_DetectionModel(weightsPath,configPath)
print(net)
net.setInputSize(320,320)
net.setInputScale(1.0/ 127.5)
net.setInputMean((127.5, 127.5, 127.5))
net.setInputSwapRB(True)
  • 辨識每一幀畫面中是否有手機和人臉
classIds, confs, bbox = net.detect(frame,confThreshold=thres)
#偵測到手機
if 77 in classIds:
    cv2.putText(frame, "Warning!!",(30,400), fontFace, 5, (0,0,255), 10, lineType) # 印出文字
    warning1.play()
    time.sleep(1)
#偵測不到手機也偵測不到人
elif 1 not in classIds:
    cv2.putText(frame, "NOT FOUND!!!",(180,400), fontFace, 5, (0, 0, 255), 8, cv2.FONT_HERSHEY_COMPLEX) # 印出文字
    warning2.play()
    time.sleep(2.5)
    warning2.stop()
#偵測到手機後匡選
if len(classIds) != 0:
    for classId, confidence,box in zip(classIds.flatten(),confs.flatten(),bbox):
        if classId == 77:
            cv2.rectangle(frame,box,color=(0,0,255),thickness=2)
            cv2.putText(frame,classNames[classId-1].upper(),(box[0]+10,box[1]+30),
                        cv2.FONT_HERSHEY_COMPLEX,1,(0,0,255),2)

寄信給使用者

def mail(count, working_time, break_time):
    today = datetime.date.today()
    content = 'working time = ' + str(working_time) + ' minutes' + '<br>'
    content += 'break_time = ' + str(break_time) + ' minutes' + '<br>'
    content += '完成：' + str(count) + '個循環'
    msg = MIMEMultipart()                          # 使用多種格式所組成的內容
    msg.attach(MIMEText(content, 'html', 'utf-8')) # 加入 HTML 內容
    # 使用 python 內建的 open 方法開啟指定目錄下的檔案
    with open('photo_0.jpg', 'rb') as file:
        img = file.read()
    attach_file = MIMEApplication(img, Name = 'photo_0.jpg') # 設定附加檔案圖片
    msg.attach(attach_file)                                  # 加入附加檔案圖片

    msg['Subject'] = "%s 番茄鐘使用時間" % today
    msg['From'] = "tomato"
    msg['To'] = 'account@mail1.ncnu.edu.tw'

    # smtp = smtplib.SMTP('smtp.gmail.com', 587)
    smtp = smtplib.SMTP('smtp.gmail.com', 25)
    smtp.ehlo()
    smtp.starttls()
    # smtp.login() 的第二個參數 : 開啟應用程式密碼
    # https://support.google.com/accounts/answer/185833
    smtp.login('account@gmail.com', '金鑰')
    status = smtp.send_message(msg)
    print(status)
    smtp.quit()

Demo Viedo

Job Assignment

  • 王詠平：program（detect gestures, detect person, Pomodoro）、program integration
  • 李亞軒：program(detect cell phone, other funtions)、program development
  • 林千榆：setting environment、program（Sent back mail）、program development
  • 張可葭：setting environment、raspi research、github content、ppt、Demo veido
  • 黃郁庭：setting environment、program(detect cell phone, other funtions)、program development、Thoughts on the topic、raspi research

Thankful

  • 感謝在MOLi學長姐的無私幫忙
  • 感謝提供設備的柏瑋助教及MOLi

References

Powerpoint

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

1 watching

Forks

1 fork
Report repository

Releases

No releases published

Packages

No packages published

Contributors 5

Languages