Skip to content

Latest commit

 

History

History
552 lines (276 loc) · 103 KB

README.md

File metadata and controls

552 lines (276 loc) · 103 KB
Raw

Mystic Mirror

Introduction

I am the guy that:

  • Never checks for delays in his work commute until he is stuck in a four-lane accident

  • Forgets his umbrella when the forecast calls for afternoon showers

  • Doesn't find out about a major news event unless the story breaks on ESPN

  • Always forgets to pull my trash bins to the curb on garbage pick-up day

In short, my morning routine is a mess (#firstworldproblems). An Amazon Echo (or a phone, for that matter) would resolve most of those problems. Unfortunately, I never seem to have my phone with me as I'm getting ready in the morning (it's usually charging). And I'm usually not asking Alexa for these details (I have Alexa enabled through my Fire TV and it's impractical to stop, cut on the tv, find the remote, and ask a question).

60% of my morning routine is centered in and around the bathroom or bedroom, so I decided to build an Alexa skill and Alexa Voice Service-enabled magic mirror – which I've titled the Mystic Mirror.

IMAGE ALT TEXT HERE

Overview

A Magic Mirror is a Raspberry Pi device connected to a monitor fronted by a two way mirror. This gives the illusion that the text/words/images appear from the mirror (similar to the Magic Mirror in Snow White and the Seven Dwarfs). People have been making magic mirrors for years; lately there have been some that are enabled with Alexa (or other voice activated services). I wanted to make my Mystic Mirror a step further by adding a few features, such as:

  • Building Alexa functionality within the mirror, controlled by a button. This would allow the magic mirror to work with specific Alexa skills that would display messages. This would also let me use the Alexa for "base functionality" (set an alarm, use another skill, etc).

  • Not having the display always on (I am very sensitive to light when I am sleeping; an "always displaying" magic mirror wasn't going to be an option). My requested information appears for a few seconds (depending on the request) then fades away.

  • Building the functionality/"modules" using Lambda, this would allow others to use the skill (even without a having a magic mirror).

  • Build functionality into my Alexa skill that would support "visual push notifications" to my Mystic Mirror. This was really centered around reminders that would appear at the appropriate time.

  • Leverage Alexa to extend functionality of a typical magic mirror. For example, most magic mirrors have the location "hard coded" into code on the Raspberry Pi. I wanted to provide flexibility to get things (like weather) for any requested location.

What the Mystic Mirror Can Do

IMAGE ALT TEXT HERE

I've built the following major requests into the Mystic Mirror:

Time

  • Question: What is the Time?

  • Response: Alexa says the time

  • Display: The day, date and time is displayed on the Mystic Mirror

Weather

  • Q: What is the Weather?

  • R: Alexa says the current weather temperature, conditions, and the day's high and low temperature

  • D: The current temperature, conditions, high and low are displayed, along with a 4 day forecast

Traffic

  • Q: What is the Traffic?

  • R: Alexa tells me the number of traffic incidents in my area.

  • D: The Mystic Mirror shows a map with traffic status and lists the incidents

News

  • Q: What are today's News Headlines

  • R: Alexa speaks one major headline.

  • D: The Mystic Mirror lists 5 headlines and a short blurb on each story (and image, if provided)

Messaging

  • Q: Send a message to a contact (e.g. – Tell John to come home)

  • R: Alexa confirms that the message was sent. The skill uses AWS SNS to send a message to a contact (note this requires that you "Add a Contact")

  • D: The display shows the message that was sent.

Take a Selfie

  • Q: Take a Selfie

  • R: Alexa tells you told still and sends a command to the Raspberry Pi to take a picture. Alexa asks if you want to send the picture to one of your saved contacts (using the messaging functionality listed above)

  • D: The picture is displayed.

Provide an inspirational quote

  • Q: Inspire Me

  • R: Alexa tells you an inspirational quote

  • D: The quote is displayed.

Set up a reminder

  • Q: Remind me to "do something"

  • R: Alexa prompts you for the frequency (single, daily, weekly), reminder time, and day to start the reminder

  • D: The reminder appears at the lower left corner of the display approximately one hour before the action needs to be taken. Reminders are removed by saying "clear reminders". Reminders can be deleted by saying "Delete reminder".

Brush Teeth Countdown

  • Q: Start teeth timer

  • R: Alexa confirms that the countdown has started.

  • D: A two minute countdown is displayed in the lower right corner. This countdown continues even if other action are requested

I also added a few Easter eggs:

Alexa Skills Kit Code

The next section provides an overview of the code that runs the Mystic Mirror. Jump to the "Build Your Own" section for details on how to install the code and build your own Mystic Mirror.

Authentication (get_profile function)

I needed a mechanism to identify the user. I first planned to build my own, but instead decided to leverage Google's OAUTH service. This was a logical choice, as I plan to display (at a later point in time) the calendar details and task lists for a user.

When the user asks to "set a default location", the user id and provided city and state are saved to a DynamoDB table. I saved this information so that the user would not need to indicate location every time a request was made for time, traffic, or weather.

The user id was also saved in second DynamoDB table for the purpose of routing the message to the correct IOT device (more on this later on).

APIs and other functionality executed in the Lambda code

The following public APIs were used to build the functionality:

Communication with the Raspberry Pi (RPI)

Messages are sent from Lambda to the RPI via MQTT. I enabled this interaction by using the AWS IoT service.

Instead of building this logic into the "main" Lambda function, I created a second Lambda function that would serve as the "Lambda based" MQTT client. This function receives (via SNS) a message from the "Magic Mirror" skill and publishes the message onto a topic. The diagram below shows the interaction in better detail.

alt text

Interaction diagram between physical device and AWS services

As started above, I used MQTT to send messages to the Raspberry Pi. These messages were sent to one of two "clients"

  • Display client - a javascript-coded website that displayed text/images on the monitor based on the commands from the Alexa skill

  • Take Selfie client - a python-coded script that took a picture if user made a request to "take a selfie"

Display Client (aws.js)

The Display client is a basic js website that uses Paho mqtt to connect to the AWS IOT service and takes action if a message is received on the "success" topic. These messages are typically the words spoken by the Raspberry Pi, but could also be additional detail (as seen in the weather, traffic, and news requests).

alt text

The client requires AWS IAM credentials.

Functionality for the "Reminders" functionality required the client to publish messages. Every 30 minutes, the client will publish the local time to an MQTT topic (reminders/time). This message triggered a Lambda function to compare the local time to the reminder time for all reminders saved in the DynamoDB table. Any reminders that need to be displayed are published to the "success" topic and displayed on the screen.

Take Selfie client (take_selfie.py):

The "take selfie" functionality required for the "fswebcam" command to be called on the RPI.

  • I created a python script that serves as an MQTT client; this code is always running on the RPI, monitoring a topic for a message.

  • When the user makes the request, the Lambda code sends a message to the MQTT client. This message triggers the camera to take a photo.

  • The photo is then uploaded to an AWS S3 bucket (using TinyS3).

  • If the user requests for the selfie be sent to a contact, the link is send (using AWS SNS) to the contact

  • Auto delete rules are enabled on the S3 object to delete it after 24 hours.

alt text

Alexa Voice Service Code and Peripherals

I looked at both the following JAVA and Python examples of the AVS code for use in the Mystic Mirror. While I liked the Python version (specifically because of the ability to restart on reboot), I ultimately decided to build on top of the JAVA example (I think it is slightly more robust and responds more quickly).

After I installed the code, I made two changes:

Physical Button

I used the code base (built by Eddie Hartmam) to configure my a physical button (used to control the requests to Alexa). The only change was that I used different pins (called out in schematics).

Indicator LED

I decided to add an indicator light to supplement the verbal ques. Since I built off of Eddie's code (which already had Pi4J referenced), all I needed to do was add a few lines to control the output to the RGB LED. This would let me know when the microphone was recording.

The microphone and camera plugged in via USB. The speakers used the 3.5 aux port.

Once I got everything plugged in, and the code running....viola!

IMAGE ALT TEXT HERE

The connected, but not complete, Mystic Mirror

The Mystic Mirror Frame

Building the frame was straight forward. I first found an inexpensive monitor (a refurbished 17 in Compaq LE 1171) and removed the frame (using a Phillips head screwdriver and a paint scraper).

Note: be careful of the wiring to the physical buttons; you'll need that intact to turn the monitor on and off.

alt text

From there, I measured the size of the monitor (without the plastic frame) and purchased a two way mirror with the same dimensions (I used a cut-to-order acrylic sheet from Tap Plastics).

From there, it was a quick trip to my local hardware store to purchase a 1x3 wood beam. I used Gorilla glue and a 24-in clamp to keep the piece in place while they dried.

Gluing the frame siding together with Gorilla Glue

alt text

After the frame siding was complete, I added the trim on the front, a panel for the speakers and the LED indicator, and a hole on the side for the push button. For the frame, I used both Gorilla Glue and small nails from a staple gun to keep things in place. I also used wood putty to cover the nails and address any cracks in the wood.

alt text

After the glue dried, I sanded and applied a couple of coats off paint, then loaded the monitor, RPI, and peripherals in the back. A few call-outs:

  • I did not add holes at the top/bottom for air flow. I have 100% landed one where I'll place the mirror (or it it will lay flush against the wall). I'll add those holes as appropriate.

  • Going with a cheaper monitor saved me $20, but cost be in real estate, as I needed a HDMI to VGA connector. I hindsight, I would have spent the extra $20 for an HDMI enabled monitor.

  • I wasn't comfortable building a single source that would power the speakers, monitor and RPI. It's a bit of an eyesore having three cables snaking out the back; I may revisit this once I decide where the Mystic Mirror will sit.

alt text

The final result - a Mystic Mirror that (a) works as a functional mirror and (b) reponds to an Alexa request (and displays the requested information if the Mystic Mirror skill is used).

alt text

Building Your Own Mystic Mirror

The Mystic Mirror is a complicated project that requires code in the AWS cloud and on the Raspberry Pi.

Part 1 - Build the Alexa Skill and supporting AWS services

Option A - Use the existing Mystic Mirror Alexa skill and build your Mystic Mirror to use this skill

For this option, all you need to do is build a Mystic Mirror display and download the provided code. (Note: There are a few variables that will nee to be updated. Also, you will need permission to connect to my AWS "infrastructure"; I can provide this on request (with restrictions that limit access to only your device).

If you want to do follow this option, skip to "Part 2".

Option B - Use my code to build your own Alexa still

This option is much more complicated, but will give you freedom to add/update logic as you see fit. The instructions for Option B are below.

Step 1 - Create and Configure your AWS IoT Device

  • Follow the instructions to Get Started with AWS IoT here

  • You can name your device anything; I named my "Mystic_Mirror"

  • Be sure to save your certificates; you'll need them later on.

Step 2 - Create the supporting DynamoDB tables

You can create tables using CLI; however, I find that it's easier just to create them via the DynamoDB GUI.

  • Create table "Magic_Mirror" with a primary partion key titled "Email" (should be a string)

  • Create table "Magic_Mirror_Contact" with a primary partiton key titled "Email" (string) and a primary sort key titled "ContactName" (string)

  • Create table "Magic_Mirror_Reminders" with a primary partiton key titled "Email" (string) and a primary sort key titled "Reminder" (string)

alt text

Step 3 - Create the SNS functions that sends messages to the Raspberry Pi

Go to the AWS SNS console and create a SNS topic called "Magic_Mirror_MQTT"

Step 4 - Download the code in the MQTT_Node/src folder

Open the config.js script and make the following changes:

  • Update the config.host with the rest api for theIoT device you created in step 1 (note, you only need part of the address, e.g. "a6i0a3yz21r4k2.iot.us-east-1.amazonaws.com")

  • Update the config.privateKey variable to the text from the "private.pem.key" file that you downloaded in Step 1

  • Update the config.certificate variable to the text from the "certificate.pem.crt" file that you downloaded in Step 1

Once that is done, use the Lambda console to create a Lambda function

  • Click the "Create A Lambda function" button.

  • Click the "skip" button

  • Select "SNS" as the trigger. In the drop down, select "Magic_Mirror_MQTT" as the SNS topic. Check the "Enable Trigger" option. Click Next.

alt text

  • Enter the name of the function as "Write_to_MQTT_Node"

  • Give the code a logic description ("Lambda code to send MQTT messages to Raspberry Pi)

  • Select Node.js 4.3 as the runtime

  • Zip the contents of the src folder into a zip file and select the "Code entry type" as "Upload a ZIP file"

  • Create a new role and attach the following policies: AmazonSNSFullAccess

  • Ensure the role has the following policy

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

 "logs:CreateLogGroup",

 "logs:CreateLogStream",

 "logs:PutLogEvents"

],

"Resource": "arn:aws:logs:*:*:*"

}

]

}

  • Click next, then click create function
Step 5 - Download the code in the Alexa_Magic_Mirror/src folder

Open the lambda_function.py script and make the following changes:

  • update TopicARN = 'placeholder' with the TopicARN you created in Step 3

  • update the bucket= 'palceholder' with the name of your S3 bucket

Once that is done, use the Lambda console to create a Lambda function

  • Click the "Create A Lambda function" button.

  • Click the "skip" button

  • Select "Alexa Skills Kit" as the trigger

alt text

  • Enter the name of the function as "Magic_Mirror"

  • Give the code a logic description ("Lambda code for the Alexa skill")

  • Select Python 2.7 as the runtime

  • Zip the lambda_function.py script into a zip file and select the "Code entry type" as "Upload a ZIP file"

  • Create a new role and attach the following policies: AmazonS3FullAccess, AmazonDynamoDBFullAccess, AmazonSNSFullAccess

  • Ensure the role has the following policy

{

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

 "logs:CreateLogGroup",

 "logs:CreateLogStream",

 "logs:PutLogEvents"

],

"Resource": "arn:aws:logs:*:*:*"

}

]

}

  • Change the timeout to 10 sec

  • Click next, then click create function

  • Save the ARN of the Lambda skill (you will need it in the next section)

Step 6 - Create the Alexa voice intents

  • Log into the Amazon Developer console, click on Alexa in the header, then click on Add a New Skill

  • Create a name for the skill (such as Mystic Mirror) and a Invocation Name (such as mystic mirror). Click Next.

  • On the "Interaction Model" page, use the code in the Alexa_Magic_Mirror/speechAssets folder and update the Intent Schema, Custom Slot Types, and Sample Utterances

  • On the "Configuration Screen" update the endpoint with the Lambda ARN you saved in Step 5.

  • Enable account linking. The code is built to authenticate using Google services. You'll need to create a login page to allow users to link their accounts. You'll also need to create a Google project. (This is a bit out of scope for these instructions; there is some good information available online, and I can provide direction if there are questions).

  • Go to the "Publishing Information" screen and update the values as directed.

  • Go to the "Privacy and Compliance" screen; you'll need to create Privacy Policy html page and provide the link. [Note - both the login and policy html pages can be hosted in S3 for ease).

  • Click save, and you're done. You can submit for certification if you desire (if you don't you'll still be able to use the skill, but others will not).

Step 7 - Create the Reminders Lambda Function

  • Download the code in the Reminders/src folder

  • Open the AWS Lambda Console and click the "Create A Lambda function" button.

  • Click the "skip" button

  • Click 'Next"

  • Enter the name as 'Get_Magic_Mirror_Reminders'

  • Enter the description as 'Get message from IoT Rule'

  • Select node.js 4.3 as the runtime.

  • Zip the downloaded code and upload it by selecting "Upload a Zip file" for Code Entry Type.

  • Select the role that you created for Mystic_Mirror lambda function (this function will need SNS access, just as the other lambda function needs)

  • Click next, then click create function

Step 8 - Create the Reminders IoT Rule

  • Open the AWS Iot Console. Click "Create a resource" then "Create a rule"

  • Call the rule "SendTimeToLambda_MM"

  • Give a description as "Get time from Magic Mirror"

  • Enter the rule query statement as: SELECT * FROM 'remindertime/#'

  • For "Choose as Action", select "Insert this message into a code function and execute it ( Lambda )"

  • Select "Function Name" as "Get_Magic_Mirror_Reminders"

  • Click "Create"

Part 2 - Configure the RPI to Display Messages

These steps will enable you to display content pushed from the Alexa Skill. The code of this section is in the Mystic_Mirror folder. You'll need to download the code to your Pi (I suggest putting it in "home/pi/Mystic_Mirror" and follow the steps below.

Step 1 - Configure the display client

  • Open the aws.js file and update the email variable to match the email address used for authentication (e.g. your Google email address)

  • In the aws.js file, update the region, secretKey, accessKey, and host If you are using my Alexa skill (Part 1, Option A), then send me a message on Twitter to request a keyfile. If you are using your own skill/Iot Thing, then use keys associated with your account.

Step 2 - Configure the take_selfie client

  • Install tinyS3 (command - sudo pip install tinys3)

  • Install paho (command - pip install paho-mqtt)

  • Open take_selfie.py file and update the email variable to match the email address used for authentication (e.g. your google email address)

  • Update the secretkey and accesskey variables. If you are using my Alexa skill (Part 1, Option A), then send me a message on Twitter to request a keyfile. If you are using your own skill/Iot Thing, then use keys associated with your account.

  • Download the IoT certificates to the the "/home/pi/certs/" location. If you are using my Alexa skill (Part 1, Option A), then send me a message on Twitter to request valid certificates. If you are using your own skill/Iot Thing, then use download the certificates associated with your IoT thing.

  • run the file by typing "python take_selfie.py". This script must be running in order for the "take a selfie" functionality to work.

  • Note - you can set up the script to start at boot; I did not because I plan to remove the camera when I install the mirror in my bedroom (maybe I've watched too much "Mr. Robot")

alt text

Step 3 - Set up the display to work in kiosk mode

Once you have the MQTT clients working, you'll need to make a few changes to get the display website to display without borders, mouse indicator, etc.

  • Install chromium (if you are on a RPI 3, then follow these instructions).

  • pen the autostart file ("/home/pi/.config/lxsession/LXDE-pi/autostart") and add the following line:

@chromium-browser--incognito--noerordialogs--disable-session-crashed-bubble--disable-infobars--kioskfile:///home/pi/Mystic_Mirror/index.html

Note - this assumes you installed the Mystic_Mirror code in the home/pi location.

Part 3 - Install AVS code and configure the Button and LED

Step 1 - Wire the LED and Button to the RPI as follows:

alt text

Step 2 - Install the AVS code

  • Download the code and follow the instructions listed by Eddie Hartman ( here)

  • Once complete, replace the AVAjava file with the version in the AVS folder (this version will have the GPIO logic based on the wiring schema above).

Part 4 - Build the Mystic Mirror Frame

I outlined how I built my frame in an earlier section. There's no perfect way to build a frame; it really comes down to the size of your monitor and how fancy you want to get (I am not very handy when it comes to woodworking, so my frame is fairly basic).

![alt text](https://hackster.imgix.net/uploads/attachments/181775/notes.png?"I have pages on top of pages of notes and measurements.")

Instead of steps, I'll call out a few pointers/gotchas:

  • Do not underestimate the space you'll need to store your pi, speakers, cables, etc.

  • Measure, Measure, Measure. Then before you cut, measure again.

  • I spent a lot of time n this site getting ideas on how to build my frame

  • Wood putty is your friend, and can fix a lot of mistakes.

  • Be sure to drill small holes in your panel so that your speaker can be clearly heard and your microphone can pick up your voice.