Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Updated video callouts and minor fixes #262

Merged
merged 4 commits into from
Jun 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions _quarto.yml
Original file line number Diff line number Diff line change
Expand Up @@ -320,5 +320,10 @@ crossref:
key: exr
latex-env: exr

- kind: float
reference-prefix: Video
key: vid
latex-env: vid

editor:
render-on-save: true
38 changes: 29 additions & 9 deletions contents/ai_for_good/ai_for_good.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ bibliography: ai_for_good.bib
# AI for Good {#sec-ai_for_good}

::: {.content-visible when-format="html"}
Resources: [Slides](#sec-ai-for-good-resource), [Exercises](#sec-ai-for-good-resource), [Labs](#sec-ai-for-good-resource)
Resources: [Slides](#sec-ai-for-good-resource), [Videos](#sec-ai-for-good-resource), [Exercises](#sec-ai-for-good-resource), [Labs](#sec-ai-for-good-resource)
:::

![_DALL·E 3 Prompt: Illustration of planet Earth wrapped in shimmering neural networks, with diverse humans and AI robots working together on various projects like planting trees, cleaning the oceans, and developing sustainable energy solutions. The positive and hopeful atmosphere represents a united effort to create a better future._](images/png/cover_ai_good.png)
Expand Down Expand Up @@ -78,7 +78,7 @@ Widespread TinyML applications can help digitize smallholder farms to increase p

With greater investment and integration into rural advisory services, TinyML could transform small-scale agriculture and improve farmers' livelihoods worldwide. The technology effectively brings the benefits of precision agriculture to disconnected regions most in need.

:::{#exr-agri.callout-caution collapse="true"}
:::{#exr-agri .callout-caution collapse="true"}

### Crop Yield Modeling

Expand Down Expand Up @@ -125,7 +125,7 @@ An on-device algorithm for early and timely life-threatening VA detection will i

The champion, GaTech EIC Lab, obtained 0.972 in $F_\beta$ (F1 score with a higher weight to recall), 1.747 ms in latency, and 26.39 kB in memory footprint with a deep neural network. An ICD with an on-device VA detection algorithm was [implanted in a clinical trial](https://youtu.be/vx2gWzAr85A?t=2359).

:::{#exr-hc.callout-caution collapse="true"}
:::{#exr-hc .callout-caution collapse="true"}

### Clinical Data: Unlocking Insights with Named Entity Recognition

Expand Down Expand Up @@ -162,18 +162,24 @@ Researchers from Moulay Ismail University of Meknes in Morocco [@bamoumen2022tin

In disaster response, speed and safety are paramount. But rubble and wreckage create hazardous, confined environments that impede human search efforts. TinyML enables nimble drones to assist rescue teams in these dangerous scenarios.

When buildings collapse after earthquakes, small drones can prove invaluable. Equipped with TinyML navigation algorithms, micro-sized drones like the [CrazyFlie](https://www.bitcraze.io/) can traverse cramped voids and map pathways beyond human reach [@duisterhof2019learning]. Obstacle avoidance allows the drones to weave through unstable debris. This autonomous mobility lets them rapidly sweep areas humans cannot access.
When buildings collapse after earthquakes, small drones can prove invaluable. Equipped with TinyML navigation algorithms, micro-sized drones like the [CrazyFlie](https://www.bitcraze.io/) can traverse cramped voids and map pathways beyond human reach [@duisterhof2019learning]. Obstacle avoidance allows the drones to weave through unstable debris. This autonomous mobility lets them rapidly sweep areas humans cannot access. @vid-l2seek presents the [@duisterhof2019learning] paper on deep reinforcement learning using drones for source-seeking.

The video below presents the [@duisterhof2019learning] paper on deep reinforcement learning using drones for source-seeking.
:::{#vid-l2seek .callout-important}

# Learning to Seek

{{< video https://www.youtube.com/watch?v=wmVKbX7MOnU >}}

Crucially, onboard sensors and TinyML processors analyze real-time data to identify signs of survivors. Thermal cameras detect body heat, microphones pick up calls for help, and gas sensors warn of leaks [@duisterhof2021sniffy]. Processing data locally using TinyML allows for quick interpretation to guide rescue efforts. As conditions evolve, the drones can adapt by adjusting their search patterns and priorities.
:::

Crucially, onboard sensors and TinyML processors analyze real-time data to identify signs of survivors. Thermal cameras detect body heat, microphones pick up calls for help, and gas sensors warn of leaks [@duisterhof2021sniffy]. Processing data locally using TinyML allows for quick interpretation to guide rescue efforts. As conditions evolve, the drones can adapt by adjusting their search patterns and priorities. @vid-sniffybug is an overview of autonomous drones for gas leak detection.

The following video is an overview of autonomous drones for gas leak detection.
:::{#vid-sniffybug .callout-important}

{{< video https://www.youtube.com/watch?v=hj_SBSpK5qg >}}

:::

Additionally, coordinated swarms of drones unlock new capabilities. By collaborating and sharing insights, drone teams comprehensively view the situation. Blanketing disaster sites allows TinyML algorithms to fuse and analyze data from multiple vantage points, amplifying situational awareness beyond individual drones [@duisterhof2021sniffy].

Most importantly, initial drone reconnaissance enhances safety for human responders. Keeping rescue teams at a safe distance until drone surveys assess hazards saves lives. Once secured, drones can guide precise personnel placement.
Expand All @@ -198,12 +204,14 @@ Technology has immense potential to break down barriers faced by people with dis

With machine learning algorithms running locally on microcontrollers, compact accessibility tools can operate in real time without reliance on connectivity. The [National Institute on Deafness and Other Communication Disorders (NIDCD)](https://www.nidcd.nih.gov/health/statistics/quick-statistics-hearing) states that 20% of the world's population has some form of hearing loss. Hearing aids leveraging TinyML could recognize multiple speakers and amplify the voice of a chosen target in crowded rooms. This allows people with hearing impairments to focus on specific conversations.

Similarly, mobility devices could use on-device vision processing to identify obstacles and terrain characteristics. This enables enhanced navigation and safety for the visually impaired. Companies like [Envision](https://www.letsenvision.com/) are developing smart glasses, converting visual information into speech, with embedded TinyML to guide blind people by detecting objects, text, and traffic signals.
Similarly, mobility devices could use on-device vision processing to identify obstacles and terrain characteristics. This enables enhanced navigation and safety for the visually impaired. Companies like [Envision](https://www.letsenvision.com/) are developing smart glasses, converting visual information into speech, with embedded TinyML to guide blind people by detecting objects, text, and traffic signals. @vid-envision below shows the different real-life use cases of the Envision visual aid glasses.

The video below shows the different real-life use cases of the Envision visual aid glasses.
:::{#vid-envision .callout-important}

{{< video https://www.youtube.com/watch?v=oGWinIKDOdc >}}

:::

TinyML could even power responsive prosthetic limbs. By analyzing nerve signals and sensory data like muscle tension, prosthetics and exoskeletons with embedded ML can move and adjust grip dynamically, making control more natural and intuitive. Companies are creating affordable, everyday bionic hands using TinyML. For those with speech difficulties, voice-enabled devices with TinyML can generate personalized vocal outputs from non-verbal inputs. Pairs by Anthropic translates gestures into natural speech tailored for individual users.

By enabling more customizable assistive tech, TinyML makes services more accessible and tailored to individual needs. And through translation and interpretation applications, TinyML can break down communication barriers. Apps like Microsoft Translator offer real-time translation powered by TinyML algorithms.
Expand Down Expand Up @@ -253,12 +261,24 @@ These slides are a valuable tool for instructors to deliver lectures and for stu

:::

:::{.callout-important collapse="false"}
#### Videos

- @vid-l2seek

- @vid-sniffybug

- @vid-envision

:::

:::{.callout-caution collapse="false"}
#### Exercises

- @exr-agri

- @exr-hc

:::

:::{.callout-warning collapse="false"}
Expand Down
12 changes: 9 additions & 3 deletions contents/benchmarking/benchmarking.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ bibliography: benchmarking.bib
# Benchmarking AI {#sec-benchmarking_ai}

::: {.content-visible when-format="html"}
Resources: [Slides](#sec-benchmarking-ai-resource), [Exercises](#sec-benchmarking-ai-resource), [Labs](#sec-benchmarking-ai-resource)
Resources: [Slides](#sec-benchmarking-ai-resource), [Videos](#sec-benchmarking-ai-resource), [Exercises](#sec-benchmarking-ai-resource), [Labs](#sec-benchmarking-ai-resource)
:::

![_DALL·E 3 Prompt: Photo of a podium set against a tech-themed backdrop. On each tier of the podium, there are AI chips with intricate designs. The top chip has a gold medal hanging from it, the second one has a silver medal, and the third has a bronze medal. Banners with 'AI Olympics' are displayed prominently in the background._](images/png/cover_ai_benchmarking.png)
Expand Down Expand Up @@ -135,7 +135,7 @@ These types of microbenchmarks include zooming into very specific operations or

Example: [DeepBench](https://github.com/baidu-research/DeepBench), introduced by Baidu, is a good example of something that assesses the above. DeepBench assesses the performance of basic operations in deep learning models, providing insights into how different hardware platforms handle neural network training and inference.

:::{#exr-cuda.callout-caution collapse="true"}
:::{#exr-cuda .callout-caution collapse="true"}

### System Benchmarking - Tensor Operations

Expand Down Expand Up @@ -449,7 +449,7 @@ Metrics: We will measure the following metrics:

By measuring these metrics, we can assess the performance of the object detection model on the edge device and identify any potential bottlenecks or areas for optimization to enhance real-time processing capabilities.

:::{#exr-perf.callout-caution collapse="true"}
:::{#exr-perf .callout-caution collapse="true"}

### Inference Benchmarks - MLPerf

Expand Down Expand Up @@ -821,6 +821,12 @@ These slides are a valuable tool for instructors to deliver lectures and for stu

:::

:::{.callout-important collapse="false"}
#### Videos

* *Coming soon.*
:::

:::{.callout-caution collapse="false"}
#### Exercises

Expand Down
16 changes: 11 additions & 5 deletions contents/data_engineering/data_engineering.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ bibliography: data_engineering.bib
# Data Engineering {#sec-data_engineering}

::: {.content-visible when-format="html"}
Resources: [Slides](#sec-data-engineering-resource), [Exercises](#sec-data-engineering-resource), [Labs](#sec-data-engineering-resource)
Resources: [Slides](#sec-data-engineering-resource), [Videos](#sec-data-engineering-resource), [Exercises](#sec-data-engineering-resource), [Labs](#sec-data-engineering-resource)
:::

![_DALL·E 3 Prompt: Create a rectangular illustration visualizing the concept of data engineering. Include elements such as raw data sources, data processing pipelines, storage systems, and refined datasets. Show how raw data is transformed through cleaning, processing, and storage to become valuable information that can be analyzed and used for decision-making._](images/png/cover_data_engineering.png)
Expand Down Expand Up @@ -123,7 +123,7 @@ In this context, using KWS as an example, we can break each of the steps out as
7. **Iterative Feedback and Refinement:**
Once a prototype KWS system is developed, it's crucial to test it in real-world scenarios, gather feedback, and iteratively refine the model. This ensures that the system remains aligned with the defined problem and objectives. This is important because the deployment scenarios change over time as things evolve.

:::{#exr-kws.callout-caution collapse="true"}
:::{#exr-kws .callout-caution collapse="true"}

### Keyword Spotting with TensorFlow Lite Micro

Expand Down Expand Up @@ -174,7 +174,7 @@ Web scraping can yield inconsistent or inaccurate data. For example, the photo i

![A picture of old traffic lights (1914). Credit: [Vox.](https://www.vox.com/2015/8/5/9097713/when-was-the-first-traffic-light-installed)](images/jpg/1914_traffic.jpeg){#fig-traffic-light}

:::{#exr-ws.callout-caution collapse="true"}
:::{#exr-ws .callout-caution collapse="true"}

### Web Scraping

Expand Down Expand Up @@ -219,7 +219,7 @@ While synthetic data offers numerous advantages, it is essential to use it judic

![Increasing training data size with synthetic data generation. Credit: [AnyLogic](https://www.anylogic.com/features/artificial-intelligence/synthetic-data/).](images/jpg/synthetic_data.jpg){#fig-synthetic-data}

:::{#exr-sd.callout-caution collapse="true"}
:::{#exr-sd .callout-caution collapse="true"}

### Synthetic Data
Let us learn about synthetic data generation using Generative Adversarial Networks (GANs) on tabular data. We'll take a hands-on approach, diving into the workings of the CTGAN model and applying it to the Synthea dataset from the healthcare domain. From data preprocessing to model training and evaluation, we'll go step-by-step, learning how to create synthetic data, assess its quality, and unlock the potential of GANs for data augmentation and real-world applications.
Expand Down Expand Up @@ -304,7 +304,7 @@ Maintaining the integrity of the data infrastructure is a continuous endeavor. T

There is a boom in data processing pipelines, commonly found in ML operations toolchains, which we will discuss in the MLOps chapter. Briefly, these include frameworks like MLOps by Google Cloud. It provides methods for automation and monitoring at all steps of ML system construction, including integration, testing, releasing, deployment, and infrastructure management. Several mechanisms focus on data processing, an integral part of these systems.

:::{#exr-dp.callout-caution collapse="true"}
:::{#exr-dp .callout-caution collapse="true"}

### Data Processing

Expand Down Expand Up @@ -493,6 +493,12 @@ These slides are a valuable tool for instructors to deliver lectures and for stu

:::

:::{.callout-important collapse="false"}
#### Videos

* *Coming soon.*
:::

:::{.callout-caution collapse="false"}
#### Exercises

Expand Down
35 changes: 32 additions & 3 deletions contents/dl_primer/dl_primer.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ bibliography: dl_primer.bib
# DL Primer {#sec-dl_primer}

::: {.content-visible when-format="html"}
Resources: [Slides](#sec-deep-learning-primer-resource), [Exercises](#sec-deep-learning-primer-resource), [Labs](#sec-deep-learning-primer-resource)
Resources: [Slides](#sec-deep-learning-primer-resource), [Videos](#sec-deep-learning-primer-resource), [Exercises](#sec-deep-learning-primer-resource), [Labs](#sec-deep-learning-primer-resource)
:::

![_DALL·E 3 Prompt: Photo of a classic classroom with a large blackboard dominating one wall. Chalk drawings showcase a detailed deep neural network with several hidden layers, and each node and connection is precisely labeled with white chalk. The rustic wooden floor and brick walls provide a contrast to the modern concepts. Surrounding the room, posters mounted on frames emphasize deep learning themes: convolutional networks, transformers, neurons, activation functions, and more._](images/png/cover_dl_primer.png)
Expand Down Expand Up @@ -91,22 +91,40 @@ Multilayer perceptrons (MLPs) are an evolution of the single-layer perceptron mo

The forward pass is the initial phase where data moves through the network from the input to the output layer. During this phase, each layer performs specific computations on the input data, using weights and biases before passing the resulting values to subsequent layers. The final output of this phase is used to compute the loss, indicating the difference between the predicted output and actual target values.

The video below explains how neural networks work using handwritten digit recognition as an example application. It also touches on the math underlying neural nets.
@vid-nn below explains how neural networks work using handwritten digit recognition as an example application. It also touches on the math underlying neural nets.

:::{#vid-nn .callout-important}

# Neural Networks

{{< video https://www.youtube.com/embed/aircAruvnKk?si=qfkBf8MJjC2WSyw3 >}}

:::

#### Backward Pass (Backpropagation)

Backpropagation is a key algorithm in training deep neural networks. This phase involves calculating the gradient of the loss function concerning each weight using the chain rule, effectively moving backward through the network. The gradients calculated in this step guide the adjustment of weights to minimize the loss function, thereby enhancing the network's performance with each iteration of training.

Grasping these foundational concepts paves the way to understanding more intricate deep learning architectures and techniques, fostering the development of more sophisticated and productive applications, especially within embedded AI systems.

The following two videos build upon the previous one. They cover gradient descent and backpropagation in neural networks.
@vid-gd and @vid-bp build upon @vid-nn. They cover gradient descent and backpropagation in neural networks.

:::{#vid-gd .callout-important}

# Gradient descent

{{< video https://www.youtube.com/watch?v=IHZwWFHWa-w&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=2 >}}

:::

:::{#vid-bp .callout-important}

# Backpropagation

{{< video https://www.youtube.com/watch?v=Ilg3gGewQ5U&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=3 >}}

:::

### Model Architectures

Deep learning architectures refer to the various structured approaches that dictate how neurons and layers are organized and interact in neural networks. These architectures have evolved to tackle different problems and data types effectively. This section overviews some well-known deep learning architectures and their characteristics.
Expand Down Expand Up @@ -282,6 +300,17 @@ These slides are a valuable tool for instructors to deliver lectures and for stu

:::

:::{.callout-important collapse="false"}
#### Videos

- @vid-nn

- @vid-gd

- @vid-bp

:::

:::{.callout-caution collapse="false"}
#### Exercises

Expand Down
6 changes: 6 additions & 0 deletions contents/efficient_ai/efficient_ai.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,12 @@ These slides are a valuable tool for instructors to deliver lectures and for stu
- [Continuous Evaluation Challenges for TinyML.](https://docs.google.com/presentation/d/1OuhwH5feIwPivEU6pTDyR3QMs7AFstHLiF_LB8T5qYQ/edit?usp=drive_link&resourcekey=0-DZxIuVBUbJawuFh0AO-Pvw)
:::

:::{.callout-important collapse="false"}
#### Videos

* *Coming soon.*
:::

:::{.callout-caution collapse="false"}

#### Exercises
Expand Down
Loading
Loading