README.md

Prerequisites | Browser Troubleshooting | Future plans | Playground Context | Playground UI | AI Basics | AI Mezzanines | AI Support | Miscellany | Object Support | Simple Examples | AI Examples | DIY AI Applications

blockly-ai-playground

experiment with AI and build your own AI agents using Blockly - even if you are a complete beginner! (also suitable for school lessons)

Build your own Copilot - with Blockly - the video will show you how well it can perform (even in comparison with the original)! (sorry for the bad quality: the video had to be created in a hurry)

(Work in progress - it was my contribution to the Backdrop Build v5 contest and had to be launched as an MVP before their deadline. I'm now working on polishing it up...)

Live Demo Warning: please read Overview and Prerequisites first. And Browser Troubleshooting may help you in case of problems

Recommendation: start with the basic Settings example and enter your APIServer, APIKey and default_model settings, at least. Then use the PoC example and run your first inferences. Later, you may also try the AI-assisted Web Search and the AI Agent to learn about the benefits of "AI agents". Finally, when you are ready, build your own AI-assisted applications - perhaps, based on the DIY AI Applications examples

Please note: the "blockly-ai-playground" has only been tested with Perplexity AI (and Ollama with Llama 3.1) on a Mac so far. Additionally, the performance of generative AI is hard to predict - as a consequence, your mileage may vary when compared to what you saw in the video

You have comments, wishes or ideas? You found a bug? Then open an issue. You like this tool and plan to use it? Then leave a star!

Overview

AI agents are "cool" these days and can be found at many places - but for most of us, they are basically just "black boxes" which do not reveal how they work internally.

Blockly is a visual programming environment developed by Google that allows users to create code using drag-and-drop "Lego"-like blocks. It is designed to simplify coding by abstracting away the syntax, making it accessible for beginners and educational purposes.

By combining these two technologies, the "blockly-ai-playground" offers beginners and casual programmers an easy way to play and experiment with AI and build their own AI agents!

Prerequisites

The "blockly-ai-playground" can be used with any AI provider offering an OpenAI compatible API - this may be OpenAI itself, Perplexity (which was used for development), Ollama (which runs LLMs locally on one's computer) and, presumably, many others.

As a consequence, you will need

• the URL of an API entry point (either locally on your machine or on the internet) and
• an API Access Key if you are using an official AI provider

Browser Troubleshooting

Modern browser security may cause a lot of trouble when a web application wants to communicate with servers that have not been configured properly. If you know what you are doing, you may lower certain security levels in order to get the experience you expect.

CORS

Cross-Origin Resource Sharing (CORS) is a security feature implemented in web browsers that allows or restricts web pages from making requests to a domain different from the one that served the web page. It is needed to prevent malicious websites from accessing sensitive data on other domains without permission, thereby protecting user data and maintaining web security.

Incorrect CORS settings of a server(!) can block legitimate requests from different domains, leading to functionality issues in web applications. This often results in CORS errors, where the browser blocks the request due to a lack of proper authorization from the server.

To get rid of CORS errors as a browser user, you may install a browser extension that adds the necessary CORS headers to server requests, allowing cross-origin requests to succeed. However, this method should be used only for development purposes, as it can introduce security risks if used indiscriminately.

Here is an example for a free extension for Chrome browsers (and compatible ones) - this one works fine and is free.

Communicating with your own PC

Modern browsers require HTTPS instead of HTTP for several security and trust reasons.

HTTPS requires servers to provide valid certificates to establish a secure, encrypted connection between the client and server, ensuring data integrity and authenticity.

However, this requirement makes it difficult for a web application to communicate with "localhost" because certificate authorities (CAs) do not issue certificates for "localhost" since it is not a unique, publicly-owned domain. Sometimes, self-signed certificates are used instead, but these are not trusted by browsers by default.

A better alternative is to configure a browser to trust "localhost" independend of a certificate:

1. Open Google Chrome (or a similar browser) and type chrome://flags in the address bar
2. Now activate the settings shown below (you may have to search for them in the list the browser will present to you)
   
   The last setting is optional and only required if you want to access a different computer in your LAN at home
3. You may have to restart your browser afterwards

Current Status and Future Plans

As you can see in the video, the "blockly-ai-playground" is principally working. However, due to the haste with which development had to take place, the result should only be considered as a "minimum viable product" (MVP) as it still requires a lot of "polishing" before it will be as easy to use as it should be for beginners.

These are my plans for the near future:

• make sure that everything is working properly in the most important browsers and on the most important platforms (including tablets, but not smartphones - their displays are just too small)
• quick notes on how to get and install Ollama (perhaps together with Open-WebUI)
• create a tutorial for real beginners (perhaps with tutorial videos?)
• use a bundler (vite) to make a real web app out of the playground
• use some of the available Blockly plugins to enhance the user interface
• create a desktop application (NW.js or Electron.js)
• create installers for Windows/Linux/macOS, provide update paths (e.g., similar to the GitHub Desktop)
• detect browser security violations (TLS, CORS etc.) and give user-friendly error messages with suggestions how to fix them

The idea is to have a beginner-friendly version ready for the next lecture season (starting in October)

Custom Blocks

Besides common blocks which can be found in many Blockly environments, the "blockly-ai-playground" also provides a set of custom blocks to manage an internal context, a simple (but dynamic) user interface, basic and enhanced functions for AI, auxiliary tools for AI agents and other useful functions.

Playground Context

The playground manages a dynamic "context" which is a collection of "items", consisting of a "value" (a string) and its unique "name". Context items can store settings (e.g., the URL of an API server, its access key or the AI model to use) or intermediate results generated by an agent or they provide a link between an agent and its user interface.

Please note, that context item names are case-sensitive.

Some context items are already used internally, these are

• APIServer - the URL (and base path) of the OpenAI-compatible API you plan to use
• APIKey - your API Access Key (if required)
• SearXNGServer - the URL of your own SearXNG server (just leave it empty if you don't have one - in that case, a random public server will be looked-up and used instead)

the inferencing parameters

• default_model - the model to use for completions (no preset, as it highly depends on the configured API Server)
• default_max_tokens - the maximum number of completion tokens to be returned (preset to 2048)
• default_num_ctx - the maximum context size (no preset, if set it must be greater than "max_tokens")
• default_temperature - the "response randomness" setting (0...2, preset to 0)
• default_top_p - the "nucleus sampling threshold" setting (0...1, no preset)
• default_top_k - the "top-k filtering setting" setting (0...2048, preset to 1)
• default_presence_penalty - the "token presence penalty" setting (-2...2, preset to 1)
• default_frequency_penalty - the "token frequency penalty" setting (0..., no preset)
• default_SystemTemplate - the default template for the "system" message in a chat (preset to Be precise and concise.)
• default_UserTemplate - the default template for the "user" message in a chat (preset to {{Prompt}})

and

• Console - the current contents of the built-in "Console" (initially empty)

In addition to the items listed above, "AI Mezzanine" blocks provide their own context items in order to allow customization of their internal API calls. To find these items, simply replace the default prefix of the inferencing parameter items by the prefix for the given mezzanine (e.g., 'summarizer_max_tokens' specifies the token generation limit for text summarization). If any mezzanine-specific context item is removed (or empty) the related default setting is used instead.

Preserve and Restore

During development it is sometimes useful to keep settings and values beyond the life-time of a single program. Typical use case are:

• you use one Blockly program to define APIServer, APIKey or other settings and use these settings in other Blockly programs
• you save intermediate results of API calls in order to avoid having to rerun them after restarting an AI-assisted application

These two blocks provide such a functionality: use the first block to preserve a given context item and the second to restore it later

Context-related Blocks

returns a list with the names of all currently defined context items (see related example)

clears the context (i.e., deletes all currently defined context items) (see related example)

returns the current value of a given context item. If the requested item does not exist, undefined is returned (see related example)

sets the given context item to the given value. If the addressed item does not exist, it will be created (see related example)

removes the given context item. It is safe to remove an item which does not exist (see related example)

preserves the given context item (as a string) in the Browser Storage. If the addressed item does not exist, an empty string will be preserved (see related example)

restores the given context item (as a string) from the Browser Storage. If the addressed item was not preserved before, the context item will be set to an empty string (see related example)

Playground UI

The playground also manages a programmable reactive "user interface" (aka "UI"). The playground UI is basically a vertical list of user interface "elements", often preceeded by a "label".

Every UI element has a unique "name" which is bound to a context item with the same name. As a consequence,

• changing the value of a context item automatically updates the contents of a bound UI element and
• entering data into a UI element automatically updates the assciated context item

This concept makes UI programming super easy.

Event Loop

Often, a program has to react on user inputs. In the simplest way, this just means to present a few input fields to the user (e.g., to enter APIServer, APIKey and a Prompt) and to start inferencing as soon as the user has pressed a button. In the meantime, the UI may also be used to inform the user about the validity of any inputs while (s)he is entering them.

This is achieved by an "event loop".

The core of any event loop is one of the blocks shown above. Each of them waits for incoming UI "events" (which indicate that the user has entered some data or clicked on a button) and then executes a sequence of inner blocks (the so called "loop body").

Within the loop body you may now check what the user actually did and react accordingly - this is what the two blocks shown above are good for:

• the first one returns true if a button with the given name was pressed and
• the second one returns true if the user entered new data into an input element with the given name

Both blocks are typically placed into the condition field of an "if-then" block to run the "then" clause only if the given button was pressed or the given input element received new data.

The loop body is executed over and over again - until the loop is exited using the block shown above.

A Blockly program may contain several event loops - but only one of them may be running at the same tine (i.e., event loops must not be nested).

Enabling

Often, a programmer wants to prevent buttons to be clicked or new data entered into input fields. Typical situations are

• users should not be able to start inferencing by pressing a "Submit" button before APIServer, APIKey and Prompt were entered (and valid)
• users should not be able to interact with the UI while an API request is running

This behaviour is achieved by "enabling" or "disabling" either the UI as a whole or specific UI elements only.

The two blocks shown above enable or disable the playground UI in its entirety. Disabling the whole UI disables all UI elements, while "enabling" the whole UI means that the enabling of every UI element depends on its own setting (i.e., elements may still be individually disabled)

These two blocks enable or disable specific UI elements.

UI-related Blocks

switches to the playground workspace pane

switches to the playground user interface pane (see related example)

(re-)enables the playground UI in its entirety. Please note that "enabling the whole UI" only means that the UI is no longer disabled in its entirety and every UI element is now enabled or disabled depending on its own setting (see related example)

disables the playground UI in its entirety (see related example)

returns a list with the names of all currently defined UI elements

returns true if the UI currently contains an element with the given name - or false otherwise

clears the user interface (i.e., deletes all currently defined UI elements)

removes the element with the given name from the UI. It is safe to remove an element which does not exist

appends a textline input element (based on an HTML input element of type "text") with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, minLength, maxLength, Pattern, SpellChecking and Enabling (see related example)

appends a password input element (based on an HTML input element of type "password") with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, minLength, maxLength, Pattern and Enabling (see related example)

appends a number input element (based on an HTML input element of type "number") with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, Minimum, Maximum, Stepping and Enabling (see related example)

appends a URL input element (based on an HTML input element of type "url") with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, minLength, maxLength, Pattern and Enabling (see related example)

appends a (multiline) text input element (based on an HTML textarea) with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, minLength, maxLength, LineWrapping, SpellChecking and Enabling (see related example)

appends an input element (with the given name and label) to the UI that can be used to listen to and recognize what a user says into a microphone. If an element with the same name already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, minLength, maxLength, LineWrapping, SpellChecking and Enabling (see related example)

Nota bene: speech recognition requires microphone access - for that reason the browser may ask for permission to use your PC's microphone. Recognition will only work if you grant that permission - or else fail.

Important: not all browsers support speech recognition (Chrome seems to, but other browsers may not - even if they are based on webkit). Additionally, the playground must be served over valid (and secure!) HTTPS

appends a checkbox element (based on an HTML input element of type "checkbox") with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended. This element only supports the option Enabling (see related example)

appends a radio button group element (based on an HTML input elements of type "radio") with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended.

Radiobutton groups are a bit special as they contain a whole group of radio buttons, each with its own label. As a consequence, after appending a new radio button group element, you will have to define a list of radio buttons using the "append Choice" block - and the value of the whole group (i.e., the associated context item) is the (0-based) index of the currently checked radio button (or -1 if no button is checked) (see related example)

appends a drop-down element with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended

After adding a new DropDown to the UI, you still have to define the entries from which the user may select. This can be done using the "append Choice" block - one per selectable entry. The context item associated with the DropDown will then contain the (0-based) index of the currently selected entry (see related example)

appends a button element with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended (see related example)

appends a textline output element with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended (see related example)

appends a number output element with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended (see related example)

appends a (multiline) text output element with the given name and label to the UI. If an element with the same name already exists, it is removed before the new one is appended (see related example)

appends an unlabelled text view element with the given name to the UI. If an element with the same name already exists, it is removed before the new one is appended (see related example)

appends an unlabelled "fine print" view element with the given name to the UI. If an element with the same name already exists, it is removed before the new one is appended (see related example)

appends a URL input element customized for the selection or input of the APIServer context item with the given label to the UI. If an element with the name APIServer already exists, it is removed before the new one is appended. This element supports the options Placeholder, readonly, minLength, maxLength, Pattern and Enabling (see related example)

sets the current value of a given option for an UI element with the given key to a given value. If no element with the given key can be found, an exception is thrown and the program aborted. Which options and option values are meaningful (and, thus, used by the UI), depends on the type of the given UI element (see related example)

returns the current value of a given option for an UI element with the given key. If no element with the given key can be found, an exception is thrown and the program aborted. The requested option does not have to exist

returns true if an UI element with the given key is enabled or false otherwise. If no element with the given key can be found, an exception is thrown and the program aborted.

enables the UI element with the given key. If no element with the given key can be found, an exception is thrown and the program aborted (see related example)

disables the UI element with the given key. If no element with the given key can be found, an exception is thrown and the program aborted (see related example)

returns true if an UI element with the given key (exists and) is enabled. If no element with the given key can be found, an exception is thrown and the program aborted.

clears all currently defined choices of a radiobutton group or a drop-down element

appends the given text to the list of choices for a radiobutton group or a drop-down element (see related example)

clears the built-in "Console" (see related example).

appends the given string to the built-in "Console" (see related example)

appends the given string to the built-in "Console" and starts a new line (see related example)

removes all pending UI events from the internal queue. If there are no pending events, this statement is simply ignored

starts an event loop and runs the given statements whenever a new UI event was received. If the given timeout is 0 the loop is exited as soon as no more UI events are pending. If the given timeout is greater than 0 the loop is exited if no event was received within the given number of seconds. With a negative timeout, the loop waits indefinitely for the next UI event

starts an event loop and runs the given statements whenever a new UI event was received, waiting indefinitely for new events (see related example)

returns true if the currently processed UI event is a click on a button with the given name - or false otherwise (see related example)

returns true if the currently processed UI event indicates that new input for the UI element with the given name was received - or false otherwise (see related example)

leaves the current event loop

AI Basics

The following blocks perform basic AI functions

performs a chat completion for the given list of messages and returns the string created by the LLM. The message list must consist of an initial "system" message, a "user" message and (optionally) a sequence of "assistant" and "user" messages. If provided, any required options (such as model, temperature, top_k etc.) are taken from context items whose names start with the given option prefix. If such context items do not exist, are empty or the prefix is missing, any options are taken from context items whose names start with default_ (see related example)

performs a text completion for the given prompt. The prompt is used to generate a "system" and a "user" message which are then used to generate a "chat completion". The "system" message is generated from the SystemTemplate context item, the "user" message from UserTemplate. If provided, any required options (such as model, temperature, top_k etc.) are taken from context items whose names start with the given option prefix. If such context items do not exist, are empty or the prefix is missing, any options are taken from context items whose names start with default_ (see related example)

AI Mezzanines

The following blocks internally use AI requests themselves to perform some tasks which are often found in AI agents:

returns a summary of the given text. The block uses its own set of context items (prefixed with Summarizer_) to configure the inferencing parameters for text summarization - missing (or empty) items are taken from the default set (see related example)

generates a sequence of steps that have to be worked through in order to achieve a given goal. A single "step" is an object with the following properties:

• Tool: name of a selected tool (either ResearchTool, ComputationTool or ResponseTool)
• Objective: a short statement of what the chosen tool should achieve
• Justification: a brief explanation of why this tool and objective were chosen

The result of this block is a list of such steps (see related example)

uses the given "objective" to generate an adequate search phrase, search the internet, fetch all documents found by the search engine, extract any relevant information from these documents and produce a common result from all individual findings. Internally, this block uses the search phrase generator, information extractor and validator, solution merger and solution encoder described below (see related example)

generates a "phrase" which can be used to perform an internet search using a common search engine (or SearXNG) (see AI Web Search example)

a web search usually returns a list of URLs - one per document that seems relevant. This block now fetches one of theses documents from the given URL, "reads" it and extracts any information that seems relevant for the given objective (see AI Web Search example)

extracting information from a document may return nonsense. In order not to dilute the final result, this block checks if the given extract is actually relevant for the given objective or not. It returns true if it is or false otherwise (see AI Web Search example)

extracting information from several documents yields a list of separate extracts which may overlap in their contents. This block takes two such extracts and combines them into a single response relevant for a given objective (see AI Web Search example)

AI responses are often "noisy", i.e. they contain additional "decorative" words besides the actually requested information. This block therefore extracts the plain response and returns it as a list of facts (see example AI Web Search)

generates a JavaScript function to achieve the given "objective", executes it and returns the result. This block internally uses the parameter list, function and argument generators described below (see related example)

uses the current "knowledge" (taken from the context item Knowledge) and generates a (possibly empty) list of parameter specifications for a Javascript function to achieve the given "objective") (see related example)

uses the given parameter specification list and generates a JavaScript function to achieve the given "objective" (see related example)

uses the current "knowledge" (taken from the context item Knowledge) and generates an argument value suitable for the given parameter specification.

uses the current "knowledge" (taken from the context item Knowledge) and generates a human-readable response to the given "task".

takes a given "fact", generates a concise description based on the given "objective" that was used to find or compute this data and appends it to the current "knowledge" of the running AI agent. The new "knowledge" will be writen to the context item Knowledge but not automatically preserved.

AI Support

The following blocks have been made to support the creation of AI agents.

returns true if the given value represents valid agent "knowledge" - or false otherwise. Agent "knowledge" is a (possibly empty) list of JavaScript objects containing the following properties:

• Description: a concise description of what the actual fact represents
• Fact: the actual knowledge as a JSON-encoded JavaScript value

The Description will help the AI to find relevant facts for its task, while the Fact contains the actual data.

returns true if the given value represents a single working step for an AI agent - or false otherwise. A "working step" is a Javascript object containing the following properties:

• Tool: name of a selected tool (either ResearchTool, ComputationTool or ResponseTool)
• Objective: a short statement of what the chosen tool should achieve
• Justification: a brief explanation of why this tool and objective were chosen

returns true if the given value represents a sequence of working steps for an AI agent - or false otherwise.

returns true if the given value a single function parameter specification - or false otherwise. A "parameter specification" is a Javascript object containing the following properties:

Name: the parameter name (a valid JavaScript identifier) Description: a short description of the parameter's purpose

both settings can be used to create a JavaScript function and derive its argument values from the already given knowledge

returns true if the given value represents a list of function parameter specifications - or false otherwise.

returns true if the given value is a string containing a Wikipedia URL - or false otherwise

returns true if the given value is a string containing a Wikipedia page name - or false otherwise

converts the given HTML into formatted text

returns the first JSON object found in a given text - or undefined if no such JSON specification could be found (or the specification was incorrect)

fetches the (english) Wikipedia article with the given page name and returns its contents as plain text

searches the web for documents related to a given search phrase and returns a list with the URLs of the first found documents. If context item SearXNGServer contains a valid URL, the search request is sent to that server and only documents with a score greater than or equal to the threshold specified in context item SearXNGScoreThreshold (a number in the range 0...10 which defaults to 2) will be listed. Otherwise, a random public server will be looked up and used to perform the search. In both cases, up to 10 URLs will be returned, at most.

searches the (english) Wikipedia for articles related to a given search phrase and returns a list with the Wikipedia page names of the first found articles

Miscellany

The following blocks do not fit into the categories shown before - may may still be helpful when creating agents:

this block has no specific functionality but serves as a visual separator between adjacent blocks only - this may help identifying related blocks

throws the given exception (which is then shown in an alert dialog) and aborts the running program (see related example)

waits for a given number of seconds and continues (see Console example)

opens a browser "alert" pop-up dialog, displays the given message and continues as soon as the user has closed the dialog (see related example)

opens a browser "confirmation" pop-up dialog, displays the given message and continues as soon as the user has responded with "ok" or "Cancel". If cancelled, the block returns false, otherwise it returns true (see related example)

opens a browser "prompt" pop-up dialog, displays the given message and continues as soon as the user has entered a response. If cancelled, the block returns an empty string, otherwise it returns the entered string (see related example)

uses text-to-speech to speak the given (english!) message (see related example)

returns true if the given value contains a string that is neither empty nor contains whitespace only - or false otherwise

returns true if the given value contains a number - or false otherwise

returns true if the given value contains a number with a value ranging from the given minimum to the given maximum - or false otherwise. The additional arguments "withMinimum" and "withMaximum" specify wether the given "minimum" and "maximum" values are themselves part of that range or not

returns true if the given value contains an integer number - or false otherwise

returns true if the given value contains an integer number with a value ranging from the given minimum to the given maximum - or false otherwise

returns true if the given value contains an "ordinal" number (i.e., an integer greater than or equal to 0) - or false otherwise

returns true if the given value contains a "cardinal" number (i.e., an integer greater than or equal to 1) - or false otherwise

returns true if the given value contains a string that matches a given JavaScript regular expression - or false otherwise

returns true if the given value contains a string that looks like a URL - or false otherwise

returns true if the given value contains a string that looks like a Wikipedia document URL - or false otherwise

Object Support

The following blocks help to create and manipulate objects

returns a newly created empty object (see related example)

returns a (possibly empty) list with the keys of all own properties of a given object (see related example)

returns true if the given object contains a proeprty with the given key - or false otherwise (see related example)

returns the current value of the given object's property with the given key (see related example)

sets the given object's property with the given key to a given value (see related example)

deletes the given object's property with the given key (see related example)

Examples

Here are a few examples which you can upload into the Blockly workspace to get familiar with Blockly itself and the custom blocks specifically made for the "blockly-ai-playground". The linked Blockly workspace files should first be downloaded onto your computer and then uploaded into your Blockly workspace

Hello, World

this is the typical "Hello, World" example (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Alert, Confirm, Prompt

if you do not want to implement a "real" user interface (with an event loop), it may already be sufficient to use the browser's built-in pop-up dialogs alert, confirm and prompt - for simple use cases, at least. This example demonstrates how to use them (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Console

this example demonstrates all Console functions (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Suspend, Resume, Abort

use this little Blockly program to get familiar with "Suspend", "Resume" and "Abort" (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

throw

this example illustrates how to use the "throw" block (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace) - please note, that all blocks following a "throw" will not be evaluated as throwing an exception will terminate a Blockly program

Context

this example demonstrates all local context functions (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

preserve and restore

this example demonstrates how to preserve and restore context items (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Input Elements

this example shows all available input elements (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Textline Input

this example shows textline input elements in several variations (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace). the other input elements work quite similarly

SpeechInput

this example demonstrates speech recognition using the Browser's built-in WebSpeech API (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Checkbox

Checkboxes support context values 'true' and 'false' (actually, any value different from 'true' is considered 'false') (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

if you want to react on changes of a checkbox's state, just check for new user input.

Radiobutton Group

Radiobutton groups may manage an arbitrary number of individual radio buttons, from which at most one may be checked at any time. These buttons have to be added using "append Choice" blocks (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace). The value of a radiobutton group is then the (0-based) index of the checked radiobutton - any invalid context value will uncheck all buttons.

if you want to react on changes of a radiobutton group's state, just check for new user input.

DropDown

Drop-down menus may manage an arbitrary number of separate choices, from which at most one may be selected at any time. These choices have to be added using "append Choice" blocks (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace). The value of a drop-down menu is then the (0-based) index of the selected choice - any invalid context value will deselect all choices.

if you want to react on changes of a drop-down menu's state, just check for new user input.

APIServer Input

the "APIServerInput" element is just a special URLInput element: in contrast to the normal one it adds a drop-down menu with predefined URls for common API servers (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace).

here you see what an APIServerInput element looks like

Output Elements

this example shows all output elements of the playground UI (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

here you see what the output elements look like

enable and disable

this example demonstrates the effects of enabling or diabling either the whole UI or individual UI elements (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Event Loop

user input and button clicks are handled within "event loops". This example illustrates how you can implement a simple program which waits for the user to enter some text, enables a "Start" button whenever such text exists and - after clicking the button - disables the whole user interface in order to avoid unwanted changes while the computer does some lengthy processing, displays any results and then re-enables the user interface again (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

here is what the user interface looks like - of course, the example does not implement any real "processing" but simply uses a short delay to create the illusion of some lengthy work.

Button Clicks

this example demonstrates how to react on a button click (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Objects

this example demonstrates how to work with objects (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Text-to-Speech

this example demonstrates text-to-speech synthesis using the Browser's built-in WebSpeech API (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

WebSpeech

this example illustrates how to use the Browser's built-in WebSpeech API (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

Web Search

the "blockly-ai-playground" can search the web for information which may be relevant to perform a given task. This example illustrates how such a search works

here is what you may get when searching for "Joseph Weizenbaum"

Web Document Retrieval

after having identified which web documents could be relevant for a certain task, this document has to be loaded and converted to plain text so that the AI will be able to process it. Binary files and other kinds of documents will have to be ignored (i.e., they will just result in an empty text string '') - right now, the text loader only supports documents with the MIME type text/*

here is the output after reading a document decribing the "Weizenbaum Institute" in Germany.

Wikipedia Search

Wikipedia is often a good source for reliable information. This example demonstrates how to search Wikipedia based on a given search phrase

here is the list of Wikipedia articles you may get when searching fo "Joseph Weizenbaum"

Wikipedia Article Retrieval

after having identified a relevant Wikipedia article, that document has to be loaded and prepared for the AI to be processed. This example demonstrates how this can be achieved (loading Wikipedia articles is a bit more complicated than loading a plain text or HTML file because the actual content is surrounded by a lot of "boilerplate" such as Wikipedia-related links)

here you see the beginnings of a Wikipedia article about Joseph Weizenbaum

Conversion of HTML to Text

HTML is not what an AI should process directly - it's actual contents should first be converted into plain text, perhaps with some proper formatting for titles, paragraphs and links to help the AI identifying them more reliably. This example demonstrates how to perform this conversion

here you see the conversion result

Extraction of JSON from a Text

even when instructed to output plain JSON only, an AI may still "decorate" JSON output with explanations and comments. In this example you will see how to extract a JSON specification from a given text

and here you see the actually extracted JSON specification

AI Examples

The following examples actually include AI-related blocks. As before, the Blockly workspace files should first be downloaded onto your computer and then uploaded into your Blockly workspace

Basic Settings

use this example to enter your basic AI-related settings - they will be stored in the browser's storage and may then be used by other programs without having to re-enter them over and over again (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

as a hint, here are the settings used by the author himself

Text Completion (Proof-of-Concept)

this is the "proof-of-concept" mentioned above - play around as you like! (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

this is what you may get as the result

Chat Completion

here is an example for a simple ChatCompletion (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

this is what you the result may look like

Text Summary

this example summarizes a given text (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

this is the author's run to let the AI summarize the beginnings of "Alice in Wonderland" - your mileage may vary, of course

AI Research Tool

this is a demonstraton of the built-in "ResearchTool" for AI Agents: enter what you are looking for and let SearXNG and the AI look for relevant documents, read them for you and present a summary of what the system learned (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

it's definitely not easy to find a good answer for the question "to which programming languages has Joseph Weizenbaum's famous Eliza program been ported?" (even if you try yourself) - here is what the author's attempt produced as output: definitely incomplete, but not bad for just a few seconds of work...

AI Web Search

you don't have to use the built-in "ResearchTool", you may also implement a similar functionality yourself: this example illustrates how to use several AI Mezzanines to implement an AI-assisted web search (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

By the way: the objective was "to which programming languages has Joseph Weizenbaum's famous "Eliza" program be ported?"

this is what the example may produce as output

AI Search Phrase Generation

if you want to conduct a web research for a given topic, you may have to generate a search phrase first - this example does that for you (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

and here you see what a search phrase may look like

AI Information Extraction

if you have a text that contains information you need for a given task, you want the AI to read that text and extract what is relevant - here is an example of how to do that (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

here you have an example for a useful information extraction

AI Information Validation

letting the AI extract relevant information from a given text may sound nice, but in fact, the AI often produces complete nonsens - here is an example how you can let the AI validate the actual relevance of its output itself (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

useful excerpts should display "true", nonsense "false"

AI Computation Tool

the "Computation Tool" takes an "Objective" and uses the already existing "Knowledge" to create and execute a JavaScript function that tries to compute the requested result

this is what you may get after running the above example

AI Parameter List Generation

letting the AI generate and execute a Javascript function to achieve a given objective first requires a well-defined list of parameters so that the function knows what data will be passed as its arguments and the AI knows what kind of argument list to create. Here is an example how you can generate such a parameter list (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

a parameter's "Index" property refers to the knowledge list entry from which the parameter was derived.

AI Code Generation and Execution

this examples shows how to let the AI generate a JavaScript function to compute the given objective based on a list of parameters (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

sometimes, the generated code is really impressive...

AI Task Decomposer (without existing knowledge)

at the core of an AI agent there is a component which breaks down a given task into smaller steps which may then be worked through. This example demonstrates how the AI does so from scratch (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

this is what you may get - not too bad!

AI Task Decomposer (using already existing knowledge)

often, an AI agent does not have to start its work from scratch but may use already existing "knowledge" - here is an example for generating the remaining steps to achieve a given objective in such a situation (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

if you already liked the previous example, this one will impress you: yes, the AI is really able to take existing knowledge into account!

AI Knowledge Generator

both the "ResearchTool" and the "ComputationTool" produce new "facts" which have to be added to existing knowledge for an AI agent to continue its work and (in the end) produce a response. This example shows how that can be done (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

this is what "knowledge" generated from a "fact" may look like. In this case, the "fact" is already part of the description - which wasn't planned, but does not hurt.

AI Answer Generator

every run of an AI agent should finally produce a response - this example demonstrates how to do so (see Blockly workspace file - download it onto your computer and then upload it into the Blockly workspace)

formulating sentences is something that AI is good at.

AI Agent

this example contains a full-blown AI agent (it's a bit like your own, personal "Copilot"): give it a "task" and watch how it decomposes it into smaller steps it is then working through. The agent has two "tools" at hand: a "ResearchTool" that can conduct web researches (like the AI Web Search from above) and a "ComputationTool" which creates and runs a JavaScript function that is used whenever some information has to be computed rather than looked up in the Internet. The outcome of each step is added to the agent's "knowledge" and stored in the browser - this gives the agent the possibility to skip over already completed steps when it was aborted and restarted later (provided that the "task" was not changed in between)

this is what you may get as a result. Feel free to compare this output with what MS Copilot or Perplexity AI will tell you - and be impressed! Nota bene: your mileage will definitely vary from run to run - use that experience to treat every response from an AI (or an AI Agent) with care, even if you asked a "professional" one!

DIY AI Applications

Now that you saw numerous examples for the custom blocks provided by the "blockly-ai-playground", here are some simple but useful examples of actual AI-assisted applications you (or your students) could build yourself/themselves.

Translator

this is a simple application which translates a given text into english. If you replace text input and output by the blocks for speech recognition and synthesis, you may even talk to that application - isn't that cool?

and here is how a translation run could look like

License

MIT License