Request for help implementing HID protocol in BrailleTouch project

[discapacidad5]

Hello,

I hope this message finds you well. I am reaching out to you because I am working on an open-source project called BrailleTouch. Our goal is to create affordable Braille displays for blind and deaf-blind individuals.

We are currently stuck and unable to get the HID Braille to receive information from the screen reader and continue programming. As someone with experience in this area, your expertise could be of great help to our project.

I believe that an open-source Braille display could be incredibly useful and have many benefits for those who are blind and deaf-blind. Your knowledge of the HID protocol could be particularly helpful in implementing the protocol in our Braille display.

Please take a moment to review our project on GitHub at https://github.com/brailletouch. If you're interested in helping, please let me know.

Thank you for your time and consideration. I look forward to hearing back from you.

Best regards,

Cergio Monasterio

[seanbudd]

Have you read our documentation on braille HID?
https://github.com/nvaccess/nvda/blob/master/devDocs/hidBrailleTechnicalNotes.md

[discapacidad5]

We have reviewed the documentation, but we have not found any practical examples that could assist us. There is no code example for a Braille display using HID anywhere on the internet. This complicates matters because without an example, it becomes difficult, especially for those who are not familiar with HID. It would be practical to have an example code. In fact, what we are looking for is to create a library for HID Braille, similar to the libraries available for Arduino and ESP32, that converts it into a keyboard. Users can simply load the keys they want, and the library will generate the reports and all other necessary tasks. It would be ideal to have a HID Braille library that can create Braille displays of different sizes, for example, a 20-cell display that uses specific pins, and the library can create the corresponding report list. If someone could provide an example code with the main buttons connected to some of the ESP32 pins, such as the button to read the next or previous message, forward and backward arrows, and serial console output that displays the Braille content. It would be very helpful to have this example code. This would allow us to connect it to the Braille reader, display the Braille on the console, and navigate the text using buttons. This would be a good starting point to continue working with all the required functions. I have been requesting assistance for over a year and have sought help from many programmers. However, none of them have been able to provide any code. Currently, we have been struggling for almost a week, spending entire nights trying to generate code and understand the explanations, but have not been successful.

[XLTechie]

CC @LeonarddeR I'm not sure, but it is possible you might have some knowledge that would guide this effort in some better direction.

[LeonarddeR]

I'm afraid not. I have only a small amount of knowledge about the display driver side of things, definitely not about how it works in the display itself (i.e. how a display should behave like a braille HID driver).

[discapacidad5]

This has been the problem we've been facing, and I've been searching for programmers, but none have been able to help because it seems that no one has seen code for HID braille on the braille display (firmware). Would it be necessary to reverse engineer a braille display? Apparently, the part that goes on the screen reader's side (the driver) is known, but no one knows how to do the firmware or the HID part that goes from the braille display side. There is no starting example. It would be a great advancement to have a sample HID braille display code in C that can display braille content on the console and have at least two buttons to navigate through the content. With this code, we could show how to pass buttons and receive and display content on the console. It would be a perfect example for developing anything, but unfortunately, it doesn't exist. I even tried using artificial intelligence to generate code, but it also failed because everyone only works on the code that goes to a controller for making a controller, and AI does the same.

[Adriani90]

cc: @FalkoBabbage

@discapacidad5 which HID library did you use on your platine?
My understanding of HID is that you wouldn't really need a screen reader at all if a HID braille display is really a HID device, so it would just have a standard for input and for output, similar to a touch screen.

Maybe these sources could help in how to design the class and the internal library on the platine to comunicate with the HiD driver in NVDA or with Windows independently. I guess you need these enhanced libraries for your platine to comunicate via HID efficiently:
https://github.com/NicoHood/HID

and this is also very useful since you can specify your own HID classes with your own enums:
https://github.com/dlkj/usbd-human-interface-device
Actually this is probably what you need. Specific Enums, standards for Input and Output for the braille display and specific HID classes. If specifics are also recognized by Windows, then NVDA should recognize them as well.

Does your braille display work already without any screen reader? Or is a screen reader required to use it?
If it is required, is there any screen reader that already works for you?

This treat might help further in finding some hints for functions required to be written to your HID device in order to comunicate with Windows.
https://forums.freebsd.org/threads/confused-about-usb-hid-devices.24477/

[discapacidad5]

"Thank you for responding, but let me tell you that I have tried all the libraries related to HID that I could find, and I have tried every possible method that I could imagine. I have also looked for programmers who could help me, but none of them have been able to provide a solution. They all give me theoretical advice on what to do, but there is a big gap between theory and practice. What I really need is someone who can apply their theory and show me an example code that actually works. The computer recognizes the device as HID, so the problem is not with the computer recognizing it. The problem is with the screen reader recognizing it correctly and sending content to it.

I have managed to write some code that makes the computer recognize the device as HID, but NVDA doesn't recognize anything. Obviously, there are specific parameters that need to be placed in certain locations, and these can only be achieved with an example code that shows how to set those parameters.

On the other hand, the HID examples that I have found online only show how to send key press data from the device to the computer, but there is no example on how to receive data from the computer to the device, which is the case for Braille where I need to receive the content of the Braille cells."

[discapacidad5]

"In the case of the projects you mentioned, such as the Android HID project and USBD-Human-Interface, they provide HID support only for USB devices and do not work with Bluetooth devices. In my case, I am facing a problem with the ESP32, which does not have native USB support for HID Braille. I can only work with it through Bluetooth.

However, as I mentioned before, the problem is not that the computer does not recognize the device as HID. The problem is that NVDA, or more specifically HID Braille, has a specific structure to define the cells and many other things, which is not the same as the structures for a keyboard or a mouse. These libraries that convert HID do not have a structure for HID Braille. Therefore, even if I manually change the descriptor and add the HID and Braille descriptor, the computer recognizes it but the screen reader does not, because there are many things that are not properly structured.

That's why it is necessary to create a specific library for HID Braille or add HID Braille support to the existing library."

[Adriani90]

It seems Helptech from Germany they claim at least some of their braille devices are pure HID devices:
https://www.helptech.eu/hid-braille-support

[Adriani90]

I guess you already used the HID conventions in this HID usage table right?
https://www.usb.org/sites/default/files/hut1_4.pdf

[discapacidad5]

"If that table specifies how to create the HID structure, it doesn't explain how to work with it, how to send content, how to receive content, or where to load the Braille cells. There is no information on how to update or send data to the screen reader, what needs to be sent, or received. Simply specifying the structure allows the device to be recognized as HID on the computer, but the screen reader will not recognize it as HID without a proper report including information on the number and type of Braille cells (whether 8 or 6 points), and other details that are needed for the screen reader to process the information correctly. None of this information is specified in the table."

[discapacidad5]

"This code using the HID structure recognizes the device as HID on the computer, but the screen reader does not recognize anything. Obviously, the HID structure is only a part of what is needed and other information is missing from the document."
mi código

[josephsl]

Hi,

Hi,

I'm sure you may have read through NVDA's own source code, but have you tried looking at HID braille standard file (source/brailleDisplayDrivers/hidBrailleStandard.py) to get a sense of how NVDA understands HID protocol?

Thanks.

[discapacidad5]

"Well, to be honest, I have read all the files related to HID personally, but I couldn't understand them because it seems to explain how the reader reads the content, but there seems to be a huge difference between how it is read and how it is sent. Apparently, the structure that one sends for HID is sent in one way for the computer to process it and delivered in another way, so apparently, the structure that is sent is not the same structure that the screen reader receives. In theory, it could be said that if I understand how the screen reader reads it, I will understand how to send it, but you can try it in practice and see that it's not as simple as the theory says. I don't know much about programming, but I have searched for many people and we have all been struggling with the same problem and none of us have been able to do anything."

[discapacidad5]

The big problem lies in the fact that those who provide explanations and the manufacturers never explain how to send the HiD report. They explain how to interpret it and there are instructions on how to interpret the HiD report to use it on a screen reader to create a driver, but there is no information on how to create or generate the HiD report itself, only its interpretation and structure. Sometimes I think there is a great interest in preventing anyone from understanding this, so that no one can develop a screen that can be compatible with all screen readers. I have always believed that Braille screens end up being a business. They are so expensive and there are ways to make Braille screens so affordable, but if there is no way to communicate with the screen reader, what is the point of manufacturing a screen if it cannot be functional with screen readers or if I have to wait a long time for a custom driver for a screen reader. And of course, if I want to try to bring it to paid readers like Jaws or iPhone or even Android, how long do I have to wait to see if my device driver can be recognized in their device list? The advantage of HiD is precisely that it would give that freedom to play with all this and start designing, testing and evaluating Braille screens that work without waiting for a custom driver for a screen reader. Having a working sample code would open the doors to many projects that could use HiD to offer more accessible devices to people with disabilities. Today, people with disabilities are not using Braille precisely because of the cost of the devices and only first-world countries that have the capacity or whose governments donate the devices have the ability to have a Braille screen. The rest of the people in the world do not have that possibility, and that is the vast majority.

[ruifontes]

Sorry if I am being dummy... The HID protocol is not only for USB devices? If you say that ESP32 only supports Bluetooth, you should not devellop a specific driver for your Braille display? Rui Fontes Às 21:16 de 15/04/2023, discapacidad5 escreveu:

…

"In the case of the projects you mentioned, such as the Android HID project and USBD-Human-Interface, they provide HID support only for USB devices and do not work with Bluetooth devices. In my case, I am facing a problem with the ESP32, which does not have native USB support for HID Braille. I can only work with it through Bluetooth. However, as I mentioned before, the problem is not that the computer does not recognize the device as HID. The problem is that NVDA, or more specifically HID Braille, has a specific structure to define the cells and many other things, which is not the same as the structures for a keyboard or a mouse. These libraries that convert HID do not have a structure for HID Braille. Therefore, even if I manually change the descriptor and add the HID and Braille descriptor, the computer recognizes it but the screen reader does not, because there are many things that are not properly structured. That's why it is necessary to create a specific library for HID Braille or add HID Braille support to the existing library."

[discapacidad5]

"The HID protocol is a communication protocol that works on both USB and Bluetooth. In fact, that's why there are Bluetooth keyboards and mice as well as gaming devices that work via Bluetooth, and the computer recognizes them perfectly as HID devices. So, HID devices are not only for USB. In fact, the code I posted in a previous response, if you put it on an SP32, the computer will recognize it as a Bluetooth HID device."

[discapacidad5]

"I think the real problem in developing HID Braille display drivers is the lack of understanding of how the HID protocol works. There isn't a deep knowledge of the protocol, which is why even very skilled programmers haven't been able to do anything with it. It would require someone who specializes in the HID protocol, and it seems that there are few people who master it or it's difficult to find someone who does. It would be interesting if some of the programmers showed more interest in this, especially since the future expects all Braille displays to exclusively use HID. The hope for the future is that even operating systems will automatically recognize a Braille display and connect it without needing a screen reader. When the HID protocol was created, it was done with that intention, just like when you connect a keyboard, whether it's Bluetooth or USB, both work with the HID protocol and are perfectly recognized by the device you connect it to. However, the population that works to develop Braille devices is very small, and in the end, the ones who end up developing direct drivers are the manufacturers. The manufacturer, as such, is not interested in having their code seen by another manufacturer. So we are in a situation where there is no example of the code currently available in open source, and my post is precisely calling for help to see if someone can do something that gives us a starting point, a code that gives us a point to build a good driver on the Braille display side."

[Adriani90]

@discapacidad5 I am totally with you, sometimes big players announce a lot of things such as standards but often they are not being thought to the end. Especially when it comes to guidance on how to achieve them.

I am not total expert, but it seems you have found some how a way to design your in report and now you need an out report for the report descriptor. Is this correct?

So can you already send key presses to an application in Windows without any screen reader?

Anyway, I found a good tutorial on how to create a USB report descriptor, it defines in general how to create it for some usual examples like gamepads and mouse. Now we have to find examples for some code that describes a report descriptor for bluetooth devices and then for a HID braille device.
https://eleccelerator.com/tutorial-about-usb-hid-report-descriptors/

And here is a tutorial which shows how to create custom HID devices with custom report descriptors in circuitpython.
https://learn.adafruit.com/custom-hid-devices-in-circuitpython/report-descriptors

I could find an usage file that has been approved from Microsoft for the HID standard, I guess this is the filtered information from the table that I have already posted above. It could make it easier to find here all the relevant info without having to scroll through the whole HID conventions:
https://usb.org/sites/default/files/hutrr78_-_creation_of_a_braille_display_usage_page_0.pdf

@discapacidad5 did you already try the win-hid-dump tool to extract the propper HID report descriptor for the custom device? I guess this will give you an in report rather than an out report but might be worthy to try.
https://todbot.com/blog/2021/01/29/get-hid-report-descriptors-with-win-hid-dump-mac-hid-dump/

Then you could use a report descriptor decoder such as this one to create C structures:
https://sourceforge.net/projects/hidrdd/

[Adriani90]

I found on stack overflow a code example for sending and receiving data on a custom HID device, but not sure if it works:

uint8_t CUSTOM_HID_ReportDesc[REPORT_DESC_SIZE] =
{
   0x06, 0x00, 0xFF,    // Global  Usage page = 0xFF00 (Vendor-defined pages are in the range 0xFF00 through 0xFFFF)
   0x09, 0x01,          // Local   Usage (vendor usage 1)
   0xA1, 0x01,          // Main    Collection (application) begin
   0x15, 0x00,          // Global  Logical minimum (0) applies to each byte
   0x26, 0xFF, 0x00,    // Global  Logical maximum (255)
   0x75, 0x08,          // Global  Report Size (8 bits)

   // 14 bytes | Output message 1 (sent from host to device)
   0x85,  1,            // Global  Report ID (cannot be 0)
   0x98, 64,            // Global  Report Count (number of Report Size fields, in this case 64 bytes)
   0x19, 0x01,          // Local   Usage Minimum (each Report Count must be associated with a Usage)
   0x19, 0x40,          // Local   Usage Maximum
   0x91, 0x02,          // Main    Output (data, array, absolute)

   // 24 bytes | Input message 1 (sent from device to host)
   0x85,  1,            // Global  Report ID (cannot be 0)
   0x98, 64,            // Global  Report Count (number of Report Size fields)
   0x19, 0x01,          // Local   Usage Minimum (each Report Count must be associated with a Usage)
   0x19, 0x40,          // Local   Usage Maximum
   0x81, 0x02,          // Main    Input (data, array, absolute)

   // 34 bytes | Output message 2 (sent from host to device)
   0x85,  2,            // Global  Report ID (cannot be 0)
   0x98, 12,            // Global  Report Count (number of Report Size fields)
   0x19, 0x01,          // Local   Usage Minimum (each Report Count must be associated with a Usage)
   0x19, 0x40,          // Local   Usage Maximum
   0x91, 0x02,          // Main    Output (data, array, absolute)

   // 44 bytes | Input message 2 (sent from device to host)
   0x85,  2,            // Global  Report ID (cannot be 0)
   0x98, 57,            // Global  Report Count (number of Report Size fields)
   0x19, 0x01,          // Local   Usage Minimum (each Report Count must be associated with a Usage)
   0x19, 0x40,          // Local   Usage Maximum
   0x81, 0x02,          // Main    Input (data, array, absolute)

   // 54 bytes | End (add one byte)
   0xC0                 // Main    Collection (application) end
}

A couple of things to note:
• More input/output pairs can easily be added: just give them another Report ID. Each message definition consists of 10 bytes, so it's simple to add up.
• We keep track of the number of bytes in the descriptor so that the size of the array can be computed (#define REPORT_DESC_SIZE (55)).

Source: https://stackoverflow.com/questions/21606991/custom-hid-device-hid-report-descriptor

[Adriani90]

@discapacidad5 did you raise your questions on this forum ast well?
https://e2e.ti.com/support/wireless-connectivity/bluetooth-group/bluetooth/f/bluetooth-forum/485430/how-to-get-hid-report-descriptor-of-connected-device

I have the impression there could be some really good experts that can help further in this regard.

[Adriani90]

For anyone's reference, here is an example of a bluetooth device that uses the Gatt profile to comunicate with the host:
https://docs.silabs.com/bluetooth/2.13/code-examples/applications/ble-hid-keyboard

It also describes how a report descriptor for a bluetooth device could look like.

[Adriani90]

And here is a complete report descriptor for a smart remote which might also help.
https://infocenter.nordicsemi.com/index.jsp?topic=%2Fcom.nordic.infocenter.sr3nrf52.firmware%2F_h_i_d.html

[Adriani90]

Finally, here is some more Info on how HID input and output can works between HID device and host, here is a dualshock game controller as an example:.
https://docs.unity3d.com/Packages/com.unity.inputsystem@1.5/manual/HID.html

[discapacidad5]

"Thank you for all the information you've sent, I will look into it to see if I can achieve something. However, I have seen many examples and have actually been able to make a HID device work. There are many examples of how a HID device works and there are libraries that do the job. But what I have noticed is that even if you have a HID device configured, it does not work with the screen reader. In fact, key presses on the HID Braille device do not go directly to the operating system like on a normal keyboard. The key presses are captured by the screen reader. Currently, the operating system does not recognize the HID Braille to work with it directly. That is the big problem. That's why I mentioned that it's not about just passing examples. There are many examples, but it's about making an example that works with HID Braille. When you start working on it, you realize that existing examples do not work with HID Braille because it's not enough to be recognized by the computer. It has to carry certain structures that the screen reader must recognize. All key presses have to go to the screen reader, and the screen reader looks for the desktop in a specific location. Therefore, both the input and output go to the screen reader, and the screen reader processes it. I will analyze all the examples you sent, but even those that claim to provide an example of a descriptor do so theoretically. No one has done it in practice. The theory is not enough when thinking about how HID works. Braille input is a bit different and requires certain different operating parameters. I encourage you to try to make a functional code that can receive the content of Braille and can be recognized by the screen reader, not just recognized by the operating system, as that can already be done. There are already libraries that help with that, and you simply need to load the library to be automatically recognized. But it's not enough to be recognized by the operating system. The goal is to make the screen reader recognize the HID Braille, and that's where we are falling short. There is a part of the code that is missing, something needs to be done, and there are some parameters that are not being set. Therefore, the screen reader is not recognizing it, and that's the unclear part. Everyone explains it theoretically, but no one provides a practical example. Sometimes, a practical example is necessary to understand."

[josephsl]

Hi,

Hmmm, even after reading NVDA's own source code? If yes, then we might as well discuss how we (NVDA folks) can make our source code better so others can understand what's going on, because when discussing HID and screen readers, NVDA's own HID standard implementation is the closest we have to seeing the protocol in action.

As long as a screen reader (or some assistive tech software) is running, the screen reader will process input before anyone can do so, and that also involves understanding input coming from a braille display as communicated by its driver. In case of HID, HID input is first processed by the screen reader, therefore the screen reader's job is to parse protocol messages from the braille device and translate it into actual input mechanism the operating system supports.

As for actually writing support for a braille display, yes, it involves looking at or asking for basic protocol documentation, and lots of debugging to get it right (more so if one wishes to support multiple models). The last time I did something with braille input and output, I have added support for specific models of HumanWare BrailleNote family, involving different models and input styles. This work involved printing protocol messages sent by the braille display, mapping messages into a form that could be understood by NVDA (key mappings, for example), and getting NVDA to recognize specific input and perform commands and/or send the equivalent command to the operating system.

As for braille and HID, what makes it a bit difficult is that the input side is not seen as a typical keyboard - it ic akin to a controller or a specialized input device. If we're dealing with a computer keyboard, then it is better to get the OS/keyboard driver to recognize the input and do whatever it needs to do. But we're talking about an input style that cannot be readily translated into scan codes and virtual key codes (that's how Windows knows about keyboard input), thus the screen reader is involved in:

1. Understand the HID protocol message.
2. Map it to its own representation.
3. Define a mapping between HID representation from the braille device and commands the screen reader will perform and/or keyboard input the screen reader will send to the operating system.
4. Parse the HID message representation looking for commands it can perform.
5. If not, feed the message representation (in dots) into Liblouis, the library responsible for back-translating braille dots into actual letters and symbols based on the active input braille table. Only after back-translating braille input into something that can be processed such as symbols can the input from the braille display that is not a screen reader command can be shown on computer output.

In other words, the way the term "human interface device" is understood, interpreted, and implemented differs based on who you ask:

• Operating system developers: a part of the OS responsible for recognizing and processing messages from devices declaring themselves as HID compatible.
• Driver developers: a way to describe the device in question as well as protocol to send input to the operating system and sometimes work with output representation.
• Braille display manufacturers and developers: describing the capabilities of the device and implementing messages at the device level.
• Screen reader developers: a way to detect braille hardware and communicate with, understand, process, and to facilitate input and output between the hardware and the operating system.

As a repository for screen reader development, we fall under the last category, with some folks wearing different hats.

Hope all of this makes sense.

Thanks.

[discapacidad5]

That's what I've been understanding over time as I've been looking into it, that depending on the perspective you approach it from, the focus is different. I see that the available documentation is for developing a driver that understands a braille display, but the perspective from the braille display (firmware) is different. From what you're saying in this message, I understand that not all braille displays send information in the same way. They use the HID protocol but have their own commands. I'm not sure if I understood that part correctly, but if so, then unification still hasn't been achieved because it would still be necessary to make the screen reader able to understand each manufacturer's commands. My proposal is to create a public unified command set that allows for specific commands to be developed for a braille display. This document should be made public so that anyone who wants to develop something with those commands can do so, and to allow the screen reader to understand those commands. This way, we would be developing a standard within the HID protocol that anyone looking for information can find and use. But as long as everyone works separately and creates their own commands, we will never achieve the true goal of the HID braille protocol, which is plug and play functionality. While it may be recognized by the operating system, being recognized by the operating system alone is not enough for it to be functional. It would be good to be able to develop a communication standard with the basic buttons required for a braille display and explained documentation on how to implement it with example code that serves as the foundation for anyone to use those commands to implement it in their development.

Something very similar is happening with regular drivers, each manufacturer has its own protocol and commands. So, for each device, an individual driver must be created. In fact, that is the big problem. Hopefully, there would be documentation with a universal driver that anyone could use code or a regular driver with all the documentation so that anyone could replicate the same driver and use it. For example, an open source driver with a recognition standard called "open" for Bluetooth, and any device that has the word "open" at the beginning of the Bluetooth name will be recognized as a device that uses that driver. I think that by implementing it in a screen reader and implementing the first braille display that uses that driver, it would open the door for the driver to be used in the rest of the screen readers. Then, you could truly have the possibility of working with an open source screen reader software and hardware.

[discapacidad5]

@Adriani90

Eso es lo que he ido entendiendo a lo largo del tiempo mientras lo investigaba, que dependiendo de la perspectiva desde la que lo abordes, el enfoque es diferente. Veo que la documentación disponible es para desarrollar un controlador que comprenda una pantalla braille, pero la perspectiva desde la pantalla braille (firmware) es diferente. Por lo que dices en este mensaje, entiendo que no todas las pantallas braille envían información de la misma manera. Usan el protocolo HID pero tienen sus propios comandos. No estoy seguro de haber entendido bien esa parte, pero si es así, todavía no se ha logrado la unificación porque aún sería necesario que el lector de pantalla fuera capaz de comprender los comandos de cada fabricante. Mi propuesta es crear un conjunto de comandos unificados públicos que permitan desarrollar comandos específicos para una pantalla braille. Este documento debe hacerse público para que cualquier persona que quiera desarrollar algo con esos comandos pueda hacerlo, y para permitir que el lector de pantalla entienda esos comandos. De esta forma, estaríamos desarrollando un estándar dentro del protocolo HID que cualquiera que busque información pueda encontrar y utilizar. Pero mientras todos trabajen por separado y creen sus propios comandos, nunca lograremos el verdadero objetivo del protocolo HID braille, que es la funcionalidad plug and play. Si bien puede ser reconocido por el sistema operativo, ser reconocido por el sistema operativo por sí solo no es suficiente para que sea funcional.

Algo muy similar está pasando con los drivers regulares, cada fabricante tiene su propio protocolo y comandos. Entonces, para cada dispositivo, se debe crear un controlador individual. De hecho, ese es el gran problema. Con suerte, habría documentación con un controlador universal que cualquiera podría usar código o un controlador normal con toda la documentación para que cualquiera pudiera replicar el mismo controlador y usarlo. Por ejemplo, un controlador de código abierto con un estándar de reconocimiento llamado "abierto" para Bluetooth y cualquier dispositivo que tenga la palabra "abierto" al comienzo del nombre de Bluetooth se reconocerá como un dispositivo que usa ese controlador. Creo que al implementarlo en un lector de pantalla e implementar la primera pantalla braille que usa ese controlador, abriría la puerta para que el controlador se use en el resto de lectores de pantalla. Entonces,

[Adriani90]

In case of HID, HID input is first processed by the screen reader, therefore the screen reader's job is to parse protocol messages from the braille device and translate it into actual input mechanism the operating system supports.

Unless there could be a Standard Windows driver that translates the commands from the braille display into windows standard Input, without having a screen reader. I was hoping USB-IF would have thought about this before shouding out such a standard promising that HID braille displays would work without a screen reader.

I think still the example that comes nearest to a Braille input / output interpretation is a dualshock game controler as a HID device. Those have input mapings that need to match some how to keyboard and mouse, and they have output mappings such as vibrations to indicate something which has been sent from the HID host to the HID device.
If we can access and understand a complete report descriptor and the comunication routines for a dualshock gamepad I guess it might be something that helps in developing such a driver which avoids the screen reader issues.

But we're talking about an input style that cannot be readily translated into scan codes and virtual key codes (that's how Windows knows about keyboard input)

This could be done via a custom hid miniport driver. Here are all the classes supported so far, but I think Microsoft didn't think of Braille driver API as an own class, so it could be a good start to also talk with them about how to create a custom HID Miniport Driver for Braille Input and Output.
Here is the drivers database where you can already find some KBF examples for HID drivers:
https://github.com/MicrosoftDocs/windows-driver-docs/blob/staging/windows-driver-docs-pr/gettingstarted/do-you-need-to-write-a-driver-.md

[josephsl]

Hi,

A completely different direction, but have you looked at BRLTTY and see how it can serve as a bridge between different braille displays and screen readers?

Yes, different braille displays implement different protocols, and some have standardized around a common set of messages (protocols) i.e. Baum protocol (talking about braille display protocols will require looking at a history of braille display development for the last twenty years or so).

Thanks.

[Adriani90]

BRLTTY causes many conflicts between screen readers, for example when Narator and braille displays are added via BRLTTY, some of the braille displays are totally overtaken by Narator ynd you cannot find them in NVDA's menu anymore. I had this problem with an user here in Germany and the solution was to remove BRLTTY and all the Braille drivers from Narator. After that the Braille displays appeared again in NVDA's menu and they have been recognized automatically again.

[kumaaarrr]

Hello, I've also been wanting to do something similar. I have used RP2040 and programmed it using Micropython. Even I have been facing issue in implementing it as Braille HID device. @discapacidad5 It would be great to know if you found out a way on how it's done. I'm very new to this HID thing and it's quiet honestly getting very complicated for me to understand.

[ghost]

Hello, I am also trying to make the braille display but I cannot establish communication.
Could you share your code?

[discapacidad5]

Hi everyone, after a year of working on this code with the help of AI, I believe I've got a good grasp on how it works. However, it's not functioning as expected. I'm hoping someone can help me identify the issue.

#include <Arduino.h>

#include "BLEDevice.h"

#include "BLEHIDDevice.h"

#include "HIDTypes.h"

#include "HIDBrailleDisplay.h"



#define DEVICE_NAME "Standard HID Braille Display"



bool isBleConnected = false;

bool hasSentIdentification = false;



BLEHIDDevice* hid;

BLECharacteristic* input;

BLECharacteristic* output;



uint8_t REPORT_MAP[] = {

    0x05, 0x41,        // Usage Page (Braille)

    0x09, 0x01,        // Usage (Braille Display)

    0xA1, 0x01,        // Collection (Application)

    0x1A, 0x01, 0x02,  // Usage Minimum (Braille Keyboard Dot 1)

    0x2A, 0x08, 0x02,  // Usage Maximum (Braille Keyboard Dot 8)

    0x75, 0x01,        // Report Size (1 bit)

    0x95, 0x08,        // Report Count (8 bits for 8 dots)

    0x15, 0x00,        // Logical Minimum (0)

    0x25, 0x01,        // Logical Maximum (1)

    0x81, 0x02,        // Input (Data, Var, Abs)

    0x09, 0x02,        // Usage (Braille Row)

    0xA1, 0x02,        // Collection (Logical)

    0x09, 0x03,        // Usage (8 Dot Braille Cell)

    0x15, 0x00,        // Logical Minimum (0)

    0x26, 0xFF, 0x00,  // Logical Maximum (255)

    0x75, 0x08,        // Report Size (8 bits)

    0x95, 0x1E,        // Report Count (30 cells)

    0x91, 0x02,        // Output (Data, Var, Abs)

    0xC0,              // End Collection (Logical)

    0xC0,              // End Collection

    0xC0               // End Collection (Application)

};



struct BrailleIdentificationReport {

    uint8_t reportID = 1;

    uint8_t page = 0x41;  // Usage Page (Braille)

    uint8_t usage = 0x01; // Braille Display

    uint8_t numCells = 30; // Número de celdas Braille

    uint8_t cellType = 0x03; // 8 Dot Braille Cell

};



class OutputCallbacks : public BLECharacteristicCallbacks {

    void onWrite(BLECharacteristic* characteristic) {

        uint8_t* data = characteristic->getData();

        int length = characteristic->getLength();



        if (length > 0) {

            Serial.print("Received Braille Cells Report: ");

            for (int i = 0; i < length; i++) {

                Serial.print(data[i], HEX);

                Serial.print(" ");

            }

            Serial.println();

        }

    }

};



class BleKeyboardCallbacks : public BLEServerCallbacks {

    void onConnect(BLEServer* server) {

        isBleConnected = true;

        BLE2902* cccDesc = (BLE2902*)input->getDescriptorByUUID(BLEUUID((uint16_t)0x2902));

        cccDesc->setNotifications(true);

        Serial.println("Client has connected");

    }



    void onDisconnect(BLEServer* server) {

        isBleConnected = false;

        hasSentIdentification = false; 

        BLE2902* cccDesc = (BLE2902*)input->getDescriptorByUUID(BLEUUID((uint16_t)0x2902));

        cccDesc->setNotifications(false);

        Serial.println("Client has disconnected");

    }

};



void sendBrailleIdentification() {

    if (!hasSentIdentification) {

        BrailleIdentificationReport idReport;

        input->setValue((uint8_t*)&idReport, sizeof(idReport));

        input->notify();

        hasSentIdentification = true;

        Serial.println("Braille Display identification sent.");

    }

}



void bluetoothTask(void*) {

    BLEDevice::init(DEVICE_NAME);

    BLEServer* server = BLEDevice::createServer();

    server->setCallbacks(new BleKeyboardCallbacks());



    hid = new BLEHIDDevice(server);

    input = hid->inputReport(1);

    output = hid->outputReport(1);

    output->setCallbacks(new OutputCallbacks());



    hid->manufacturer()->setValue("hidBrailleStandard");

    hid->pnp(0x02, 0xe502, 0xa111, 0x0210);

    hid->hidInfo(0x00, 0x2);



    BLESecurity* security = new BLESecurity();

    security->setAuthenticationMode(ESP_LE_AUTH_BOND);



    hid->reportMap(REPORT_MAP, sizeof(REPORT_MAP));

    hid->startServices();

    hid->setBatteryLevel(100);



    BLEAdvertising* advertising = server->getAdvertising();

    advertising->setAppearance(0x03C9);

    advertising->addServiceUUID(hid->hidService()->getUUID());

    advertising->addServiceUUID(hid->deviceInfo()->getUUID());

    advertising->addServiceUUID(hid->batteryService()->getUUID());

    advertising->start();



    Serial.println("BLE ready");



    while (true) {

        if (isBleConnected) {

            sendBrailleIdentification();

        }

        delay(1000); 

    }

}



void setup() {

    Serial.begin(19200);

    xTaskCreate(bluetoothTask, "bluetooth", 20000, NULL, 5, NULL);

}



void loop() {

    // Nada en el loop

}

[discapacidad5]

HIDBrailleDisplay.h

#ifndef HID_BRAILLE_DISPLAY_H
#define HID_BRAILLE_DISPLAY_H

#define REPORT_ID_BRAILLE_DISPLAY 1

// Define la estructura BRAILLE_KEYMAP
typedef struct {
    uint16_t usage;
    uint8_t modifier;
} BRAILLE_KEYMAP;

enum BRAILLE_DISPLAY_USAGE {
    BRAILLE_DISPLAY = 0x01,         // Braille Display
    BRAILLE_ROW = 0x02,             // Braille Row
    BRAILLE_CELL_8_DOT = 0x03,      // 8 Dot Braille Cell
    BRAILLE_CELL_6_DOT = 0x04,      // 6 Dot Braille Cell
    NUMBER_OF_BRAILLE_CELLS = 0x05, // Number of Braille Cells
    SCREEN_READER_CONTROL = 0x06,   // Screen Reader Control
    SCREEN_READER_IDENTIFIER = 0x07,// Screen Reader Identifier

    // Router Usages
    ROUTER_SET_1 = 0xFA,
    ROUTER_SET_2 = 0xFB,
    ROUTER_SET_3 = 0xFC,
    ROUTER_KEY = 0x100,

    // Braille Keyboard Dots
    BRAILLE_KEYBOARD_DOT_1 = 0x201,
    BRAILLE_KEYBOARD_DOT_2 = 0x202,
    BRAILLE_KEYBOARD_DOT_3 = 0x203,
    BRAILLE_KEYBOARD_DOT_4 = 0x204,
    BRAILLE_KEYBOARD_DOT_5 = 0x205,
    BRAILLE_KEYBOARD_DOT_6 = 0x206,
    BRAILLE_KEYBOARD_DOT_7 = 0x207,
    BRAILLE_KEYBOARD_DOT_8 = 0x208,
    BRAILLE_KEYBOARD_SPACE = 0x209,
    BRAILLE_KEYBOARD_LEFT_SPACE = 0x20A,
    BRAILLE_KEYBOARD_RIGHT_SPACE = 0x20B,

    // Joystick Controls
    BRAILLE_JOYSTICK_CENTER = 0x210,
    BRAILLE_JOYSTICK_UP = 0x211,
    BRAILLE_JOYSTICK_DOWN = 0x212,
    BRAILLE_JOYSTICK_LEFT = 0x213,
    BRAILLE_JOYSTICK_RIGHT = 0x214,

    // D-Pad Controls
    BRAILLE_D_PAD_CENTER = 0x215,
    BRAILLE_D_PAD_UP = 0x216,
    BRAILLE_D_PAD_DOWN = 0x217,
    BRAILLE_D_PAD_LEFT = 0x218,
    BRAILLE_D_PAD_RIGHT = 0x219,

    // Pan Controls
    BRAILLE_PAN_LEFT = 0x21A,
    BRAILLE_PAN_RIGHT = 0x21B,

    // Rocker Controls
    BRAILLE_ROCKER_UP = 0x21C,
    BRAILLE_ROCKER_DOWN = 0x21D,
    BRAILLE_ROCKER_PRESS = 0x21E,
};

#define BRAILLE_KEYMAP_SIZE (28)

const BRAILLE_KEYMAP braille_keymap[BRAILLE_KEYMAP_SIZE] = {
    {BRAILLE_KEYBOARD_DOT_1, 0},
    {BRAILLE_KEYBOARD_DOT_2, 0},
    {BRAILLE_KEYBOARD_DOT_3, 0},
    {BRAILLE_KEYBOARD_DOT_4, 0},
    {BRAILLE_KEYBOARD_DOT_5, 0},
    {BRAILLE_KEYBOARD_DOT_6, 0},
    {BRAILLE_KEYBOARD_DOT_7, 0},
    {BRAILLE_KEYBOARD_DOT_8, 0},
    {BRAILLE_KEYBOARD_SPACE, 0},
    {BRAILLE_KEYBOARD_LEFT_SPACE, 0},
    {BRAILLE_KEYBOARD_RIGHT_SPACE, 0},
    {BRAILLE_JOYSTICK_CENTER, 0},
    {BRAILLE_JOYSTICK_UP, 0},
    {BRAILLE_JOYSTICK_DOWN, 0},
    {BRAILLE_JOYSTICK_LEFT, 0},
    {BRAILLE_JOYSTICK_RIGHT, 0},
    {BRAILLE_D_PAD_CENTER, 0},
    {BRAILLE_D_PAD_UP, 0},
    {BRAILLE_D_PAD_DOWN, 0},
    {BRAILLE_D_PAD_LEFT, 0},
    {BRAILLE_D_PAD_RIGHT, 0},
    {BRAILLE_PAN_LEFT, 0},
    {BRAILLE_PAN_RIGHT, 0},
    {BRAILLE_ROCKER_UP, 0},
    {BRAILLE_ROCKER_DOWN, 0},
    {BRAILLE_ROCKER_PRESS, 0},
};

#endif // HID_BRAILLE_DISPLAY_H

[Scherady]

Thank you @discapacidad5 for sharing your code.

I can run it and get it recognised by VoiceOver but nothing else.

Are you able to receive Braille Cells Reports?

[FalkoBabbage]

I have looked at implementing a hid braille display with a RP2040 as well last week since I was tagged in this discussion and I am relatively interested in HID devices in general. I'll see if I can make some time this week to see if I can make a minimal example and put it online.

[ghost]

Hello, I am also trying to make the braille display but I cannot establish communication.
Could you share your code?

[blasthenwaytech]

Hello guys, I'm working on a new multi-line Braille display project that intends to make use of the "Standard HID Braille Display" option for NVDA. So, the idea is to implement the HID Braille Display usage page so the NVDA software can automatically detect my display and work right away.

I'm using an ATSAMD51J20A microcontroller from Microchip and I'm developing the firmware using the Harmony 3 framework for MPLABX IDE.

I've created a very simple demo that configures my USB device as a Braille Display with only 1 row of 8 cells, no other feature. However, when I open the NVDA settings, I can't get NVDA to detect my USB Braille Display.

By opening the Windows Device Manager tool I can see that my USB device is correctly enumerated.

---

Below is the value of each of the descriptors sent to the USB host.

Device Descriptor

    0x12,                     // bLength: Size of this descriptor in bytes (18 bytes)
    (uint8_t)USB_DESCRIPTOR_DEVICE,  // bDescriptorType: DEVICE descriptor type (0x01)
    0x0200,                   // bcdUSB: USB version (USB 2.0)
    0x00,                     // bDeviceClass: Class is defined at the interface level
    0x00,                     // bDeviceSubClass: Subclass defined in interface descriptor
    0x00,                     // bDeviceProtocol: Protocol defined in interface descriptor
    USB_DEVICE_EP0_BUFFER_SIZE, // bMaxPacketSize0: Max packet size for endpoint 0 (control) is defined in configuration.h
    0x04D8,                   // idVendor: Microchip's Vendor ID (0x04D8)
    0x003F,                   // idProduct: Product ID for this device (0x003F)
    0x0100,                   // bcdDevice: Device release number (1.00)
    0x01,                     // iManufacturer: Index of string descriptor for manufacturer (0x01)
    0x02,                     // iProduct: Index of string descriptor for product (0x02)
    0x00,                     // iSerialNumber: No serial number
    0x01                      // bNumConfigurations: Number of possible configurations (1)

Configuration Descriptor

    0x09,                                                   // Size of this descriptor in bytes
    (uint8_t)USB_DESCRIPTOR_CONFIGURATION,                  // Descriptor Type
    USB_DEVICE_16bitTo8bitArrange(41),                      // Total size of the configuration descriptor and its associated descriptors (41 bytes)
    1,                                                      // Number of interfaces in this configuration
    0x01,                                                   // Index of this configuration
    0x00,                                                   // Configuration string index
    USB_ATTRIBUTE_DEFAULT | USB_ATTRIBUTE_SELF_POWERED,      // Attributes: Self-powered
    50,                                                     // Maximum Power: 100mA

Interface Descriptor

    0x09,                           // Size of this descriptor in bytes
    (uint8_t)USB_HID_DESCRIPTOR_TYPES_HID,   // HID descriptor type
    0x11, 0x01,                      // HID Spec Release Number in BCD format (1.11)
    0x00,                           // Country Code (0x00 for Not supported)
    1,                              // Number of class descriptors
    (uint8_t)USB_HID_DESCRIPTOR_TYPES_REPORT,// Report descriptor type
    USB_DEVICE_16bitTo8bitArrange(sizeof(hid_rpt0)),   // Size of the report descriptor

Endpoint Descritpors

    0x07,                           // Size of this descriptor in bytes
    USB_DESCRIPTOR_ENDPOINT,        // Endpoint Descriptor
    1 | USB_EP_DIRECTION_IN,    // EndpointAddress ( EP1 IN )
    (uint8_t)USB_TRANSFER_TYPE_INTERRUPT,    // Attributes (Interrupt transfer)
    0x40,0x00,                      // Maximum packet size (64 bytes)
    0x01                            // Polling interval (1 ms)

    0x07,                           // Size of this descriptor in bytes
    USB_DESCRIPTOR_ENDPOINT,        // Endpoint Descriptor
    1 | USB_EP_DIRECTION_OUT,   // EndpointAddress ( EP1 OUT )
    (uint8_t)USB_TRANSFER_TYPE_INTERRUPT,    // Attributes (Interrupt transfer)
    0x40,0x00,                      // Maximum packet size (64 bytes)
    0x01                            // Polling interval (1 ms)

Report Descriptor (Here's were the Braille Display usage is defined)

    0x05, 0x41,        // Usage Page (Braille Display)
    0x09, 0x01,        // Usage (Braille Display)
    0xA1, 0x01,        // Collection (Application)

    0x09, 0x02,        // Usage (Braille Row)
    0xA1, 0x02,        // Collection (Logical)

        0x09, 0x03,        // Usage (8 Dot Braille Cell)
        0x15, 0x00,        // Logical Minimum (0)
        0x26, 0xFF, 0x00,  // Logical Maximum (255)
        0x75, 0x08,        // Report Size (8 bits per cell)
        0x95, 0x28,        // Report Count (40 cells)
        0x91, 0x02,        // Output (Data, Var, Abs)

    0xC0,              // End Collection (Logical)

0xC0               // End Collection (Application)

---

Am I configuring something wrong? What could be happening?