Feature suggestions/development/maintenance: bringing ideas/features/code from add-ons to NVDA Core

[josephsl]

Hi,

Stemming from #16056, this discussion is intended to serve as a venue for NVDA community and development stakeholders (NV Access, broader NVDA community, observers, and others) to offer thoughts on the overall direction of NVDA development/maintenance and its connection with add-ons and bringing ideas/features from them to the screen reader core:

Context

One of the strengths of NVDA is add-ons, a small, self-contained module that adds features and workarounds at the global and app levels. Over the years, NVDA community members (including NV Access) has developed several add-ons that has enriched the lives of NVDA users with features such as online-based image recognition and generative AI powered descriptions, computer resource monitoring, numerous speech synthesizers and braille displays support, support for apps such as audio editors, and other enhancements and workarounds. Recently, several features from these add-ons have made their way to NVDA Core. These include audio output splitting, visual highlight and screen curtain, and checking for and applying add-on updates.

With requests to bring add-on features/ideas/code to NVDA Core, several community members have commented on the scope of a screen reader in recent issues. In #16056, for example, while discussing a request to add a command to mute microphones, a discussion arose around the need to bring add-on features/ideas/code to the core screen reader. Some community members believed that it is okay to bring add-on features to solve real-life issues, while some felt that screen readers should focus on the core task of screen reading. While some participants insisted on keeping screen readers to their minimum and let add-ons take care of various feature requests, some felt that it is time to reassess use cases and think about development and maintenance strategy.

Stemming from the above discussion, another conversation arose around who is responsible for maintaining NVDA project. Some argued that NV Access is in charge because pull requests are accepted and merged by NV Access staff even though pull requests may come from the broader community. Some said the broader NVDA Community because NVDA itself is a screen reader with community sentiments, needs, and wishes embedded in its code and ideas, with add-ons representing how community members use NVDA across contexts.

Key questions and discussion points

Definitions of screen readers and screen reading

• What is the definition and core nature of a screen reader in 2024?
• How did the definition and core nature of screen readers change over time?
• If the nature of screen reading has changed, what caused these changes recently?

Features/ideas/code development/maintenance between screen readers and add-ons

• Can screen reader code and developers set boundaries between core functionality and add-ons, or is setting the boundary even desired in the long-term?
• In addition to the features listed above, what are features/idea/code from other add-ons completely or partially included in NVDA?
• What are features of add-ons (registered or not on the NV Access add-on store) that are essential for screen reading in 2024?
• If add-on features are included in NVDA Core, who should be responsible for its maintenance and upkeep?

Thanks.

[seanbudd]

These are some great questions Joseph, I look forward to the discussion here.

Just to add to some of the simpler ones first:

In addition to the features listed above, what are features/idea/code from other add-ons completely or partially included in NVDA?

Off the top of my head:

• OCR
• no beeps speech mode
• many items from Windows app essentials
• many key commands for navigation, either with apps or for general NVDA actions
• several app modules

What are features of add-ons (registered or not on the NV Access add-on store) that are essential for screen reading in 2024?

Here are a couple I think we see in the scope of core, and are valuable to have:

• NVDA remote (this is harder as we would likely have to host our own infrastructure)
• AI Image description tools
• mathCAT (already planned for inclusion)

If add-on features are included in NVDA Core, who should be responsible

This is going to always fall back on NV Access as the maintainers of the project. However, in practice of course there would be collective efforts to fix or back out code. Generally with PRs we will revert it if there is no capacity to fix blocking issues, and so it becomes on the original author or the community to decide if they want to take responsibility.

[lukaszgo1]

What are features of add-ons (registered or not on the NV Access add-on store) that are essential for screen reading in 2024?

Here are a couple I think we see in the scope of core, and are valuable to have:

• NVDA remote (this is harder as we would likely have to host our own infrastructure)

• AI Image description tools

• mathCAT (already planned for inclusion)

I'm surprised by what I saw, or rather what I haven't seen here. I'd say there are several add-ons, or in some cases categories of add-ons missing which are much less controversial to be included:

• Stuff which makes inefficient parts of core (configuring synth settings ring, which is unusable for all but the simplest of cases), selecting with the review cursor (again the core experience is far from delightful)
• Add-ons which contain app modules for most apps should be at least considered for inclusion as long as applications they support are not moving targets. I'd say improving accessibility of all but the most niche programs is in scope of NVDA.
• Auto updates for add-ons
• Support for Remote Desktop / Citrix, where sighted people can just work with these with no additional effort.

[seanbudd]

Stuff which makes inefficient parts of core (configuring synth settings ring, which is unusable for all but the simplest of cases), selecting with the review cursor (again the core experience is far from delightful)
Add-ons which contain app modules for most apps should be at least considered for inclusion as long as applications they support are not moving targets. I'd say improving accessibility of all but the most niche programs is in scope of NVDA.

Important fixes for the UX and accessibility of applications really should have been proposed via issues first and fixed via PR, than ever becoming add-ons in the first place.
It would be great to get a list of these add-ons in this discussion, or even better, issues filed.

Auto updates for add-ons

This is already triaged: #3208

Support for Remote Desktop / Citrix, where sighted people can just work with these with no additional effort.

I was mistaken with what our preference is, but yes the features of rdAccess (RDP/Citrix) are more likely the solution we will go with rather than NVDA Remote (with relay server infrastructure)

[lukaszgo1]

Important fixes for the UX and accessibility of applications really should have been proposed via issues first and fixed via PR, than ever becoming add-ons in the first place. It would be great to get a list of these add-ons in this discussion, or even better, issues filed.

I'll try to create issues for stuff I consider important in the coming days - we will see how subjective that list will become.

Support for Remote Desktop / Citrix, where sighted people can just work with these with no additional effort.

I was mistaken with what our preference is, but yes the features of rdAccess (RDP/Citrix) are more likely the solution we will go with rather than NVDA Remote (with relay server infrastructure)

Note that these two solutions serve different, but important purposes:

• RDP / Citrix - used mostly in corporate environments where employees need to work on a single server
• NVDA remote - necessary for remote training, providing help support to users (where
  RDP / Citrix will not work due to costs, but also since when machine is accessed via RDP its operator cannot see what the helper is doing). There is also a matter of smaller enterprises who prefer to use Team Viewer or some VNC based solutions for remote access, where NVDA Remote is indispensable. All other screen readers have some form of a remote access not tied to RDP / Citrix / whatever, and the fact that we have RIM which is a commercial product shows that there is a real need for such solutions still.

[Qchristensen]

A couple of questions I think worth asking:

Are there barriers for people using add-ons? And is this a key reason there is sometimes a push for functionality already available in an add-on to be included in core?

Where something is developed as an add-on first for whatever reason, and is found to be really beneficial for the majority of users, there are multiple examples where it then has moved into core - screen curtain for instance.

In some cases, I think even if something may be widely regarded as beneficial, there can be arguments for keeping it as an add-on. Two mentioned already could be:

NVDA Remote
Online tools such as AI Image recognition.

Given the online nature of features such of those and the potential for user data to be sent over the internet - these raise security and privacy questions for some users and companies, and I believe "This functionality is available to be downloaded separately" is a better approach than "This functionality is built-in, and you can either disable it or not use it if you prefer".

The question then becomes "as a user, if I want those features, is it hard for me to get them?" Either in terms of discoverability (finding out that such a feature even exists as an add-on) or setup (can I easily access the add-on and install it?)

I think having the add-on store built-in to NVDA now should help to a lesser or greater extent with both of those issues, but is there more that could / should be done to make it easier?

[gerald-hartig]

Thanks for reaching out and providing the context for this discussion. I appreciate the opportunity to address these important questions regarding the direction of NVDA development and the role of add-ons.

First and foremost, the core purpose of NV Access is to support the health and longevity of NVDA, with the ultimate goal of maximising the benefit that NVDA brings to the visually impaired community. The best way to accomplish this is by being deeply attuned to the community's needs, working in a collaborative manner with them, and being guided and bound by our product vision and guiding principles (which can be found at https://github.com/nvaccess/nvda/blob/master/projectDocs/product_vision.md). These principles emphasise accessibility, user empowerment and flexibility to meet individual needs.

It's important to recognise that user needs and the technological environment are constantly evolving, and NVDA is no exception. As such, there is no one-size-fits-all equation for balancing the various needs of different user groups, or even for determining the appropriate balance between infrastructure improvements and new features. This is part of the complex decision-making process that the NV Access staff are tasked with.

Regarding the definition and core nature of a screen reader, it is true that this has evolved over time. We recognise that the lines of "core" screen reader functionality have never been static throughout NVDA's history. Screen readers aren't just about reading text. They're about empowering visually impaired individuals to interact with and control a digital world. As that world expands, so must NVDA's capabilities.

With respect to the development and maintenance of features and code between the core NVDA application and add-ons, there is no strict boundary or hard line. While add-ons provide fantastic experimentation and niche solutions and have played a crucial role in extending NVDA's capabilities and addressing specific use cases, there are instances where it may be appropriate to incorporate certain add-on features or concepts into the core NVDA codebase. This decision is made on a case-by-case basis, taking into account factors such as the potential benefit to a broader user base, the maturity and stability of the feature, and the long-term maintenance implications.

Ultimately, the responsibility for maintaining and supporting features that are incorporated into NVDA core rests with NV Access. However, we highly value and encourage community contributions, whether in the form of bug reports, feature requests or code contributions. This collaborative approach has been a key strength of the NVDA project, and we are committed to continuing to foster an open and inclusive development process.

In conclusion, while there may be differing perspectives on the appropriate scope and direction of NVDA development, our commitment remains unwavering: to provide a robust and accessible screen reader that empowers users and meets their evolving needs. NV Access will continue to engage with the community, leverage their insights and expertise, and make thoughtful decisions that align with our product vision and guiding principles.

[lukaszgo1]

Are there barriers for people using add-ons? And is this a key reason there is sometimes a push for functionality already available in an add-on to be included in core?

Yes, there are several:

• In a corporate environments you may not be allowed to install add-ons, or if you are this is probably because your IT admin does not know about them, rather than permits you to install them
• Add-ons are broken yearly so if the given functionality is important to me I'm very likely not to update NVDA, so that my productivity is not negatively impacted. Obviously many add-ons are updated sooner or later, but there always many which are not
• This is similar to the above and affects mostly power users, but if the basic stuff I need to use gets broken every year I'm very unlikely to use Alpha versions of NVDA at least during the .1 releases, which in turn means these version get much less testing than they could. Note that I'm guilty of this myself - I've been using Alpha versions of NVDA as my daily driver since 2016 or so, but stopped after 2019.3 got released exactly due to add-ons stopping working way too often.

The question then becomes "as a user, if I want those features, is it hard for me to get them?" Either in terms of discoverability (finding out that such a feature even exists as an add-on) or setup (can I easily access the add-on and install it?)

I'd say add-on store should really have an ability to display the documentation for a given add-on - this is covered in #13454

[seanbudd]

In a corporate environments you may not be allowed to install add-ons, or if you are this is probably because your IT admin does not know about them, rather than permits you to install them

Isn't it the responsibility of the workplace to ensure their employees have access to the tools they need to do their job?
My feeling is that there will always be certain cases where something would never be welcome in core, but necessary for employment, e.g. a niche dev tools add-on.
If an employer is refusing to let you install such tools, I think that is a case of discrimination.
I understand that the add-on infrastructure is a little riskier which causes hesitation unless you trust the add-on publisher, but over time we plan to create a much secure and trustworthy environment for add-ons.
For other items, yes this is a motivating factor for inclusion in core, i.e. support for RDP/Citrix is pretty essential in corporate environments and well within scope.

Add-ons are broken yearly so if the given functionality is important to me I'm very likely not to update NVDA, so that my productivity is not negatively impacted. Obviously many add-ons are updated sooner or later, but there always many which are not

Does the ability to override incompatible add-ons help reduce this friction point much? We've certainly seen a large reduction in complaints and frustration since that feature was introduced.

[lukaszgo1]

Isn't it the responsibility of the workplace to ensure their employees have access to the tools they need to do their job? My feeling is that there will always be certain cases where something would never be welcome in core, but necessary for employment, e.g. a niche dev tools add-on.

I'm not sure I follow this point. On one hand you said that accessibility improvements for applications are welcomed in core, but here you're saying they're not. While I understand stuff like this will need to be decided on a case by case basis to some extend, knowing what qualifies as too niche for inclusion in general terms may be helpful.

If an employer is refusing to let you install such tools, I think that is a case of discrimination.

In some cases it may be, but even if it is then what? You're very likely looking at this from the US (or perhaps Australian, I'm not sure what anti discrimination laws are there) perspective, but most places, including where I live, in cases of employer discrimination I have to either keep working somehow, or try looking for another job, which is pretty difficult. There is also a case of employer disagreeing to allow installation of NVDA add-on, but permitting, or even encouraging, to use different screen reader, where the same feature exists natively. To summarize this point: my opinion is that as long as a given feature is supported by default by at least one screen reader on a main OS, we should seriously consider including it into the core.

I understand that the add-on infrastructure is a little riskier which causes hesitation unless you trust the add-on publisher, but over time we plan to create a much secure and trustworthy environment for add-ons.

I'm looking forward into developments in this area.

Add-ons are broken yearly so if the given functionality is important to me I'm very likely not to update NVDA, so that my productivity is not negatively impacted. Obviously many add-ons are updated sooner or later, but there always many which are not

Does the ability to override incompatible add-ons help reduce this friction point much? We've certainly seen a large reduction in complaints and frustration since that feature was introduced.

I'm not in a position to judge this generally. My, very subjective, opinion is that allowing to override compatibility simply shows the weakness of the add-ons system. On one hand it is pointed out that developers should not lie in the manifest about compatibility info, but since this is too annoying for users we give them ability to override this, rather than try to avoid yearly breakages. I'm not saying this will be easy, but working on the root cause is always better.

[seanbudd]

I'm not sure I follow this point. On one hand you said that accessibility improvements for applications are welcomed in core, but here you're saying they're not. While I understand stuff like this will need to be decided on a case by case basis to some extend, knowing what qualifies as too niche for inclusion in general terms may be helpful.

I think there is a difference between an application being accessible (the purpose of a screen reader) and additional/advanced functionality to improve usability of certain tasks, that only apply to a subset of users.

In some cases it may be, but even if it is then what? You're very likely looking at this from the US (or perhaps Australian, I'm not sure what anti discrimination laws are there) perspective, but most places, including where I live, in cases of employer discrimination I have to either keep working somehow, or try looking for another job, which is pretty difficult.

The point here is that employers could discriminate by preventing you from using a screen reader, or other tools for accessibility. It is really the fault and responsibility of the organisation. we can't do anything if they block people from using NVDA and similarly we can't do anything if they prevent employees from using other essential tools such as add-ons.

[lukaszgo1]

One thing which both @LeonarddeR and I mentioned briefly in #16056 which has not really been discussed here, is the impact of yearly compatibility breaks on add-ons. I'd say that if we could limit them we would get much less requests about including stuff into the core, just because users would be much less annoyed. Is this a direction which has at least been considered by NV Access?

[seanbudd]

We are looking towards building a stable, restricted add-on API over time.
This will likely start with a stable API for synth drivers and braille display drivers.
Reducing breakages isn't really feasible with the current infrastructure, otherwise we would just be endlessly blocked on core development.
We are hoping that the ability to override incompatibility holds people over in the mean time, as this lets users use add-ons where the API hasn't actually broken the add-on.

[Adriani90]

@seanbudd wrote:

Isn't it the responsibility of the workplace to ensure their employees have access to the tools they need to do their job?

In general yes, but regarding extensions system admins are very strict. It is not only NVDA that is impacted by this.

If an employer is refusing to let you install such tools, I think that is a case of discrimination.

In some cases that might indeed be the case, but I wouldn't generalize it otherwise it becomes a really hard statement. There are also valid reasons why an employer would not allow add-ons to be installed, and I think there is too less knowledge about accessibility in IT departments so they will rather make a requirement for all instead of looking for add-ons case by case. I think though there are companies who are more open minded and hear the arguments of a person who needs the add-on for their job. But there are also such add-ons like "speak passwords" which I guess no system admin would allow them to be installed.

I understand that the add-on infrastructure is a little riskier which causes hesitation unless you trust the add-on publisher, but over time we plan to create a much secure and trustworthy environment for add-ons.

That's a really good goal, though I think add-ons will always remain a thorn in the side of system admins and IT security departments. Not only in the case of NVDA. So there will always arise cases where an add-on's functionality will be proposed for core inclusion. And there will be add-ons which are so impactful, that a single add-on author with a small community behind him or her will never be able to use its full potential. So having certain features in NVDA that can impact positively the whole community will always be a wish case by case.

Does the ability to override incompatible add-ons help reduce this friction point much? We've certainly seen a large reduction in complaints and frustration since that feature was introduced.

It certainly improved the situation for some users, but certainly not for the majority of people in the community:

• Showing incompatible add-ons is disabled by default in the updatable add-ons tab, so their updates are not visible
• Updates are issued in a random period of time, you never have a certainty whether you should update NVDA or not especially when there is an API breaking change

@gerald-hartig wrote:

First and foremost, the core purpose of NV Access is to support the health and longevity of NVDA, with the ultimate goal of maximising the benefit that NVDA brings to the visually impaired community. The best way to accomplish this is by being deeply attuned to the community's needs, working in a collaborative manner with them, and being guided and bound by our product vision and guiding principles

That's a very strong and valid statement in my view. This makes it already very obvious that discussions with the community and understanding the use cases and the potential of a feature will help in deciding whether it is elligible to be part of the core or not.

There are also add-ons which only solve bugs in NVDA. Such add-ons are for sure easier to integrate into the core and will generate less discussions than an add-on which brings a totally new feature.

[Adriani90]

My conclusion after more than 10 years of power using NVDA and building the german and romanian NVDA community is that now we have more than 250 add-ons out there, some people have installed more than 50 add-ons and they will never be able to reach out to each of the add-ons authors to request a new functionality or to request solving a bug. Merging some of the add-ons which are very important for the community will become more and more important, because it opens up the potential that the whole community can easily contribute to these features within one single repository.

The challenge will be to discuss the UX design and the documentation of each proposal in such a way that NVDA still remains an easy to use screen reader and does not overwelm people.
But in my view this community is already managing such discussions in a very impressive way and I am really glad to see more and more people contributing to such discussions with great arguments.
Users become more educated the more they are empowered to do things. 10 years ago almost no one I know used object navigation. Now almost everyone knows what it is and how it works.

[josephsl]

Hi,

While I can't comment on all discussion posts so far, to offer context around Windows App Essentials:

I created Windows App Essentials add-on in 2015 to respond to Windows as a Service (WaaS) where things can change without notice. This was evident in Windows Insider Program (WIP) where new builds were released monthly or so. The build release frequency has increased significantly in recent years (close to a weekly release). I still participate in Windows Insider Program and have made changes to the add-on in response to Windows feature changes (I am planning to submit several pull requests once Windows 11 24H2/2024 Update/Server 2025 final build is revealed to Insiders).

From its first (stable) release in December 2015, I knew that parts of Windows App Essentials add-on must be made available in NVDA Core. Sooner or later NVDA users upgrading to Windows 10 and later will meet changes proposed and implemented by Microsoft with community feedback, and NVDA users should be able to access and use built-in Windows features (including included apps) without help from add-ons. More importantly, Windows Insiders will be testing builds where features can come and go without notice, including bug fixes dealing with accessibility and usability. While it was a minor factor, since 2018 Narrator has shown remarkable progress, and I could not ignore it - I consider Narrator to be the gold standard on Windows 10 and later as far as feature support is concerned.

The above three things - continuous development and delivery for Insiders, feature availability and accessibility on stable Windows 10/11 releases, and Narrator development - are what keeps Windows App Essentials and my contribution to NVDA Core going (of course I consider this an opportunity to further develop research topics in graduate school). Therefore, I realized early on that parts of this add-on must be made available to NVDA users. As Sean noted, many features from this add-on have made their way to NVDA Core, including:

• Looping selectors: in early Windows 10 releases, a looping selector was used in various apps, and NVDA did not handle them well.
• App module name handling: because some apps have a period (.) in the middle of their executable names, I patched app module handler to let NVDA load additional app modules.
• App enhancements, most notably Calculator.
• Play sounds when suggestions appear: this was originally a part of the add-on, and I remember a lively discussion about which sound to use when this became part of NVDA.
• Emoji panel support: I remember spending many nights trying to let NVDA navigate emoji panel in 2017, and the rest is history.
• Clipboard history: took me several tries in summer 2018, and the feature is now part of NVDA.
• Virtual desktops reporting: I and Jamie Teh have worked on it - I did the initial work and Jamie provided refinements.
• Accessible drag and drop: took a while, but now part of NVDA.
• Mouse and touch navigation in some Windows 11 UI elements: caused by XAML Islands not recognized as UIA elements. I think some cases were handled (particularly File Explorer) but expect a pull request from me as Windows 11 2024 Update has introduced a redesigned Action Center with mouse and touch navigation problem for NVDA users.

All of these features have one thing in common: built into Windows at one time or another. For me, an important criteria for bringing add-ons into NVDA Core is support for features and apps built into Windows. We have various add-ons supporting various apps and scenarios, and many do not come with the operating system by default; for these, I think they should remain add-ons; this is more so for apps that change their interfaces and experiences quite frequently. We also have add-ons adding enhancements and offering fixes at the global level, and these are sites of tensions and discussions. For these, I think we need to think about where these features are encountered and needed, with ones built into Windows having priority along with improving usage of various apps (say, supporting a common set of controls found across apps, both built into Windows or obtained elsewhere). In some ways, folks could say that I'm basing my thoughts on Windows App Essentials because the add-on is an example of my thinking in this regard (and is structured and coded to follow these principles); this is why I keep an eye on Windows changes (through Insider Preview builds) and features I helped write in NVDA Core to see if things must be refined from time to time.

Postscript: there is one thing I will NOT transfer from Windows App Essentials to NVDA Core: Windows 10 Settings app and live region change event handling in Windows Update interface. This feature will be removed from Windows App Essentials either in late 2025 (when Windows 10 support ends) or the add-on is declared end of life, whichever happens earlier (I did say that add-on development is tied to Windows Insider Program, but since 2023 I began the slow process of retiring the add-on; I don't know when the add-on will be declared EOL, and that's the day I plan to step aside from most NVDA activities).

Thanks.