Windows: Systray: The current process has used all of its system allowance of handles for Window Manager objects

[d1ss0nanz]

Checklist

• I have searched the issue tracker for open issues that relate to the same problem, before opening a new one.
• This issue only relates to a single bug. I will open new issues for any other problems.

Describe the bug

Systray Menu becomes unresponsive. The following line is logged:
systray error: unable to addOrUpdateMenuItem: The current process has used all of its system allowance of handles for Window Manager objects.

How to reproduce

1. Build the attached program
2. ran the resulting exe file
3. wait a few minutes
4. the systray menu becomes unresponsive
5. UserCacheDir/test.log shows the message systray error: unable to addOrUpdateMenuItem: The current process has used all of its system allowance of handles for Window Manager objects.

Screenshots

No response

Example code

import (
	"log"
	"os"
	"strconv"
	"time"

	"fyne.io/fyne/v2"
	"fyne.io/fyne/v2/app"
	"fyne.io/fyne/v2/driver/desktop"
)

func main() {
	StartLogging("test.log")
	log.Println("starting")
	app := app.New()

	systrayMenu := fyne.NewMenu("Test", &fyne.MenuItem{Label: "Click"})
	if desk, ok := app.(desktop.App); ok {
		desk.SetSystemTrayMenu(systrayMenu)
	}

	systrayMenu.Items = append(systrayMenu.Items, fyne.NewMenuItemSeparator(), &fyne.MenuItem{Label: "a"}, &fyne.MenuItem{Label: "b"}, fyne.NewMenuItemSeparator(), &fyne.MenuItem{Label: "c"})

	log.Println("AddSubMenu")
	AddSubMenu(systrayMenu.Items[0], 1)
	log.Println("AddSubMenu done")

	go func() {
		for {
			time.Sleep(5 * time.Second)
			log.Println("AddSubMenu")
			AddSubMenu(systrayMenu.Items[0], 1)
			log.Println("AddSubMenu done")
			systrayMenu.Refresh()
		}
	}()
	app.Run()
}

func AddSubMenu(parent *fyne.MenuItem, level int) {
	if level > 4 {
		return
	}
	parent.ChildMenu = &fyne.Menu{Label: "Level " + strconv.Itoa(level)}
	menu := parent.ChildMenu
	for i := 0; i < 5; i++ {
		menuItem := &fyne.MenuItem{Label: strconv.Itoa(level) + "." + strconv.Itoa(i), Disabled: false}
		menu.Items = append(menu.Items, menuItem)
		menu.Items = append(menu.Items, fyne.NewMenuItemSeparator())
		AddSubMenu(menuItem, level+1)
	}
}

func StartLogging(logFile string) (*os.File, error) {
	path, _ := os.UserCacheDir()
	_, err := os.Lstat(path)
	if err != nil {
		if os.IsNotExist(err) {
			err = os.MkdirAll(path, 0755)
			if err != nil {
				log.Printf("MkdirAll failed with: %s\n", err.Error())
			}
		} else {
			log.Println(err.Error())
			return nil, err
		}
	}

	filename := path + "/" + logFile
	f, err := os.OpenFile(filename, os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
	if err != nil {
		log.Fatalf("error opening file: %v", err.Error())
	}
	log.SetOutput(f)
	return f, nil
}

Fyne version

2.3.2

Go compiler version

1.20

Operating system and version

Windows 11

Additional Information

No response

[andydotxyz]

Confirmed locally

[andydotxyz]

Looks like the reset of the menu is leaking window handles (as your error rather implied)

If you are able to test then the systray_windows.go file of the systray packge being updated with the following might confirm a potential fix:

func resetMenu() {
	const MF_BYCOMMAND = 0x00000000
	for id, res := range wt.menus {
		_, _, _ := pDeleteMenu.Call(
			uintptr(res),
			uintptr(id),
			MF_BYCOMMAND,
		)
	}
	wt.createMenu()
}

[d1ss0nanz]

I've add

func resetMenu() {
	const MF_BYCOMMAND = 0x00000000
	for id, res := range wt.menus {
		_, _, _ = pDeleteMenu.Call(
			uintptr(res),
			uintptr(id),
			MF_BYCOMMAND,
		)
	}
	log.Println("systray: resetMenu")
	wt.createMenu()
}

Fixes the system allowance for handles issue.

But still get this after around 5 minutes:

2023/03/30 07:48:24 AddSubMenu
2023/03/30 07:48:24 AddSubMenu done
2023/03/30 07:48:24 systray: resetMenu
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: The operation completed successfully.
2023/03/30 07:48:24 systray error: unable to addSeparator: The operation completed successfully.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: The operation completed successfully.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addSeparator: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addSeparator: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Invalid menu handle.```

And soon after:
```2023/03/30 07:48:24 systray error: unable to addSeparator: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addSeparator: The operation completed successfully.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: The operation completed successfully.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Not enough memory resources are available to process this command.
2023/03/30 07:48:24 systray error: unable to addSeparator: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Not enough memory resources are available to process this command.
2023/03/30 07:48:24 systray error: unable to addSeparator: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Not enough memory resources are available to process this command.
2023/03/30 07:48:24 systray error: unable to addSeparator: Invalid menu handle.
2023/03/30 07:48:24 systray error: unable to addOrUpdateMenuItem: Not enough memory resources are available to process this command.```

[andydotxyz]

Thanks, I think maybe it was a too naïve fix. Will dig into it a bit more, perhaps we are leaking multiple handles, or maybe delete needs to be called in a certain way.

[d1ss0nanz]

Would using DestroyMenu work?

u32 = windows.NewLazySystemDLL("User32.dll")
var pDestroyMenu = u32.NewProc("DestroyMenu")

func resetMenu() {
	log.Println("systray: resetMenu")
	_, _, _ = pDestroyMenu.Call(uintptr(wt.menus[0]))
	wt.visibleItems = make(map[uint32][]uint32)
	wt.menus = make(map[uint32]windows.Handle)
	wt.menuOf = make(map[uint32]windows.Handle)
	wt.menuItemIcons = make(map[uint32]windows.Handle)
	wt.createMenu()
}

[d1ss0nanz]

With the changes above, I can't reproduce any of the issues.

[andydotxyz]

Thanks, can you check out the PR and see if that's correct now?

[d1ss0nanz]

Looks good. Can't reproduce the handle issues.

[andydotxyz]

Thanks for your help, this is now merged on develop