Simple vanilla WebAPI causes problems in containers - inotify limits

[benclerix-cegeka]

Hi,

We have the following setup:

• Docker version 18.09.6-dev, build 95c7371304f9cef494efe93f0a8ffd53a75eac21
• .NET Core 2.2
• Unix system (based on Yocto OS)

We measured the number of notify instances with:
for foo in /proc/*/fd/*; do readlink -f $foo; done | grep inotify | sort | uniq -c | sort -nr

Once we hit the configured number of inotify instances we see the following errors:

Unhandled Exception: System.IO.IOException: The configured user limit (128) on the number of inotify instances has been reached.
   at System.IO.FileSystemWatcher.StartRaisingEvents()
   at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
   at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.TryEnableFileSystemWatcher()
   at Microsoft.Extensions.FileProviders.Physical.PhysicalFilesWatcher.CreateFileChangeToken(String filter)
   at Microsoft.Extensions.FileProviders.PhysicalFileProvider.Watch(String filter)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.<.ctor>b__0_0()
   at Microsoft.Extensions.Primitives.ChangeToken.OnChange(Func`1 changeTokenProducer, Action changeTokenConsumer)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider..ctor(FileConfigurationSource source)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationSource.Build(IConfigurationBuilder builder)
grabbing logs: error opening journald follow descriptor: "Too many open files"
grabbing logs: error opening journald follow descriptor: "Too many open files"

Temporary we fixed the error by configuring a larger number: sysctl -w fs.inotify.max_user_instances=512.

But this is not ok having into account that containers can be restarted multiple times and we will still hit this limit after 10 restarts.

What we tried:

1. Disable reloadOnChange by using custom WebHostBuilder

  public static IWebHostBuilder DefaultWebHostBuilder(string[] args)
        {
            return new WebHostBuilder()
                .UseKestrel()
                .UseContentRoot(Directory.GetCurrentDirectory())
                .ConfigureAppConfiguration((hostingContext, config) =>
                {
                    config.AddJsonFile("appsettings.json", optional: false, reloadOnChange: false);
                    config.AddEnvironmentVariables();
                    if (args != null)
                    {
                        config.AddCommandLine(args);
                    }
                })
                .ConfigureLogging((hostingContext, logging) =>
                {
                    logging.AddConfiguration(hostingContext.Configuration.GetSection("Logging"));
                    logging.AddConsole(options => options.IncludeScopes = true);
                })
                .UseDefaultServiceProvider((context, options) =>
                {
                    options.ValidateScopes = context.HostingEnvironment.IsDevelopment();
                });
      }

This didn’t help. The number of inotify instances still increased after container restart.

2. Configure ENV DOTNET_USE_POLLING_FILE_WATCHER=true in dockerfile.

This didn’t help either. The number of inotify objects increased after container restart.

We don't need any file watching, the file handle should be closed properly after the file has been read. How can we accomplish this? Updating to asp .netcore 3.0 isn't an option, a fix should be provided.

Any info and update is welcome, we can provide more info if needed.

This is related to: https://github.com/dotnet/corefx/issues/32024 however this was closed without a proper fix to turn off the file watching.

[JeremyKuhne]

This is an ask to backport #928 to 2.2.

[ericstj]

The fix is in a nuget package that already shipped. Can you just reference that nuget package?

<PackageReference Include="Microsoft.Extensions.Configuration.FileExtensions" Version="3.1.2" />

[davidfowl]

How are you hurting that limit? Are you creating multiple web host builders within the same process?

[analogrelay]

Closing, as 2.2 is end-of-life and no longer supported.