Get-BcContainerTenants fails if called immediately after Restore-DatabasesInBcContainer

[Sven-Niehus]

Describe the issue
The commandlet Get-BcContainerTenants fails if called immediately after Restore-DatabasesInBcContainer with the following error message:

Cannot refresh the tenant state because the tenant is being mounted.  CorrelationId:  bd29782a-5e6a-4bc7-9bcf-56c265a5a6b9.
Status: 400 (Cannot refresh the tenant state because the tenant is being mounted.)
ErrorCode: Internal_TenantUnavailable

Content:
{"error":{"code":"Internal_TenantUnavailable","message":"Cannot refresh the tenant state because the tenant is being mounted.  CorrelationId:  bd29782a-5e6a-4bc7-9bcf-56c265a5a6b9."}}

Scripts used to create container and cause the issue
Script used to create the container and to backup the database:

$containerName = "BACKUPTEST-BC"
$multiTenant = $false
$dnsServer = "8.8.8.8"
$memoryLimit = "6G"

$securePassword = try { "pass" | ConvertTo-SecureString } catch { ConvertTo-SecureString -String "pass" -AsPlainText -Force }
$credential = New-Object PSCredential -ArgumentList "user", $SecurePassword
$auth = 'UserPassword'

$artifactType = "Sandbox"
$artifactCountry = "de"
$artifactSelect = "Latest"
$artifactVersion = ""

$tenantName = "default"

if($artifactVersion -ne "") {
    $artifactUrl = Get-BcArtifactUrl -type $artifactType -country $artifactCountry -version $artifactVersion -select $artifactSelect
} else {
    $artifactUrl = Get-BcArtifactUrl -type $artifactType -country $artifactCountry -select $artifactSelect
}
Write-Host "Container will be using artifact ""$artifactUrl""."

if(-Not (Test-BCContainer -containerName $containerName)){
    Write-Host "Container doesn't exist, creating container"
    New-BcContainer `
        -accept_eula `
        -containerName $containerName `
        -credential $credential `
        -auth $auth `
        -artifactUrl $artifactUrl `
        -multitenant:$multiTenant `
        -assignPremiumPlan `
        -dns $dnsServer `
        -isolation 'process' `
        -memoryLimit $memoryLimit `
        -updateHosts `
        -additionalParameters '--net="pub"' `
        @optionalParameter
        Write-Host "Creating backup"
        Backup-BCContainerDatabases -containerName $containerName -bakFolder $containerName -tenant $tenantName
}

Script used to restore the database:

$containerName = "BACKUPTEST-BC"
$tenantName = "default"
Restore-DatabasesInBcContainer -containerName $containerName -bakFolder $containerName -tenant $tenantName

$Tenant = Get-BcContainerTenants -containerName $containerName -tenant $tenantName
$TenantState = $Tenant.State.ToString()
if($TenantState -ne "Operational") {
    do {
        Write-Host "Waiting for Tenant to be Operational..sleeping for 5 seconds"
        Start-Sleep -Seconds 10
        $Tenant = Get-BcContainerTenants -containerName $containerName -tenant $tenantName
        $TenantState = $Tenant.State.ToString()
        Write-Host "Tenant State: $($TenantState)"
        if($TenantState -eq "Mounted") {
            Write-Host "Tenant is in ""Mounted"" state, forcefully refreshing.."
            $Tenant = Get-BcContainerTenants -containerName $containerName -tenant $tenantName -ForceRefresh
        } else {
            Write-Host "Refreshing Tenant.."
            $Tenant = Get-BcContainerTenants -containerName $containerName -tenant $tenantName
        }
    } while ($TenantState -ne "Operational")
}
$User = Get-BcContainerBcUser -containerName $containerName -tenant $tenantName | Where-Object { $_.Username -eq $credential.UserName }
if(!$User) {
    Write-Host "Creating user: $($credential.UserName)"
    New-BcContainerBcUser -containerName $containerName -tenant $tenantName -credential $credential
} else {
    Write-Host "User already exists. Nothing to do."
}

Full output of scripts
Output for container creation:

BcContainerHelper is version 6.0.17
BcContainerHelper is running as administrator
HyperV is Disabled
Host is Microsoft Windows Server 2022 Standard Evaluation - 10.0.20348.2113
UsePsSession is True
UsePwshForBc24 is True
UseWinRmSession is allow
UseSslForWinRmSession is True
Docker Client Version is master-dockerproject-2022-03-26
Docker Server Version is master-dockerproject-2022-03-26
Removing Session BACKUPTEST-BC
Removing container BACKUPTEST-BC
Removing entries from hosts
Removing BACKUPTEST-BC from container hosts file
Removing BACKUPTEST-BC-* from container hosts file
Removing Desktop shortcuts
Removing C:\ProgramData\BcContainerHelper\Extensions\BACKUPTEST-BC
Fetching all docker images
Fetching all docker volumes
Using image mcr.microsoft.com/businesscentral:ltsc2022
Disabling Health Check (always report healthy)
Creating Container BACKUPTEST-BC
Style: sandbox
Multitenant: Yes
Version: 24.0.16410.19234
Platform: 24.0.19178.0
Generic Tag: 1.0.2.20
Container OS Version: 10.0.20348.2402 (ltsc2022)
Host OS Version: 10.0.20348.2113 (ltsc2022)
WARNING: Container and host OS build is 20348 or above, defaulting to process isolation. If you encounter issues, you could try to install HyperV.
Using process isolation
Using locale de-DE
Disabling the standard eventlog dump to container log every 2 seconds (use -dumpEventLog to enable)
Using license file REDACTED
Downloading C:\ProgramData\BcContainerHelper\Extensions\BACKUPTEST-BC\my\license.bclicense
Additional Parameters:
--volume "C:\ado:C:\Agent"
--expose 5986
--env customNavSettings=EnableTaskScheduler=False
Files in C:\ProgramData\BcContainerHelper\Extensions\BACKUPTEST-BC\my:
- AdditionalOutput.ps1
- AdditionalSetup.ps1
- HelperFunctions.ps1
- license.bclicense
- MainLoop.ps1
- SetupVariables.ps1
- updatehosts.ps1
Creating container BACKUPTEST-BC from image mcr.microsoft.com/businesscentral:ltsc2022
34376c0f7d15b4734147cc8dd88d759127a435a4d706775a8a0709e5ecfcdfb7
Waiting for container BACKUPTEST-BC to be ready
Using artifactUrl https://bcartifacts-exdbf9fwegejdqak.b02.azurefd.net/sandbox/24.0.16410.19234/de
Using installer from C:\Run\240
Installing Business Central: multitenant=True, installOnly=False, filesOnly=False, includeTestToolkit=False, includeTestLibrariesOnly=False, includeTestFrameworkOnly=False, includePerformanceToolkit=False, appArtifactPath=c:\dl\sandbox\24.0.16410.19234\de, platformArtifactPath=c:\dl\sandbox\24.0.16410.19234\platform, databasePath=c:\dl\sandbox\24.0.16410.19234\de\BusinessCentral-DE.bak, licenseFilePath=c:\dl\sandbox\24.0.16410.19234\de\Cronus.bclicense, rebootContainer=True
Installing from artifacts
Starting Local SQL Server
WARNING: Waiting for service 'SQL Server (SQLEXPRESS) (MSSQL$SQLEXPRESS)' to 
start...
Starting Internet Information Server
Copying Service Tier Files
c:\dl\sandbox\24.0.16410.19234\platform\ServiceTier\Program Files
c:\dl\sandbox\24.0.16410.19234\platform\ServiceTier\System64Folder
Copying Web Client Files
c:\dl\sandbox\24.0.16410.19234\platform\WebClient\Microsoft Dynamics NAV
Copying ModernDev Files
c:\dl\sandbox\24.0.16410.19234\platform
c:\dl\sandbox\24.0.16410.19234\platform\ModernDev\program files\Microsoft Dynamics NAV
Copying additional files
Copying ConfigurationPackages
C:\dl\sandbox\24.0.16410.19234\de\ConfigurationPackages
Copying Test Assemblies
C:\dl\sandbox\24.0.16410.19234\platform\Test Assemblies
Copying Extensions
C:\dl\sandbox\24.0.16410.19234\de\Extensions
Copying Applications
C:\dl\sandbox\24.0.16410.19234\platform\Applications
Copying Applications.DE
C:\dl\sandbox\24.0.16410.19234\de\Applications.DE
Copying dependencies
Importing PowerShell Modules
Restoring CRONUS Demo Database
Setting CompatibilityLevel for tenant on localhost\SQLEXPRESS
Exporting Application to CRONUS
Removing Application from tenant
Modifying Business Central Service Tier Config File for Docker
Creating Business Central Service Tier
Installing SIP crypto provider: 'C:\Windows\System32\NavSip.dll'
Starting Business Central Service Tier
Importing license file
Copying Database on localhost\SQLEXPRESS from tenant to default
Taking database tenant offline
Copying database files
Attaching files as new Database default
Putting database tenant back online
Mounting tenant database
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Stopping Business Central Service Tier
Installation took 631 seconds
Installation complete
Initializing...
Setting host.containerhelper.internal to 172.25.80.1 in container hosts file
Starting Container
Hostname is BACKUPTEST-BC
PublicDnsName is BACKUPTEST-BC
Using NavUserPassword Authentication
Creating Self Signed Certificate
Self Signed Certificate Thumbprint FDEC19FE4E1540CB5488A981421B2CBA78449A65
DNS identity BACKUPTEST-BC
Modifying Service Tier Config File with Instance Specific Settings
Modifying Service Tier Config File with settings from environment variable
Setting EnableTaskScheduler to False
Starting Service Tier
Registering event sources
Creating DotNetCore Web Server Instance
Using application pool name: BC
Using default container name: NavWebApplicationContainer
Copy files to WWW root C:\inetpub\wwwroot\BC
Create the application pool BC
Create website: NavWebApplicationContainer without SSL
Update configuration: navsettings.json
Done Configuring Web Client
Enabling Financials User Experience
Using license file 'c:\run\my\license.bclicense'
Import License
Dismounting Tenant
Mounting Tenant
Mounting Database for default on server localhost\SQLEXPRESS with AllowAppDatabaseWrite = False
Sync'ing Tenant
Tenant is Operational
Creating http download site
Setting SA Password and enabling SA
Creating agent as SQL User and add to sysadmin
WARNING: This license is not compatible with this version of Business Central.
Creating SUPER user
WARNING: This license is not compatible with this version of Business Central.
WARNING: This license is not compatible with this version of Business Central.
Enable PSRemoting and setup user for winrm
Creating self-signed certificate for winrm
Container IP Address: 172.25.81.197
Container Hostname  : BACKUPTEST-BC
Container Dns Name  : BACKUPTEST-BC
Web Client          : http://BACKUPTEST-BC/BC/?tenant=default
Dev. Server         : http://BACKUPTEST-BC
Dev. ServerInstance : BC
Dev. Server Tenant  : default
Setting BACKUPTEST-BC to 172.25.81.197 in host hosts file
Setting BACKUPTEST-BC-default to 172.25.81.197 in host hosts file
Setting BACKUPTEST-BC-default to 172.25.81.197 in container hosts file

Files:
http://BACKUPTEST-BC:8080/ALLanguage.vsix

Container Total Physical Memory is 6.0Gb
Container Free Physical Memory is 1.1Gb

Initialization took 201 seconds
Ready for connections!
Reading CustomSettings.config from BACKUPTEST-BC
Synchronizing Permissions Mock on default
App successfully synchronized
Installing Permissions Mock on default
App successfully installed
Synchronizing Test Runner on default
App successfully synchronized
Installing Test Runner on default
App successfully installed
Synchronizing Any on default
App successfully synchronized
Installing Any on default
App successfully installed
Synchronizing Library Assert on default
App successfully synchronized
Installing Library Assert on default
App successfully installed
Skipping app 'C:\Applications.DE\Microsoft_Permissions Mock_24.0.16410.19234.app' as it is already installed
Synchronizing Library Variable Storage on default
App successfully synchronized
Installing Library Variable Storage on default
App successfully installed
Publishing C:\ProgramData\BcContainerHelper\Extensions\BACKUPTEST-BC\9ab65b5c-893c-491d-97b9-ab81e2f51591\Microsoft_System Application Test Library_24.0.16410.19234.app
Synchronizing System Application Test Library on tenant default
Installing System Application Test Library on tenant default
App Microsoft_System Application Test Library_24.0.16410.19234.app successfully published
Publishing C:\ProgramData\BcContainerHelper\Extensions\BACKUPTEST-BC\58ee019b-b291-4abc-ac7e-c7dc31b18719\Microsoft_Business Foundation Test Libraries_24.0.16410.19234.app
Synchronizing Business Foundation Test Libraries on tenant default
Installing Business Foundation Test Libraries on tenant default
App Microsoft_Business Foundation Test Libraries_24.0.16410.19234.app successfully published
Publishing C:\ProgramData\BcContainerHelper\Extensions\BACKUPTEST-BC\fe163d9d-1fef-40c8-a8c2-ce06f3a681ec\Microsoft_Tests-TestLibraries_24.0.16410.19234.app
Synchronizing Tests-TestLibraries on tenant default
Installing Tests-TestLibraries on tenant default
App Microsoft_Tests-TestLibraries_24.0.16410.19234.app successfully published
TestToolkit successfully imported
Cleanup old dotnet core assemblies
Container BACKUPTEST-BC successfully created

Backing up CRONUS to C:\ProgramData\BcContainerHelper\onprem-24.0.16410.19234-DE-bakFolders\BACKUPTEST-BC\app.bak
Backing up default to C:\ProgramData\BcContainerHelper\onprem-24.0.16410.19234-DE-bakFolders\BACKUPTEST-BC\default.bak

Output for restoring the database

Cannot refresh the tenant state because the tenant is being mounted.  CorrelationId:  6d3b0a0a-369b-4609-bf0e-0a2ee7f0bfb7.
Status: 400 (Cannot refresh the tenant state because the tenant is being mounted.)
ErrorCode: Internal_TenantUnavailable

Content:
{"error":{"code":"Internal_TenantUnavailable","message":"Cannot refresh the tenant state because the tenant is being mounted.  CorrelationId:  6d3b0a0a-369b-4609-bf0e-0a2ee7f0bfb7."}}

Additional context
It was working until the 29th and thus was likely broken with the 6.0.17 update of the BcContainerHelper on the 30th.

[Sven-Niehus]

Using Get-BcContainerTenants used to work without throwing any error and I found a work around with waiting anywhere between 30s and 2 minutes using a do/while loop. It would still be great if the commandlet wouldn't throw the Cannot refresh the tenant state because the tenant is being mounted. error as using it is the only way to get the tenants state. It should fail soft instead with giving back $null for example.

[freddydk]

Did you test whether this is related to 6.0.17 or BC 24.0 ?
I would assume the problem is in 24 - and 6.0.17 would still work with 23.5?

You could try to use Wait-BcContainerReady, which also waits for tenants to be mounted instead of your loop.

[Sven-Niehus]

As far as I'm aware it only happens with the latest version of the BCContainerHelper and the latest version of BC yes.

If I understood Wait-BcContainerReady right it waits for the first occurence of Ready for connections!, but does that even get logged again if the database is restored via Restore-DatabasesInBcContainer? In my tests the log didn't show Ready for connections! again after the initial creation of the container.

[freddydk]

Wait-BcContainerReady might actually have a bug as it doesn't wait for a new occurance of Ready for conenctions - but accepts an earlier one - will test restart-bccontainer.

But anyway, you can run this:

        Invoke-ScriptInBcContainer -containerName $containerName -scriptblock {
            $boo = $true
            while (Get-NAVServerInstance | Get-NavTenant | Where-Object { $_.State -eq "Mounting" }) {
                if ($boo) { Write-Host "Waiting for tenants to be mounted"; $boo = $false }
                Start-Sleep -Seconds 1
            }
        }

To wait for all tenants are mounted.

That is being run in the end of Wait-BcContainerReady as well.

[freddydk]

Restart seems to work

will check why...:-)

[freddydk]

Ahh - I added a parameter called startLog - which is the part of the log to be ignored.
So, indeed you can just call Wait-BcContainerReady - it will wait for container and tenants ready.