Microsoft Azure WebJobs SDK 'Storage' connection string is missing or empty

[marcelvandendungen]

I'm trying out the latest Azure-functions-cli, but cannot seem to get past this issue. When I run 'func function run , the web service window is launched and an error appears:

A ScriptHost error has occurred
Microsoft.Azure.WebJobs.Host: Microsoft Azure WebJobs SDK 'Storage' connection string is missing or empty. The Microsoft Azure Storage account connection string
can be set in the following ways:

1. Set the connection string named 'AzureWebJobsStorage' in the connectionStrings section of the .config file in the following format
   , or
2. Set the environment variable named 'AzureWebJobsStorage', or
3. Set corresponding property of JobHostConfiguration.

I tried both 1 and 2, but neither seems to have any effect.

Azure Functions Cli (1.0.0-beta.93)
Function Runtime Version: 1.0.10774.0

Thanks,
-- Marcel

[ahmelsayed]

You need to set that value in the appsettings.json file. or through the command func settings add AzureWebJobsStorage <connectionString>

[ahmelsayed]

@lindydonna
we currently favor appsettings.json over set environment variables. Maybe it should be the other way around?

[lindydonna]

@marcelvandendungen Unfortunately, the error message string comes from the WebJobs SDK (a dependency of Azure Functions), so the message is not helpful. We have a tracking bug to improve this: #38

@ahmelsayed That's a good question--I don't know which direction is the right one. My gut feeling is that our current approach is the right one (appsettings.json is more specific than environment variables), but we should get more feedback.

[lindydonna]

@mabdelmonemali Thanks for reporting the error, we're going to also make it so that environment variables take precedence over appsettings.json. See #91

[marcelvandendungen]

Thanks, I've got it working. Yes, environment variables before appsettings.json makes sense.

[lindydonna]

@marcelvandendungen Thanks for confirming; there was some discussion in the team about the right order of convention. :)

[davidsk]

There is a requirement for an AzureWebJobsStorage property in the appsettings.json purely for compatibility with the WebJobs SDK, it's not required by the azure function cli, correct?

[markmonster]

@davidsk actually the AzureWebJobsStorage is required when you want to run a Timer triggered Azure Function. You don't need it for the http-triggered Azure Functions indeed.

[davidsk]

And this should be set to the actual value even when developing locally?

[markmonster]

I'm using the Azure Storage Emulator during development.

"AzureWebJobsStorage": "UseDevelopmentStorage=true"

Works like a charm.

[lindydonna]

@davidsk You can use the Azure storage emulator, but it may not work exactly like Azure Storage. As @mkamonster explained, the setting is required for timer triggers, and in fact all triggers aside from HTTP.

[davidsk]

@lindydonna I've got the storage emulator installed now. It's not particularly clear in the docs or blog posts that storage is necessary for timer triggers and easy to confuse the requirement with storage based triggers

[lindydonna]

@davidsk Thanks for the feedback, we're going to improve the documentation and CLI error messages around this. See Azure/Azure-Functions#108 and #38

[markmonster]

but it may not work exactly like Azure Storage

@lindydonna Can you eloborate a bit on this statement? Should we prevent testing against the Azure Storage Emulator because of anomalies?

[lindydonna]

@mkamonster It's hard to enumerate all the differences, but in general, you should not assume that if something worked correctly with the Storage Emulator, that it will also work correctly in Azure. If you want to use the emulator, then you should do another round of testing with a real storage account before deploying to Azure.

[securityvoid]

FYI, this is still pretty confusing. I set the following in my function.json:

      "connection":"AZURE_STORAGE_CONNECTION_STRING"

At least for local development this made it so I had to specify both variables in appsettings.json: AZURE_STORAGE_CONNECTION_STRING
AzureWebJobsStorage

I'll be able to test later if when it goes to the actual app if its smart enough to only use one or the other, or if I really have to specify both. If so, then mine as well stick with the default variable.

It is also worth noting that the default appsettings.json has "IsEncrypted": true set, which made it completely ignore my unencypted settings when I added them manually, which kept this error from going away.

Once I changed it over to "IsEncrypted" : false, it took my plaintext values and this error went away.

[lindydonna]

@securityvoid You can use the value "connection": "AzureWebJobsStorage" if you want the default. AzureWebJobsStorage is required as the runtime creates queues for internal use.

[jfoshee]

I've tried everything I can think of w/ web.config, appsettings.json and local.settings.json and I still cannot resolve the error connection string is missing or empty...
I download the func tool yesterday: 1.0.0-beta.96.

[ahmelsayed]

@jfoshee can you check whether the IsEncrypted flag represent the actual state of the file?

[jfoshee]

Yes, thanks @ahmelsayed and @securityvoid, the IsEncrypted flag was not present and defaulting to true. But there were multiple layers of confusion for me. In the hopes that my feedback can be useful, here are all the ways I was stymied:

1. The error message specifically first recommends updating web.config XML in the traditional way:
   1. Set the connection string named 'AzureWebJobsAzureWebJobsDashboard' in the connectionStrings section of the .config file in the following format <add name="AzureWebJobsAzureWebJobsDashboard" connectionString="DefaultEndpointsProtocol=http|https;AccountName=NAME;AccountKey=KEY" />
   As am working in a web project, as directed by this guide, I already have a web.config file, so I added my connection string there.... I was never able to get this to work.

2. The second thing that's confusing is obviously the weird AzureWebJobsAzureWebJobsStorage connection string name... My function.json includes "connection": "AzureWebJobsDashboard", which is what I grabbed from an existing .csx AzureFunction. So it's pretty confusing whether I need to prepend my connection string name, or remove AzureWebJobs from my connection: value. /shrug

3. When I tried to use appsettings.json I overlooked the error at the top of the output that says it should be named local.settings.json. I saw the discussion about renaming that file in another issue. Clearly it's frustrating if these patterns keep changing. Especially given that I have live functioning Core web apps that are still using appsettings.json. So is this change peculiar to func? Or will it be changing for standard Core web apps too?

4. When I tried to add my connection string to local.settings.json I overlooked the tell-tale error at the top of the output:
   The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.
   Clearly it would be better if the error specifically pointed out I was providing an unencrypted connection string while IsEncrypted was set to true. And that I could encrypt the string from the command line or explicitly set the flag to false. I lost the most time to this issue. I recommend scanning the provided connection string for the tokens DefaultEndpointsProtocol, AccountName, AccountKey, ... If one of those tokens is there, the string is not encrypted.
   I also recommend IsEncrypted default to false. I presume the goal is to help people not commit secrets to source control. But if this encryption is using the machine key, will the encrypted string work on other developers' machines? More importantly though, new users want to first get this working, then think about key control. Moreover web.config and appsettings.json default to unencrypted connection strings.

A minor confusion which persists with the .json files is whether connection strings should go under values: or connectionStrings:... /shrug

The feeling I'm left with is that the func team is departing from established patterns used in web projects. But I'm not on the bleeding edge of that stuff, so I don't know for sure. My ultimate recommendation would be to stay as consistent as possible with typical web projects so that developers can bring over that experience. I can add a connection string to a web.config in my sleep, and that experience did not help me here.

I'm a fan of Azure Functions, which is the reason I persisted this far. So I am very glad you are continuing to improve the tooling. Thanks for listening. Now, back to watching MS Build. ;-)

[lindydonna]

@jfoshee Thanks for the detailed feedback. I agree that the error messages are quite confusing!

To answer your questions:

1. We're going to ship a fix soon to provide a detailed warning when connection strings are missing. And, we won't even launch the functions host if that connection string is missing (we can't control that error message, because it comes from the underlying functions runtime, which uses WebJobs). See When connection strings are missing, provide a tip on how to fetch app settings #38

2. This error should go away once we fix issue 1.

3. This file is specific to func.exe, and we found that the previous name caused a lot of confusion. The old filename will still work, it's just deprecated. Core appsettings.json are a different format, which is another reason we changed our name.

4. We have two fixes for this: Provide a better error message when failing to decrypt a setting #95 and Change default of func init to IsDecrypted=false #140

You're right that we have a different pattern than web projects, but we're working on making the error messages and help more clear so that customers can learn how we do things.

[eryon]

@lindydonna I'm encountering the AzureWebJobsAzureWebJobsStorage missing error message, running with host CLI 1.0.10726.0. I'm trying to run two function projects locally, so I'm using func host start -p 7072 for the second one and that's where it's failing. Is that the wrong way to do this?

[ahmelsayed]

@eryon can you verify the IsEncrypted flag in local.settings.json is correctly reflecting the state of the file?

[eryon]

@ahmelsayed sure, it is false, and my values are not encrypted

[ahmelsayed]

What about the path you're launching func host start -p 7072 from. Is that the root of your project where local.settings.json is? There was a bug that when launching the command from a sub directory in your function app it won't load the settings. it should be fixed in 1223f59 which will go out with the next update

[eryon]

I was launching it from the bin\Debug\net461\ directory, as I'm using the VS2017 pre-compiled stuff and that's where all the generated stuff ends up (from the project root, it couldn't detect any functions as function.json hasn't been generated until build).

[ahmelsayed]

can you set your local.settings.json to be copied to the output? it just needs to have that file in the CurrentDirectory of the process when launching func host start

[eryon]

It's there, and is where I checked first to ensure the normal storage key was in place :)

[ahmelsayed]

hmmmm, then I'm puzzled. I don't really know what could be going wrong. If you do func settings list (optional --showValue) does it correctly list the settings you have?

[eryon]

@ahmelsayed it does not! See below:

D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461 [master ≡ +4 ~1 -0 !]> func settings list
App Settings:
Connection Strings:
D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461 [master ≡ +4 ~1 -0 !]> ls


    Directory: D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         6/2/2017     10:44                Spawner
-a----         6/2/2017     11:06              4 host.json
-a----         6/2/2017     08:51            778 local.settings.json

And just to prove the settings file really has content:

D:\dev\limn\RenderSpawner\RenderSpawner\bin\Debug\net461 [master ≡ +4 ~1 -0 !]> cat .\local.settings.json
{
  "IsEncrypted": false,
  "Values": {
    "ApplicationId": "<privacy>",
    "ApplicationSecret": "<privacy>",
    "AuthenticationAuthority": "https://login.windows.net/<privacy>",
    "AuthenticationTokenEndpoint": "https://management.azure.com/",
    "AzureSubscriptionId": "<privacy>",
    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=<privacy>"
  }
}

[lindydonna]

Maybe it's a file encoding issue? Can you open it in VS code and ensure it has the right encoding and line endings?

[eryon]

@lindydonna just the local.settings.json? Sure, it's UTF-8, CRLF all around.

[lindydonna]

@eryon Try this: rename your current file to local.settings.json.bak and add a couple of settings manually through the CLI (with encryption off). Then diff the two files.

[eryon]

@lindydonna when I do that, it generates an appsettings.json instead. Encoding and line endings are also UTF-8 and CRLF in the new file.

Interestingly, when I then run func host start I get a different error:

A ScriptHost error has occurred
Microsoft.Azure.WebJobs.Host: Error indexing method 'Functions.Spawner'. Microsoft.Azure.WebJobs.Host: Cannot bind parameter 'log' to type
TraceWriter. Make sure the parameter Type is supported by the binding. If you're using binding extensions (e.g. ServiceBus, Timers, etc.) m
ake sure you've called the registration method for the extension(s) in your startup code (e.g. config.UseServiceBus(), config.UseTimers(),
etc.).

My function has the following signature:

[FunctionName("Spawner")]
public static async Task Run([QueueTrigger("to-render", Connection = "AzureWebJobsStorage")]string item, TraceWriter log)

And for what it's worth, this does run from VS2017 Preview 3 without a hitch. It's only having these problems when I try to use the CLI directly.

[ahmelsayed]

huh, what version are you using? the current version is 1.0.0-beta.97, you can get that just by running func

[eryon]

@ahmelsayed VS2017 installed beta97, so I assumed that was it, but when I run it in powershell, it's reporting beta91! Is there an easy way to upgrade the CLI used by powershell?

[eryon]

@ahmelsayed @lindydonna thanks for the support. I uninstalled azure-functions-cli and installed azure-functions-core-tools using npm -g and now I'm running the beta97 CLI, and it's working fine!

I also found this article indicating that I can specify the host port in the settings file. I tried that, in order to run it from VS2017, but it's not having any effect... still launches on 7071 when my local.settings.json has

  "Host": {
    "LocalHttpPort": 7072
  }

Maybe a separate issue?

[ahmelsayed]

yeah, unfortunately VS passes --port 7071 which takes precedence over the file.

[eryon]

@ahmelsayed Is there a github repo I can open a feature request for to stop that? :)

[ahmelsayed]

@eryon yes, this repo :)

I think VS should not pass that port number and let the cli or the user decide. But feel free to open an issue on this repo and will discuss it.

[lindydonna]

@ahmelsayed Good point about VS, I'll open a bug to track that feature.

[lindydonna]

@eryon You can change the port that VS uses by modifying the launch options. Change it to "host start -p 7072"

[lindydonna]

I opened this bug for the change to VS: Azure/Azure-Functions#325

[eryon]

@lindydonna thanks so much! By launch options, I took you to mean Application arguments in the Project properties > Debug section, and that works just fine :)

[keithdhodo]

I am running into this same issue when trying to write to TableStorage with the following binding:

[Table("Chapter01", Connection = "AzureWebJobsStorage")] IAsyncCollector outputTable

Any thoughts? I definitely have the value AzureWebJobsStorage in my appsettings.json file.

[lindydonna]

@ssfcultra Please open a new issue as we generally don't monitor closed ones.

Can you paste the contents of appsettings.json, eliding any connection strings? Also, note that you should rename your settings file to local.settings.json, as the older name is deprecated.

[keithdhodo]

Thanks @lindydonna - I have added a new issue here: 204. I have also added an issue for binding to Blob Storage using the Web Jobs bindings syntax. That issue is here: 201.

I am including both here since I imagine that once I solve one they both will make sense.

Thanks!

[ConnorMcMahon]

@ahmelsayed, it looks like this issue isn't fixed with dockerized Azure Functions.

[ahmelsayed]

@ConnorMcMahon are you referring to the error message?

[ConnorMcMahon]

Yes. I think that this issue would have been a better place for me to link that post.

[ahmelsayed]

This has nothing to do with the cli (core tools) the cli doesn't run in the docker container, just the runtime. The runtime outputs this bogus error message. The issue should be on the runtime

[pilarodriguez]

I have a WebJob with some scheduled jobs (using TimerTrigger). I've specified the AzureWebJobsStorage connection string in the connectionStrings.config file
<add name="AzureWebJobsStorage" connectionString="UseDevelopmentStorage=true" />
But I get this error

The listener for function 'ScheduleTask.CronJob' was unable to start.
Value cannot be null.
Parameter name: connectionString

If I add it as an appSetting, it works fine
<add key="AzureWebJobsStorage" value="UseDevelopmentStorage=true" />

In some other places in my code I use the AzureWebJobsStorage as a connection string and I don't want to add an extra app setting. Is there a way to make this work specifying the value as a connection string and not as app setting?

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.