Support Azure Storage Emulator #53

rustd · 2014-08-08T16:02:33Z

We should be able to run locally on the emulator. Today the emulator has some limitations as compared to Azure Storage service. Maybe we throw exceptions for the cases where we don't work.
The getting started experience is hard without this support

marckassay · 2015-05-29T17:37:51Z

I seem to be experiencing this with Emulator 4.0.0.0. Has this issue been resolved yet?

rustd · 2015-06-10T03:49:46Z

Not yet. The SDK only works with Azure Storage

niclaslindberg · 2015-07-01T09:07:25Z

Are there any known workarounds for this?

mathewc · 2015-07-01T19:12:23Z

Unfortunately no. Local emulator is missing features that the SDK requires to function (e.g. blob logs).

agrigg · 2015-08-21T16:27:44Z

+1

MarkStokes · 2015-08-26T12:52:27Z

+1 We need this ASAP.

Forcing developers to pay for azure storage during their development is not cool when we used to be able to do it locally on our dev machines.

mathewc · 2015-09-03T23:44:39Z

We hear you guys, and we'll try to take a look at this soon. While there are some features of storage the SDK leverages (e.g. Blob logs) that don't work in the emulator, we may be able to provide a downgraded experience when running locally (you can probably live w/o having BlobTrigger trigger immediately for new blobs, which requires Blob logs).

jculverwell · 2015-09-10T17:29:28Z

Even a downgraded experience would be extremely useful.

gweinhold · 2015-09-10T20:31:38Z

I've been able to work around this by including an #if DEBUG statement to manually pull from a queue instead of the lack of support for the JobHost class (and corresponding QueueTrigger) in development.

https://gist.github.com/gweinhold/58f4e928378ab6329a3f

jculverwell · 2015-09-12T04:37:25Z

Thanks gweinhold, this is a great workaround

spewu · 2015-12-04T11:34:28Z

@mathewc Any news on this?

mathewc · 2015-12-04T17:49:53Z

I've just moved this from backlog to the v1.2.0 milestone for "soonish" investigation.

dkallen · 2015-12-22T19:25:23Z

I would use this. But it's not essential as I can use Azure accounts in the meantime.

bmrpatel · 2016-01-05T00:16:24Z

👍 Cool, it will be true dev experience if supported locally. Thanks

tyeth · 2016-01-21T13:55:53Z

I can't use Azure to test due to multiple MS accounts that are f-d and still linked to my card, so the local emulator is my only hope. Shame it's lacking... Maybe time to consider Azure/C# alternatives, everyone knows and loves JS

fwd079 · 2016-02-19T11:39:31Z

I tried this link:
https://github.com/Azure/azure-webjobs-sdk/wiki/Running-Locally

And my code is:

Environment.SetEnvironmentVariable("AzureWebJobsEnv", "Development");
            if (!VerifyConfiguration())
            {
                Console.ReadLine();
                return;
            }
            CreateDemoData();
            var config = new JobHostConfiguration();

            if (config.IsDevelopment)
            {
                config.UseDevelopmentSettings();
            }

            JobHost host = new JobHost(config);
            host.RunAndBlock();

But it failed with this :

Exception thrown: 'System.InvalidOperationException' in mscorlib.dll

Additional information: Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure.

If I stop at: if (config.IsDevelopment) the value is true but I still get the exception.

Please can you help? Thanks.
.

NBelham · 2016-02-23T06:59:30Z

When developing my WebJob with a local development queue, mscorlib.dll is throwing an InvalidOperationException with message "Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure".

So forcing me to use an Azure hosted Queue when developing my Web Job seems a rather awkward step I must take and goes against the tenant of the Azure SDK's capabilities of emulating Azure Storage capabilities locally.

Please can you provide a time frame when I can develop my WebJob without a dependency on a live Azure Storage account?

dkallen · 2016-02-23T14:56:13Z

For those who are developing locally and don't want to pay for an Azure account to develop, I just learned of the free Developer program that offers $25/month in free Azure services. https://www.visualstudio.com/en-us/products/visual-studio-dev-essentials-vs.aspx. That doesn't sound like much, but a Queue or Blob does not cost that much to operate with amounts of data and transaction volumes typical of development activities. So it may be a good option for many people.

fwd079 · 2016-02-26T19:44:30Z

I have a paid account on azure :) but I'd rather test locally before
putting it up. Thanks for the info though.

On Tuesday, 23 February 2016, David Kreth Allen notifications@github.com
wrote:

For those who are developing locally and don't want to pay for an Azure
account to develop, I just learned of the free Developer program that
offers $25/month in free Azure services.
https://www.visualstudio.com/en-us/products/visual-studio-dev-essentials-vs.aspx.
That doesn't sound like much, but a Queue or Blob does not cost that much
to operate with amounts of data and transaction volumes typical of
development activities. So it may be a good option for many people.

—
Reply to this email directly or view it on GitHub
#53 (comment)
.

https://www.profiles.google.com/fwd079

LeeCampbell · 2016-04-06T04:52:40Z

This isn't just a cost issue. Not all development (or testing) should happen with network connectivity required. Other platforms allow you to test or dev locally or even in in-memory mode. It is an insight to the culture of the team releasing this software that this wasn't in by default and still hasn't been actioned in 20months.

plaurin · 2016-04-14T18:18:18Z

My problem is that I am working in a team currently working with the Storage Emulator was easy for us now I need to setup all members of my team with proper Azure Subscriptions or find a way to all work on the same Subscription but different Storage Account or Queues. Storage Emulator would be easier than configuring all our web apps and tooling to work in Azure in separated environments.

ghost · 2016-04-19T13:36:49Z

Any update on when this issue will be resolved? We are starting a new project, and will be relying heavily on Azure storage with WebJobs - not being able to run them locally is a HUGE issue for us, as we have times where we need to develop/demo our product without internet connections being available.

How has this been able to happen for so long? Many people rely on Azure for large scale apps, and issues like this really slow down development velocity!

ScottyMac52 · 2016-04-21T21:53:19Z

I personally think it is ridiculous and unprofessional for this pretty major issue to go 2 years without being resolved or at least seriously looked at, I have plenty of evidence to support this claim:

This issue is not even assigned to anyone
The milestone hasn't moved
No labels have been applied

I mean, what gives??? If you are NOT going to fix the issue then say so and then allow someone from the community to come up with a work around solution.

christopheranderson · 2016-04-21T22:43:01Z

Sorry for the lack of a clear answer here. It's a tricky issue because, for one reason or another, the SDK was never designed with the emulator in mind and there are a lot of gaps between the emulator and the actual Storage products which would impact the SDK.

I don't think that this is a good time for the Azure WebJobs team to investigate this any further, thus I'm moving this into backlog for the time being.

With that said, we're always open for community requests on issues we haven't closed. If you're interested in investigating the gaps and finding work arounds for them, we're happy to work with you to do that in the best way possible.

You can always reach out to me at chrande (at) microsoft (dot) com if you want to discuss this or any other feature and its relative priority.

ctolkien · 2016-04-21T22:47:45Z

Should this discussion be flipped around? Instead of getting WebJobs SDK working on the emulator, should the emulator be improved to close the gap between itself and "real" Azure storage?

Doesn't look like the emulator bits are on github however.

PureKrome · 2016-08-10T22:56:02Z

Thanks kindly @mathewc :) Looking forward to the 2.0 RTM/RTW.

Keep up the good work, MS / webjobs-team. 🍰

xt0rted · 2016-08-20T04:12:51Z

Watch out if you're using the myget feed to get this update. My builds just started failing because older packages were removed from the feed.

mathewc · 2016-08-22T15:33:57Z

@xt0rted What myget package are you saying doesn't work? Both the nuget and myget packages should work just fine. I'm not aware of any issues :)

xt0rted · 2016-08-22T18:48:20Z

@mathewc I was using version 2.0.0-alpha-10313 of the Microsoft.Azure.WebJobs package. My build server started failing on the 19th so this package was probably removed earlier that day or later on the 18th when 2.0.0-beta1-10381 was published.

mathewc · 2016-08-22T19:03:04Z

I think myget is doing auto-purge on pre-release versions - we haven't been deleting those ourselves. We need to look into this more - might be something we can configure.

mathewc · 2016-08-22T19:14:48Z

Yes, I confirmed that a default package retention policy that we never created was in effect, and it was configured to retain only 20 versions. I've removed that restriction so now all versions will be kept going forward. Thanks for reporting this :)

danpetitt · 2016-11-07T15:03:45Z

Any news on a v2 RTM?

martinfletcher · 2016-11-16T12:38:15Z

Could this feature be added to a 1.x release rather than waiting for 2.0?

This has been a known issue for a long time and has put a burden on developers wanting to use queues locally! Pre-Release packages are fine when checking out new features or getting to know a new SDK, but developing against them for production apps is far from ideal!

If 2.0 is not right around the corner from being release (judging from the milestone, it seems far from it!), please release a patch to fix this issue for the current 1.x.

zeeshanejaz · 2016-11-21T02:33:10Z

How come this issue is closed? I don't see the emulator support in the SDK :-/ Am I missing something?

xt0rted · 2016-11-21T02:59:17Z

@zeeshanejaz sadly it never made it into v1. It's in v2 but there hasn't been a stable release yet. You can get it on NuGet in one of the the 2.0.0-beta2 builds. I've been using the preview builds ever since this was merged and haven't had any issues with them.

It would be nice if this could be pulled into v1 since we've been waiting 6 months for a release and there hasn't been one.

dapug · 2017-01-14T03:23:38Z

I'm not getting this to work at all using 2.0.0-beta2, Emulator v4.5. Should I open a new issue?

error: System.InvalidOperationException: 'Invalid storage account 'devstoreaccount1'. Please make sure your credentials are correct.'

string: "UseDevelopmentStorage=true;"

Total loss here, reading this thread it seems this should totally work.

mathewc · 2017-01-20T01:40:17Z

Yes, if you're having issues with the emulator in the beta builds, please create a new issue.

zeeshanejaz · 2017-01-30T20:59:56Z

In my experience WebJobs SDK 2.0.0-beta2 works with Emulator if you are using WindowsAzure.Storage 7.2.1 but doesn't work if you are using WindowsAzure.Storage 8.0.1. In the latter case, it throws up an exception "Invalid storage account: 'devstoreaccount1'".

I hope this helps someone

martinfletcher · 2017-02-07T12:41:49Z

Any update on release date for V2 - This fix is much needed by many people!

NathanNZ · 2017-02-17T01:11:11Z

@zeeshanejaz : This is resolved with 8.x of the WindowsAzure.Storage library if you grab the latest emulator (v4.6 at time of writing) https://go.microsoft.com/fwlink/?linkid=717179&clcid=0x409

ScottyMac52 · 2017-02-17T17:07:06Z

Wow, that is certainly great news, we don't need it anymore but I do remember being chastised back on 4/16/2016 so maybe I am owed an apology for that since I am the one that got this whole thread kick started? Regards, Scott McIntosh Principal Software Engineer

…

On Feb 16, 2017 5:11 PM, "Nathan Holland" ***@***.***> wrote: @zeeshanejaz <https://github.com/zeeshanejaz> : This is resolved with 8.x of the WindowsAzure.Storage library if you grab the latest emulator (v4.6 at time of writing) https://go.microsoft.com/fwlink/?linkid=717179&clcid= 0x409 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANTXFGpRzeN5RP6WAvXSRKCOJksPJ8bMks5rdPOxgaJpZM4CVjrK> .

mchudinov · 2017-02-28T15:18:59Z

I'm running emulator v 4.6. Still does not work at least from VS 2015 ASP.NET MVC 4.6.2 project.
WindowsAzure.Storage v 8.1.1

Same error "Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure." from my WebJob

connection strings
<connectionStrings> <add name="AzureWebJobsDashboard" connectionString="UseDevelopmentStorage=true" /> <add name="AzureWebJobsStorage" connectionString="UseDevelopmentStorage=true" /> </connectionStrings>

isaacabraham · 2017-02-28T16:02:15Z

This seems to work reasonably well with 4.6. However, if you give it a reasonable amount of blobs to process in parallel - say, 10, it often just pops with a 500 server error. Not been able to trace it to anything more than that.

HerrKater · 2017-03-02T12:33:38Z

Still not working for me on .net 4.6 with the latest storage SDK & emulator. I can't believe that this has been an issue for 3 years.

isaacabraham · 2017-03-02T21:53:40Z

The latest emulator seems to have some breaking changes in it as well - at last one has broken the F# Storage Type Provider - that lead to null reference exceptions being thrown.

StefH · 2017-03-06T08:22:38Z

I'm using version 4.6 in VS 2015, and I can connect (no exceptions), however I don't know how I can define to which storage account I want to connect to.

Note that my local storage account is define like: http://127.0.0.1:10000/devstoreaccount1/webjob-test

And my handler method looks like:

public static void RenameFile(
    [BlobTrigger("input/{blobName}.exe")] Stream input,
    [Blob("output/processed_{blobName}_final.exe", FileAccess.Write)] Stream output)
{
    input.CopyTo(output);
}

Can you please help me?

mathewc · 2017-03-08T01:13:03Z

I vote that we leave this issue and open new issues as needed for new emulator problems. This issue was all about removing some explicit restrictions we had in place, which we've done. This will make it easier to track things for us, thanks.

isaacabraham · 2017-03-08T08:29:46Z

@MatthewC do you want me to raise another one then related to many blobs created at once?

mathewc · 2017-03-09T02:40:05Z

Yes, I want this issue to die so we can start fresh, and track any new issues people have in new issues :) However, note that if you're running into bugs in the emulator or limitations with the emulator, we can't fix those - you should instead log bugs against the storage SDK / emulator itself. Our statement on emulator support as resolved for this issue is that we no longer block it as we were before, however we make no strong guarantees on how well it will work for you. E.g. we know there are features of the WebJobs SDK that use storage features that the emulator doesn't support (e.g. blob leases, at least in the past that was an issue) so YMMV. I view the emulator as a way to try out simple scenarios, however I always use a full test account for my work.

alexjustus · 2017-09-26T01:00:33Z

Regarding the invalid operation exception:

Exception thrown: 'System.InvalidOperationException' in mscorlib.dll

Additional information: Failed to validate Microsoft Azure WebJobs SDK Dashboard account. The Microsoft Azure Storage Emulator is not supported, please use a Microsoft Azure Storage account hosted in Microsoft Azure.

One way to get around this limitation is to create a Console app rather than an Azure WebJob project. You can then add the Microsoft.Azure.WebJobs nuget package manually, add the standard WebJob boilerplate code to Program.cs and everything should just work.

Visual Studio even lets you publish it as an Azure WebJob like normal via the right-click menu (at least VS 2017 does). It also runs just fine in Azure.

jdvanegas · 2018-05-07T16:54:55Z

I found the correct way to connect, I have not seen any similar comments so I share it.

Note: Not work with UseDevelopmentStorage=true

https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string?toc=%2fazure%2fstorage%2fqueues%2ftoc.json

JustinBeckwith · 2019-03-28T04:59:57Z

ping is this fixed yet?

rustd added this to the Backlog milestone Aug 20, 2014

mathewc modified the milestones: 1.2.0, Backlog Dec 4, 2015

christopheranderson modified the milestones: Backlog, 1.2.0 Apr 21, 2016

mathewc removed this from the Backlog milestone Aug 10, 2016

isaacabraham mentioned this issue Mar 13, 2017

WebJobs Blob Trigger does not work correctly with Storage Emulator Azure/azure-storage-net#430

Closed

cori mentioned this issue Mar 19, 2017

Seems like this extension doesn't support WebJobs 2.0 christopheranderson/azure-webjobs-sdk-extensions-slack#2

Open

Support Azure Storage Emulator #53

Support Azure Storage Emulator #53

Comments

rustd commented Aug 8, 2014

marckassay commented May 29, 2015

rustd commented Jun 10, 2015

niclaslindberg commented Jul 1, 2015

mathewc commented Jul 1, 2015

agrigg commented Aug 21, 2015

MarkStokes commented Aug 26, 2015

mathewc commented Sep 3, 2015

jculverwell commented Sep 10, 2015

gweinhold commented Sep 10, 2015

jculverwell commented Sep 12, 2015

spewu commented Dec 4, 2015

mathewc commented Dec 4, 2015

dkallen commented Dec 22, 2015

bmrpatel commented Jan 5, 2016

tyeth commented Jan 21, 2016

fwd079 commented Feb 19, 2016

NBelham commented Feb 23, 2016

dkallen commented Feb 23, 2016

fwd079 commented Feb 26, 2016

LeeCampbell commented Apr 6, 2016

plaurin commented Apr 14, 2016

ghost commented Apr 19, 2016 • edited by ghost Loading

ScottyMac52 commented Apr 21, 2016 • edited Loading

christopheranderson commented Apr 21, 2016

ctolkien commented Apr 21, 2016

PureKrome commented Aug 10, 2016

xt0rted commented Aug 20, 2016

mathewc commented Aug 22, 2016

xt0rted commented Aug 22, 2016

mathewc commented Aug 22, 2016

mathewc commented Aug 22, 2016

danpetitt commented Nov 7, 2016

martinfletcher commented Nov 16, 2016

zeeshanejaz commented Nov 21, 2016

xt0rted commented Nov 21, 2016

dapug commented Jan 14, 2017

mathewc commented Jan 20, 2017

zeeshanejaz commented Jan 30, 2017

martinfletcher commented Feb 7, 2017

NathanNZ commented Feb 17, 2017

ScottyMac52 commented Feb 17, 2017 via email

mchudinov commented Feb 28, 2017 • edited Loading

isaacabraham commented Feb 28, 2017

HerrKater commented Mar 2, 2017

isaacabraham commented Mar 2, 2017

StefH commented Mar 6, 2017

mathewc commented Mar 8, 2017 • edited Loading

isaacabraham commented Mar 8, 2017

mathewc commented Mar 9, 2017 • edited Loading

alexjustus commented Sep 26, 2017

jdvanegas commented May 7, 2018 • edited Loading

JustinBeckwith commented Mar 28, 2019

ghost commented Apr 19, 2016 •

edited by ghost

Loading

ScottyMac52 commented Apr 21, 2016 •

edited

Loading

mchudinov commented Feb 28, 2017 •

edited

Loading

mathewc commented Mar 8, 2017 •

edited

Loading

mathewc commented Mar 9, 2017 •

edited

Loading

jdvanegas commented May 7, 2018 •

edited

Loading