Enable ASP.NET Project System to be Serialized in Any Data Format

[Mike-E-angelo]

After reading aspnet/Announcements#24 I wanted to make a suggestion that the project loading system be expanded so that it can use other data formats if preferred by the developer and/or development team.

Some great data formats were mentioned in that thread:

• YAML
• JSON5
• XML
• TOML
• INI

And I would even recommend XAML as well as it features mature, built-in designer and tooling support right in Visual Studio. If the project.json (and supporting files) are just serialized objects, then they should be able to be easily defined/serialized in any format as long as there is a serializer/deserializer registered and defined for it.

That way, every developer can design their project files the way they prefer (comments and all) and the project system still works as expected. Everyone wins. :)

Thank you for any consideration!

[Mike-E-angelo]

Incidentally, this is an issue that is also facing MSBuild vNext, with a good discussion found at dotnet/msbuild#16 . Maybe there can be some inter-project collaboration to create an innovative solution to that can be beneficial to both projects (and others).

[Eilon]

The classic problem here is: who loads the loader? And how do other tools integrate with the system? Though there is certainly value in having more than one way to do something, there are often also very large costs to having extensibility. The core issue here is that DNX itself has to read project.json and has its own JSON parser (which is what that Announcements thread is all about). If it were to use another format, how would it even know where to load that loader?

[Mike-E-angelo]

True there are costs... but are they greater than the costs of potentially turning away developers because you are championing (correctly perceived or not) one format over the other? Or better yet, is it worth the cost to enable developers to bring-your-own-format (BYOF haha) to your platform, thereby increasing attractiveness and (therefore) adoption? I especially think of the cataclysmic divide right now in MSFT's own ecosystem of UWP/.NET vs., well ASP.NET. Everything seems to be Xaml/Xml in MSFT except what is being developed by this group/division. Offering this capability would certainly ease this dichotomy (and perhaps further encourage innovation solutions to reduce it even more).

As for actual implementation, that is up for the smartypants on your team here to (theoretically) think about, IMO. 😄 But, I would say that this would be a setting in the .xproj file. In fact, IMO we shouldn't even see a "project.json" file in the application node tree, but should see a "Project" node under "Properties" folder (since there is already a launchSettings file there already, which itself should be "Launch Settings" rather than "launchSettings.json -- again IMO). If you select the node, and look at the Properties of this file, there could be a "Format" drop down that the developer/team could use to select their preferred format. It could default to the setting in the .xproj file, but could further be overridden here in the properties pane for each file on a granular level, if desired.

Lots of options here. If it helps, I would be interested in assisting in this endeavor in any way possible.

[davidfowl]

I can tell you with great certainty that we're not going to support multiple project file formats in a first class way. There's little to gain from it at a large cost. Consistency, tooling costs, whenever we add a new feature we now have to think about how it;s expressed in json, xml, yaml, ini, toml? That's a bit absurd IMO.

But, I would say that this would be a setting in the .xproj file

Nope, the xproj file is a visual studio artifact. Project.json is where the meat is (and project.lock.json)

In fact, IMO we shouldn't even see a "project.json" file in the application node tree, but should see a "Project" node under "Properties" folder (since there is already a launchSettings file there already, which itself should be "Launch Settings" rather than "launchSettings.json -- again IMO)

That's not a crazy idea for visual studio, but we run in more editors. Plus I think hand editable files is actually a nice change from having purely heavyweight tool operated files (ala csproj)

If you select the node, and look at the Properties of this file, there could be a "Format" drop down that the developer/team could use to select their preferred format. It could default to the setting in the .xproj file, but could further be overridden here in the properties pane for each file on a granular level, if desired.

That leads to inconsistencies across the ecosystem and complexity in tooling that wants to support all of them.

We are OSS so you could take the runtime code and read any file format but we're not going to do anything like that out of the box.

[Mike-E-angelo]

There's little to gain from it at a large cost

Again, as large of a cost of losing out on developers who don't like working with your "preferred" format? A format that can be viewed as a step backwards as it does not support a fundamental concept such as comments?

I would also like to point out that you are already supporting multiple formats in your solutions: xml in the .xproj, .json for the project (and launch settings) configuration, and now .ini for hosting configuration. Hardly a consistent model, and rather confusing to someone looking at it for the first time. I think users would appreciate a consistent, configurable system that meets them where they are and allow them to develop in the format(s) that they prefer.

There is lots to gain from increased adoption. It's worth the effort. IMO, of course. Besides, it would be yet another cool, innovative feature you could hang your hat on. :)

[davidfowl]

I would also like to point out that you are already supporting multiple formats in your solutions: xml in the .xproj, .json for the project (and launch settings) configuration, and now .ini for hosting configuration. Hardly a consistent model, and rather confusing to someone looking at it for the first time.

We aren't. We support a project format, project.json, xproj is for visual studio and it barely does anything. It's actually not a project format at all. It's there to keep visual studio infrastructure happy. The configuration itself is a library, the format hardly matters as much as this is user code. We'll likely make it hosting.json eventually or something similar.

I think users would appreciate a consistent, configurable system that meets them where they are and allow them to develop in the format(s) that they prefer.

Sure, flexible project file format isn't a requirement for that. It's something new. If people want XAML project files, fork the code but we're not supporting that 😄.

There is lots to gain from increased adoption. It's worth the effort. IMO, of course. Besides, it would be yet another cool, innovative feature you could hang your hat on. :)

Maybe, but we're not investing in that.

[Mike-E-angelo]

We'll likely make it hosting.json eventually or something similar.

That's good to hear. But even then, we will still be left with a minimum of two formats in a typical solution now, Xml and JSON. I just created my first ASP.NET solution and I see there is a global.json and NuGet.config placed in the Solution Items now. ALL THE FRANKENSTEIN!!! I am not sure that this inconsistency, fragmentation, and incompatibility is a good thing for our development environments, processes, and resources. It would be like having two totally incompatible programming languages that cannot share code between each other in the same project. I mean, can you imagine someone forcing you to have to use, say, JavaScript and C# in the same project and having to duplicate your code (and efforts learning each language) all over the place in a hell of unincapsulation??

Oh wait. :)

Thinking about this some more, the core issue is not the file format, per se, but the tooling that is built around it. Consider a UWP .appxmanifest (an XML file):

Pretty. :) If the project.json loaded up like this in Visual Studio, I think it's safe to say that everyone would shut up about the format it is in. (Of course, it would be easier to design the above screen as a Xaml file. But I digress. 😛)

They haven't had that with csproj for the last 15 years or so, so I'm going to say that's not really a requirement.

I would say that is due to the fact that for the past 15 years things have been very consistent and aligned, with all configuration files being defined in Xml (and Xaml). Now that this (disruptive -- as now many resources and hours have to duplicate/recreate the efforts around tooling that have already been done around existing formats) change has been made, it might become more of a requirement as more and more traditional .NET developers take a look at it for possible adoption.

If people want XAML project files, fork the code but we're not supporting that 😄.

I hear what you are saying, and I appreciate that you are leveraging the power of OSS to appease my own (what must feel like disruptive) requests. Even though you are the only group in MSFT actively fragmenting the ecosystem with .json files, you are open source, so that does allow the possibility of the discontented developers (who value consistency) to offset the overall damage incurred.

[davidfowl]

Pretty. :) If the project.json loaded up like this in Visual Studio, I think it's safe to say that everyone would shut up about the format it is in. (Of course, it would be easier to design the above screen as a Xaml file. But I digress. 😛)

FYI the image is not loading for me but I can imagine what it looks like. We use more tools than just visual studio. This new one came up recently that's optimized for file editing VS Code 😄 . That said, I'm not against a designer for certain things where it makes sense. We also have awesome package completion intellisense and JSON schema intellisense for project.json, so showing a designer might in fact hamper the experience we're trying to build.

I would say that is due to the fact that for the past 15 years things have been very consistent and aligned, with all configuration files being defined in Xml (and Xaml). Now that this (disruptive -- as now many resources and hours have to duplicate/recreate the efforts around tooling that have already been done around existing formats) change has been made, it might become more of a requirement as more and more traditional .NET developers take a look at it for possible adoption.

Agreed. XML is old and crusty, JSON is all the rage (on the web), in another 15 years there'll be a new format that's the hawtness. Server side/web technologies tend to be on the bleeding edge more so than thick client development so when you say that "our group" (ASP.NET) is the one fragmenting .NET, it's because we're moving with the industry while doing our best to not completely alienate our existing .NET ecosystem.

Side question:

Are you a web developer?

[Mike-E-angelo]

so showing a designer might in fact hamper the experience we're trying to build.

Yes we are all so very impressed with JSON-based intellisense. They said it was impossible, and it could never be done, but you proved them wrong! :) At the end of the day, you are still editing a text file, which is just data entry. A well-designed UI that has great UX behind it yields more productivity over hand-editing a text file any day of the week. I mean, that's what Windows was built for, right? All of the neat tricks you have done with JSON intellisense could be implemented better in a clever UI.

when you say that "our group" (ASP.NET) is the one fragmenting .NET, it's because we're moving with the industry...

First off, what ever happened to leading the industry? 😄 Secondly, if you understand that you are fragmenting (or at the very least, confusing) our ecosystem, then I (for one) would feel better if "your group" took some responsibility to help mitigate the challenges and disruptions that occur from doing so (hence the original request of this issue). I think you will get better acceptance from developers (and people in general) if you offer them options to your platform, rather than saying it is one way, take-it-or-leave-it and have them deal with it from there.

Are you a web developer?

I started out as one, but then moved into Silverlight and have been suffering in the wilderness since its brutal death in 2011, haha. I believe one of the great challenges in MSFT's domain currently is the differentiation between "web" and "native" developer. I believe they are one in the same, but the provided tooling is hurting way more than helping (MSFT should officially incorporate and enable C# to transpile into JavaScript, and/or cross-platform UWP, as examples).

[davidfowl]

Yes we are all so very impressed with JSON-based intellisense. They said it was impossible, and it could never be done, but you proved them wrong! :) At the end of the day, you are still editing a text file, which is just data entry. A well-designed UI that has great UX behind it yields more productivity over hand-editing a text file any day of the week. I mean, that's what Windows was built for, right? All of the neat tricks you have done with JSON intellisense could be implemented better in a clever UI.

I disagree. To each his own. Some people prefer dialogs and designers and I respect that.

First off, what ever happened to leading the industry? 😄 Secondly, if you understand that you are fragmenting (or at the very least, confusing) our ecosystem, then I (for one) would feel better if "your group" took some responsibility to help mitigate the challenges and disruptions that occur from doing so (hence the original request of this issue). I think you will get better acceptance from developers (and people in general) if you offer them options to your platform, rather than saying it is one way, take-it-or-leave-it and have them deal with it from there.

I actually think we should go the opposite direction personally. I'd like to see simpler file formats across all project types. I'd love to see editor friendly files, with designers where it made sense but experiences built with the command line comes first. That's what we're doing here. Starting a trend. It's no mistake that VS Code supports ASP.NET 5 projects easily. We intended to build simpler file based experiences that with a more enhanced visual studio experience on top. Unlike the current ecosystem, we try very hard not to build experiences that require a designer.

[Mike-E-angelo]

I disagree. To each his own. Some people prefer dialogs and designers and I respect that.

Have you seen consumer apps lately? That's all they are. I contend that this translates to the developer, but obviously you want to leave a robust text-editing solution as a backup. So, I guess what we're seeing here is another Code-First/Data-First paradigm comparison/philosophy like what EntityFramework has, but here it's Text-First/Designer-First. :)

Unlike the current ecosystem, we try very hard not to build experiences that require a designer.

Well as you can guess, I am in total disagreement with that strategy/vision. But, you're one smart cookie, @davidfowl. I was a big fan of SignalR2.0. If this is where you think we should go, then I am not going to stop you (not that I could, anyways, LOL). As you've prescribed, the community can clean up after your mess... if there is one, of course! 😄

FWIW, I've created a vote for this issue here in case anyone would like to vote for it:
https://aspnet.uservoice.com/forums/252111-asp-net-vnext/suggestions/9243402-enable-asp-net-json-project-files-to-be-serialize

[Mike-E-angelo]

As you've prescribed, the community can clean up after your mess... if there is one, of course! 😄

I slept on this and I now regret saying that. 😛 Looking at my new solution this morning with fresh eyes, I feel a little like Moby now in this new world.

When taken in with the context of appealing to developers outside of MSFT, your efforts do make sense and should be commended. Features such as .json intellisense should indeed be appreciated and are, yes, impressive in that context. However, taken in the context of Visual Studio and the tooling that has been around for over a decade, these features seem counterproductive, somewhat unnecessary, and even a step back (again considering the tooling already in place and resources invested in learning this tooling).

Part of the problem I feel is that I have seen posts on the web and in videos espousing (bragging, even) that Visual Studio will "light up" on the experiences that are now provided by (or "just work" in) regular text editors. But I (for one) do not feel that's the case at all. Solutions were consistent and had a nice feel to them. Now, everything feels (as I have already stated) fragmented, dependent on some mysterious project.json file, and I would say even a little hijacked. Additionally, it seems we now have project management concerns separated over two files now: the .*proj file, and the project.json file. Messy.

Case in point: the References node in a Visual Studio Project file properly displays NuGet packages (thereby eliminating an unsightly assembly list found in previous VS versions, yay!), but I cannot right-click and remove them (boo!). I now have to go into a project.json (and somehow know to do this, double boo!) and remove an item from the dependencies node found in this arcane file (even if it does have intellisense 😄). Hardly intuitive or consistent.

So, I believe it's not just a Designer-First/Text-First discussion, but also a VisualStudio-First/VisualStudio-Second discussion, as well.

With all of this said, I am not crying (much) over spilled milk here, and am interested in taking you up on your OSS initiative/challenge and try to help out if I can. Would it be possible to get more context around project.json, please? Is there a magic process that is used with this now? Or is it used in conjunction with the .xproj? Is .xproj used at all or is it just there for VS decoration as alluded to earlier? I see that project.json is also used in UWP projects as well, so this makes me believe that it is something that started out in the ASPNET group but then maybe got adopted in VS from there?

Any pointers/documentation for the use of .lock.json and global.json (along with the repo/code used to load/process/utilize them) would be great, as well. I would like to try poking around to see if it's feasible to fix these eyesores that have hijacked our once pretty and consistent environment, haha. 😘

Thank you in advance!

[davidfowl]

Part of the problem I feel is that I have seen posts on the web and in videos espousing (bragging, even) that Visual Studio will "light up" on the experiences that are now provided by (or "just work" in) regular text editors. But I (for one) do not feel that's the case at all. Solutions were consistent and had a nice feel to them. Now, everything feels (as I have already stated) fragmented, dependent on some mysterious project.json file, and I would say even a little hijacked. Additionally, it seems we now have project management concerns separated over two files now: the .*proj file, and the project.json file. Messy.

I totally agree, but we're not done yet and the goal is to make VS feel like it always did for common things. There's are tons of gaps right now but they are being worked on (you identified a few but there's so much more 😄 ).

With all of this said, I am not crying (much) over spilled milk here, and am interested in taking you up on your OSS initiative/challenge and try to help out if I can. Would it be possible to get more context around project.json, please? Is there a magic process that is used with this now? Or is it used in conjunction with the .xproj? Is .xproj used at all or is it just there for VS decoration as alluded to earlier? I see that project.json is also used in UWP projects as well, so this makes me believe that it is something that started out in the ASPNET group but then maybe got adopted in VS from there?

This might be a good place to start http://docs.asp.net/en/latest/dnx/overview.html. TL;DR With DNX projects, NuGet packages are the unit of reference. A project is seen as a nuget package that hasn't been built yet, but the capabilities are identical. project.json in that sense can be seen as a nuspec file but with some additional properties that have to do with compiling the project. We've unified package authoring and project authoring into a single file. The xproj is there to for the visual studio tooling to keep track of any information it needs as well as a way to integrate with the VS build process. For example, if you build the solution file, the xproj will be used to kick off builds for dnx projects. Those builds do something different but the xproj is basically the machinery used to integrate with VS as it exists today. That's why the file is so empty 😄. Omnisharp based editors don't use any of this and focus on the project.json since that's where most of the relevant data is. dnu restore is used to download transitive dependencies and it also creates the project.lock.json file with the full list of referenced packages.

In the UWP/PCL the project.json is used for NuGet dependencies http://docs.nuget.org/consume/ProjectJson-Intro as it relates to csproj (which is different from xproj).

Any pointers/documentation for the use of .lock.json and global.json (along with the repo/code used to load/process/utilize them) would be great, as well. I would like to try poking around to see if it's feasible to fix these eyesores that have hijacked our once pretty and consistent environment, haha. 😘

The global.json file has alot less documentation. But I'll try to dig something up.

[Mike-E-angelo]

There's are tons of gaps right now but they are being worked on (you identified a few but there's so much more 😄 ).

OK, thanks for letting me know and for your explanations.

Another thing: I see that project.lock.json is viewed as a "dependent upon" node in the project tree, but it is not defined anywhere in the project (as you say it is quite empty and there to make VS happy). So, what magic is taking place to render those nodes as such? :)

I do like where you are going with a project build file. As I mentioned, this is a discussion that is happening @ MSBuild vNext. The VS build system is actually quite terrible. I absolutely hate the XML that is used to build our projects (not to be confused with hating XML -- in fact they should be using Xaml and serialized POCOs for their build files which would greatly improve the current system). That and the .SLN files (y-u-c-k) are due for a massive overhaul.

I am also still at a loss at why the Visual Studio team allowed this to happen. Why not make this a change across the board? We now have inconsistent projects in solutions now. What is used for simply package management in one project is used for compilation management in another. Insanity.

That said, I am actually very much open to new build files (granted they can be defined in any format, getting back to to the original topic here). This would be easier to accept if they acknowledged that, yes, we're updating our project system, and yes, it's going to be messy for a few years. But there hasn't been any of that and it's just mass confusion right now.

Anyways, I will also look into these links. It would be great to see if you can get anything around global.json as well. Thanks again.

[Mike-E-angelo]

FWIW, after the great discussions here and elsewhere, I have created a vote for this issue in Visual Studio that helps better describe and articulate the issue. The ASP.NET group is not to blame here, as they are really trying to do what should have been done long ago: reboot/refine Visual Studio's project structure. For that, you should be commended -- even if it does leave us in a mess for the time being. :)

It took a lot of discussion (on my part) to land where I am now, but I feel a lot better about where I feel about this issue. Thank you @davidfowl for being a great communicator (and organizer) in this regard. Much appreciated to have your time and thoughts around this.

Here is the Visual Studio vote:
http://visualstudio.uservoice.com/forums/121579-visual-studio/suggestions/9347001-improve-reboot-visual-studio-project-system

Additionally, I feel this issue unveils a deeper, bigger issue within application development that I have attempted to tackle in this blog post (which also has a corresponding vote for it):
http://blog.developers.win/2015/10/data-asts-and-data-views/

I feel that with these votes out, that this issue is no longer valid. However, I will make that the decision of the team here whether to close it or not. Thanks again for your patience in dealing with me!

[sayedihashimi]

Hi @Michael-DST thanks for your feedback. I don't think that we have all the right folks on this repo for this item. I think this would be better to discuss at https://github.com/aspnet/home or https://github.com/aspnet/dnx. I'll close this now.