Add .Net Core Support

[gaoxiaojun]

Microsoft has been released .Net Core 1.0.0 and SDK Preview2,

[jaredbroad]

Thanks @gaoxiaojun! Do you mind giving an example use case to use it in LEAN?

[bnmanager]

@jaredbroad @gaoxiaojun

Universal Windows Apps (UWP), (of which I am currently developing an app for), use .NET Core as their foundation. Therefore, many useful libraries found in console or WPF programs are simply incompatible with .NET Core. Currently, QuantConnect is one of those.

Also,.NET Core is open source and can run on Windows/Linux/Apple.

[jaredbroad]

Thank you @gaoxiaojun @bnmanager. I'll leave this open but from my reading it seems like a mammoth & risky task to convert the platform across. We can start a branch if there is someone to lead the effort.

For now LEAN is cross platform Windows/Linux/Apple via Mono. The only limitation to date is the winforms UX project which is windows only.

[omencat]

Instead of targeting .net core, the easiest way today to go about this would be to target the .net standard 2.0 surface. Using the .net portability analyzer extension, it looks like QuantConnect.Algorithm.CSharp is 98.47% complete. QuantConnect.Indicators is already 100% complete.

[lnaie]

yes, just wait for dns 2 release and you should be 99% good to go.

[wklm]

Any news on that?

[jaredbroad]

If you click subscribed you'll be notified when there are updates to the ticket

…

On Jan 27, 2018 12:17, "Wojtek" ***@***.***> wrote: Any news on that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#452 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACI6meA3_Vq0CrDOK8wlVVnzOwV7lzFlks5tOxPKgaJpZM4JECId> .

[wklm]

I was asking, because if there's no one currently working on it I could give it a try

[snielsson]

Hi, what is the current status of this ?

[mamatopian]

Hey guys, any news?

[omidkrad]

Hi, is migrating to .NET Core in the plans at all?

[omidkrad]

Or maybe a more appropriate compilation target is .NET Standard 2.0+.

[Leonardo-Ferreira]

I think this can be re-prioritized. @jaredbroad mentioned in several places that Mono 5.12+ has displayed abnormal memory leaks, so why don't we discard mono altogether and go for .Net Core?
This could happen in phases, porting first the C#/F# algorithms support and latter the python support.
Microsoft has already announced that .Net 5 will be based on .Net Core and not the old framework.
I think I can greatly support this effort, I've been through this scenario a few times and was very successful.

[cmoski]

Totally on board for this transition, all (of my) development is in dotnet core. Currently have a port of LEAN to DOTNET 4.7.2, but that is simply to allow compatibility with the Dotnet Standard 2.0+ surface without totally breaking Python. Happy to contribute dev time to this initiative.

[Leonardo-Ferreira]

Porting to core can't be done by simply changing the target of compilation. There are several restrictions and caveats there, specially regarding Threads (Thread.Kill doesn't exist for instance), which is something that we see here quite often (not thread.kill but the direct reference to threads)

If I were in charge here, the first step would be creating a new repo, forked from this one, something like QuantConnect/Lean2 or QuantConnect/LeanOnCore. This way the development can happen without disturbance to the main project, but we could feed on any bug correction and etc...

Second, I would create duplicate the project tree, in a way that code could be moved from the regular framework to Core on a easy manner (drag-n-drop). This would leave the project uncompilable for quite some time, but it's a lesson learned... once the code is moved, then we would work on the quirks to get it compiling. Once this is over, delete the original tree

Third phase would be the unit tests... all of them...

I must already advice that this is not an easy undertake and there will be a lot of merge conflicts and lots of headache, but something definitively worth doing.

[Leonardo-Ferreira]

It's also important to note that we should try to reach as many people as we can, thus targeting .Net Standards such as 1.6 as much as possible.

[Leonardo-Ferreira]

You can follow my work here: https://github.com/Leonardo-Ferreira/Lean/tree/feature-452-dotNetCore3
ps: I created a channel for net core implementation discussion on our Slack

Update 1 (2020-03-20): Happy to report that a compiling and running project is up. There might be bugs and everything but it's up. It's nice to have something I can hit F5 and have running on .net core

[snielsson]

Great effort thanks. I hope you get some support. I might be able to help later on.

[happysammy]

You can follow my work here: https://github.com/Leonardo-Ferreira/Lean/tree/feature-452-dotNetCore3
ps: I created a channel for net core implementation discussion on our Slack

Update 1 (2020-03-20): Happy to report that a compiling and running project is up. There might be bugs and everything but it's up. It's nice to have something I can hit F5 and have running on .net core

how to join the slack channel?

[Leonardo-Ferreira]

@yuandong618 Check this out: https://www.quantconnect.com/slack

[Leonardo-Ferreira]

Happy to report that the tests are running!
All python tests are broken because it's not the scope yet to run python lags (but I hope to get this up again soon) and Interactive brokers is also out, but things are moving along!

[happysammy]

Happy to report that the tests are running!
All python tests are broken because it's not the scope yet to run python lags (but I hope to get this up again soon) and Interactive brokers is also out, but things are moving along!

Hey guys, what's the status of tests?

[Leonardo-Ferreira]

couldn't work on this any further @happysammy. I noticed they upgraded the framework a bit to comply to .net standard 2.0 I think though

[eramosr16]

Four years and counting...

[kgoderis]

@Leonardo-Ferreira @jaredbroad As I discover more and more on QuantConnect, I just stumbled on https://docs.microsoft.com/en-us/dotnet/core/dotnet-five ... Time to take a quantum leap ? ;-)

[Leonardo-Ferreira]

I backed out a bit from this project... I did went quite far transitioning it to .net core 3.1 but didn't manage to finish it cause they just wouldn't accept the PR... me and @jaredbroad had a long talk @ slack, where I explained what I believed we should do, and how I was proposing to do it. Long story short, I was proposing a incremental approach, where not all features/brokers would be available at day 1, but would be added over time, and this was deemed unacceptable given the relatively small gains...

As I mentioned before, migrating to .Net Core or to .Net 5 is the basically the same effort...

[eramosr16]

@xposure I just wanna say thank you for you work here. Keep it on, this is a great library.

[xposure]

@itsbalamurali PR created, lets hope it gets some traction

[xposure]

@Martin-Molinero I hope you take the time to look at my PR that was rejected as all this work was already done in mine.

[jaredbroad]

.NET Core Update

Hi All, it has been a year or so working on this so I think it is time for an official update. TLDR: A shotgun approach will not work with a delicate financial engine. Contributors welcome but accept it will be a daily grind, not a home run.

We have been carefully upgrading and testing each library change and putting it into production brick by brick - for about 3-4 months with 2-3 FTE engineers. This is the true grind of professionals; it is easy to mass delete things not understood, and ignore and not test each component. But as LEAN hosts more and more assets it's critical the mentality is one of perfection, precision, and complete awareness of changes and the impact they may have on the users of the platform.

Even in the last few months since we started this work, it has touched more than 2M algorithms - without a blink from the community. Our core team has been upgrading each library, maintaining backward compatibility, running full regression tests, and deploying it all the way through to production. Each upgrade then gets a few days of live deployment through the QC platform -- exposing the new code to tens of thousands of users and live testing. Any issues are quickly rolled back and fixed before moving to the next step.

LEAN needs professionals to continue trail-blazing an institutional-caliber quantitative engine and we'd welcome contributors. The contributors will be given a single library to upgrade, test, and submit back for merge. LEAN has the trust of 140,000 engineers and investors because we do things the right way, the hard way. Perhaps one thing missing is a way for the community to run the regression algorithm suite so we'll try and make this accessible to help improve the iteration cycle.

We estimate the .NET core project will be another month or so before it's merged. By the time is merged it would have been tested on millions of algorithms, and live traded across thousands of accounts. In the meantime thank you for your patience and support as we continue pushing this one over the finish line.

[Midaroh]

@jaredbroad When you make these framework changes, is there some automated regression step in your build process to ensure that the output of all user algorithms are not changed by refactoring?

[jaredbroad]

It is not possible to check them all or even a significant sample (would take years of computing to check every algorithm).

However, it would be ideal to run the LEAN regression algorithms for open-source PR's via a GH actions event; we can't ship all the data required to run them with the open-source project. We can put the data somewhere semi-private to install with a GH action regression test setup (and GH secret) -- or we can generate random data which covers the same test cases and host that to run the tests.

Edit: @Midaroh You can actually run almost all the basic regressions with the data included in LEAN =). Once those pass the other basic test you could do locally is running a live strategy in python. Then it should be good enough for us to deploy to a branch and test it in the cloud.

[biiiipy]

Thank you for the update! I do hope that .net5 support comes soon, but it's okey to take time, as long as the plans are publicly shared so we know what to expect :)