Comparisons with other libraries

[netcorefan1]

I couldn't find a way for a better title. As you saw I am trying all the possible alternatives. The most recent one is gRPC and I think it's time for me to go into a conclusion because I have lost enough time.
This is the time I experienced with PolyMessage to exchange data.

polymessage.mp4

And this with grpc-dotnet using grpc-dotnet-namedpipes

gRPC_namedPipes.mp4

I am not sure if such big difference is due to TCP vs Pipes. I also tried gRPC (IPC over Unix domain socket), a bit slower, but still fast. Regular Http/2 was still faster than PM.
vs-streamjsonrpc over named pipes was very slow compared to all (as you saw in the video) and I reported this to their repo.
Sincerely I start to think that gRPC is the best alternative, it's under active development (it's google) and there are so many extensions in Nuget that I was not able to reach the end of the list. It's fully integrated with VS and recommended from Microsoft (although I don't understand why they also recommends streamjsonrpc while to me seems two similar technonologies.
gRPC also allow an easy connection with lot of online services which all uses this techonogy.
NanoMsg you recommended me was also good and fast like gRPC (may be a bit less than gRPC), but extremely difficult to use. The learning curve is very long and honestly I don't know what kind of advantaged can offer compared to the others. It's like playing with pinvoke with the only documentation available is the one that refers to the native version and this add additional level of complexity.
I entered into this world from a few days and I have learned very little, but what I learned for sure is that the wrong decision will cause a big amount time to lose during development.
Let me know what you think.
At this point I am asking myself if it's really worth that you try to reinvent the wheel considering what there is out there?
What advantages you think to gain compared to other third parties?

Let me write you some similar projects I bookmarked time ago, but never tried after discovering those giants projects.
https://github.com/spazzarama/SharedMemory
https://github.com/furesoft/Furesoft.Rpc.MMF
https://github.com/seaeagle1/EasyPipes
https://github.com/cloudtoid/interprocess
https://github.com/hhblaze/SharmIPC
https://github.com/HavenDV/H.Pipes
https://github.com/RandomEngy/PipeMethodCalls
And there are others...
Some of them seems to have an architecture similar to yours.

Now I think that my last try is MagicUnion, a framework based on gRPC made from the creator of MessagePack. He also created ZString a Zero Allocation StringBuilder for .NET Core, ZLogger, another zero allocation logger built on top of a Microsoft.Extensions.Logging. I suppose all of this technogies are integrated in the Framework. I am curious to see how it performs.

Let me know what you think. I am at the beginning and the maximum I could do is make the samples working and see for which I feel better. I don't have the technical skills to go deeper and dissect every solution for pros and cons.
You are a guru in this kind of technology and I am sure that there is much more that I should know.
I will put here an update as soon as I finish to experiment with MagicOnion. After that It will be time for me to choose for a technology because I really can't afford more time in researches. Let me know your goals on PolyMessage and how you would compare with the above solutions, but first of all let know why is so slow compared to all the others.
Thanks
Thanks

[cvetomir-todorov]

Thanks for the extensive list of libs which are comparable to PolyMessage.

Not sure what to say about your conclusions though. You are free to choose whichever library you see fit and gRPC and libs under active development are always a better choice.

Problem is I don't see a proper benchmark and your conclusion seems far-fetched based on a subjective observation using the loopback interface even. Look at dev\load-testing for a more real benchmarking code. Deploy the server on a machine different from the one where the clients are and test it that way. You can adapt it to different libs, appropriate message size, expected clients count, desired transport, preferred format etc. Based on numbers you can choose what's best for you. Good luck!

[netcorefan1]

MagicOnion.mp4

MagicOnion is also pretty fast, but unfortunately does not support named pipes. Yes, you're right, I have no experience in this side and the maximum I can do is observe the time to establish a connection, call a method and return. I should also see how it performs on heavy load, but since I now need quick short terms connections I didn't thought about it. Simplicity is also another factor I have to take into account. Unfortunately I don't have another machine where I can test, but thanks.

I wanted to try again PM and compiled the IPC variant to see how it performs. Unfortunately seems to be that PM don't like me as its user.
I spent hours in trying to find an uri accepted from IPC constructor and then I ended up with the client stuck forever in "await client.ConnectAsync();" (in debug mode). Without debugger attached I get this:

polymessageV2.mp4

Client and server:
PolyTransport transport = new IpcTransport(new Uri("net.pipe://localhost/ProductService"), loggerFactory);

Wouldn't be better an overload that accepts just a name like does the original namedPipeServerStream and other libs? I know, an user should know how to compose an appropriate url, but someone who enter IPC stuff for the first time can really have very hard time.
For the rest I like your library, is very intuitive. I just don't understand the difference with other libraries and this is my fault because not being experienced I know perfectly well that I could be very wrong in all my assumptions and conclusions I may end up with comparisons.

[cvetomir-todorov]

I don't think I've been correctly understood. Your testing methodology is a bit naive. Investigate the load testing code I gave you a reference to and try to understand it.

As an example I could let you know that gRPC uses automatically generated code, while Protobuf-net format used by PolyMessage relies on code generation at runtime. Additionally PolyMessage generates code at runtime as well for each endpoint. Establishing a connection is also something that may vary and has no too big of an effect on the throughput. Essentially warming up is required.

Investigate the code I gave you for load testing. Your decision for a communication library is as good as your understanding how benchmarking happens.

[cvetomir-todorov]

Also, and I can't stress it enough, as I already said - the code in the IPC transport branch is not yet finished. So no point toying with it and making conclusions.

[netcorefan1]

I don't think I've been correctly understood. Your testing methodology is a bit naive. Investigate the load testing code I gave you a reference to and try to understand it.

As an example I could let you know that gRPC uses automatically generated code, while Protobuf-net format used by PolyMessage relies on code generation at runtime. Additionally PolyMessage generates code at runtime as well for each endpoint. Establishing a connection is also something that may vary and has no too big of an effect on the throughput. Essentially warming up is required.

Investigate the code I gave you for load testing. Your decision for a communication library is as good as your understanding how benchmarking happens.

Many thanks! I think that I start to understand something on how it works the whole thing. Essentially, the delay I see on startup is due to dynamic generation of code at runtime (Anonymously Hosted DynamicMethods Assembly or similar). Warming up helps a bit, but it's still much slower than non dynamic code. I am talking about grpc-dotnet-namedpipes vs other libs that uses dynamic generation. I didn't had chance to compare with PolyMessage IPC. In all the cases, as you says, there should not be a big effect on the throughput, that's why I didn't take care about benchmarks, but I only focused on startup time. And for this, eyes is more than enough, very naive for sure, but it works (eyes and sixth sense sometimes are more reliable than anything else).

I have discovered another library which may be of interest for you. It's called SciTech.Rpc. Just like your project, the best projects are the ones that one discovers by accidence once researches ends up. And just like your project, even this one was completely hidden (I have been the first one to open an issue). They were actively developing in private repo and only recently I saw a constant activity. May be they are going to release.
I see your project has missing features which available in their project (IPC to mention one) while they have missing features that your project has (all the custom serialization stuff).
Would you to take in consideration the possibility to cooperate in one shared big project? I played a little with their code and from what I see, both the projects has a very similar common base. Merging the two projects together would be great given the features provided by each project. It will complement each other.
The delay due to dynamic generation is something that they are concerned of (see performance paragraph) and they have worked around with a tool (not sure if it's ready to be used or not). It may be of help for you.

I know that developers prefers to have their own project with full control, but GH is overflowing of similar RPC projects (the list I have given to you was only a very minimal list where I have been forced to exclude other great projects). Final users like me are scared to refactor code in order to implement an open source project and then, after a while, find the main repository archived as read only.
You previously asked me how to attract community. Well, I have already given some suggestion in my other thread and I have given here other ones. Today when people search for some kind of project (included me), the first thing they do after pressing the search button, is filtering by the most recent updated repos. Then, they look at the activity (issues, unanswered issues if any, pull requests, number of collaborators if any etc etc). Paradoxically, checking the description of a project to see if matches requirements, is the last step.
You should take all of this into account. In this particular case, under the hood there is google, so joining something like that will bring popularity easier.

Let me know what are your plans and final goals with your project and whatever they will be let remind you to consider me available for any help.

[netcorefan1]

Oh! I believed it was named pipes, but from the log it is showing memory mapped files. Is this shared memory which is supposed to be even faster than named pipes. This is really a great thing!
What you think about making your PM as an extension of gRPC dotnet instead of a standalone? Or some other kind of integration? Wouldn't this expand possibilities? It will attract for sure the community something like that. I am not sure if you had chance to read my previous thread, I recommend you to look at SpanJson and BinaryPack.

[cvetomir-todorov]

Yes, as the documentation says - named pipes is used only for the control data. The payload data is transferred via shared memory using memory mapped files. Thanks for the suggestion.

[netcorefan1]

Also, and I can't stress it enough, as I already said - the code in the IPC transport branch is not yet finished. So no point toying with it and making conclusions.

Yes, as the documentation says - named pipes is used only for the control data. The payload data is transferred via shared memory using memory mapped files. Thanks for the suggestion.

I have read that named pipes alone does not make big difference. What really makes difference is Shared Memory, but there are lot of dangers and things to take care about. It's good that your lib frees the final user from from all of this.
IPC transport is my only need for now (while I still want to have all the rest at hands for future needs).
Well, when something does not work I always assume that it is my fault. I am glad to see that this time couldn't. Hope to see soon the IPC transport ready.

[netcorefan1]

I am afraid that you are busy and I have abused enough your time. As of now, I like PM, but it's not of much use for me due to lack of IPC which is the main feature I need. I'll put your repo in my favourite and I will check every while to see if there are good news.
Thanks for now