Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Linux #15

Open
lewissbaker opened this issue May 27, 2017 · 37 comments
Open

Add support for Linux #15

lewissbaker opened this issue May 27, 2017 · 37 comments

Comments

@lewissbaker
Copy link
Owner

lewissbaker commented May 27, 2017

While most of cppcoro is platform agnostic, some of the thread-pooling and I/O code is OS-specific.

This issue is for providing an implemention of the cppcoro I/O abstractions for Linux.

Will need to get it building under Clang as a first step. See #3.
Then:

  • Look into writing an epoll I/O event loop, based io_service implementation.
  • Port file and socket (once written) interfaces to Linux.
  • Add build-system support for Linux
@lewissbaker
Copy link
Owner Author

cppcoro is now at least buildable under Linux/Clang on the clang branch.

Still outstanding is the need to work on an implementation of io_service using epoll and then subsequent file/socket implementations.

@lewissbaker
Copy link
Owner Author

For async file I/O for io_service I'm thinking of using kernel aio (ie. io_submit, io_getevents and friends).
Although from my research this seems to be somewhat limited in that it's only truly async for files opened with O_DIRECT.

For async socket I/O it seems like epoll is the go.

Need to investigate how to mix the two approaches (aio/epoll) in the same event loop.

@neo5167
Copy link

neo5167 commented Nov 3, 2017

So, did you guys start working on this?

@lewissbaker
Copy link
Owner Author

I haven't started on Linux async I/O support yet.

@neo5167
Copy link

neo5167 commented Nov 7, 2017

I will work on it. But before I start can you help me compile the current code on linux? The build instructions aren't working for me.

@lewissbaker
Copy link
Owner Author

Sure. Can you create a new issue to work through the build issue?

@neo5167
Copy link

neo5167 commented Dec 20, 2017

scheduler_operator_tests.cpp and io_service_tests.cpp are now working in linux. I will get file tests working as well and send a PR. I am using linux message queues as a sub for iocp, timerfd/eventfd for WaitableTimer/AutoresetEvent, and epoll for WaitForMultipleObjects. For files, I plan to send messages to the message queue from AIO's signal handling functions.

@lewissbaker
Copy link
Owner Author

Great! I look forward to the PR :)

For AIO, can you use an eventfd as the aio_resfd parameter of the iocb block for an I/O request as a way to have it wake up the epoll() loop when the operation completes. When epoll() indicates that the AIO eventfd is signalled then you know you need to call io_getevents() to retrieve pending I/O completion events. This should avoid the need for signal-handling functions.

Or are you talking about something different?

@neo5167
Copy link

neo5167 commented Dec 21, 2017

That's right! I will try the resfd way. I remember a while ago that it wasn't stable. Its been long. They should've worked it out by now...

@lewissbaker
Copy link
Owner Author

@neo5167 How are you going with the epoll/aio io_service for Linux? Do you need any help?

@neo5167
Copy link

neo5167 commented Jan 12, 2018

Was on vacation. How about I send a PR for everything without files and then send the files one once its done. WIN32 APIs are more prevalent in the files code than in timer/scheduler code, so taking longer than I expected...

@tavi-cacina
Copy link

Have you seen AFIO already? https://ned14.github.io/afio/
It states it supports coroutine TS (async_file_handle) and @ned14 seems very convinced he is on the right path (I watched few presentations from him).

@ned14
Copy link

ned14 commented Jan 12, 2018

I intend to propose AFIO as the File I/O TS in the June WG21 meeting. AFIO is the complete redesign after a Boost peer review in 2013, and therefore ought to carry some weight with regards to design with WG21 (but we shall see!). As you'll notice in AFIO, we don't prioritise async file i/o in the design, mainly because it isn't faster for most use cases, is definitely not faster on SSDs and is positively retrograde on SCM. I suspect you'll thus want to stick with your implementation. But keep at your library, looks valuable and I'd encourage you to submit it to Boost soon. We sorely need abstractions built on top of Coroutines both for Boost, and thence the C++ library.

@lewissbaker
Copy link
Owner Author

@neo5167 A PR with the basic io_service event loop and support for scheduling coroutines sounds like a great first step! I'd be happy to review it at that stage.
I've also been working on adding some socket APIs for Win32 in the sockets/sockets2 branch - this does some refactoring of the file_read/write_operation classes to eliminate some of the boiler-plate and also to add separate cancellable and non-cancellable variants of the operations (non-cancellable version generates much tighter assembly). Maybe check that out and see if a similar approach can be taken for the Linux implementations.

@tavi-cacina Thanks for pointing AFIO out. I had heard of it but haven't had a chance to look into it in any depth yet. I'll check it out.

@ned14 Thanks for the encouragement :) There's still a lot I want to add to it and a few things I want to revisit before I feel it will be baked enough for submitting to Boost.

@neo5167
Copy link

neo5167 commented Jan 17, 2018

Great. Some info on timeline:
I work for Microsoft Research and we want to use cppcoro for some internal R&D on linux. I will share the PR as soon as our open source team approves the project. Should be less than a week. Thanks!

@neo5167
Copy link

neo5167 commented Jan 24, 2018

Good to go on my end. Can I have write perms to create a branch to push my changes to? I can generate a PR after that.

@neo5167
Copy link

neo5167 commented Jan 25, 2018

@lewissbaker ping. :)

@ericniebler
Copy link
Collaborator

@neo5167 The typical GitHub work flow is to create a fork of the repo, make your branch there, and make a pull request from there back to the original repo.

@neo5167
Copy link

neo5167 commented Jan 25, 2018

Got it.

@lewissbaker
Copy link
Owner Author

@neo5167 My apologies for the delay in getting back to you. I've been travelling this week.

As @ericniebler suggested, it's probably best if you create your own fork of cppcoro and push your changes to a branch of that fork and then create a PR to merge those changes back into this repository.

I'm looking forward to seeing the PR, I think a lot of people are hanging out for it!

@neo5167
Copy link

neo5167 commented Jan 27, 2018

@lewissbaker I created the PR.

@lewissbaker
Copy link
Owner Author

@neo5167 Great, I'll check it out.

@neo5167
Copy link

neo5167 commented Feb 7, 2018

@lewissbaker Got a chance to try it out? I have a fixed few things. Will send another commit into the PR soon.

@lewissbaker
Copy link
Owner Author

@neo5167 I have started reviewing the code and am about 50% through.
I have had quite a number of other distractions the last couple of weeks, though, so it's taking a little longer than I'd hoped. I hope to finish the initial pass of the review soon.

@lni
Copy link

lni commented Mar 29, 2018

@lewissbaker @neo5167 thanks for the library and the port for Linux!

I grabbed a copy it and used it in the prototype application I am playing with. One issue I noticed is that the POSIX message queue based message_queue is pretty slow, it became a bottleneck when I need to do lots of io_service.schedule()/resume_on(). I plugged in Cameron Desrochers's lock free queue with built-in block queue wrapper, it solved my problem.

I guess this might be useful. Thanks.

@neo5167
Copy link

neo5167 commented Apr 2, 2018

@lni Thanks for the input. I will take a look at that and see how we can use that.

@neo5167
Copy link

neo5167 commented Apr 2, 2018

@lewissbaker Is it possible to get a private email address to chat with you? I wanted to go over somethings but prefer doing it in a private setting. Thanks!

@lewissbaker
Copy link
Owner Author

@neo5167 Sure, no worries. My email is my github username at gmail dot com.

@neo5167
Copy link

neo5167 commented Jun 26, 2018

The linux scheduler and timer support code (beta) is at the following fork if anyone wants to try:
https://github.com/neo5167/cppcoro

@JasonDictos
Copy link

Hows linux support as of today?

@neo5167
Copy link

neo5167 commented Dec 21, 2018

I did implement the linux support for an older version that is here:
https://github.com/neo5167/cppcoro

@JasonDictos
Copy link

There any important upstream changes in cppcoro branches that I'd be missing out on? It work? ;)

@JasonDictos
Copy link

oh btw one thing I noticed with your repo @neo5167 is the definition of the linux inner namespace conflicts with some other projects pre-processor definition of linux=1.

@JasonDictos
Copy link

Alright cool @neo5167 got it building with vcpkg, will report back if I find anything, here's the port file info if anyone is needing it: https://bitbucket.org/aparavi/vcpkg/src/aparavi/ports/cppcoro/

@dawidpilarski
Copy link

Hi there! Am I right to see, that the work on the Linux support is suspended right now (last comment on December 21st)? Any chance I can offer my help?

@Garcia6l20
Copy link

Hi there !
I just implemented the io_service, file and net APIs using liburing.
There are some requirements: GCC10 (10.2 fixes some compilation errors in lambda captures within a coroutine) and kernel >= 5.4 (5.5 for accept, connect, send, recv operations).
This is still a work-in-progress stuff but it is almost functional.

Check my fork.

Tell me what you think about it.

@Garcia6l20
Copy link

All tests passing !

make -j && ctest
[...]
100% tests passed, 0 tests failed out of 160

And my cppcoto-http extension works pretty well :)

tavi-cacina pushed a commit to tavi-cacina/cppcoro that referenced this issue Dec 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants