Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace Typhoeus with async-http #669

Closed
wants to merge 1 commit into from

Conversation

jasl
Copy link
Contributor

@jasl jasl commented Jan 11, 2021

typhoeus is the only blocking problem that preventing CocoaPods running on M1 natively

I have read many GH issues offfi and ethon, it looks like the segfault problem is hard to fix, so I decide to rewrite it.

I'm using async-http to replace typhoeus + concurrent-ruby

Advantage:

  • can running on native M1 Ruby
  • the abstraction is simpler than typhoeus + concurrent-ruby
  • the author ioquatix is also the author of Ruby 3 Fiber scheduler

Disadvange:

  • Minimal Ruby version require 2.5+

Once blocking PRs getting merged, I'll cleanup commits and ready for reviewing

@jasl
Copy link
Contributor Author

jasl commented Jan 11, 2021

image

It works on my M1 mac, but lacking

  • Complete codes
  • Update tests

@amorde
Copy link
Member

amorde commented Jan 12, 2021

Thanks for the PR! this has been top of mind recently with the M1 Mac releases.

There's an existing PR that attempts to achieve the same result here #639 - I'm not not familiar enough with either of these APIs to pick one, but I'd love to move at least one of these PRs forward. @igor-makarov what do you think?

@igor-makarov
Copy link
Contributor

First of all, thanks for all the work you've put in, @jasl!
I've got several comments on your proposed change:

  • Bumping Ruby/Rubocop version should probably be done as a separate PR. It seems like a big deal.
  • async relies on another gem with native extensions, nio4r. Does it work on M1?
  • GitHub Actions doesn't have Apple Silicon machines yet. Official support for native M1 should probably wait until we can actually run tests on it.
  • typhoeus is also used in the CocoaPods repo. I think the cdn_url? function can be moved to Core, but this will involve integration work in any case.
  • I've tried running basic operations install, repo update and I've noticed that it fails on responses that have more than one chunk. I think that async-http returns the response without the body being complete, see here for an example of collecting chunks.
  • Considering the previous item, may I suggest using the already-built concurrency and synchronization and just change the implementation of download_typhoeus_impl_async to use async-http and resolve Futures with it?
  • There's going to be a lot of work getting the tests necessary for this transition.

@jasl
Copy link
Contributor Author

jasl commented Jan 12, 2021

First of all, thanks for all the work you've put in, @jasl!
I've got several comments on your proposed change:

  • Bumping Ruby/Rubocop version should probably be done as a separate PR. It seems like a big deal.
  • async relies on another gem with native extensions, nio4r. Does it work on M1?
  • GitHub Actions doesn't have Apple Silicon machines yet. Official support for native M1 should probably wait until we can actually run tests on it.
  • typhoeus is also used in the CocoaPods repo. I think the cdn_url? function can be moved to Core, but this will involve integration work in any case.
  • I've tried running basic operations install, repo update and I've noticed that it fails on responses that have more than one chunk. I think that async-http returns the response without the body being complete, see here for an example of collecting chunks.
  • Considering the previous item, may I suggest using the already-built concurrency and synchronization and just change the implementation of download_typhoeus_impl_async to use async-http and resolve Futures with it?
  • There's going to be a lot of work getting the tests necessary for this transition.

Bumping Ruby/Rubocop version should probably be done as a separate PR. It seems like a big deal.

Yes, async require at least Ruby 2.5.0, depends on CocoaPods drop support of 10.14 and below,
that's should be decide by the Core team.

but for now, I add these changes in the branch for not blocking my work.

async relies on another gem with native extensions, nio4r. Does it work on M1?

It works, my branch can natively running on M1 (see the screenshot above).
Besides nio4r has a pure Ruby implementation (but not enable in M1)

GitHub Actions doesn't have Apple Silicon machines yet. Official support for native M1 should probably wait until we can actually run tests on it.

For now, I can only run tests on my M1 Mac, Homebrew also suffer this,
but I think this is fine, if CI green, most probably no issue on M1...

typhoeus is also used in the CocoaPods repo. I think the cdn_url? function can be moved to Core, but this will involve integration work in any case.

I've send another PR CocoaPods/CocoaPods#10346 (this PR is ready)

I've tried running basic operations install, repo update and I've noticed that it fails on responses that have more than one chunk. I think that async-http returns the response without the body being complete, see here for an example of collecting chunks.

I'll take a look, the PR not ready yet, hopefully I can complete this in this or next week.

Considering the previous item, may I suggest using the already-built concurrency and synchronization and just change the implementation of download_typhoeus_impl_async to use async-http and resolve Futures with it?

I can try, but I don't prefer this idea,

because:

  • aysnc-http is combine of async + http which is competitive with concurrent-ruby + typhoeus
    • if we have to keeping concurrent-ruby, your branch is better than me in my view
  • Ruby 3 has built-in non-blocking solution called Fiber scheduler, async already use that (they are the same author)
    • I believe nio4r will be removed for Ruby 3.0 implementation
    • I heard macOS will not bundle Ruby in future, when the times come, Cocoapods can migrate to Ruby 3.0

There's going to be a lot of work getting the tests necessary for this transition.

Agree, made Cocoapods running on M1 natively is a long-term work, but we can learn from Homebrew:

  • Make the codebase compatible with M1 in theroy
  • Ensure not break x86 platform
  • Let M1 users test in their own risk

@jasl
Copy link
Contributor Author

jasl commented Jan 12, 2021

Considering the previous item, may I suggest using the already-built concurrency and synchronization and just change the implementation of download_typhoeus_impl_async to use async-http and resolve Futures with it?

@amorde could you consider this suggest? I left some opinion in #669 (comment)

but if it worthy keeping concurrent-ruby, I prefer #639

@igor-makarov
Copy link
Contributor

@jasl We aren't tied to Concurrent-Ruby - it's an implementation detail of CDNSource.

The function it serves in the current implementation is structuring the concurrent HTTP/2 requests so that as many as possible can be performed in parallel.

For what I gather, async provides a different way to structure this concurrency - so my guess is that the whole implementation needs to be rewritten.

I'm not very familiar with this type of concurrency so I don't know how to structure it correctly. However, the passing of context seems like it could be factored out somehow.
In the original implementation, all requests run on the libcurl reactor - even when there is only one.

@igor-makarov
Copy link
Contributor

@jasl I've managed to fix the partial HTTP body issue without hurting concurrency. I've pushed the change to your branch.

I'm repeating the list from above to focus the discussion:

  • Bumping Ruby/Rubocop version should probably be done as a separate PR. It seems like a big deal.
  • async relies on another gem with native extensions, nio4r. Does it work on M1?
  • GitHub Actions doesn't have Apple Silicon machines yet. Official support for native M1 should probably wait until we can actually run tests on it.
  • typhoeus is also used in the CocoaPods repo. I think the cdn_url? function should be moved to Core.
  • There's going to be a lot of work getting the tests necessary for this transition.

@jasl
Copy link
Contributor Author

jasl commented Jan 12, 2021

@jasl I've managed to fix the partial HTTP body issue without hurting concurrency. I've pushed the change to your branch.

I'm repeating the list from above to focus the discussion:

  • Bumping Ruby/Rubocop version should probably be done as a separate PR. It seems like a big deal.
  • async relies on another gem with native extensions, nio4r. Does it work on M1?
  • GitHub Actions doesn't have Apple Silicon machines yet. Official support for native M1 should probably wait until we can actually run tests on it.
  • typhoeus is also used in the CocoaPods repo. I think the cdn_url? function should be moved to Core.
  • There's going to be a lot of work getting the tests necessary for this transition.

Thank you! I'll check tomorrow

Bumping Ruby/Rubocop version should probably be done as a separate PR. It seems like a big deal.

Will separate another PR for this before this one ready to review

async relies on another gem with native extensions, nio4r. Does it work on M1?

It works, I've tested in my M1 Mac Mini, I'll do double check

GitHub Actions doesn't have Apple Silicon machines yet. Official support for native M1 should probably wait until we can actually run tests on it.

We should ensure all exist targets pass, I'll also run test on my M1 Mac

typhoeus is also used in the CocoaPods repo. I think the cdn_url? function should be moved to Core.

A PR replace typhoeus to Ruby open-uri have sent CocoaPods/CocoaPods#10346
I'll consider how to achive moving cdn_url? to Core later, I think it will be another PR

In addition, some minor PRs

@jasl jasl force-pushed the replace-typhoeus-to-async-http branch from f246bf9 to 1327f3c Compare January 12, 2021 20:04
@igor-makarov
Copy link
Contributor

@jasl I've reviewed CocoaPods/CocoaPods#10347.

I'll be waiting for the other items for now.

@jasl
Copy link
Contributor Author

jasl commented Jan 13, 2021

@jasl I've reviewed CocoaPods/CocoaPods#10347.

I'll be waiting for the other items for now.

Thank you, when #670 getting merged, I'll separate a PR including:

  • Require minimal Ruby to 2.5
    • newer Rubocop no longer support Ruby 2.3
    • async require Ruby 2.5+
    • 2.5 is an important version of Ruby
  • Upgrade to Rubocop to 1.8
    • 0.38 too old that made my RubyMine unhappy

Important: if it getting merged which means CocoaPods no longer support macOS 10.14 system Ruby (2.3.7 if my memory correct)

@igor-makarov
Copy link
Contributor

Ok, so to review (each item ideally should be separate PR to review):

  1. Use Ruby bundled json gem instead of forked #670
  2. After that is taken care of, create a PR with just the Ruby version deprecation.
  3. Typhoeus removal PR in Core would consist at this point of only Typhoeus removal.
  4. Because https://github.com/CocoaPods/CocoaPods gem is pinned to Core, deprecate Ruby version there as well and update the version pin.
  5. Update the version pin of https://github.com/CocoaPods/CocoaPods again, this time removing Typhoeus transitive dependency.

@jasl
Copy link
Contributor Author

jasl commented Jan 13, 2021

Ok, so to review (each item ideally should be separate PR to review):

  1. Use Ruby bundled json gem instead of forked #670
  2. After that is taken care of, create a PR with just the Ruby version deprecation.
  3. Typhoeus removal PR in Core would consist at this point of only Typhoeus removal.
  4. Because https://github.com/CocoaPods/CocoaPods gem is pinned to Core, deprecate Ruby version there as well and update the version pin.
  5. Update the version pin of https://github.com/CocoaPods/CocoaPods again, this time removing Typhoeus transitive dependency.

no problem

@amorde
Copy link
Member

amorde commented Jan 14, 2021

I think we can bump Ruby to 2.6 which is shipped with 10.15. If we're going to drop support for 10.14 system ruby, we might as well update to 10.15 system Ruby and use the newest version possible.

@jasl
Copy link
Contributor Author

jasl commented Jan 14, 2021

I think we can bump Ruby to 2.6 which is shipped with 10.15. If we're going to drop support for 10.14 system ruby, we might as well update to 10.15 system Ruby and use the newest version possible.

I'm hoping this, I'd like to do some contributions to help CocoaPods future-proof!

@igor-makarov
Copy link
Contributor

@jasl @amorde agreed about 10.15 - we aren't even testing on 10.14 because GH doesn't have the runners for it.

@jasl
Copy link
Contributor Author

jasl commented Jan 15, 2021

Since CocoaPods/CocoaPods#10346 nearly ready

I'll back to work on this PR on weekend,
I plan to refactor download_file_async and download_impl_async resolving propagating issue first (current impl can't rescue error such as Timeout, NetworkError, etc. in async context)
then update and fix tests

@igor-makarov
Copy link
Contributor

igor-makarov commented Jan 15, 2021

@jasl could you start with creating the Ruby deprecation PR? It will get the ball rolling and begin getting reviewed.

Minimum Ruby version = 2.6.

@igor-makarov igor-makarov changed the title [WIP][Proposal] Replace typhoeus to async-http Replace Typhoeus with async-http Jan 15, 2021
@jasl
Copy link
Contributor Author

jasl commented Jan 15, 2021

@jasl could you start with creating the Ruby deprecation PR? It will get the ball rolling and begin getting reviewed.

Minimum Ruby version = 2.6.

No problem, one more question, can I upgrade bundler in that PR? since Ruby 2.6 bundled bundler 2.0, and 1.8.7 won't compatible with 2.0's Gemfile.lock

@igor-makarov
Copy link
Contributor

@jasl not sure I understand, what isn't compatible with current Bundler 1.17.3? BTW, that's just the version for installing on CI, I believe users can use other versions.

@jasl
Copy link
Contributor Author

jasl commented Jan 15, 2021

@jasl not sure I understand, what isn't compatible with current Bundler 1.17.3? BTW, that's just the version for installing on CI, I believe users can use other versions.

This change not affect users, they install CocoaPods via Rubygems.

I see CocoaPods/CocoaPods#10324 also upgrade Bundler.

Bundler only be used in development and CI,
upgrade to 2.x means CocoaPods developers and CI no longer need gem install bundler -v "~> 1.17".

Besides, Bundler 1.17 has deprecation warning on Ruby 3.0

/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:29: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:35: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:35: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:35: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:44: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:35: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:44: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.
/Users/jasl/.rvm/gems/ruby-3.0.0/gems/bundler-1.17.3/lib/bundler/shared_helpers.rb:118: warning: Pathname#untaint is deprecated and will be removed in Ruby 3.2.

@igor-makarov
Copy link
Contributor

@jasl I think you're right. Also, it seems that going to Ruby 2.6 can allow us to finally get rid of the janky activesupport version pin:

  # 6.0 requires Ruby 2.5.0
  s.add_runtime_dependency 'activesupport', '> 5.0', '< 6'

@jasl
Copy link
Contributor Author

jasl commented Jan 15, 2021

@jasl I think you're right. Also, it seems that going to Ruby 2.6 can allow us to finally get rid of the janky activesupport version pin:

  # 6.0 requires Ruby 2.5.0
  s.add_runtime_dependency 'activesupport', '> 5.0', '< 6'

Yes, I believe AS 5.x has problem on Ruby 2.7+

@igor-makarov
Copy link
Contributor

Cool, so waiting for a PR with:

  • Ruby minimum 2.6.0
  • Bundler bump (CI only)
  • Rubocop bump (dev only)
  • ActiveSupport version specification bumped to '~> 6'

@jasl
Copy link
Contributor Author

jasl commented Jan 15, 2021

Cool, so waiting for a PR with:

  • Ruby minimum 2.6.0
  • Bundler bump (CI only)
  • Rubocop bump (dev only)
  • ActiveSupport version specification bumped to '~> 6'

Correct: Bundler bump (CI & Dev only)

will do later

@igor-makarov
Copy link
Contributor

Thanks for all the work!

@jasl jasl mentioned this pull request Jan 15, 2021
@jasl jasl force-pushed the replace-typhoeus-to-async-http branch from 029a79e to c30e8eb Compare January 17, 2021 21:09
@jasl
Copy link
Contributor Author

jasl commented Apr 2, 2021

long time no follow this, any news? @igor-makarov

@jasl jasl force-pushed the replace-typhoeus-to-async-http branch from 534e239 to 2cfa702 Compare April 2, 2021 08:50
@jasl
Copy link
Contributor Author

jasl commented Apr 2, 2021

Just rebased master, squashed all bundle update

@jasl
Copy link
Contributor Author

jasl commented Jun 15, 2021

Any news here?

@igor-makarov
Copy link
Contributor

Hi @jasl - long time!

I'm currently a bit in a dilemma - on the one hand, this PR contains much work, replacing one library with another. On the other hand - the libraries that are being replaced have already been updated for Apple Silicon - and I can personally confirm that under a correct Ruby installation, Typhoeus works natively.

There's a whole separate issue regarding the term "correct Ruby installation" as apparently the System Ruby on Apple Silicon can't work with native extensions because they get build according to RbConfig which is incorrect.

@jasl
Copy link
Contributor Author

jasl commented Jun 15, 2021

Hi @jasl - long time!

I'm currently a bit in a dilemma - on the one hand, this PR contains much work, replacing one library with another. On the other hand - the libraries that are being replaced have already been updated for Apple Silicon - and I can personally confirm that under a correct Ruby installation, Typhoeus works natively.

There's a whole separate issue regarding the term "correct Ruby installation" as apparently the System Ruby on Apple Silicon can't work with native extensions because they get build according to RbConfig which is incorrect.

I believe nio4r can handle both x86 and arm64 well, this can easily test

@igor-makarov
Copy link
Contributor

@jasl this isn't what worries me - I have tested nio4r on both. What worries me is that we're talking about shipping a HUGE change in a core module - while we actually don't have to.

@segiddins what do you think?

@ioquatix
Copy link

ioquatix commented Jun 15, 2021

If it’s any help we found problems with Typhoeus and it seems largely unmaintained.

I haven’t looked closely at this PR for a while but I would say you could make it lower impact/risk if you build an abstract interface for doing the requests and then support both. Such an approach might make migration easier. We can also compare performance too which should be interesting.

Bearing in mind that you'll want to put as much of the request handling in an async block to get as much concurrency as possible - happy to provide advice here, but downloading lots of streams concurrently should be advantageous for a tool like this.

@igor-makarov
Copy link
Contributor

@ioquatix I'm less worried about the performance impact - there are well-defined places where concurrency is utilized and it's easy to see that it works correctly. I'm more worried about the fact that it's a huge change in a core component and the impact of even a small bug can be catastrophic.

Having 2 backends will increase the amount of work even more that has been invested right now and increase the test surface even more.

The Typhoeus implementation works for the project's needs and it's not a problem to load it on ASi architecture anymore.

@amorde
Copy link
Member

amorde commented Jul 27, 2021

Having ffi has a dependency has introduced friction in more ways than just the introduction of Apple Silicon, so I'd love to remove it. That said I'm not personally familiar with the differences in these two libraries and how much risk there is. I'd suggest releasing it in a few betas to get a sense of stability, and then ship it, as long as we are available to triage any issues that come up

@dnkoutso dnkoutso removed this from the 1.11.0 milestone Aug 3, 2021
@MaxDesiatov
Copy link

Is there anything blocking this PR? Is anything else needed other than resolving conflicts?

@amorde
Copy link
Member

amorde commented Nov 4, 2021

It's mostly a matter of our ability to maintain this going forward + push out potential bug fix releases if we find issues after releasing it.

There's been more issues with ffi recently with the release of macOS Monterey and Xcode 13. I think it is still worthwhile to pursue removing our dependency on ffi

@jasl
Copy link
Contributor Author

jasl commented Nov 4, 2021

I have business traval these days, and will back home next week, I can help to rebase to master branch to fix conflicts

@igor-makarov
Copy link
Contributor

@amorde can you point me to the new ffi issues? Haven't heard about any of it unfortunately.

@amorde
Copy link
Member

amorde commented Nov 5, 2021

We had some internal issues with our CI machines that seem to be related to this https://twitter.com/churowa/status/1455971248555511808

@igor-makarov
Copy link
Contributor

I've gone over this PR one more time and there are several issues still related to using async-http:

  • There's no easy way to use an outgoing proxy like there is with libcurl/typhoeus. I've researched this and it seems that the async-http library added minimal support for this, that cannot work with HTTPS at all.
  • There's no library support for netrc - the hand-rolled implementation may break and we may be required to fix it.

In addition to these gaps, the change is huge and we definitely don't have sufficient tests in place to verify everything.

Much time has passed since the M1 Macs first emerged and there exist versions of ffi and typhoeus that are working correctly on the architecture, even if some CI vendors are having trouble deploying them.

It is my opinion that because of all the reasons I've listed above, we should not commit to making this change.

Big thanks to @jasl for all the hard work done on this, and my sincere apology for not wanting to merge it. 🙏🏻

@jasl
Copy link
Contributor Author

jasl commented Nov 7, 2021

I've gone over this PR one more time and there are several issues still related to using async-http:

  • There's no easy way to use an outgoing proxy like there is with libcurl/typhoeus. I've researched this and it seems that the async-http library added minimal support for this, that cannot work with HTTPS at all.
  • There's no library support for netrc - the hand-rolled implementation may break and we may be required to fix it.

In addition to these gaps, the change is huge and we definitely don't have sufficient tests in place to verify everything.

Much time has passed since the M1 Macs first emerged and there exist versions of ffi and typhoeus that are working correctly on the architecture, even if some CI vendors are having trouble deploying them.

It is my opinion that because of all the reasons I've listed above, we should not commit to making this change.

Big thanks to @jasl for all the hard work done on this, and my sincere apology for not wanting to merge it. 🙏🏻

it's OK, the best is FFI can working good on M1.

For your opinions about async, I should forward to @ioquatix , maybe can be improve in the future, personally proxy support is also important to me.

@amorde
Copy link
Member

amorde commented Nov 7, 2021

Thanks for taking another look @igor-makarov. Do the same points apply to your original PR that uses net-http2? #639

@igor-makarov
Copy link
Contributor

@amorde To be honest it was a draft and I don't think I was close to finishing it 🤷🏻‍♂️

@amorde
Copy link
Member

amorde commented Nov 7, 2021

Ok makes sense. I still do wish we didn't depend on ffi as it tends to be a common source of issues but I understand the reason we aren't there yet.

@dnkoutso
Copy link
Contributor

dnkoutso commented Nov 7, 2021

Should we close those PRs?

@jasl
Copy link
Contributor Author

jasl commented Nov 7, 2021

Ok makes sense. I still do wish we didn't depend on ffi as it tends to be a common source of issues but I understand the reason we aren't there yet.

As a Rubyist, I wish ffi gem can running on M1 smoothly, many gems depend on it.

But for a CLI tool, less C-ext dependencies, less trouble to end-users

@igor-makarov
Copy link
Contributor

Yes, I would have also preferred a Ruby-native implementation, but it's very hard to compete with libcurl in terms of feature completeness. It's truly a Swedish army knife.

@ioquatix
Copy link

ioquatix commented Nov 7, 2021

There's no easy way to use an outgoing proxy like there is with libcurl/typhoeus. I've researched this and it seems that the async-http library added minimal support for this, that cannot work with HTTPS at all.

async-http has full support for HTTP/1 and HTTP/2 proxies including TLS.

There's no library support for netrc - the hand-rolled implementation may break and we may be required to fix it.

Is this what you are referring to? https://everything.curl.dev/usingcurl/netrc

It seems awfully insecure. I don't think we'd want to support this by default.

@igor-makarov
Copy link
Contributor

@ioquatix I've seen that the library does offer proxy support. However, I've spent most of yesterday trying to allow to users to enable it as a simple http_proxy env var, and I've not been successful.
Specifically, I've gotten to the point where the outgoing connection was being established, and then failing due to TLS validation. I wasn't able to figure out how to disable certificate validation for my localhost proxy.

As for netrc - a lot of the users are using it, and depend on it to authenticate with their private spec repos and Artifactories. It's a def facto standard for basic auth and CocoaPods needs to support it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants