stream client proxy not working

[kares]

I've tried setting up a proxy with the streaming client and it does not work, sample script:

require 'twitter'
require 'twitter/version'

puts "twitter gem: #{Twitter::Version.to_s}"

def configure(client)
  client.consumer_key = '...'
  client.consumer_secret = '...'
  client.access_token = '...'
  client.access_token_secret = '...'
  client
end

stream_client = configure Twitter::Streaming::Client.new
stream_client.proxy = { proxy_address: '192.168.56.1', proxy_port: 3128 }

rest_client  = configure Twitter::REST::Client.new
rest_client.proxy = { host: '192.168.56.1', port: 3128 }

user = rest_client.user('tenderlove')
p user; puts user.uri # WORKS

puts "filtering (using stream client): #{stream_client}"
stream_client.filter({ track: "xxx" }) do |tweet|
  p tweet.full_text; puts tweet.uri # FAILS WITH A PROXY
  exit 1
end

Have tested this setup using twitter gem 6.2.0 as well as latest (7.0.0) on MRI 2.5.7 and JRuby 9.2.

Twitter::Error::BadRequest: 
  on_headers_complete at /home/borg/jruby-9.2.13.0/lib/ruby/gems/shared/gems/twitter-7.0.0/lib/twitter/streaming/response.rb:24
                   << at org/ruby_http_parser/RubyHttpParser.java:370
                   << at /home/borg/jruby-9.2.13.0/lib/ruby/gems/shared/gems/twitter-7.0.0/lib/twitter/streaming/response.rb:19
               stream at /home/borg/jruby-9.2.13.0/lib/ruby/gems/shared/gems/twitter-7.0.0/lib/twitter/streaming/connection.rb:25
                 loop at org/jruby/RubyKernel.java:1442
               stream at /home/borg/jruby-9.2.13.0/lib/ruby/gems/shared/gems/twitter-7.0.0/lib/twitter/streaming/connection.rb:21
              request at /home/borg/jruby-9.2.13.0/lib/ruby/gems/shared/gems/twitter-7.0.0/lib/twitter/streaming/client.rb:123
               filter at /home/borg/jruby-9.2.13.0/lib/ruby/gems/shared/gems/twitter-7.0.0/lib/twitter/streaming/client.rb:38

Believe the problem might be with the http gem, potentially not handling the proxy setup for streaming connection?
I was able to verify the proxy isn't an issue, I've tried Python's (tweepy) client with a similar setup using a proxy.
The proxy is a standard Squid installation on ubuntu, listening on its default 3128 port.

The Ruby client seems to be trying a POST request for streaming which gets rejected:

2020/09/29 14:58:54.410 kid1| 11,2| client_side.cc(1302) parseHttpRequest: HTTP Client local=192.168.56.1:3128 remote=192.168.56.102:33692 FD 11 flags=1
2020/09/29 14:58:54.410 kid1| 11,2| client_side.cc(1303) parseHttpRequest: HTTP Client REQUEST:
---------
POST /1.1/statuses/filter.json?track=xxx HTTP/1.1
User-Agent: TwitterRubyGem/7.0.0
Authorization: OAuth oauth_consumer_key="...", oauth_nonce="...", oauth_signature="...", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1601384334", oauth_token="16308531-...", oauth_version="1.0"
Host: stream.twitter.com
Content-Length: 0


----------
2020/09/29 14:58:54.411 kid1| 11,2| Stream.cc(266) sendStartOfMessage: HTTP Client local=192.168.56.1:3128 remote=192.168.56.102:33692 FD 11 flags=1
2020/09/29 14:58:54.411 kid1| 11,2| Stream.cc(267) sendStartOfMessage: HTTP Client REPLY:
---------
HTTP/1.1 400 Bad Request
Server: squid/4.10
Mime-Version: 1.0
Date: Tue, 29 Sep 2020 12:58:54 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 3575
X-Squid-Error: ERR_INVALID_URL 0
Vary: Accept-Language
Content-Language: en
X-Cache: MISS from precision
X-Cache-Lookup: NONE from precision:3128
Via: 1.1 precision (squid/4.10)
Connection: close


----------

While the Python client simply issues a CONNECT:

2020/09/29 15:25:15.642 kid1| 11,2| client_side.cc(1302) parseHttpRequest: HTTP Client local=192.168.56.1:3128 remote=192.168.56.102:33704 FD 11 flags=1
2020/09/29 15:25:15.642 kid1| 11,2| client_side.cc(1303) parseHttpRequest: HTTP Client REQUEST:
---------
CONNECT stream.twitter.com:443 HTTP/1.0


----------

---

Also the proxy settings are bit confusing as they're different based on the client being used.
Which is a shame considering most of the configuration is the same for both the rest and streaming client:

if client.is_a?(Twitter::REST::Client)
  client.proxy = { host: @proxy_address, port: @proxy_port }
else
  client.proxy = { proxy_address: @proxy_address, proxy_port: @proxy_port }
end

[kares]

an older version of the gem (5.15.0 + http.rb 0.9.9) seems to be working, proxy log:

2020/10/05 11:54:47.807 kid1| 11,2| client_side.cc(1302) parseHttpRequest: HTTP Client local=127.0.0.1:3128 remote=127.0.0.1:58600 FD 11 flags=1
2020/10/05 11:54:47.807 kid1| 11,2| client_side.cc(1303) parseHttpRequest: HTTP Client REQUEST:
---------
POST https://stream.twitter.com/1.1/statuses/filter.json?track=xxx HTTP/1.1
Authorization: OAuth oauth_consumer_key="...", oauth_nonce="...", oauth_signature="...", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1601891687", oauth_token="...", oauth_version="1.0"
Host: stream.twitter.com
User-Agent: http.rb/0.9.9


----------
2020/10/05 11:54:48.274 kid1| 11,3| http.cc(2310) httpStart: POST https://stream.twitter.com/1.1/statuses/filter.json?track=London,Barcelona
2020/10/05 11:54:48.275 kid1| 11,2| http.cc(2266) sendRequest: HTTP Server local=192.168.0.25:46560 remote=199.16.156.200:443 FD 14 flags=1
2020/10/05 11:54:48.275 kid1| 11,2| http.cc(2267) sendRequest: HTTP Server REQUEST:
---------
POST /1.1/statuses/filter.json?track=xxx HTTP/1.1
Authorization: OAuth oauth_consumer_key="...", oauth_nonce="59e85cfa31e58cb41085595bb3688f74", oauth_signature="...", oauth_signature_method="HMAC-SHA1", oauth_timestamp="1601891687", oauth_token="...", oauth_version="1.0"
User-Agent: http.rb/0.9.9
Host: stream.twitter.com
Via: 1.1 precision (squid/4.10)
X-Forwarded-For: 127.0.0.1
Cache-Control: max-age=0
Connection: keep-alive


----------

the only difference seems to be Content-Length: 0
and sending a full URL: POST https://stream.twitter.com/1.1/statuses/filter.json?track=xxx HTTP/1.1

[kares]

🥗 confirmed, patching HTTP to send full URIs via proxy seems to resolve the issue:
(note that to connect to twitter's streaming API we're using https://)

::HTTP::Request.class_eval do
  def headline
    request_uri =
        if using_proxy? #&& !uri.https?
          uri.omit(:fragment)
        else
          uri.request_uri
        end

    "#{verb.to_s.upcase} #{request_uri} HTTP/#{version}"
  end
end

... the patch essentially undoes: httprb/http#333 (which has been shipped since http.rb 2.0.0)

[kares]

to recap, this is a (squid) proxy issue (httprb/http#333 seems legit),
still, annoying that things got broken with various proxies out there in the 6.x/7.0 line ...

[cmirnow]

Working through a proxy, I get an error: cannot interpret as DNS name: nil (gem 'twitter' 7.0.0).
Reproduce:

def twi_sclient(t)
   Twitter::Streaming::Client.new config(t)
 end

 def config(t)
   {
     consumer_key: t.key,
     consumer_secret: t.secret,
     access_token: t.token,
     access_token_secret: t.token_secret,
     proxy: proxy(t)
   }
 end

 def proxy(t)
     {
       host: t.host,
       port: t.port,
       username: t.username,
       password: t.password
     }
 end

[ActiveJob] [RetweetsJob] [d564b4c5-6f04-4191-bd69-bba02a4965e0] Error performing RetweetsJob (Job ID: d564b4c5-6f04-4191-bd69-bba02a4965e0) from Async(default) in 15.55ms: ArgumentError (cannot interpret as DNS name: nil)

Any ideas? Thank you.