maven.twttr.com outage - 503 errors - breaks builds of downstream projects #148

mmisiewicz-yext · 2020-12-01T17:04:31Z

It seems that maven.twttr.com is currently returning 503 when attempting to retrieve artifacts. This prevents any builds which depend on this artifact from succeeding

Describe the bug
maven.twttr.com returns 503 Service Temporarily Unavailable.

To Reproduce
curl -vvv https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.pom

Expected behavior
Twitter Maven runs correctly.

Environment
I have been able to reproduce from multiple different ISPs.

Additional context
This outage has been ongoing for at least 24h now.

The text was updated successfully, but these errors were encountered:

mattburgess · 2020-12-02T12:00:10Z

A reoccurence of #138 by the looks of it. As mentioned there, it would be really handy if these artifacts were made available on maven central as well/instead of maven.twtrr.com.

willnorris · 2020-12-02T21:05:33Z

Things seem to be back up now. I'll follow up internally to make sure we've got appropriate monitoring on this service as well as see if there's a reason these artifacts aren't also mirrored to maven central.

mattburgess · 2020-12-03T10:07:46Z

Thanks @willnorris - it appears to be down again this morning.

nenad-spuzic · 2020-12-03T22:29:22Z

@willnorris It still appears to be down

willnorris · 2020-12-03T22:57:19Z

hmm... so far I've been unable to catch it while it is down, and I don't see anything in our monitoring about 500s (though I suspect I'm just looking at the wrong monitoring). Are you seeing it down consistently or just intermittently?

mattburgess · 2020-12-03T23:11:23Z

It's down for me right now. I wonder whether it's geographical? Just in case it is, I happen to be in the UK. I'm not polling it continuously, but when I do hit it it's always been down.

nenad-spuzic · 2020-12-04T10:24:39Z

@willnorris Consistently down. Whenever I load https://maven.twttr.com/ I get a 503 error. I've loaded it at least 30 times now, general time frames:

2 hours before I send my previous comment
at the time of my previous comment
3 hours before my current comment
now

Additionally, I have 9 different apps in the cloud whose builds are failing since they depend on this repository being online, at it is returning 503 for different artifacts during the build process.

It doesn't seem to be a geographical problem since, it does not work from me in Serbia, does not work over a US proxy, and builds hosted in France are failing (that depend on this repository)

I need this repository back online as soon as possible, if you need any additional information from me to start troubleshooting this, please let me now.

mmisiewicz-yext · 2020-12-04T17:07:07Z

Consistently down here from a US ISP. Here's the output from curl -vvv if that's any help:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 199.16.156.89...
* TCP_NODELAY set
* Connected to maven.twttr.com (199.16.156.89) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [229 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [66 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2861 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; OU=atla; CN=maven.twttr.com
*  start date: Feb  6 00:00:00 2020 GMT
*  expire date: Feb  5 12:00:00 2021 GMT
*  subjectAltName: host "maven.twttr.com" matched cert's "maven.twttr.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd044808200)
> GET /com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.pom HTTP/2
> Host: maven.twttr.com
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 4294967295)!
< HTTP/2 503
< cache-control: no-cache, no-store, max-age=0
< content-length: 5670
< content-security-policy: default-src 'none'; img-src https://abs.twimg.com https://ssl.google-analytics.com http://www.google-analytics.com; script-src https://abs.twimg.com https://ssl.google-analytics.com https://ajax.googleapis.com http://www.google-analytics.com about:; style-src https://abs.twimg.com https://fonts.googleapis.com 'unsafe-inline'; font-src https://abs.twimg.com https://twitter.com; connect-src 'none'; object-src 'none'; media-src 'none'; frame-src 'none'; report-uri https://twitter.com/i/csp_report?a=ORTGK%3D%3D%3D&ro=false
< content-type: text/html;charset=utf-8
< date: Fri, 04 Dec 2020 17:05:23 GMT
< server: tsa_b
< strict-transport-security: max-age=631138519
< x-connection-hash: 630d602d603279d7c161aa24f7686abf
< x-response-time: 74
< x-xss-protection: 0
<
{ [4087 bytes data]
100  5670  100  5670    0     0  14575      0 --:--:-- --:--:-- --:--:-- 14575
* Connection #0 to host maven.twttr.com left intact
* Closing connection 0

dig output:

➜  dig maven.twttr.com

; <<>> DiG 9.10.6 <<>> maven.twttr.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 29943
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;maven.twttr.com.		IN	A

;; ANSWER SECTION:
maven.twttr.com.	203	IN	CNAME	maven.twitter.com.
maven.twitter.com.	203	IN	A	199.16.156.89

brahma19 · 2020-12-06T13:39:54Z

This seems to be working from IND / US ISP as of 06/12 , 19:05 IST.

willnorris · 2020-12-09T19:09:10Z

as @brahma19 noted, the immediate problem should be resolved. I'm still working on the longer term solution to try to make sure this doesn't happen again.

pjfanning · 2022-03-23T13:52:22Z

@willnorris this is happening again today

Would it be possible to publish hadoop-lzo to oss.sonatype.org like most companies do? Jars published to oss.sonatype.org end up being duplicated around many maven repos, most notably https://repo1.maven.org/

2022-03-23T12:57:08.1776904Z [ERROR] Failed to execute goal on project zeppelin-scalding_2.10: Could not resolve dependencies for project org.apache.zeppelin:zeppelin-scalding_2.10:jar:0.11.0-SNAPSHOT: Could not transfer artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.19 from/to twitter (https://maven.twttr.com): transfer failed for https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/hadoop-lzo-0.4.19.jar, status: 503 Service Unavailable -> [Help 1]

willnorris · 2022-03-23T14:17:19Z

Gah! Thanks for letting me know. I'll get into it today.

willnorris · 2022-03-23T17:31:16Z

okay, it's back up. Immediate fire extinguished, but I'll be continuing to work on removing the reliance on maven.twttr.com and to just get this (and others) published into maven central.

willnorris · 2022-05-05T17:34:09Z

Filed an issue with Hadoop project to track discussion of publishing on maven central with the current groupId: https://issues.apache.org/jira/browse/HADOOP-18223

pedroslopez · 2023-01-05T18:57:27Z

This has been happening once again - maven.twttr.com is down with a 503

curl -vvv https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.20/hadoop-lzo-0.4.20.pom
*   Trying 199.59.149.208:443...
* Connected to maven.twttr.com (199.59.149.208) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; CN=maven.twttr.com
*  start date: Nov 14 00:00:00 2022 GMT
*  expire date: Nov 14 23:59:59 2023 GMT
*  subjectAltName: host "maven.twttr.com" matched cert's "maven.twttr.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x15b011a00)
> GET /com/hadoop/gplcompression/hadoop-lzo/0.4.20/hadoop-lzo-0.4.20.pom HTTP/2
> Host: maven.twttr.com
> user-agent: curl/7.77.0
> accept: */*
> 
< HTTP/2 503 
< content-length: 91
< content-type: text/plain
< x-connection-hash: 282072b4961e6f43bb64596f79d8f79f8f4e3784c228582ad8222bc439c34a87
< date: Thu, 05 Jan 2023 18:56:29 GMT
< server: tsa_a
< 
* Connection #0 to host maven.twttr.com left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure

@willnorris was this issue ever resolved?

mmisiewicz-yext · 2023-01-05T21:16:59Z

Maybe Elon auctioned off the server.

…

On Thu, Jan 5, 2023 at 1:57 PM Pedro S. Lopez ***@***.***> wrote: This has been happening once again - maven.twttr.com is down with a 503 curl -vvv https://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.20/hadoop-lzo-0.4.20.pom * Trying 199.59.149.208:443... * Connected to maven.twttr.com (199.59.149.208) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/cert.pem * CApath: none * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=California; L=San Francisco; O=Twitter, Inc.; CN=maven.twttr.com * start date: Nov 14 00:00:00 2022 GMT * expire date: Nov 14 23:59:59 2023 GMT * subjectAltName: host "maven.twttr.com" matched cert's "maven.twttr.com" * issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS RSA SHA256 2020 CA1 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x15b011a00) > GET /com/hadoop/gplcompression/hadoop-lzo/0.4.20/hadoop-lzo-0.4.20.pom HTTP/2 > Host: maven.twttr.com > user-agent: curl/7.77.0 > accept: */* > < HTTP/2 503 < content-length: 91 < content-type: text/plain < x-connection-hash: 282072b4961e6f43bb64596f79d8f79f8f4e3784c228582ad8222bc439c34a87 < date: Thu, 05 Jan 2023 18:56:29 GMT < server: tsa_a < * Connection #0 to host maven.twttr.com left intact upstream connect error or disconnect/reset before headers. reset reason: connection failure @willnorris <https://github.com/willnorris> was this issue ever resolved? — Reply to this email directly, view it on GitHub <#148 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALSZMZT4U5O5PWJU3LUUTUTWQ4KSFANCNFSM4UJJKCNQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- *Michael Misiewicz* Director, Data Science ***@***.***

willnorris · 2023-01-05T23:02:25Z

The general response from the Hadoop project was that this shouldn't be published in Maven Central with the current artifact ID in the com.hadoop namespace. We had begun the work on moving it to a new com.twitter ID and publish it under that, but it wasn't completed before all of the involved folks left Twitter. I am not optimistic that this would rise to anyone's attention that is still at the company.

My recommendation would be:

short term, someone that is sufficiently interested republish this in Maven Central under some other artifact ID. The license lets you do it... the hardest part is probably just getting an account setup with Sonatype so you can publish (maybe that's easier now?)
long term, you should probably strongly reconsider having this in your dependency tree. I have no idea how hard that would be for different projects, but it's work that should be budgeted for. Twitter was already not using this package much (or at all?) internally, and now there is very little hope of any support from Twitter if major issues are discovered or other maintenance work is needed.

As for getting maven.twttr.com back online... I'm honestly not sure what to tell you. Every single one of the people I would have reached out to was laid off. I don't even know who is left at Twitter or how to get in touch with them. If I recall, it wasn't super hard to get it back online last time (restarting an Apache server maybe? perhaps something related to an SSL cert).

I guess my biggest regret is that I didn't do more to shore up some of this stuff before I left. 😕 🤷🏻

davinchia · 2023-01-06T23:01:41Z

We managed to compile a version of this jar internally and are considering making this public in our own hosted repository if there is enough interest. Would it be helpful to folks?

willnorris · 2023-01-07T18:41:11Z

Actually, now that I think more about it... I believe maven.twttr.com was likely only running in the SMF1 data center in Sacramento (that's where a lot of internal tools ran). If reports of Twitter shutting down that data center are true, then it's very possibly that's why this went offline, and the chances of getting it back online are rather slim.

saurabh-deochake · 2023-01-10T20:18:26Z

Hi all, apologies for the inconvenience. We did a configuration change and the site should be up now. I came across this thread and was able to find folks who could help. Please note that I am not active support for this site. But, happy to help check internally. Thanks!

zman0900 · 2024-01-02T22:01:53Z

Looks like it is dead again with 503 error. Plus the status page https://status.twitterstat.us/ that is linked from that error page has an expired certificate.

vxrl-esimplicity · 2024-01-04T17:50:22Z

Looks like it is dead again with 503 error. Plus the status page https://status.twitterstat.us/ that is linked from that error page has an expired certificate.

@zman0900 Hello. Any updates on this error? TIA.

rohitdogra99 · 2024-01-08T04:22:55Z

Hi Team, Any upates

pjfanning mentioned this issue Mar 23, 2022

ZEPPELIN-5690: use jinjava 2.5.4 due to cves apache/zeppelin#4329

Merged

1 task

willnorris self-assigned this Mar 23, 2022

willnorris removed their assignment Jun 17, 2022

davinchia mentioned this issue Jan 6, 2023

Could not find artifact com.hadoop.gplcompression:hadoop-lzo:jar:0.4.16 in Twitter public Maven repo (http://maven.twttr.com) #140

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

maven.twttr.com outage - 503 errors - breaks builds of downstream projects #148

maven.twttr.com outage - 503 errors - breaks builds of downstream projects #148

mmisiewicz-yext commented Dec 1, 2020

mattburgess commented Dec 2, 2020

willnorris commented Dec 2, 2020

mattburgess commented Dec 3, 2020

nenad-spuzic commented Dec 3, 2020

willnorris commented Dec 3, 2020 •

edited

Loading

mattburgess commented Dec 3, 2020 •

edited

Loading

nenad-spuzic commented Dec 4, 2020 •

edited

Loading

mmisiewicz-yext commented Dec 4, 2020

brahma19 commented Dec 6, 2020

willnorris commented Dec 9, 2020

pjfanning commented Mar 23, 2022

willnorris commented Mar 23, 2022

willnorris commented Mar 23, 2022

willnorris commented May 5, 2022

pedroslopez commented Jan 5, 2023

mmisiewicz-yext commented Jan 5, 2023 via email

willnorris commented Jan 5, 2023

davinchia commented Jan 6, 2023 •

edited

Loading

willnorris commented Jan 7, 2023

saurabh-deochake commented Jan 10, 2023

zman0900 commented Jan 2, 2024

vxrl-esimplicity commented Jan 4, 2024

rohitdogra99 commented Jan 8, 2024

maven.twttr.com outage - 503 errors - breaks builds of downstream projects #148

maven.twttr.com outage - 503 errors - breaks builds of downstream projects #148

Comments

mmisiewicz-yext commented Dec 1, 2020

mattburgess commented Dec 2, 2020

willnorris commented Dec 2, 2020

mattburgess commented Dec 3, 2020

nenad-spuzic commented Dec 3, 2020

willnorris commented Dec 3, 2020 • edited Loading

mattburgess commented Dec 3, 2020 • edited Loading

nenad-spuzic commented Dec 4, 2020 • edited Loading

mmisiewicz-yext commented Dec 4, 2020

brahma19 commented Dec 6, 2020

willnorris commented Dec 9, 2020

pjfanning commented Mar 23, 2022

willnorris commented Mar 23, 2022

willnorris commented Mar 23, 2022

willnorris commented May 5, 2022

pedroslopez commented Jan 5, 2023

mmisiewicz-yext commented Jan 5, 2023 via email

willnorris commented Jan 5, 2023

davinchia commented Jan 6, 2023 • edited Loading

willnorris commented Jan 7, 2023

saurabh-deochake commented Jan 10, 2023

zman0900 commented Jan 2, 2024

vxrl-esimplicity commented Jan 4, 2024

rohitdogra99 commented Jan 8, 2024

willnorris commented Dec 3, 2020 •

edited

Loading

mattburgess commented Dec 3, 2020 •

edited

Loading

nenad-spuzic commented Dec 4, 2020 •

edited

Loading

davinchia commented Jan 6, 2023 •

edited

Loading