A handy lookup table of similar technology and services to help ex-googlers survive the real world :) pull-requests are very welcomed. Please do not list any confidential projects!
For a working example of (some) of these technologies integrated together, see: https://github.com/google/startup-os
See also: System Design Primer, The Hadoop Ecosystem Table, Map AWS services to GCP products, Database of Databases, Google Open Source Glossary
| Google Internal | Google External | Open Source / Real-World |
|---|---|---|
| MapReduce | Apache Hadoop, Spark | |
| Flume | DataFlow | Apache Beam |
| Protocol Buffer | Protobuf, FlatBuffers | Cap'n Proto, Thrift, Avro, Amazon Ion, CBOR, kryo |
| Stubby | gRPC | Thrift, Bolt |
| Chubby | Apache Zookeeper, etcd, HashiCorp Consul | |
| Goops / PubSub | Cloud Pub/Sub | Apache Kafka, Apache Pulsar, Facebook LogDevice |
//base |
abseil |
| Google Internal | Google External | Open Source / Real-World |
|---|---|---|
| Borg | Kubernetes | Apache Mesos, Apache Aurora, HashiCorp Nomad |
| GSLB | Cloud Load Balancing - Internal | AWS ELB, Istio, linkerd |
| OnePlatform | API Gateway | Swagger |
| GFE, Maglev, uberproxy | Cloud Load Balancing - HTTPS / External | envoy, AWS ALB, HAProxy, nginx, F5 |
| uberproxy (sso proxy) | Identity-Aware Proxy | buzzfeed-sso, Pomerium |
| Zanzibar | Zanzibar Research Paper | SpiceDB/authzed, Ory Keto, topaz, Opal, (iam)Keycloak, Warrant |
| data center hardware | open compute | |
| Jupiter, Starblaze | ||
| B4, Stargate, TE | ||
| USPS, Andromeda | ||
| ESDN | Faucet | |
| broccoli man |
| Google Internal | Google External | Open Source / Real-World |
|---|---|---|
| GFS/Colossus | HDFS, Ceph, GlusterFS, MooseFS | |
| BigTable | Cloud BigTable | PrestoDB, Cassandra, HBase, Accumulo, DynamoDB, ScyllaDB |
| Spanner | Cloud Spanner | Vitess, CockroachDB, TiDB |
| ColumnIO / Capacitor | Apache Parquet, ORC | |
| sstable | levelDB | RocksDB, pebble |
| zippy | Snappy | lz4 |
| RecordIO | Riegeli, TFRecords, & in OR-Tools, szl | stuffed-record-stream |
| Google Internal | Google External | Open Source | SaaS |
|---|---|---|---|
| Dremel | BigQuery | Apache Drill, Presto, Spark(sort-of), | AWS Athena, Redshift Spectrum, Snowflake |
| Dremel UI | Redash, Metabase, Apache Superset | ||
| Search (Mustang, Alexandria) | Elasticsearch, OpenSearch, Solr, Lucene | algolia | |
| pubsub | pubsub | NATS.io, RabbitMQ, PubNub | AWS SQS/SNS, AWS AppSync |
| MillWheel | Cloud Dataflow | Apache Flink, beam | |
| Colab | Colaboratory | Jupyter | Observable |
| PLX | Google Data Studio | Mode | |
| Monarch | paper | ||
| Napa | paper | ||
| MakerSuite | MakerSuite |
| Google Internal | Google External | Real-World |
|---|---|---|
| Assimilator | Dominator | |
| Blaze | Bazel | Buck, Pants, please.build, Blade, recc, BuildBuddy, flare.build |
| Oncall | PagerDuty, OpsGenie, VictorOps | |
| varz/borgmon/monarch | Cloud Monitoring | Datadog, Prometheus, M3, librato, newrelic, skylight, scout, Scotty/tricorder, netdata, bosun, also this and this |
| Viceroy | Cloud Monitoring | Grafana |
| Exception/Error Tracking (thirdeye) | Sentry.io, Raygun.io, Rollbar, Honeybadger, Airbrake, OverOps, ELK stack | |
| styleguides | google styleguides | PEP-8, HoundCI auto-style-reviewer |
| Sponge |