proposal: Use opendal to handle the cache storage operations

[Xuanwo]

Background

Hi, I'm the maintainer of opendal, a rust lib that focuses on accessing data freely, painlessly, and efficiently.

OpenDAL now supports almost all storage services, including s3, gcs, azblob, oss, obs, hdfs, redis, fs and so on.

use anyhow::Result;
use opendal::Operator;
use opendal::Scheme;

#[tokio::main]
async fn main() -> Result<()> {
    // Init Operator
    let op = Operator::from_env(Scheme::S3)?;

    // Create object handler.
    let o = op.object("test_file");

    // Write data info object;
    o.write("Hello, World!").await?;

    // Read data from the object;
    let bs = o.read().await?;

    // Delete object.
    o.delete().await?;

    Ok(())
}

I came across sccache last week and found opendal is a great choice for sccache, which can help resolve many issues like #1384, #1180, #1167.

Apart from various storage backend supports, opendal also has the following benefits:

• Full observability support: logging, tracing, metrics.
• Automatic retry support
• Used in production by databases like databend

Proposal

So I propose to replace the storage implementation with opendal instead.

I built a quick demo here: https://github.com/Xuanwo/cccache/tree/debug. Most of the change will be like:

- let credentials = self
-     .provider
-     .credentials()
-     .await
-     .context("failed to get AWS credentials")?;
- 
- let bucket = self.bucket.clone();
- let _ = bucket
-     .put(&key, data, &credentials)
-     .await
-     .context("failed to put cache entry in s3")?;
+ self.op.object(&key).write(data).await?;

If this proposal is accepted, I plan to

• Replace them one by one (s3, gcs, ...)
• Make sure all existing config work as before
• Then adding new services support

---

Any comments are welcome! Thanks in advance.

[sylvestre]

Looks interesting
Did you look at the performance impact?

[sylvestre]

Also, do you know any project using opendal ?

[Xuanwo]

Did you look at the performance impact?

Take s3 as an example, read and write are mapped to put_obejct and get_object directly. I think no performance impact will be observed by users.

https://github.com/datafuselabs/opendal/blob/39dc79e42050af463a33755cffb6d4aec1e96758/src/services/s3/backend.rs#L886-L903

Also, do you know any project using opendal ?

databend is a cloud data warehouse which uses opendal heavily.

[sylvestre]

I think no performance impact will be observed by users.

Sure but I would like to see some benchmark :) The devil is in the details

databend is a cloud data warehouse which uses opendal heavily.

Are you aware of other projects?

[Xuanwo]

Sure but I would like to see some benchmark :) The devil is in the details

Agreed.

How about this benchmark plan?

• Compile different projects with no cache, sccache and sccache (opendal powered)
• Recode the finish durations for analysis.

Are you aware of other projects?

Small project like deepeth/mars.

No others so far to my knowledge. I'm seeking to expand the usage too.

[PsiACE]

opendal was created precisely to simplify access to data.

I think opendal is a good fit for sccache, taking over all the data storage services that might need to be considered without having to bother with abstractions and complex dependencies. And vice versa, sccache is a good project to test and validate opendal designs and implementations.

Although there may be a lack of more projects using opendal at the moment, I thought it would be worthwhile to try it out on sccache.

[sylvestre]

Sure, I just don't want to add a critical dependency which could be unmaintained in a few months

[Xuanwo]

I did a simple benchmark.

Setup

Test with databend at commit b24be85

Both of sccache and cccache are using the same local minio with different bucket.

Sccache

sccache v0.3.1, built with release profile under stable rust

export RUSTC_WRAPPER=/home/xuanwo/Code/mozilla/sccache/target/release/sccache
export CARGO_INCREMENTAL=0

export SCCACHE_BUCKET=sccache
export SCCACHE_ENDPOINT=127.0.0.1:9900

export AWS_ACCESS_KEY_ID=minioadmin 
export AWS_SECRET_ACCESS_KEY=minioadmin

cargo build

Cccache

Cccache at commit c1a725f0, built with release profile under stable rust.

cccache remove the dist-server related features which is not covered by this proposal, but I think they should not affect the perf.

export RUSTC_WRAPPER=/home/xuanwo/Code/xuanwo/cccache/target/release/cccache
export CARGO_INCREMENTAL=0

export SCCACHE_BUCKET=cccache
export SCCACHE_ENDPOINT=127.0.0.1:9900

export AWS_ACCESS_KEY_ID=minioadmin 
export AWS_SECRET_ACCESS_KEY=minioadmin

cargo build

Result

Without any cache

Finished dev [unoptimized + debuginfo] target(s) in 2m 45s
cargo build  1722.26s user 174.20s system 1147% cpu 2:45.30 total

With sccache

First build

Finished dev [unoptimized + debuginfo] target(s) in 3m 45s
cargo build  407.65s user 76.37s system 214% cpu 3:46.04 total

Build after cargo clean

Finished dev [unoptimized + debuginfo] target(s) in 2m 38s
cargo build  325.56s user 59.88s system 243% cpu 2:38.17 total

With cccache

First build

Finished dev [unoptimized + debuginfo] target(s) in 3m 34s
cargo build  557.05s user 99.79s system 305% cpu 3:34.94 total

Build after cargo clean

Finished dev [unoptimized + debuginfo] target(s) in 2m 31s
cargo build  427.92s user 70.19s system 327% cpu 2:31.96 total

---

The result is stable within couple of seconds.

[sylvestre]

Could you please try benchmark with hyperfine ?
I don't trust single run compilation

[Xuanwo]

On the same setup:

Sccache

hyperfine --warmup 3 "cargo clean; cargo build" --export-markdown=/tmp/sccache.md

Command	Mean [s]	Min [s]	Max [s]	Relative
cargo clean; cargo build	185.520 ± 6.944	176.236	198.428	1.00

Cccache

hyperfine --warmup 3 "cargo clean; cargo build" --export-markdown=/tmp/cccache.md

Command	Mean [s]	Min [s]	Max [s]	Relative
cargo clean; cargo build	169.907 ± 5.350	160.920	177.485	1.00

[Xuanwo]

Sure, I just don't want to add a critical dependency which could be unmaintained in a few months

We have great news related to this to share!

The newly open-source project greptimedb is using OpenDAL too.

@sunng87 from the @GreptimeTeam is one of the contributors to OpenDAL.

---

We do know the concern of the long-term maintenance status of the project. And we are working hard to grow our community and introduce more maintainers (outside of our own team) to address that.

[Xuanwo]

In PR #1412 I built a quick demo in about 5 minutes, please take a look and leave any comments so we can improve.

[kesavkolla]

@Xuanwo does OpenDAL uses connection pool? All the calls will be to the same backend S3. It would be nice to re-use the connections so that whole SSL handshakes are minimized and also avoiding any DNS lookups.

@sylvestre I know for sure the aws sdk doesn't use any connection reuse. I had used very similar project to OpenDAL for replacing the AWS SDK and gained tremendous performance improvement. I used object_store from apache arror-rs and the performance to interact with S3 compatible is very better than standard AWS SDK.

I am assuming OpenDAL also must be similiar to object_store if so sccache would benefit a lot with that implementation.

[Xuanwo]

does OpenDAL uses connection pool?

Yes.

also avoiding any DNS lookups

We also support caching DNS lookups by enabling trust-dns features.

[sylvestre]

reopening as we "only" have s3 & azure.

@Xuanwo I would be curious if you could do GCS next. I would like to test it for Firefox CI.

[Xuanwo]

@Xuanwo I would be curious if you could do GCS next. I would like to test it for Firefox CI.

Yes, absolutely! I'm working on migrating gcs now. The only thing blocks me is apache/opendal#1062

I expect to implement it before tomorrow.

[Xuanwo]

@Xuanwo I would be curious if you could do GCS next. I would like to test it for Firefox CI.

Done by #1474

[Xuanwo]

Most opendal related work has been done. Let's close this issue.