Setting up Distributed sccache on EC2 instances.

[rahulbansal16]

Hi,

I am trying to set up the sccache on the Ec2 instances and I am not able to get it up and running. I have tried few techniques to make it work but I have not got any success.

I thought of documenting all the steps that I took and will keep on posting the new ones.

Intended Setup
I want to put the scheduler server on a dedicated machine and connect the build servers to the scheduler. I want the scheduler to be accessible by the General Public with the required token.

This means that the build server should be running on the public IP instead of the private one.

Approaches tried

1. Setting the public_addr value to the public IP in the build server

   The public IP needs to be associated with the OS for this approach to work. I did not have much idea about doing that so I thought of trying the next approach. Refer to the StackOverflow question for more details.

2. Listening on the Private IP

   I tried listening on the Private IP of the build server to at least get the sccache working for the subnet. I set the value of the public_addr value to 172.23.23.22:10501. This leads to an error of invalid_bearer_token_mismatched_address. I have set the scheduler auth type to DANGEROUSLY_INSECURE.

3. Setting the public_addr to 127.0.0.1:10501 and doing a Port Forwarding from Scheduler and Client

   In this, I thought of keeping the public_addr value to 127.0.0.1:10501 and doing a port forwarding from the client and scheduler to build server port 10501. This gives the error channel 3: open failed: connect failed: Connection refused

4. Hardcoding the ip addresses

   I hardcoded the address in the line

   sccache/src/dist/http.rs

   Line 772 in 915e977

   let server = rouille::Server::new(public_addr, move |request| {

   with the public_addr "0.0.0.0:10501" to make it possible to expose the port outside VM and returning server_id in the line

   sccache/src/dist/http.rs

   Line 719 in 915e977

   return make_401("invalid_bearer_token_mismatched_address");

   EDIT
   I got it working via the 4th approach

[luser]

I want to put the scheduler server on a dedicated machine and connect the build servers to the scheduler. I want the scheduler to be accessible by the General Public with the required token.

FYI, sccache's distributed compilation support was not designed to be used with untrusted users. While it does sandbox compile jobs on the build servers, I'd caution against relying on that to provide access to arbitrary users.

[rahulbansal16]

@luser Thanks for the information. I am trying to setup a one-click distributed sccache setup for the organizations. Just for testing purposes, I was trying to provide access to the untrusted users.

[aidanhs]

This is essentially a feature request to allow build servers to inform the scheduler of an address that they can be connected to at, rather than (re)using the listen address. This seems reasonable.