Listing of S3 bucket content does not work when using cloudFront

[ash-dey]

I am trying to using this code for listing S3 bucket content. Works perfect when s3 content is published directly. However, in my case S3 is not configured direct for public access and is controlled via Origin Access Identity of the cloudFront.

How can I use this when cloudFront is used in front of S3?

[rufuspollock]

@AshAbc not sure here as I've never tried this route! good luck getting it working and less us know if you succeed.

[az0]

For what is worth, I am using this code with Amazon S3 and Cloudfront. However, my S3 is configured for public access.

[luismanuel001]

@AshAbc did you ever get this to work with CloudFront + Origin Access Identity?

[rusik69]

1 year later i need to get this working too:)

[rusik69]

actually it works for the top bucket directory, but doesn't work when i'm trying to access subdirectories

[mlissner]

@az0 if you have a sec to share a link to your list.html, that'd be really helpful. I'm trying to get this going too, but feel like I'm going in circles.

[shurrman]

I am also on the list of those who try to get it working with CloudFront.
I only get "Error: [object Object]" on the result page :-(
Is there any way to debug?

[anthonynguyen394]

I also tried with setting up S3 bucket + cloudfront. But using this project, it goes in a loop for me when trying to access index.html file.

[haarch]

I just got this working for a us-east-1 s3 bucket...

Some ideas:

• from the instruction page, use the 'Use Amazon S3 in website mode with URL navigation' configuration.
• put the CORS rule on the bucket per the instruction page.
• adjust the permissions on the bucket via ACL or bucket policy per the instruction page. This includes GetObject and ListBucket.
• create the CloudFront distro with a CUSTOM origin, not an S3 bucket origin. The 'origin domain name' should be '<bucket_name>.s3-website-.amazonaws.com'...don't use the dropdown list to get the bucket.
• set the default root object to 'index.html' in CloudFront.
• in your index.html, set "var BUCKET_URL = 'https://<bucket_name>.s3.amazonaws.com';", and "var S3BL_IGNORE_PATH = false;"
• wait a bit for the CloudFront cache to reset and the CloudFront distro to deploy after making changes.

I hope one or more of these ideas helps.

[oguzhanaygn]

Hi @haarch does your guideline support CloudFront + ACM ?

[haarch]

@oguzhanaygn , I'm unsure whether it works with https, as the site I used it for does not require SSL.

[dylanhitt]

For anyone that hits this problem, I found myself here today. I used a different index.html and script from this project but I don't see why it wouldn't work.

To be clear I am using a completely private bucket where the only access comes from cloudfront using OAI.

From my understanding the reason why my config didn't work is because I set the default_root value to index.html. This causes cloudfront to send all request that are `` or / to index.html meaning you will never be able to grab the result list of the s3 bucket. To get around this I removed the `default_root` object and used a lambde edge function that looks like this

function pathRewrite(uri) {
    if (uri == "/object-list") {
        return "/"
    }
    if (uri == "/" || uri == "") {
        return "/index.html"
    }
    return uri
}

exports.handler = (event, context, callback) => {
    const request = event.Records[0].cf.request;
    request.uri = pathRewrite(request.uri)
    return callback(null, request);
};

Now as I mentioned I used a different project to accomplish my goal. I think something like this:

<script type="text/javascript">
  // var S3BL_IGNORE_PATH = true;
  // var BUCKET_NAME = 'BUCKET';
  var BUCKET_URL = 'https://<cloudfront-alais>/object-list';
  // var S3B_ROOT_DIR = 'SUBDIR_L1/SUBDIR_L2/';
  // var S3B_SORT = 'DEFAULT';
  // var EXCLUDE_FILE = 'index.html';  // change to array to exclude multiple files
  // var AUTO_TITLE = true;
  // var S3_REGION = 's3'; // for us-east-1
</script>

would probably do the trick.. Hopefully this helps someone. This is by no means a clean solution

🤷

[freefly42]

I solved this using an origin response lambda function that embeds the directory listing as variables in the HTML, and editing the javascript to use the built in variables rather than reading them from the bucket.

const vers = process.env.AWS_LAMBDA_FUNCTION_VERSION;
const debug = true;
const S3 = require('aws-sdk').S3;
var s3 = new S3({apiVersion: '2006-03-01'});

// Looks up the bucket to list based on the Cloudfront instance
function getS3BucketParams(cf){
    var name = '';
    var path = '';
    switch (cf.config.distributionId) {
            
        case 'CAAFAFYUYLSGS':  //other.example.com
            name = 'other-bucket';
            break;
        
        case 'XXFAFAVHAFDAA':   //whatever.example.com
            name = 'example-bucket';
            break;
                
        default:
            return undefined;
    }
    var bucketRoot = '';
    if (path != '') bucketRoot = path + '/';
    path += cf.request.uri.replace(/\/index.js$/,"/");
    path = path.replace(/\/index.html$/,"/");
    path = path.replace(/\/$/,"") + '/';  //ensure it ends with a /
    path = path.replace(/^\//,"");  //ensure it doesn't start with a /
    var params = {
        Bucket: name,
        Prefix: path,
        Delimiter: '/'
        //MaxKeys: 50
    };
            
    //if (debug) console.log("getS3BucketData returning ", params);
    return { params: params, bucketRoot: bucketRoot};
}//function getS3BucketParams(cf)

function genBody(cf, bucketList, config){
    return '<!DOCTYPE html><html><head><title>S3 Bucket Listing Generator</title></head>\n'
             + '<body><div id="navigation"></div><div id="listing"></div>\n'
             + '<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>\n'
             + '<script type="text/javascript">\n'
             + '// var S3BL_IGNORE_PATH = true;\n'
             + 'var DIRLIST_VERSION = ' + vers + ';\n'
             + 'var S3B_ROOT_DIR = \'' + config.bucketRoot +'\';\n'
             + 'var S3B_SORT = \'DEFAULT\';\n'
             + 'var AUTO_TITLE = true;\n'
             + 'var S3B_DATA = ' + JSON.stringify(bucketList) + ';\n'
             + '</script><script type="text/javascript" src="https://js.example.com/scripts/list.js"></script>\n'
             + '</body></html>';
}//genBody(cf, bucketList)

exports.handler = async (event, context, callback) => {
    var cf = event.Records[0].cf;
    let bucketList;
    
    cf.response.headers['dirlist-version'] = [{key: 'Dirlist-Version', value: vers}];
    if (((cf.response.status == 403) || (cf.response.status == 404))
            && (cf.request.uri.endsWith("/index.html") || cf.request.uri.endsWith("/") || cf.request.uri.endsWith("/index.js"))) {
        var config = getS3BucketParams(cf);
        var returnStatus = 'unprocessed';
        if (config == undefined){
            if (debug) console.log('no bucuket configured for ' + cf.config.distributionId);
        }
        else {
            try {
                bucketList = await s3.listObjectsV2(config.params).promise();
                returnStatus = 'received';
                //if (debug) console.log("bucketList received:  ", bucketList);
                cf.response.status = 200;
                cf.response.statusDescription = 'OK';
                cf.response.body = genBody(cf, bucketList, config);
            } catch (e) {
                returnStatus = 'error';
                cf.response.body = '<body><pre>params:  ' + JSON.stringify(config.params) + '\n'
                    + 'cf.config.distributionId' + cf.config.distributionId + '\n'
                    + e.stack + '</pre></body>';
                cf.response.headers['error-message'] = [{key: 'Error-Message', value: e.message}];
                console.log("listObjectsV2.error ", e.stack); // an error occurred
            }
            if (debug) console.log("bucketList:  ", bucketList);
            cf.response.headers['content-type'] = [{key: 'Content-Type', value: 'text/html'}];
            cf.response.headers['return-status'] = [{key: 'Return-Status', value: returnStatus}];
        }
    }
    else {
        if (debug) {
            console.log("response " + cf.response.status);
            console.log("request.uri " + cf.request.uri);
        }
    }

    callback(null, cf.response);
};

[isuftin]

Just got this working via CloudFront + OAI with no changes to the JS besides setting the variables.

var S3BL_IGNORE_PATH = true;
var BUCKET_URL = 'https://my.cloudfront.domain';

In CloudFront, I empty out "Default root object". Under behavior I switch to Legacy Cache Settings and choose "All" for Query Strings since CloudFront by default will cut off query strings to the S3 server.

Finally, under Origin I set "my-bucket.s3-us-west-2.amazonaws.com", choose OAI.

That's about it.