Skip to content

Commit

Permalink
Option to maintain local cache file in Remote (#428)
Browse files Browse the repository at this point in the history
With `remote_file_list_cache_file_path` options we could be able to reduce and sync time by 30 minutes.

Maintaining the local file is difficult in the docker environment, so in this PR providing the option to have the cache file in remote (`remote_file_list_remote_path`) as well. Which will be downloaded during asset sync if path is configured and its creates a local file for further references in the sync. Once after the sync is completed, updated local cache file will be uploaded back to configured remote path. 

This remote cache file will be maintained in the path configured using `remote_file_list_remote_path` inside `fog_directory`
  • Loading branch information
PikachuEXE authored Aug 26, 2022
2 parents 7857a55 + 83b5792 commit d261cd4
Show file tree
Hide file tree
Showing 6 changed files with 42 additions and 0 deletions.
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,9 @@ AssetSync.configure do |config|
# Path to cache file to skip scanning remote
# config.remote_file_list_cache_file_path = './.asset_sync_remote_file_list_cache.json'
#
# Path on remote storage to persist remote file list file
# config.remote_file_list_remote_path = '/remote/asset_sync_remote_file.json'
#
# Fail silently. Useful for environments such as Heroku
# config.fail_silently = true
#
Expand Down Expand Up @@ -384,6 +387,7 @@ AssetSync.config.gzip_compression == ENV['ASSET_SYNC_GZIP_COMPRESSION']
* **concurrent_uploads**: (`true, false`) when enabled, will upload the files in different Threads, this greatly improves the upload speed. **default:** `'false'`
* **concurrent_uploads_max_threads**: when concurrent_uploads is enabled, this determines the number of threads that will be created. **default:** `10`
* **remote_file_list_cache_file_path**: if present, use this path to cache remote file list to skip scanning remote **default:** `nil`
* **remote_file_list_remote_path**: if present, use this path to download remote file list file to cache file list in local to skip scanning remote. useful in container environment where you cannot maintain the local file, remote_file_list_cache_file_path also needed to make use of this option **default:** `nil`
* **enabled**: (`true, false`) when false, will disable asset sync. **default:** `'true'` (enabled)
* **ignored\_files**: an array of files to ignore e.g. `['ignore_me.js', %r(ignore_some/\d{32}\.css)]` Useful if there are some files that are created dynamically on the server and you don't want to upload on deploy **default**: `[]`
* **cache\_asset\_regexps**: an array of files to add cache headers e.g. `['cache_me.js', %r(cache_some\.\d{8}\.css)]` Useful if there are some files that are added to sprockets assets list and need to be set as 'Cacheable' on uploaded server. Only rails compiled regexp is matched internally **default**: `[]`
Expand Down
3 changes: 3 additions & 0 deletions lib/asset_sync/config.rb
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ class Invalid < StandardError; end
attr_accessor :concurrent_uploads
attr_accessor :concurrent_uploads_max_threads
attr_accessor :remote_file_list_cache_file_path
attr_accessor :remote_file_list_remote_path

# FOG configuration
attr_accessor :fog_provider # Currently Supported ['AWS', 'Rackspace']
Expand Down Expand Up @@ -107,6 +108,7 @@ def initialize
self.concurrent_uploads = false
self.concurrent_uploads_max_threads = 10
self.remote_file_list_cache_file_path = nil
self.remote_file_list_remote_path = nil
@additional_local_file_paths_procs = []

load_yml! if defined?(::Rails) && yml_exists?
Expand Down Expand Up @@ -254,6 +256,7 @@ def load_yml!
self.concurrent_uploads = yml['concurrent_uploads'] if yml.has_key?('concurrent_uploads')
self.concurrent_uploads_max_threads = yml['concurrent_uploads_max_threads'] if yml.has_key?('concurrent_uploads_max_threads')
self.remote_file_list_cache_file_path = yml['remote_file_list_cache_file_path'] if yml.has_key?('remote_file_list_cache_file_path')
self.remote_file_list_remote_path = yml['remote_file_list_remote_path'] if yml.has_key?('remote_file_list_remote_path')

self.azure_storage_account_name = yml['azure_storage_account_name'] if yml.has_key?("azure_storage_account_name")
self.azure_storage_access_key = yml['azure_storage_access_key'] if yml.has_key?("azure_storage_access_key")
Expand Down
1 change: 1 addition & 0 deletions lib/asset_sync/engine.rb
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ class Engine < Rails::Engine
config.include_manifest = (ENV['ASSET_SYNC_INCLUDE_MANIFEST'] == 'true') if ENV.has_key?('ASSET_SYNC_INCLUDE_MANIFEST')
config.concurrent_uploads = (ENV['ASSET_SYNC_CONCURRENT_UPLOADS'] == 'true') if ENV.has_key?('ASSET_SYNC_CONCURRENT_UPLOADS')
config.remote_file_list_cache_file_path = ENV['ASSET_SYNC_REMOTE_FILE_LIST_CACHE_FILE_PATH'] if ENV.has_key?('ASSET_SYNC_REMOTE_FILE_LIST_CACHE_FILE_PATH')
config.remote_file_list_remote_path = ENV['ASSET_SYNC_REMOTE_FILE_LIST_REMOTE_PATH'] if ENV.has_key?('ASSET_SYNC_REMOTE_FILE_LIST_REMOTE_PATH')
end
end

Expand Down
27 changes: 27 additions & 0 deletions lib/asset_sync/storage.rb
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,10 @@ def remote_file_list_cache_file_path
self.config.remote_file_list_cache_file_path
end

def remote_file_list_remote_path
self.config.remote_file_list_remote_path
end

def ignored_files
expand_file_names(self.config.ignored_files)
end
Expand All @@ -72,6 +76,16 @@ def remote_files
return [] if ignore_existing_remote_files?
return @remote_files if @remote_files

if remote_file_list_remote_path && remote_file_list_cache_file_path
log "Downloading file list file from remote"
remote_cache_file = bucket.files.get(remote_file_list_remote_path)
if remote_cache_file
File.open(remote_file_list_cache_file_path, 'w') do |local_file|
local_file.write(remote_cache_file.body)
end
end
end

if remote_file_list_cache_file_path && File.file?(remote_file_list_cache_file_path)
begin
content = File.read(remote_file_list_cache_file_path)
Expand All @@ -94,6 +108,18 @@ def update_remote_file_list_cache(local_files_to_upload)
end
end

def update_remote_file_list_in_remote
return if ignore_existing_remote_files?
return unless remote_file_list_remote_path
return unless remote_file_list_cache_file_path
log "Updating file list file in remote"
remote_file = bucket.files.new({
:key => remote_file_list_remote_path,
:body => File.open(remote_file_list_cache_file_path)
})
remote_file.save
end

def always_upload_files
expand_file_names(self.config.always_upload) + get_manifest_path
end
Expand Down Expand Up @@ -317,6 +343,7 @@ def upload_files
end

update_remote_file_list_cache(local_files_to_upload)
update_remote_file_list_in_remote
end

def sync
Expand Down
3 changes: 3 additions & 0 deletions lib/generators/asset_sync/templates/asset_sync.rb
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,9 @@
# Path to cache file to skip scanning remote
# config.remote_file_list_cache_file_path = './.asset_sync_remote_file_list_cache.json'
#
# Path on remote storage to persist remote file list file
# config.remote_file_list_remote_path = '/remote/asset_sync_remote_file.json'
#
# Fail silently. Useful for environments such as Heroku
# config.fail_silently = true
#
Expand Down
4 changes: 4 additions & 0 deletions spec/unit/asset_sync_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,10 @@
it "should default asset_regexps to empty array" do
expect(AssetSync.config.cache_asset_regexps).to eq([])
end

it "should default remote_file_list_remote_path to nil" do
expect(AssetSync.config.remote_file_list_remote_path).to be_nil
end
end

describe 'from yml' do
Expand Down

0 comments on commit d261cd4

Please sign in to comment.