Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add file locking to support parallel runs. #427

Merged
merged 34 commits into from
Jan 28, 2023
Merged

Add file locking to support parallel runs. #427

Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
da8ba91
Add file locking to support parallel runs.
PercentBoat4164 Nov 27, 2022
6abda4b
Fixed formatting.
PercentBoat4164 Nov 28, 2022
218c079
Prevent double locking file.
PercentBoat4164 Nov 28, 2022
ea6411f
Fix SegFault from test 2.
PercentBoat4164 Nov 28, 2022
665ef68
Remove unnecessary debugging messages.
PercentBoat4164 Nov 28, 2022
f4ad1d6
Lock the package directory rather than the cache directory.
PercentBoat4164 Nov 29, 2022
d657584
Lock the version specific cache entry rather than the package specifi…
PercentBoat4164 Nov 29, 2022
773228e
Remove unnecessary arguments in conditional statements.
PercentBoat4164 Nov 29, 2022
3ecd96a
Change back to locking entire cache directory.
PercentBoat4164 Nov 29, 2022
83a4a51
Only check CPM_HAS_CACHE_LOCK.
PercentBoat4164 Nov 29, 2022
b2fba4e
Lock on a per-package basis rather than the entire cache.
PercentBoat4164 Dec 1, 2022
9d72172
Clean up the locked file.
PercentBoat4164 Dec 1, 2022
2250917
Unlock then remove to fix Windows.
PercentBoat4164 Dec 1, 2022
9f1747f
Merge branch 'cpm-cmake:master' into master
PercentBoat4164 Dec 6, 2022
9f4a952
Specify use of cmake.lock as the lock file.
PercentBoat4164 Dec 6, 2022
9660df6
Merge branch 'master' into master
TheLartians Jan 10, 2023
5ec1db4
Merge branch 'cpm-cmake:master' into master
PercentBoat4164 Jan 10, 2023
579f272
- Changed CPM_HAS_CACHE_LOCK to ${CPM_ARGS_NAME}_CPM_HAS_CACHE_LOCK.
PercentBoat4164 Jan 10, 2023
85f3ebb
Add unit test.
PercentBoat4164 Jan 10, 2023
70e4f57
Actually test if resulting git cache is clean in unit test.
PercentBoat4164 Jan 11, 2023
b2eb097
- Added comments
PercentBoat4164 Jan 11, 2023
937abb1
convert parallelism test to integration test
TheLartians Jan 24, 2023
a5a33b8
remove comment
TheLartians Jan 24, 2023
82c05f5
- Removed now unnecessary variable.
PercentBoat4164 Jan 25, 2023
0397a72
Forgot to change variable name.
PercentBoat4164 Jan 25, 2023
dd2ef70
Add similar changes to the missed section.
PercentBoat4164 Jan 25, 2023
0bc1f2d
Fixed formatting.
PercentBoat4164 Jan 25, 2023
c031a3d
Unlock the file, but do not delete it.
PercentBoat4164 Jan 25, 2023
88592ef
Only unlock the file if it exists.
PercentBoat4164 Jan 25, 2023
42ef7b6
Changed cache.cmake test to ignore non-directory entries.
PercentBoat4164 Jan 25, 2023
bc00dd7
Integration test lib make_project:
iboB Jan 26, 2023
dce53a1
- Moved checks to function.
PercentBoat4164 Jan 26, 2023
5789a87
- Fix formatting
PercentBoat4164 Jan 26, 2023
dffafed
Switch to snake case.
PercentBoat4164 Jan 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions cmake/CPM.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -712,6 +712,12 @@ function(CPMAddPackage)
# relative paths.
get_filename_component(download_directory ${download_directory} ABSOLUTE)
list(APPEND CPM_ARGS_UNPARSED_ARGUMENTS SOURCE_DIR ${download_directory})

if(NOT ${CPM_ARGS_NAME}_CPM_HAS_CACHE_LOCK)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't this (local) variable useless since you are always going to lock and release the file anyways ?
Besides this doesnt need to be done when clones are operated into different directories (i.e. when CPM_SOURCE_CACHE is not used, which could be a somewhat decent condition to check ?).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It looks like these variables are useless. The reason it exists is that in the past I had come across an error that occurred because the point of execution entered the locked block then left it and went back into it without going past the unlocking line. This would cause an error along the lines of file already locked! to occur. The local variable was used to solve that problem. I just tried it again without the ${CPM_ARGS_NAME}_CPM_HAS_CACHE_LOCK and it succeeded, so I will happily remove the checks.

As for the CPM_SOURCE_CACHE check, a while ago I moved all of my changes to within a block that checked for this. see line 700. If this is not a good place for it, I can move it elsewhere and perform my own checks.

file(LOCK ${download_directory}/../cmake.lock)
set(${CPM_ARGS_NAME}_CPM_HAS_CACHE_LOCK ON)
endif()

if(EXISTS ${download_directory})
cpm_store_fetch_properties(
${CPM_ARGS_NAME} "${download_directory}"
Expand Down Expand Up @@ -790,6 +796,12 @@ function(CPMAddPackage)
cpm_get_fetch_properties("${CPM_ARGS_NAME}")
endif()

if(${CPM_ARGS_NAME}_CPM_HAS_CACHE_LOCK)
file(LOCK ${download_directory}/../cmake.lock RELEASE)
file(REMOVE ${download_directory}/../cmake.lock)
TheLartians marked this conversation as resolved.
Show resolved Hide resolved
set(${CPM_ARGS_NAME}_CPM_HAS_CACHE_LOCK OFF)
endif()

set(${CPM_ARGS_NAME}_ADDED YES)
cpm_export_variables("${CPM_ARGS_NAME}")
endfunction()
Expand Down
4 changes: 3 additions & 1 deletion test/integration/lib.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ def startup
tr("-", "_").
downcase
)
@@proj_count = 0
end
end

Expand All @@ -184,7 +185,8 @@ def make_project(template_dir = nil)
test_name = local_name
test_name = test_name[5..] if test_name.start_with?('test_')

base = File.join(cur_test_dir, test_name)
@@proj_count += 1
base = File.join(cur_test_dir, test_name + '-' + @@proj_count.to_s)
src_dir = base + '-src'

FileUtils.mkdir_p src_dir
Expand Down
24 changes: 24 additions & 0 deletions test/integration/test_parallelism.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
require_relative './lib'

class Parallelism < IntegrationTest
def setup
@cache_dir = File.join(cur_test_dir, 'cpmcache')
ENV['CPM_SOURCE_CACHE'] = @cache_dir
end

def test_populate_cache_in_parallel

[*1..4]
.map{ |i|
prj = make_project 'using-fibadder'
prj.create_lists_from_default_template package: 'CPMAddPackage("gh:cpm-cmake/testpack-fibadder@1.0.0")'
prj
}
.map{ |prj| Thread.new do
assert_success prj.configure
assert_success prj.build
end }
.map { |t| t.join }

end
end