Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unconstrained Refinement #160

Merged
merged 152 commits into from
Sep 19, 2023
Merged
Show file tree
Hide file tree
Changes from 144 commits
Commits
Show all changes
152 commits
Select commit Hold shift + click to select a range
a972b3b
refactor gain computation for use by jet refiner
N-Maas May 12, 2023
856da31
jet context and implementation first phase
N-Maas May 11, 2023
6caf0fb
command line / factories integration
N-Maas May 11, 2023
d8810e7
rebalancing and rollback
N-Maas May 11, 2023
33c0f46
support for randomized vertex locking
N-Maas May 12, 2023
8046690
rework jet refiner to use multiple internal rounds
N-Maas May 15, 2023
a6f58be
jet refiner fixes
N-Maas Jul 27, 2023
c998bdc
use factory for rebalancer
N-Maas May 16, 2023
3d17dcc
jet rebalancer initial implementation and tests
N-Maas May 23, 2023
b9483b6
register jet balancer, use global rebalancer for jet refiner
N-Maas May 25, 2023
ef19c63
improved GraphToHgr tool
N-Maas May 25, 2023
35e9cf5
initial implementation of unconstrained fm strategy
N-Maas Aug 29, 2023
c33a672
fixes for merge to master
N-Maas Aug 29, 2023
af10979
unconstrained FM strategy and integration to kway FM
N-Maas May 31, 2023
a359d32
refactoring for shared data
N-Maas Jun 1, 2023
b64a361
edge case
N-Maas Jun 5, 2023
c7eb45c
add even more functionality to unconstrained data
N-Maas Jun 5, 2023
3253601
implement global rollback for unconstrained FM
N-Maas Jun 6, 2023
2324016
minor
N-Maas Jun 6, 2023
1fa9cf4
optimization: precompute incident weights
N-Maas Jun 6, 2023
20d6ee8
minor
N-Maas Jun 15, 2023
882c9ce
introduce IRebalancer interface and implement it for JetRebalancer
N-Maas Jun 6, 2023
1603327
add option to allow part weight scaling as in global rollback
N-Maas Jun 6, 2023
c46d04e
implement interleaving of move sequence
N-Maas Jun 14, 2023
bb690c2
apply strategy for interleaved rebalancing moves in kway fm
N-Maas Jun 14, 2023
273265b
jet rebalancer return moves
N-Maas Jun 14, 2023
61de936
add cooling fm algorithm (unconstrained with increasing penalty)
N-Maas Jun 15, 2023
e8229b9
add options for including border nodes
N-Maas Jun 15, 2023
62b656e
JET: option for fixed number of rounds and rollback after each iteration
N-Maas Jun 20, 2023
704047b
special case for penalty 0 in unconstrained FM
N-Maas Jun 20, 2023
a4d226c
implement FM vertex locking
N-Maas Jun 21, 2023
c3f94cd
add upper bound option (replaces or supplements penalty) for unconstr…
N-Maas Jun 22, 2023
ac1dcab
option for only appending rebalancing moves
N-Maas Jun 26, 2023
dcc01a9
fix for direct rollback in jet refiner: correctly compute active nodes
N-Maas Jun 26, 2023
80205b1
swap jet and label propagation so that jet is executed first
N-Maas Jun 26, 2023
64729a5
implement dynamic activation/deactivation of unconstrained FM
N-Maas Jun 26, 2023
2e8f989
Compile Time: reduce template usage in jet refiner
N-Maas Jun 28, 2023
c0764ba
minor: fix rebalancer test
N-Maas Jun 28, 2023
16efa66
remove approximate rollback strategy
N-Maas Jun 28, 2023
88bd7b5
compile time: de-template the FM strategies, call with dynamic dispatch
N-Maas Jun 28, 2023
5c99ac3
upper bound option for jet refiner
N-Maas Jun 29, 2023
7d2ac82
fix for skipping moves
N-Maas Jun 30, 2023
180e3db
allow to correctly update the penalty for locally reverted moves
N-Maas Jun 30, 2023
95cb724
disambiguate timing keys
N-Maas Jul 7, 2023
00df782
geometric interpolation in cooling fm
N-Maas Jul 7, 2023
f02462d
recompute penalties before rebalancing in FM
larsgottesbueren Jul 11, 2023
6fe20f8
let jet refiner perform gain cache updates
larsgottesbueren Jul 11, 2023
0da85c8
[Parallel Gain Recalculation] establish the zero-init requirement (ra…
larsgottesbueren Jun 21, 2023
a9af5f4
bootstrap new rebalancer
larsgottesbueren Jun 20, 2023
713cf97
add parallel rebalancer implementation
larsgottesbueren Jul 10, 2023
032b326
add pcg random number generator code
larsgottesbueren Jul 10, 2023
db8b1d3
fixes to make rebalancer_v2 work with jet refiner
larsgottesbueren Jul 11, 2023
3639f68
move impl details to impl namespace in rebalancer_v2
larsgottesbueren Jul 11, 2023
a406a7d
remove old debug output and fix bug
larsgottesbueren Jul 12, 2023
7004109
fix warning in pcg lib
larsgottesbueren Jul 13, 2023
3e9f0e1
fix mem ordering in work container test
larsgottesbueren Jul 13, 2023
bf73d0c
refactor parallel rebalancer. can now retain data structures to avoid…
larsgottesbueren Jul 14, 2023
7f47eb4
rebalancer: use new sync_update struct
N-Maas Jul 27, 2023
135096a
feed num nodes to the rebalancer
larsgottesbueren Jul 14, 2023
66c5a09
fix fm strategy test
larsgottesbueren Jul 14, 2023
2286e54
fix compilation jet_rebalancer_test
larsgottesbueren Jul 14, 2023
e805dbd
fix compilation jet_refiner_test
larsgottesbueren Jul 14, 2023
ee15e38
fix compilation multitry_fm_test
larsgottesbueren Jul 14, 2023
64d3762
reset top key of PQs
larsgottesbueren Jul 14, 2023
acf4fe7
switch to ds:Array instead of vec for large data structures to benefi…
larsgottesbueren Jul 14, 2023
5ce78ec
JET: option for rollback after each iteration
N-Maas Jun 20, 2023
682004e
add HgrToGraph tool
N-Maas Jul 11, 2023
2999995
Remove some parameters from Unconstrained FM that are not needed anymore
N-Maas Jul 12, 2023
197fddf
Revert "upper bound option for jet refiner"
N-Maas Jul 12, 2023
6f99c89
minor
N-Maas Jul 13, 2023
10cdd6d
implement unconstrained LP as variant of the LP refiner
N-Maas Jul 13, 2023
14ed735
add option to match jet implementation from paper more exactly
N-Maas Jul 17, 2023
d9b332a
add matrix market to graph converter tool
N-Maas Aug 29, 2023
a9cab51
another fix for merge to master
N-Maas Aug 29, 2023
34c9036
report stats regarding dynamic UFM activation
N-Maas Jul 20, 2023
2dd497d
cast before divide -.-
larsgottesbueren Jul 19, 2023
06ee1c3
some cleanup
N-Maas Jul 26, 2023
ecccbe9
rework how bucket weights / virtual weight is tracked
N-Maas Jul 26, 2023
4b2448c
remove some unnecessary options
N-Maas Jul 27, 2023
2d536d3
fix jet "afterburning" ...
N-Maas Jul 28, 2023
4e729d1
dumbo bug
larsgottesbueren Jul 31, 2023
df39f55
fix zero degree node bug
larsgottesbueren Aug 1, 2023
1a2c652
Revert "rebalancing and rollback"
N-Maas Aug 30, 2023
12fc59c
remove jet refiner
N-Maas Aug 29, 2023
c73972b
remove jet refiner parameters
N-Maas Aug 29, 2023
8ae89de
remove jet rebalancer
N-Maas Aug 29, 2023
d5f86f8
remove jet rebalancer parameters
N-Maas Aug 29, 2023
2a2cdf5
remove unnecessary parameter for unconstrained LP
N-Maas Aug 30, 2023
5caeb83
use :: for constexpr check (. notation seems to fail with older compi…
N-Maas Aug 30, 2023
27aaa0a
fix FM strategy and LP tests
N-Maas Aug 30, 2023
58f5930
fix multitry fm test
N-Maas Aug 30, 2023
858b569
formatting
N-Maas Aug 30, 2023
4b4721b
more LP test cases
N-Maas Aug 30, 2023
3ef9746
fix for changeNumberOfBlocks
N-Maas Aug 30, 2023
941e740
note some TODOs
N-Maas Aug 29, 2023
0a277dc
remove combined fm strategy
N-Maas Aug 30, 2023
7ab4329
remove geometric interpolation parameter
N-Maas Aug 30, 2023
b66b531
add fixed vertex support to the rebalancer
larsgottesbueren Aug 30, 2023
53fd49c
remove only-append-rebalancing-moves option
N-Maas Aug 30, 2023
6c09c66
remove code for impossible cases in localized FM
N-Maas Aug 30, 2023
d8d1504
remove unconstrained strategy, rename cooling to unconstrained
N-Maas Aug 30, 2023
2f83938
fix silly bug from refactoring
N-Maas Aug 31, 2023
a61e464
extend FM test
N-Maas Aug 31, 2023
01f528e
fix for rebalancer
N-Maas Aug 31, 2023
4ad1a8b
resolve issues with gain cache updates
N-Maas Aug 31, 2023
ef77456
resolve more gain cache issues
N-Maas Aug 31, 2023
b04d998
fix gain recalculation edge case with invalidated moves (and add asse…
N-Maas Aug 31, 2023
4a1ab97
extend sql_plottools output and remove one more parameter
N-Maas Aug 31, 2023
b04b5f6
remove set max part weights for round function
larsgottesbueren Aug 31, 2023
5ced03c
[Rebalancer] replace (potentially modified) max part weights with tho…
larsgottesbueren Sep 1, 2023
573cff9
don't include more invalid moves then necessary
N-Maas Sep 1, 2023
9eb00eb
avoid additions by computing a prefix sum at initialization
N-Maas Sep 1, 2023
e6b2389
Revert changes to concurrent bucket map
N-Maas Sep 1, 2023
0fe6b8d
minor
N-Maas Sep 1, 2023
b49a19e
minor
N-Maas Sep 1, 2023
a441c2d
unconstrained data minor
N-Maas Sep 1, 2023
1237d55
implement fallback for buckets
N-Maas Sep 1, 2023
0943608
actually use the fallback (+ test)
N-Maas Sep 4, 2023
1a6c8a7
re-enable (modified) test for changing number of blocks
N-Maas Sep 4, 2023
8e8087a
[rebalancer] check for invalid target part
N-Maas Sep 4, 2023
770ee87
remove old TODOs
larsgottesbueren Sep 4, 2023
50c3b4d
remove power of two choices gain balancing during insertion
larsgottesbueren Sep 4, 2023
ff41863
[rebalancer] don't insert nodes that can't move
larsgottesbueren Sep 4, 2023
2bc6d42
fix typo
larsgottesbueren Sep 4, 2023
f423340
[rebalancer] remove nodes with invalid target part from PQs
larsgottesbueren Sep 4, 2023
f18b989
make gain cache usable in bucket initialization
N-Maas Sep 4, 2023
c8209cc
use gain cache for bucket initialization
N-Maas Sep 5, 2023
0781e22
[LP] fix gain cache updates & refactoring
N-Maas Sep 5, 2023
28884a8
fix FM test
N-Maas Sep 5, 2023
01445f7
use gain cache constant to determine whether entries must be recomputed
N-Maas Sep 7, 2023
314c74b
minor
N-Maas Sep 7, 2023
ac76973
minor
N-Maas Sep 7, 2023
d4e6a0a
Revert "refactor gain computation for use by jet refiner"
N-Maas Sep 7, 2023
b30c8a6
update includes/license
N-Maas Sep 7, 2023
73ba8ca
[rebalancer] allow to return moves in single vector
N-Maas Sep 8, 2023
8810fa5
[LP] final refactoring: compile times
N-Maas Sep 8, 2023
facb8be
remove old rebalancer
N-Maas Sep 7, 2023
ac66406
test fixes
N-Maas Sep 11, 2023
9966303
[LP] use lazy initialization for n-level/localized variant (don't sca…
N-Maas Sep 11, 2023
b915a25
cleanup TODOs
N-Maas Sep 13, 2023
755ff05
use unconstrained refinement in default and quality preset
N-Maas Sep 13, 2023
0f1cb8e
minor fixes
N-Maas Sep 13, 2023
4097024
update readme
N-Maas Sep 13, 2023
14ee858
fix windows build
N-Maas Sep 13, 2023
b42dce3
small changes from review
N-Maas Sep 18, 2023
6a6804c
split files for fm strategies and i_rebalancer
N-Maas Sep 18, 2023
43a0a0f
restore old rebalancer
N-Maas Sep 18, 2023
a28240c
rename rebalancers
N-Maas Sep 18, 2023
f1197ae
add tests for new rebalancer
N-Maas Sep 18, 2023
cd719d6
fix
N-Maas Sep 18, 2023
4460e6e
minor
N-Maas Sep 18, 2023
41003aa
[rebalancer] don't assume phg.k() == context.partition.k
N-Maas Sep 19, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -147,6 +147,7 @@ include_directories(SYSTEM ${gtest_SOURCE_DIR}/../googlemock/include ${gtest_SOU
include_directories(external_tools/kahypar-shared-resources)
include_directories(external_tools/growt)
include_directories(external_tools/WHFC)
include_directories(external_tools/pcg)

if(KAHYPAR_DOWNLOAD_BOOST)
# Download Boost
Expand Down
17 changes: 14 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -489,7 +489,7 @@ If you use Mt-KaHyPar in an academic setting please cite the appropriate papers.
@inproceedings{MT-KAHYPAR-SDET,
author = {Lars Gottesb{\"{u}}ren and
Michael Hamann},
title = {{Deterministic Parallel Hypergraph Partitioning}},
title = {Deterministic Parallel Hypergraph Partitioning},
booktitle = {European Conference on Parallel Processing (Euro-Par)},
volume = {13440},
pages = {301--316},
Expand All @@ -498,6 +498,17 @@ If you use Mt-KaHyPar in an academic setting please cite the appropriate papers.
doi = {10.1007/978-3-031-12597-3\_19},
}

// Unconstrained Refinement (Under Review)
@article{MT-KAHYPAR-UNCONSTRAINED,
title = {Parallel Unconstrained Local Search for Partitioning Irregular Graphs},
author = {Nikolai Maas and
Lars Gottesb{\"{u}}ren and
Daniel Seemaier},
institution = {Karlsruhe Institute of Technology},
year = {2023},
url = {https://arxiv.org/abs/2308.15494}
}

// Dissertation of Lars Gottesbüren
@phdthesis{MT-KAHYPAR-DIS-GOTTESBUEREN,
author = {Lars Gottesb\"{u}ren},
Expand All @@ -523,11 +534,11 @@ If you use Mt-KaHyPar in an academic setting please cite the appropriate papers.
title = {Scalable High-Quality Hypergraph Partitioning},
author = {Lars Gottesb{\"u}ren and
Tobias Heuer and
Nikolai Mass and
Nikolai Maas and
Peter Sanders and
Sebastian Schlag},
institution = {Karlsruhe Institute of Technology},
year = {2023}
year = {2023},
url = {https://arxiv.org/abs/2303.17679}
}

Expand Down
15 changes: 11 additions & 4 deletions config/default_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -57,22 +57,29 @@ i-r-fm-iter-moves-on-recalc=true
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=false
# main -> refinement -> label_propagation
r-lp-type=label_propagation
r-lp-unconstrained=true
r-lp-maximum-iterations=5
r-lp-rebalancing=true
r-lp-rebalancing=false
r-lp-he-size-activation-threshold=100
r-lp-relative-improvement-threshold=0.001
# main -> refinement -> fm
r-fm-type=kway_fm
r-fm-type=unconstrained
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rename this to "unconstrained_fm"

r-fm-multitry-rounds=10
r-fm-unconstrained-rounds=8
r-fm-perform-moves-global=false
r-fm-rollback-parallel=true
r-fm-rollback-balance-violation-factor=1.25
r-fm-rollback-balance-violation-factor=1.0
r-fm-threshold-border-node-inclusion=0.7
r-fm-imbalance-penalty-min=0.2
r-fm-imbalance-penalty-max=1.0
r-fm-seed-nodes=25
r-fm-release-nodes=true
r-fm-min-improvement=-1.0
r-fm-unconstrained-min-improvement=0.002
r-fm-obey-minimal-parallelism=true
r-fm-time-limit-factor=0.25
r-fm-iter-moves-on-recalc=true
Expand Down
2 changes: 1 addition & 1 deletion config/deterministic_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ i-fm-refinement-rounds=3
i-lp-maximum-iterations=20
i-lp-initial-block-size=5
# main -> initial_partitioning -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
i-r-refine-until-no-improvement=false
# main -> initial_partitioning -> refinement -> label_propagation
i-r-lp-type=deterministic
Expand Down
2 changes: 1 addition & 1 deletion config/highest_quality_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ i-r-use-global-fm=false
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=true
r-relative-improvement-threshold=0.0025
r-max-batch-size=1000
Expand Down
2 changes: 1 addition & 1 deletion config/large_k_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ i-r-fm-type=do_nothing
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=false
# main -> refinement -> label_propagation
r-lp-type=label_propagation
Expand Down
2 changes: 1 addition & 1 deletion config/old_configs/default_flow_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ i-r-fm-iter-moves-on-recalc=true
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=true
r-relative-improvement-threshold=0.0025
# main -> refinement -> label_propagation
Expand Down
2 changes: 1 addition & 1 deletion config/old_configs/default_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ i-r-fm-iter-moves-on-recalc=true
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did we remove the "simple_rebalancer"? These configs should be not modified as they refer to the state in our dissertations.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated the PR so that it doesn't remove the simple rebalancer anymore

r-refine-until-no-improvement=false
# main -> refinement -> label_propagation
r-lp-type=label_propagation
Expand Down
2 changes: 1 addition & 1 deletion config/old_configs/quality_flow_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ i-r-use-global-fm=false
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=true
r-relative-improvement-threshold=0.0025
r-max-batch-size=1000
Expand Down
2 changes: 1 addition & 1 deletion config/old_configs/quality_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ i-r-flow-algo=do_nothing
# main -> initial_partitioning -> refinement -> global fm
i-r-use-global-fm=false
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=true
r-max-batch-size=1000
r-min-border-vertices-per-thread=50
Expand Down
13 changes: 10 additions & 3 deletions config/quality_preset.ini
Original file line number Diff line number Diff line change
Expand Up @@ -58,23 +58,30 @@ i-r-fm-iter-moves-on-recalc=true
# main -> initial_partitioning -> refinement -> flows
i-r-flow-algo=do_nothing
# main -> refinement
r-rebalancer-type=simple_rebalancer
r-rebalancer-type=rebalancer
r-refine-until-no-improvement=true
r-relative-improvement-threshold=0.0025
# main -> refinement -> label_propagation
r-lp-type=label_propagation
r-lp-unconstrained=true
r-lp-maximum-iterations=5
r-lp-rebalancing=true
r-lp-he-size-activation-threshold=100
r-lp-relative-improvement-threshold=0.001
# main -> refinement -> fm
r-fm-type=kway_fm
r-fm-type=unconstrained
r-fm-multitry-rounds=10
r-fm-unconstrained-rounds=8
r-fm-perform-moves-global=false
r-fm-rollback-parallel=true
r-fm-rollback-balance-violation-factor=1.25
r-fm-rollback-balance-violation-factor=1.0
r-fm-threshold-border-node-inclusion=0.7
r-fm-imbalance-penalty-min=0.2
r-fm-imbalance-penalty-max=1.0
r-fm-seed-nodes=25
r-fm-release-nodes=true
r-fm-min-improvement=-1.0
r-fm-unconstrained-min-improvement=0.002
r-fm-obey-minimal-parallelism=true
r-fm-time-limit-factor=0.25
r-fm-iter-moves-on-recalc=true
Expand Down
Loading
Loading