Implement N4409 on top of HPX

[hkaiser]

Implement N3989 (http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n3989.html) on top of HPX. This finally would be the first step to expose parallel algorithms to application developers.

There is an updated version of the proposal document here (N4409): http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2015/n4409.pdf.

As always, this implementation will add HPX specific functionality. Several extensions are obvious right away:

• Add a new execution policy task_execution_policy on top of what the proposal mandates. The difference to the already described parallel_execution_policy would be that all algorithms would return a future<> representing the original result.
• Add possibility to specify HPX executors to be used with the par and task execution policies. This could be done by adding par(exec) and task(exec) as valid execution policy arguments.
• Add functionality enabling the use of all algorithms for distributed use cases (see also Extend parallel algorithms to work with hpx::partitioned_vector et.al. #1338).

Another possible extension needs more investigation:

• Allow for all algorithms to be invoked with sequences of futures themselves, where the predicates/operators are invoked only after the corresponding future became ready.

Here is the list of algorithms mandated by the proposal:

• adjacent_difference inner_product (see Implemented inner_product and adjacent_diff algos #1659)
• adjacent_find (see Remaining find algorithms implemented, N4071 #1223)
• all_of any_of none_of (see Adding all_of, any_of, and none_of and corresponding documentation #1181)
• copy copy_if copy_n (see N3960 copy_if and copy_n implemented and tested #1147)
• move
• count count_if
• equal mismatch (see Parallel equal  #1193, N4071 additional algorithms implemented #1206)
• exclusive_scan inclusive_scan (see Exclusive scan #1341, Inclusive scan #1342)
• reduce transform (see Extended parallel::reduce #1183)
• fill fill_n
• find find_end find_first_of find_if find_if_not (see N4071 additional algorithms implemented #1206, Remaining find algorithms implemented, N4071 #1223)
• for_each for_each_n
• generate generate_n (see N4071 additional algorithms implemented #1206)
• is_heap is_heap_until (see Implement parallel is_heap and is_heap_until. #2654)
• is_partitioned is_sorted is_sorted_until (see Is_sorted, Is_sorted_until and Is_partitioned Algorithms #1382)
• lexicographical_compare (see Lexicographical Compare completed #1354)
• max_element min_element minmax_element (see Parallel minmax #1231)
• make_heap (Adding parallel make_heap  #4964)
• partial_sort (Adding Francisco Tapia's implementation of partial_sort #5041)
• partial_sort_copy nth_element (Add nth_element #5592, Add partial_sort_copy and adapt partial sort to c++ 20 #5630)
• sort (see Fixing 1836, adding parallel::sort #1850)
• stable_sort (see Adding implementation of stable_sort #4803)
• partition (see Implement parallel::partition. #2778)
• partition_copy (see Implement parallel::partition_copy. #2716)
• stable_partition (see Implemented parallel::stable_partition #2345)
• remove remove_if (see Implement parallel::remove and parallel::remove_if #3086)
• remove_copy remove_copy_if (see Copy algorithms #1385)
• replace replace_copy replace_copy_if replace_if (see Adding parallel::replace etc. #1225)
• reverse reverse_copy (see Adding parallel::reverse and parallel::reverse_copy #1208)
• rotate rotate_copy (see Parallel rotate #1212)
• search search_n (see N4071 Search/Search_n finished, minor changes #1317)
• set_difference set_intersection set_symmetric_difference set_union includes (see Set operation algorithms #1410)
• inplace_merge (see Implement parallel::inplace_merge. #2978)
• merge (see Implement parallel::merge. #2833)
• swap_ranges
• uninitialized_copy uninitialized_copy_n (see Uninitialized copy #1284)
• uninitialized_fill uninitialized_fill_n (see Uninitialized copy #1284)
• uninitialized_default_construct uninitialized_default_construct_n (see Adding uninitialized_default_construct and uninitialized_default_construct_n #2669)
• uninitialized_value_construct uninitialized_value_construct_n (see Adding uninitialized_value_construct and uninitialized_value_construct_n  #2671)
• uninitialized_move uninitialized_move_n (see Adding uninitialized_move and uninitialized_move_n #2664)
• destroy destroy_n (see Adding parallel::destroy and destroy_n #2676)
• unique (see Implement parallel::unique. #2867)
• unique_copy (see Implement parallel::unique_copy. #2754)

These were added by N4310:

• transform_reduce
• transform_exclusive_scan transform_inclusive_scan (see Exclusive scan #1341, Inclusive scan #1342)

These were added to C++20:

• shift_left shift_right (Add shift_left and shift_right algorithms #5466)
• Add starts_with and ends_with algorithms (Add starts_with and ends_with algorithms #5381)

[diehlpk]

@hkaiser Could you please add a project description here https://github.com/STEllAR-GROUP/hpx/wiki/GSoC-2017-Project-Ideas

[taeguk]

I'm preparing GSoC. I have a question.
When implementing parallel algorithms, can I allocate additional memory for optimization or parallelization of algorithms?
For some algorithms, there may be differences in implementation depending on whether additional memory allocation is allowed or not.

[hkaiser]

When implementing parallel algorithms, can I allocate additional memory for optimization or parallelization of algorithms?

That's a very good and controversial question. The maximum memory requirements for the parallel algorithms are not specified, only the computational complexity. While allocating memory itself does not change the computational complexity of an algorithm, often the fact that you allocate some intermediate buffer requires more data copying which in turn may change the complexity.

So the first rule of the game is not to exceed the complexity requirements as specified (e.g. don't make an algorithm O(N) if it's supposed to be O(logN), etc.). If an additional allocation does not (indirectly) change the complexity, then please make sure (that even for large data arrays) this does not blow the memory requirements out of proportion.

Generally, I'd suggest to try to avoid memory allocations as much as possible (in the first step) and use the implementation allowing to do things without. I understand that additional allocations may improve the algorithm performance, or even it's complexity, but I'd like to make an implementation correct first before diving into possible optimizations.

I know I have not given a concrete answer to your question. I guess it's a case by case decision we'll have to make as we go.

[hkaiser]

@taeguk Most of the missing algorithms are usually implemented based on a variation of a parallel scan. We already have a handful algorithms based on a scan_partitioner (e.g. copy_if, remove_copy, inclusive_scan, etc.). I'd expect for the rest of those to be easily implementable using this very same (and already existing) scan_partitioner. I'd suggest for you to familiarize yourself with how we have implemented those existing algorithms as reusing the partitioner would significantly simplify implementing the missing algorithms.

[msimberg]

Marked inplace_merge as done because #2978 was merged.

[victor-ludorum]

Hello @hkaiser !! As Many algorithms are already implemented . But I have made one list of the unimplemented algorithms .
copy_backward
equal_range
is_permutation
lower_bound
upper_bound
move_backward
prev_permutation
next_permutation
nth_element
partition_point
pop_heap
push_heap
sort_heap
stable_sort
partial_sort

and as we have one numeric adjacent_difference (numeric algorithm) ,
These three numeric algorithms can also be implemented
accumulate
inner_product
partial_sum
As I have started learning about HPX , I have checked that these algorithms haven't been implemented yet. Is the implementation of these algorithms are not important for parallelism and concurrency ?

[hkaiser]

@victor-ludorum the algorithms listed here above are the ones specified for C++17.

accumulate, inner_product, and partial_sum are listed under a different name (reduce, transform_reduce, and inclusive_scan).

Somebody already tried to implement the heap algorithms, but that was abandoned (see #1914), feel free to revive that effort.

The algorithms related to sort are listed as not-implemented in our list above (nth_element, partial_sort, etc.), feel free to take those on.

I'm not sure if it's possible to parallelize the permutation algorithms.

copy_backwards and move_backwards can easily be implemented on top of the existing copy and move algorithms (it requires at least or bi-directional iterators, so you could wrap the given iterators into reverse_iterator), alternatively we'd need a separate (but similar to copy/move) implementation.

I don't know what is the difference between partition and partition_point.

[victor-ludorum]

Thanks @hkaiser sir !! I will definitely work on these algorithms. So numeric algorithms are already implemented . Remaining algorithms which is important can be implemented , I hope.

[hkaiser]

@fjtapia Just out of curiosity, we still have a couple of algorithms missing that are related to sorting (partial_sort, partial_sort_copy, and nth_element). Do you happen to have implementation available for those? Even some initial code would be very helpful. Our plan is to finally have all of the algorithms as specified by C++20 in place and these are the last missing pieces. Any help you could give would be most appreciated!

[fjtapia]

Hi Hartmut Glad to contact you again. I will prepare the implementation of the functions and send it to you to examine. But it will be at the end of August because in two days I am going on vacation. Please send me a list of the functions to implement. If they are single-thread or parallel, and if they have any conditions that must be taken into consideration. Regards Francisco El jue., 30 jul. 2020 a las 17:45, Hartmut Kaiser (<notifications@github.com>) escribió:

…

@fjtapia <https://github.com/fjtapia> Just out of curiosity, we still have a couple of algorithms missing that are related to sorting ( partial_sort, partial_sort_copy, and nth_element). Do you happen to have implementation available for those? Even some initial code would be very helpful. Our plan is to finally have all of the algorithms as specified by C++20 in place and these are the last missing pieces. Any help you could give would be most appreciated! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1141 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA5O5GCKZPO5VOA4QRB67UTR6GITHANCNFSM4AP742IQ> .

[hkaiser]

Hi Hartmut Glad to contact you again. I will prepare the implementation of the functions and send it to you to examine. But it will be at the end of August because in two days I am going on vacation. Please send me a list of the functions to implement. If they are single-thread or parallel, and if they have any conditions that must be taken into consideration. Regards Francisco

Francisco, thanks for getting back so quickly, and thanks for your interest in helping! As said in my initial message, we are missing the implementations for the parallel versions of the following algorithms: partial_sort, partial_sort_copy, and nth_element (I believe I would be able to derive the partial_sort_copy from a partial_sort implementation myself, if needed - if that simplifies things). Also, we can always fall back to the std library versions for the sequential algorithms, so there is no need for you to look into those.

[hkaiser]

This has finally been done! Thanks to everybody who contributed to this task!