From 59b1aeca2479000b4c029364c20d6b6eb3633a81 Mon Sep 17 00:00:00 2001 From: "Documenter.jl" Date: Sun, 5 Nov 2023 19:00:09 +0000 Subject: [PATCH] build based on 7f481eb --- .../comparison_to_iterators/index.html | 2 +- dev/explanation/glossary/index.html | 2 +- dev/explanation/internals/index.html | 26 +-- dev/explanation/state_machines/index.html | 2 +- dev/howto/empty_result_handling/index.html | 2 +- dev/howto/reducibles/index.html | 2 +- dev/howto/transducers/index.html | 2 +- dev/howto/useful_patterns/index.html | 2 +- dev/index.html | 2 +- dev/parallelism/index.html | 2 +- dev/reference/interface/index.html | 16 +- dev/reference/manual/index.html | 166 +++++++++--------- dev/search/index.html | 2 +- dev/tutorials/tutorial_missings/index.html | 2 +- dev/tutorials/tutorial_parallel/index.html | 2 +- dev/tutorials/words/index.html | 8 +- 16 files changed, 120 insertions(+), 120 deletions(-) diff --git a/dev/explanation/comparison_to_iterators/index.html b/dev/explanation/comparison_to_iterators/index.html index d42249b6..bc2a7d2e 100644 --- a/dev/explanation/comparison_to_iterators/index.html +++ b/dev/explanation/comparison_to_iterators/index.html @@ -61,4 +61,4 @@ return acc end -@assert foldl(*, xf, 1:10) == map_filter_cat_transducers(1:10, 1)

It is not straightforward to implement an iterator like Cat that can output more than one items at a time. Such an iterator has to track the state of the inner (y1 in above) and outer (xs in above) iterators and conditionally invoke the outer iterator once the inner iterator terminates. This generates a complicated code and the compiler would have hard time optimizing it.

+@assert foldl(*, xf, 1:10) == map_filter_cat_transducers(1:10, 1)

It is not straightforward to implement an iterator like Cat that can output more than one items at a time. Such an iterator has to track the state of the inner (y1 in above) and outer (xs in above) iterators and conditionally invoke the outer iterator once the inner iterator terminates. This generates a complicated code and the compiler would have hard time optimizing it.

diff --git a/dev/explanation/glossary/index.html b/dev/explanation/glossary/index.html index 4af3985e..5eeba06d 100644 --- a/dev/explanation/glossary/index.html +++ b/dev/explanation/glossary/index.html @@ -18,4 +18,4 @@ \mathrm{xf}_1(\mathrm{xf}_2(...(\mathrm{xf}_{n}(\mathrm{rf}_0))))\]

which is equivalent to the following forms in Transducers.jl

rf = xf₁'(xf₂'(...(xfₙ'(rf₀))))
 rf = (xf₁' ∘ xf₂' ∘ ... ∘ xfₙ')(rf₀)
 rf = (xfₙ ∘ ... ∘ xf₂ ∘ xf₁)'(rf₀)
-rf = (xf₁ ⨟ xf₂ ⨟ ... ⨟ xfₙ)(rf₀)

Inner transducer

Given a composition xf₁' ∘ xf₂', transducer xf₂ is said to be the inner transducer of xf₁' ∘ xf₂'. Likewise, xf₂'(rf₀) is an inner reducing function of xf₁'(xf₂'(rf₀)).

Reducible collection

Reducible collection (or just Reducible): Any object that can be passed to foldl and alike is reducible. A reducible collection knows how to apply reducing function to its elements. Iterators are automatically reducible as this is the canonical fallback implementation.

Transducible process

A function that can foldxt reducible collections using transducers is a transducible process. Examples are foldl and foldxt. Find more in Transducible processes.

Executor

An executor such as SequentialEx, ThreadedEx and DistributedEx specifies the execution mechanism of a fold. These executors provide a unified mechanism for choosing underlying execution mechanism for Transducers.jl and its related packages such as Folds.jl and FLoops.jl. Typically, the API functions take an executor as the last optional argument. In addition to the executors provided by Transducers.jl (see Executors section in the manual), additional executors are provided from external packages such as FoldsThreads.jl (various thread-based executors) and FoldsCUDA.jl (a CUDA-based executor).

Transducers.jl's executor is a concept similar to KernelAbstractions.jl' device.

+rf = (xf₁ ⨟ xf₂ ⨟ ... ⨟ xfₙ)(rf₀)

Inner transducer

Given a composition xf₁' ∘ xf₂', transducer xf₂ is said to be the inner transducer of xf₁' ∘ xf₂'. Likewise, xf₂'(rf₀) is an inner reducing function of xf₁'(xf₂'(rf₀)).

Reducible collection

Reducible collection (or just Reducible): Any object that can be passed to foldl and alike is reducible. A reducible collection knows how to apply reducing function to its elements. Iterators are automatically reducible as this is the canonical fallback implementation.

Transducible process

A function that can foldxt reducible collections using transducers is a transducible process. Examples are foldl and foldxt. Find more in Transducible processes.

Executor

An executor such as SequentialEx, ThreadedEx and DistributedEx specifies the execution mechanism of a fold. These executors provide a unified mechanism for choosing underlying execution mechanism for Transducers.jl and its related packages such as Folds.jl and FLoops.jl. Typically, the API functions take an executor as the last optional argument. In addition to the executors provided by Transducers.jl (see Executors section in the manual), additional executors are provided from external packages such as FoldsThreads.jl (various thread-based executors) and FoldsCUDA.jl (a CUDA-based executor).

Transducers.jl's executor is a concept similar to KernelAbstractions.jl' device.

diff --git a/dev/explanation/internals/index.html b/dev/explanation/internals/index.html index 860641f0..17c45cd9 100644 --- a/dev/explanation/internals/index.html +++ b/dev/explanation/internals/index.html @@ -1,5 +1,5 @@ -Internals · Transducers.jl
Edit on GitHub

Internals

Transducers.DefaultInit — Constant
DefaultInit(op)

DefaultInit is like Init but strictly internal to Transducers.jl. It is used for checking if the bottom reducing function is never called.

source
Transducers.AdHocRF — Type
AdHocRF(next; oninit, start = identity, complete = identity, combine = next)

Define an ad-hoc reducing function rf.

Note

Use wheninit etc. instead of this constructor.

Arguments

  • next: binary function

Keyword Arguments

  • oninit: nullary function that generates an initial value for next
  • start: unary function that pre-process the initial value for next
  • complete: unary function that post-process the accumulator
  • combine: (approximately) associative binary function for combining multiple results of next (before post-processed by complete).

Examples

julia> using Transducers
+Internals · Transducers.jl
Edit on GitHub
Internals
Transducers.DefaultInit — Constant
DefaultInit(op)
DefaultInit is like Init but strictly internal to Transducers.jl.  It is used for checking if the bottom reducing function is never called.
source
Transducers.AdHocRF — Type
AdHocRF(next; oninit, start = identity, complete = identity, combine = next)
Define an ad-hoc reducing function rf.
Note
Use wheninit etc. instead of this constructor.
Arguments
  • next: binary function
Keyword Arguments
  • oninit: nullary function that generates an initial value for next
  • start: unary function that pre-process the initial value for next
  • complete: unary function that post-process the accumulator
  • combine: (approximately) associative binary function for combining multiple results of next (before post-processed by complete).
Examples
julia> using Transducers
        using Transducers: AdHocRF
 
 julia> rf = AdHocRF(push!, combine = append!);
@@ -9,7 +9,7 @@
  1
  2
  3
- 4
source
Transducers.AdHocXF — Type
AdHocXF(f, init, [onlast])
Examples
julia> using Transducers
        using Transducers: AdHocXF, @next
        using Setfield: @set!
 
@@ -46,7 +46,7 @@
  (name = "Map", lines = ["name: Map", "type: onetoone"])
  (name = "Cat", lines = ["name: Cat", "type: expansive"])
  (name = "Filter", lines = ["name: Filter", "type: contractive"])
- (name = "Cat |> Filter", lines = ["name: Cat |> Filter", "type: chaotic"])
source
Transducers.ReduceSplitBy — Type
Transducers.ReduceSplitBy(f, rf, init)
Split chunks by elements evaluated to true by f and reduce each chunk by reducing function rf.
Note
This is an internal implementation detail of SplitBy for now.
The reducing function rf receives either a Bulk(x) if !f(x) or a End(x) if f(x) returns true. Just just after rf is called with End(x), its accumulator is finalized by complete and then passed to the downstream transducer/reducing function.
Examples
julia> using Transducers
+ (name = "Cat |> Filter", lines = ["name: Cat |> Filter", "type: chaotic"])
source
Transducers.ReduceSplitBy — Type
Transducers.ReduceSplitBy(f, rf, init)
Split chunks by elements evaluated to true by f and reduce each chunk by reducing function rf.
Note
This is an internal implementation detail of SplitBy for now.
The reducing function rf receives either a Bulk(x) if !f(x) or a End(x) if f(x) returns true. Just just after rf is called with End(x), its accumulator is finalized by complete and then passed to the downstream transducer/reducing function.
Examples
julia> using Transducers
        using Transducers: ReduceSplitBy, Bulk, End
 
 julia> 1:5 |> ReduceSplitBy(isodd, Map(getindex)'(string), "") |> collect
@@ -67,13 +67,13 @@
 3-element Vector{String}:
  ""
  "2"
- "4"
source
Transducers.UseSIMD — Type
UseSIMD{ivdep}()
Tell the reducible to run the inner reducing function using @simd. The reducible can support it using @simd_if.
source
Transducers._set_joiner_value — Method
_set_joiner_value(ps::PrivateState, x) :: PrivateState
Set .state field of the PrivateState of the first "unbalanced" Joiner.  A Joiner matched with preceding Splitter would be treated as a regular reducing function node.  Thus, private state ps must have one more Joiner than Splitter.
source
Transducers._unzip — Method
_unzip(xs::Tuple)
Examples
julia> _unzip(((1, 2, 3), (4, 5, 6)))
-((1, 4), (2, 5), (3, 6))
source
Transducers.air — Method
air.(broadcasting_expression) :: Broadcasted
Broadcast without materialization.
The idea is taken from @dawbarton's _lazy function: https://discourse.julialang.org/t/19641/20.
source
Transducers.asfoldable — Method
Transducers.asfoldable(x) -> foldable
By default, this function does nothing, but it can be overloaded to convert an input into another type before reducing over it. This allows one to implement a foldable in terms of transducers over an existing type. For instance,
struct VectorOfVectors{T}
+ "4"
source
Transducers.UseSIMD — Type
UseSIMD{ivdep}()
Tell the reducible to run the inner reducing function using @simd. The reducible can support it using @simd_if.
source
Transducers._set_joiner_value — Method
_set_joiner_value(ps::PrivateState, x) :: PrivateState
Set .state field of the PrivateState of the first "unbalanced" Joiner.  A Joiner matched with preceding Splitter would be treated as a regular reducing function node.  Thus, private state ps must have one more Joiner than Splitter.
source
Transducers._unzip — Method
_unzip(xs::Tuple)
Examples
julia> _unzip(((1, 2, 3), (4, 5, 6)))
+((1, 4), (2, 5), (3, 6))
source
Transducers.air — Method
air.(broadcasting_expression) :: Broadcasted
Broadcast without materialization.
The idea is taken from @dawbarton's _lazy function: https://discourse.julialang.org/t/19641/20.
source
Transducers.asfoldable — Method
Transducers.asfoldable(x) -> foldable
By default, this function does nothing, but it can be overloaded to convert an input into another type before reducing over it. This allows one to implement a foldable in terms of transducers over an existing type. For instance,
struct VectorOfVectors{T}
    v::Vector{Vector{T}}
 end
 
 Transducers.asfoldable(vov::VectorOfVectors{T}) = vov.v |> Cat()
Now we can do things like
julia> foldxl(+, VectorOfVectors([[1,2], [3, 4]]))
-10
source
Transducers.combine — Method
Transducers.combine(rf::R_{X}, state_left, state_right)
This is an optional interface for a transducer.  If transducer X is stateful (i.e., wrap is used in start), it has to be able to combine the private states to support fold functions that require an associative reducing function such as foldxt. Typical implementation takes the following form:
function combine(rf::R_{X}, a, b)
+10
source
Transducers.combine — Method
Transducers.combine(rf::R_{X}, state_left, state_right)
This is an optional interface for a transducer.  If transducer X is stateful (i.e., wrap is used in start), it has to be able to combine the private states to support fold functions that require an associative reducing function such as foldxt. Typical implementation takes the following form:
function combine(rf::R_{X}, a, b)
     #   ,---- `ua` and `ub` are the private state of the transducer `X`
     #  /  ,-- `ira` and `irb` are the states of inner reducing functions
     # /  /
@@ -82,7 +82,7 @@
     irc = combine(inner(rf), ira, irb)
     uc = # somehow combine private states `ua` and `ub`
     return wrap(rf, uc, irc)
-end
See ScanEmit, etc. for real-world examples.
source
Transducers.completebasecase — Method
Transducers.completebasecase(rf, state)
Process basecase result state before merged by combine.
For example, on GPU, this function can be used to translate mutable states to immutable values for exchanging them through (un-GC-managed) memory.  See whencompletebasecase.
Note
This function is an internal experimental interface for FoldsCUDA.
source
Transducers.completebasecase — Method
Transducers.completebasecase(rf, state)
Process basecase result state before merged by combine.
For example, on GPU, this function can be used to translate mutable states to immutable values for exchanging them through (un-GC-managed) memory.  See whencompletebasecase.
Note
This function is an internal experimental interface for FoldsCUDA.
source
Transducers.extract_transducer — Method
extract_transducer(foldable) -> (xf, foldable′)
"Reverse" of eduction.
Examples
julia> using Transducers
 
 julia> double(x) = 2x;
 
@@ -94,7 +94,7 @@
 true
 
 julia> foldable == xs
-true
source
Transducers.initialize — Method
initialize(initializer, op) -> init
 initialize(init, _) -> init
Return an initial value for op.  Throw an error if initializer (e.g., Init) creates unknown initial value.
Examples
julia> using Transducers
        using Transducers: initialize
 
@@ -109,8 +109,8 @@
 julia> initialize(Init, unknown_op)
 ERROR: IdentityNotDefinedError: `init = Init` is specified but the identity element `InitialValue(op)` is not defined for
     op = unknown_op
-[...]
source
Transducers.initvalue — Method
initvalue(initializer::AbstractInitializer) -> init
-initvalue(init) -> init
Materialize the initial value if the input is an AbstractInitializer. Return the input as-is if not.
source
Transducers.is_prelude — Function
is_prelude(::T)
Return true if it is better to tail-call when the accumulator or the private state changes its type from T.
source
Transducers.issmall — Function
Transducers.issmall(reducible, basesize) :: Bool
Check if reducible collection is considered small compared to basesize (an integer).  Fold functions such as foldxt switches to sequential __foldl__ when issmall returns true.
Default implementation is amount(reducible) <= basesize.
source
Transducers.maybe_usesimd — Method
maybe_usesimd(xform, simd)
Insert UseSIMD to xform if appropriate.
Arguments
  • xform::Transducer
  • simd: false, true, or :ivdep.
Examples
julia> using Transducers
+[...]
source
Transducers.initvalue — Method
initvalue(initializer::AbstractInitializer) -> init
+initvalue(init) -> init
Materialize the initial value if the input is an AbstractInitializer. Return the input as-is if not.
source
Transducers.is_prelude — Function
is_prelude(::T)
Return true if it is better to tail-call when the accumulator or the private state changes its type from T.
source
Transducers.issmall — Function
Transducers.issmall(reducible, basesize) :: Bool
Check if reducible collection is considered small compared to basesize (an integer).  Fold functions such as foldxt switches to sequential __foldl__ when issmall returns true.
Default implementation is amount(reducible) <= basesize.
source
Transducers.maybe_usesimd — Method
maybe_usesimd(xform, simd)
Insert UseSIMD to xform if appropriate.
Arguments
  • xform::Transducer
  • simd: false, true, or :ivdep.
Examples
julia> using Transducers
        using Transducers: maybe_usesimd
 
 julia> maybe_usesimd(reducingfunction(Map(identity), right), false)
@@ -164,7 +164,7 @@
                     Reduction(
                         Map(tan),
                         BottomRF(
-                            Transducers.right)))))))
source
Transducers.retransform — Method
Transducers.retransform(rf, itr) -> rf′, itr′
Extract transformations in rf and itr and use the appropriate adjoint for better performance.
Note that the reducing function extracted from a comprehension such as (f(x) for x ∈ itr) may not be == to f because of the way generator comprehensions work in Julia. Use Iterators.map to specify an exact mapping function.
Examples
julia> using Transducers
+                            Transducers.right)))))))
source
Transducers.retransform — Method
Transducers.retransform(rf, itr) -> rf′, itr′
Extract transformations in rf and itr and use the appropriate adjoint for better performance.
Note that the reducing function extracted from a comprehension such as (f(x) for x ∈ itr) may not be == to f because of the way generator comprehensions work in Julia. Use Iterators.map to specify an exact mapping function.
Examples
julia> using Transducers
 
 julia> double(x) = 2x;
 
@@ -178,11 +178,11 @@
 true
 
 julia> rf == reducingfunction(Map(double), +)
-true
source
Transducers.simple_transduce — Method
simple_transduce(xform, step, init, coll)
Simplified version of transduce.  For simple transducers Julia may be able to emit a good code.  This function exists only for performance tuning.
source
Transducers.usesimd — Method
usesimd(rf::Reduction, xfsimd::UseSIMD)
Wrap the inner-most loop of reducing function rf with xfsimd. xfsimd is inserted after the inner-most Cat if rf includes Cat.
source
Transducers.simple_transduce — Method
simple_transduce(xform, step, init, coll)
Simplified version of transduce.  For simple transducers Julia may be able to emit a good code.  This function exists only for performance tuning.
source
Transducers.usesimd — Method
usesimd(rf::Reduction, xfsimd::UseSIMD)
Wrap the inner-most loop of reducing function rf with xfsimd. xfsimd is inserted after the inner-most Cat if rf includes Cat.
source
Transducers.whencompletebasecase — Function
whencompletebasecase(completebasecase, rf) -> rf′
 whencompletebasecase(completebasecase) -> rf -> rf′
Add completebasecase protocol to arbitrary reducing function.
The function completebasecase is used as follows in the basecase implementation of reduce as follows:
init′ = oninit()
 acc = start(init′)
 for x in collection
     acc += rf(acc, x)
 end
 result = completebasecase(acc)
-return result
The result₁ from basecase 1 and result₂ from basecase 2 are combined using combine protocol:
combine(result₁, result₂)
Note
This function is an internal experimental interface for FoldsCUDA.
source
Transducers.@default_finaltype — Macro
@default_finaltype(xf::Transducer, coll)
Infer the type of the object that would be fed into the second argument input of the bottom reducing function rf(acc, input).
See: Base.@default_eltype
source
Transducers.@simd_if — Macro
@simd_if rf for ... end
Wrap for-loop with @simd if the outer most transducer of the reducing function rf is UseSIMD.
source

+return result

The result₁ from basecase 1 and result₂ from basecase 2 are combined using combine protocol:

combine(result₁, result₂)
Note

This function is an internal experimental interface for FoldsCUDA.

source
Transducers.@default_finaltype — Macro
@default_finaltype(xf::Transducer, coll)

Infer the type of the object that would be fed into the second argument input of the bottom reducing function rf(acc, input).

See: Base.@default_eltype

source
Transducers.@simd_if — Macro
@simd_if rf for ... end

Wrap for-loop with @simd if the outer most transducer of the reducing function rf is UseSIMD.

source
diff --git a/dev/explanation/state_machines/index.html b/dev/explanation/state_machines/index.html index d3f91f1c..4940ecb2 100644 --- a/dev/explanation/state_machines/index.html +++ b/dev/explanation/state_machines/index.html @@ -48,4 +48,4 @@ y === nothing && return acc (x, istate) = y acc = rf(acc, x) -end

Note that this code snippet is only for explaining the concept. For example, it is not possible to know how many while loops to generate a priori. It also assumes that the transition of the prelude accumulator which is not the case (see, e.g., ReduceSplitBy). Instead, several __foldl__ methods implement this strategy using what we call tail-call function-barrier pattern (ref: an example).

For a previous discussion on this strategy, see also: Tail-call optimization and function-barrier -based accumulation in loops - Internals & Design - JuliaLang.

+end

Note that this code snippet is only for explaining the concept. For example, it is not possible to know how many while loops to generate a priori. It also assumes that the transition of the prelude accumulator which is not the case (see, e.g., ReduceSplitBy). Instead, several __foldl__ methods implement this strategy using what we call tail-call function-barrier pattern (ref: an example).

For a previous discussion on this strategy, see also: Tail-call optimization and function-barrier -based accumulation in loops - Internals & Design - JuliaLang.

diff --git a/dev/howto/empty_result_handling/index.html b/dev/howto/empty_result_handling/index.html index 05a012e8..5b3103a5 100644 --- a/dev/howto/empty_result_handling/index.html +++ b/dev/howto/empty_result_handling/index.html @@ -11,4 +11,4 @@ xs[1] = Init(+) xs
2-element Vector{Bool}:
  0
- 1

They can be converted into numbers also by using Integer:

Integer(Init(+))
0

or float:

float(Init(*))
1.0

This page was generated using Literate.jl.

+ 1

They can be converted into numbers also by using Integer:

Integer(Init(+))
0

or float:

float(Init(*))
1.0

This page was generated using Literate.jl.

diff --git a/dev/howto/reducibles/index.html b/dev/howto/reducibles/index.html index 77206b54..18bfde99 100644 --- a/dev/howto/reducibles/index.html +++ b/dev/howto/reducibles/index.html @@ -69,4 +69,4 @@ 2 1 2 - 3

This page was generated using Literate.jl.

+ 3

This page was generated using Literate.jl.

diff --git a/dev/howto/transducers/index.html b/dev/howto/transducers/index.html index 0ddaaacd..48e55b63 100644 --- a/dev/howto/transducers/index.html +++ b/dev/howto/transducers/index.html @@ -74,4 +74,4 @@ 4 1 2 - 5

This page was generated using Literate.jl.

+ 5

This page was generated using Literate.jl.

diff --git a/dev/howto/useful_patterns/index.html b/dev/howto/useful_patterns/index.html index c6f1df83..9de2c392 100644 --- a/dev/howto/useful_patterns/index.html +++ b/dev/howto/useful_patterns/index.html @@ -156,4 +156,4 @@ 3 1 3

When input is a row: DataTools.oncol

oncol from DataTools.jl is like ProductRF but acts on NamedTuple (as well as any Setfield.jl-compatible possibly nested objects).

using DataTools
-foldxl(oncol(a = +, b = *), [(a = 1, b = 2), (a = 3, b = 4)])
(a = 4, b = 8)

This page was generated using Literate.jl.

+foldxl(oncol(a = +, b = *), [(a = 1, b = 2), (a = 3, b = 4)])
(a = 4, b = 8)

This page was generated using Literate.jl.

diff --git a/dev/index.html b/dev/index.html index f3454456..4e733be1 100644 --- a/dev/index.html +++ b/dev/index.html @@ -27,4 +27,4 @@ 4 8 12

An efficient way to use transducers is combination with foldl. The computation is compiled down to an efficient loop you would write by hand::

julia> foldl(+, 1:6 |> Filter(iseven) |> Map(x -> 2x))
-24

For more detailed discussions on the difference to iterators, see Comparison to iterators.

List of transducers

Here is the list of pre-defined transducers:

TransducerSummary
Broadcasting()Broadcast inner reducing function over elements in the input. Roughly speaking, it transforms the inner reducing function op to op′(a, b) = op.(a, b). However, it has a better memory usage and better initial value handling.
Cat()Concatenate/flatten nested iterators.
Consecutive(size, step = size)Sliding window of width size and interval step. Yield tuples.
Count([start[, step]])Generate a sequence start, start + step, start + step + step, and so on.
Dedupe()De-duplicate consecutive items. Comparison operator which identifies duplicates can be specified by the eq parameter, which defaults to == (equal).
Drop(n)Drop first n items.
DropLast(n)Drop last n items.
DropWhile(pred)Drop items while pred returns true consecutively. It becomes a no-op after pred returns a false.
Enumerate([start[, step]])Transducer variant of Base.enumerate. The start and step arguments are optional and have the same meaning as in Count.
Filter(pred)Skip items for which pred is evaluated to false.
FlagFirst()Output (isfirst, input) where isfirst::Bool is true only for the first iteration and input is the original input.
GroupBy(key, xf::Transducer, [step = right, [init]])Group the input stream by a function key and then fan-out each group of key-value pairs to the eduction xf'(step). This is similar to the groupby relational database operation.
Interpose(sep)Interleave input items with a sep.
Iterated(f, init)Generate a sequence init, f(init), f(f(init)), f(f(f(init))), and so on.
KeepSomething(f = identity)Pass non-nothing output of f to the inner reducing step after possibly unwrapping Some.
Map(f)Apply unary function f to each input and pass the result to the inner reducing step.
MapCat(f)Concatenate output of f which is expected to return an iterable.
MapSplat(f)Like Map(f) but calls f(input...) for each input and then pass the result to the inner reducing step.
NondeterministicThreading(; basesize, ntasks = nthreads())Parallelize inner reducing function using ntasks.
NotA(T)Skip items of type T. Unlike Filter(!ismissing), downstream transducers can have a correct type information for NotA(Missing).
OfType(T)Include only items of type T.
Partition(size, step = size, flush = false)Sliding window of width size and interval step. Yield vectors.
PartitionBy(f)Group input sequence into chunks in which f returns a same value consecutively.
ReduceIf(pred)Stop fold when pred(x) returns true for the output x of the upstream transducer.
ReducePartitionBy(f, rf, [init])Reduce partitions determined by isequal on the output value of f with an associative reducing function rf. Partitions are reduced on-the-fly and no intermediate arrays are allocated.
Replace(assoc)Replace each input with the value in the associative container assoc (e.g., a dictionary, array, string) if it matches with a key/index. Otherwise output the input as-is.
Scan(f, [init = Init])Accumulate input with binary function f and pass the accumulated result so far to the inner reduction step.
ScanEmit(f, init[, onlast])Accumulate input x with a function f with the call signature (u, x) -> (y, u) and pass the result y to the inner reduction step.
TCat(basesize::Integer)Threaded version of Cat (concatenate/flatten).
Take(n)Take n items from the input sequence.
TakeLast(n)Take last n items from the input sequence.
TakeNth(n)Output every n item to the inner reducing step.
TakeWhile(pred)Take items while pred returns true. Abort the reduction when pred returns false for the first time.
Unique(by = identity)Pass only unseen item to the inner reducing step.
Zip(xforms...)Zip outputs of transducers xforms in a tuple and pass it to the inner reduction step.
+24

For more detailed discussions on the difference to iterators, see Comparison to iterators.

List of transducers

Here is the list of pre-defined transducers:

TransducerSummary
Broadcasting()Broadcast inner reducing function over elements in the input. Roughly speaking, it transforms the inner reducing function op to op′(a, b) = op.(a, b). However, it has a better memory usage and better initial value handling.
Cat()Concatenate/flatten nested iterators.
Consecutive(size, step = size)Sliding window of width size and interval step. Yield tuples.
Count([start[, step]])Generate a sequence start, start + step, start + step + step, and so on.
Dedupe()De-duplicate consecutive items. Comparison operator which identifies duplicates can be specified by the eq parameter, which defaults to == (equal).
Drop(n)Drop first n items.
DropLast(n)Drop last n items.
DropWhile(pred)Drop items while pred returns true consecutively. It becomes a no-op after pred returns a false.
Enumerate([start[, step]])Transducer variant of Base.enumerate. The start and step arguments are optional and have the same meaning as in Count.
Filter(pred)Skip items for which pred is evaluated to false.
FlagFirst()Output (isfirst, input) where isfirst::Bool is true only for the first iteration and input is the original input.
GroupBy(key, xf::Transducer, [step = right, [init]])Group the input stream by a function key and then fan-out each group of key-value pairs to the eduction xf'(step). This is similar to the groupby relational database operation.
Interpose(sep)Interleave input items with a sep.
Iterated(f, init)Generate a sequence init, f(init), f(f(init)), f(f(f(init))), and so on.
KeepSomething(f = identity)Pass non-nothing output of f to the inner reducing step after possibly unwrapping Some.
Map(f)Apply unary function f to each input and pass the result to the inner reducing step.
MapCat(f)Concatenate output of f which is expected to return an iterable.
MapSplat(f)Like Map(f) but calls f(input...) for each input and then pass the result to the inner reducing step.
NondeterministicThreading(; basesize, ntasks = nthreads())Parallelize inner reducing function using ntasks.
NotA(T)Skip items of type T. Unlike Filter(!ismissing), downstream transducers can have a correct type information for NotA(Missing).
OfType(T)Include only items of type T.
Partition(size, step = size, flush = false)Sliding window of width size and interval step. Yield vectors.
PartitionBy(f)Group input sequence into chunks in which f returns a same value consecutively.
ReduceIf(pred)Stop fold when pred(x) returns true for the output x of the upstream transducer.
ReducePartitionBy(f, rf, [init])Reduce partitions determined by isequal on the output value of f with an associative reducing function rf. Partitions are reduced on-the-fly and no intermediate arrays are allocated.
Replace(assoc)Replace each input with the value in the associative container assoc (e.g., a dictionary, array, string) if it matches with a key/index. Otherwise output the input as-is.
Scan(f, [init = Init])Accumulate input with binary function f and pass the accumulated result so far to the inner reduction step.
ScanEmit(f, init[, onlast])Accumulate input x with a function f with the call signature (u, x) -> (y, u) and pass the result y to the inner reduction step.
TCat(basesize::Integer)Threaded version of Cat (concatenate/flatten).
Take(n)Take n items from the input sequence.
TakeLast(n)Take last n items from the input sequence.
TakeNth(n)Output every n item to the inner reducing step.
TakeWhile(pred)Take items while pred returns true. Abort the reduction when pred returns false for the first time.
Unique(by = identity)Pass only unseen item to the inner reducing step.
Zip(xforms...)Zip outputs of transducers xforms in a tuple and pass it to the inner reduction step.
diff --git a/dev/parallelism/index.html b/dev/parallelism/index.html index 284c0657..a30b260c 100644 --- a/dev/parallelism/index.html +++ b/dev/parallelism/index.html @@ -1,2 +1,2 @@ -Parallelism · Transducers.jl
Edit on GitHub

Overview of parallel processing in Transducers.jl

Transducers.jl supports thread-based (foldxt) and process-based (foldxd) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing foldl is useful. Using multiple cores or machines for your computation is as easy as replacing foldl with foldxt or foldxd; you don't need to re-write your transducers or reducing functions.

See also:

Thread-based parallelism

Note

To use multiple threads, the julia process must be started with appropriate environment variable JULIA_NUM_THREADS. See Julia manual for more information about how to enable and verify multi-threading in Julia:

Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing foldl with foldxt. With Julia ≥ 1.3, Transducers.jl supports early termination to avoid unnecessary computation while guaranteeing the result to be deterministic; i.e., it does not depend on how computation tasks are scheduled.

Process-based parallelism

Transducers.jl supports process-based parallelism using Distributed.jl. You can use it by replacing foldl with foldxd. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not "thread-safe."

Note that early termination is not supported in foldxd yet.

+Parallelism · Transducers.jl
Edit on GitHub

Overview of parallel processing in Transducers.jl

Transducers.jl supports thread-based (foldxt) and process-based (foldxd) parallelisms with the same composable API; i.e. transducers. Having a uniform API to cover different parallelisms as well as sequential processing foldl is useful. Using multiple cores or machines for your computation is as easy as replacing foldl with foldxt or foldxd; you don't need to re-write your transducers or reducing functions.

See also:

Thread-based parallelism

Note

To use multiple threads, the julia process must be started with appropriate environment variable JULIA_NUM_THREADS. See Julia manual for more information about how to enable and verify multi-threading in Julia:

Transducers.jl supports thread-based parallelism for Julia ≥ 1.0. You can use it by replacing foldl with foldxt. With Julia ≥ 1.3, Transducers.jl supports early termination to avoid unnecessary computation while guaranteeing the result to be deterministic; i.e., it does not depend on how computation tasks are scheduled.

Process-based parallelism

Transducers.jl supports process-based parallelism using Distributed.jl. You can use it by replacing foldl with foldxd. It can be used for horizontally scaling the computation. It is also useful for using external libraries that are not "thread-safe."

Note that early termination is not supported in foldxd yet.

diff --git a/dev/reference/interface/index.html b/dev/reference/interface/index.html index a21afad6..41f646aa 100644 --- a/dev/reference/interface/index.html +++ b/dev/reference/interface/index.html @@ -1,8 +1,8 @@ -Interface · Transducers.jl
Edit on GitHub

Transducer interface

Core interface for transducers

Transducers.R_ — Type
Transducers.R_{X}

When defining a transducer type X, it is often required to dispatch on type rf::R_{X} (Reducing Function) which bundles the current transducer xform(rf)::X and the inner reducing function inner(rf)::R_.

source
Transducers.xform — Function
Transducers.xform(rf::R_{X}) -> xf :: X

Return the transducer xf associated with rf. Returned transducer xf is "atomic"; i.e., it is not a Composition transducer type.

source
Transducers.start — Function
Transducers.start(rf::R_{X}, state)

This is an optional interface for a transducer. Default implementation just calls start of the inner reducing function; i.e.,

start(rf::Reduction, result) = start(inner(rf), result)

If the transducer X is stateful, it can "bundle" its private state with wrap:

start(rf::R_{X}, result) = wrap(rf, PRIVATE_STATE, start(inner(rf), result))

where PRIVATE_STATE is an initial value for the private state that can be used inside next via wrapping.

See Take, PartitionBy, etc. for real-world examples.

Side notes: There is no related API in Clojure's Transducers. Transducers.jl uses it to implement stateful transducers using "pure" functions. The idea is based on a slightly different approach taken in C++ Transducer library atria.

source
Transducers.next — Function
Transducers.next(rf::R_{X}, state, input)

This is the only required interface. It takes the following form (if start is not defined):

next(rf::R_{X}, result, input) =
-    # code calling next(inner(rf), result, possibly_modified_input)

When calling next, it is almost always a better idea to use the macro form @next. See the details in its documentation.

See Map, Filter, Cat, etc. for real-world examples.

source
Transducers.@next — Macro
@next(rf, state, input)

It is expanded to

result = next(rf, state, input)
+Interface · Transducers.jl
Edit on GitHub
Transducer interface
Core interface for transducers
Transducers.R_ — Type
Transducers.R_{X}
When defining a transducer type X, it is often required to dispatch on type rf::R_{X} (Reducing Function) which bundles the current transducer xform(rf)::X and the inner reducing function inner(rf)::R_.
source
Transducers.xform — Function
Transducers.xform(rf::R_{X}) -> xf :: X
Return the transducer xf associated with rf.  Returned transducer xf is "atomic"; i.e., it is not a Composition transducer type.
source
Transducers.start — Function
Transducers.start(rf::R_{X}, state)
This is an optional interface for a transducer.  Default implementation just calls start of the inner reducing function; i.e.,
start(rf::Reduction, result) = start(inner(rf), result)
If the transducer X is stateful, it can "bundle" its private state with wrap:
start(rf::R_{X}, result) = wrap(rf, PRIVATE_STATE, start(inner(rf), result))
where PRIVATE_STATE is an initial value for the private state that can be used inside next via wrapping.
See Take, PartitionBy, etc. for real-world examples.
Side notes: There is no related API in Clojure's Transducers. Transducers.jl uses it to implement stateful transducers using "pure" functions.  The idea is based on a slightly different approach taken in C++ Transducer library atria.
source
Transducers.next — Function
Transducers.next(rf::R_{X}, state, input)
This is the only required interface.  It takes the following form (if start is not defined):
next(rf::R_{X}, result, input) =
+    # code calling next(inner(rf), result, possibly_modified_input)
When calling next, it is almost always a better idea to use the macro form @next.  See the details in its documentation.
See Map, Filter, Cat, etc. for real-world examples.
source
Transducers.@next — Macro
@next(rf, state, input)
It is expanded to
result = next(rf, state, input)
 result isa Reduced && return result
-result
This is usually the best way to call next as checking for Reduced is required to support early termination.
See also: next, Reduced, @return_if_reduced.
source
Transducers.complete — Function
Transducers.complete(rf::R_{X}, state)
This is an optional interface for a transducer.  If transducer X has some internal state, this is the last chance to "flush" the result.
See PartitionBy, etc. for real-world examples.
If start(rf::R_{X}, state) is defined, complete must unwarp state before returning state to the outer reducing function.
Transducers.jl 0.3
In Transducers.jl 0.2, complete had a fallback implementation to automatically call unwrap when wrap is called in start. Relying on this fallback implementation is now deprecated.
source
Transducers.combine — Function
Transducers.combine(rf::R_{X}, state_left, state_right)
This is an optional interface for a transducer.  If transducer X is stateful (i.e., wrap is used in start), it has to be able to combine the private states to support fold functions that require an associative reducing function such as foldxt. Typical implementation takes the following form:
function combine(rf::R_{X}, a, b)
+result
This is usually the best way to call next as checking for Reduced is required to support early termination.
See also: next, Reduced, @return_if_reduced.
source
Transducers.complete — Function
Transducers.complete(rf::R_{X}, state)
This is an optional interface for a transducer.  If transducer X has some internal state, this is the last chance to "flush" the result.
See PartitionBy, etc. for real-world examples.
If start(rf::R_{X}, state) is defined, complete must unwarp state before returning state to the outer reducing function.
Transducers.jl 0.3
In Transducers.jl 0.2, complete had a fallback implementation to automatically call unwrap when wrap is called in start. Relying on this fallback implementation is now deprecated.
source
Transducers.combine — Function
Transducers.combine(rf::R_{X}, state_left, state_right)
This is an optional interface for a transducer.  If transducer X is stateful (i.e., wrap is used in start), it has to be able to combine the private states to support fold functions that require an associative reducing function such as foldxt. Typical implementation takes the following form:
function combine(rf::R_{X}, a, b)
     #   ,---- `ua` and `ub` are the private state of the transducer `X`
     #  /  ,-- `ira` and `irb` are the states of inner reducing functions
     # /  /
@@ -11,7 +11,7 @@
     irc = combine(inner(rf), ira, irb)
     uc = # somehow combine private states `ua` and `ub`
     return wrap(rf, uc, irc)
-end
See ScanEmit, etc. for real-world examples.
source
Helpers for stateful transducers
Transducers.wrap — Function
wrap(rf::R_{X}, state, iresult)
Pack private state for reducing function rf (or rather the transducer X) with the result iresult returned from the inner reducing function inner(rf).  This packed result is typically passed to the outer reducing function.
This is intended to be used only in start.  Inside next, use wrapping.
Implementation detail
If iresult is a Reduced, wrap actually unwraps all internal state iresult recursively.  However, this is an implementation detail that should not matter when writing transducers.
Consider a reducing step constructed as
rf = opcompose(xf₁, xf₂, xf₃)'(f)
where each xfₙ is a stateful transducer and hence needs a private state stateₙ and this stateₙ is constructed in each start(::R_{typeof(xfₙ)}, result).  Then, calling start(rf, result)) is equivalent to
wrap(rf,
+end
See ScanEmit, etc. for real-world examples.
source
Helpers for stateful transducers
Transducers.wrap — Function
wrap(rf::R_{X}, state, iresult)
Pack private state for reducing function rf (or rather the transducer X) with the result iresult returned from the inner reducing function inner(rf).  This packed result is typically passed to the outer reducing function.
This is intended to be used only in start.  Inside next, use wrapping.
Implementation detail
If iresult is a Reduced, wrap actually unwraps all internal state iresult recursively.  However, this is an implementation detail that should not matter when writing transducers.
Consider a reducing step constructed as
rf = opcompose(xf₁, xf₂, xf₃)'(f)
where each xfₙ is a stateful transducer and hence needs a private state stateₙ and this stateₙ is constructed in each start(::R_{typeof(xfₙ)}, result).  Then, calling start(rf, result)) is equivalent to
wrap(rf,
      state₁,                     # private state for xf₁
      wrap(inner(rf),
           state₂,                # private state for xf₂
@@ -20,16 +20,16 @@
                result)))
or equivalently
result₃ = result
 result₂ = wrap(inner(inner(rf)), state₃, result₃)
 result₁ = wrap(inner(rf),        state₂, result₂)
-result₀ = wrap(rf,               state₁, result₁)
The inner most step function receives the original result as the first argument while transducible processes such as foldl only sees the outer-most "tree" result₀ during the reduction.
See wrapping, unwrap, and start.
source
Transducers.unwrap — Function
unwrap(rf, result)
Unwrap wraped result to a private state and inner result. Following identity holds:
unwrap(rf, wrap(rf, state, iresult)) == (state, iresult)
This is intended to be used only in complete.  Inside next, use wrapping.
source
Transducers.wrapping — Function
wrapping(f, rf, result)
Function f must take two argument state and iresult, and return a tuple (state, iresult).  This is intended to be used only in next, possibly with a do block.
next(rf::R_{MyTransducer}, result, input) =
+result₀ = wrap(rf,               state₁, result₁)
The inner most step function receives the original result as the first argument while transducible processes such as foldl only sees the outer-most "tree" result₀ during the reduction.
See wrapping, unwrap, and start.
source
Transducers.unwrap — Function
unwrap(rf, result)
Unwrap wraped result to a private state and inner result. Following identity holds:
unwrap(rf, wrap(rf, state, iresult)) == (state, iresult)
This is intended to be used only in complete.  Inside next, use wrapping.
source
Transducers.wrapping — Function
wrapping(f, rf, result)
Function f must take two argument state and iresult, and return a tuple (state, iresult).  This is intended to be used only in next, possibly with a do block.
next(rf::R_{MyTransducer}, result, input) =
     wrapping(rf, result) do my_state, iresult
         # code calling `next(inner(rf), iresult, possibly_modified_input)`
         return my_state, iresult  # possibly modified
-    end
See wrap, unwrap, and next.
source
Interface for reducibles
Transducers.__foldl__ — Function
__foldl__(rf, init, reducible::T)
Left fold a reducible with reducing function rf and initial value init.  This is primary an API for overloading when the reducible "container" or "context" (e.g., I/O stream) of type T can provide a better reduction mechanism than the default iterator-based one.
For a simple iterable type MyType, a valid implementation is:
function __foldl__(rf, val, itr::MyType)
+    end
See wrap, unwrap, and next.
source
Interface for reducibles
Transducers.__foldl__ — Function
__foldl__(rf, init, reducible::T)
Left fold a reducible with reducing function rf and initial value init.  This is primary an API for overloading when the reducible "container" or "context" (e.g., I/O stream) of type T can provide a better reduction mechanism than the default iterator-based one.
For a simple iterable type MyType, a valid implementation is:
function __foldl__(rf, val, itr::MyType)
     for x in itr
         val = @next(rf, val, x)
     end
     return complete(rf, val)
-end
although in this case default __foldl__ can handle MyType and thus there is no need for defining it.  In general, defining __foldl__ is useful only when there is a better way to go over items in reducible than Base.iterate.
See also: @next.
source
Transducers.@return_if_reduced — Macro
@return_if_reduced expr
It transforms the given expression to:
val = expr
+end
although in this case default __foldl__ can handle MyType and thus there is no need for defining it.  In general, defining __foldl__ is useful only when there is a better way to go over items in reducible than Base.iterate.
See also: @next.
source
Transducers.@return_if_reduced — Macro
@return_if_reduced expr
It transforms the given expression to:
val = expr
 val isa Reduced && return val
 val
See also @next.
Transducers.jl 0.3
In v0.2, the calling convention was @return_if_reduced complete(rf, val) and it was transformed to val isa Reduced && return reduced(complete(rf, unreduced(val))).  For the rationale behind the change, see this commit message.
Examples
julia> using Transducers: @return_if_reduced
 
@@ -42,4 +42,4 @@
         #= ... =#
         #158#val
     end
-end
source

+end
source
diff --git a/dev/reference/manual/index.html b/dev/reference/manual/index.html index 3b08cd1d..29213b65 100644 --- a/dev/reference/manual/index.html +++ b/dev/reference/manual/index.html @@ -1,5 +1,5 @@ -Manual · Transducers.jl
Edit on GitHub

Transducers and Transducible processes

Transducible processes

Transducers.foldxl — Function
foldxl(step, xf::Transducer, reducible; init, simd) :: T
+Manual · Transducers.jl
Edit on GitHub
Transducers and Transducible processes
Transducible processes
Transducers.foldxl — Function
foldxl(step, xf::Transducer, reducible; init, simd) :: T
 foldxl(step, reducible; init, simd) :: T
 foldl(step, xf::Transducer, reducible; init, simd) :: T
 foldl(step, ed::Eduction; init, simd) :: T
@@ -21,8 +21,8 @@
 (2, 8)
The unary method of foldlx is useful when combined with |>:
julia> (1:5, 4:-1:1) |> Cat() |> Filter(isodd) |> Enumerate() |> foldxl() do a, b
            a[2] < b[2] ? b : a
        end
-(3, 5)
source
Base.foldl — Function
foldl(step, xf::Transducer, reducible; init, simd) :: T
-foldl(step, ed::Eduction; init, simd) :: T
See foldxl.
source
Base.foldl — Function
foldl(step, xf::Transducer, reducible; init, simd) :: T
+foldl(step, ed::Eduction; init, simd) :: T
See foldxl.
source
Base.foreach — Function
foreach(eff, xf::Transducer, reducible; simd)
 foreach(eff, ed::Eduction; simd)
Feed the results of xf processing items in reducible into a unary function eff.  This is useful when the primary computation at the bottom is the side-effect.  It is also equivalent to foreach(eff, eduction(xf, coll)).  Note that
foreach(eduction(xf, coll)) do x
     ...
 end
can be more efficient than
for x in eduction(xf, coll)
@@ -81,7 +81,7 @@
 true
 
 julia> simpler_has2([1, missing])
-false
source
Transducers.foldxt — Function
foldxt(step, xf, reducible; [init, simd, basesize, stoppable, nestlevel]) :: T
eXtended threaded fold (reduce).  This is a multi-threaded reduce based on extended fold protocol defined in Transducers.jl.
The "bottom" reduction function step(::T, ::T) :: T must be associative and init must be its identity element.
Transducers composing xf must be stateless (e.g., Map, Filter, Cat, etc.) except for ScanEmit. Note that Scan is not supported (although possible in theory).  Early termination requires Julia ≥ 1.3.
Use tcollect or tcopy to collect results into a container.
See also: Parallel processing tutorial, foldxl, foldxd.
Keyword Arguments
  • basesize::Integer = amount(reducible) ÷ nthreads(): A size of chunk in reducible that is processed by each worker.  A smaller size may be required when:
    • computation time for processing each item fluctuates a lot
    • computation can be terminated by reduced or transducers using it, such as ReduceIf
  • stoppable::Bool: [This option usually does not have to be set manually.]  The threaded fold executed in the "stoppable" mode used for optimizing reduction with reduced has a slight overhead if reduced is not used.  This mode can be disabled by passing stoppable = false.  It is usually automatically detected and set appropriately.  Note that this option is purely for optimization and does not affect the result value.
  • nestlevel::Union{Integer,Val}: Specify how many inner Cat (flatten) transducers to be multi-threaded (using TCat). It must be a positive integer, Val of positive integer, or Val(:inf).  Val(:inf) means to use multi-threading for all Cat transducers.  Note that Cat transducer should be statically known. That is to say, the fold implementation sees two Cats in ... |> Map(f) |> Cat() |> Cat() but only one Cat in ... |> Map(x -> f(x) |> Cat()) |> Cat() even though they are semantically identical.
  • For other keyword arguments, see foldl.
Transducers.jl 0.4.23
Keyword option stoppable requires at least Transducers.jl 0.4.23.
Examples
julia> using Transducers
+false
source
Transducers.foldxt — Function
foldxt(step, xf, reducible; [init, simd, basesize, stoppable, nestlevel]) :: T
eXtended threaded fold (reduce).  This is a multi-threaded reduce based on extended fold protocol defined in Transducers.jl.
The "bottom" reduction function step(::T, ::T) :: T must be associative and init must be its identity element.
Transducers composing xf must be stateless (e.g., Map, Filter, Cat, etc.) except for ScanEmit. Note that Scan is not supported (although possible in theory).  Early termination requires Julia ≥ 1.3.
Use tcollect or tcopy to collect results into a container.
See also: Parallel processing tutorial, foldxl, foldxd.
Keyword Arguments
  • basesize::Integer = amount(reducible) ÷ nthreads(): A size of chunk in reducible that is processed by each worker.  A smaller size may be required when:
    • computation time for processing each item fluctuates a lot
    • computation can be terminated by reduced or transducers using it, such as ReduceIf
  • stoppable::Bool: [This option usually does not have to be set manually.]  The threaded fold executed in the "stoppable" mode used for optimizing reduction with reduced has a slight overhead if reduced is not used.  This mode can be disabled by passing stoppable = false.  It is usually automatically detected and set appropriately.  Note that this option is purely for optimization and does not affect the result value.
  • nestlevel::Union{Integer,Val}: Specify how many inner Cat (flatten) transducers to be multi-threaded (using TCat). It must be a positive integer, Val of positive integer, or Val(:inf).  Val(:inf) means to use multi-threading for all Cat transducers.  Note that Cat transducer should be statically known. That is to say, the fold implementation sees two Cats in ... |> Map(f) |> Cat() |> Cat() but only one Cat in ... |> Map(x -> f(x) |> Cat()) |> Cat() even though they are semantically identical.
  • For other keyword arguments, see foldl.
Transducers.jl 0.4.23
Keyword option stoppable requires at least Transducers.jl 0.4.23.
Examples
julia> using Transducers
 
 julia> foldxt(+, 1:3 |> Map(exp) |> Map(log))
 6.0
@@ -98,7 +98,7 @@
 9
 
 julia> foldxt(TeeRF(min, max), [5, 2, 6, 8, 3])
-(2, 8)
source
Transducers.foldxd — Function
foldxd(step, xform::Transducer, array; [init, simd, basesize, threads_basesize, pool])
eXtended distributed fold (reduce).  This is a distributed reduce based on extended fold protocol defined in Transducers.jl.
Unlike foldxt, early termination by reduced is not supported yet.
Use dcollect or dcopy to collect results into a container.
See also: Parallel processing tutorial, foldxl, foldxt.
Transducers.jl 0.4.3
New in version 0.4.3.
Keyword Arguments
  • pool::AbstractWorkerPool: Passed to Distributed.remotecall.
  • basesize::Integer = amount(array) ÷ nworkers(): A size of chunk in array that is processed by each worker.  A smaller size may be required when computation time for processing each item can fluctuate a lot.
  • threads_basesize::Integer = basesize ÷ nthreads(): A size of chunk in array that is processed by each task in each worker process. The default setting assumes that the number of threads used in all workers are the same.  For heterogeneous setup where each worker process has different number of threads, it may be required to use smaller threads_basesize and basesize to get a good performance.
  • For other keyword arguments, see foldl.
Examples
julia> using Transducers
+(2, 8)
source
Transducers.foldxd — Function
foldxd(step, xform::Transducer, array; [init, simd, basesize, threads_basesize, pool])
eXtended distributed fold (reduce).  This is a distributed reduce based on extended fold protocol defined in Transducers.jl.
Unlike foldxt, early termination by reduced is not supported yet.
Use dcollect or dcopy to collect results into a container.
See also: Parallel processing tutorial, foldxl, foldxt.
Transducers.jl 0.4.3
New in version 0.4.3.
Keyword Arguments
  • pool::AbstractWorkerPool: Passed to Distributed.remotecall.
  • basesize::Integer = amount(array) ÷ nworkers(): A size of chunk in array that is processed by each worker.  A smaller size may be required when computation time for processing each item can fluctuate a lot.
  • threads_basesize::Integer = basesize ÷ nthreads(): A size of chunk in array that is processed by each task in each worker process. The default setting assumes that the number of threads used in all workers are the same.  For heterogeneous setup where each worker process has different number of threads, it may be required to use smaller threads_basesize and basesize to get a good performance.
  • For other keyword arguments, see foldl.
Examples
julia> using Transducers
 
 julia> foldxd(+, 1:3 |> Map(exp) |> Map(log))
 6.0
@@ -107,7 +107,7 @@
 9
 
 julia> foldxd(TeeRF(min, max), [5, 2, 6, 8, 3])
-(2, 8)
source
Transducers.eduction — Function
eduction(xf::Transducer, coll)
 xf(coll)
 coll |> xf
Create a iterable and reducible object.
This API is modeled after eduction in Clojure.
Note
Even though eduction returns an iterable, it is highly recommended to use the foldl-based method provided by Transducers.jl when the performance is important.
Examples
julia> using Transducers
 
@@ -123,7 +123,7 @@
        end;
 x = 1
 x = 3
-x = 5
source
eduction(iterator::Iterators.Generator)
+x = 5
source
eduction(iterator::Iterators.Generator)
 eduction(iterator::Iterators.Filter)
 eduction(iterator::Iterators.Flatten)
Convert an iterator to an eduction.  The iterators that are typically used in the generator comprehensions are supported.
Transducers.jl 0.3
New in version 0.3.
Examples
julia> using Transducers
 
@@ -132,7 +132,7 @@
 julia> ed = eduction(iter);
 
 julia> collect(iter) == collect(ed)
-true
source
Base.map! — Function
map!(xf::Transducer, dest, src; simd)
Feed src to transducer xf, storing the result in dest. Collections dest and src must have the same shape.  Transducer xf may contain filtering transducers.  If some entries src are skipped, the corresponding entries in dest will be unchanged. Transducer xf must not contain any expansive transducers such as MapCat.
See also copy!.
Examples
julia> using Transducers
+true
source
Base.map! — Function
map!(xf::Transducer, dest, src; simd)
Feed src to transducer xf, storing the result in dest. Collections dest and src must have the same shape.  Transducer xf may contain filtering transducers.  If some entries src are skipped, the corresponding entries in dest will be unchanged. Transducer xf must not contain any expansive transducers such as MapCat.
See also copy!.
Examples
julia> using Transducers
 
 julia> xs = collect(1:5)
        ys = zero(xs)
@@ -145,14 +145,14 @@
  5
 
 julia> ans === ys
-true
source
Base.copy! — Function
copy!(xf::Transducer, dest, src)
Feed src to transducer xf, storing the result in dest. Collections dest and src may have the same shape.  Source src must be iterable.  Destination dest must implement empty! and push!.
See also map!.
Examples
julia> using Transducers
+true
source
Base.copy! — Function
copy!(xf::Transducer, dest, src)
Feed src to transducer xf, storing the result in dest. Collections dest and src may have the same shape.  Source src must be iterable.  Destination dest must implement empty! and push!.
See also map!.
Examples
julia> using Transducers
 
 julia> copy!(opcompose(PartitionBy(x -> x ÷ 3), Map(sum)), Int[], 1:10)
 4-element Vector{Int64}:
   3
  12
  21
- 19
source
Base.copy — Function
copy(xf::Transducer, T, foldable) :: Union{T, Empty{T}}
+ 19
source
Base.copy — Function
copy(xf::Transducer, T, foldable) :: Union{T, Empty{T}}
 copy(xf::Transducer, foldable::T) :: Union{T, Empty{T}}
 copy([T,] eduction::Eduction) :: Union{T, Empty{T}}
Process foldable with a transducer xf and then create a container of type T filled with the result.  Return BangBang.Empty{T} if the transducer does not produce anything.  (This is because there is no consistent interface to create an empty container given its type and not all containers support creating an empty container.)
For parallel versions, see tcopy and dcopy.
Transducers.jl 0.4.4
New in version 0.4.4.
Transducers.jl 0.4.8
copy now accepts eductions.
Examples
julia> using Transducers
        using BangBang: Empty
@@ -176,7 +176,7 @@
 julia> @assert copy(
            Map(x -> (A = x.a + 1, B = x.b + 1)),
            DataFrame(a = [1], b = [2]),
-       ) == DataFrame(A = [2], B = [3])
source
Transducers.tcopy — Function
tcopy(xf::Transducer, T, reducible; basesize) :: Union{T, Empty{T}}
+       ) == DataFrame(A = [2], B = [3])
source
Transducers.tcopy — Function
tcopy(xf::Transducer, T, reducible; basesize) :: Union{T, Empty{T}}
 tcopy(xf::Transducer, reducible::T; basesize) :: Union{T, Empty{T}}
 tcopy([T,] itr; basesize) :: Union{T, Empty{T}}
Thread-based parallel version of copy. Keyword arguments are passed to foldxt.
See also: Parallel processing tutorial (especially Example: parallel collect).
Transducers.jl 0.4.5
New in version 0.4.5.
Transducers.jl 0.4.8
tcopy now accepts iterator comprehensions and eductions.
Examples
julia> using Transducers
 
@@ -228,9 +228,9 @@
            1:2;
            basesize = 1,
            # init = Empty(Table),
-       ) == Table(a = [1, 2])
source
Transducers.dcopy — Function
dcopy(xf::Transducer, T, reducible; [basesize, threads_basesize]) :: Union{T, Empty{T}}
+       ) == Table(a = [1, 2])
source
Transducers.dcopy — Function
dcopy(xf::Transducer, T, reducible; [basesize, threads_basesize]) :: Union{T, Empty{T}}
 dcopy(xf::Transducer, reducible::T; [basesize, threads_basesize]) :: Union{T, Empty{T}}
-dcopy([T,] itr; [basesize, threads_basesize]) :: Union{T, Empty{T}}
Distributed.jl-based parallel version of copy.  Keyword arguments are passed to foldxd.  For examples, see tcopy.
See also: Parallel processing tutorial (especially Example: parallel collect).
Transducers.jl 0.4.5
New in version 0.4.5.
Transducers.jl 0.4.8
dcopy now accepts iterator comprehensions and eductions.
source
Base.append! — Function
append!(xf::Transducer, dest, src) -> dest
This API is modeled after into in Clojure.
Warning
The performance of append!(dest, src::Eduction) is poor. Use append!! instead if two-argument form is preferred.
Examples
julia> using Transducers
+dcopy([T,] itr; [basesize, threads_basesize]) :: Union{T, Empty{T}}
Distributed.jl-based parallel version of copy.  Keyword arguments are passed to foldxd.  For examples, see tcopy.
See also: Parallel processing tutorial (especially Example: parallel collect).
Transducers.jl 0.4.5
New in version 0.4.5.
Transducers.jl 0.4.8
dcopy now accepts iterator comprehensions and eductions.
source
Base.append! — Function
append!(xf::Transducer, dest, src) -> dest
This API is modeled after into in Clojure.
Warning
The performance of append!(dest, src::Eduction) is poor. Use append!! instead if two-argument form is preferred.
Examples
julia> using Transducers
 
 julia> append!(Drop(2), [-1, -2], 1:5)
 5-element Vector{Int64}:
@@ -238,7 +238,7 @@
  -2
   3
   4
-  5
source
BangBang.append!! — Function
BangBang.append!!(xf::Transducer, dest, src) -> dest′
 BangBang.append!!(dest, src::Eduction) -> dest′
Mutate-or-widen version of append!.
Transducers.jl 0.4.4
New in version 0.4.4.
Transducers.jl 0.4.37
Performance optimization for append!!(dest, src::Eduction) requires version 0.4.37.
Examples
julia> using Transducers, BangBang
 
 julia> append!!(opcompose(Drop(2), Map(x -> x + 0.0)), [-1, -2], 1:5)
@@ -247,7 +247,7 @@
  -2.0
   3.0
   4.0
-  5.0
source
Base.collect — Function
collect(xf::Transducer, itr) :: Vector
 collect(ed::Eduction) :: Vector
Process an iterable itr using a transducer xf and collect the result into a Vector.
For parallel versions, see tcollect and dcollect.
Transducers.jl 0.4.8
collect now accepts eductions.
Examples
julia> using Transducers
 
 julia> collect(Interpose(missing), 1:3)
@@ -256,7 +256,7 @@
   missing
  2
   missing
- 3
source
Transducers.tcollect — Function
tcollect(xf::Transducer, reducible; basesize) :: Union{Vector, Empty{Vector}}
 tcollect(itr; basesize) :: Union{Vector, Empty{Vector}}
Thread-based parallel version of collect. This is just a short-hand notation of tcopy(xf, Vector, reducible). Use tcopy to get a container other than a Vector.
See also: Parallel processing tutorial (especially Example: parallel collect).
Transducers.jl 0.4.5
New in version 0.4.5.
Transducers.jl 0.4.8
tcollect now accepts iterator comprehensions and eductions.
Examples
julia> using Transducers
 
 julia> tcollect(Map(x -> x^2), 1:2)
@@ -267,8 +267,8 @@
 julia> tcollect(x^2 for x in 1:2)
 2-element Vector{Int64}:
  1
- 4
source
Transducers.dcollect — Function
dcollect(xf::Transducer, reducible; [basesize, threads_basesize]) :: Union{Vector, Empty{Vector}}
-dcollect(itr; [basesize, threads_basesize]) :: Union{Vector, Empty{Vector}}
Distributed.jl-based parallel version of collect. This is just a short-hand notation of dcopy(xf, Vector, reducible). Use dcopy to get a container other than a Vector.
See also: Parallel processing tutorial (especially Example: parallel collect).
Transducers.jl 0.4.5
New in version 0.4.5.
Transducers.jl 0.4.8
dcollect now accepts iterator comprehensions and eductions.
source
Transducers.dcollect — Function
dcollect(xf::Transducer, reducible; [basesize, threads_basesize]) :: Union{Vector, Empty{Vector}}
+dcollect(itr; [basesize, threads_basesize]) :: Union{Vector, Empty{Vector}}
Distributed.jl-based parallel version of collect. This is just a short-hand notation of dcopy(xf, Vector, reducible). Use dcopy to get a container other than a Vector.
See also: Parallel processing tutorial (especially Example: parallel collect).
Transducers.jl 0.4.5
New in version 0.4.5.
Transducers.jl 0.4.8
dcollect now accepts iterator comprehensions and eductions.
source
Base.Channel — Type
Channel(xf::Transducer, itr; kwargs...)
 Channel(ed::Eduction; kwargs...)
Pipe items from an iterable itr processed by the transducer xf through a channel.  Channel(xf, itr) and Channel(eduction(xf, itr)) are equivalent.  Note that itr itself can be a Channel.
Keyword arguments are passed to Channel(function; kwargs...).
Examples
julia> using Transducers
 
 julia> ch1 = Channel(Filter(isodd), 1:5);
@@ -285,7 +285,7 @@
 input = [1, 1]
 input = [2, 3, 4, 5]
 input = [1]
-input = [2, 3, 4, 5, 6, 7, 8, 9]
source
Experimental transducible processes
Experimental transducible processes
Transducers.channel_unordered — Function
channel_unordered(xf, input; eltype, size, ntasks, basesize) :: Channel{eltype}
 channel_unordered(itr; eltype, size, ntasks, basesize) :: Channel{eltype}
Provide elements in input processed by a transducer xf through a Channel.
Unary method channel_unordered(itr) produces a Channel that provides elements in the input iterator itr with possibly different order.  Iterator comprehensions and eductions can be passed as the input itr.
Use append_unordered! to send outputs to an existing channel.
Transducers.jl 0.4.8
New in version 0.4.8.
Transducers.jl 0.4.9
Unary method channel_unordered(itr) requires Transducers.jl 0.4.9.
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
Keyword Arguments
  • eltype::Type: element type of returned channel
  • size: The size of Channel. A non-negative Int or Inf.
  • ntasks::Int: Number of concurrent tasks.  Default to Threads.nthreads().
  • basesize: The "batch size" of the processing; i.e., the number of elements to be processed in a single task.  Default to 1.
Examples
julia> using Transducers: Map, channel_unordered
 
 julia> sort!(collect(channel_unordered(Map(x -> x + 1), 1:3)))
@@ -297,7 +297,7 @@
 julia> sort!(collect(channel_unordered(x + 1 for x in 1:3 if isodd(x))))
 2-element Vector{Any}:
  2
- 4
source
Transducers.append_unordered! — Function
append_unordered!(output, xf, input; ntasks, basesize)
 append_unordered!(output, itr; ntasks, basesize)
Process input elements through a transducer xf and then push! them into output in undefined order.
Binary method append_unordered!(output, itr) is like append!(output, itr) but without order guarantee.  Iterator comprehensions and eductions can be passed as the input itr.
output (typically a Channel) must implement thread-safe push!(output, x) method.
See also channel_unordered.
Transducers.jl 0.4.8
New in version 0.4.8.
Transducers.jl 0.4.9
Binary method append_unordered!(output, itr) requires Transducers.jl 0.4.9.
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
Examples
julia> using Transducers: Map, append_unordered!
 
 julia> input = Channel(Map(identity), 1:3);
@@ -331,7 +331,7 @@
  1
  1
  2
- 3
source
Transducers
Transducers.Transducer — Type
Transducer
The abstract type for transducers.
A transducer xf can be used as both iterator transformation xf(itr) and reducing function transformation xf'(rf).
See also adjoint for xf'(rf).
Transducers.jl 0.4.39
The call overload xf(rf) requires Transducers.jl 0.4.39 or later.
Note
The call overload xf(rf) requires Julia 1.3 or later. For older Julia versions, use eduction.
Examples
julia> using Transducers
+ 3
source
Transducers
Transducers.Transducer — Type
Transducer
The abstract type for transducers.
A transducer xf can be used as both iterator transformation xf(itr) and reducing function transformation xf'(rf).
See also adjoint for xf'(rf).
Transducers.jl 0.4.39
The call overload xf(rf) requires Transducers.jl 0.4.39 or later.
Note
The call overload xf(rf) requires Julia 1.3 or later. For older Julia versions, use eduction.
Examples
julia> using Transducers
 
 julia> xs = Map(inv)(2:2:4)
 2-element StepRange{Int64, Int64} |>
@@ -349,16 +349,16 @@
         +))
 
 julia> rf(1, 4)  # +(1, inv(4))
-1.25
source
Base.:∘ — Function
f ⨟ g
 g ∘ f
 opcompose(f, g)
-compose(g, f)
Composition of transducers.
Transducers.jl 0.4.39
Transducers.jl 0.4.39 or later is required for composing transducers with ∘ and other operators and functions derived from it.
Transducers written as f |> g |> h in previous versions of Transducers.jl can now be written as f ⨟ g ⨟ h (in Julia 1.5 or later) or opcompose(f, g, h).
Note
"op" in opcompose does not stand for operator; it stands for opposite.
source
Base.adjoint — Function
xf'
xf'(rf₁) is a shortcut for calling reducingfunction(xf, rf₁).
More precisely, adjoint xf′ of a transducer xf is a reducing function transform rf₁ -> rf₂.  That is to say, xf' a function that maps a reducing function rf₁ to another reducing function rf₂.
Examples
julia> using Transducers
+compose(g, f)
Composition of transducers.
Transducers.jl 0.4.39
Transducers.jl 0.4.39 or later is required for composing transducers with ∘ and other operators and functions derived from it.
Transducers written as f |> g |> h in previous versions of Transducers.jl can now be written as f ⨟ g ⨟ h (in Julia 1.5 or later) or opcompose(f, g, h).
Note
"op" in opcompose does not stand for operator; it stands for opposite.
source
Base.adjoint — Function
xf'
xf'(rf₁) is a shortcut for calling reducingfunction(xf, rf₁).
More precisely, adjoint xf′ of a transducer xf is a reducing function transform rf₁ -> rf₂.  That is to say, xf' a function that maps a reducing function rf₁ to another reducing function rf₂.
Examples
julia> using Transducers
 
 julia> y = Map(inv)'(+)(10, 2)
 10.5
 
 julia> y == +(10, inv(2))
-true
source
Transducers.Broadcasting — Type
Broadcasting()
Broadcast inner reducing function over elements in the input.  Roughly speaking, it transforms the inner reducing function op to op′(a, b) = op.(a, b).  However, it has a better memory usage and better initial value handling.
If the input is an array, the array created at the first iteration is reused if it can hold the element types of subsequent iterations. Otherwise, the array type is widen as needed.
If init passed to the fold function is a lazy "initializer" object such as OnInit, it is initialized independently for each item in the first input array.  This makes using Broadcasting for (possibly) in-place functions safe.
Transducers.jl 0.4.32
New in version 0.4.32.
Note
Broadcasting transducer is not supported in Julia 1.0.
Examples
julia> using Transducers
+true
source
Transducers.Broadcasting — Type
Broadcasting()
Broadcast inner reducing function over elements in the input.  Roughly speaking, it transforms the inner reducing function op to op′(a, b) = op.(a, b).  However, it has a better memory usage and better initial value handling.
If the input is an array, the array created at the first iteration is reused if it can hold the element types of subsequent iterations. Otherwise, the array type is widen as needed.
If init passed to the fold function is a lazy "initializer" object such as OnInit, it is initialized independently for each item in the first input array.  This makes using Broadcasting for (possibly) in-place functions safe.
Transducers.jl 0.4.32
New in version 0.4.32.
Note
Broadcasting transducer is not supported in Julia 1.0.
Examples
julia> using Transducers
 
 julia> foldl(+, Broadcasting(), [[1, 2], [3, 4], 5])
 2-element Vector{Int64}:
@@ -395,10 +395,10 @@
            add!(add!([0, 0], [1]), [4, 5]),
            add!(add!([0, 0], [2, 3]), 6),
        ]
-true
source
Transducers.Cat — Type
Cat()
Concatenate/flatten nested iterators.
This API is modeled after cat in Clojure.
Examples
julia> using Transducers
 
 julia> collect(Cat(), [[1, 2], [3], [4, 5]]) == 1:5
-true
source
Transducers.Consecutive — Type
Consecutive(size, step = size)
 Consecutive(size; step = size)
Sliding window of width size and interval step. Yield tuples.
This transducer is like Partition but feeds tuples to the downstream transducers instead of vectors.
If step == 1, this transducer supports parallel reduction for any collections; i.e., Consecutive(size, 1)'(op) is associative if op is associative.
Warning
Currently, in parallel folds, Consecutive(size, 1) cannot be used with reducing functions that can produce a Reduced.
If step > 1, this transducer can, in principle, support parallel reduction if the input collection allows random access (e.g., arrays). However, this feature is not implemented yet.
Examples
julia> using Transducers
 
 julia> 1:8 |> Consecutive(3) |> collect
@@ -413,7 +413,7 @@
  (3, 4, 5)
  (4, 5, 6)
  (5, 6, 7)
- (6, 7, 8)
source
Transducers.Count — Type
Count([start[, step]])
Generate a sequence start, start + step, start + step + step, and so on.
Note that input is ignored.  To use the input in the downstream reduction steps, use Zip.
start defaults to 1 and step defaults to oneunit(start).
See also: Iterators.countfrom. Enumerate
Examples
julia> using Transducers
+ (6, 7, 8)
source
Transducers.Count — Type
Count([start[, step]])
Generate a sequence start, start + step, start + step + step, and so on.
Note that input is ignored.  To use the input in the downstream reduction steps, use Zip.
start defaults to 1 and step defaults to oneunit(start).
See also: Iterators.countfrom. Enumerate
Examples
julia> using Transducers
 
 julia> collect(Zip(Map(identity), Count()), -3:-1)
 3-element Vector{Tuple{Int64, Int64}}:
@@ -425,7 +425,7 @@
 
 julia> 1:3 |> Zip(Map(identity), Count(Day(1))) |> MapSplat(*) |> collect ==
        [Day(1), Day(4), Day(9)]
-true
source
Transducers.Dedupe — Type
Dedupe()
 Dedupe(eq)
De-duplicate consecutive items.  Comparison operator which identifies duplicates can be specified by the eq parameter, which defaults to == (equal).
This API is modeled after dedupe in Clojure.
Examples
julia> using Transducers
 
 julia> collect(Dedupe(), [1, 1, 2, 1, 3, 3, 2])
@@ -434,12 +434,12 @@
  2
  1
  3
- 2
source
Transducers.Drop — Type
Drop(n)
Drop first n items.
This API is modeled after drop in Clojure.
Examples
julia> using Transducers
 
 julia> 1:5 |> Drop(3) |> collect
 2-element Vector{Int64}:
  4
- 5
source
Transducers.DropLast — Type
DropLast(n)
Drop last n items.
This API is modeled after drop-last in Clojure.
Examples
julia> using Transducers
 
 julia> 1:5 |> DropLast(2) |> collect
 3-element Vector{Int64}:
@@ -451,7 +451,7 @@
 true
 
 julia> 1:0 |> DropLast(2) |> collect == []
-true
source
Transducers.DropWhile — Type
DropWhile(pred)
Drop items while pred returns true consecutively.  It becomes a no-op after pred returns a false.
This API is modeled after drop-while in Clojure.
Examples
julia> using Transducers
 
 julia> collect(DropWhile(x -> x < 3), [1:5; 1:2])
 5-element Vector{Int64}:
@@ -459,7 +459,7 @@
  4
  5
  1
- 2
source
Transducers.Enumerate — Type
Enumerate([start[, step]])
Transducer variant of Base.enumerate. The start and step arguments are optional and have the same meaning as in Count.
Examples
julia> using Transducers
+ 2
source
Transducers.Enumerate — Type
Enumerate([start[, step]])
Transducer variant of Base.enumerate. The start and step arguments are optional and have the same meaning as in Count.
Examples
julia> using Transducers
 
 julia> collect(Enumerate(), ["A", "B", "C"])
 3-element Vector{Tuple{Int64, String}}:
@@ -474,17 +474,17 @@
  (2, "A")
  (5, "B")
  (8, "C")
-
source
Transducers.Filter — Type
Filter(pred)
Skip items for which pred is evaluated to false.
This API is modeled after filter in Clojure.
Examples
julia> using Transducers
 
 julia> 1:3 |> Filter(iseven) |> collect
 1-element Vector{Int64}:
- 2
source
Transducers.FlagFirst — Type
FlagFirst()
Output (isfirst, input) where isfirst::Bool is true only for the first iteration and input is the original input.
See also: IterTools.flagfirst
Examples
julia> using Transducers
 
 julia> collect(FlagFirst(), 1:3)
 3-element Vector{Tuple{Bool, Int64}}:
  (1, 1)
  (0, 2)
- (0, 3)
source
Transducers.GroupBy — Type
GroupBy(key, xf::Transducer, [step = right, [init]])
 GroupBy(key, rf, [init])
Group the input stream by a function key and then fan-out each group of key-value pairs to the eduction xf'(step).  This is similar to the groupby relational database operation.
For example
[1,2,1,2,3] |> GroupBy(string, Map(last)'(+)) |> foldxl(right)
returns a result equivalent to Dict("1"=>2, "2"=>4, "3"=>3) while
[1,2,1,2,3] |> GroupBy(string, Map(last) ⨟ Map(Transducers.SingletonVector), append!!) |> foldxl(right)
returns a result equivalent to Dict("1"=>[1,1], "2"=>[2,2], "3"=>[3,3]).
Alternatively, one can provide a reducing function directly, though this is disfavored since it prevents results from being combined with Transducers.combine and therefore cannot be used with foldxt or foldxd.  For example, if GroupBy is used as in:
xs |> Map(upstream) |> GroupBy(key, rf, init) |> Map(downstream)
then the function signatures would be:
upstream(_) :: V
 key(::V) :: K
 rf(::Y, ::Pair{K, V}) ::Y
@@ -520,7 +520,7 @@
 julia> unreduced(result)
 Transducers.GroupByViewDict{String,Any,…}(...):
   "1" => 2
-  "2" => 4
source
Transducers.Interpose — Type
Interpose(sep)
Interleave input items with a sep.
This API is modeled after interpose in Clojure.
Examples
julia> using Transducers
 
 julia> collect(Interpose(missing), 1:3)
 5-element Vector{Union{Missing, Int64}}:
@@ -528,7 +528,7 @@
   missing
  2
   missing
- 3
source
Transducers.Iterated — Type
Iterated(f, init)
Generate a sequence init, f(init), f(f(init)), f(f(f(init))), and so on.
Note that input is ignored.  To use the input in the downstream reduction steps, use Zip.
Pass OnInit or CopyInit object to init for creating a dedicated (possibly mutable) state for each fold.
See also: Scan, ScanEmit.
The idea is taken from IterTools.iterated
Examples
julia> using Transducers
+ 3
source
Transducers.Iterated — Type
Iterated(f, init)
Generate a sequence init, f(init), f(f(init)), f(f(f(init))), and so on.
Note that input is ignored.  To use the input in the downstream reduction steps, use Zip.
Pass OnInit or CopyInit object to init for creating a dedicated (possibly mutable) state for each fold.
See also: Scan, ScanEmit.
The idea is taken from IterTools.iterated
Examples
julia> using Transducers
 
 julia> collect(Iterated(x -> 2x, 1), 1:5)
 5-element Vector{Int64}:
@@ -544,7 +544,7 @@
  (2, 2)
  (3, 4)
  (4, 8)
- (5, 16)
source
Transducers.KeepSomething — Type
KeepSomething(f = identity)
Pass non-nothing output of f to the inner reducing step after possibly unwrapping Some.
This API is modeled after keep in Clojure.
Examples
julia> using Transducers
+ (5, 16)
source
Transducers.KeepSomething — Type
KeepSomething(f = identity)
Pass non-nothing output of f to the inner reducing step after possibly unwrapping Some.
This API is modeled after keep in Clojure.
Examples
julia> using Transducers
 
 julia> xf = KeepSomething() do x
            if x == 0
@@ -568,13 +568,13 @@
 julia> [Some(1), 2, nothing] |> NotA(Nothing) |> collect
 2-element Vector{Any}:
   Some(1)
- 2
source
Transducers.Map — Type
Map(f)
Apply unary function f to each input and pass the result to the inner reducing step.
This API is modeled after map in Clojure.
Examples
julia> using Transducers
+ 2
source
Transducers.Map — Type
Map(f)
Apply unary function f to each input and pass the result to the inner reducing step.
This API is modeled after map in Clojure.
Examples
julia> using Transducers
 
 julia> collect(Map(x -> 2x), 1:3)
 3-element Vector{Int64}:
  2
  4
- 6
source
Transducers.MapCat — Type
MapCat(f)
Concatenate output of f which is expected to return an iterable.
This API is modeled after mapcat in Clojure.
Examples
julia> using Transducers
 
 julia> collect(MapCat(x -> 1:x), 1:3)
 6-element Vector{Int64}:
@@ -583,13 +583,13 @@
  2
  1
  2
- 3
source
Transducers.MapSplat — Type
MapSplat(f)
Like Map(f) but calls f(input...) for each input and then pass the result to the inner reducing step.
Examples
julia> using Transducers
+ 3
source
Transducers.MapSplat — Type
MapSplat(f)
Like Map(f) but calls f(input...) for each input and then pass the result to the inner reducing step.
Examples
julia> using Transducers
 
 julia> collect(MapSplat(*), zip(1:3, 10:10:30))
 3-element Vector{Int64}:
  10
  40
- 90
source
Transducers.NondeterministicThreading — Type
NondeterministicThreading(; basesize, ntasks = nthreads())
Parallelize inner reducing function using ntasks.
Given
#                   ,-- Not parallelized
 #            ______/__
   foldxl(rf, xs |> xfo |> NondeterministicThreading() |> xfi)
 #        ==                                              ===
@@ -603,17 +603,17 @@
 julia> collect(1:4 |> NondeterministicThreading() |> Filter(isodd))
 2-element Vector{Int64}:
  1
- 3
source
Transducers.NotA — Type
NotA(T)
Skip items of type T.  Unlike Filter(!ismissing), downstream transducers can have a correct type information for NotA(Missing).
See also: OfType
Examples
julia> using Transducers
+ 3
source
Transducers.NotA — Type
NotA(T)
Skip items of type T.  Unlike Filter(!ismissing), downstream transducers can have a correct type information for NotA(Missing).
See also: OfType
Examples
julia> using Transducers
 
 julia> [1, missing, 2] |> NotA(Missing) |> collect
 2-element Vector{Int64}:
  1
- 2
source
Transducers.OfType — Type
OfType(T)
Include only items of type T.
See also: NotA
Examples
julia> using Transducers
 
 julia> [1, missing, 2] |> OfType(Int) |> collect
 2-element Vector{Int64}:
  1
- 2
source
Transducers.Partition — Type
Partition(size, step = size, flush = false)
 Partition(size; step = size, flush = false)
Sliding window of width size and interval step. Yield vectors.
Note: step = size is the default.  Hence, the default behavior is non-overlapping windows.
For small size, see Consecutive for a similar transducer that yields tuples instead.
Warning
The vector passed to the inner reducing function is valid only during its immediate reduction step.  It must be reduced immediately or copied.
This API is modeled after partition-all in Clojure.
Examples
julia> using Transducers
 
 julia> 1:8 |> Partition(3) |> Map(copy) |> collect
@@ -634,17 +634,17 @@
  [3, 4, 5]
  [4, 5, 6]
  [5, 6, 7]
- [6, 7, 8]
source
Transducers.PartitionBy — Type
PartitionBy(f)
Group input sequence into chunks in which f returns a same value consecutively.
Warning
The vector passed to the inner reducing function is valid only during its immediate reduction step.  It must be reduced immediately or copied.
This API is modeled after partition-by in Clojure.
Examples
julia> using Transducers
+ [6, 7, 8]
source
Transducers.PartitionBy — Type
PartitionBy(f)
Group input sequence into chunks in which f returns a same value consecutively.
Warning
The vector passed to the inner reducing function is valid only during its immediate reduction step.  It must be reduced immediately or copied.
This API is modeled after partition-by in Clojure.
Examples
julia> using Transducers
 
 julia> 1:9 |> PartitionBy(x -> (x + 1) ÷ 3) |> Map(copy) |> collect
 4-element Vector{UnitRange{Int64}}:
  1:1
  2:4
  5:7
- 8:9
source
Transducers.ReduceIf — Type
ReduceIf(pred)
Stop fold when pred(x) returns true for the output x of the upstream transducer.
Examples
julia> using Transducers
+ 8:9
source
Transducers.ReduceIf — Type
ReduceIf(pred)
Stop fold when pred(x) returns true for the output x of the upstream transducer.
Examples
julia> using Transducers
 
 julia> foldl(right, ReduceIf(x -> x == 3), 1:10)
-3
source
Transducers.ReducePartitionBy — Type
ReducePartitionBy(f, rf, [init])
Reduce partitions determined by isequal on the output value of f with an associative reducing function rf. Partitions are reduced on-the-fly and no intermediate arrays are allocated.
Examples
Consider the input 1:6 "keyed" by a function x -> x ÷ 3:
julia> map(x -> x ÷ 3, 1:6)
+3
source
Transducers.ReducePartitionBy — Type
ReducePartitionBy(f, rf, [init])
Reduce partitions determined by isequal on the output value of f with an associative reducing function rf. Partitions are reduced on-the-fly and no intermediate arrays are allocated.
Examples
Consider the input 1:6 "keyed" by a function x -> x ÷ 3:
julia> map(x -> x ÷ 3, 1:6)
 6-element Vector{Int64}:
  0
  0
@@ -663,7 +663,7 @@
 3-element Vector{Int64}:
   3
  12
-  6
source
Transducers.Replace — Type
Replace(assoc)
Replace each input with the value in the associative container assoc (e.g., a dictionary, array, string) if it matches with a key/index. Otherwise output the input as-is.
This API is modeled after replace in Clojure.
Examples
julia> using Transducers
+  6
source
Transducers.Replace — Type
Replace(assoc)
Replace each input with the value in the associative container assoc (e.g., a dictionary, array, string) if it matches with a key/index. Otherwise output the input as-is.
This API is modeled after replace in Clojure.
Examples
julia> using Transducers
 
 julia> collect(Replace(Dict('a' => 'A')), "abc")
 3-element Vector{Char}:
@@ -685,7 +685,7 @@
   'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
   'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)
   'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)
- 4
source
Transducers.Scan — Type
Scan(f, [init = Init])
Accumulate input with binary function f and pass the accumulated result so far to the inner reduction step.
The inner reducing step receives the sequence y₁, y₂, y₃, ..., yₙ, ... when the sequence x₁, x₂, x₃, ..., xₙ, ... is fed to Scan(f).
y₁ = f(init, x₁)
+ 4
source
Transducers.Scan — Type
Scan(f, [init = Init])
Accumulate input with binary function f and pass the accumulated result so far to the inner reduction step.
The inner reducing step receives the sequence y₁, y₂, y₃, ..., yₙ, ... when the sequence x₁, x₂, x₃, ..., xₙ, ... is fed to Scan(f).
y₁ = f(init, x₁)
 y₂ = f(y₁, x₂)
 y₃ = f(y₂, x₃)
 ...
@@ -707,7 +707,7 @@
 3-element Vector{Int64}:
  10
  20
- 60
source
Transducers.ScanEmit — Type
ScanEmit(f, init[, onlast])
Accumulate input x with a function f with the call signature (u, x) -> (y, u) and pass the result y to the inner reduction step.
The inner reducing step receives the sequence y₁, y₂, y₃, ..., yₙ, ... computed as follows
u₀ = init
+ 60
source
Transducers.ScanEmit — Type
ScanEmit(f, init[, onlast])
Accumulate input x with a function f with the call signature (u, x) -> (y, u) and pass the result y to the inner reduction step.
The inner reducing step receives the sequence y₁, y₂, y₃, ..., yₙ, ... computed as follows
u₀ = init
 y₁, u₁ = f(u₀, x₁)
 y₂, u₂ = f(u₁, x₂)
 y₃, u₃ = f(u₂, x₃)
@@ -720,7 +720,7 @@
 3-element Vector{Int64}:
  0
  1
- 2
source
Transducers.TCat — Type
TCat(basesize::Integer)
Threaded version of Cat (concatenate/flatten).
To use this transducer, all the downstream (inner) transducers must be stateless (or of type ScanEmit) and the reducing function must be associative.  See also: Parallel processing tutorial.
Note that the upstream (outer) transducers need not to be stateless as long as it is called with non-parallel reduction such as foldl and collect.
Examples
julia> using Transducers
+ 2
source
Transducers.TCat — Type
TCat(basesize::Integer)
Threaded version of Cat (concatenate/flatten).
To use this transducer, all the downstream (inner) transducers must be stateless (or of type ScanEmit) and the reducing function must be associative.  See also: Parallel processing tutorial.
Note that the upstream (outer) transducers need not to be stateless as long as it is called with non-parallel reduction such as foldl and collect.
Examples
julia> using Transducers
 
 julia> 1:3 |> Map(x -> 1:x) |> TCat(1) |> tcollect
 6-element Vector{Int64}:
@@ -742,7 +742,7 @@
  3
  4
  5
- 6
source
Transducers.Take — Type
Take(n)
Take n items from the input sequence.
This API is modeled after take in Clojure.
Examples
julia> using Transducers
 
 julia> 1:10 |> Take(2) |> collect
 2-element Vector{Int64}:
@@ -761,7 +761,7 @@
 Reduced([1, 2, 3])
 
 julia> transduce(Take(3), Completing(push!!), Init, 1:0)
-InitialValue(push!!)
See also transduce, Reduced and Completing.
source
Transducers.TakeLast — Type
TakeLast(n)
Take last n items from the input sequence.
Examples
julia> using Transducers
 
 julia> 1:10 |> TakeLast(2) |> collect
 2-element Vector{Int64}:
@@ -771,18 +771,18 @@
 julia> 1:2 |> TakeLast(5) |> collect
 2-element Vector{Int64}:
  1
- 2
source
Transducers.TakeNth — Type
TakeNth(n)
Output every n item to the inner reducing step.
This API is modeled after take-nth in Clojure.
Examples
julia> using Transducers
 
 julia> 1:9 |> TakeNth(3) |> collect
 3-element Vector{Int64}:
  1
  4
- 7
source
Transducers.TakeWhile — Type
TakeWhile(pred)
Take items while pred returns true.  Abort the reduction when pred returns false for the first time.
This API is modeled after take-while in Clojure.
Examples
julia> using Transducers
 
 julia> [1, 2, 3, 1, 2] |> TakeWhile(x -> x < 3) |> collect
 2-element Vector{Int64}:
  1
- 2
source
Transducers.Unique — Type
Unique(by = identity)
Pass only unseen item to the inner reducing step.
The item is distinguished by the output of function by when given.
This API is modeled after distinct in Clojure.
Transducers.jl 0.4.2
New in version 0.4.2.
Examples
julia> using Transducers
+ 2
source
Transducers.Unique — Type
Unique(by = identity)
Pass only unseen item to the inner reducing step.
The item is distinguished by the output of function by when given.
This API is modeled after distinct in Clojure.
Transducers.jl 0.4.2
New in version 0.4.2.
Examples
julia> using Transducers
 
 julia> [1, 1, 2, -1, 3, 3, 2] |> Unique() |> collect
 4-element Vector{Int64}:
@@ -795,13 +795,13 @@
 3-element Vector{Int64}:
  1
  2
- 3
source
Transducers.Zip — Method
Zip(xforms...)
Zip outputs of transducers xforms in a tuple and pass it to the inner reduction step.
Warning
Head transducers drive tail transducers.  Be careful when using it with transducers other than Map, especially the contractive ones like PartitionBy and the expansive ones like MapCat.
Examples
julia> using Transducers
+ 3
source
Transducers.Zip — Method
Zip(xforms...)
Zip outputs of transducers xforms in a tuple and pass it to the inner reduction step.
Warning
Head transducers drive tail transducers.  Be careful when using it with transducers other than Map, especially the contractive ones like PartitionBy and the expansive ones like MapCat.
Examples
julia> using Transducers
 
 julia> collect(Zip(Map(identity), Map(x -> 10x), Map(x -> 100x)), 1:3)
 3-element Vector{Tuple{Int64, Int64, Int64}}:
  (1, 10, 100)
  (2, 20, 200)
- (3, 30, 300)
source
Experimental transducers
Transducers.ZipSource — Type
ZipSource(xform::Transducer)
Branch input into two "flows", inject one into xform and then merge (zip) the output of xform with the original (source) input.
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
To illustrate how it works, consider the following usage
collection |> xf0 |> ZipSource(xf1) |> xf2
where xf0, xf1, and xf2 are some transducers.  Schematically, the output yn from xfn flows as follows:
xf0      xf1                       xf2
+ (3, 30, 300)
source
Experimental transducers
Transducers.ZipSource — Type
ZipSource(xform::Transducer)
Branch input into two "flows", inject one into xform and then merge (zip) the output of xform with the original (source) input.
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
To illustrate how it works, consider the following usage
collection |> xf0 |> ZipSource(xf1) |> xf2
where xf0, xf1, and xf2 are some transducers.  Schematically, the output yn from xfn flows as follows:
xf0      xf1                       xf2
 ---- y0 ------ y1 ---.-- (y0, y1) ----->
       |              |
        `-------------'
Examples
julia> using Transducers
@@ -811,7 +811,7 @@
 3-element Vector{Tuple{Int64, Int64}}:
  (1, 2)
  (3, 4)
- (5, 6)
source
Transducers.GetIndex — Type
GetIndex(array)
 GetIndex{inbounds}(array)
Transform an integer input i to array[i].
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
Examples
julia> using Transducers
        using Transducers: GetIndex
 
@@ -825,7 +825,7 @@
 3-element Vector{Char}:
  'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)
  'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)
- 'd': ASCII/Unicode U+0064 (category Ll: Letter, lowercase)
source
Transducers.SetIndex — Type
SetIndex(array)
 SetIndex{inbounds}(array)
Perform array[i] = v for each input pair (i, v).
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
Examples
julia> using Transducers
        using Transducers: SetIndex
 
@@ -837,7 +837,7 @@
 3-element Vector{Float64}:
  11.1
   0.0
- 33.3
source
Transducers.Inject — Type
Inject(iterator)
Inject the output from iterator to the stream processed by the inner reduction step.
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
Examples
julia> using Transducers
+ 33.3
source
Transducers.Inject — Type
Inject(iterator)
Inject the output from iterator to the stream processed by the inner reduction step.
Warning
This API is experimental.  Backward incompatible change, including the removal of this API, is more likely to occur than other parts of this package.
Examples
julia> using Transducers
        using Transducers: Inject
 
 julia> collect(Inject(Iterators.cycle("hello")), 1:8)
@@ -865,7 +865,7 @@
  (3, (1, 4))
  (4, (2, 4))
  (5, (1, 5))
- (6, (2, 5))
source
Other reducing function combinators
Other reducing function combinators
Transducers.TeeRF — Type
TeeRF(reducing_functions::Tuple)
 TeeRF(reducing_functions...)
Combine multiple reducing functions into a new reducing function that "multicast" the input to multiple reducing functions.
Roughly speaking, TeeRF(op₁, op₂, ..., opₙ) is equivalent to
((a₁, a₂, ..., aₙ), x) -> (op₁(a₁, x), op₂(a₂, x), ..., opₙ(aₙ, x))
For combine, it behaves like ProductRF.
Transducers.jl 0.4.32
New in version 0.4.32.
Examples
julia> using Transducers
 
 julia> extrema′(xs, xf = Map(identity)) = foldl(TeeRF(min, max), xf, xs);
@@ -880,7 +880,7 @@
 ERROR: EmptyResultError: ...
 
 julia> foldl(TeeRF(min, filtering_max), Map(identity), 2:2:8; init = Init)
-(2, InitialValue(max))
source
Transducers.ProductRF — Type
ProductRF(reducing_functions::Tuple)
 ProductRF(reducing_functions...)
Combine N reducing functions into a new reducing function that work on N-tuple.  The i-th reducing function receives the i-th element of the input tuple.
Roughly speaking, ProductRF(op₁, op₂, ..., opₙ) is equivalent to
((a₁, a₂, ..., aₙ), (b₁, b₂, ..., bₙ)) -> (op₁(a₁, b₁), op₂(a₂, b₂), ..., opₙ(aₙ, bₙ))
Transducers.jl 0.4.32
New in version 0.4.32.
Examples
Like TeeRF, ProductRF can be used to drive multiple reducing functions.  ProductRF is more "low-level" in the sense that TeeRF can be defined in terms of ProductRF (other direction is much harder):
julia> using Transducers
 
 julia> TeeRF′(fs...) = reducingfunction(
@@ -893,7 +893,7 @@
 (false, 24)
 
 julia> foldl(TeeRF(reducingfunction(Map(isodd), &), +), Map(identity), [5, 2, 6, 8, 3])
-(false, 24)
source
Transducers.wheninit — Function
wheninit(oninit, rf) -> rf′
 wheninit(oninit) -> rf -> rf′
 whenstart(start, rf) -> rf′
 whenstart(start) -> rf -> rf′
@@ -946,7 +946,7 @@
 3.0
 
 julia> foldxt(averaging2, Filter(isodd), 1:50; basesize = 1)
-25.0
source
Early termination
Transducers.Reduced — Type
Reduced
The type signaling transducible processes to abort.
Note
Call reduced function for aborting the transducible process since reduced makes sure x is not doubly wrapped.  Reduced is meant to be used as x isa Reduced for checking if the result from transduce is due to early termination.
See reduced, unreduced.
Examples
julia> using Transducers
+25.0
source
Early termination
Transducers.Reduced — Type
Reduced
The type signaling transducible processes to abort.
Note
Call reduced function for aborting the transducible process since reduced makes sure x is not doubly wrapped.  Reduced is meant to be used as x isa Reduced for checking if the result from transduce is due to early termination.
See reduced, unreduced.
Examples
julia> using Transducers
 
 julia> function step_demo(y, x)
            if x > 5
@@ -972,7 +972,7 @@
 false
 
 julia> unreduced(result)
-10
source
Transducers.reduced — Function
reduced([x = nothing])
Stop transducible process with the final value x (default: nothing).  Return x as-is if it already is a reduced value.
See Reduced, unreduced.
This API is modeled after ensure-reduced in Clojure.
Examples
julia> using Transducers
 
 julia> foldl(Enumerate(), "abcdef"; init=0) do y, (i, x)
            if x == 'd'
@@ -989,7 +989,7 @@
            end
        end;
 1 a
-2 b
source
Transducers.ifunreduced — Function
ifunreduced(f, [x])
Equivalent to unreduced(x) if x is a Reduced; otherwise run f(x).  Return a curried version if x is not provided.
See: foreach.
Examples
julia> using Transducers
 
 julia> 1 |> ifunreduced() do x
            println("called with x = ", x)
@@ -1001,16 +1001,16 @@
        end
 1
Notice that nothing is printed in the last example.
Implementation
ifunreduced(f) = x -> ifunreduced(f, x)
 ifunreduced(f, x::Reduced) = unreduced(x)
-ifunreduced(f, x) = f(x)
source
Executors
Transducers.SequentialEx — Type
SequentialEx(; simd)
Sequential fold executor.  It can be passed to APIs from packages such as Folds.jl and FLoops.jl to run the algorithm sequentially.
See also: foldxl, ThreadedEx and DistributedEx.
Keyword Arguments
  • simd: If true or :ivdep, enable SIMD using Base.@simd.  If :ivdep, use @simd ivdep for ... end variant.  Read Julia manual of Base.@simd to understand when it is appropriate to use this option.  For example, simd = :ivdep must not be used with stateful transducer like Scan.  If false (default), Base.@simd is not used.
Examples
julia> using Folds
+ifunreduced(f, x) = f(x)
source
Executors
Transducers.SequentialEx — Type
SequentialEx(; simd)
Sequential fold executor.  It can be passed to APIs from packages such as Folds.jl and FLoops.jl to run the algorithm sequentially.
See also: foldxl, ThreadedEx and DistributedEx.
Keyword Arguments
  • simd: If true or :ivdep, enable SIMD using Base.@simd.  If :ivdep, use @simd ivdep for ... end variant.  Read Julia manual of Base.@simd to understand when it is appropriate to use this option.  For example, simd = :ivdep must not be used with stateful transducer like Scan.  If false (default), Base.@simd is not used.
Examples
julia> using Folds
 
 julia> Folds.sum(1:3, SequentialEx())
-6
source
Transducers.ThreadedEx — Type
ThreadedEx(; basesize, stoppable, nestlevel, simd)
Multi-threaded fold executor.  This is the default [1] parallel executor used by Folds.jl and FLoops.jl.
See also: foldxt, SequentialEx and DistributedEx.
Keyword Arguments
  • basesize::Integer = amount(reducible) ÷ nthreads(): A size of chunk in reducible that is processed by each worker.  A smaller size may be required when:
    • computation time for processing each item fluctuates a lot
    • computation can be terminated by reduced or transducers using it, such as ReduceIf
  • stoppable::Bool: [This option usually does not have to be set manually.]  The threaded fold executed in the "stoppable" mode used for optimizing reduction with reduced has a slight overhead if reduced is not used.  This mode can be disabled by passing stoppable = false.  It is usually automatically detected and set appropriately.  Note that this option is purely for optimization and does not affect the result value.
  • nestlevel::Union{Integer,Val}: Specify how many inner Cat (flatten) transducers to be multi-threaded (using TCat). It must be a positive integer, Val of positive integer, or Val(:inf).  Val(:inf) means to use multi-threading for all Cat transducers.  Note that Cat transducer should be statically known. That is to say, the fold implementation sees two Cats in ... |> Map(f) |> Cat() |> Cat() but only one Cat in ... |> Map(x -> f(x) |> Cat()) |> Cat() even though they are semantically identical.
  • simd: If true or :ivdep, enable SIMD using Base.@simd.  If :ivdep, use @simd ivdep for ... end variant.  Read Julia manual of Base.@simd to understand when it is appropriate to use this option.  For example, simd = :ivdep must not be used with stateful transducer like Scan.  If false (default), Base.@simd is not used.
Examples
julia> using Folds
+6
source
Transducers.ThreadedEx — Type
ThreadedEx(; basesize, stoppable, nestlevel, simd)
Multi-threaded fold executor.  This is the default [1] parallel executor used by Folds.jl and FLoops.jl.
See also: foldxt, SequentialEx and DistributedEx.
Keyword Arguments
  • basesize::Integer = amount(reducible) ÷ nthreads(): A size of chunk in reducible that is processed by each worker.  A smaller size may be required when:
    • computation time for processing each item fluctuates a lot
    • computation can be terminated by reduced or transducers using it, such as ReduceIf
  • stoppable::Bool: [This option usually does not have to be set manually.]  The threaded fold executed in the "stoppable" mode used for optimizing reduction with reduced has a slight overhead if reduced is not used.  This mode can be disabled by passing stoppable = false.  It is usually automatically detected and set appropriately.  Note that this option is purely for optimization and does not affect the result value.
  • nestlevel::Union{Integer,Val}: Specify how many inner Cat (flatten) transducers to be multi-threaded (using TCat). It must be a positive integer, Val of positive integer, or Val(:inf).  Val(:inf) means to use multi-threading for all Cat transducers.  Note that Cat transducer should be statically known. That is to say, the fold implementation sees two Cats in ... |> Map(f) |> Cat() |> Cat() but only one Cat in ... |> Map(x -> f(x) |> Cat()) |> Cat() even though they are semantically identical.
  • simd: If true or :ivdep, enable SIMD using Base.@simd.  If :ivdep, use @simd ivdep for ... end variant.  Read Julia manual of Base.@simd to understand when it is appropriate to use this option.  For example, simd = :ivdep must not be used with stateful transducer like Scan.  If false (default), Base.@simd is not used.
Examples
julia> using Folds
 
 julia> Folds.sum(1:3, ThreadedEx(basesize = 1))
-6
source
Transducers.DistributedEx — Type
DistributedEx(; pool, basesize, threads_basesize, simd)
Distributed fold executor.  It can be passed to APIs from packages such as Folds.jl and FLoops.jl to run the algorithm sequentially.
See also: foldxd, SequentialEx and ThreadedEx.
Keyword Arguments
  • pool::AbstractWorkerPool: Passed to Distributed.remotecall.
  • basesize::Integer = amount(array) ÷ nworkers(): A size of chunk in array that is processed by each worker.  A smaller size may be required when computation time for processing each item can fluctuate a lot.
  • threads_basesize::Integer = basesize ÷ nthreads(): A size of chunk in array that is processed by each task in each worker process. The default setting assumes that the number of threads used in all workers are the same.  For heterogeneous setup where each worker process has different number of threads, it may be required to use smaller threads_basesize and basesize to get a good performance.
  • simd: If true or :ivdep, enable SIMD using Base.@simd.  If :ivdep, use @simd ivdep for ... end variant.  Read Julia manual of Base.@simd to understand when it is appropriate to use this option.  For example, simd = :ivdep must not be used with stateful transducer like Scan.  If false (default), Base.@simd is not used.
Examples
julia> using Folds
+6
source
Transducers.DistributedEx — Type
DistributedEx(; pool, basesize, threads_basesize, simd)
Distributed fold executor.  It can be passed to APIs from packages such as Folds.jl and FLoops.jl to run the algorithm sequentially.
See also: foldxd, SequentialEx and ThreadedEx.
Keyword Arguments
  • pool::AbstractWorkerPool: Passed to Distributed.remotecall.
  • basesize::Integer = amount(array) ÷ nworkers(): A size of chunk in array that is processed by each worker.  A smaller size may be required when computation time for processing each item can fluctuate a lot.
  • threads_basesize::Integer = basesize ÷ nthreads(): A size of chunk in array that is processed by each task in each worker process. The default setting assumes that the number of threads used in all workers are the same.  For heterogeneous setup where each worker process has different number of threads, it may be required to use smaller threads_basesize and basesize to get a good performance.
  • simd: If true or :ivdep, enable SIMD using Base.@simd.  If :ivdep, use @simd ivdep for ... end variant.  Read Julia manual of Base.@simd to understand when it is appropriate to use this option.  For example, simd = :ivdep must not be used with stateful transducer like Scan.  If false (default), Base.@simd is not used.
Examples
julia> using Folds
 
 julia> Folds.sum(1:3, DistributedEx())
-6
source
Transducers.PreferParallel — Type
PreferParallel(; simd, basesize)
A "placeholder" executor that indicates preference to parallel execution.
This lets the input data collection decide preferred execution strategy (e.g., CUDAEx for CuArray when FoldsCUDA.jl is available), assuming that the reducing function is associative. The default executor is ThreadedEx. As an optional feature, some input data collections support (e.g., AbstractChannel) automatically demoting the execution strategy to SequentialEx.  An error is thrown if the automatic detection fails,
source
Miscellaneous
Transducers.SplitBy — Type
SplitBy(f; [keepend = false,] [keepempty = false,])
Split input collection into chunks delimited by the elements on which f returns true. This can be used to implement parallel and lazy versions of functions like eachline and split.
If keepend is true (or Val(true)), include the "delimiter"/end element at the end of each chunk. If keepempty is true (or Val(true)), include empty chunks. When keepend is true, the value of keepempty is irrelevant since the chunks cannot be empty (i.e., it at least contains the end).
The input collection (xs in SplitBy(...)(xs)) has to support eachindex and view or SubString.
Extended Help
Examples
For demonstration, consider the following input stream and SplitBy(iszero; ...) used with the following options:
input     keepend=false       keepend=false        keepend=true
+6
source
Transducers.PreferParallel — Type
PreferParallel(; simd, basesize)
A "placeholder" executor that indicates preference to parallel execution.
This lets the input data collection decide preferred execution strategy (e.g., CUDAEx for CuArray when FoldsCUDA.jl is available), assuming that the reducing function is associative. The default executor is ThreadedEx. As an optional feature, some input data collections support (e.g., AbstractChannel) automatically demoting the execution strategy to SequentialEx.  An error is thrown if the automatic detection fails,
source
Miscellaneous
Transducers.SplitBy — Type
SplitBy(f; [keepend = false,] [keepempty = false,])
Split input collection into chunks delimited by the elements on which f returns true. This can be used to implement parallel and lazy versions of functions like eachline and split.
If keepend is true (or Val(true)), include the "delimiter"/end element at the end of each chunk. If keepempty is true (or Val(true)), include empty chunks. When keepend is true, the value of keepempty is irrelevant since the chunks cannot be empty (i.e., it at least contains the end).
The input collection (xs in SplitBy(...)(xs)) has to support eachindex and view or SubString.
Extended Help
Examples
For demonstration, consider the following input stream and SplitBy(iszero; ...) used with the following options:
input     keepend=false       keepend=false        keepend=true
           keepempty=false     keepempty=true
 
 1           `.                  `.                  `.
@@ -1046,7 +1046,7 @@
  [1, 2, 3, 0]
  [1, 2, 3, 4, 0]
  [0]
- [1, 2]
source
Transducers.Transducer — Method
Transducer(iterator::Iterators.Generator)
 Transducer(iterator::Iterators.Filter)
 Transducer(iterator::Iterators.Flatten)
Extract "processing" part of an iterator as a Transducer.  The "data source" iterator (i.e., xs in (f(x) for x in xs)) is ignored and nothing must be used as a place holder (i.e., (f(x) for x in nothing)).
See also eduction.
Transducers.jl 0.3
New in version 0.3.
Examples
julia> using Transducers
 
@@ -1056,7 +1056,7 @@
 
 julia> xs = 1:10
        collect(xf1, xs) == collect(xf2, xs)
-true
source
Transducers.reducingfunction — Function
reducingfunction(xf, step; simd)
 xf'(step; simd)
Apply transducer xf to the reducing function step to create a new reducing function.
Transducers.jl 0.3
New in version 0.3.
Warning
Be careful using reducingfunction with stateful transducers like Scan with mutable init (e.g., Scan(push!, [])).  See more in Examples below.
Arguments
  • xf::Transducer: A transducer.
  • step: A callable which accepts 1 and 2 arguments.  If it only accepts 2 arguments, wrap it with Completing to "add" 1-argument form (i.e., complete protocol).
Keyword Arguments
Examples
julia> using Transducers
 
 julia> rf = reducingfunction(Map(x -> x + 1), *);
@@ -1084,7 +1084,7 @@
 CopyInit(Any[])
 
 julia> transduce(rf_good, "", 1:3)
-"112123"
source
Transducers.Init — Constant
Init
 Init(op) :: InitialValues.InitialValue
The canonical initializer; i.e., a singleton placeholder usable for init argument of foldl for binary functions with known initial values.
When init = Init is passed to foldl etc., Init(op) is called for the bottom reducing function op during the start phase. Init(op) returns InitialValue(op) which acts as the canonical initial value of op.
Examples
julia> using Transducers
 
 julia> foldl(+, 1:3 |> Filter(isodd); init = Init)
@@ -1104,7 +1104,7 @@
 julia> acc = Transducers.start(rf, Init);
 
 julia> Transducers.unwrap(rf, acc)
-(3, InitialValue(+))
source
Transducers.OnInit — Type
OnInit(f)
Call a callable f to create an initial value.
See also CopyInit.
OnInit or CopyInit must be used whenever using in-place reduction with foldxt etc.
Examples
julia> using Transducers
+(3, InitialValue(+))
source
Transducers.OnInit — Type
OnInit(f)
Call a callable f to create an initial value.
See also CopyInit.
OnInit or CopyInit must be used whenever using in-place reduction with foldxt etc.
Examples
julia> using Transducers
 
 julia> xf1 = Scan(push!, [])
 Scan(push!, Any[])
@@ -1144,7 +1144,7 @@
 3-element Vector{Char}:
  'a': ASCII/Unicode U+0061 (category Ll: Letter, lowercase)
  'b': ASCII/Unicode U+0062 (category Ll: Letter, lowercase)
- 'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)
source
Transducers.CopyInit — Type
CopyInit(value)
This is equivalent to OnInit(() -> deepcopy(value)).
Transducers.jl 0.3
New in version 0.3.
Examples
julia> using Transducers
+ 'c': ASCII/Unicode U+0063 (category Ll: Letter, lowercase)
source
Transducers.CopyInit — Type
CopyInit(value)
This is equivalent to OnInit(() -> deepcopy(value)).
Transducers.jl 0.3
New in version 0.3.
Examples
julia> using Transducers
 
 julia> init = CopyInit([]);
 
@@ -1158,21 +1158,21 @@
 3-element Vector{Any}:
  1
  2
- 3
source
Transducers.right — Function
right([l, ]r) -> r
It is simply defined as
right(l, r) = r
 right(r) = r
This function is meant to be used as step argument for foldl etc. for extracting the last output of the transducers.
Transducers.jl 0.3
Initial value must be manually specified.  In 0.2, it was automatically set to nothing.
Examples
julia> using Transducers
 
 julia> foldl(right, Take(5), 1:10)
 5
 
 julia> foldl(right, Drop(5), 1:3; init=0)  # using `init` as the default value
-0
source
Transducers.setinput — Function
setinput(ed::Eduction, coll)
Set input collection of eduction ed to coll.
Transducers.jl 0.3
Previously, setinput combined with eduction was a recommended way to use transducers in a type stable manner.  As of v0.3, all the foldl-like functions and eduction are type stable for many cases.  This workaround is no more necessary.
Examples
julia> using Transducers
+0
source
Transducers.setinput — Function
setinput(ed::Eduction, coll)
Set input collection of eduction ed to coll.
Transducers.jl 0.3
Previously, setinput combined with eduction was a recommended way to use transducers in a type stable manner.  As of v0.3, all the foldl-like functions and eduction are type stable for many cases.  This workaround is no more necessary.
Examples
julia> using Transducers
 
 julia> ed = eduction(Map(x -> 2x), Float64[]);
 
 julia> xs = ones(2, 3);
 
 julia> foldl(+, setinput(ed, xs))
-12.0
source
Transducers.AdHocFoldable — Type
AdHocFoldable(foldl, [collection = nothing])
Provide a different way to fold collection without creating a wrapper type.
Arguments
  • foldl::Function: a function that implements __foldl__.
  • collection: a collection passed to the last argument of foldl.
Examples
julia> using Transducers
+12.0
source
Transducers.AdHocFoldable — Type
AdHocFoldable(foldl, [collection = nothing])
Provide a different way to fold collection without creating a wrapper type.
Arguments
  • foldl::Function: a function that implements __foldl__.
  • collection: a collection passed to the last argument of foldl.
Examples
julia> using Transducers
        using Transducers: @next, complete
        using ArgCheck
 
@@ -1269,7 +1269,7 @@
        end;
 i = 1
 i = 2
-i = 3
source
Transducers.withprogress — Function
withprogress(foldable) -> foldable′
Wrap a foldable so that progress is shown in logging-based progress meter (e.g., Juno) during foldl, foldxt, foldxd, etc.
For parallel reduction such as foldxt and foldxd, reasonably small basesize and threads_basesize (for foldxd) must be used to ensure that progress information is updated frequently.  However, it may slow down the computation if basesize is too small.
Keyword Arguments
  • interval::Real: Minimum interval (in seconds) for how often progress is logged.
Examples
julia> using Transducers
+i = 3
source
Transducers.withprogress — Function
withprogress(foldable) -> foldable′
Wrap a foldable so that progress is shown in logging-based progress meter (e.g., Juno) during foldl, foldxt, foldxd, etc.
For parallel reduction such as foldxt and foldxd, reasonably small basesize and threads_basesize (for foldxd) must be used to ensure that progress information is updated frequently.  However, it may slow down the computation if basesize is too small.
Keyword Arguments
  • interval::Real: Minimum interval (in seconds) for how often progress is logged.
Examples
julia> using Transducers
 
 julia> xf = Map() do x
            sleep(0.01)
@@ -1288,7 +1288,7 @@
        );
 
 julia> foldxt(+, xf, withprogress(1:10; interval=1e-3); basesize=1)
-220
source
Base.mapfoldl — Function
mapfoldl(xf::Transducer, step, reducible; init, simd)
Warning
mapfoldl(::Transducer, rf, itr) is deprecated.  Use foldl(rf, ::Transducer, itr) if you do not need to call single-argument rf on complete. Use foldl(whencomplete(rf, rf), ::Transducer, itr) to call the single-argument method of rf on complete.
Like foldl but step is not automatically wrapped by Completing.
Examples
julia> using Transducers
+220
source
Base.mapfoldl — Function
mapfoldl(xf::Transducer, step, reducible; init, simd)
Warning
mapfoldl(::Transducer, rf, itr) is deprecated.  Use foldl(rf, ::Transducer, itr) if you do not need to call single-argument rf on complete. Use foldl(whencomplete(rf, rf), ::Transducer, itr) to call the single-argument method of rf on complete.
Like foldl but step is not automatically wrapped by Completing.
Examples
julia> using Transducers
 
 julia> function step_demo(state, input)
            @show state, input
@@ -1304,4 +1304,4 @@
 (state, input) = (0.0, 1)
 (state, input) = (1.0, 3)
 Finishing with state = 4.0
-4.0
source
Base.mapreduce — Function
mapreduce(xf, step, reducible; init, simd)
Warning
mapreduce(::Transducer, rf, itr) is deprecated.  Use foldxt(rf, ::Transducer, itr) if you do not need to call single-argument rf on complete. Use foldxt(whencomplete(rf, rf), ::Transducer, itr) to call the single-argument method of rf on complete.
Like foldxt but step is not automatically wrapped by Completing.
source

+4.0
source
Base.mapreduce — Function
mapreduce(xf, step, reducible; init, simd)
Warning

mapreduce(::Transducer, rf, itr) is deprecated. Use foldxt(rf, ::Transducer, itr) if you do not need to call single-argument rf on complete. Use foldxt(whencomplete(rf, rf), ::Transducer, itr) to call the single-argument method of rf on complete.

Like foldxt but step is not automatically wrapped by Completing.

source
diff --git a/dev/search/index.html b/dev/search/index.html index c9b02d1c..dfb43962 100644 --- a/dev/search/index.html +++ b/dev/search/index.html @@ -1,2 +1,2 @@ -Search · Transducers.jl

Loading search...

    +Search · Transducers.jl

    Loading search...

      diff --git a/dev/tutorials/tutorial_missings/index.html b/dev/tutorials/tutorial_missings/index.html index 95eba6bf..b1e8a7be 100644 --- a/dev/tutorials/tutorial_missings/index.html +++ b/dev/tutorials/tutorial_missings/index.html @@ -182,4 +182,4 @@ collect
      3-element Vector{Pair{Int64, Int64}}:
  0 => 0
  2 => 1
- 2 => 1

      This page was generated using Literate.jl.

      + 2 => 1

      This page was generated using Literate.jl.

      diff --git a/dev/tutorials/tutorial_parallel/index.html b/dev/tutorials/tutorial_parallel/index.html index dd4d3169..0aeb86ff 100644 --- a/dev/tutorials/tutorial_parallel/index.html +++ b/dev/tutorials/tutorial_parallel/index.html @@ -144,4 +144,4 @@ # # Uncomment for demo: # x == 3 ? sleep(0.1) : @show x # give other tasks a chance to finish first return x -end
      3

      This snippet always returns 3, even though the reductions for c = 6 and c = 9 may finish first.

      This page was generated using Literate.jl.

      +end
      3

      This snippet always returns 3, even though the reductions for c = 6 and c = 9 may finish first.

      This page was generated using Literate.jl.

      diff --git a/dev/tutorials/words/index.html b/dev/tutorials/words/index.html index 7e9f33b1..5ff1e156 100644 --- a/dev/tutorials/words/index.html +++ b/dev/tutorials/words/index.html @@ -35,7 +35,7 @@ @test collectwords(" ") == [] @test collectwords("") == [] end
      Test Summary: | Pass  Total  Time
-test set      |    5      5  1.8s

      String-splitting transducer

      Let's make it re-usable by packaging it into a transducer.

      Rather than accumulating words into a vector, we are going to write a transducer that "emits" a word as soon as it is ready. The downstream transducer may choose to record everything or only aggregate, e.g., reduced statistics. To this end, we replace Segment in the original algorithm with

      struct Vacant
+test set      |    5      5  1.9s

      String-splitting transducer

      Let's make it re-usable by packaging it into a transducer.

      Rather than accumulating words into a vector, we are going to write a transducer that "emits" a word as soon as it is ready. The downstream transducer may choose to record everything or only aggregate, e.g., reduced statistics. To this end, we replace Segment in the original algorithm with

      struct Vacant
     l::String
     r::String
 end

      and output the words in the "middle" without accumulating it. So, instead of segment_or_chunk, we now have:

      vacant_or_chunk(c::Char) = c == ' ' ? Vacant("", "") : Chunk(string(c))

      The idea is to create a custom transducer WordsXF that is used as in

      ... |> Map(vacant_or_chunk) |> WordsXF() |> Filter(!isnothing) |> ...

      so that the whole transducer streams non-empty words to the downstream. That is to say, the input stream is first processed by vacant_or_chunk which returns either a Vacant or a Chunk. This is processed by WordsXF() which outputs either a word (a String) or nothing. We are using Filter(!isnothing) in the downstream to simplify the definition of WordsXF.

      We define a function extract(x::Union{Chunk,Vacant}, y::Union{Chunk,Vacant}) -> (output, state). It is something like ⊕ but works with Chunk and Vacant:

      extract(x::Chunk, y::Chunk) = nothing, Chunk(x.s * y.s)
@@ -78,7 +78,7 @@
     @test collect(wordsxf, " ") == []
     @test collect(wordsxf, "") == []
 end
      Test Summary: | Pass  Total  Time
-test set      |    5      5  0.5s

      Word-counting transducer

      We can pipe the resulting words into various transducers.

      using MicroCollections: SingletonDict
+test set      |    5      5  0.6s

      Word-counting transducer

      We can pipe the resulting words into various transducers.

      using MicroCollections: SingletonDict
 
 processcount(word) = SingletonDict(word => 1)
 countxf = opcompose(wordsxf, Map(processcount))
      Map(vacant_or_chunk) ⨟
@@ -99,8 +99,8 @@
     @test countwords("あああ いい あああ あああ いい いい あああ", basesize=basesize) ==
         Dict("あああ" => 4, "いい" => 3)
 end
      Test Summary: | Pass  Total  Time
-basesize = 1  |    7      7  0.6s
+basesize = 1  |    7      7  0.7s
 Test Summary: | Pass  Total  Time
 basesize = 2  |    7      7  0.0s
 Test Summary: | Pass  Total  Time
-basesize = 4  |    7      7  0.0s

      This page was generated using Literate.jl.

      +basesize = 4 | 7 7 0.0s

      This page was generated using Literate.jl.