From 8c600065b08c3e10047ad7a8ca2aa60223be3098 Mon Sep 17 00:00:00 2001 From: Michael Abbott <32575566+mcabbott@users.noreply.github.com> Date: Sun, 8 Dec 2024 12:41:49 -0500 Subject: [PATCH] Re-write "basics" page of docs (#2535) * heading id * re-write of basics page * change readme too? * fix some doctests * Update docs/src/guide/models/basics.md Co-authored-by: Christian Guinard <28689358+christiangnrd@users.noreply.github.com> * evalpoly, Enzyme, heading, withgradient, glorot_uniform, etc * more doctests, shorten fitting example, and a regex I copied from somewhere * maybe I should remember how to run documenter locally * fixes? * bytes on 1.11, and revert a fix * logos & links * move custom-models / advanced.md back to tutorial section * tweaks * more tweaks * more tweaks & typos --------- Co-authored-by: Christian Guinard <28689358+christiangnrd@users.noreply.github.com> --- README.md | 18 +- docs/make.jl | 2 +- docs/src/assets/zygote-crop.png | Bin 0 -> 63244 bytes docs/src/guide/models/basics.md | 489 +++++++++++++----- docs/src/reference/models/layers.md | 13 +- .../models => tutorials}/custom_layers.md | 0 6 files changed, 367 insertions(+), 155 deletions(-) create mode 100644 docs/src/assets/zygote-crop.png rename docs/src/{guide/models => tutorials}/custom_layers.md (100%) diff --git a/README.md b/README.md index 1973874dd0..cc0a7f1803 100644 --- a/README.md +++ b/README.md @@ -22,19 +22,25 @@ Flux is an elegant approach to machine learning. It's a 100% pure-Julia stack, a Works best with [Julia 1.10](https://julialang.org/downloads/) or later. Here's a very short example to try it out: ```julia -using Flux, Plots -data = [([x], 2x-x^3) for x in -2:0.1f0:2] +using Flux +data = [(x, 2x-x^3) for x in -2:0.1f0:2] -model = Chain(Dense(1 => 23, tanh), Dense(23 => 1, bias=false), only) +model = let + w, b, v = (randn(Float32, 23) for _ in 1:3) # parameters + x -> sum(v .* tanh.(w*x .+ b)) # callable +end +# model = Chain(vcat, Dense(1 => 23, tanh), Dense(23 => 1, bias=false), only) opt_state = Flux.setup(Adam(), model) -for epoch in 1:1000 +for epoch in 1:100 Flux.train!((m,x,y) -> (m(x) - y)^2, model, data, opt_state) end -plot(x -> 2x-x^3, -2, 2, legend=false) -scatter!(x -> model([x]), -2:0.1f0:2) +using Plots +plot(x -> 2x-x^3, -2, 2, label="truth") +scatter!(model, -2:0.1f0:2, label="learned") ``` +In Flux 0.15, almost any parameterised function in Julia is a valid Flux model -- such as this closure over `w, b, v`. The same function can also be implemented with built-in layers as shown. The [quickstart page](https://fluxml.ai/Flux.jl/stable/guide/models/quickstart/) has a longer example. See the [documentation](https://fluxml.github.io/Flux.jl/) for details, or the [model zoo](https://github.com/FluxML/model-zoo/) for examples. Ask questions on the [Julia discourse](https://discourse.julialang.org/) or [slack](https://discourse.julialang.org/t/announcing-a-julia-slack/4866). diff --git a/docs/make.jl b/docs/make.jl index 6bdfcbb638..4367639d8e 100644 --- a/docs/make.jl +++ b/docs/make.jl @@ -30,7 +30,6 @@ makedocs( "Quick Start" => "guide/models/quickstart.md", "Fitting a Line" => "guide/models/overview.md", "Gradients and Layers" => "guide/models/basics.md", - "Custom Layers" => "guide/models/custom_layers.md", "Training" => "guide/training/training.md", "Recurrence" => "guide/models/recurrence.md", "GPU Support" => "guide/gpu.md", @@ -63,6 +62,7 @@ makedocs( # Or perhaps those should just be trashed, model zoo versions are newer & more useful. "Linear Regression" => "tutorials/linear_regression.md", "Logistic Regression" => "tutorials/logistic_regression.md", + "Custom Layers" => "tutorials/custom_layers.md", "Model Zoo" => "tutorials/model_zoo.md", #= # "Multi-layer Perceptron" => "tutorials/mlp.md", diff --git a/docs/src/assets/zygote-crop.png b/docs/src/assets/zygote-crop.png new file mode 100644 index 0000000000000000000000000000000000000000..ddc04b3d176836816d31530ec2a7231b1179ef7d GIT binary patch literal 63244 zcmeFZRaBf!5GaVdYjAh>;1CFIL4v!xyE_C35+t}oaMxjQCqQub;O-7PB>AtMy$}1o z!1wKU|`77QsT;BU=XcfVBmLfFrYVup$c1|3%Ikg96xK{?J{nrKp3@qFd z4C-GSEztGtV*vVsdjGpZC4_;&f}S`)-?$v`{|P+@;=i5{tvQhY-GkqO+JT9xh)GL> zt{zT&CVb+e67uXHEk(_2METemt@t>&cv(3)*f=@3I2c{{*x6Zm+1Xg%gBg8qx&n29 zcaYL_1_MLsefxqZF`y8DmO$N7Rl`L?UXIV$-j>h3zBQdE0K!XJDHMlGqWh5zWoKb$X9Bfga`v=yG4x=vbEfzU z%b1&ynn$|4j=tL6)~)SlF0Z zS^nR;nR;0MU%I{h^0(WcdHo$u;0+j`qLZa52;n!hgxCcB4Di3b`=96uyg}hpb2fDn zv$wScb^2dmfgt`_EK!^Pn2a}1ZnW3wV3%St$2R0C3d0RFc z%l~88KX3o{aRpfZZ>au-@xPGbFP;Cu01;XUL4f6dFpUsGid-&1}sUjtZ1J$Ey`QOGpnz%WE0q`=@w!6^fv{{M&ne+~XO zJ_vkI6^B*4)on-?_ygnY-Y4(o<_4?y7x|%s7o||F*v=JNEVtVSHA zBLdY2^IDPLln^y4$y_vlp`~rZ?~?swqdwI5z(<=KqhwU*y5CSNu7; z+&v_67N(cW@+xdRl4_hZz7^)yFD$Cg+t@PN+1k@NcrQB&rek?pX*co}^J%AQr&kPB z?iVsG%rZ2b775|`y5(aK&dG)ci~W@yVlu?V#|`9Hr&OZHQY&53abe9EPoKT z&MC1KEGlvRI;oY#jEN_e7P;WO_x37GI$@MQ1_Eh;Cf29*e%gqf(A$yR>KUm2#4%6` zUF0K=vPsQK9Gp^9u#E9`PUfigfW7@9x10H8W4vq8;$$4Ye84b$>LM$v^;s(!@4yr9 zd*`#Nf!`#zG#Y-k@T^=J5dT4|#son-lB(%4fD15USk>v+=pBtoAD>@%xP|9WcPF%W zb{ouP5@kKUPUQO#_)8^i)_i2WJogZG2T4H`wB+ix-!u< zPA-!(rZea5JGywuJzF8@EZd{M#iyG&gnnEN%ZW0pJ4pjvE80R-(BUn$_??i;OEKH& zokh&*nnC^_ZvLQ|-s(%8#YmxpmDye#lxC0XsW-hWBR5#nbQab4>>-YKNr8vEVH*{0 z&j-Z|$1oB$Jk2Lej5vLBwx*aT;*R;JxJWDEauU$hN8&w#!M8u(JKqmq`c-J%`yX1( zWk3@Qm_+i(zWn@QFYzj0oT#ATAu|-B=%nmade7lRNcW!>JOT0J_85{i6lu=ozH0jI z6W8|Kmn2;KANuaCJ;zebKj6U8&NjDRQXzSQCLgcw^osqG(-X9NqC~ey&@E!gE{+JcS?U+wKQwCf%iQwFGnT3vkn$V1}vf+)$rdDJ~R**w(lVmG8j zP;YRn#wDL#TeJ2VNXYcE!##ZT-iP4|FhJId>>vB~tM4Xe-4ffGZ7pnWIQ$)K{Rh;` z{k2djoL>OcSA`s$4}MHb4R?*(rfd2Wv$^8}ttK`?xd^2iv}7285hAU6?{UySB}Bl9 z=3>L&2=1|ov*cLWYQRV*=}O@q`8P*;uW_`wiC!Q`(H;4_b#f#{Hj(3e5!{kh&}H>J zo8}4ICVup8i$NU-{Dr%yYr$D{tSA@F@y0(l6f z;*@0Uxru`}!tQiU0hiSpijmhOZ0&8-v+Kv>YTBAr1z_o;y5u~C)rP@rNw zHCqAS=Y;^01IF%)jr}~|e|&lNCV@VTn2m+<^63f>UYEtUcmZ z3kM0$lV;hEuUN~NlxrzWz|F%B;q1@33lmiV_{jb6?vmXTNWr%ii}R4@CVXir^b?LGy}KGlVb77eN)- za*sBbPk1oe+Pyp}6Y>T!H9MUIMv^pwhJ%9`UTsDOrrPHPUj0;xKM@RB%7hr@#g|4( zv61$Fc1q)Isb%QF zf4`LK`yVXNQ~{MLS@@}!r{bd$q-AQ83#8OuI9oUDNS2})!STDgHZ5Qa=l0eHamH3* z%#bw&eU~AdZbuH9xc*n55N`~PmJ4;)CZj%MznWbItZsaWbvEEL)ONN4%0p)_ae7+C zezi{rD;^|M$=Q!Y=B~b29PK?uj7`GLv%aH7ZQq+2>DRlLl28u2jzc+z{!dik(BaZ2 zF>}ui>&VbpAEFmg{nB}M0hv*iw;qDsXlZ-}r5dC#+n;48P3Ye3-k`BZUI$U(i7cx+Di)`2MLJrDCA=o7e{yd@$cV*ke4r$+`o{pVrXu&<3* z4)VdRGzG!epCM1+Ba@wYbfPh8hz@2&D7fnvuQtAn9%c&s?&dLz2E6LcH}6P^o4u#` zOSK48h#!0X1AHO2SX@PE3F`jYm$Utg$i?AhpQtFE2NTTiIFdrkzHX zV>sCytt4@H1UtTXF?u~lshp?){SD|>U8@abwT+}N1z)uln-5n@#8@8m)gSMo|6^Dk z5W^zl4!vuvD|8&0$My5vT+uSNU2{MZL*D3O?+RiV=tbG+Os%#JKxDCWekUy}$#Cug zBpY>>JHHD>&A8l)Nu!J*D+sG*|8EV0L2KxZhSW|JgsskQe_S^1)4n~$-zHS`e)?Pm zH`E4N&0!*lfPN%PHJdue!p5G8+QAyGuz-R6fSC8WL!IQC|MGZleX#3);_3ooq^2mS z@hk>pK-f<`p~I=t#^J796kY0s1VU5lPmxKv+@aUrtbr)SF`V^0=zsw{PH54DPVb(* zB{PslJm3}O3Bvvl+7S)NT#hS1S8pPax}J)W*!et?fy^Yb>77wvgbDqmbg5j_C#(

1ivI4j6| z-iJ7~y^oBvNgMQ!Fzfmt!aU<9-1!*yWsxv@pQjY>F6wZ*(o(3(s!}SZ%J+WwL_2Dr zOW7%&GinF%x{{nz0Vuhs?_V$<=k-^*slJvY%0v6#X%g)rP(U!IcYnN~j$`|JkgU`% ziLG6_uKW=f5hH>cJbXaZiV?Z@1cjL=M1$6C)qB|APM~b9AU2QJ?g;U3&~~Q?a%tb| z^_ZzRS4pM_pQ)wV?M4`c!wY?-Wa3RS5fjjv-@~-w&_IO`5f6Ac;TAQ4@ijl2=&>Vo zJ^ab=6>zd22NRU^ml!#cZz@6|#an*+6f^r}`}`gU@DcL}&GLJ-fGf8&P8gNHcy&*w zm()5Kc+`~4L4|zp@in1G>`2!%87@%3b{Xj215GG?-uK^{<#`HY}>i7Ib_Y4?&{o*@w!l=`g@sFx6vP>q2YTu z0?h`0%vxC^T1E6+N-+9cTpS^CB(Rr@as?#)Ws@99klW6qkevFkC3N;&Z#olJ`qG6G zspVj4R!4ak4h?@67%*WMNQoKKD|-p=%viYZ%Pru0BKxg2QImVmJAM{{H&no9TX4eZE6DyGJp;qR&_+(_`j zo95(8N3VN#PJS;dHPV}pGwhD?Giav+Q=Co9{{^t{Z*k5$=2QN|*V~^11j22wVV#~T zq^^pCYNqj=nleb{0lE}{XXrU(gL~9mbv@lJ6E3C_qa!M-d_WNurLGqC&wrh+zz`}B zPDsf(ZqZR$I`{Li3^zXho~$dfAsEU9V-!Q=NCY@Y!sVE2P!`k;2Aju4LsQn-*>?|k zRCrXT=YnE{M1L1aJPKstC&+;vBuY-E94RYx2J+XZn8Ev6Oll9$j-lV5$lm#AgwsG- zaKhjx1zJn170h(JcRUHv+XuQiS43U>HH>i{Z^nExZSu^p4rlbq&U&ioyvrp$f3SDH zxhtTDuZ_~6I$au)G9`x@Sx2T+pm3v5`jQ%}825Bj8TFsHL3$3-h6IIN4H}I5?yn9@ zircNVxjKM?5^uyVHBlybs}B6R6D4B!US#m7I-)1HL*8(b{NWR8%-yWC>`$#UcGSU; zqJP4^z#nf>6~Z6^KLO&YzQ>%RYf;*&4&WPHZ-Iukh{*>zu&yMcVH%h~l%|*?u1}^Z z+KUDUb&tdMmkK?|R*{ax>x_RGvFnWu-&qkJ)PJ{mfwn3SB%5#c{(k>sP zAqU^hLDB(y@T5eWgR&@Jp_gSk_u!=me7Vw!=y$NnZLDUmq>1?L%4(B*E2@smOlS-s$7A}w~XF$tDDF6-_5G|E! z2%-MIu56(k$;0fkRnO_^Cz#?z`_CnPq9$FxJ==1=YDLjK(M7a>5`0I+qn;WnW9P%n z@Rc3~PD%e-S)L5t*=5(Vk^36xypbu~*x2OcM)oDxatrORw313eSWF;eEKih9kW6~F z18rR^YKneauuFPnoAHgPZzo`-K1h*ld4o8GP`vZRG03zjtj@HFw$Nkb@Y^cCSfTPh ziWLg#UqweM3ZL_lCmK;!EMn#HIJ3h+&8dXME}j}SX_>VWnmcJ8$|R1cT3ox=Gv6$x z=HWXad9`i&H;3WLQLoTP2e&`mC=9auJuc)T{TzLQ!prTo{Tef|vfn^K-9B`a(F_K5 ziWO-7UIO5yN6A)*ybJ&}+wFn1ruE8G7a=0QeXl3Y5HQfpq+CG$NSI-$=n(cwhsk2M zVCzaxGbare+_UgBLQRG&SK>zqDFb0}y1S#~&d@x#c3GWa;+0CaSSFt{0GI-PN5f`%f#uUP6ETcFGN_m0I6}+H)^S zDGGptUVbUCwVd~ueU?3WfJoFGn$R9l{7b8VDk#>24{!{($6xz+I?hJERf%E#ETXg; zDimtrJj6?$$2dbLaTFE!+%zDcp_k$h+Yw%rdfnxJHVU>43j?Qo6)AM;9c0czd4U$B zOhrlVQIvYPS8?f!ita-l@DBfOCa*b9*XG5g%e8mw(&^x#H?Kxt(DfuLg_?@W5%JeZ z2`-kzUlFyy?9|^PjjXED^F8rz@7T4LlBPGc4U$mLkzs!LU5kOJI|dF9!?)tL8Q`F* zn(Hcxu)tiHY&{9ioR3a*tm0>CL>CWlY5)v3ujX@#c^xiPOk9!7=`p479&P9aoFatc zjWj2y2zhUo^Qs5cjxde0I-}A?{Ph?ns6na|t(O4_JNWeOpmip*Qeh{jNp*-ROuI~m z?Zg)z!n2cJwYbLnxc8(MlYq7{Gmd%ox}I3!PeY1w{CIi2sWLz6K^swFG8 z)qy(`96B_a#JH9X4fiCf`-QmDbhTJ#*B@o==UTcRlS@GyG$qmqG4Pilek&rH1GEmhg)?On$Sz9K%c;;YaEedO^Ke`1F1xl)AN6=@oP@ zuUwg`F|#w_z2iFTX^2QNzqMHne}d{|E+< zE?{RIco{Xf52kgFQ#i2J%o#JZAo|#bhA<4(bQ0r+najUZLzoJ-VlZQOLTZ#7Q81(o za*(%BZQV>PeQpgPEDg|MdzdXPc7Gh5=y^d}v_dGk+4mw_!D99q{~?a>o0>4lT=cO$ z;1_D~!?O`GcB@V5`ubqxO02a4QGE=St!-Q!`CW8q8~>J5WD5Sd2C=(<%e_7UCmqGd zlZ#R|>*V%Oeos2w&}lk!5OL8?#T&7BwzJlosCgnnbODHyxRSLY+!Kk0@FzstOaWPb z@Y$W1B$iQYP97{MH2yO-dALsnqqj6z@6V`-2}wns?PAnov%qNe{1}>t z3?UBf5QbC>Fxf&_dYoJ!@PZb*H$fiuJNYjF2gRntK#{Moz2G?JK&qXUs>?YQwUv{hXcqvs6~y zgwuSm4qpXp@pGWa@1QnwyIeG4h#Fb(gk|!0xUZoKq6zcImk&@l;dE2WmOojSM)9b_ z*+z{xiqnpZIO}>ctQxfIStTD<@-Sl--J|=h6ktCxLo&8q1tE@1z$;mM$cDe{K}g=4 zsGdhg?PgrEkLH*Oe}QZNm$q~iC$$ppcH}fK%AHQLlC^)i7?$TwiQ%Oa=e7|T386*Q zcp)=zc73p~nfl&TL(3^>?`So2(~kGdgIk1iI+$oaQ^YWJQ1{#OS~IO2MoFXF?lTOP z`Y)?WCT9CG0kha#r(QkZdp2xa?=Wsne8jxm-&51QoQsltcjwC6xFhlKzYwdEQC8xG zk9JlJTPQvYu$$gM<4NPq+V!4M{1azLh&u+4kdI)d;w~dW*0ZOx$Apd{v9WhJ$lz*F z;0HBF6u&m=HJIUH3s>s?E zM5$6bBaidD&Ftl|$-3~-%V^;HMNrteUXxtt1JZ{TK%&@9-{Y3m%)#izTOlHq}dy9F8 z2j8{j_MmH=I#Ow=hoY;4bHJ+Q?~u>c>ywFX@r?J2-CyEEX&8lTJbor@T}W0z^l=*o z9XHlDnb?+V@|((!lI+@+>-*QPn5KkBQMqmms+fshz?&&%IJk~RxoL?`*rIgDkf~)& zp8f7k87KJ8=77mU{UuoZjr*@yLCS-;e*J3p;v-3RQ}N8=PeQOJrOZRra~CBJ!GkzE zc54fJ2t+#*ES(am1H3H$&D~EQjgyJ=T{hjJ(}KQafxR31*vG?&YncCbDld-B1S;k~_O3O#7op6ll7vYbKG@X?k_qdp3>)T3kIzS8k=UHt+bzf}Z zoqv)j2~P3Lyqw)U)?eve)J1?gVo}wX4^5eJa-a+cE1jgo!Tf{|`wgq4Q zuWD0a7Eng7^JIW|dY zZ-Ru@)KCIU@@U(L$;{4^n#rDcJi6aSd4HSvmf`V$q?Cbd(5k?@Z1TVB+_cqS2Dd|= zR00$(bUUe~8qlA3?mD<1EgngX?FRuC0Z;Q~+3l8~OdddQi{l>>$si;;N$fC3Dxa>p zk6qD}qfL1&dW%-m^e8nMjnNg~;Q^L5+*S7Awj?+X`1xA*da2A&zCWEbZ#u38j@x=P z?1tpNj*I_bImOl95w0~E$V(@*la#{IYmKi(%L^S`??xUGhKhDWENSBYKx%TrmaRxH zxE-ZwKGs4!w!IfhcyGSv5%Sj+fdv^ZTmy%g+Biu5dCQE+Eaf&K)MmOp_y(66yGg`e z7!0m=EQDS|vMLXmkI6DgqfGK@IsM*)yI$!+r};6b0*x-aJaf&0Cpp-9_)L_jPi`d) zD9QeC8X3n0{W>Ixwc3eVwbw(=Ssm5IcZsiJ$bY5u{A11lL+f@0f7sg7_4VNfx4^f& zoJ3lw$RgJE&ul^YZ2XJ{eYRwFQY!>s6N;ql+pfNk=(jn{Z##Ewnj%W33j$++55Q;r zd*iX?QDpy2LIgzouxJUhPFBxXrGoREvI^JrqOIp$`F@H*)dlX$D^2w+YAy)_+OW52 zM}P$qSL_A3Dtq(>$xwn$m^nK!!+?bbFSZS+Vq`H47{CO&{?V3# zaUx2>A(5R!Ml}VxFWZSSQ9Z-IztiHPB+z7?T8ia-OvSQ0q_#Ro zs}e3q7_vjL{4%s@vZOY!%6d47uHGJjvmB=srlYw#RkBm@u$DgJ&H`YZ@1OjpBKDV937VWSQ;RV|YTr5>g;UqFpi|JvTMacyUIGPddj=BJm_t%5w&3VAkwGO{ajriV z7)qKrG1915zk~MbGUtf=M<^U_w|J{h!gd(2eu7n_ys4ucSTrf6>*GC!rc$52k9Eo8 zPn&xf;eM0a+Hybq?eefo=eae~BgmlCp-ZH8+N^pTUV%@|*4d#XRwvq%n8;Qc72}63aL^aT<{p!@o3q`Sj)0;Zjj9XE1$9+p@y18S*^Ix^F?bep| zJ3dr2A*V3N!V=$#4erB5qU05|_*`Dw(0e~gYwhpsZ=I&Yd|q7({_$1q@7&i}Ap|8R zpsg#7h9Avm#aMldJftnH#}E&J&zIxxT_fA~DtQ!MD|DJ>z5*CD;W zM+Xj9lk#$=c+0fBIJi^g$I5k3rw8WFU$0!W=L4%Qzh~@~Qh=fpv~Vzz^QdsL+urR7Y%68RUac0)CKRKITj1uA%hICL(YYy>^0 zwfp3-;0Q~{!8b&A|JMC{*bVYbI>CG4{LX)#;~x1h3dYvAEbYvxuBv&1_OveM=}GEp z@{E+)O61uANBYvv8Y^aWQse{5s`G`v9bd&b9-;@AB_SgX1Y+aQj@@gnDj`@I?A;Fd zmSM)hci)FgI&N&#&eScpxQ|b^5e=?tAt8}oRv28O7+SeC9ub-b_Y-#Ic^0PtgAZfj8o(9AS2lm zd>-S)dyswe$-6+ zT8aeJ)v_$zXuBj}=V~#_;y(PM@+f!5UJBViw)s^gy@KgA%iiwQuqxLHZLaH+zwSX( z)b<(_$Rn||u&3&0wc8{}^Zv;)N9@0Kgnm!q|KJ{b_g9OVV@g?kyzPlOeYpkYr-Zs| zdtEN^Rzmu3KZr%By|ae%-BJFH5qce2m!0aP^Rk!SGA*{yOJ#6|$BD|)cKb*)x}>nq zC#&iPppoE-;MUOY-UKYoOm!aRma*Ww?OW5NWZiHrr#iX=W5@r@G)s=u5Le*RK%295jT>#5ULk78`EpPE_0V?%%$NDHJ z7&$mGIsa2sC6b3JrSZ{ysPZF^VADZKlLihey7Kd6eY@KFx1}z&EzKPT^b0&aw9Q$beVXMT3jnKzx4FBwS2U?(mu)Zx-e61?)*H}q`Hu0Cb zWNv%w5HO}1o<*`MOIMC zQ(>S|mx8bC_U5*AMYSa6^SW<}67^x+?Ma>dOvkH!T8(jvMHuQUHWu>gssc>Wrlg<8 z`sG@gpHTK8Mo|w;gU)ge4jKWn=B+nMxp#8=zz6?1y!Q`0Sf$!B5F;<*}sUrk0hvc~;dxFgN(-$g_Sufi)&l3Jg z@x2|Faq#2_f0Fn&m{L+7-4{30U%|4w*%-@~`3oZkKgx;XX{1H9B8|3B!+j=YrqWlb zJ{%n)$?A2NWu|Q^-4XT!YIPiimo0+W0vu+dcHXJ$nME^DYC4yg?di6DirOdMaaJ?C zr%z&U0qSK&)?^jTlxP25Kg-8^Xw1HHcTubelo00aUohQ}(2c)xIF~U#ZKcRLn`G_VSIO_gBk%vva1fBnB2Icef%|1wrOM zajORA%jL*ZbA_xY2`4){c^#AsxL~jQxm`W|`sWet_56Ufy5KVNU_>!$GbR3a(oF-^ zcnR;0Bs#hY;1TWBca`I({p40_I->Wv?DoIw(Xf2D4P$2ivroSUPnnU&Z$5)b!nAaK zoWzKzQa(gu75U{O=ZT2`TC8(y&6j3@qb44*knj7~m#YGiu@l&x3A9$)FFo6%QSB{5 zt=G4B$c^HE(PJ$@yTz-lV?}1z)yy%R!}US{H0TatJj-f*JqX$oJ( z%SC+<c5XC(LhbC`l#aG|LuGbwjo*R+J1`wqkvhP$?xZq|;4X{GnXH zjctWsEy`=w6d^*R5hp^9{(cenON#T8-9Wc?*6a3u3uExON*8KHbhdDP1vk*Y4q-%~ z1nDWBB3ACZ?}VlAm`0}-4e{^jGXHd)#~Z+FgPIz0O!&FDH_|Yfm>)yi^H*3-NZ86K zw>6yy?~~nyQ>d*v?VMpUI(DX%N+ma8@PkSdhj-6pLq{ij;f`Z!yJ{$-yk^&NZH8sx zCMYlYtPSqaPnnL?n-CLG$FpB&<)Y-Xm!>DQre`Zz0cX|67@ZS%ORbUT4yIPoEl)8txTrhIIBFWoRM}F|`8Hw~Voz{Ijm;?IV*5xy zu#uugu^9Fid0qyU@toSNNb^v+YKu#gvy%c{UO&1t3n6@p|&;G|IhkeM%;>f~+@m{Q>L{Ins@Gn$BvrRqZSZe+aE zbWhL$uQhq(-5#_?E`5qcffe(N(hVyF=ZF?TGuL+_oDaTU#-BP5WNT-5)u0#iE_`D* zS3af%5#5*cnc^83M)!T!c;LL3qbuT8aZ3kMN!rg~*SqZ@Agcx*Ta3=Q?l~>-T`k19 z(Bg7KQ74vRnNnax2%t&&Dq4DFAB#DEefD)Y`K9JN`EX>g5ymbRpa_K_5&47j2;XVN zT-_G@4xDE|=~d)_X0Yjly>>FeEe11AMoBP>kL!aps@ztOjxS7b(Z#Xn_K5lyTq=($)aWlZE-!6)eMEh@Cwl)Fxr#ltZEh^u$R6SS<2 zV;OYc*67=u7sYZ&`&UjptggLwc3<0Hdn&2aoi{`$Ca0Htu9J4z=cGoIJACt3klNcM zjBa~l9o3kVwGrz3uv^mF=wsxJ1HwO1P3>(=q><0m(-mrE(reBXO;|)*<#Bi4uUwBB zOGu0h3r;XZ#?acjseGAP%`$S*nB>ctb*WoPw5%qmWh5$c$?r)7|6*N~o6j^dZ#f8c z8(S&Prt3@n)1I88Ef|W6y~0o9gslvA_v@Xp_(`=Ia0=EY@LDROmj_lJ6V>}&V9334 z$J0`VXH=uFE5*peb+Ai^GI7zP?TN*;y@X*ZMH1cUUfuc1>!S(MQ|^~+ePBM)>e=Jz zx108>O`T%BYdq!rM~j1fsgTMgB~MC#E1dk5q&52~@Zzv$DG~t8 zY_1^cJw&J?AdLAH=R8Ekg0t#0Vtnp~E|#>iz(9O&@)kMQZss{%&ALYx#r-;^Nd^^t zeAdIS=-Oj-ZpFlEM>mj#7XxDi1lNU^ljwULbUQ*9`=q9)X{WO~_e4^128JlPs3$Uk zg74u0Ars9KA6FFVqzh|EIaX%G+f4S$ut><__K|MN&+I$bhvSU?PB)Uw-{?4WjM3e4pq|rH9rwtT+gd18NyUT)Om~ z(WP;Qy4jFywpbxhVuQaQeTHbdf*0%$g8D%AOW3(nKj(^>Sl?@L7dq{3 zMHp?Eu*S{^O$zSgZ6MW@dU=_quU8ILptin=ffh+M@wxRP-=N=YFWKV)9gyay|I^Rn z_bWmuz0AwmJY(6f2FLvNuXDTCVU;lm(BgR$Xf>b3pfIGH!|`Wt+T36r@#T9>1FH)R z=iftV&f4_3ulm`~P9vt~#e}%BuW-T)SKb8{PT^KnX*w(L7`;A7QKBU%+oH4RM!j&Q zzV{2ZvnoWW>Rb4ly!u{`CJSkFjz51bx$ZIZcEeJP#^-xNqS5A1v*UNc5U>u;b_&KX z6Xw%;ceA{scCiE}^6vT;%L@1%-xNd5#OOIxyX~wRpTT%bmshHE@5cr`Mh#IB;(W%p zV%zQS?o*;Z=)xYDGe}Qhur*T?&Wqq!+Gs`mUkE|(6o@9!hsH7u^B z_xk`#c@48L%`1b+iEvm@b7d-Pggms~^IkZ_6UjAA^@)js4zGDiTbs)M+R|qDcWXgG z>0o`fH5lP~sOgIzF1Ke0g)bh#In4DE5^;nz%lAS@i4d#wYk;0hkvhpT1Hbu6_1q1cok}a z=tHl=lcOtqFq^7Q{$6q^S9j|+r_U*Dec4b#qQm;U3o8AfXuguA$WOFFv}{uZ z*+K4Ts6Derx6-?y!Z{e3s+E-?f{zT*@4lx8?V-I~Cbh0dY_x?AqQA^t`H~5S!TJ`x z>`9-Yec+U9MgL}f&d^r9KZAS-$>#Tr>UA4S3}_?XJa37_>?Y;w!sVa+!bVx{5Oe5y z(Jd>XbK~n-6 z;+w^*E-L-#=wj_T-7W6t>$#HcSI;wsrWImUdcKEX(xIPV!a8uJT^z3m%rNES>veJ& zwlJHV?}BeOu&o;&gcOCHE6sl82x~sZ)Mr5FSueNz?q4g7QG;x`qQd-S+B8^2=ui?~ z8&4(z#OD*tESNIAWoSogv>x#;0rehx@%dE6 z+NKS=cn%(yR$(&$$&0hSmT2}Te3fAIiNi+5_^;)S4n5AJVch1K?U8jntnZqad~ zdgKK*K5!Lw3n>h2!(}#)RUM-3v&-rI$(R~}^;{-~#pE>oG*76$9in%%+D|~PByOyx z!0vKNG0Y~~@i-ZLZ|}tb$s<-G_gFiO^oLyiE_rbtXy(Ip8;*x(_u$B1q{<~oka~}6 zbmv%pH>gCII;Zobc-mT?o2EA!`2+wP zvk&d&Ug^HC-Uiq=-Hz3v^c9@7D1wsW(@bFH73A-6lnRqtk(EM|=_{{1Uab$5158Q4 zQ|7kSy)0|d37X5qDF9jAyoIjAaj)t8+s%mj{T9d`)DmC7?6Gyz@}Giu#_eEaKZtPH z;xsdvoDbS|L3OQIGsVbRqxiRFlIRKfOSQMUpsIw@!teXvn_6xAoBJZsIrA0+^4J_q zc^Bf6as5N~oF<~JGv!Cof>Qt+V&xNikHD)7y3$o(lceQ7q??L)<)QOjH^FO8i3TJB z31v>)pfn4Ynw#gN`C-n&`bIpQQr~xaz!}v2`1W$P*sO_x)L>mydIoD@pfl-E8 zSGgjwPL!?4{6kqqOv~#~Fw2vE8(mA0@T))txfGRpT9!=0<)zY);2sj1NFuzaW4v(U zqNqIwQR~m0t6IRR-u678Nf$863V{YCeeqFOe?U@kq@EmX@wH^4q-+JpOCdW{aWEUK*!ELZ-*UXSRR-*U%2FYv;x}lo!L2BH@Q}qRV|Rp z%-(y%fZXUA2b~^*Q&PvQ%O}G4`kjniQc}D>j#{w_VADIpqQiQ%*1 zr7F{mvl8|rVg!9C>Q|wd5MgO(n-kx2dQDc`;$)jF_Pf{b-U>Y)S=#``C9U4h&nRZyFPk(o zjncJ2T@#g?Um3@a$yMW0{tjI5my=eYzk-n_wO0}yY0f^-_HuKom8Gt z?WT;M0^n!`bUxdmp?k3-(2IVWX#x~2)d6+2OSLXT*Gy43D_G(w=w(gCCUZ=O~UIti+#2= zNQYWGVMl%a<4EF&uqP}YOdY}nlOV>0lN6h~y2J@?)ZtxGLVe!TV_DVH?;u?gTc1^UoYQMKKZCnN$Vz_h z*3;J3y4+dj{e^hhLMf-PTAX8eR~?5-1HX*>+h9Q+qvml?bdbli&p6iaIM~LN43#t* zIb(S@8u~W-;TN*bsp>mW_1qS{fr=EV0F=eMPR`WiD^OiAfQz>GqMCDg|3I|^;CF(V zGMe4$`gRhR@5kI59q-b zy&u=}C`vX{&4tcf+^jVDFV~#L$G6*3zcf5&(MtNdw$K91WR|d40Y6Ig%NF)=-L=KA)QLJv6vuxi4eyP_Jr%o(xm{ zG@%hIX|6fv?+Aa97|hPM(A)~yl7W)9^_YSfDy9CXKyO_M3YE`%i~ddu+!7iQy&s?~ zRKR2^8NYd^MpU$VDIMOf>7(Z?eE0R~`JqjMBC$}oRs~Gf+T6hDB{RG1{TeT#NF#DO zSVU7SqEWm0a%Pfy#_EcHNvy?ZJXF>J6k++z_H0JW>Zy|gBIMU1+k6d;k-jGf?Y%Pq z6YG9uyNY-A3--t|1>Kgl<&f3O=A~u1uhgXS1`eO_Y>=yhPXcC%ns|YUTRt_iK}k4C zaPtb&yYkv4PLrN8bJn&bxLy&|8+=M^R{-JHQVuxSXc3Ii= z3?xPp?%3fTlt=y^H$Fb%l8sl9h9^Q6Z7YuYwY4hk|{_q}Tw{d3YMJeAfSmrf*=ctLwUs(b#Hiv#}c6PEL%*R%6??)7U(* z8>_Kx+wZxb@A`hho(psCi802aU^UNk=HPm=ifz2_4P%B=cdNy9yig76AwF)N+>oZ5 z{4ofP2Uic8T=Wt{aEL%YY%wYwg(hpXsc8DH%NrQ9?{r)s*zZrDI~MPd^b&}-j6*mH)TTIvg7;QqDZ1{=C6Yalq znZJKnn)fD(i!MP)kcnVIV8}0|5GB{2!R=reGGa0P;qy2OU}&gMZnxWKjN9M_Rl#~A z+xKn4E7hbqOngffpa2vv91R~18s%^&94Yetj0<8Mck^+duQo8~5Lqe+V&Kz4&jItp z1ejo{zU%$Y3PKs`|23%&0-cGng?-_T^UMZA0I|lI&G8|{I^xX@lcA|Jlra;{>pBuR z{JM@g>)ZxEwnG2MT+>y^_9Zsl$JFRU>IVLc33%Yc9Wv{$0f8T&GFb9)D)x1Eyk?$L?uFfVuQOd;q|X{*Mcf5dg680^^L}8>PS02n|LGIk%Krr z=vi!IU{tBu!I(9_n-|@<$H$^5yq%qy5tck&f=J(Y-K7F+MwV)Q0oKetgqB*_FoVzc zd&j2@XgfB!4&IK}o7c|bRXxf3oZ*_*YdjE8ln5<=Q1 z{_rNN8-t$T&TFPlQad*YDTyTbh04RqL zYP8`m&r^SSR))KP`8h9DL&)+T8+Rmw6-Q6hTxD-0q&}ND%y>k>}!3khU4^%1* zAGlhacHHe4g)G7KZ{eJ-kNy$Fg1kGUQ9~lEM>*!Uh6~l4^MmNDU5@C=g3H|KQl=37 z6r>=?f^Z>xI2+_wAWdLNM|YR!aBHj2ACL&c&wnFyp#K6+q+N6QDK2&>8=8<^hP!YX zd1~UHQD)h+KU{TxC1fO?E_QVbY)gB-?`aC& z#$IX0tC`K7JL~f+(xuk>^dV;=$M~y&XZ_38pOV-K(&-Zg4Iz!CNd$OOX4*RXO3^@C z88;KK-EDUkUGjN2!Sx%Qv&7n+uK2|H$tKyjPQ|9BkFOX0fp2GuI|q!&b&7 z@4iHOIB_5V3y7o4mVh}2ha;--qb2HfKoYQ`;dx$d0JzhY*L~?75E@u|r|3lx$9Z>t zYuG-B^0WIBH7b;bI7Wm&{|kYUfKv`eJd8FvL@R1hFV&fY$W$!a`hYEqeS&~9IxdXW z8E+u}{O6g$XHV;}d3l3%`@f{^m7$wB5(s2eW!!@s?U`)*;=0yvmki_<`GN3A2!3!B z@F8Nw#^X$;S{>%-e&%k+QtX&`G1Gwe*S0+bi7$s^YMa^#WP}xy^|R@+VGYO^`d`|CCJ@FX6T$pc4e3ueFlM)9@oLNE2Y>tx||dSJc-?Svaf5)~zO-3!sM6R5Mq zX>BL1vsH~M#AVJJx9a=xAXhuz*Yu`VY>Xa7>306c0zrnfmdTTubytOj6@*ebm6c{G zZJByR)y3;L21;B5e>x*q3bW&;W>rMY+U-teJLZGXDc@y;vnc{SI<{N0CN}KTHg`N< zT6yffn;#n6rg1%wt8UbB0@^fPD%UDjs@mg`mB6&yce|DaY-ZzRRDNGXALz+hYbqL@ z7?33GQgL7_IZ}}aDw+Ii7*`rRY_^^@iI37n%XBasmPXlhI%?~<3PKIe9-w>f^(mNI z6+7M8AwM2NZAFNI6ouG8%|HE_%a*kdd(!&0O3d7S`*Ij(NhQI+PmcN&X(#I?3eSS{ zYXuYGxqff!(%cz9BYj0E+ZsYKpV@5DkUee;t*fVz+M+}i3AJ6@twd^ZI@9?Y7a1q)@pJe^(# zp~d+82E0(j-GY~Yx4}CYoH)dh2z$vdVrQZ;=9g=b?S>VenK}cmA7toPu)*X91>H0> zpMnUVKz*Z<4&A6 zLSBFV>ZO$&hEsp}YWA@94Gk}rIQ_>jAC{b%zqInGDFMUlX0GDjN{OpLsw~3$aL-)j zmQfLEGzaUWk_20x=NAvP(}#L!^n=0w-A>Og88 zp#x6gV>8C{eNVj08g)dVz)?>o{17^qECSKfo*l;Gw+D;Twfd*a>JxN|rR26-)OD!=E zSE0LHeAs#PFN15teT<)!M@rIrgukNNtY1i=#swPt4&2}>qty!w-99?Ol54wzAq4b(!y^leMBNx>4; zdc~2NwG9YSP55bT{bO-f+yQ*nW4j-1_I~bGa)c3+%y_=rx94@TWO4YZ(*6F0?28h&7xmu(V0Z8sVFEx! zjY@;MHWGtVLVN+iEPqZ;fDig{jZe$kW&$o%3mz2&1Z|djcVj7#Sr4inKFrc))~3`? z;w$D#;pIM^wQJZ9;R6OsG+M{qwS`lPO<7tl-Ub2zwEgd-@laly2n(M1=M_9i_zl$G za-UD?6&Dw3_<~f$F=O!qo;6(vv8k4z3rId0fp)~Yov(6c3KN(BSu@alAFd5e9?sGa zen6tMEN66CSqUmw1-=*(`Q4)T=qAz@3kx#nG?+T&KOq#=qfVBWn);`8j4FD_9J(%N zRfLnE6Z2*ecVE#wZUDqdG0bbol5Juu{jLt{-!5QxV%5FhKxAX=^;IMI=z>1=Yq&9hjZeWxi%Xwm& zfza0n#^wE9sQ>R2R14Y2*?i&?&?VA49#5)zGCC3wHvThAtZ_iL`k4 z)~tLUz?dGCIB%hLb!LCOM(eaOqQ*l^hD&@6c{w(QVBc9%uC{1R4q3A7*WWY75BdSk zQa$q#jV5iV9awV-B6^$b&9@kd4O+IBLG%CLket;a93^fRVM4r+&9@D&Ct?3cH9(|O z5>p81_@6PA4c5)N?8JJUwgp4R(g>bl%ficWgUy9&mZzpbzh2l@XrOU4_69|a9wEE< zQuUHWV>W;zlqq$Pq!*aVlv5+A<RkQ2Mo0qp0TrghTM_vLI9vp zf?PAlMoa2dU2>u$YrwDoKs9(}t);nwCHA2v1vna5dE@`uf5ZUbb~=4r$8Tg)72`GK z-%1}rD3-#ZEehl-wq+y>sl3Ln&BX7lME{k~7Es?7+Qle$uz%a=&M&Iwy`J2(J94HF z@nyGhj`Q$$`)3s%kA?iUX4R=wbT?~ER^sfhYud^|T+cOcI7U}~xBon|%Y9xEf=Xuu z!nEu&el=xtOI~vggEx9QfYQ+EypNZjtJJe`iC|v7sG^*iFf|&|@=;m}EN_SqX-8Wx zIM*?FUB(9o$c49KP2l!p|!d=GH5ZEw6ik18Sh(eXGp2dn1NF7b23S zxL?Qn*P{@^wli=mcsK-wO2AeGEVfa4=Aocqq^!1U(#(Wt1H}DY@&U01Q#6UZE>tH{ zlvv3AyEv|17jhk@$9c$t$v%G9yI#c#qPo1}aN5=W47LSwHt-az`c2$x_wDtBmgio7 zx;yd*qqBQJz-Yl$mMhjjBY3wL(qSVfinQwxtR5$JS1#Ig2@&a^_d`Td5HbFtelDJX z!Gi#cWkqnllc%oGr*h-7Pz&I6;H*9x8(syJYDijX>yr?|Nqunm2|EQ(zIK-q9x@)g zDx?ds60?#U{p63&^cA@j;p3bK=e*};r-jQT8AZTf9oU0@MUrkTz@MxONhk#o!Q@{b z0f?#4@BRpjIUIu5%nZ$f!mnBk5kG~>8f+cj=wuN~h*nrcH&x3wBgEyro|;$lImRK~mP7qeSyKmr|b)Ipgd#953gG2XNQQGf5He=I%f`v$^4SzReQQX4~eYbU9! z>#W!Pv5`%BYud{5v<3S!(!8ChGBm@tkf>D$5luk};tkx` zoL7yy7tKM3_+?iQ_~+$XjGJ|CIG=7_Ka|ezPr3`!INe>B4=c^XnZB?--{(k?BqfL$ zIlHz3|D=|^%hZY#_kv`jNM#vo)W98k2cW+(`oKhW7_^sYtco$XOleU2ykpd60EPYz zjf#;5hKVWcU(0O`p!0%TNk7qdgyQ$O=9P$p3T;oDp`g5`msJHhS(yPjHinsg28^Z1 znxzBdhGP|Cd_hbNM({(7B@7u6Ksw4F-RF_xhn?C3w6a)co&);2ZbluT*UiIKt>2rn zPgpFi*cMtQ@2;+zbP>Vwu_#zM2$~hLwMvgO9sy-%{gf=ixCuzrf^Gf_e4=lBt8QW_ zpz(sB9}5U=Dnfx^D!E?@&ZVb|=3bJ-_D9)8Ib8!J;a!bk6~z%Hg^4Hev{R)j1H>QtJ9>I5QHE-YKTXx~elezr z1Qt&W*k@U4A)D4gc>>)e?GH5sS%;GqXZ{xf{JxUcm1cCheF(_n=O zY@nen!D?^>V>2PsZ`u}z5g4#YgsIQ>mmg`=?$Ed1?Nbz?)yI>19ccQ>hh26-?xVmZ z**s8f-Sc4A1Gdqun4;fvMJ48v@0$%E$TY@s&bAxeXk9^=!13)__Q~pc)c#d|RW;tT zawg?G_|z}!UlY`a+`WCBe-tTE7}^0vS14AJhHTo^!Sm>9_Hjd*kHu#}T2`4)=asdO!uC-Nw>Eyp&i2q6l+>+4!a8I?$Qoi0WD z`JVTkS9kM|Hj9&k{A-DW+o34q72%~hOoe+dS|7*gM7 zj~iE0_!;MtNn|_L>wG&FOVmRqO6@%%YO*zFUx{Q$WkjOh*P%HmG15N?pxy$gw0<^5)cHU-TuZ1ZnLgG>8<{` zr}>*TaG){rkMh%2R$dZg>a6InV9@sjo&m(>ZThFFaaqklq8q!Q4K?V7o98Jm3 zRY+`&uxo!LaV>uxIjzWz3$xYWkvVKlyaX9DXWi;s;RkOb6eD>K)H23*{$VIw_u{A@ znW?yO$*mh#t=EFEewDcf292-h;(XniE#2mmP#%OBA!)57j@xL|+bGVhTg?Fy_ENC) z%<@6w7^3onFW)!8V}AYQ`jbNN$hOt~TzNU|EV#w?Mf3~d!xaS(zM=DjS)#g}~CSfiQq_fSFWY(HdHu%Ww{^c1=Dw9ftcI0W}EQr!JL2JUexD zf-Hm93gBT#^CN&FSQCk`ki-YMd3=Ozi2*dAM5uz6Kybm642-h{+U*mUq9{N zrQ;}o2=;pirJ-4x&Uq@PkC%0Bjc6K54HK&nu}zJF3Ia+(eoFBuiI+jcVowI69X7cF z3|W8_BJ?T0ifxSv(X7|mR<`=4#-~qSlOXF$wFS{H2)NBLH=l0s)aX zq^~AQ8-~u&N{4+#SAb=v{ldA?lK|EVRwzyY{P%3X%HYXK*>Quvaq{RKwtB!ZVw+MP z{MaQH{g_{yjrG7`{q{Wml3yC$pAkOR46VbzYUYa;bd3aan0S+Tk3Q8u1lN{D5kN<`($cI7sLR1r_KTj_stK=& z)U8i1XZWmy38@-&DdY9pe3V^jMp|H1a&!Pc=dI%L z#*=_>{D8&kKdjSsF(QW|jzl#!Y{2Yl{!!(~t^InQm{9zObx78z8N#hOYWhaj|DjMk zF>}hrZtzNO&)M{vd15^fk4K$n?7d67;=D`yH2Rdm(l0&h~0FV=h>kkav=(9c#%~6e~C~CVV?~)Ah*pq3Kt6yG#QVz|2tGqJ~?%E za4vi)1}C2(`=rCm|As^|K-`Q3GMlV4q5TtUXyv!)@RUx3`_%2(*5Ef+GauVi!{)Nc^% zzN}aR2^?+Cx0w91`?#F-TdWdj6zQ=LUb5KnF7@rKCj9M(j#05*BCYCR&jwcNI}x(o zuVlAA{+;P6iji2Ltu=q|V+m9XgX z*bT{^WeI2g$x8)%OAA5EtWr3k`ac}^VTCjOZ4)$Bdt~4kgS-Zk*2v3^32K4(xb$dw zXz9rwf=B%D)NtXC13$ew^Q+n;d)rlx;mNj}2d89)YF5ca`RKyF`XP;T zrITs{9S1Gm0;rpXEjUH5VKN0`ga&jQ6r=p^wyDq(I*F^;V$w3`a1#k@6fkrlTyzw5 zqE<1f37ha9rGUlM523MSR*SkQ(l_)taZQJ5B&l-!3UUn-os)%`0xLt&NRS~l8Y1n)_d zH6eqgAi{s$rN?f^wlgK(~M{W zz}=9A8^qf*&KDuXG8m=4LC^30xr?pM7T{~EbGaGk_YeuoGLFUPfrNv~2Kck?WQTh~ zds|a__IGLB^LT-n67NQn9l{RRHNnd~R#WmQ{;@iYV~q?Wdn&fXYsqQfVck7FRCDol z018JZ?qnyqX|(DZKsVt%I+VnS*7o8xy`Aar-R&u-Lgp`*j7hwJ;Zs-caHY-0+I2|Q z$fs$69Glda{Y{^ifv0-OE$rB*=Hk7zo&Zap?{gzP-=058xm-C9PHY*69FEH1$U66V za;8ji9HjwvApN0~wKL2h5rz=8dgDK9l#ZSN^f*`|!CECcOg=G@azf5;e1j0Q!Sm>R z@1#v)Y#nkG7fMp<5+!OzN~0XMIuR1J@)}%!WExy$N>ose6u0Rs{9sK-B(=QZkZ ze1$CdF{{a#e)zMOV&u0Oh4wx2rAU6ccx8?#ZOow96|Um#r@*YC6FOI9e5m5OKXoA}xy*tgX_4?WO`!-iuP#5&D_eb>?qh@%qU;sgT66 z(Gen|-H_(*PsW6a3Jn_>>GkQrj0uee8;L<3c#VUK^;38#z~BG8%UbqHUpk4y>wOwY zjUL-cKl4W4$zJG%!slw;dHQ2>zw2&FU?8LkCN~OJ07nNZ+dOe)%uDsaZQ~**Qbh@h z2Etn$EShC;ydwtieBv7Q=Er|?SCHn?eoIiVw0;T5mMCjTh1G_Oel+hEnP+Opz;fK~ zh~9d6H(vU~yqI-5zxIRX;++|ayY{%k%qlsLi5L?Q*r`VIu>_HQkXMHBgBm9HK(7~t z9_Y3agg3Z}2wn*HrGUsRgW(Oe?#8#*Z>Hx^0d(IY@PhsuQAu-JqtBc z?N-d2ah|h4mD*rx#c&9@F(n1Lm@w6*+u2ls|KJg81cGQOFBZdUo69m)hxEr(>C)Dn zr|{~H5S)YlY)pbB{uj9`0qk?4=g&}8myGvSjEhZPD@dw;1;^`ph5tNIhQh*q!r(2? z#E|470wetg!W--0q)Bw#-1L<~ih`6un?ouIMmMA`^m}Lii3xa;RYUG1QEgc({BS67 z$Xs`s`Mqt~|1owXD{U5pX0IlIhlPtyb#=-APc7V0F?Qx#$Xw-I;4Pfiu~NGZ9skq4z>AXg!8mY4eEd}0&ux9U;Oe!{TnmuY zgLwYl@ZS!^DbNXpz5C@gyz!ttZbg5*gImus+S6S?7&vvB#N{1C6LQ{LAj(qD(S`4b zTLy-KNh2Mx6*(n!{2*uk)k;b2!CplE-!24*}}?kvozkl}&f^>RI}4?saB&QyzT zJWix;hfIMLIqc5wHXO7<%w@X1bJ>08XFaEpk4JR~^!qW~h0oBQy5A?d1S8GBizx0- z)(Z(GaTu+0b0ELv=4ZOkKF{GKpHxehB-||G178gLph#q3Z~&(v1dTZt9>##V+~RT- zXyIZ9U`WlKYmNas!rCyEhw-$!KZUao8)NYPB|`NL0we>27T9wdi-f$8&EfNAhgP0U z(~1^0b#$gzR*G}8w~7RV2{XF^yu_fOUdj)rMFZJ^$_m#jT6dSJk0GSxv@ymsz^8wF zH{j4XIk;~fg-72|2eAdNpiCwDNU3M88MJ`<<)@3?+55y3rsF$B52T%?6$eYyTmLB9#nI;m&fx~w~n@1G%6A0zPOv_)vVG(u=4%ONrl=@m<-XWJ(@m!^W_|G-6 z;L6l0R#!B;e|(EBD(|8@=k>umjj%%l6T_u`N_^8z=p!d}(>2$5Vwh_$$}>1pL|+QnbSl%SV8N=QmA9mo&I#3~k4#DCS`lywEXP zR>c=06;_E%h(<_H<@a5OKqM?F0GUApQH|Et^0e>Xo!;4I@w87=wkulwmN@MiZ=Ysr z3U@?>Gmaa>03$H;(jn*xFE0oG6#^!+_Np0&`kquF!`O#teV~kf`l~qaa&dQ3)98?h zCUYP@M6nAt`d7HBV`p6@KOCyXcr{?m2j{ACor^8|2JYXwN5QeV6>olv;v|3M zs+z2N*$)AwdUp?+yAwKRC*<(w+Nr2Z764*%}j* zCo1LtojWQySVv$@Sf~*1#Z6Rh$8;K7|DZu@KW_I}4Ta3apeQN4&70Gb1qUP~c9n(; zXpGB8wSR6;d8aLu7)kGKX~n8A3xZUoGeQ$EBOP?A3>yd#^L1Af9n3|UNuIn1gN;sG zREp}Z+e%|ldGG%zpe2X|6DR3M!}Z$?Ry>N%a*vQH9ZoT^3DFGFCRh~gkdB0s;-a$o zvDfIgu37JXA222`Fs<-&x+w&VDMMFRAjqT2MDHjleMc%1z3gc)Uawit9;C8I+6^?n zy2K8=pwz7meyIXE{w{ORd)6+>Q5Ul1;9TeEw!a7za#92ut%%0I$*gcR&rR?QY(38P zp!^DpTA>s`N1?AX8a!1Fhxj%JgsWZan1-c%<-(e3^KUPmuFZ_i8Pn;R)0}d*u<;}o z!f%Zpc2d_&BpXl=!zF6p2zJ#AE1+VE88{`>%9X!Ws{9D7Xyf)I)cQHVwrnIk?2mNt zdJC>t>+?L~1hBN=G$7k;l=YX58UAx%JRp?ZM6rRZ>_(pe?h)n2OrfzcE<($`HT?K~ z{Y-M-2Ay_%pGsAmyO#5V5Z6W{NNKvGv^}?+b-@!$lst~{^YhWbNebp$?I1WAjc(I* z$Zd$hM1LL5m{~L#AA+ciWnci9sSZmng9vPaYsO7K(oLHFt2~QXk%YXB3Y#$0 zC6ISvlh2MBtdi+8K_@kFBK(Jbo-70fquClh_^yI+HEa}IAuU^D?2~>&G1t15qoy8| zBFLk*$sznw2;MIFm%Sef-VdX0$0tN%sHkgvJMySyleg|Eq@~jshH5?mDQ8nx1=?5y z^xMr}{DV2+o9i52pK&7^ie?Bn@Y`hAGY-}dhRwq|uCS)LUuSE3M*t!_TqrBSP6q%< z(pCJ-4Y$N>Gxu}jndn?pyo!Ig2u^?XkY6k;u1M+eH*^dnbpt;ESUud$es8>LSJ!AM zyh%8*&X-7|(jh+lKMTWJ1Wj7)fRm?9z8t!Ctu`6}H!0Xyn(31AM;W1H^q%|oIYnXq*BFUpXk3G zF6?+cjttw}ydQpJDR&-|TgMK6gD}9C&I>GuN2LgoB%w7s`3YthU%HDPO4E^E%3FI>noI1*|4mfZk_KYOA53o^ ziVqoY__LwtCbnxN#$-|h=PGxCJ+VYw)T>fcPmtm=3fXdQ&s#Tby2bB(c+LkQ&zQUw z?4SR4d2ApMmteD?xH_!2L&h{UV!Xqk+2eJH{ZFWhes}cbM_Ki})FFk-1}yP(`98V| z`nvc#;Cb#JI;G6N*F&H}ypIt>B)70wzNs#oT^29F_@)sD8?r#nH=I=Jk^yqVYI|Y& z_TGhD&n=(n-F8)(NH9M%7Tom+S-+Yb-@7}{1$IGN{l*7>UkKt9y#@%wlZ)pIdsoHH z2dB;YkuBQwHqV|@R?%P;eX)+y2I%X@Pb*!JeV%bp& zhut~^4N_KE%7BXMNE_k54UDL?p5ttDb`xu{4h9xSa><(LlX;7otbaFqUQ)$?0cX~ro%0%;|bot*U}MUzbq z7Z5EHA!KgDWvb1h`7_a#;&5AXBirZ7XWGO=6+aS;7;V{o!ebP&7U$e+x2q5->+#gi zfqOub$I4pZ)a9`_#JbX;{UoHQ*`9&UKx^ffM0-~A5WUt25z)`Jx|t6xN>Db zu6{}2UZ19B_(bK~$L!T5#9nSNl6ivEIDvQ?gF;;L<{E31sZ%yIFqk5rEZ^{9HVL@T}2Q+x)V5^EHQ{B<3=3f$Jo*uzA(y?uAnj11^w=EYQ zub1>4agtlQa7sF|7^2>~10@NGy<#gv968ciud16UTqeKBGeR{Z?P`++by=d!b`!+Y zCa86)3{6G!4l@lD-dUhA^}n;iFc4evZum?mB?bq7vvK!I%B>fjg;<4%@-JzLb>!Gn zwWH&8<&F1z+F*v(pc~5}L)C-2t*klxz$cZyK7yse?BF(+a>SSfA)wT0mBqQ#{JDWrEkGNEm8z&?v3Qr{I+xS}k z=Y^(J-}^3GD36lk$I;{N_6y7Rx3Y}`LmP!o@w2Ti(K~}|h$kpmarQaCska;8jMU3? zSzudRY4%7S$>+}+zu4Tjq~_98$IO8vW0%Vnm@fv;_&Vzw6bxnKdRB{9XVqT95Cfsq zZBbdteRv2YyVlnWN~}DA=P?7x7UF-3aG}e_3H}MPMYt(?e}AcwQISea6c0ybL_@(R zCD44lr6A`CFmvaAr#Ig<5tEZ^idP;h#4V?YjQG~S(YA7~#`NC(+B`b1m|VS9lY@`w zy&Ma^ME=xRWRSheY|=}-+)cqWAv}ZIzj8Pjy~2p*{&F+wBCR1PwyFGCn;U^Km%V(!?AN?^(B3u1wEKDwikgPdWVwc0Q`>H z?dcrz-f*ZZ=Zg^mg0(mN!eI0vVzlm(tV+pHBG8L&9NLZyJ|k8 z1d>4MTATQEwC|1^rXv(O^bX!!PuZi~We8oSVH3dzpM!t9_)my)Em{RuAswIR{vvfr zSXeCz%r}L3B2THzWVJ{#7&1i%9c{egt_I@{C0=GYZh5NfaWkCjKc_3sva9>F&Run- ze`q1HCG*Qi37<}JC}nfs%1oPkQuGRxw(EhVL0Ftzs#E*LW9dyE@EEYEy3ksVAD|GQ zAK|K10(z|mpEW6k!pw=?R8Ao?S~~Cxi;dKYouJD!keoUzZ0qB7bin%aR~yC#y;QJO zGtl=D%XvgroDJ4Z3O!rH!mc;bIa7f&k-eA+tkjmb-IuJbNDSn538C))xt%YK5TGc< zPj`Qi8Fo!@?T0;fDrdY)V0!v+4Reo^Eoga_tg^w5A{zb;PQxJ#rl?XcFH>1sPSP3( zTnLqMG&%)m@AvGgMo}|wkKkM2g|}fpAP62$s5XDRJx3sQWJ(Fg&JZV2YUmdJHK&%% zm^xu|5RLapR-Ew^oukNqA1)Wqf0p6DMHSDY8TeFbAoX*&e;cP4^fK3<&9n;QqoqjV zAJ1=>Zhy;bJ8tKjaso~cG*2qsu1Cmx`)BUQL?{BKrCP{*Jg}LE- z66EXedJ9b3bKvtxQB3oGG9|+k&D~|W5Pr#}e@wo@4Jkn4kau~y^x_kEd>&wlUE&{M zY-l4?P;w)MEo9_pl4893cGxlCu?hg6i!dT+i%oYn$aV!`$2(Bh!se8r+CstLOEcfh zR*q_?m_Kx^9Vl9K1(l2=;%uq3lp_{6;k|!FY}tmptTifxX{9f0FUc}Yh`}pHV6dww zO?=E0K7RiZ-Qjmw-@|!62e^Xu^)+$SUZU^r?0zy;!NOXQ(%)c%Ja-8B8I@2rTR!Z| z+mgBf`4hJ{C>H{>yzIg3P71Sc6z~t5i2Tk5B<#i}3pqV|d)k}IaLD5Q>8!FU)cS8| zS+jci=b19SF##}T^#PNj)?B!K0PsY866b?lqTYm$ZBa$bP{fy%2mdBWaes2Se+%vN zrCpv*Yd~9_hK)@o`WiUQ5h%ECXNZ^JClTmW_z(lnbK%x?612v=fyN3|U4vt5`jhL1 zPQ#tA;wN)YJvAOEL{!|6!;k~SK23hZ`7z7Qq#0@bqzAaB_$J0?Z8@6*>aff-)^y`T z#+cGB*)L>`;64tAFK4XxFa&~==%EHIzc|?5G1en8O~^Tq?Av&8os5S6=05&4IqmkD z#UbyX2kF%4^u^zsH0_|){owfR?X+^zhM4625Ui+5EuR@i5TXJ)o3P#+@A(eQ&pF`?9sM+R`{*I$`-*Z*wqpPxinT;Pbh0|3e+|@fJ zvIbdD>H|uBmiwCqcEYxwuMDazyklZ2p{7Z_O$N8VzzF(RFo{jW*=v>AizUl#X5~Pd zT3;YVYi{3^P^BHenhXYlc-fV(qsfl&-%42sR_<9SRN6Jy%`luXm3f)vZ>H8WZ03K7 zgk_fJ9_^u|N*WInRfg?Y-rt~hg~Y>hVW~mkV8h<5;a3 z-FPw*`_*u3FPps&Ca1BzM~{7#`^CXOw8LS7`0Jurd2l4n-Ro&1syIUW*tRs4lyKW? zRZnYzJ4>E1jO=rxfXgFU&!9n##>wgNA@^zLm&!KzZa^{EH8QwyPG(0FXe|$93who^ z16tC*4AkmlgQABJer9yr2=T?J;xJIlXvF%gtQK!G_Y#u*>{mFp%XOMw)hHVun+LfC z^Q9Ib)9Rxuqg|Dx0=K50@yp;zP`4ND2S$;&jO!yXdA~b-7c+67V#*&j{1NyTsJX{{ zWR8nqU{fv?m{=)euyVI1IxdyMHU)Y=&B`&(eXA>7K4TkuX9F45B=Ft@l-ZSaV7D&< z0)yH+16egSJiRk?ulu^`ot@nue4n~x!&3og#^O-EhMXyE6u8YaAs&HP3? zthO;>-^xu&)4nWYnH9w;Ck&49Q>;_nCCsh!nooBawv=Jy?!5S%Br) zu|hw+k{*8sEun+jZ7ZuvCgfdaQ#UE4%b>`;Cg4waLWMxPD=WG7x%s4SmW4A@p?+#o z($ZayJ9BDD36{yBBUQ|}T^OdR;Rf^;0Oz?p)XQ0$A-Wv%P*)vNLvO2U-c4r zKvT&*#Nn$Y_~<=6@$L3~?ttL&c&21XH=abJ`=3WWwL0!7kL;Qs8i`=XAh43^}-uthOPcvwpDO zvby}il3;7eF=!zUbBcuN+vHFQBJIhUiSrOL;iFF+iV#o+m3Ug{$c30%z9R$&q7f(6 zr`^+Uo}#6ZOk?ca6kf~GXD!!)k0|dXLRS(WGlv9QzaU`5L{M#E&6&#&__H*PD;-whIa zV$<{WzI^3r;c%tJf7ryP*`K?YVEpr;Ryl%mq+b2u>Q+zUmu=sleooN2wa7Qp_ZHd>We z)xY9>TaYGe2G#?rgn1~i7Ubd$Qf*$mDk>ehWg$mVZ04;5PUgdjnWRe=iJ;RAsi!9< z&hBogMRz5lwbqyzEG;61;n27>O!3*=Engr|!PBe`o2qV<7ADdzo=_E1l0;>PAW9)= zItJOMN-dFM2Qv`9X$BblsB|@959QKPk=hLv2h}9*uH(3+*>+T! zO(lFzUFuH$YyypHs4fxotF`QuoLsR#OVQ<(saO+fSbsww+Iqe*cIoH&U!t1!FYgR2 zA-)?cD^A822=&ig_(G8W)GG$!*WaScmW$RN1zST(qe(PKrjOGLG-z9&l9Wv`o6%mt zMfdLZD1(kfbl z6p$XJFYYXiUC736N@pwfPS{wZQJqRG%C7mutLxEJk^{U%`PLz4xci?K8b|kZepXEoJ$?HNd(P4y zvY@zVF6}MyLu-)wa4MT8#9TwT7LjtF1i}zrt@HWkORXWEoDsP`I!${q7?ka&xPLLp zc5A&&59lKv(^x;FnC$Q7VlT$NKtJ%Tm2SMV2xhWGX=EL~Zw%;;XWKkO;i# zYd;(P5I48L7JO^=xnLJQyQ-FP>k}2tZ{`Xd!ahl@O?a zr!>tjbv3ige)DUJznG|!DSDjo^@Wscdw7E}0XYry@5dddD9HOKVU73`af3@wT`yf? zX6SV0VaMR!NJ?=K#DEa3SP?t>l_`3qU?YXWskiJR!{i?+uDK2SI%?9rP~!Ya*g1o< z86G%M&v-{aa!tSgO$@yz>Fwk=e1IqC)9+m6dzw-4zojYD@4Eer42d!iAj4N{Mfj

qcUKcAyC)jC5+c_!U1rlRg1$~HXcowxWxZo3t+loklXR0 zK@)h;??X+|4Q?Il>vy#--ze0VUzA#wj?9XOt1>nFFY{D+VUt9w=YDO?cUz6LT*-gb zD~bu6k)5hBU(+g&5Eo23B-!xX(8Tm6Bvee#yzIKed+_#o^;H`o>NE=Y!luvihI;0G z8XQ54q*cd4eYdD_P=io6yL z5Xzr*qlOBtLqBW&z%s9QwDHGyA~oy9`s2_D;Op-6{NMKLhhWTQ2ii*QkbqNui&L zXxsD{dC>zAWwYasWC&G#OIAQl|a#GNP zQ>tO>Z)B-vkhtB?%a~+5i--zHve4w%Ee{A*2MI*8#lK!yMO(>-rGc5Yuiwo9>E_9rgFpGcv7;{I5KjuXX!oon357`z45TsK=4vK7U}V$_Kz`D=+j zQ6Ql2^?1xN@^>;dP%UDy;dA8-sZlFE?c+PlICd$}66Epj(g4v{R9IM*c?bIiR^C7S zpj%%kI?UMGMJJ_DvxB;?h8gK^fcAX)RDmZi0sC5*0*8lfsx;x>GD9w`0;Dy41`IA_ z;S=#VHFzq~Q1sAQtRJBlK)KrgVzCw_HSQQui*&@^;5D)R9aNs3r|f6J($aK4tciz|5_GE#DML9Imbh&Iwd)|8dwDN4dw4Wk`E${dB=h+d>@??>w zBNN}2gOV}|Z}#KZ^g6t0{2*q;VXK$v#JaKZK6Vr$1)C-)zX4_b!~+~@!vK+sCi~_Z-0iZD*2+ZQHhO+qP{xnb@4zwl%TtckVyvXLon)s#-B&v0<(}rZR62f{e1P^TxClYWO*5i|APGOU3~S^#iQfJ4YVurbE0!gmIOcA9X(cpR zEtwyj!`HLn+cr)WXXrOVj&tb{XC;`yqf1MI*4NQ1Wwk!+iKGuPsJn6OJfl>x{}l4v zxoBbrj}?WRXpXjRXf^hGo9sjdybJJ#z8TKwVn^k#)8~9@I@)u1nk|=ewwj~=mk>rI zp&a;AHS5bH(&!pVl=f*}*ZD=<{k)C8OFaJi9#ij?_MX^$F7CMiDg+W_Ogalv#=F?z zw@$v|hBb}`!t#9JXJr^a$eUNMhrr|~aNnLr z@-Bh+4u+a0X^%tfm`B&WAkF2g?af+krFh51*|PK>uL*$!SyT=bd_^9uSxpAn}+shpksL?JBW`nh0&@jk;lf=B~ehS zjx_PVmE6voD7iV^^XX_W$>#P750t<_qWY6>G3s{n&GBNfMM{ypl(Y;jsAVmXfTnnq zHB3iNO5El8NH4>pAuW4h+ghySd|vvu`1O8}Snc8{q7{Wab@|v)H1rE-pJ|#TsG92g z+_Pce7?9Jek^@lFqU>rLIgZQ7QKhcrx>v{b4h|RSdr^Uq3H0z<%OEN;1{nuPc{hA3dq1rWUk^_uz@llmW z$jFxmMT24MkPSOXT%`6x`frM5jsI(KVVoRY6Qj^sflyGU>Lvy`3`WU2TeAJi(4wc= ziI$X4!l27qFQu(v^qZPTH0W>TeV?2YSIxGWk+Q!+L2FcCN7s*;!0j{0p=8Jqzj)+n zF${1U*t2QbXf-4-vwI#)h#FYbm(i`n!k!R((n9_g72nJx9QPw1E)t>hnn_ zuKthLOylj%KaRQ`D=^VjT=e$YWR@qV#2DfcA|*w*1ZLA5YOh^-oW;ew zlG_%T`IjL1%s(=!+^DfKybBA950(rqy$59S-Pufj|6+#tqZfqU=S|=d#5bp?E5`~mXg_AHsoCPQm8 z{c=$qvp*!B)u8I)ob2?1IE~YT+)PDg=~5h;-}!GNi%{}sK`?oOYK^A5&1|NbpYM6m zJv{xVp1$`V7J%jwKsE^ON9u-v+WTSs6}E{GRZq^!m&>OqWmWq z|9zXz;h@~2Q8^Y+)xPg8#QYf~NQx?{IF!PM@hpHGMvQy`GQ8@!%d&Ta+0e9_iQB(9 z5=(x{lZ$ji&&7-#qeEv*UcGo6HZC{g-7RnXA zma2j60<+EZ1!VxugchiZi+kI+rb7dVa&Og0i3AT2;X6VDqzFdF$$zOH%ms%W;Fjv3l!B4mLRvLy+xXuh)lS_PtbW+EN-l<}0lAN}3>!%^}+ zuIopIBMWhq zz)di&Y()o7O{bLDjxyr};N4{1zOo%y_(VBVz%aCu?ymJS^R?h;*^I|PDV(zHU`-MTwT0D$b(O4t`G(1f@zm#jT&o?ga)sDs zuWHS8X8zwM@Izg3+4R&j&1S_W{(_kNR9e-6Q`#zxgs5h4{~Hr({wqHukm>7^vGgTbU)TCQ>Rf6oM*#t1sF?_WzU_*SD1?1jCZ zR3*P;MeJa3N>qE>=F?h#IvhP3DA`&L#|fE?acn5yu*)rugmONFGTSO}%fS$nn2-z4 zo~Q6yD&@6)=&@uZvl=f>xZ$X8EqEpwRQ&HpnqWsbO$fiB1S3peFTnCOzN?2@GCk8)k{86C_{r8+Wj6p#0r)IG&opth-k{ z)y`(DWP&Hx+Dk)nyN&D|+WGhinK$G!0UbNLFr0CXLB~~pHmqn=)?5Z!obsB6{b6w2 zD}AfO?7161*4r1Yj#@sb1UP(xf_vkVkAOTXApn~$jY_;F6kv%{7@#=e=-03rcUBMm zT*7M1^0;Ws0?Tx01jA(T3KlI!GyEo@j!VM2Lt%ur&1dP`7Bk&x!TubrzOEBYevF^1 zfHdL~GhuSzTLWRxU6c-L5O^R}!p&8E#nN}1vE4dx~K1403{%Eh#V3x7;S^)&BL z880zPlmbG>j2mikGlyaC@17FkQqaXzYMCsWk<%8w&FD5g=a*IGXWMCg{>XG9g~I>h zULPsY=E5PfjA~Ov*Ovj&KCf&GniCLF`qLb!Y{hLPzuh1}nM+aI8R#$ZL)o z{1{*_7i|Iyk)pyxIJW!YK+sGgY=U}`mK}1ZaiV|UV@uqve!di_;Ya2U8j8+3gvi8B z7HVzlZpP6wgUQ8*sk@kiGDfu_w(fB-_8_9;n7Lll?ZAHbVQ^EIP!bDT%^QhM5wF~u zb!)a-%$+joHJ?oyS0mPwb)qP$^Q?f*BmB1#O5|UT3oIyLk6!x3QVdGFdq+H#3g+Vd z#`B_QNe=!7aVZz!Il0L{a{)=#(@Q4bq5C$DceF&hL2we)^Qta`)BHIWJ$ea)mQD1| zoxQg+w`Em}(OTax&w6$hW?OfdH?l{SMQI}d9A=YUwYHM)@_MG;*Jo3G^?FyQ#Zjm` zV}AC(W!NK))3U;kLP>99Fj3tnhrOQBB#;L_M$4Q+dZ+24cwG&}OthmJuLZPlJNx38 zeea8)CPp#vEUy!E?ahlD5G!ubt`AlO3!x0yk8SnpXkX(x>?>3hbA>;Ga(F3cvXQz1_CaXbsWyfcrJk#as0V^ch=V z=M!cn=2{tjY-GSdvvYiq3ps!kECW5-LMJ8V)_!lhCwMZSKY@;~^K4 zomd#!X)}sit?7t62#m4@7SO^AVJUIyF-sk?t}>k?)rZsNd4KXU0XgL64_UijZsopB z`OtGmmWgS^<@m6~vR|gLq_ATB%eRN|PU<6Lm52Y-)(gy2sP5Nb2Frgyu~3+RUnaKc z{0NO8aW)iw$2Jrcac!7JNYh65!{n9kCNB{zL}ZXiuvVEE`9yoSg5`^548bp@M4;2I zg66Y|BHRwsJ&Myy@7;fH=^=HHjSI^e7g)Wb#zv=Z4BQW-lWOIxnRkPHTzS!iT;@&L zOS|}5eAjX0oOvETPs056)LSeFH9Zy8#hg3;c>ZJD0nm?>V-!Q{8HBs85PKG!6?N%yb3^CBXFU%D0mm#Mi|!9w@^eOx}`+X(m*nM_p18+X64+~@T8D78e2;mhOWV^?qCEy=Vq zvg2&+IZpngSsZ9ZMq_MgBGXLlyn%~MQIVrimk5pPS}u`vk~pU6cM+W;L*nLsm^oRX z@aT9}>AaG13;hNbDynhdZf@pRMGIs)a>e3?GSkBZS)%*)@E(dCn@KMG4y$ulb!*#| z!5pt1w!Ur&{n@^)4!}Qi&kAZ)wf@DVdZ;MivK`IhEtP6*(+PIjaWtAzMnz2Ax9&)m$k;d45AKWxvlbyrK-x(Dy z7^vx(mD**C%hn0mv@jJa@7x-8PBP(sp`zI$*}sAO=>s+K%v$iwV%d7|*0s#**~h<8I`Dtq|3H&v3Ut0ps;2b%6ew28BA|?W#2w>!ktIK zoG0tQ`JB%gDgAmTHZuT7H3d2MJTA8RETQddG;#09&PG%ve<`IHrJ7eUZ2BXz{k*N_ zcz8Jf+ge@Hb-Fsh&DDa-t?q@}8MHV_j8)3<2m?4dszgTIm15C`lUa3!eJ^Tn7i{xb z)^xj<>ki$MmkwDfi%|2czwMDV{CkJr_ij!%`TmCDRBOzEZ*|X3;S%?srIxg=S8F3j z_kc$FjzN`du+}N{9aro8Qo@3Pl|mu8TFH|8SfZ;2EET^A(BO~{}Il_2e)lA>yw%SabZmV*bir@#PRppzESTgF{qYR>@~y}py^ zDH;e9xb?q`e|}CFpgX;PQ$GJfus+k5{q2y@n(eHttgQ2=ndOhvZo;}j#Y`2ey7}_C z^`do_wen?5!8FHenf}K-I+<5P>qzU)YvF97;ZNwq>K>k_RTFT+YxmGP{2r(DpWoKrlQ1DnN%oqxrzw6CGy!&fs z9x3do#cUAT&(5)!$A75yD{q9d|LOP)UgN2{XxCcY9_?)MZ}Q#NmVRLHZ(rPL@^kAT z+^yCcII(FWpVxa+Fp<#)YZaN{(8bEsR^~YpwJFRV1%CC-m&pEH*Sog&Uz(kf0*U*a z+D_H|R2vJOl6FGtBDhCg_Aqi*eh?>Vf=(sFx(vHlX{qOG*;mU~b1s=IbFP{$Wm~aa z#==vVqiZM_T`r6=c7|veMh3G@_zR=0MW_%HUFWxS6gDt=)=p`A|K0-%l_N|0P?zCd zqVGJ-3j~Z0;;tAD*B87hk^CYbdCMM|ziAOj=cD-)}iIfpw!!BLpw z>AaBL;lgVby#rK*i(aI0Zw;a}5N)+h6IYRxBI(|+mR2DkKZgT=pP{L|pswDnTKL=> zGqlu6nE*TU#STG_~~vt9R4H#2HNe{iRtBxB1) zk48_r(1Mr$e#7AZ(1Ct|K?M*t7#ge;z+<}LwR(wcgjyT^`SYHnw6{jw;G+BVs9#g* zyRKdD(HbGS)KY&?)Xse7c|(+-Z)lh^>ulX6vhkCho1Ke6$q1iz)ZqR5pFi!*)`>08 zLEb7l4CQ@k?oz6U4)mXik8k$;n5$$CgJ{k!2QGQ~F7q0)^#?i&x{kMRd%I20&C-hU zJgxCs*ig#;EXhm4mVRt-RV|c#U?M@}F)SOg_(AK2LM!K%hP$cXl_1W;J|1_S8G%3L zvZV2~!O>ZwbZDDoib~BIWNne5DAgih^;C_jBq>^=B$cXXW<-er!bc9=!*1=$S1#)# za^<=3TTbkrf5;=8zU-HI zwZ&Yp?hg~&yuW|B?ROKBew5rx5cV2QRXUP?iz1eKjpNICFz z)dAf|K^$lYOoJUmq{B%Pjh@oTSDm#rvg96e6TV&ar!)FJoPN}|J9fAR}@AI%Zf4t0lwbOsA zn=ecWuQ!HGz=6Num{v!s*!mF-IEB4=xN*JT5lq5yz|<0}-?~&J37n zD-;51dnjr{`*yr8t|rv+zig5WSHsi!W|Ex4pz$1Z)yX@D-|vFU$;Rvma#SLB3kHss zhnEs77!K8)E+|hCfyN*rsnX2>)ku|6<#e9Dk}RS^=EiB6u(~T*7Li!glnywgrXc(hpZEkd-~Zb`fy9n04DDs{_#v+3)H^Kj9YuxEbdtu1a@&>;)W z?$xm=qfKtn*p>(|&(E8tYqU+CPLw=!1Z~8+mW|z<53AV1>?1-*;6ta9oMNzayMfUqPxB0ugyxyJpeAzve6QX2jBTR}evv3sk z-%$i$Cq?3pL`=z9NhfTS(01%5js!Raw{d-$Ygfyz?#vBV5Q6y4YaVRV9zk-O@>-qU|}uX+#O zT;+Y;PYXWGU1ZKYNYFAIJ!Jf@@13y59Z+JcVMuVJ8PrezuDpO%o+kYU@=EWjb$oS; zq3Ie&_coetZpUDl;MXXvA0KrgmRAFi+RiqRYa6^%+qgp!3m$FV#sdq|dL@CRyhku> ztQZO|<5O1+5VHIGlTHZ*phX>CI*>e*bdIpUN~cjUft_-C21}9CAs>>0rNO9W78Yvm zbS)IUWDy>68-kuN&6h3q2M%-_N*xYMY)7&wd+tA^qs!p_JeInMdSOR@8%o2>l~Mf# zZD%utrLU(9gF88BfhA`##iU$;&L$V4!VptA9>42Eg?P2bpp3Adte@#S8Ble%3GT_E z61P*L>~5$GIR~4p!>TcFd4mLwC#32KYq2Z2Y8Zi*V(j2T63w9M+jH2>3gOI`r*4iK(_r`%cfHQ;$Rkyb_%d&wOh1mu5=Y-Nh{cCBh8_I;rZq~kuYe*+zmr$)78_Q{nqdWD}s01@MpI@AZli!efFGD;C6 zhu4_TYcmUJJ$sOxc$!9iBcn!+`0_WjfwhRkD|VO0fKAL!>Gn6?f7dqq;`yRgx`$az zr7{rhQYurHaT{-oV0L=Q4bzyJh(5ZqwUoR2aRK@sUJmRbYI^x+{<}jsVXoXqDAEaS z+-D2C!BsgBH{EOSg_%m+P+6p2be=bOK=f7(c8mYlYCtGVFg^f$jJ^ly-L zIr@ckR;~o8DMCS>HAm811fep4KpxZk--z{C z3Q+P+qPO#BHx%xs*cG(?1@27zX@~SP+rlB{6N{OwwkXeH037*%1tepJ!TvLh=pwxP zdk__1NdHJ$mI#F~8WS%I|Hx%OK8bAjD=4v)qAAPWjOThUgI`-LceCk?w=5^L99#Ql z<0%16(Xac>hSHW_0YK2BB;R$zBr;nh5rkBd(WPkOtYlx9?MVLcibeD{ zzpuNy+x~ov;oc4eD~T+F#>YkqX&zPNUDn&DvP4u+S&ylugIIro~6$JG{-8N^NO%Qq~IvL1UEkvQ2y8KekK*r5l=dT zmT7;TB^Xf$N2L3?5#*;`!4Z|+oX9Gc? zl;d&mhJKfww^Ef#+yC3m>T5umNt6EU#wWJ?n z(|nTQDj_J21@w3KAV6p;QE2{|vTPS9=@;?LGN!$13Co_Euk!TVq&~6s{=Iy<-^#Gu zE0K^)EY!ioLkg28fUtn|2eZ+Lwu|ZD2E2Pv8wNI=Q7H8%BC(bphZ3ycIEzedA)*i# z69Rc9G!SRv9w+G8o;aq(6r+x&wpKO?EBi=u1EBN4@stHybKbT#q)R5CT_&v)NBNaguaspo z`iklAX?7g0ecB6M2lvU0?V0ItDsyZ{bZ_0$g&FH4OE$%ZZw=khw%kNQf8IErz^W@-;Oz?DF$f=Hj1_vLiIKdVn3tKFEZR+K4 z8SmVi=gCDVl&z%-ZD=wCE&mx=kW_?<4l(^~8@X4?C( zC&k||-RZ<)QqfNmoOz??Xu1tT9FE-HTQ`e;WP(XlF=J36$D$ScvlPWYc2<&rq@Oht zx`=y#<|rtCieUu@_!|_UcX*En&+TxW-KeiMwR6|X1lDCRbM=^C7*1vXUv2~Ndn${h zrGHqLDr4%oI3?A>&iVNs7ORAJd5-kYgXU_b+j3sdy=-U9a-Jq=Nu~O^z^0LG&WN7; z|FlMe@Z@>?-5k$L?!Va#@v{jdq2W&$K5U>(K7p;c7q^5;F4pK)+TNpZ4r5c;;q&Tghd-6zfE13UA?noJ= ze5^PiCyOrS)XLEwEcLz z5cxm=S$lHr>>XGQw9mioubaOQV`cT;SGWpp=v(>s{^WRD%3swgD4~ z5YU8`kZ0oH#HQlr7~oUzbdIIzy9Bi|BYo|BUglx*{l?QN*C3kHsBLJ>uGmP06V##+ zgpn->K}@7( zo~P~(-=KLZeSaM$soOzNx^J~NEw9bE{Wrf?BxqI3R7gk-q9YbiLNKLP?j0rjin63+ zU2(%^d#sC#sCI)taO{~!y*a+)SiNs{!Cnh8AL1%@(d$6r=kwiHiH&Y@cC{({#_A&; zMBwP#@q7o=y?C=sDF}34@OD&e?Ox_gE`peB%B;TzV3XBj~iLV3oXq#Md%LcbZFv{QQdg!t(cZ(&>c2{A( zdO7HwLDNY^bCrU0tKiRupD;TrHv=wM)_{<`a2Gw?EqZelyuJTCK_!m^a#*?!8K`PH zKXW0*_)3W7#KjT|oNmXwf+!~#?FOgu-snqRE;S;yRX zAE&Q_K(71TM=ioJKf4Bt%)XY981Ognv4>I@fKDV}fP@4sA~TjDLm^jZ$O}}0ht~W3 zF+Bp1wL0=$Pmx?YI{XJujTBSBSz8b)i3~jyv;GnpxOZTgw1J=QC6sOT;sD|Fm${! ze6QCl*`#$vyPC{}fJ6##PHe)>sFX98&d64bk`_Ac?fnjm?Gfobb6ih}+$Chtu*CkU z`-WpbFQv7sr|SW34k`&)i-*4T%PZ_5KTK%Gf2IOi(;FhwYx5+2rZ3IE91Qt$leqDQ zr}?)m0IJ3TdMxJf=Cy}$IHh02wQ~88t+(-;V%Hw=o=YcojVd3n=gS}VimkKB$m^3ugeJ}%9sap* znyDx3dj-Bb%_e^5`~CJ<0fOiO z?gsbmbw;_1gEFBRv?DOus-$WDY>KfTiFcuY>mxYfNgsyKD1 zyC2g{x!dWSZRb-*gh93a6be9#=#CRCNY{?O;FjBzxKY^oX1Ge4V#t9M8GYoNLc^+27@zWZ8r>D<%~G zA83LnD1bt4$Zg$(ja|ZKw25)^YI1V0y`}0l79XKolK`z)i6&XfmYm90(ENoElGKYp zAQy4_tB6YsF-jK-LliY=<(5obj-UqxPO#$BPYSy4(|HC#he#{@HK*7QGK~`O|D75L zB@whDl-F??$gn%ac%va`yh%Ho+E*P&nEQVQuq7JM!RP|P8D>Ff*97bC%aU9TFdtdV zpw!E`!@zjsuEm#@Iv0uwSiUj(*8zgO7tLT$M}Qvxy#D;M#DV_lMjBQwL}x7FhQ`N^ zik3XsA_tv|XmB=|2+02MbX5Y$IL(O{14vpiZ|{Wh84o zd~+wi`CyL|vH1pT55c6{>!CEV-YC2>6b^3qC7I)fL_q2*sl_1!;w{CtJw_f%bML4u zxi7w*o2~)|iA*3xz82eZ24FizB_AzKd%U>yNlmNj@)cZrxThNpRnxg-If2$i}0jF!T>cvcr%B)rgLL1-5ZNj+vu5i~eAomOx%RxwJ z97lXJEXM7w;cktkSa-%WyfhgN0{}}VL z*5jN#`4~%IIA_yH#GdMmcAoU18o;_l9imm6E;4x5Y)PO=MS`tFj zwd6K}$*-l6p};`$22n29#7JWeXZ&tPSNXr#{kPFp@IluCb^7eEliBcpgEeAYWS|=-96<9E!@8VVVzJ)5m z(>awz*R8Ani~RsULjbU|evxf`pi>Y~6`i~SNx`RS8 zINa6vS@^M~-f@xLD;PxbtC&raMVGB*@O{O^YQ zi3Lz~YZ|P_rZzeo2guL{Y4Xeec$rf(K#k2AwND__blrs8Pj-OH73KVv9I1`+!sYN) z%{E>L{uT}}(X3e9Z#E(uk(9FUAR{)(#2|vAzX&(3z8NMW3gub(bk=Sta5EtmZ?6=0 zy7k4v8%zaUIv&0C|1z46qmx^waF>p~)n~cj*W0|MRHje`hry8Psns3k|F<@Q!Ue*j zI+z=j-3F4#^BQl?RUv5*;l^T7JgNZpK7IP0f}bE)5@zGoQ-g$PQ)vs0B76Da2rlB6 z!mH&wkcgZfGgvb?;vJn1harHDK15j7wLR<*>t8kqaaOAc2FY!N5GQ;EA&86!nl507 zUw^M8XSX@)yj~to88cnAV`LU@@Qg}A9DqAm(JrSf!oma?36ABF1xOqwV~Jx8yZ=ZP z&6NMVo@IS*eEN+Sg2-_kpR|k-G1X%HMv~+#V^#9krPK2(2fa;{YN*+~^59IOWr+AN z(#X%q|6+iJ4cjHuyhjSl`D{BlYQ$2Ko5!mCCHm5HG>#xb8ZCydE9dL#i+Qq8r~bXW zaH;Nwz5KR--s}8UUcO`X&%}aAl(Hi9g;3{=GD@HV>8^vhf(0uzMYnFwESvc`l91rD zQbB)>U8Sj=fK!r)iI0cN*6PxDhY!^s(626~e}z)9wmnlZ66refkeoQ=EaxZik^I4K ztmv)S?0wiS{^oLembZH4jVv^P&CgndR|UiG5JLV^`mjG(+yWs61PIVTs|b%OfVEv_ zdY{+!6|=o=Zj~J4eK=N03{&GBN;?EEyw)RRNeZNBzd0Du48x)z{3OHF@H2Na$MJ;f z&qt?{cINftp5N847We6{`C#L|;U+Ruv&g=>ZNXGsII8n4_UB+ZLPVAKv) zaIJp>u39kvDwcJUrYjuMy)PsBUctCdWXFa zcuq3p<=suy{)@4F4?i%x=uH1%Wjh92ks^gfDy@$c5JVQS`bUr!y~tnQPzkslhOxqU zN(-YHyF%eLGbDV;pEU{VoUT;xl&t@|}&qFZ)SqQ;NSyH*s*@78QpQ4!Y3RijC|DY)4AvBopY5MtIoBZ9c3#ocQr#HaAs{;b0H1CQy zOYoen-cJ$`K+|Q|I}RNc5`~>Ull3`J0zTA5{xYF#5KG{{G_QW$u>uNBJr}!M~*n%o{l+3iVg(jbZmVusU?k zRMs(En2d`Pt%ose2FRt;zHdI-0i+JB)nTV@6t zHVz+S#QIIL@61sn2@$~JU|axBzE*IO+Xh`VpEzl5lOr4=6zDsOpQ&yRFzNClByK>n zj<4@&*HPcVW|hLD>J%avJv25pJwS&}<#OCcFl>Nw2Q>Y{K49{h`>MTSK0dRxS5J6& z4jRV}(C*LB_{|$EAN3znG_9?GXzc?j3GItZBZ=n=|DOj13JZWCALg=Y#%PQ)IQh7r z$2kIM!IK|>b@(^mz5R*GpWpS$|2R0P5g9XQ1w1Yh*Y|+$8ow3k=Fk1w{G8<>U_30}6iP-{x3E)qPKZDhvetBVQy)#!`6vG0GIFMx;G`BcKh*ANfA9;&*x1>$mpqx1t!xX0{q6mECVMJm7L9D3#fYn z{E?BwBL|wYuP5SywpRW@HB=f5#X1i+Pvk!8{YVYR;jTY$U~R2#eO*Kb159pU+S;DrGcILh4E>00) zRZ3&v*U2bewj4_PGKDaOhqfdzX4qRM0VS(0q+dV=3RUm)xDBH0<8AooSiFpdFTukj zRWEX^=<}A=&6Logu!9EdPh{Yxl_4Yne&*8H?4-B)QPxg~b8jKxoS*b|nyxYDZMHK( z4;U-@?TouMfnq!?)04#~(fme`Xk&}+s+RABhImu??)MCeUl$sV#*v3$Gc3G#vDLLk z4O>Ugp#fj<5QtBdpghAON~o0qgtC%xK?K#aa|vaj&{W%TTYjV=STjrO2!`}md=E;) z#9lcBHr4T>zgtho>2xyYH^la1lNdp?+#3Ku&pef^+y_)sua76J{(^pb51C+1iF+LX zrDB24#6*~rM00i+sG=x0O%}TZTF+~kzxi_~ilhZ3oSRFizCV!R)WFaY_q8QgXr@@`IPr5SE0zQwzc)M z{u0f}mYrKObU;o5+`%G&)fuwkINKJ-O$8?Cbky|ahvdiWoSMDk*>b3%;KY@f-#W0A z!2C8<4O@diOYMxqG~sW_d~bg z^Yj2JoyV=x>96s7?R7~NW}?TG1iHrrPV!2YFZt}jL3u5LeG_;+RYwl<&|!$Oead*B zBG9K?kD);+(fBUN_=uG!#^X$4AK^8m%g|Frx8dIaJPe(mV!`Ji;TsBT{loq8T^K^qyAI`iUi}!8yr;L6Y*dTpy{+X-si4(!O zS=#Sy8^bCwtz7d#l&Hkd_C{ zYHDU@j^2ln|6C3M61Gjgq^Mw$e^xvo0(hzk-+0->R~4@JcDw6Fi4_>O(;mb!$ImH? zrjJEq%FI7j#lgK_{B1gIL$kv}9AF^+*g*=tOaY;L>WP-8H9?8^MH!Wegw~yk)S{q+A^kV_v7Y-c)teG zFkAxeY9jFsua6C&ii4_BZOZpPzNYnk6UJ6{5%e5<^+Y}n|+ z6S*A7NY%En1_hQ*LnE^2f+$U0!DQl8!Un~opD2n^F(7jJ&#_=g4Y+Ngcmaaax2m9@ zH3z((@J)K1$6*#Mcbv{lnLd$4d^#JV$A0mGtJWXhdCc!&Dg0_QFOX%T(`rUKkL~ah z{c2#^I&2>&U)x~sD&jl34~YIGldF9hZ;NAIf4sU4eiTw9YJ|j6si6d?xG=QH@$d`` z8E?0LuSH8w~mo&`R0RBJ3=d+|lFhx?Mrx})zzvFko_16&hHJxKZZA3vRpVY|H z_%YCNzTBnqd$z^KBQrbFIG9HS{3di%u2|i~=0yv%5O0ph#&~%%R$d^4AWv6L4e0&d z@tOG2J}df`ocwfyvx%*W?e=+e=z3Yx5yLGmj%wW~z0yIrn;8$+p`}hee(be`% ztlUL=ZLP+kIy{LAe|>)7Ef?zukmouJ%5ZcgXKQz5xU3qr75t!}A2cXrBc&RnMJPQ? zOvLZ8OsJg(W((Un$UcGycdZtR64%9KD&DI zwvytchRDH8_HLz9^w`l*i2*bqEXxtf3Vl-wHVsxSxVmO#X}k^{XOHJoxcWb-n~^Ff zO=*As-ry$MR~OTnP1=>!Z^YN4PnJ0jzKPia#mEe-QYbgvS+SA)Lyqul;&n8MP*(jf z2W>OlYK+zC9@dYG$1Wo_=~Oc0A=sFu(GB%exE zBlP0LN-T60`XC)7+r)X}YmopjnKU+44=43z!%`&LfA+CIXHAbvvxgxEkB|8qo2S0Y z_k6GUH0#sF^jZfC0x7!iJtaP{TEWs)16Tj3+9{%M`8sMH99^#q)|d|i{U0ovt!6Fx%Z%u;bUeDEOl1HN5UZA?h@eU)#G5r- zC|%UNNOXpU$EnjE&VAo-hXz&n`TQ}i;GqJW_56_TJ7*Zx2z z^79v!EtP4`)#X+(Vh7<2o9ZaK8f+!AT4!{fqkC&~3)|iodb2kAOUfzMDKlPiq!@#u zXQXK1juk;8-9#Y(3_l74!{DfVXY4E-7&|ytULt$FsrqL*-sfxfV*mDb8|oz{x*_Jb zpd-`)!N{r_c&hN_bcawWEai{>t(@aF;-hO_U`jSkSD;AxgBURoC++?GBxvx9>h+&1 zgX8b$_*FdIB^%bFs;vTu2qY5uM##c(VD#hfxHyQZ7}y5_=?L)L+w-}IQckfdqXVJ_ zGOgj=Y)tjuII@mX1@+Mkyq$JY?I{_$ZhQTwfn!}|?;AT`o|JspshU6f+By%eilL?2 zL`Osn@GuHKKYf2bfNZKZet`To%-XSS&-GH^gH(V8heIofgcWo;?tbt(+7AZl5y8-g z=^1?N92-m{3ibp9#K5Hh4>xSINN&V@`%@~JQNNhAVDDaDN@s03-|*(+0$e+1-|$Q6 zVyKUhW8!b0jz`5kHEre0Z0!tN3?Qsa451MSYi#id2+Vq~dKl`rIk@hz{eLFCC+Zk) zGW#57eF|>^d4OZv6aBHawY~dO!@o>yZ;~A*Z|&v39o8nEVV6&RKkC_4ROjvD!6)fV z0q|xg=!3O$v(tIGK>lj|c*5@3Ndx${b(&e*W($mMt61T3H98LHQ`U0l&D7Gn#Cv3t zbxdNEL@_r>6@yYC3k{j#6HVx(4DBd;80`>lk!FkNxO(irKp~|etCT2+&;>GP>Ku;p z!37mU)y)cdHV*70_}JD+uW058gZVM+nRgE@E!J6s<+0VwSlv&0i4)r65h_G`-n{htGfsdIjR>YjVevW5SuZ;H1` zPWyfrN(&0s=KzL(1sg)Vd@OBY#4=&P5Iy&Nn|OqSa=py2(S|q9+9I8H9(vk+EP~WO zY~t;6NO=_-Zxb(-?5-Ahoi6X?KUPV-Ctq?LcVF7Sx2`NOmaGZSgfTk!rd>!ga8R4-MsE+j-YC&E=xd+1^Y;Rclv`idcNPnK6^#qKX+W!it85p&-)G8in+5! zA4V1Wryie-P#ODwW{&O+R*CKC10*z6wxx~rvC~s?j@9+uSC=xBG3aYO9k26UITx(e zc_PBeU$bclcc+5&@^+)Y-#V;H_z~-3^OJK=V!3+Q+Mai{z<>HkrO*}Kk3)6MbOF^A z?Z5WZzwP`%)@OV8)wtEk_8%8v*ZU)mO`JihB zE95J#-S6`CD%f$a;l+Rn5)}7v_3bNVD;v+0W!02pmd(^SH*i|nJOynn+e9?a+`9qPt z6Fv2~zh+qHm9H*O(XK*V9VO3p0X)Ka1qK+NG9tObLsbm@46k~m!%N^aX{!t%uX5BE zcFzG+mPS%?2|RfuXz3@P6mw z{0Zmco2$9n`?ps->sf2UHNVgyFJV4Ye~<@%tn{JOFkb|FO%*O%ZX9UCR?&J$m~0KO7yqb|os@y13Z z?ES{$5Cv*DcurJ}b=F0l2KzziPe)p>L}v*vuf?&v*v7Z9DPO8;d|V--2ri3#9ePmYj9B`GB@RG1`dt6TtF0@Tz|q zBS8rg#p^$7C$1$Pj<71E!!{aE^!R*GK79rO`NU^fv*C(ar3l&$*$O`1G;VlF?9OBS zN+`fG8rsQtrxm#H+1=Tsw4tdh=LB^u4!OxJac};EdG0*~b+4~OI5(nfI>k=Brv(j^ z%1ImepeN|W&!2`E2r2ld=udpM^+|DQx0v`z!uc}y`M05|#wKee*+lFC=f8=Z zs3ApovnT3tDcXj`k>bC3;Ir3w+`K>pX5HM}PMC(*_n{;cmXwYSP_~XsV*RX|v5x ziji9h&8B!sF3PB4)%w(*B)L{mzkCu9Jp9yOVy>K`V!Rkz)z!?$9Q=KI_l|bL6A!bG z3NH9NPA)i=p6nv9)!_h2@8!ULw`Fv{QjT17>B>gW6a1-F2s}r|mb7Sa#Fi7z{mocA zq)9?Qq1JgfRgcm0cy^|NRjeQ_S=mSRa%XvVvd{xxW+Q!(Jbi{+z~L2RDFxfBRaMq89;m`Dv+XlX&-C-Rqduurb4>ijq07n~JF7HoQIE4z zW1w$qm+zprrZPJ>BXkFUSaSg_{$QqhG|^E;T82w$1R5x!@i)WbM*gnVS50>xMEWkH z&&g6a+)qk%qous7-BO=)#|f)M+ckuAuOKV*(c*G4?-M&B6&(BxBGW>A)wn*dkL~r( zrKXf;&qm*d_kbt_biL=5+d&StO_#lYXCmS2v}y(%LODMd1poa}-`+bIfQsyGwDsq> zEgJTyDS%ZiU3Ii-R|qNktlJwJQ=rXI5&97lc28mpi=?DW5Tw2Ey`g#qp0iKMb>RtNm7>pMv5vh_1p*3=HM_u0rr#-EgA>~oakGYbbVZ;uIduKj7E z5Is42VCqKyrf3JP$FU4W)fUiZ5N*GtKy$kQp|PHwn7Bp8TuoR`$xg`>i2lL9Y4aZy zh_(+#p=K=lDVHwZb@x!UKT44A>7CP^Q{>&u-i!>75}GS{lL~DsSxni}*}HmPyN=7g zWA?hzNerAW2uk6M9ELsqI1JG*`+&Mgg4tPhyPfmS5sj5whwJMs)BS8C-$R@2;;ir6 zY3xL1+kf@%Ckb*THBe5omm^YEpXwH<1ZhMwJ!gNtRgE9!w34$(TDJ%F1uhm_hC!%{=JV3~z7ChT^*{&%Zp zKQC`{-PJe9)5A1ThlgV;huD19ITIc+LUcgjLtkGYJ1FKEBPvSd7J0Y|e_Q_Tr}$wo z@ryucAS=z6Y32gf&Nf?5tb54?rZ0gM9Y(MXJjz%3GT)jth)7Gcm(XRFB|?4 z8=S8D{^`&tVKzQ(1Mru@x{mdU81Q>0i(Qk@((g8e(McF~V?B6}_K@Bb#>PqJghk%V@6-?lp z^0ek_Xp*CEUv*~Fz^@+~V_Bk=s!4iV&f~KMnnsk=HO;@Y7~PzXYoK`9T(yvBy$E@r z><-P){n6dYUYA%ne!H<@Da`+X{LL|vb+C}T<%bj>L(^>Tpmu)@-;u&#a*5i9)4Qv+ z8-*fVY6HiUuaZrUM51W$D2O$1S3GhZb(THufaAx_0Maf9Q2Agqf-vo)3p?!*kX}kV z-a3}uk%TBVuzDiKsspZAlY6!pQbk^Ld>pf?3}mGHf>;-n&yqgdR)qgIf(EDKN0>Esey21c{Rqma9j{D~ z{PHeKU~5RLc%)#}{ia|y_vh7G|D*Bs%_1k*_xkJR{U#KeNe?I~WZkjFEW$Te_@JH6 zOb;liPV`Hho48xmq0F9RFB%`Cg>?I$A)t8lW#l0uQelT8FeESsTyu_h)Nwoa7D5H$ zZsZHg@Y2+(Je#6&S@ulTbd{o&QOM)G^ZRciPnvdm+;f@>^!&i&jQ^11yt3i0mRU?* zJXepLfdM{@s?~1xe|f@WdwBS5W2^;G`(xE-mYoNQGNYU~gNE1IUosE85DVlKnoxtC z6r#7hv4esD=oSB(Hp9a*cG>v&)0R!BY1BxzhKAPXF^;h5fxCyYe0W zW07wdy2L%#E62-c?aX}AmPHApP>B|dTc0Y@$V_gl|9qa2@h&a6nVFi%;q&CdFhp66 z-wvt?Iy-BT7HE*xdg%JA46HOq(QQMPR8@s zIK(&97=5H^wd(jBdHRnjJZcvz`4aOIaktwf5`*;$6IsX@vD!f!mexjMybNse3o};< zMy#UjCPC`^eij$yoPJL?d+kK?$4(1>24Op4aK!+skEmYek)+Yt;`=&i;$MBoeb!fp z-x}3P!d_TjsDeXS0s5TM!4xutAaQiJ+b9@ld>XZA1+-UWK#E=3!L$FhF{NQ8Is?oW zOa%Y3st0vOmCEOpwz;i)x`WC1JTYU}264Jf~KO93o1b*?{;Q-cTTBTW$bsj%NA-OYB~4k zmc=GlACQotdr|h4*iDXTr|;10ufd(rCy7dhbeYyR4yI0Em$H=RTV$9CFzDfUqz?h)!$St= zw(j0#ar#QS!yHKdQ=&i&atn?bP9+x7v13C^a+07fs?%EHv~#NeQks_pOEVl0pAP1& z*m%GYI7F)Mh4iQLqLC#ocN`2$mvUB?;C@;L#5-@<8Ze%g)*B4#5>wb=2WN-Lvea|s zdle&5;Cg5nb6KLf^9)Iu=TYgc@4g`nU>oWj9$KcD?mHLt6E z<0(2EQa!CYTWB4~vjdqvkV*H2*r1E3e1wKc3es_kkYd22G_%G764v(8D2oHNOa~!* zYUDMS@;SIk`)ZT9?7ag}U5#@lxy%zJMYx}#3F=U2{8mm-W^4cNiPircM$sHansyxxf{-WQpyg50>%etqJ!=`i{ zSS;R_pN3MUT<5f(2D?*sh$O~*sI8>viWfQm;>z|)Ql9Bf4*--B=VK@6)}Pi{@9S@I zi;bCSWe}0xEMTI$46!IC(j#{aheK>R}BN*F4rkzv@ALd#sYNEziM=+;ZRNQoYoD8NSQ75HP;C-zcilDH@{t zfCLU~of<4K!W-Mhp}>M1n^KQs4Gip@&yN3j*VeLo39(icxyU}qrX=UDXAF{T5+fId zQCvyMTuDz0&nc~8%KPV>Lx;&bU?C3f;o_)i6n0t#ZTw})N=1FGfiO~gk=0s2uX2lu z;eG$j5)Yll?Z1jY6&N#14`f&ENXsGkIv6^UT|~|#UG#xl&0R{*#I_CkI*5l2*-NyX zn3UnYyey|1XnCy!b14%M+zIQuH;PA1JC(!Yq7f#Bz@D(`0z+8%TQQfRel?~6bdB*e zxBjvNO6L(niMw0trNB@GwmH?5u|%AF1<87VYrNRfN4z=j@NoHz78wJE$oK`{{;O4> zo%Gypz~46lUeO0F?8YhCOVr3j-q`-pCWabrRICa)ymU<;@6?~LWMCwLad{b4Kxx|9bC1#rOvNG1=K z&=By;v(Y7+oj5mpLO6G*+xX{6WSn(zA^%tT zIidPBI-V6czg+(dWER5F$wqD?;ggC5Q-vLGBUrVC&1ycwUIAjL^buC>{Tm1U_to0T z1t~SLZGMuTrHW6Sr&G2#rZiUf&y0%Y%(4q-+Nys$Ti6=aFsW$R^yQt5u6Tel!S)=oIj|jE zsUA+F(c$RiH}ypnsksJ?oS%3@CZR5kd}1%!U>Zvyv~e} z+OL0DJnKUnFk5o4I*Gll!rp{STX~|4-({AmKJCrguaS@v5i}#|FB`1N25n)5(@Vt)l~pnwGuxDV(Hpyp4O=0$$x;#qlI_ zA-o3K?_l1y4hSZ<6p-#Hgruv|}Eg z^#R@Bg=4#7^H!ILNwXz;S=3CZ5O3DerBB1C|2U|}1=#Yi>z>D^@+{1YzzlzV zXQ*fGc0U$x-OTo^Z!arRxo@`?R83I93*-JsuPJS!?y{WS_c+oI^^57%; zGwGJFiv__XF^4;y3favtbDK23r9O7uYr>v39lmna{5ps4>m1gR>Hjl_h9+PN(&_B{ z5J{N)(aCGTBKud`Bb!c^tOXtS@DnlVWY>9aQ_tK-DezzZ`rv-`=qg%bw9|R~oYO%G z$x`OQq|iuGd5U+4`Agl5V#xy9-&nll@6(2@;xZ(H+xte92t4tRma}1m?Aqh>{i{82 zq9>9KwTV|hUgr!QDz!ybe$dHM0Wi}p5J2Wup9u}Vg~aXbNLaOU3Mk9_dOj?<;||Tf zr;qhOYV6YU;iB(H#7%=DkKFOX04D+T@IN}!*LT*KL@V>;~yndDZ0WQ?R-{ap=QPk6{ko0 zOve~7KwH(2C4V*eWo)iS;sV&cAyChUvYTv%MsGK51W0}ij^VopU#9M_fbz;T{rC8n znw=)ubxn(A?kndMW&r1nL{uqxbit4Gn)7axu<{BsQ@759n5cFhqIJ+lR zwE(%F7#wgSU%RRG66t^!f=_`3>UQi`tMSMW>XZZfZ)PCtS?fV5Sg!eg(roExDrw^) z*FophVA&+IA5yxR|~ z-z5nv<8#u*rGP3O6G$~}@F~W`RVK-OOye_(a&4fgiePLzKl|q+znZ|x!M-vhB`6U% zUA<)G?dFhp1*7`LC!l?_S$1z7|ocLLRS|(J}S1o}^<$?e& zF2y|#yZ)WgOJxl;x+u=?a2HQIIzmJdF4&2&XRG>Eq~Gy#P4zt2Zl;runO1<$KjP3^ zDAjwMXhMvswHJ^Pj?tP!jKqzkwh?F(*pQQr$6+HVZ_Ngl$-;H(%{qB>3YwXw?r~ z;`h3QSYU78pA4gjwLbegx>&)8{x>v@@v5U50xZ9}!u1qabJToWDb#FxBHO}0LuM}; z1Y8qS6ZAzq>1<3+XSJ96F&)k^0PiY;tj`2xz~Bid#MWS!B$w~7fd)}Es#&vY4u5b8 z>^i2>HJ%x?q!-$2Vjo3?^*&QGYRIgGCh}`XIeC={uUKVHa>!ERYxC}7q(?QAayC@8qOu7Q|nK=6-HMF&_;)UKwIc%-nVYKOE$yalYv3PSB-pfnBnj9!M`)!=L5Zm@!6mtpA!*CIW^XnyQ2=3jg`5g9aYAgSC{r;1GZ>l zS7op!&ef5()#Vb;Dq+XYYp=tQWr%<{AZ5EcAjIi@z$2^WWH`o~W^_ z5oUJ{bXWIH-c|?y{TdKpw|TL(^?OqV9vyp=@w=qp9{`+#-W9z%mk4~Jj>&$jV#t4h|yLKkg$dzOB!{}vG@cq-7;DRxUsuwCN;=@_^MABD2 zT!(e^vjv#X)CqoyXLTy&Yx*88-HG3wBLq0icX~wual+5)!a#tL;1+LNF9%5RhRK$0 zIM=le$1FekzFDV)0s)FXV}MaHqRk8{tec}X5wtlBbS44rSyXmy2~@+9_P5tpy^F6l zVcJFT1f1k8r?`VB-)Vdx5=I*b@W`Yr$N_R;6FC5JQnLUZS zzye>ud5s}KZam;MIt&koB>Wd1)Jy?)P-@so%0S1gStd}CzR=ZvQAxsQK8DgrnSs>+ z+KG=ou6N$cRuQ)|X6-m13dc@2`|1X6)P;}rx@_^){;?(79l@ zXV#!DtN)7;aEP+#+Wgw|;tl5;mcHe+$dMc>NHm2k$yKUmQ(P6lcE3{&d1MC7uKtNC zAxc?mZNpHd>^z@z<*oXT5lzrVQQd|t*MdnPnY)y?EJXb3nrrA2KYNBqU{Z6iSPfTR z5hew2r0y;04AZPIf9JtIZGk)KspQh*1y#0rvJDk=SBXD`r#O>=_B>ioJj!wL(NJJ+ z3GbTb^79pt|DbBXVPE&}>H+TD28wT@Sbe-tKJRqNklae+El{mvArM9+CC%w^)vat^ zACwS~PH*jr@mhM{)TOi-qoTcyfqMoajTHWllzzn@&eIhr_{6B%c1${Ay2B$E7W@ho z3b_ikJ*1J(7^88qg@WOBqKYV%Np~hW(;-_h(?TbW-gBamH;gabb4qTJ-3G*%_`+s^5sO&x$1d>mri` zw7D{Rp`RKO?$rafR^2IDa>7}fo1#J$6cjebaA5+}KNsSF2{0HFpQ(x!%5AaJyS6jC zo4pA&)6t|?o~GmVX&zZ&(ZECqLGm^oeq%(j52`Kw#WMV6PYiew)*&Bws3^unmI^bI zHy5oN6|g=e$~Quo<(KWvi#FI8j~$lF5s7~e+FOp@>u5teT#^ig&;BnjPZ85vH;Ad+vAFIPY%tp39|$%_7-Mqf32nFFsKaaNmz!0uXN9J zTwght{wQ9r2K(@I=vmSzjNFSctoPbg4YfLrIa;q?Au?q8jF9jVz*;8h%8}tP{D4L1 zpkgXt_lrlp3W0Kkm0GuXmvF?QA95{TG)x3Muq-S|XEFrqN)^Mz)*a;0zluEWd!Age zT>)s?$9InYWM=Yw_HxDj_KL&+xgDMe(+0|$KF z=x-M0ZhoIU`4a|v>zCB_Cn9?_OVjnX$Oe?X z?66m%Pj9gnYX~;IUcU_+#UWUXz&E_J{<>6MiRq?F&>;U8tUD$Y95hds9;}5tp06?g z#hl-&LqYkPVri3D$KeXr^YEfP-=tvuy0-LcvB_u@UtwN=y@?>Y2YrmK+Y9Kj{>#_- zSSBbyae6*G>s|D=qIwtO(@RC~wi=5ZV~8|kbjIC2m#V1;($Vx7vz5O&{Or7ctR8zHcKv3kXks%%H+N1078uFU^hg~4 z3)6DJ?L+==0C%2azq58k_)_>Zp`YaIst3ulK&MzKejPOgDw?)RVUtvj_4$ zZ2zi9%Q_pXu;0@ds`kNOH}G@A)+W`20fNk-TS3k1^4l4!1+1?cC?B7%-4nC&GH&d| zjPQ^?sL<>1WU2Ie;NQxJ%DoT3z4;*9t%NRey59I5p@~ z8_IFk=u;b6Dp*!l9Mrlz_Wtbgm)Yo|{ay(uyEk&uEA3Xys1 z_#_aKmF9p^H9JJ=5IeH81zrl-YHKBL^E>kN6Ynf0WV% zUJvUeSVv3z?&66N9RsODM7pq{Nl-~miu3CJoc`O7;+sF`qT}p<(uy8}<#hdo6)zdw z?_$vu#-JzvyKZ~J3A#G<{nk6H+>;vQz7uM8m~eh^N|0i`l6gS{{L}DCk<+cj@kW2q zC&|(0=T2dzrH1fnF62%jIXYE4Eu}<{@^=S{j(!g8SPpk_eO6Ywl&W<04|ZI7&JU+C zbtwSu)58}1(yxIZCIh_RBXqvmEuy;8ZbGpB2%2w!X`Dc4Ftd4S0Wos=mFB z=V!!;b}^Bx&WnLMu8K+5)3wiOzzzQh^szs@;@R@c`he`?Fho{6Z%J`IDVuYi(#wf|<;|vfeia8PPOhD3V`qwTQ|4bC=A=pIG#oL;QVZ zXE5Q%5lv#%DI2DDp#8v4w`GA#Wg$xlbL5-G z%xqEb(e6a4E)8&HWfqM$8Uj3XkOU&U&!WrJ>%yoelHsnPWsONrWXZ-Y7e^~QfpH63 zE%vE@4Jc>lI`b|+mkCs!CjQb2JjoGSPkB}>e##`tYKcZL^azWXil&lXijowRn6pRngRDn3Y5=vP#g za4l=GaPNBcaCPZ*j~8cLbSw2b_;tBgL+kDTEw|emVdkP-NF!@SP%kG%r(~Q=Ll-dF z(VkafpHxb-Vmb%$1DbMbv*q=%ZWiDw+5G-&_VYv9%l4Q4%Y{YWM#y(>N02zkt|KUh zB}5)ZvnGs`_z?foX*+fI*L6Rw0QsaEmV$rUbCvC#JawIai2>G?(cJtbo>z1CHl&z1 zs$ByDiQAIe zy-NOYhzf)epPqoQN%8+wf0_n|)OSwmt{T&^$l)cMYDt)(HS~Uz$Dm42K4EiPl^v6$ zZn02>rRMMN-x+ipQC0@h3oLQ*f~<6C<+2R6$%~e%b`Ad|d&+yQ8Ka_J!|W?%dDY using Flux +This page describes Flux's take on how to construct such flexible functions +containing many parameters, and how to handle their gradients. -julia> f(x) = 3x^2 + 2x + 1; +## Parameterised Functions -julia> df(x) = gradient(f, x)[1]; # df/dx = 6x + 2 +Let's start with very simple functions. This is a polynomial in `x::Real`, +returning another real number `y` which depends on some coefficients stored in a vector: -julia> df(2) -14.0 -``` +```jldoctest poly; output = false +θ = [10, 1, 0.1] + +poly1(x::Real) = θ[1] + θ[2]*x + θ[3]*x^2 -When a function has many parameters, we can get gradients of each one at the same time: +poly1(5) == 17.5 # true -```jldoctest basics -julia> f(x, y) = sum((x .- y).^2); +# output -julia> gradient(f, [2, 1], [2, 0]) -([0.0, 2.0], [-0.0, -2.0]) +true ``` -These gradients are based on `x` and `y`. Flux works by instead taking gradients based on the weights and biases that make up the parameters of a model. +Here the parameters are a global variable `θ`. They could be handled in other ways, +for instance by explicitly passing them as an additional argument to the function: + +```jldoctest poly; output = false +poly2(x::Real, θ2) = evalpoly(x, θ2) # built-in, from Base.Math + +poly2(5, θ) == 17.5 # true -Machine learning often can have *hundreds* of parameter arrays. -Instead of passing them to `gradient` individually, we can store them together in a structure. -The simplest example is a named tuple, created by the following syntax: +# output -```jldoctest basics -julia> nt = (a = [2, 1], b = [2, 0], c = tanh); +true +``` + +Flux chooses a third path, by *encapsulating* the parameters within the function. +The simplest way to do this is a *closure*, an anonymous function which Julia knows +to depend on some local variable `θ3`: -julia> g(x::NamedTuple) = sum(abs2, x.a .- x.b); +```jldoctest poly; output = false +poly3 = let θ3 = [10, 1, 0.1] + x -> evalpoly(x, θ3) +end -julia> g(nt) -1 +poly3(5) == 17.5 # true -julia> dg_nt = gradient(g, nt)[1] -(a = [0.0, 2.0], b = [-0.0, -2.0], c = nothing) +# output + +true ``` -Notice that `gradient` has returned a matching structure. The field `dg_nt.a` is the gradient -for `nt.a`, and so on. Some fields have no gradient, indicated by `nothing`. +An equivalent, but tidier, way is to construct a `struct` in which to store the parameters. +Any struct can be made callable, allowing its instances to act just like function: -Rather than define a function like `g` every time (and think up a name for it), -it is often useful to use anonymous functions: this one is `x -> sum(abs2, x.a .- x.b)`. -Anonymous functions can be defined either with `->` or with `do`, -and such `do` blocks are often useful if you have a few steps to perform: +```jldoctest poly; output = false +struct Poly3{T} # container struct + θ3::T +end +(p::Poly3)(x::Real) = evalpoly(x, p.θ3) # make this callable -```jldoctest basics -julia> gradient((x, y) -> sum(abs2, x.a ./ y .- x.b), nt, [1, 2]) -((a = [0.0, 0.5], b = [-0.0, -1.0], c = nothing), [-0.0, -0.25]) +poly3s = Poly3([10, 1, 0.1]) # construct an instance -julia> gradient(nt, [1, 2]) do x, y - z = x.a ./ y - sum(abs2, z .- x.b) - end -((a = [0.0, 0.5], b = [-0.0, -1.0], c = nothing), [-0.0, -0.25]) +poly3s(5) == 17.5 # true + +# output + +true ``` -Sometimes you may want to know the value of the function, as well as its gradient. -Rather than calling the function a second time, you can call [`withgradient`](@ref Zygote.withgradient) instead: +Internally, there is little difference between a closure and a struct. +They have the same fields, and equivalent methods: +```julia +dump(poly3), dump(poly3s) # both contain θ3: Array +poly3s.θ3 == poly3.θ3 == θ # field called :θ3 has same value +methods(poly3) +methods(poly3s) # each has 1 method, accepting x ``` -julia> Flux.withgradient(g, nt) -(val = 1, grad = ((a = [0.0, 2.0], b = [-0.0, -2.0], c = nothing),)) + +The virtue of encapsulation is that it makes composition very easy. +We can make more complicated functions by combining simple ones, +and each will keep track of its own parameters. +Juia writes function composition as `∘`, for instance `(inv ∘ sin)(pi/6) ≈ 2`, +and we can use exactly this for our parameterised polynomials: + +```jldoctest poly; output = false +poly4 = Poly3([1, 0.5, 0]) ∘ Poly3([10, 1, 0.1]) + +poly4 isa ComposedFunction # ∘ creates another struct... +poly4.outer.θ3 == θ # which has fields :inner & :outer + +poly4(5) == 9.75 # true + +# output + +true ``` -## Building Simple Models +Flux models are precisely made by such function composition. +In fact, `poly3` and `poly4` are already valid Flux models. -Consider a simple linear regression, which tries to predict an output array `y` from an input `x`. -```julia -predict(W, b, x) = W*x .+ b +## [Structural Gradients](@id man-taking-gradients) -function loss(W, b, x, y) - ŷ = predict(W, b, x) - sum((y .- ŷ).^2) -end +The derivative of a scalar function is its slope: how fast the output changes as the input is changed slightly. +This may be found approximately by evaluating at two nearby points, and exactly by taking the limit in +which the distance between them approaches zero: -x, y = rand(5), rand(2) # Dummy data -W = rand(2, 5) -b = rand(2) +```jldoctest poly +julia> (poly1(5 + 0.1) - poly1(5)) / 0.1 +2.010000000000005 -loss(W, b, x, y) # ~ 3 +julia> (poly1(5 + 0.001) - poly1(5)) / 0.001 # answer is getting close to 2 +2.000100000003613 ``` -To improve the prediction we can take the gradients of the loss with respect to `W` and `b` and perform gradient descent. +Flux's `gradient(f, x)` works this out for `f(x)`, and gives exactly `∂f/∂x = 2.0` here: -```julia -using Flux +```jldoctest poly +julia> using Flux -dW, db = gradient((W, b) -> loss(W, b, x, y), W, b) +julia> gradient(poly1, 5) +(2.0,) ``` -Now that we have gradients, we can pull them out and update `W` to train the model. +The reason `gradient` returns a tuple, not just the number `2.0`, is to allow for +functions taking several arguments. (That's also why it's not called "derivative".) +For instance, this returns `∂f/∂x, ∂f/∂y, ∂f/∂z`: -```julia -W .-= 0.1 .* dW +```jldoctest poly +julia> gradient((x,y,z) -> (x*y)+z, 30, 40, 50) +(40.0, 30.0, 1.0) +``` + +For our parameterised polynomial, we have `∂f/∂x` but we are really more interested +in `∂f/∂θ`, as this will tell us about how the parameters are affecting the answer. +It is not impossible to track gradients with respect to global `θ`, but much clearer to track explicit arguments. +Here's how this works for `poly2` (which takes `θ` as a 2nd argument) and `poly3` (which encapsulates `θ`): + +```jldoctest poly +julia> grad2 = gradient(poly2, 5, θ) +(2.0, [1.0, 5.0, 25.0]) + +julia> grad3 = gradient((x,p) -> p(x), 5, poly3s) +(2.0, (θ3 = [1.0, 5.0, 25.0],)) +``` + +The first entry is `∂f/∂x` as before, but the second entry is more interesting. +For `poly2`, we get `∂f/∂θ` as `grad2[2]` directly. +It is a vector, because `θ` is a vector, and has elements `[∂f/∂θ[1], ∂f/∂θ[2], ∂f/∂θ[3]]`. + +For `poly3s`, however, we get a `NamedTuple` whose fields correspond to those of the struct `Poly3`. +This is called a *structural gradient*. And the nice thing about them is that they work for +arbitrarily complicated structures, for instance: -loss(W, b, x, y) # ~ 2.5 +```jldoctest poly +julia> grad4 = gradient(|>, 5, poly4) +(1.0, (outer = (θ3 = [1.0, 17.5, 306.25],), inner = (θ3 = [0.5, 2.5, 12.5],))) ``` -The loss has decreased a little, meaning that our prediction `x` is closer to the target `y`. If we have some data we can already try [training the model](../training/training.md). +Here `grad4.inner.θ3` corresponds to `poly4.inner.θ3`. +These matching nested structures are at the core of how Flux works. + +!!! note "Implicit gradients" + Earlier versions of Flux used a different way to relate parameters and gradients, + which looks like this: + ```julia + g1 = gradient(() -> poly1(5), Params([θ])) + g1[θ] == [1.0, 5.0, 25.0] + ``` + Here `Params` is a set of references to global variables using `objectid`, + and `g1 isa Grads` is a dictionary from these to their gradients. + This method of `gradient` takes a zero-argument function, which only *implicitly* + depends on `θ`. + +```@raw html +

 Zygote.jl

+``` -All deep learning in Flux, however complex, is a simple generalisation of this example. Of course, models can *look* very different – they might have millions of parameters or complex control flow. Let's see how Flux handles more complex models. +Flux's [`gradient`](@ref) function by default calls a companion packages called [Zygote](https://github.com/FluxML/Zygote.jl). +Zygote performs source-to-source automatic differentiation, meaning that `gradient(f, x)` +hooks into Julia's compiler to find out what operations `f` contains, and transforms this +to produce code for computing `∂f/∂x`. -## Building Layers +Zygote can in principle differentiate almost any Julia code. However, it's not perfect, +and you may eventually want to read its [page about limitations](https://fluxml.ai/Zygote.jl/dev/limitations/). +In particular, a major limitation is that mutating an array is not allowed. -It's common to create more complex models than the linear regression above. For example, we might want to have two linear layers with a nonlinearity like [sigmoid](https://en.wikipedia.org/wiki/Sigmoid_function) in between them. We could write this as: +Flux can also be used with other automatic differentiation (AD) packages. +It was originally written using [Tracker](https://github.com/FluxML/Tracker.jl), a more traditional operator-overloading approach. +The future might be [Enzyme](https://github.com/EnzymeAD/Enzyme.jl), and Flux now builds in an easy way to use this instead, turned on by wrapping the model in `Duplicated`. (For details, see the [Enzyme page](@ref autodiff-enzyme) in the manual.) ```julia -using Flux +julia> using Enzyme: Const, Duplicated -W1 = rand(3, 5) -b1 = rand(3) -layer1(x) = W1 * x .+ b1 +julia> grad3e = Flux.gradient((x,p) -> p(x), Const(5.0), Duplicated(poly3s)) +(nothing, (θ3 = [1.0, 5.0, 25.0],)) +``` -W2 = rand(2, 3) -b2 = rand(2) -layer2(x) = W2 * x .+ b2 +`Flux.gradient` follows Zygote's convention that arguments with no derivative are marked `nothing`. +Here, this is because `Const(5.0)` is explicitly constant. +Below, we will see an example where `nothing` shows up because the model struct has fields containing things other than parameters, such as an activation function. +(It also adopts the convention that `gradient(f, x, y)` returns a tuple `(∂f/∂x, ∂f/∂y)`, without a "`∂f/∂f`" term for the function. This is why we had to write `gradient(|>, 5, poly4)` above, not just `gradient(poly4, 5)`.) -model(x) = layer2(sigmoid.(layer1(x))) +Finally, the function [`withgradient`](@ref) works the same way, but also returns the value of the function: -model(rand(5)) # => 2-element vector +```jldoctest poly +julia> Flux.withgradient((x,p) -> p(x), 5.0, poly3s) +(val = 17.5, grad = (2.0, (θ3 = [1.0, 5.0, 25.0],))) ``` -This works but is fairly unwieldy, with a lot of repetition – especially as we add more layers. One way to factor this out is to create a function that returns linear layers. +## Simple Neural Networks -```julia -function linear(in, out) - W = randn(out, in) - b = randn(out) - x -> W * x .+ b -end +The polynomial functions above send a number `x` to another a number `y`. +Neural networks typically take a vector of numbers, mix them all up, and return another vector. +Here's a very simple one, which will take a vector like `x = [1.0, 2.0, 3.0]` +and return another vector `y = layer1(x)` with `length(y) == 2`: -linear1 = linear(5, 3) # we can access linear1.W etc -linear2 = linear(3, 2) +```jldoctest poly; output = false +W = randn(2, 3) +b = zeros(2) -model(x) = linear2(sigmoid.(linear1(x))) +sigmoid(x::Real) = 1 / (1 + exp(-x)) +layer1(x) = sigmoid.(W*x .+ b) -model(rand(5)) # => 2-element vector +# output + +layer1 (generic function with 1 method) ``` -Another (equivalent) way is to create a struct that explicitly represents the affine layer. +Here `sigmoid` is a nonlinear function, applied element-wise +because it is called with `.()`, called broadcasting. -```julia -struct Affine - W - b -end +Like `poly1` above, this `layer1` has as its parameters the global variables `W, b`. +We can similarly define a version which takes these as arguments (like `poly2`), +and a version which encapsulates them (like `poly3` above): + +```jldoctest poly; output = false +layer2(x, W2, b2) = sigmoid.(W2*x .+ b2) # explicit parameter arguments -Affine(in::Integer, out::Integer) = - Affine(randn(out, in), zeros(out)) +layer3 = let + W3 = randn(2, 3) + b3 = zeros(2) + x -> sigmoid.(W3*x .+ b3) # closure over local variables +end -# Overload call, so the object can be used as a function -(m::Affine)(x) = m.W * x .+ m.b +layer3([1.0, 2.0, 3.0]) isa Vector # check that it runs -a = Affine(10, 5) +# output -a(rand(10)) # => 5-element vector +true ``` -Congratulations! You just built the [`Dense`](@ref) layer that comes with Flux. Flux has many interesting layers available, but they're all things you could have built yourself very easily. +This third way is precisely a Flux model. And we can again make a tidier version +using a `struct` to hold the parameters: -(There is one small difference with `Dense` – for convenience it also takes an activation function, like `Dense(10 => 5, sigmoid)`.) +```jldoctest poly; output = false, filter = r"[+-]?([0-9]*[.])?[0-9]+(f[+-]*[0-9])?" +struct Layer # container struct + W::Matrix + b::Vector + act::Function +end -## Stacking It Up +(d::Layer)(x) = d.act.(d.W*x .+ d.b) # make it callabale -It's pretty common to write models that look something like: +Layer(in::Int, out::Int, act::Function=sigmoid) = + Layer(randn(Float32, out, in), zeros(Float32, out), act) -```julia -layer1 = Dense(10 => 5, relu) -# ... -model(x) = layer3(layer2(layer1(x))) +layer3s = Layer(3, 2) # instance with its own parameters + +# output + +Layer(Float32[0.6911411 0.47683495 -0.75600505; 0.5247729 1.2508286 0.27635413], Float32[0.0, 0.0], sigmoid) ``` -For long chains, it might be a bit more intuitive to have a list of layers, like this: +The one new thing here is a friendly constructor `Layer(in, out, act)`. +This is because we anticipate composing several instances of this thing, +with independent parameter arrays, of different sizes and different +random initial parameters. -```julia -using Flux +Let's try this out, and look at its gradient: + +```jldoctest poly; output = false, filter = r"[+-]?([0-9]*[.])?[0-9]+(f[+-]*[0-9])?" +x = Float32[0.1, 0.2, 0.3] # input -layers = [Dense(10 => 5, relu), Dense(5 => 2), softmax] +layer3s(x) # output, 2-element Vector{Float32} -model(x) = foldl((x, m) -> m(x), layers, init = x) +Flux.gradient((x,d) -> d(x)[1], x, layer3s)[2] # NamedTuple{(:W, :b, :act)} -model(rand(10)) # => 2-element vector +# output + +(W = Float32[0.024975738 0.049951475 0.07492722; 0.0 0.0 0.0], b = Float32[0.24975738, 0.0], act = nothing) ``` -Handily, this is also provided for in Flux: +This `∂f/∂layer3s` is a named tuple with the same fields as `Layer`. +Within it, the gradient with respect to `W` is a matrix of seemingly random numbers. +Notice that there is also an entry for `act`, which is `nothing`, +as this field of the struct is not a smoothly adjustible parameter. -```julia -model2 = Chain( - Dense(10 => 5, relu), - Dense(5 => 2), - softmax) +We can compose these layers just as we did the polynomials above, in `poly4`. +Here's a composition of 3 functions, in which the last step is the function `only` +which takes a 1-element vector and gives us the number inside: + +```jldoctest poly; output = false, filter = r"[+-]?([0-9]*[.])?[0-9]+(f[+-]*[0-9])?" +model1 = only ∘ Layer(20, 1) ∘ Layer(1, 20) + +y = model1(Float32[0.1]) # output is a Float32 number + +grad = Flux.gradient(|>, [1f0], model1)[2] -model2(rand(10)) # => 2-element vector +# output + +(outer = (outer = nothing, inner = (W = Float32[0.058179587 0.1276911 … 0.08071162 0.034993216], b = Float32[0.14223717], act = nothing)), inner = (W = Float32[-0.048111934; -0.0008379104; … ; 0.017658396; -0.015104223;;], b = Float32[-0.048111934, -0.0008379104, 0.017207285, 0.026828118, -0.024858447, -0.015956078, 0.0020494608, -0.012577536, -0.044770215, 0.01478136, 0.034534186, -0.004748393, 0.026848236, -0.016794706, -0.041044597, 0.016186379, -0.036814954, 0.034786277, 0.017658396, -0.015104223], act = nothing)) ``` -This quickly starts to look like a high-level deep learning library; yet you can see how it falls out of simple abstractions, and we lose none of the power of Julia code. +This gradient is starting to be a complicated nested structure. +But it works just like before: `grad.outer.inner.W` corresponds to `model1.outer.inner.W`. + +We don't have to use `∘` (which makes a `ComposedFunction` struct) to combine layers. +Instead, we could define our own container struct, or use a closure. +This `model2` will work the same way (although its fields have different names): + +```jldoctest poly; output = false, filter = r"[+-]?([0-9]*[.])?[0-9]+(f[+-]*[0-9])?" +model2 = let + lay1 = Layer(1, 20) # local variables containing layers + lay2 = Layer(20, 1) + function fwd(x) # equivalent to x -> only(lay2(lay1(x))) + mid = lay1(x) + lay2(mid) |> only + end +end + +model2(Float32[0.1]) -A nice property of this approach is that because "models" are just functions (possibly with trainable parameters), you can also see this as simple function composition. +Flux.gradient(|>, [1f0], model2)[2] -```julia -m = Dense(5 => 2) ∘ Dense(10 => 5, σ) +# output -m(rand(10)) +(lay2 = (W = Float32[0.051824596 0.03971491 … 0.038365345 0.051143322], b = Float32[0.09477656], act = nothing), lay1 = (W = Float32[-0.00049770635; 0.002891017; … ; -0.0022540581; 0.0039325757;;], b = Float32[-0.00049770635, 0.002891017, -0.00865399, -0.015051818, -0.005504916, -0.004188145, -0.01533527, -0.0059600063, -0.003092169, -0.00697084, -0.012470333, -0.0048766206, -0.010671042, -0.006604657, -0.0086712, -0.0044975257, -0.0028462198, -0.009992857, -0.0022540581, 0.0039325757], act = nothing)) ``` -Likewise, `Chain` will happily work with any Julia function. +```@raw html +

 Flux's layers

+``` -```julia -m = Chain(x -> x^2, x -> x+1) +Rather than define everything from scratch every time, Flux provides a library of +commonly used layers. The same model could be defined: + +```jldoctest poly; output = false +model3 = Chain(Dense(1 => 20, σ), Dense(20 => 1), only) -m(5) # => 26 +# output + +Chain( + Dense(1 => 20, σ), # 40 parameters + Dense(20 => 1), # 21 parameters + only, +) # Total: 4 arrays, 61 parameters, 452 bytes. ``` -## Layer Helpers +How does this `model3` differ from the `model1` we had before? + +* Flux's [`Chain`](@ref Flux.Chain) works left-to-right, the reverse of Base's `∘`. + Its contents is stored in a tuple, thus `model3.layers[1].weight` is an array. +* Flux's layer [`Dense`](@ref Flux.Dense) has only minor differences from our `struct Layer`: + - Like `struct Poly3{T}` above, it has type parameters for its fields -- the compiler does not know exactly what type `layer3s.W` will be, which costs speed. + - Its initialisation uses not `randn` (normal distribution) but [`glorot_uniform`](@ref) by default. + - It reshapes some inputs (to allow several batch dimensions), and produces more friendly errors on wrong-size input. + - And it has some performance tricks: making sure element types match, and re-using some memory. +* The function [`σ`](@ref NNlib.sigmoid) is calculated in a slightly better way, + and has a rule telling Zygote how to differentiate it efficiently. +* Flux overloads `Base.show` so to give pretty printing at the REPL prompt. + Calling [`Flux.@layer Layer`](@ref Flux.@layer) will add this, and some other niceties. + +All Flux layers accept a batch of samples: Instead of mapping one sample `x::Vector` to one output `y::Vector`, they map columns of a matrix `xs::Matrix` to columns of the output. This looks like `f(xs) ≈ stack(f(x) for x in eachcol(xs))` but is done more efficiently. + +If what you need isn't covered by Flux's built-in layers, it's easy to write your own. +There are more details [later](@ref man-advanced), but the steps are invariably those shown for `struct Layer` above: +1. Define a `struct` which will hold the parameters. +2. Make it callable, to define how it uses them to transform the input `x` +3. Define a constructor which initialises the parameters (if the default constructor doesn't do what you want). +4. Annotate with `@layer` to opt-in to pretty printing, and other enhacements. + +```@raw html +

 Functors.jl

+``` -We can give our layer some additional functionality, like nice printing, using the [`@layer`](@ref Flux.@layer) macro: +To deal with such nested structures, Flux relies heavily on an associated package +called Functors. Its basic function is [`fmap`](@ref Functors.fmap), +which generalises `map(f, x)` to work on almost anything. + +For example, this is how [gpu](@ref Flux.gpu) moves all arrays within a model to the GPU, +reconstructing another `only ∘ Layer(...) ∘ Layer(...)` (or a `Chain` etc.) around the new `CuArray`s: ```julia -Flux.@layer Affine +using CUDA, Functors +fmap(cu, model1) ``` -Finally, most Flux layers make bias optional, and allow you to supply the function used for generating random weights. We can easily add these refinements to the `Affine` layer as follows, using the helper function [`create_bias`](@ref Flux.create_bias): +And this is a very simple gradient update of the parameters, walking over `model` and `grad` simultaneously: ```julia -function Affine((in, out)::Pair; bias=true, init=glorot_uniform) - W = init(out, in) - b = Flux.create_bias(W, bias, out) - return Affine(W, b) +fmap((x, dx) -> x isa Array ? (x - dx/100) : x, model, grad) +``` + +!!! note + Before Flux v0.15 (and Functors v0.5), this exploration of structs was opt-in. + After defining `struct Layer` it was necessary to call `@functor Layer` (or `@layer Layer`) before Flux would look inside. + This has now changed to be opt-out: Functors (and hence Flux) will explore arbitrary structs, unless told not to (using `Functors.@leaf`). + This is why even "anonymous structs" created by closures, like `poly3` and `layer3` above, are now valid Flux models, although the use of named structs is still recommended practice. + +## Curve Fitting + +Above we took gradients of the output, or sometimes to the first element +of the output -- it must be a number, not a vector. Adjusting the parameters +to make this smaller won't lead us anywhere interesting. Instead, we should minimise +some *loss function* which compares the actual output to our desired output. + +Perhaps the simplest example is curve fitting. The [previous page](@ref man-overview) fitted +a linear model to data. With out two-layer model, we can fit a nonlinear function. +For example, let us use `f(x) = 2x - x^3` evaluated at some points `x in -2:0.1:2` as the data, +and adjust the parameters of `model3` from above so that its output is similar. + +```jldoctest poly; output = false +data = [([x], 2x-x^3) for x in -2:0.1f0:2] # training points (x, y) + +for _ in 1:1000 # adjust parameters to minimise the error: + Flux.train!((m,x,y) -> (m(x) - y)^2, model3, data, Descent(0.01)) end -Affine(3 => 1, bias=false) |> gpu +# output + +``` + +The same code will also work with `model1` or `model2` instead. +Here's how to plot the desired and actual outputs: + +```julia +using Plots +plot(x -> 2x-x^3, -2, 2, label="truth") +scatter!(x -> model3([x]), -2:0.1f0:2, label="fitted") ``` +More detail about what exactly the function `train!` is doing, and how to use rules other than simple [`Descent`](@ref Optimisers.Descent), is what the next page in this guide is about: [training](@ref man-training). diff --git a/docs/src/reference/models/layers.md b/docs/src/reference/models/layers.md index b798a35291..8e5c0e873c 100644 --- a/docs/src/reference/models/layers.md +++ b/docs/src/reference/models/layers.md @@ -1,9 +1,4 @@ -```@meta -CurrentModule = Flux -CollapsedDocStrings = true -``` - -# Built-in Layer Types +# [Built-in Layer Types](@id man-layers) If you started at the beginning of the guide, then you have already met the basic [`Dense`](@ref) layer, and seen [`Chain`](@ref) for combining layers. @@ -58,7 +53,7 @@ documented in NNlib's [Attention](@ref) section. ```@docs MultiHeadAttention -``` +``` ### Pooling @@ -75,7 +70,7 @@ GlobalMeanPool ## Upsampling -The opposite of pooling, these layers increase the size of an array. They have no trainable parameters. +The opposite of pooling, these layers increase the size of an array. They have no trainable parameters. ```@docs Upsample @@ -135,7 +130,7 @@ Flux.normalise ### Test vs. Train -Several normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. +Several normalisation layers behave differently under training and inference (testing). By default, Flux will automatically determine when a layer evaluation is part of training or inference. !!! warning This automatic train/test detection works best with Zygote, the default diff --git a/docs/src/guide/models/custom_layers.md b/docs/src/tutorials/custom_layers.md similarity index 100% rename from docs/src/guide/models/custom_layers.md rename to docs/src/tutorials/custom_layers.md