VpfSPdZddlmZddlmZddlmZmZddlm Z ddl Z ddl m Z ddl m ZddlmZmZmZdd lmZdd lmZmZmZmZeZeZe d gd Zeed deZeZeZeZ eegefZ!e"Z#ee#e egefZ$eegefZ%GddeZ&ee#ge'fZ(d9dZ)e)dZ*e)d:dZ+e)d:dZ,e)d;dZ-e)dd#Z0e)d>d$Z1e)d;d%Z2d?d&Z3d'Z4d@d)Z5dAd+Z6dBd/Z7dCd2Z8d3Z9d4Z:Gd5d6Z;d7Zrzs"R["/$BCcHt|d|d|dS)Nr)r)datars rrr{s^BqE47DG<<rc.eZdZUded<ded<ded<dS) OptimizerInitFninit_fnUpdateFn update_fnParamsFn params_fnN)__name__ __module__ __qualname____annotations__rrr r s3///rr opt_makeruCallable[..., tuple[Callable[[Params], State], Callable[[Step, Updates, Params], Params], Callable[[State], Params]]]returnCallable[..., Optimizer]cFtjfd}|S)a Decorator to make an optimizer defined for arrays generalize to containers. With this decorator, you can write init, update, and get_params functions that each operate only on single arrays, and convert them to corresponding functions that operate on pytrees of parameters. See the optimizers defined in optimizers.py for examples. Args: opt_maker: a function that returns an ``(init_fun, update_fun, get_params)`` triple of functions that might only work with ndarrays, as per .. code-block:: haskell init_fun :: ndarray -> OptStatePytree ndarray update_fun :: OptStatePytree ndarray -> OptStatePytree ndarray get_params :: OptStatePytree ndarray -> ndarray Returns: An ``(init_fun, update_fun, get_params)`` triple of functions that work on arbitrary pytrees, as per .. code-block:: haskell init_fun :: ParameterPytree ndarray -> OptimizerState update_fun :: OptimizerState -> OptimizerState get_params :: OptimizerState -> ParameterPytree ndarray The OptimizerState pytree type used by the returned functions is isomorphic to ``ParameterPytree (OptStatePytree ndarray)``, but may store the state instead as e.g. a partially-flattened data structure for performance. c|i|\tjfd}tjfd}tjfd}t|||S)Nct|\}}fd|D}ttt|\}}t|||S)Nc&g|] }|Sr+r+).0x0inits r zHoptimizer..tree_opt_maker..tree_init..s!333RR333r)rr mapr)x0_treex0_flattreeinitial_states states_flatsubtreesr6s r tree_initz4optimizer..tree_opt_maker..tree_inits["7++mgt33337333n$S~%F%FGGk8 Kx 8 88rc|\}}}t|\}}||kr%d}t|||tt||} tt ||| } t tt| \} } t|| D]0\} }|| kr%d}t|| |1t| ||S)Nzoptimizer update function was passed a gradient tree that did not match the parameter tree structure with which it was initialized: parameter tree {} and grad tree {}.zvoptimizer update function produced an output structure that did not match its input structure: input {} and output {}.) r TypeErrorformatr8rrr zipr)i grad_tree opt_stater=r;r> grad_flattree2msgstates new_statesnew_states_flat subtrees2subtreesubtree2updates r tree_updatez6optimizer..tree_opt_maker..tree_updates$-!k4%i00i $B 4//000>8[99fwvq))9f==j#)#lJ*G*G#H#H oy"8Y7799 '8 w  N##**Wh7788 8 OT8 < <rJparams get_paramss rtree_get_paramsz:optimizer..tree_opt_maker..tree_get_paramss?$-!k4>8[99f:v&&f D& ) ))r) functoolswrapsr ) argskwargsr?rQrUrTr6rPr,s @@@rtree_opt_makerz!optimizer..tree_opt_makers(y$9&99D&*_T99999 _V=====$_Z  ****! * Y _ = ==r)rVrW)r,rZs` r optimizerr[s=H ?9$>$>$>$>$>J rcZtd}fd}d}t|||S)aConstruct optimizer triple for stochastic gradient descent. Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. Returns: An (init_fun, update_fun, get_params) triple. c|Srr+)r5s rr6zsgd..inits Irc&|||zz Srr+)rDgx step_sizes rrPzsgd..updates yy||a rc|Srr+)r`s rrTzsgd..get_paramss Hr) make_scheduler )rar6rPrTs` rsgdrds[I&&)        4 , ,,rraSchedulemassfloatcFtd}fd}d}|||fS)aFConstruct optimizer triple for SGD with momentum. Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. mass: positive scalar representing the momentum coefficient. Returns: An (init_fun, update_fun, get_params) triple. c2tj|}||fSrjnp zeros_liker5v0s rr6zmomentum..init   B r6MrcH|\}}|z|z}|||zz }||fSrr+rDr_stater`velocityrfras rrPzmomentum..updates;KAxh"H IIaLL8 ##A h;rc|\}}|Srr+rrr`_s rrTzmomentum..get_params DAq Hrrcrarfr6rPrTs`` rmomentumrz^I&&)     vz !!rcFtd}fd}d}|||fS)aOConstruct optimizer triple for SGD with Nesterov momentum. Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. mass: positive scalar representing the momentum coefficient. Returns: An (init_fun, update_fun, get_params) triple. c2tj|}||fSrrjrms rr6znesterov..initrorcT|\}}|z|z}|||z|zzz }||fSrr+rqs rrPznesterov..updatesDKAxh"H IIaLLD8Oa/ 00A h;rc|\}}|Srr+rus rrTznesterov..get_params"rwrrxrys`` rnesterovr r{r?cFtd}fd}d}|||fS)aConstruct optimizer triple for Adagrad. Adaptive Subgradient Methods for Online Learning and Stochastic Optimization: http://www.jmlr.org/papers/volume12/duchi11a/duchi11a.pdf Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. momentum: optional, a positive scalar value for momentum Returns: An (init_fun, update_fun, get_params) triple. c\tj|}tj|}|||fSrrj)r5g_sqms rr6zadagrad..init9s, >"  D rA tQ;rc|\}}}|tj|z }tj|dkdtj|z d}dz ||zz|zz}|||zz }|||fS)Nr?g)rksquarewheresqrt) rDr_rrr`rr g_sq_inv_sqrtrzras rrPzadagrad..update>sJAtQCJqMMDIdQhSXd^^(;SAAM h1},-1 .get_paramsFGAq! Hrrx)rarzr6rPrTs`` radagradr(s^I&&)     vz !!r:0yE>cJtd}fd}d}|||fS)a7Construct optimizer triple for RMSProp. Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. gamma: Decay parameter. eps: Epsilon parameter. Returns: An (init_fun, update_fun, get_params) triple. c2tj|}||fSrrj)r5 avg_sq_grads rr6zrmsprop..init[s.$$K {?rc|\}}|ztj|dz zz}|||ztj|zz z }||fSNrrkrr)rDr_rrr`repsgammaras rrPzrmsprop..update^saNA{% 1 e(DDK IIaLL1 sx c(9:: ::A k>rc|\}}|Srr+rus rrTzrmsprop..get_paramscrwrrx)rarrr6rPrTs``` rrmsproprMsdI&&)     vz !!rcNtd}fd}d}|||fS)aConstruct optimizer triple for RMSProp with momentum. This optimizer is separate from the rmsprop optimizer because it needs to keep track of additional parameters. Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. gamma: Decay parameter. eps: Epsilon parameter. momentum: Momentum parameter. Returns: An (init_fun, update_fun, get_params) triple. c\tj|}tj|}|||fSrrj)r5rmoms rr6zrmsprop_momentum..init{s-.$$K .  C {C rc|\}}}|ztj|dz zz}|z ||ztj|zz z}||z }|||fSrr) rDr_rrr`rrrrrzras rrPz rmsprop_momentum..updatestA{C% 1 e(DDK S.99Q<.get_paramsrrrx)rarrrzr6rPrTs```` rrmsprop_momentumrisj"I&&)        vz !!r+?cNtd}fd}d}|||fS)a|Construct optimizer triple for Adam. Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. b1: optional, a positive scalar value for beta_1, the exponential decay rate for the first moment estimates (default 0.9). b2: optional, a positive scalar value for beta_2, the exponential decay rate for the second moment estimates (default 0.999). eps: optional, a positive scalar value for epsilon, a small constant for numerical stability (default 1e-8). Returns: An (init_fun, update_fun, get_params) triple. c\tj|}tj|}|||fSrrj)r5m0rns rr6zadam..init,   B   B r2:rc`|\}}}dz |z|zz}d z tj|z |zz}|dtj|j|dzzz z }|dtj |j|dzzz z }| ||ztj| zz z }|||fSNr)rkrasarraydtyper) rDr_rrr`rvmhatvhatb1b2rras rrPzadam..updatesGAq! R1 rAvA R3:a== 26)A CKAG,,Q77 8D CKAG,,Q77 8D IIaLL4 38D>>C#7 88A a7Nrc|\}}}|Srr+rus rrTzadam..get_paramsrrrxrarrrr6rPrTs```` radamrsj"I&&)    vz !!rcNtd}fd}d}|||fS)aConstruct optimizer triple for AdaMax (a variant of Adam based on infinity norm). Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. b1: optional, a positive scalar value for beta_1, the exponential decay rate for the first moment estimates (default 0.9). b2: optional, a positive scalar value for beta_2, the exponential decay rate for the second moment estimates (default 0.999). eps: optional, a positive scalar value for epsilon, a small constant for numerical stability (default 1e-8). Returns: An (init_fun, update_fun, get_params) triple. c\tj|}tj|}|||fSrrj)r5ru0s rr6zadamax..initrrc|\}}}dz |z|zz}tj|ztj|}| |dtj|j|dzzz z |z|zz z }|||fSr)rkmaximumabsrr) rDr_rrr`rurrrras rrPzadamax..updatesGAq! R1 rAvA BFCGAJJ''A iilla#+b!'":":q1u"EEF! K G A a7Nrc|\}}}|Srr+rus rrTzadamax..get_paramsrrrxrs```` radamaxrsj"I&&)    vz !!rc^tdfdd}fd}d}|||fS)aConstruct optimizer triple for SM3. Memory-Efficient Adaptive Optimization for Large-Scale Learning. https://arxiv.org/abs/1901.11150 Args: step_size: positive scalar, or a callable representing a step size schedule that maps the iteration index to a positive scalar. momentum: optional, a positive scalar value for momentum Returns: An (init_fun, update_fun, get_params) triple. c8t|}||||dz<|Sr)list)seqrDr`lsts rsplicezsm3..splices# s))CC!A#J Jrcldg|z|tdg}|t|Sr)slicetuple)ndimr`axisidxrs rbroadcast_intozsm3..broadcast_intos4 &$$uT{{m 4 4C U3ZZ=rcj}tjfdjD}tj||fS)NcFg|]}tj|jS))r)rkzerosr)r4szr5s rr7z%sm3..init..s* ; ; ;B#)Bbh ' ' ' ; ; ;r)shaperk atleast_1drl)r5x_shapevss` rr6zsm3..initsMhG   B ; ; ; ;"( ; ; ;B s~b!!2w ..rc|\}}} fdt|D}tjtj|tjztjdkdtjz d}d z |zz |zz} ||zz  fdtj D}|||fS)Nc:g|]\}}j||Sr+)r)r4rDrrr_s rr7z'sm3..update..s- A A A41a..A & & A A Arrrc tg|]4}tj|g5Sr+)maxranger)r4jaccumrr`s rr7z'sm3..update..s; I I Ia%))FF5==!R00 1 1 I I Ir) enumeraterVreducerkminimumrrrrr) rDr_rrrrraccum_inv_sqrtrr`rrzrras ` @@rrPzsm3..updatesAq"g A A A A A9R== A A AB  S[" - - 1 =EYuqy"sx*>BBN h1~-.A=A IIaLL1 A I I I I I I5== I I IB aW rc:|\}}}}||Sr)reshape)rrr`rvrs rrTzsm3..get_paramss"Aq!W 99W  rrx)rarzr6rPrTrrs`` @@rsm3rsI&&) ///  vz !!rcfd}|S)NcSrr+)rDras rschedulezconstant..schedules rr+)rars` rconstantrs! /rcfd}|S)Nc|z zzSrr+rD decay_rate decay_stepsras rrz#exponential_decay..schedule s za+o6 66rr+)rarrrs``` rexponential_decayr s-7777777 /rFc.|rfd}nfd}|S)NcDdtj|z zzz Sr)rkfloorrs rrz$inverse_time_decay..schedules& !j39Q_+E+EEE FFrc d|zz zz Srr+rs rrz$inverse_time_decay..schedules !j1n{:: ;;rr+)rarr staircasers``` rinverse_time_decayrse<GGGGGGGG<<<<<<< /rrcfd}|S)NcZtj|}d|z z z}|z zzSr)rkr)step_num step_multrfinal_step_sizepowerras rrz"polynomial_decay..schedules={8[11HX ++5I  O3 4 FFrr+)rarrrrs```` rpolynomial_decayrs=GGGGGGGG /r boundariesrvaluesctjtjjjcxkrdksntdjdjddz kstdfd}|S)Nrz'boundaries and values must be sequencesrz8boundaries length must be one shorter than values lengthc@tj|kSr)rksum)rDrrs rrz$piecewise_constant..schedule(s #'!j.)) **r)rkarrayr ValueErrorr)rrrs`` rpiecewise_constantr sy$$* 9V  & FK , , , ,1 , , , , > ? ??  !  Q! 3 3 3 O P PP++++++ /rscalar_or_schedulefloat | Schedulect|r|Stj|dkrt|St t |)Nr)callablerkrrrAtype)rs rrcrc,sU  !!.  x"##q(( & ' '' D+,, - --rc|t|\}}tjtd|DS)zCCompute the l2 norm of a pytree of arrays. Useful for weight decay.c3@K|]}tj||VdSr)rkvdot)r4r`s r zl2_norm..:s,55chq!nn555555r)rrkrr)r;leavesrvs rl2_normr7s:4  )&! #55f55555 6 66rcPt|fd}t||S)zGClip gradients stored as a pytree of arrays to maximum norm `max_norm`.cDtjk||z zSr)rkr)r_max_normnorms rrzclip_grads..?s" $/1a8d?6KLLr)rr )rEr  normalizer s ` @r clip_gradsr <s2   $LLLLL) )Y ' ''rceZdZdZdZdZdS) JoinPointz7Marks the boundary between two joined (nested) pytrees.c||_dSrrN)selfrNs r__init__zJoinPoint.__init__Gs DLLLrc#K|jVdSrr)rs r__iter__zJoinPoint.__iter__Ks ,rN)r'r(r)__doc__rrr+rrrrEs8??rrc||\}}}tt||}d|D}tj||S)aConverts an OptimizerState to a marked pytree. Converts an OptimizerState to a marked pytree with the leaves of the outer pytree represented as JoinPoints to avoid losing information. This function is intended to be useful when serializing optimizer states. Args: opt_state: An OptimizerState Returns: A pytree with JoinPoint leaves that contain a second level of pytrees. c,g|]}t|Sr+)r)r4rNs rr7z*unpack_optimizer_state..\s :::gy!!:::r)r8rr )rFr=rrr> sentinelss runpack_optimizer_staterNsG)2%+x { ; ;(:::::)  !(I 6 66rct|\}}td|DsJd|D}ttt|\}}t |||S)aConverts a marked pytree to an OptimizerState. The inverse of unpack_optimizer_state. Converts a marked pytree with the leaves of the outer pytree represented as JoinPoints back into an OptimizerState. This function is intended to be useful when deserializing optimizer states. Args: marked_pytree: A pytree containing JoinPoint leaves that hold more pytrees. Returns: An equivalent OptimizerState to the input argument. c3@K|]}t|tVdSr) isinstancerr4ss rrz'pack_optimizer_state..ms, 9 9!Z9 % % 9 9 9 9 9 9rcg|] }|j Sr+rrs rr7z(pack_optimizer_state..ns + + +Aai + + +r)rallr r8r) marked_pytreerrr>r=rs rpack_optimizer_stater"_sz%]33)X 9 9y 9 9 9 9 9999 + + + + +($Sx%@%@AA+|  X| < <r __future__rcollections.abcrtypingrr collectionsrrVr jax.numpynumpyrk jax._src.utilr r r jaxr jax.tree_utilr rrrr8rCrArrayParamsStateUpdatesr!intStepr#r%r rgrer[rdrzrrrrrrrrrrrrrcrr rrr"r+rrr2sKKZ#"""""$$$$$$""""""""""""""4444444444111111111111,HHHJJCC<<>>>     6(N* +  T7N3^C D ^$f, -  TFEM "JJJJ^ -- -& """ "4 """ "4 !"!"!" !"H """ "6 """ "B  " " "  "F """ "D -"-"-" -"d      ....777 (((777"=====r