Vpf$UdZddlmZddlZddlmZmZddlZddlm Z m Z ddl Z ddl Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZddlZddlZiZd ed<eZ eZ!eZ"dwdxdZ#dZ$dZ%dZ&dZ'dZ(dyd Z)dzd#Z*d{d$Z+d%Z,d&Z-d'Z.d|d*Z/d+Z0d,Z1d-Z2d}d;Z3e3eej4<d~d<Z5e5eej6<d=Z7dd>Z8e8eej9<ddFZ:e e:dGeej;<e e:dHeej<<dIZ=dJZ>dKZ?ddLZ@ddOZAe e@dPQeejB<e e@dRQeejC<e e@dSQeejD<eAeejE<e'dTeejF<e'dUeejG<e'dVeejH<dd[ZIejJGd\d]ZKdd`ZLddbZMeLeMddcZNdddZOeLeOddeZPddfZQddgZReLeQddhZSddkZTeLeRddlZUddmZVeVeejW<ddnZXeXeejY<ddoZZeZeej[< dddrZ\ddsZ]e]dteej^<e]ej_ej j`eeja<e]ejbej jceejd<e]ejeej jfeejg<e]ejhej jieejj<e'dueejk<e'dveejl<dS)zNNN#7 6  6 6 67C S ! !!c"td|S)Nconv_general_dilatedrrs rr!9s&!7==rc"td|S)N reduce_windowrr s rr!r!:sF?C88rc"td|S)Nz%scatter_(update/add/multiply/min/max)rr s rr!r!;sV$KS%%rcfd}|S)Nc"tNr)argkwargsnames ropz_unimplemented..op@s ,,r)r*r+s` r_unimplementedr->s# )rcntfdttDS)Nc3BK|]}|VdSr')index).0iperms r z&_invert_permutation..Js-77tzz!}}777777r)tuplerangelenr3s`r_invert_permutationr9Is4 7777eCII&6&6777 7 77rxTfValx_shape core.Shapetuple[TfVal, core.Shape]cftj||tfd|DfS)zComputes transposition of x and its shape. x_shape matches x.shape in the known dimensions, and it has dimension polynomials elsewhere, while x.shape has None. r8c3(K|] }|V dSr'r,)r1r2r<s rr4z(_transpose_with_shape..Ss'1R1R'!*1R1R1R1R1R1Rr)tf transposer5)r:r< permutations ` r_transpose_with_shaperDMs< ak * * *E1R1R1R1Rk1R1R1R,R,R RRr lhs_shape rhs_shapec|\}}}t|||\}}t|dkr.|ddddddtjf}t |dz}t||d\}}t|||\}}t|dkr.|ddddddtjf}t |dz}t||d\}}t j||t j||||||fS)zTransposes lhs and rhs to respectively NHWC and HWIO so they can be passed to TF functions. The shapes passed in and returned may contain polynomials, and thus may be different than lhs.shape and rhs.shape. NrrHrJ)rLrHrJr)rDr7npnewaxisr5r_assert_matching_abstract_shape)lhsrErhsrFdimension_numberslhs_permrhs_perm_s r_transpose_for_tf_convrVVs',(Ha)iBB.#y]]a aaaAAArz! "Ci  4'I(iFF.#y)iBB.#y]]a aaaAAArz! "Ci  4'I(iFF.#y(i888(i888 ii ''rcdD]=}tj||||}t|t|kr|cS>dS)N)VALIDSAMEEXPLICIT)r padtype_to_padslist)in_shape window_shapewindow_stridespaddingpad_strpadss rpads_to_padtypercxsQ"g  x~w O OD DzzT']]"" nnn# rcrt|}t|t|dz kr d}||z|z}tj||}t|jt|ksJtdt ||D}t j||||fS)MPads `x` using `padding`, which specifies padding for the spatial dimensions.rLrrc32K|]\}\}}||z|zVdSr'r,r1xsp0p1s rr4z$_pad_spatial_dims..2II<2xB"r'B,IIIIIIr)r5r7rApadshapeziprrOr:r<r`no_pads r_pad_spatial_dimsrss 'NN'\\S\\A%%% Fw'GfQ! QWW % % % % II3w3H3HIII I I'(G444 Grc t|}t|t|dz kr d}||z|z}t|jt|ksJtdt||D}|||fS)rerLrfc32K|]\}\}}||z|zVdSr'r,ris rr4z*_check_pad_spatial_dims..rmr)r5r7rorprqs r_check_pad_spatial_dimsrvs 'NN'\\S\\A%%% Fw'G QWW % % % % II3w3H3HIII I I' GW rc d}d}t|t|cxkrt|ksntd|d|d|dt|||D]\}}\}}||zdz tj||z dz} |dz } | | z } || ks|| krd }||zdz } ||dz kr|dz } n$t t j| dz } | | z } || ks|| krd }|rd S|rd Std ) z3Finds the padding type for a transpose convolution.Tz*Found different lengths for kernel_sdims (z), lhs_dilation (z), and padding (z).rLrrJFrXrYz=Transpose convolution padding mode must be `SAME` or `VALID`.)r7 ValueErrorrpbuiltinsmaxintrMceil) kernel_sdims lhs_dilationr`is_validis_sameksbeginend pad_len_validpad_apad_b pad_len_sames r_conv_transpose_pads_to_padtypers( ' \  c,// ? ? ? ?3w<< ? ? ? ? 1&211&211&-111 2 22 lGDDaLUCEAI QUA 6 66M EE E !E ~~hq519L1q5yy!eee"',*++,,e 5 E ~~g  7 6( ) ))rrPrQctj||tj||t|dz }|dkrtd|ddS)z Check spatial dimension support.rLz1We only support 1D or 2D convolutions, but found .N)rrOr7 _conv_error)rPrErQrFnr_spatial_dimensionss r_validate_spatial_dimensionsrsy (i888(i888i..1,Q  $  $ $ $ % %%rc|r9t|dgz}t|dgz}t|dgz}|||fS)NrJrg)r\)r`r~ rhs_dilation is_conv1ds r _normalize_padding_and_dilationsrsS' %%+L %%+L 7mmvh&G , ,,rct|dkrt|dgz}t|dkrdgt|zdgz}|S)z#Ensure window_strides has length 4.rJrL)r7r\)r_s r_normalize_window_stridesrsa A.))QC/NAS4///1#5N rc |dkr|std|r|r|sn5|||gddkrtd|d|d|d|dkrtd|d|||krtd dSdS) NrJz$Grouped convolutions are unsupportedTzCan only do one of depthwise (z ), atrous (z) and transposed convolutions ()z8Unimplemented support for batch_group_count != 1 (found z0Unimplemented support for preferred_element_type)rcount) is_transpose is_atrous is_depthwisefeature_group_countbatch_group_countpreferred_element_type lhs_dtypes r_validate_conv_featuresrs1\ < = ==9y9,9i.44T::Q>>  8 8 8) 8 8(4 8 8 8 9 99 ! /+/// 0 00( )) H I II)())rrRlax.ConvDimensionNumbersrr{r precision*tuple[PrecisionType, PrecisionType] | Noner DType | None _in_avalsSequence[core.ShapedArray] _out_avalcore.ShapedArrayc  ~ | dj| dj}} | jt|| ||t| dz dk}t|}t ||||\}}}t || |||\}} }}| d}|^}}}t d|D}t d|D}||ko|dk}t|||||| |jj dt||D}|d}|rt|||}n7t| dd|||}|d krt|| |\}} }|}|d krtt d t| dd|DrEtjtjdtj t%jS|rat)||||zfz}tj|tj|t%j|||| }n|rtj|ddg}tj|d}|rt)fd|Ddz nt)fd|D t) fddD tj||t%j ||}n$tj|||||}tj|d}|r|dddddddf}t9|}tj||}|S)z;Implementation of lax.conv_general_dilated_p using XlaConv.rrJrLc3"K|] }|dkV dSrJNr,r1ds rr4z(_conv_general_dilated..s&22Q!V222222rc3"K|] }|dkV dSrr,rs rr4z(_conv_general_dilated..s&//Q!q&//////rc*g|]\}}|dz |zdzSrIr,)r1rrs r z)_conv_general_dilated..s5!Qq1ukAorrHrZrYc3(K|] \}}||kVdSr'r,)r1lrs rr4z(_conv_general_dilated..,s*#]#]daAE#]#]#]#]#]#]rdtype)inputfilterstridesr` dilations)rrJrHrLc3(K|] }|V dSr'r,r1r2 out_shapes rr4z(_conv_general_dilated..H'==A9Q<======rrIc3(K|] }|V dSr'r,rs rr4z(_conv_general_dilated..Jrrc3(K|] }|V dSr'r,)r1r2 tf_out_shapes rr4z(_conv_general_dilated..Ls'??Qa??????rrK)rfilters output_shaperr`)rrrr`r)rrHrJrLN)rorr7rrrVanyrras_numpy_dtyperprrcrvrA broadcast_toconstantfloat32r _eval_shaper5nndepthwise_conv2dreshapereverserBconv2d_transposeconv2dr9)!rPrQr_r`r~rrRrrrrrrrErFrtf_window_strides in_channelsrhs_spatial_shapesrUrhs_out_channelrrrrhs_dilated_shape output_perm padding_type new_rhs_shapeoutputrhs_t inverse_permrrs! @@r_conv_general_dilatedrs("1+Yq\-?Y)o)sIsI>>>)nnq A%)/??(H |\9)6)6%'<$:#y:=y:K$M$M #y#y" +,5)q/22\22222,//,/////) 33O8Ka8O,, <-/@0#)2JLLL"%&8,"G"G"!$+ 2L'33LL#!n'BBLz!! 7Y P Pc9glV#]#]c)AaC.J[6\6\#]#]#] ] ] ?2;q ;;;!-i88 : ::' ,--1@K1O1QQM U # #z#v1-@@AA! $  FF JsQF # #E L - -E>=========Dll=========l????,?????L U # #' 55 $FFU\\!   F < - -& AAAqqq!!!QJ F%[11, < - -& -rc~~tj||d||d|\}}}|\\}} \} } t|jt|j} } | | cxkr.t t t| krnn| | z dvrd| t| z cxkrdkrwnnsd| t| z cxkrdkrUnnQ|t|jdz fkr4| t| fkrg}| t| z dkrBt j|| dz }|t|jdz | t| z dkr?t j|| }|t|jdz t j ||t|dkr2tfd|DsJt j ||Sttjfd|jD}fd|jD}|dd}|dd}t!|| D](\}}t#}|||<|||<d||<d||<)g}t!| | D]=\}}t#}|||<|||<d||<d||<||>d }t%t'|||z|z}|j|jksJd d |d |d |}|t j |||S) zAImplementation of lax.dot_general_p in terms of tf.linalg.einsum.rrJ)rrrJrLc3:K|]}j|dkVdSrro)r1r2results rr4z_dot_general..s.<.2221$v,,222rc.g|]}tSr,rrs rrz _dot_general..rrNc |duSr'r,)r:s rr!z_dot_general..s q}rz {},{}->{}r)r$_dot_general_convert_to_common_dtyper7ror5r6rA expand_dimsappendlinalgmatmulallsqueezeiterstring ascii_lettersrprr\rrformatjoineinsum)rPrQrRrrrrconvert_resultlhs_contractingrhs_contracting lhs_batch rhs_batchlhs_ndimrhs_ndim squeeze_idxs lhs_axis_ids rhs_axis_idslhs_out_axis_idsrhs_out_axis_idslhs_axisrhs_axis shared_id batch_idsnot_none out_axis_idsspecrrs @@r _dot_generalrls#H1sIaL)55#sN@Q<$?O&& ! !! $ % %&2222 222,2222 222,!!!!_!!!!_AA&&hV I&L&L!%X!%X) 955  hV I&L&L!%X!%X Y $ $( Xy#336FFGGII, ci       BGGL11277<3H3HGGL11 3 3$  ((sC88 9 99rcg}|dz}g}tt||D]\}\}\} } } | dz} ||| z| z tj|| z} dgdt |zz} t ddd| |<|tj| | |tj|t |}tj |||}tj |tj |tj |}tj |||S)NrIrJaxisr) enumeraterprrAr6r7slicerconcat scatter_nd ones_likerMbool_where)operand padding_valuepadding_config operand_shapeindices_by_dim indices_shaperrdszrUr2dilation_factorindices expansionindices_cartesian scatteredmasks r_interior_paddingrsI.$&-,&s=.'I'IJJNNa #y1a!eOo-1222hsmmo-G!c-0001ItT**IaL"/')*.s,..GAq!a......rrc>g|]\}}}|dkr|nd|dkr|ndfSrr,r1lohirUs rrz_pad..sK;;;'b"a#%''BBq"''BBqJ;;;rCONSTANT)modeconstant_valuesc36K|]\}}}|dkp|dkVdSr"r,r%s rr4z_pad..s5::ib"aa 26::::::rc,g|]\}}}|dkr| ndSr$r,)r1r&rUs rrz_pad..s, D D DQb1ffss! D D Dr)rrrrrorArnr )rrrrrnon_negative_paddingrbeginss r_padr/s ..~.....H & 29Q<3E F FHHG;;+9;;; F  #  % % %'  ::>:::::6%io66L D D^ D D DFhw 55G .ris_minboolraxes Sequence[int] index_dtypeDTypec&|\}tj}tj|jdkr tj}|rtjjntjj}||||} tj | tj |S)N )r  output_type) rAint32r iinfobitsint64mathargminargmaxcastr _to_tf_dtype) r0rr2r4rrr r8fnrs r _argminmaxrCsz %$+ \+#b(((K3rw~~RW^" 2gDk : : :& ,[99 : ::rTFc|dvrtd|dvrM|tjtjtjtjtjfvrtd||dkr,|tjtjfvrtd||dkrX|tj tj tj tj tj tjfvrtd|t|t|cxkr't|cxkrt|krnntd d }t|d krtd t|d krDd}d|fd|fd|ffD]6\}} | d| dcxkrdkrn!td|d| 7t!|dgt|zkrtd|S)Nminrzadd5Reduction function should be either min, max, or add.)rFrzz/Min/max pool does not support operands of type rFz+Min pool does not support operands of type rGz.Add pooling does not support operands of type zgInput shapes, window dimensions, window stride dimensions, and window dilation dimensions should match.Tz"Only 1D or 2D input are supported.rLFwindow_dimensionsr_window_dilationrrrJz,Only 1D or 2D input are supported, expected z$=(1, spatial_dims, ..., 1), but got z(Unimplemented support for base dilation.) _reduce_errorrAr1uint32uint64 complex64 complex128uint8uint16bfloat16float16rfloat64int16r9r7r\) rcomputation_namerrJr_ base_dilationrKhas_only_spatial_dimsr*values r_validate_reduce_window_inputsr[s222 O P PP''Egry")R\2=6-- # ## $ $$5RXry,A#A#A MeMM N NN5kjjjhh 1$$ # ## $ $$ -C 122c.6I6I / ! " ""! < = ==!!,.?@)>:*O<>(( e qU2Y # # # #! # # # # #'#''$''(( ( $  -QC#m"4"4444 B C CC rct||||}|dko|dko|dgt|zk}|s|dkrt|||\}}d}|||fS)NrYrGrJrZrX)rcr7rs)rrrWrJr_r`rneeds_manual_paddings r_padding_reduce_windowr^Ms 0A!/::, f4!1U!:4A3]!3!333\Z77.w wOOG]L - --rc|r0tj|}tj|d|zdz}n|dd}|dd}|dd}||||fS)NrIrJr)rrrAr)rrrJr_rKrYs r_reshape_reduce_windowr`^sx  ,&}55Mj$"6"=>>GG *!B$/#AbD)N%ad+O #^_ DDrc  |j |dj} t|  ||t|| |\}} t || |\}  fd} fd} dkr | |dSdkr| | | |dSdkr1t j| |d tjSdS) Nr)rYctd fDrtd d d|jtjtjfvr.dkr(|j}tj|tj}nd}tj | | }|rtj||}rtj |dd g}tj |j |S) Nc3@K|]}tj| VdSr')r is_constant_shape)r1rs rr4z2_reduce_monoid..tf_pool..s@ ; ;Qt%a(( ( ; ; ; ; ; ;ru<TODO: use tf.nn.pool with dynamic shapes¨window_dimensions=z window_strides=z dilations=rGr)r^ pooling_typer`rrrr)rrrrArVr9r@rrpoolrrrOro) inputsreoriginal_dtyperrrWrrYrrJr_s rtf_poolz_reduce_monoid..tf_pools@  ; ; ~y 9 ; ; ;;;.  -7H - - - - ) - - . ..  28,,,2Be2K2K|nwvRZ000ffn UZZ&! F5wv^444f+z&1b'**f *69?CCC MrcTtj|tjdS)Nr)rAmultiplyr)r:rs rr!z _reduce_monoid..sR[BKE$:$:;;rrzMAXrFrGAVG) rror[r^r`rArkr=prod)rrJr_r`rXrKrWrrrrinegaterrrYrs `` ` ` @@@@r_reduce_monoidrptsr -%A,$-9%u.?_&&*@ }.0Ag**&'=,;Q 1 ;3;3;37' ni> < ; ; ;& 77E " ""5  6''&&//511 2 225   ;www.. :K0L0L M MM ! rtuple[core.ShapedArray, ...]tuple[TfVal, ...]c t|dks Jdtj| t| dzg\} } t| dkst| dkrtd| d| d} } t|jdks_t|jdjdks.s>44qDJ222222244r)r5r6r7rrors`r batch_dimszGatherArgs.batch_dimssO 4444E#dm&9":":;;444 4 44rN)__name__ __module__ __qualname____annotations__rrpropertyrr,rrrr s...####!!!!777///  44 8444rrprecondition_fnCallable[[GatherArgs], None]cdfd }|S)a Decorator for specifying a precondition function. This decorator should be put on a function with argument `arg` of type `GatherArgs`. It will first call `precondition_fn` with `arg` (which may throw an exception), and then call the function it is decorating with `arg` as well. gather_fnCallable[[GatherArgs], Any]cBtdfd }|S)Nr}rc0||Sr'r,)r}rrs rwrapperz7gather_precondition..decorator..wrapper*s#od Yt__rr}r)r)rrrs` r decoratorz&gather_precondition..decorator(sA 9 Nr)rrr,)rrs` rgather_preconditionr s) rr}cTt|jdkrtddS)zReturns True if this call to gather represents scalar indexing into arrays. E.g., op[2], op[:, :5, :], jnp.take(op, 0, axis=0). rJzstart_indices shape should be 1N)r7rrxr}s r_pre_gather_for_scalar_indexingr5s1  !""a'' 6 7 77('rctj|jjd}t j|j}t j|j}tj||j t|g}t|||}||z}td|jj D}tj|j|||}t j|t j|j}|S)zImplements 'scalar indexing into arrays' cases of lax.gather using tf.slice. E.g., op[2], op[:, :5, :], jnp.take(op, 0, axis=0). rJc3 K|] }d|zV dS)rLNr,r1r:s rr4z._gather_for_scalar_indexing..Ws&BBQAqDBBBBBBr)shrink_axis_mask)rArrstart_index_maprrrrrrr7rsumcollapsed_slice_dims strided_slicer_ensure_tf_shape_if_dynamic_aval_to_tf_shaper)r}rrslice_sizes_tfrr shrink_maskress r_gather_for_scalar_indexingr@s N4:5q 9 9'  . .(%d&677. -!3c(mm_ E E% % 0 0%#BB$*"ABBBBB+ ucKPPP#* 6 #DM 2 2  # *rc ^|j}|jj}|jj}|jj}t |dkr[t |dkrHt |dkr5|d|dkr#t |t |dz kst dtj|j ddst d|d}t |j dz }ttt|tt||zt ||zdz z}||krt d|d|dz||dzdz}tj |j|st d dS) zReturns True if this call to gather represents multi-dimensional indexing. E.g., jnp.take(op, [[0], [1]], axis=0). Note we currently only support multi-dimensional indexing if the last dimension is 1. rJrzunsupported dimension numbersrz#start_indices shape[-1] should be 1zunsupported offset_dimsNrIzunsupported slice_sizes)rrrrrr7rxr definitely_equalrr5r\r6definitely_equal_shaper) r}rrrrr  index_dimsexpected_offset_dimsexpected_slice_sizess r!_pre_gather_for_multidim_indexingr`s ](J./8 &+ h--1  _!5!5!:!: " # #q ( (-A!-D Q . . $' $4$4H 8I$I$I 4 5 55 t7;Q ? ?< : ; ;; a $4+,,q0* 5;; 5 "X+a/ 1 122233((( . / //!%4%4/(4!8992EE $T%57K L L0 . / //00rc|jjd}tj|jd}t j|j}t||f|d}tj |j ||dS)zImplements 'multi-dimensional indexing into arrays' cases of lax.gather using tf.gather. E.g., jnp.take(op, [[0], [1]], axis=0). rrrI)r r) rrrArrrrrrgatherr)r}r squeezed_indicesrrs r_gather_for_multidim_indexingrsn  ( +$Z 2B77   . .($)+;TBB- 4<Ta H H HHrc|t|jdkr%tdt|jd|jjt t |jdkrtdtj |j d|jdstddS) zReturns True if this call to gather has non-empty batch dimensions. This is for instance triggered when doing jax.vmap(lax.dynamic_slice). rJzbatch_dims is z but should be 1rzunsupported start_index_maprz9Batch dimensions in operand and start_indices don't agreeN) r7rrxrrr5r6rr rrrs r_pre_gather_with_batch_dimrs  Q Lc$/&:&:LLL M MM  Z5t/G/K)L)L#M#MMM 2 3 33  t}Q/1I!1L M M   rct|jjdkrtdt|jdvr%tdt|jddS)zReturns True if this call to gather has non-empty 2D batch dimensions. This is for instance triggered when doing jax.vmap(jax.vmap(lax.dynamic_slice)). rz0only len(collapsed_slice_dims) == 0 is supported)rJrLrHzSize of batch_dims is z but should be up to 3N)r7rrrxrrs r_pre_gather_with_batch_dimsrsv   ())Q.. G H HH  ** MT_!5!5MMM  +*rc>tjj}t|jj}t jfd|tjj j }t j |tjj j }|S)zoImplements call to gather with non-empty batch dimensions. E.g., when doing `jax.vmap(lax.dynamic_slice). cFtjj|jS)N)rsize)rAr rr)idxsr}s rr!z(_gather_with_batch_dim..s$,d9IJJJr)fn_output_signature)rrrrrrrAmap_fnrArrrrro)r}rrrs` r_gather_with_batch_dimrs   . .($"4d6FGG- 9JJJJ+DL,>??   & :ff01DEE F F& -rrotuple[int, ...]c tjtjtjd|Dddiddt |fS)a Returns the indices of the according to `shape`: each element in the output is the index of an element of an array of the provided shape. The result's shape is (math.prod(shape), len(shape)) For example, given shape (2,2) it returns (0,0),(0,1),(1,0),(1,1) c:g|]}tjd|S)rstartlimit)rAr6rs rrz,_gather_generate_indices..s':::qq***:::rindexingijrr )rArstackmeshgridr7rs r_gather_generate_indicesrsi h +::E::: EI     3u::  rctj|j}tj|jjt j|jjd}ttfd|j D}tj|jd}ttfd|jj D}tj|jd}|j |jj z}tfd|D}d} t j |j|} | ||t|jj} t j| | |t|g} t#|| |j} t jt j| |dt j||df}t j |j|}t j||}t j|t j| S) z=Implements call to gather with non-empty 2D batch dimensions.rJc3(K|] }|V dSr'r,r1r2rs rr4z*_gather_with_batch_dims..s' 5 5LO 5 5 5 5 5 5rrc3(K|] }|V dSr'r,rs rr4z*_gather_with_batch_dims..s' < <LO < < < < < .s'>>LO>>>>>>rc tjtj|d|ddg}tjtjd||}tj|||df}tj||gdS)zGGenerate the start indices of each slice, which index into the operand.rrJrrr )rAtilerrepeatr6rr )rbatch_indices_sizesize_of_index_map tiled_indicestemp_batch_indicess rget_scatter_indicesz4_gather_with_batch_dims..get_scatter_indicessG w""%7A$>M  q 23335F/1BAF 9(-8r B B BBrr rr8)rrrrrorArrrrr5rr gather_ndrr7rrrrGrrrrrBr=invert_permutation)r}rr batch_indicesroffset_indicesoffset_indices_sizedim_maskmask_output_shaperslice_start_indicesscatter_indicesindices_in_operandclipped_start_indicesslice_element_indicesresultstemprs @r_gather_with_batch_dimsrs8  . .(#DM$788, N4:5q 9 9'+ 5 5 5 5T_ 5 5 555-)-*=>>qA, < < < +?@@C_tz5 5(>>>>X>>>>>CCC$ T%7GG'' !3tz'A#B#B/},/A3x==.Q  *NOO &i%':CCCgn11566 L'< = =' G#4 5 5 5$ d!;!;H!E!E F F FFrc  |tjjkr:tjt jd} | |||||||| j||  St|||||dj|dj| } g} ttttfD]@} | | cS#t$r$}| | d|Yd}~9d}~wwxYwd | d d | z}t!d |) z$Tensorflow implementation of gather.F)multiple_results)rRrunique_indicesindices_are_sorted fill_valuerrrrrJ)rrrrrrrz: Nz"Unsupported arguments for gather: z , errors:  r)r GatherScatterMode FILL_OR_DROPr_convert_jax_impl lax_slicing _gather_fillrorrrrrrxrrr)rrrRrrrr)rrrgather_fill_fn gather_argserrorsre error_msgs r_gatherr 1sp  S " ///-k.F?DFFFN >2C-*_ Y  P P PP! |!#A,,+ &"#  + +i + Y{ # #### +++ mmy))A))********+MKLLLyy  !) x###s B(( C2CCctj|}tj|}tj|dj}t |||}tj|||S)Nr)r)rArrrrorr )rrrrrrrs r_dynamic_slicer`sb(=))-%k22.$Yq\%788- }nEE- '=~ > > >>rcRtj|}tj|dj}tj|}tj|dj}t |||}tj||}tjtj ||} tj | ||} tj tj | d} tj |} tj| | |f}tj||}tj| tj} tj| | |f} tj| |} tj| ||S)NrrJrr)rArrrrorrrGrr6rrnestflattenrrr1r)rupdaterrrrop_sizeupdate_shape_tf end_indices id_tensorscattered_indices flat_indices flat_update update_masks r_dynamic_update_slicernsS(=))-   ! 2 3 3( GG  '&y|'9::/-AA-}o66+j'**H55)&y-MM0A B BBGG,''+ ={WJ ? ?& :fh ' '& [888+ lK'DD+ ;11+ +vw / //rinverseforwardctfdtt|jD}|r|zn|z}|s|nt |}t j||S)z1Shifts the tuple of axes to the front of an arrayc3$K|] }|v|V dSr'r,)r1r2r2s rr4z%shift_axes_forward..s'KK1Qd]]Q]]]]KKr)r5r6r7ror9rArB)rr2rr other_axes fwd_orderorders ` rshift_axes_forwardr$s{ KKKKc'-&8&8 9 9KKKKK*#*AdZ T0A)" F))(;I(F(F% gu % %%rc&d fd d fd }|S) Nrrrrc|pt|jdk}|r`tj||} ||}t|jdkr|d}|d}tj|||} nZ|jddkr: r8 |tj|d|jd}  || } nt d| S)avImplementation of scatter specialised to indexing from the front axes. This covers unique indices and non-unique indices of single depth. Note on unique indices: `tf.tensor_scatter_nd_update` interprets indices thusly: every axis except the final one encodes a batch dimension, the final axis encoding the actual indices to scatter in to. It enforces, at least one, batch dimension so we add an empty dimension to indices and updates if lacking. Note on non-unique indices: There is no tf op for non-single depth indexing, but if indexing is single depth, this can be viewed as a segment op. rJNrrzgScatter only supports non-unique indices with indexing into only one dimension for (add, mul, min, max))r7rorArtensor_scatter_nd_updater_scatter_error) rrupdatesrr)rr suboperandupdated_suboperandyoperand_updateunsorted_segment_op update_ops r_sparse_scatterz2convert_scatter_jax_to_tf.._sparse_scatters $HO,A(B(Ba(GN<99j$9Z99 _ " # #q ( ()$//5 %g@R S Saa   #q ( (.A (,,W-/Z-L-L-4]1-=?? Ig~ . .   Hrrr1rc ~~~|j} |j} |j} |j}|tjtjfvrtd|| | krtd|tj j kr$|tj j krtdtt| }t|d}t|| d}|||||||| | S) a) Wrapper around the scatter function. The underlying tf ops `tf.tensor_scatter_nd_update` and `tf.math.unsorted_segment_*` index from the front dimensions. `tf.math.unsorted_segment_*` indexes to a depth 1 from the front. `tf.tensor_scatter_nd_update` indexes from the front dimensions onwards, with no ability to skip a dimension. This function shifts the axes to be indexed to the front then calls a front-specific implementation, then inverse-shifts the output. scatter_dims_to_operand_dims: dimensions which the scatter indexes in to. We shift these to the front to match tf syntax. All other dims are batch update_window_dims: dimensions which are not batch dimensions. We shift these to the back as the remaining dimensions are batch dimensions. z*Scatter does not support operands of type z"Complex scatters are not supportedzHOnly scatter modes `FILL_OR_DROP` and `PROMISE_IN_BOUNDS` are supported.)r2T)rF)r2r)update_window_dimsinserted_window_dimsscatter_dims_to_operand_dimsrrAr1rOr(r rrPROMISE_IN_BOUNDSrr$)rrr) update_jaxpr update_constsrRrrr)rrr2r3scatter_to_operand_dimsrfwdinvupdates_shiftedr0s rsparse_scatterz1convert_scatter_jax_to_tf..sparse_scatters7( m%7*=,A/L ME ",''' OOO P PP666 ? @ @@ %222 %777 @ A AA $+B C C CC #t $ $ $C)(%999O 3 CLL/?N )Y ( ( ) ))rrrrr)rr1rr1rrrrr,)r/r.r<r0s`` @rconvert_scatter_jax_to_tfr>sP% % % % % % % N7)7)7)7)7)7)p rc|Sr'r,)r:r,s rr!r!srsortreduce_precision)r)rrrrrr)r:r;r<r=rr>)rEr=rFr=)rr)rPr;rEr=rQr;rFr=)rRrrr{rr{rrrrrrrr)rrrrrrrrr=) r0r1rr;r2r3r4r5rrrr)rrrrqrrr)rrrrrr)rrr)ror)rr=rrrr)FT)r2rrr1rr1r')m__doc__ __future__rrycollections.abcrr dataclasses functoolsrrr=rtypingrjax._srcr jaxr jax._src.laxr rr r jax.experimental.jax2tfrnumpyrM tensorflowrArrr;r5 PrecisionTyperrrLr(r-r9rDrVrcrsrvrrrrrrconv_general_dilated_pr dot_general_prr/pad_prCargmin_pargmax_pr[r^r`rprreduce_window_min_preduce_window_max_preduce_window_sum_preduce_window_preduce_pselect_and_scatter_add_prng_bit_generator_pr dataclassrrrrrrrrrrrr gather_prdynamic_slice_prdynamic_update_slice_pr$r> scatter_prGunsorted_segment_sum scatter_add_prkunsorted_segment_prod scatter_mul_pryunsorted_segment_min scatter_min_przunsorted_segment_max scatter_max_psort_preduce_precision_pr,rrrjs+CBB""""""........$$$$$$$$ //////******<>====   """"">= 88 %%   888SSSS((((D      !)!)!)F % % % % - - -   JJJ.kkkk\.Cs)*Y:Y:Y:Y:x%1s !222<0!sy ; ; ; ; 'wz488s|&wz599s|444n..."EEE,CNCNCNCNL''''V GNU333s&' GNU333s&' GNU333s&''5s"#-~h77s|/=~00s+,+9.9L*M*Ms&'PPPP( 44444444,*8888455   65 >&0&0&0&0R677 I I I87 I,&/00   10  &011JGJGJG21JGX)$)$)$)$X 's|????'5s"#0000>.Cs)* (-'+&&&&&aaaaH!: 9N!!s}$=$=bf27Kg$h$hs !$=$=bk27Kh$i$is !$=$=bj27Kg$h$hs !$=$=bj27Kg$h$hs !+^F33sz)78J)K)Ks%&&&r