VpfhdZddlmZddlmZddlmZddlZddlZ ddl m Z m Z ddl Z ddlZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZmZdd lmZddl m!Z!m"Z"ddl#m$Z%GddZ&e&Z'eej(dgdZ)e)*dej(dhdidZ+ej(dgdZ,ej(dgdZ-ej(dgdZ.eej(d dgd!Z/ej(dgd"Z0ej(dgd#Z1e1Z2ej(dgd$Z3ej(dgd%Z4ej(djdkd(Z5ej(dldmd+Z6ej(dgd,Z7ej(djdkd-Z8ej(dgd.Z9dndod1Z:eej(d23dpdqd7Z;e%Z$eej(d23d4de'fdrd=ZZ=eejd?@d4dej> fdsdAZ?e?j@dBZAd4dej> fdsdCZBeej(d23 dtdudHZCeej(dI3dvdNZDejEd4dOdvdPZFejej(dgdQZGeG*dRej(dgdSZHej(dgdTZIeIZJdUZKdVZLdwd`ZMddddaddbdxdfZdS)yz6Shared neural network activations and other functions.) annotations)Sequence)partialN)AnyLiteral) custom_jvp)lax)config)core)dtypes)util)AxisName)dot_product_attentionMaskType)Array ArrayLike) logsumexpceZdZdZdS) UnspecifiedcdS)N _UNSPECIFIED)selfs U/var/www/html/nettyfy-visnx/env/lib/python3.11/site-packages/jax/_src/nn/functions.py__repr__zUnspecified.__repr__+s >N)__name__ __module__ __qualname__rrrrrr*s#rrxrreturnrc,tj|dS)uJRectified linear unit activation function. Computes the element-wise function: .. math:: \mathrm{relu}(x) = \max(x, 0) except under differentiation, we take: .. math:: \nabla \mathrm{relu}(0) = 0 For more information see `Numerical influence of ReLU’(0) on backpropagation `_. Args: x : input array Returns: An array. Examples: >>> jax.nn.relu(jax.numpy.array([-2., -1., -0.5, 0, 0.5, 1., 2.])) Array([0. , 0. , 0. , 0. , 0.5, 1. , 2. ], dtype=float32) See also: :func:`relu6` r)jnpmaximumr s rrelur&2sB Q  rc\tj|dk|tj|dS)Nrr select full_likegansr s rr.Us$sz!a%CM!Q4G4GHHrbctjd|tjd|tj|}tj|}|tjtj||zz}|dz S)zSquareplus activation function. Computes the element-wise function .. math:: \mathrm{squareplus}(x) = \frac{x + \sqrt{x^2 + b}}{2} as described in https://arxiv.org/abs/2112.11687. Args: x : input array b : smoothness parameter squareplus) numpy_utilcheck_arrayliker#asarraysqrtsquare)r r0ys rr2r2Wsn \1--- \1--- k!nn! k!nn!#(3:a==1$ % %%! Q,rc,tj|dS)zSoftplus activation function. Computes the element-wise function .. math:: \mathrm{softplus}(x) = \log(1 + e^x) Args: x : input array r)r# logaddexpr%s rsoftplusr<ms q!  rc tjd|tj|}tj|dkdtj|dk||dzdzdz S)aSparse plus function. Computes the function: .. math:: \mathrm{sparse\_plus}(x) = \begin{cases} 0, & x \leq -1\\ \frac{1}{4}(x+1)^2, & -1 < x < 1 \\ x, & 1 \leq x \end{cases} This is the twin function of the softplus activation ensuring a zero output for inputs less than -1 and a linear output for inputs greater than 1, while remaining smooth, convex, monotonic by an adequate definition between -1 and 1. Args: x: input (float) sparse_plusg?r3r/r4r5r#r6wherer%s rr>r>{sZ, ]A... k!nn! 19c39Q#Xq1s7Q,q.#I#I J JJrctjd|tj|}|tj|dzz S)zSoft-sign activation function. Computes the element-wise function .. math:: \mathrm{soft\_sign}(x) = \frac{x}{|x| + 1} Args: x : input array soft_sign)r4r5r#r6absr x_arrs rrDrDs< [!,,, +a..% #'%..1$ %%rT)inlinec*tj|S)zSigmoid activation function. Computes the element-wise function: .. math:: \mathrm{sigmoid}(x) = \frac{1}{1 + e^{-x}} Args: x : input array Returns: An array. See also: :func:`log_sigmoid` )r logisticr%s rsigmoidrLs& arc:dtj|dzddzS)aSparse sigmoid activation function. Computes the function: .. math:: \mathrm{sparse\_sigmoid}(x) = \begin{cases} 0, & x \leq -1\\ \frac{1}{2}(x+1), & -1 < x < 1 \\ 1, & 1 \leq x \end{cases} This is the twin function of the ``sigmoid`` activation ensuring a zero output for inputs less than -1, a 1 output for inputs greater than 1, and a linear output for inputs between -1 and 1. It is the derivative of ``sparse_plus``. For more information, see `Learning with Fenchel-Young Losses (section 6.2) `_. Args: x : input array Returns: An array. See also: :func:`sigmoid` ?r@r?g@)r#clipr%s rsparse_sigmoidrPs!< sxCc** **rcxtjd|tj|}|t |zS)aHSiLU (aka swish) activation function. Computes the element-wise function: .. math:: \mathrm{silu}(x) = x \cdot \mathrm{sigmoid}(x) = \frac{x}{1 + e^{-x}} :func:`swish` and :func:`silu` are both aliases for the same function. Args: x : input array Returns: An array. See also: :func:`sigmoid` silu)r4r5r#r6rLrGs rrRrRs5( VQ''' +a..%  rctjd|tj|}|tjt |zS)aMMish activation function. Computes the element-wise function: .. math:: \mathrm{mish}(x) = x \cdot \mathrm{tanh}(\mathrm{softplus}(x)) For more information, see `Mish: A Self Regularized Non-Monotonic Activation Function `_. Args: x : input array Returns: An array. mish)r4r5r#r6tanhr<rGs rrTrTs?& VQ''' +a..% (5//** **rcvtjd|tj|}t |  S)zLog-sigmoid activation function. Computes the element-wise function: .. math:: \mathrm{log\_sigmoid}(x) = \log(\mathrm{sigmoid}(x)) = -\log(1 + e^{-x}) Args: x : input array Returns: An array. See also: :func:`sigmoid` log_sigmoid)r4r5r#r6r<rGs rrWrW s7$ ]A... +a..% E6   rr@alphac tjd|tj|}tj|dk||tjtj|dkd|zS)akExponential linear unit activation function. Computes the element-wise function: .. math:: \mathrm{elu}(x) = \begin{cases} x, & x > 0\\ \alpha \left(\exp(x) - 1\right), & x \le 0 \end{cases} Args: x : input array alpha : scalar or array of alpha values (default: 1.0) Returns: An array. See also: :func:`selu` elurr?)r4r5r#r6rBexpm1)r rXrHs rrZrZ!sh, UA&&& +a..% 51939SYuqy"e%D%DEEE G GGr{Gz?negative_slopectjd|tj|}tj|dk|||zS)aLeaky rectified linear unit activation function. Computes the element-wise function: .. math:: \mathrm{leaky\_relu}(x) = \begin{cases} x, & x \ge 0\\ \alpha x, & x < 0 \end{cases} where :math:`\alpha` = :code:`negative_slope`. Args: x : input array negative_slope : array or scalar specifying the negative slope (default: 0.01) Returns: An array. See also: :func:`relu` leaky_relurrA)r r]rHs rr_r_=sA0 \1--- +a..% 5A:unu&< = ==rc tjd|tj|}tj|dkdtj|dkd|S)aHard :math:`\mathrm{tanh}` activation function. Computes the element-wise function: .. math:: \mathrm{hard\_tanh}(x) = \begin{cases} -1, & x < -1\\ x, & -1 \le x \le 1\\ 1, & 1 < x \end{cases} Args: x : input array Returns: An array. hard_tanhrErArGs rraraYsN& [!,,, +a..% 519a52:r5!A!A B BBrctj|d|tjtj|d|z zzS)aContinuously-differentiable exponential linear unit activation. Computes the element-wise function: .. math:: \mathrm{celu}(x) = \begin{cases} x, & x > 0\\ \alpha \left(\exp(\frac{x}{\alpha}) - 1\right), & x \le 0 \end{cases} For more information, see `Continuously Differentiable Exponential Linear Units `_. Args: x : input array alpha : array or scalar (default: 1.0) Returns: An array. r?)r#r$r[minimum)r rXs rcelureps;. Q  usyQ1D1Du1L'M'MM MMrc0d}d}|t||zS)a Scaled exponential linear unit activation. Computes the element-wise function: .. math:: \mathrm{selu}(x) = \lambda \begin{cases} x, & x > 0\\ \alpha e^x - \alpha, & x \le 0 \end{cases} where :math:`\lambda = 1.0507009873554804934193349852946` and :math:`\alpha = 1.6732632423543772848170429916717`. For more information, see `Self-Normalizing Neural Networks `_. Args: x : input array Returns: An array. See also: :func:`elu` g,x?g2֫?)rZ)r rXscales rselurhs!8 ,% +% Q r approximateboolc tjd|\}|rdtjdtjz |j}ddtj||d|dzzzzzz}||zStjd|j}tj |tj ||z dzzdz |jS) aGaussian error linear unit activation function. If ``approximate=False``, computes the element-wise function: .. math:: \mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{erf} \left( \frac{x}{\sqrt{2}} \right) \right) If ``approximate=True``, uses the approximate formulation of GELU: .. math:: \mathrm{gelu}(x) = \frac{x}{2} \left(1 + \mathrm{tanh} \left( \sqrt{\frac{2}{\pi}} \left(x + 0.044715 x^3 \right) \right) \right) For more information, see `Gaussian Error Linear Units (GELUs) `_, section 2. Args: x : input array approximate: whether to use the approximate or exact formulation. gelur3rNr@gHm?rEdtype) r4promote_args_inexactnpr7piastyperor#rUarrayr erf)r rirHsqrt_2_over_picdfsqrt_2s rrlrls,  +FA 6 6'5SWQY''..u{;;N sx%(eqj:Q2Q RSSS TC 3; WQZZ  u{ + +F 9Ucgefn559:Q>ek R R RRraxis)static_argnamesrbrzintctjd|tj|}|j|}|dzdks Jdtj|d|\}}|t |zS)a Gated linear unit activation function. Computes the function: .. math:: \mathrm{glu}(x) = x\left[\ldots, 0:\frac{n}{2}, \ldots\right] \cdot \mathrm{sigmoid} \left( x\left[\ldots, \frac{n}{2}:n, \ldots\right] \right) where the array is split into two along ``axis``. The size of the ``axis`` dimension must be divisible by two. Args: x : input array axis: the axis along which the split should be computed (default: -1) Returns: An array. See also: :func:`sigmoid` glur3rz axis size must be divisible by 2)r4r5r#r6shapesplitrL)r rzrHsizex1x2s rr~r~sp0 UA&&& +a..% T $ Q: 9UAt $ $&"b gbkk rint | tuple[int, ...] | NonerBArrayLike | NoneinitialArrayLike | None | Unspecifiedc,|turtjdtd~t jd|t j|}t j|||t j d}||n t j ||t j }|tj |z }t j t jt j|||d}||z } |!t j || t j S| S) aLog-Softmax function. Computes the logarithm of the :code:`softmax` function, which rescales elements to the range :math:`[-\infty, 0)`. .. math :: \mathrm{log\_softmax}(x)_i = \log \left( \frac{\exp(x_i)}{\sum_j \exp(x_j)} \right) Args: x : input array axis: the axis or axes along which the :code:`log_softmax` should be computed. Either an integer or a tuple of integers. where: Elements to include in the :code:`log_softmax`. Returns: An array. Note: If any input values are ``+inf``, the result will be all ``NaN``: this reflects the fact that ``inf / inf`` is not well-defined in the context of floating-point math. See also: :func:`softmax` zPThe initial argument to log_softmax is deprecated, and no longer has any effect.r3 stacklevel log_softmaxTrBrkeepdimsNrBr)rwarningswarnDeprecationWarningr4r5r#r6maxinfrBr stop_gradientlogsumexp) r rzrBrrHx_maxx_safeshiftedshifted_logsumexpresults rrrs< L   Md$4444  ]A... +a..% '%USWHt L L L%M55syx'H'H& S&u-- -'g gcggEDAAACC & &&  9UFSWH - -- -rc|turtjdtd~tjjrt|||St|||S)aSoftmax function. Computes the function which rescales elements to the range :math:`[0, 1]` such that the elements along :code:`axis` sum to :math:`1`. .. math :: \mathrm{softmax}(x) = \frac{\exp(x_i)}{\sum_j \exp(x_j)} Args: x : input array axis: the axis or axes along which the softmax should be computed. The softmax output summed across these dimensions should sum to :math:`1`. Either an integer or a tuple of integers. where: Elements to include in the :code:`softmax`. Returns: An array. Note: If any input values are ``+inf``, the result will be all ``NaN``: this reflects the fact that ``inf / inf`` is not well-defined in the context of floating-point math. See also: :func:`log_softmax` zLThe initial argument to softmax is deprecated, and no longer has any effect.r3r) rrrrr softmax_custom_jvpvalue_softmax_softmax_deprecated)r rzrBrs rsoftmaxr!sg: L   M`$4444  $/ AtU # ## q$ . ..r)rE)nondiff_argnumsctj||||d}||ntj|||}tj||z }|tj|||dz }|tj||d}|SNTrrr)r#rrBrrr rzrBrrr unnormalizedrs rrrLs '!T$ G G G% 1139UAw#?#?&%((, #',EDQQQ Q&  Yufa ( (F -rc||c\}}}\}}}t||||}|||||z||dz zfS)NTr)rr) rzprimalstangentsr rBrx_dot_r9s r _softmax_jvprZsY'.$1eW}q!q$w''! A!e)UTJJJ K KKrc$tj||||d}||ntj|||}tj|t j|z }|tj|||dz }|tj||d}|Sr)r#rrBrr rrrs rrr`s '!T$ G G G% 1139UAw#?#?&#"3E":"::;;, #',EDQQQ Q&  Yufa ( (F -rh㈵>meanvarianceepsilonctjd|tjd||||tj||d|}|?tjtj||d|tj|z }tj|tj|tj tj||zzS)z]Normalizes an array by subtracting ``mean`` and dividing by :math:`\sqrt{\mathrm{variance}}`. standardizeNT)rrB) r4r5check_arraylike_or_noner#rr8subtractr6r rsqrt)r rzrrrrBs rrrns ]A... $]D(EJJJ \ 8Atd% 8 8 8D  x 1 td%999;>:d;K;KLH aT** + +ci H8M8MPW8W.X.X XXr) num_classesrorzrrroint | AxisNamec tj|d}tj|}t j|} t j||jdz}nr#t$retj d|}||krtd|d|d|dtj |}t j||k|cYSwxYwtj|}tj||f}dg|jz} | ||tj|j| |} t j|| k|S)N9The error arose in jax.nn.one_hot argument `num_classes`.rEz/Expected num_classes to match the size of axis z, but z != rn)r concrete_dim_or_errorr canonicalize_dtyper#r6r canonicalize_axisndim TypeErrorr psum ValueError axis_indexoperatorindex expand_dimsinsertbroadcasted_iotaro) r rrorzrHoutput_pos_axis axis_sizeaxis_idxlhs rhs_shaperhss r_one_hotrsz*ACC+  #E * *% +a..%7,T5:>BBOO 777D!!Ii ;;;);;/8;; < >> jax.nn.one_hot(jnp.array([0, 1, 2]), 3) Array([[1., 0., 0.], [0., 1., 0.], [0., 0., 1.]], dtype=float32) Indices outside the range [0, num_classes) will be encoded as zeros:: >>> jax.nn.one_hot(jnp.array([-1, 3]), 3) Array([[0., 0., 0.], [0., 0., 0.]], dtype=float32) Args: x: A tensor of indices. num_classes: Number of classes in the one-hot dimension. dtype: optional, a float dtype for the returned values (default :obj:`jnp.float_`). axis: the axis or axes along which the function should be computed. rr)r rr)r rrorzs rone_hotrs72*ACC+ ![D 9 9 99rcRtjtj|ddS)anRectified Linear Unit 6 activation function. Computes the element-wise function .. math:: \mathrm{relu6}(x) = \min(\max(x, 0), 6) except under differentiation, we take: .. math:: \nabla \mathrm{relu}(0) = 0 and .. math:: \nabla \mathrm{relu}(6) = 0 Args: x : input array Returns: An array. See also: :func:`relu` r@)r#rdr$r%s rrelu6rs": S[A&& + ++rcjtj|dk|dkz|tj|dS)Nrr(r+s rr.r.s1j!a%AE*As}Q/B/BCCrc,t|dzdz S)zHard Sigmoid activation function. Computes the element-wise function .. math:: \mathrm{hard\_sigmoid}(x) = \frac{\mathrm{relu6}(x + 3)}{6} Args: x : input array Returns: An array. See also: :func:`relu6` g@r)rr%s r hard_sigmoidrs$ q2v rcxtjd|tj|}|t |zS)aMHard SiLU (swish) activation function Computes the element-wise function .. math:: \mathrm{hard\_silu}(x) = x \cdot \mathrm{hard\_sigmoid}(x) Both :func:`hard_silu` and :func:`hard_swish` are aliases for the same function. Args: x : input array Returns: An array. See also: :func:`hard_sigmoid` hard_silu)r4r5r#r6rrGs rrrs7* [!,,, +a..% e$$ $$rcftj|j}tjd|z|S)Ngffffffrn)r#finforr6)ro dtype_maxs r_get_large_negativer s.i") TI%U 3 3 33rctjtj||ftj}tj|tjd|t |}|tjtjddddfS)Nrnr?)r#trilonesbool_rBr6rnewaxis)TSropredmasks r_get_causal_maskrsn #(Aq6333 4 4$ 4S%002Ee2L2L M M$ ck3;111, --rquerykeyrbias Array | Noner is_causalrgfloatctj|jtj}tjd|||}|tj||jz}|||z|j}|B|jtjksJt|j} tj ||| } n|} |r5|j d|j d} } t| | |j}| |z} | tj} tj | d|j} tjd| |}|S)NzBTNH,BSNH->BNTS)preferred_element_typernrbryzBNTS,BSNH->BTNH)r# promote_typesrofloat32einsumrtrsrrrBrrjaxnnr)rrrrrrrg logits_dtypelogitslarge_negative_number padded_logitsrrprobsencodeds r_dot_product_attention_xlarsM"5; <<, :'-9 ; ; ;& CIe6< 0 0 00& tm # #FL 1 1F  : " " " "/ ==IdF,ABBMMM) ;r?CIbMqA Aq&, / /D!D(M &&s{33- &..R. 0 0 7 7 B B% J(% 7 7' .rF)rrrgrimplementation float | NonerLiteral['xla', 'cudnn'] | Nonec \dd }tj|}tj|}tj|}||ntj|}||ntj|}|j\} } } } ||| | | | gd ||| d | | gd |dtj| z n|} |j|jcxkr |jks*nt d|jd|jd|jd|-|jtjkrt d|jd|xdkrt||||||| Sxdkr1|r tj n tj }t|||||| |St||||||| S t d|)aScaled dot product attention function. Computes the attention function on Query, Key, and Value tensors: .. math:: \mathrm{Attention}(Q, K, V)=\mathrm{softmax}(\frac{QK^T}{\sqrt{d_k}})V If we define :code:`logits` as the output of :math:`QK^T` and the :code:`probs` as the output of :math:`softmax`. Throughout this function, we utilize the following uppercase letters to represent the shape of array: B = batch size S = length of the key/value (source) T = length of the query (target) N = number of attention heads H = dimensions of each attention head Args: query: query array; shape :code:`(BTNH)` key: key array; shape :code:`(BSNH)` value: value array; shape :code:`(BSNH)` bias: optional, bias array to be added to logits; shape broadcastable to :code:`(BNTS)`. mask: optional, mask array used to filter out logits. It is a boolean mask where `True` indicates the element should take part in attention. For an additive mask, users should pass it to `bias`. The shape is broadcastable to :code:`(BNTS)`. scale: scale for the logits. If None, the scale will be set to 1 divided by the square root of query's head dimension (i.e. H). is_causal: If true, causal attention will be applied. Note, some implementations like `xla` will generate a mask tensor and apply it to the logits to mask out the non-causal parts of the attention matrix, but other implementations like `cudnn` will avoid computing the non-causal regions, providing speedups. implementation: A string to control which implementation backend to use. Supported strings are `xla`, `cudnn` (cuDNN flash attention). It defaults to `None`, which will automatically select the best available backend. Note, `cudnn` supports only a subset of shapes/dtypes, and an exception will be thrown if its not supported. Returns: An array of the attention output with the same shape as :code:`query`. trr Sequence[int]namestrr!Nonec4|jt|kr)t|dt|d|jt|jD]A}||dkr3|j|||krt|d|d|jBdS)Nz ndim should be z , but got rbz shape should be z : but got )rlenrranger)rrris r_check_has_shapez/dot_product_attention.._check_has_shapeqsvU $NNE NNafNN O OO 16]]OO qRAGAJ%(22DMM5MMAGMMNNNOOrNrrbrr@z4query/key/value should have the same dtype, but got z vs .z$Mask must be boolean dtype, but got xla)rrgcudnn)rg mask_typez#Unsupported implementation option: )rrrrrrr!r) r#r6rrqr7rorrrrCAUSALNO_MASKcudnn_dot_product_attention)rrrrrrgrrr BrNH scale_valrs rrr9sHpOOOO +e  % C# +e  %3;t#4#4$3;t#4#4$y*!Q151aA,00051b!Q-111$)MsRWQZZu) + 1 1 1 1ek 1 1 1 1 G GG),GG8= GGG H HH $* 11 IDJIII J JJ  ' eT49I %.D(//H4Di ( eT4yI ( eT49I M^MM N NNr)r rr!r)r/)r rr0rr!r)r@)r rrXrr!r)r\)r rr]rr!r)T)r rrirjr!r)rb)r rrzr|r!r) r rrzrrBrrrr!r) r rrzrrBrrrr!r)rbNNrN)r rrzrrrrrrrrBrr!r) r rrr|rorrzrr!r)rrrrrrrrrrrrjrgr)rrrrrrrrrrrgrrrjrrr!r)N__doc__ __future__rcollections.abcr functoolsrrnumpyrqtypingrrrr jax.numpyr#rr jax._srcr r r r jax._src.corer(jax._src.cudnn.fused_attention_stablehlorrrjax._src.numpyr4jax._src.typingrrjax._src.ops.specialr _logsumexprrjitr&defjvpsr2r<r>rDrLrPrRswishrTrWrZr_rarerhrlr~rrrrdefjvprrrrfloat_rrrr hard_swishrrrrrrr,s"=<""""""$$$$$$ """"""DDDDDDDD------,,,,,,,,888888{}}    B H HIII *     KKK K2 & & &  & (+++ +>     . +++ +, *GGGG G6>>>> >6CCC C,NNNN N0 BSSSSS@ ),,,-,@   ),,,57*.:F,,,,-,,d24&*6B(/(/(/(/(/V ...*,"!$    /.  LLL*,"!$      ),,,35'++/#'(, YYYY-,Y* "BCCC...DC.2B::::::>,,, ,8 DDEEE &%%% %0 444... """"R"!59`O`O`O`O`O`O`O`Or