VpfcdZddlmZddlmZddlZddlZddlmZmZm Z ddl Z ddl m Z ddlmZddlmZddlmZdd lmZdd lmZmZdd lmZed ZeZeZeZeZeZeejGd de Z ee j!fdCdZ"ee j!fdCdZ#ee j!fdDdZ$ede j!fdEdZ%ede j!fdFd!Z&ede j!d"d#fdGd&Z'e dHdId0Z(dJd1Z)dKd2Z*ed'd(d)e j!fdLd7Z+ed'd(d)e j!fdMd8Z,e,Z-ed'd(d)e j!fdMd9Z.e.Z/ed'd(d)e j!fdMd:Z0ed'd(d)e j!fdMd;Z1ed'd(d)e j!fdMd<Z2e2Z3ed'd(d)e j!fdMd=Z4e4Z5ed>d(e j!fdNdAZ6ed>d(e j!fdNdBZ7dS)Oza Common neural network layer initializers, consistent with definitions used in Keras and Sonnet. ) annotations)SequenceN)AnyLiteralProtocol)lax)random)core)dtypes)Array ArrayLike) set_modulezjax.nn.initializersc4eZdZeejfd d Zd S) InitializerkeyKeyArrayshape core.ShapedtypeDTypeLikeInexactreturnr ctN)NotImplementedErrorrrrs X/var/www/html/nettyfy-visnx/env/lib/python3.11/site-packages/jax/_src/nn/initializers.py__call__zInitializer.__call__2s  Nrrrrrrrr )__name__ __module__ __qualname__ staticmethodjnpfloat_rrrrr/s@*-<rrrrrrrrrr cPtj|tj|S)a An initializer that returns a constant array full of zeros. The ``key`` argument is ignored. >>> import jax, jax.numpy as jnp >>> jax.nn.initializers.zeros(jax.random.key(42), (2, 3), jnp.float32) Array([[0., 0., 0.], [0., 0., 0.]], dtype=float32) )r$zerosr canonicalize_dtypers rr(r(8s! 5&3E:: ; ;;rcPtj|tj|S)aAn initializer that returns a constant array full of ones. The ``key`` argument is ignored. >>> import jax, jax.numpy as jnp >>> jax.nn.initializers.ones(jax.random.key(42), (3, 2), jnp.float32) Array([[1., 1.], [1., 1.], [1., 1.]], dtype=float32) )r$onesr r)rs rr+r+Gs! %2599 : ::rvaluer c|fd fd }|S) aBuilds an initializer that returns arrays full of a constant ``value``. Args: value: the constant value with which to fill the initializer. dtype: optional; the initializer's default dtype. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.constant(-7) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) Array([[-7., -7., -7.], [-7., -7., -7.]], dtype=float32) rrrrrrrr cZtj|}tj||S)Nr)r r)r$full)rrrr,s rinitzconstant..initgs,  %e , ,E 8E5 . . ..rrr&)r,rr1s` rconstantr2Ws2$&+/////// +rg{Gz?scale RealNumericc|fd fd }|S) aYBuilds an initializer that returns real uniformly-distributed random arrays. Args: scale: optional; the upper bound of the random distribution. dtype: optional; the initializer's default dtype. Returns: An initializer that returns arrays whose values are uniformly distributed in the range ``[0, scale)``. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.uniform(10.0) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[7.298188 , 8.691938 , 8.7230015], [2.0818567, 1.8662417, 5.5022564]], dtype=float32) rrrrrrrr ctj|}tj|||t j|zSr)r r)r uniformr$array)rrrr3s rr1zuniform..inits:  %e , ,E >#ue , ,sy/F/F FFrrr&)r3rr1s` rr7r7n9*&+GGGGGGG +rstddevc|fd fd }|S) axBuilds an initializer that returns real normally-distributed random arrays. Args: stddev: optional; the standard deviation of the distribution. dtype: optional; the initializer's default dtype. Returns: An initializer that returns arrays whose values are normally distributed with mean ``0`` and standard deviation ``stddev``. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.normal(5.0) >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 3.0613258 , 5.6129413 , 5.6866574 ], [-4.063663 , -4.4520254 , 0.63115686]], dtype=float32) rrrrrrrr ctj|}tj|||t j|zSr)r r)r normalr$r8)rrrr:s rr1znormal..inits:  %e , ,E =eU + +ci.F.F FFrrr&)r:rr1s` rr=r=r9rg@lowerupperc |fd fd }|S) aBuilds an initializer that returns truncated-normal random arrays. Args: stddev: optional; the standard deviation of the untruncated distribution. Note that this function does not apply the stddev correction as is done in the variancescaling initializers, and users are expected to apply this correction themselves via the stddev arg if they wish to employ it. dtype: optional; the initializer's default dtype. lower: Float representing the lower bound for truncation. Applied before the output is multiplied by the stddev. upper: Float representing the upper bound for truncation. Applied before the output is multiplied by the stddev. Returns: An initializer that returns arrays whose values follow the truncated normal distribution with mean ``0`` and standard deviation ``stddev``, and range :math:`\rm{lower * stddev} < x < \rm{upper * stddev}`. >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.truncated_normal(5.0) >>> initializer(jax.random.PRNGKey(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 2.9047365, 5.2338114, 5.29852 ], [-3.836303 , -4.192359 , 0.6022964]], dtype=float32) rrrrrrrr ctj|}tj|||t j|zSr)r r)r truncated_normalr$r8)rrrr?r:r@s rr1ztruncated_normal..initsK  %e , ,E  " UE5% ) )+.9VU+C+C DDrrr&)r:rr?r@r1s` `` rrCrCsH@&+DDDDDDDDD +rr& Sequence[int]in_axisint | Sequence[int]out_axis batch_axistuple[float, float]c4tdkr tdtdt|tr |}n t jfd|D}t|tr |}n t jfd|D}t|tr |}n t jfd|D}t j|z |z |z }||z}||z} || fS)z Compute effective input and output sizes for a linear or convolutional layer. Axes not in in_axis, out_axis, or batch_axis are assumed to constitute the "receptive field" of a convolution (kernel spatial dimensions). z*Can't compute input and output sizes of a z1-dimensional weights tensor. Must be at least 2D.c g|] }| Sr&r&.0irs r z!_compute_fans..s333aq333rc g|] }| Sr&r&rOs rrRz!_compute_fans..s555q%(555rc g|] }| Sr&r&rOs rrRz!_compute_fans..s999E!H999r)len ValueError isinstanceintmathprod) rrGrIrJin_sizeout_size batch_sizereceptive_field_sizefan_infan_outs ` r _compute_fansrasN ZZ1__ I#e**III J JJ5GnGGi3333733344G#7XHHy5555H55566H C  ;z"JJ9999j999::J5))G3h>K ) )& + +' rctj|\}}tjd|jj}t j|}tj dtj |||z |}dtj ztj ||| |z}|tj d|zzS)zj Sample uniform random values within a disk on the complex plane, with zero mean and unit variance. r?)r splitnpr8realrr to_complex_dtyper$sqrtr7astypepiexp)rrrkey_r key_theta real_dtyperthetas r_complex_uniformrrs\#&&%x5!!&,*  !* - -% hq6>% ;;;<<CCEJJ! cf*v~i CCJJ5QQ Q% SWR%Z  rc@tj|\}}tjd|jj}t j|}dtj tj|dz |z tj ||| |z}tj tj d|z  }dtjztj ||| |z} |tj d| zzS)z Sample random values from a centered normal distribution on the complex plane, whose modulus is truncated to `upper`, and the variance before the truncation is one. rrMrcrd)r rerfr8rgrr rhr$rlr7rjrilogrk) rr@rrrmrnrotrprqs r_complex_truncated_normalrvs\#&&%x5!!&,*  !* - -% CGCI mU33 4 44 uj 1 1 8 8 ? ?@! hA! cf*v~i CCJJ5QQ Q% SWR%Z  rmode;Literal['fan_in'] | Literal['fan_out'] | Literal['fan_avg'] distributionDLiteral['truncated_normal'] | Literal['normal'] | Literal['uniform']c,|fd fd }|S) a Initializer that adapts its scale to the shape of the weights tensor. With ``distribution="truncated_normal"`` or ``distribution="normal"``, samples are drawn from a (truncated) normal distribution with a mean of zero and a standard deviation (after truncation, if applicable) of :math:`\sqrt{\frac{scale}{n}}`, where `n` is: * the number of input units in the weights tensor, if ``mode="fan_in"``, * the number of output units, if ``mode="fan_out"``, or * the average of the numbers of input and output units, if ``mode="fan_avg"``. This initializer can be configured with ``in_axis``, ``out_axis``, and ``batch_axis`` to work with general convolutional or dense layers; axes that are not in any of those arguments are assumed to be the "receptive field" (convolution kernel spatial axes). With ``distribution="truncated_normal"``, the absolute values of the samples are truncated at 2 standard deviations before scaling. With ``distribution="uniform"``, samples are drawn from: * a uniform interval, if `dtype` is real, or * a uniform disk, if `dtype` is complex, with a mean of zero and a standard deviation of :math:`\sqrt{\frac{scale}{n}}` where `n` is defined above. Args: scale: scaling factor (positive float). mode: one of ``"fan_in"``, ``"fan_out"``, and ``"fan_avg"``. distribution: random distribution to use. One of ``"truncated_normal"``, ``"normal"`` and ``"uniform"``. in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. rrrrrrrr ctj|}tj|}t | \}} dkr|}n* dkr|}n! dkr ||zdz }nt d t j |z |} dkrt j|t j rEt j |t jd|z }tj |d d|||zSt j |t jd |z }t|d|||zS d kr+tj|||t j |zS d krtt j|t j r/tj|||d t j d|zzSt!|||t j |zSt d )Nr_r`fan_avgrcz/invalid mode for variance scaling initializer: r/rCg۶%?rDgVr?r=r7rEz7invalid distribution for variance scaling initializer: )r canonicalize_shaper r)rarVr$r8 issubdtypefloatingrir rCrvr=r7rr)rrrr_r` denominatorvariancer:rJryrGrwrIr3s rr1zvariance_scaling..init=s  #E * *E  %e , ,E#E7HjIIOFG xv   'KK   6G+;q*@KK @$@@ B BBy,E:::H))) s| , ,H(##ci0BE&J&JJ&sB5%@@6II(##ci0BE&J&JJ(a>>GG  ! ! ]3u - -0B0B BB  " " s| , ,H~c5%44sxH 7M7MMMUE22SXh5G5GGG _Q]__ ` ``rrr&)r3rwryrGrIrJrr1s`````` rvariance_scalingrs^n&+aaaaaaaaaaaaB +rc .tddd||||S)aBuilds a Glorot uniform initializer (aka Xavier uniform initializer). A `Glorot uniform initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 1.0``, ``mode="fan_avg"``, and ``distribution="uniform"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.glorot_uniform() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.50350785, 0.8088631 , 0.81566876], [-0.6393332 , -0.6865721 , 0.11003882]], dtype=float32) .. _Glorot uniform initializer: http://proceedings.mlr.press/v9/glorot10a.html ?r}r7rGrIrJrrrs rglorot_uniformr`s/B #y)W#+ % Q Q QQrc .tddd||||S)aBuilds a Glorot normal initializer (aka Xavier normal initializer). A `Glorot normal initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 1.0``, ``mode="fan_avg"``, and ``distribution="truncated_normal"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.glorot_normal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.41770416, 0.75262755, 0.7619329 ], [-0.5516644 , -0.6028657 , 0.08661086]], dtype=float32) .. _Glorot normal initializer: http://proceedings.mlr.press/v9/glorot10a.html rr}rCrrrs r glorot_normalrs0B #y*>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.lecun_uniform() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.56293887, 0.90433645, 0.9119454 ], [-0.71479625, -0.7676109 , 0.12302713]], dtype=float32) .. _Lecun uniform initializer: https://arxiv.org/abs/1706.02515 rr_r7rrrs r lecun_uniformr/B #xG#+ % Q Q QQrc .tddd||||S)aBuilds a Lecun normal initializer. A `Lecun normal initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 1.0``, ``mode="fan_in"``, and ``distribution="truncated_normal"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.lecun_normal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.46700746, 0.8414632 , 0.8518669 ], [-0.61677957, -0.67402434, 0.09683388]], dtype=float32) .. _Lecun normal initializer: https://arxiv.org/abs/1706.02515 rr_rCrrrs r lecun_normalr0B #x);W#+ % Q Q QQrc .tddd||||S)aBuilds a He uniform initializer (aka Kaiming uniform initializer). A `He uniform initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 2.0``, ``mode="fan_in"``, and ``distribution="uniform"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.he_uniform() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.79611576, 1.2789248 , 1.2896855 ], [-1.0108745 , -1.0855657 , 0.17398663]], dtype=float32) .. _He uniform initializer: https://arxiv.org/abs/1502.01852 r>r_r7rrrs r he_uniformrrrc .tddd||||S)aBuilds a He normal initializer (aka Kaiming normal initializer). A `He normal initializer`_ is a specialization of :func:`jax.nn.initializers.variance_scaling` where ``scale = 2.0``, ``mode="fan_in"``, and ``distribution="truncated_normal"``. Args: in_axis: axis or sequence of axes of the input dimension in the weights array. out_axis: axis or sequence of axes of the output dimension in the weights array. batch_axis: axis or sequence of axes in the weight array that should be ignored. dtype: the dtype of the weights. Returns: An initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.he_normal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 0.6604483 , 1.1900088 , 1.2047218 ], [-0.87225807, -0.95321447, 0.1369438 ]], dtype=float32) .. _He normal initializer: https://arxiv.org/abs/1502.01852 r>r_rCrrrs r he_normalrrrr column_axisrXc|fd fd }|S) a Builds an initializer that returns uniformly distributed orthogonal matrices. If the shape is not square, the matrices will have orthonormal rows or columns depending on which side is smaller. Args: scale: the upper bound of the uniform distribution. column_axis: the axis that contains the columns that should be orthogonal. dtype: the default dtype of the weights. Returns: An orthogonal initializer. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.orthogonal() >>> initializer(jax.random.key(42), (2, 3), jnp.float32) # doctest: +SKIP Array([[ 3.9026976e-01, 7.2495741e-01, -5.6756169e-01], [ 8.8047469e-01, -4.7409311e-01, -1.3157725e-04]], dtype=float32) rrrrrrrr c tj|}t|dkrtdt j|| z| }}||kr||fn||f}t j|||}tj |\}}tj tj tj||j} || z}||kr|j}tj|t%t'j| | fz}tj|d }tj ||zS)Nrcz3orthogonal initializer requires at least a 2D shape)rankrE)r r)rUrVrYrZr r=r$linalgqrrbroadcast_to_ranksigndiagndimTreshapetuplerfdeletemoveaxisr8) rrrn_rowsn_cols matrix_shapeAQR diag_signrr3s rr1zorthogonal..initZs8  %e , ,E 5zzA~~ L M MMYu%%{);;U;=OFF'-FF##VVSSTTA QK((A 9UE " "Q &&rrr&r3rrr1s`` r orthogonalr@s88&+'''''''' +rc|fd fd }|S) ae Builds an initializer for delta orthogonal kernels. Args: scale: the upper bound of the uniform distribution. column_axis: the axis that contains the columns that should be orthogonal. dtype: the default dtype of the weights. Returns: A `delta orthogonal initializer`_. The shape passed to the initializer must be 3D, 4D, or 5D. Examples: >>> import jax, jax.numpy as jnp >>> initializer = jax.nn.initializers.delta_orthogonal() >>> initializer(jax.random.key(42), (3, 3, 3), jnp.float32) # doctest: +SKIP Array([[[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ]], [[ 0.27858758, -0.7949833 , -0.53887904], [ 0.9120717 , 0.04322892, 0.40774566], [-0.30085585, -0.6050892 , 0.73712474]], [[ 0. , 0. , 0. ], [ 0. , 0. , 0. ], [ 0. , 0. , 0. ]]], dtype=float32) .. _delta orthogonal initializer: https://arxiv.org/abs/1806.05393 rrrrrrrr ctj|}t|dvrtd|d|dkrtdt |}|||dd}t j||}t|dkr0|d }|j|d z d zd f|St|d kr<|dd \}}|j|d z d z|d z d zd f|S|dd\}}} |j|d z d z|d z d z| d z d zd f|S)N)r~z;Delta orthogonal initializer requires a 3D, 4D or 5D shape.rErDz/`fan_in` must be less or equal than `fan_out`. )r3rrr/r~rrMrc.r) r r)rUrVrr$r(atset) rrr ortho_init ortho_matrixWkk1k2k3rr3s rr1zdelta_orthogonal..inits  %e , ,E 5zz""   ! !! Ry59 H I II%[NNNJ:c5:..L %u%%%A 5zzQ (a T1Q3(C- $ $\ 2 22 UqRaRyfb" T2a4!)bdQY+ , 0 0 > >>!9jb"b T2a4!)bdQYA 36 7 ; ;L I IIrrr&rs`` rdelta_orthogonalrlsAN&+JJJJJJJJ* +rr)r,r rrrr)r3r4rrrr)r:r4rrrr) r:r4rrr?r4r@r4rr)rDrEr&) rrFrGrHrIrHrJrHrrK)rrrrFrrrr ) rrr@r rrFrrrr )r3r4rwrxryrzrGrHrIrHrJrFrrrr) rGrHrIrHrJrFrrrr)r3r4rrXrrrr)8__doc__ __future__rcollections.abcrrYtypingrrrnumpyrf jax.numpyr$jaxrr jax._srcr r jax._src.typingr r jax._src.utilrexportrDTypeLikeFloatDTypeLikeComplexrr4runtime_checkablerr%r(r+r2r7r=rCrarrrvrrxavier_uniformr xavier_normalrrrkaiming_uniformrkaiming_normalrrr&rrrs #"""""$$$$$$ )))))))))),,,,,,,,$$$$$$ ) * *  (%(J < < < < <$': ; ; ; ; ;'*z,!%&)j2!%%(Z2+//2z*.*-#####J132446@ ! ! ! !!!!!" "$"$ JUUUUUn2435/1-0Z!Q!Q!Q!Q!QF 1324.0,/J!Q!Q!Q!Q!QF 1324.0,/J!Q!Q!Q!Q!QF0213-/+.:!Q!Q!Q!Q!QF.0/1+-),!Q!Q!Q!Q!QF-/.0*,(+ !Q!Q!Q!Q!QF$'"$),)))))VJ9999999r