Vpf!tddlmZddlmZddlZddlZddlmZddlZ ddl m Z ddl m Z ddl mZee jcZZejZejZejZeZejZejZeeefZejZdd Zd Zee_ee_eeee eeefdffZ!ddZ"ddZ# dd dZ$dS)!) annotations)SequenceN)Union)config)util)pmap_libshapetuple[int, ...]return np.ndarrayct|t|jksJ||jfg}g}t|jD]a\}}||}t|tr$|t dgGt|trW||jksJ|d|j|t|||t|trtj |j }t||\}|r J|||f|fdt|D||j Mt!j|ct%jtj |gt$j} tt+j|D] \} } | | | < | |} t|} d|jD} t3| t3t| krt| | ksJt5d|jDt+jdt9| cfd|jD}t%j| | jz|S) aReturns NumPy-style indices corresponding to a sharding spec. Args: shape: The shape of the logical array being sharded. Returns: An ndarray with the same shape as the logical mesh (as derived form `mesh_mapping`). Each entry is a NumPy-style index selecting the subset of the data array to be placed on a corresponding device. The indices can be ints, slice objects with step=1, or tuples of those. Nz != cBg|]}t|z|dzzS))slice).0i shard_sizes W/var/www/html/nettyfy-visnx/env/lib/python3.11/site-packages/jax/_src/sharding_specs.py z*_sharding_spec_indices..Xs@999 !Z!a%:1EFF999)dtypecFg|]}t|t|jS isinstance ShardedAxisaxisras rrz*_sharding_spec_indices..is)VVV:a;U;UVafVVVrc3NK|] }t|t|jV!dSN)r Replicatedreplicasrs r z)_sharding_spec_indices..ms3[[qAzAZAZ[ [[[[[[rrcg|]E}t|trtnttzFSr)rr"nextlen)rr replica_dim replica_sizes sharded_dims rrz*_sharding_spec_indices..os] & & & *!Z88 1${    m  tK00 0 & & &r) r'sharding enumerater NoShardingappendr UnstackedsizerangeChunkedmathprodchunksdivmodextendrassert_unreachablenpemptyobject_ itertoolsproductreshape mesh_mappingsettuplecountiter broadcast_tor transpose)selfr axis_indicesshard_indices_shapedimr+ axis_size total_chunksragged shard_indicesridxsnum_sharded_dimsharded_dim_permpermr(r)rr*s @@@@r_sharding_spec_indicesrR:s< Us4=)) ) ) )E4=+A ) ) )(*, //((mc8c I(J''(5;;-(((( Hi ( ( ( (- ' ' 'I)J)J8=)J)J ' ' '% **+++  ++++ Hg & &(Yx//l!)\::j&77)\377779999$),$7$7999:::  1111 h'''' (DI&9::;2:NNN-9,l;<<gaM!''(;<<-+,,/VVd&7VVV   3u_'='=#>#> > >   ? 2 2 2 2[[D,=[[[[[-&_Q//6F1G1G+{ & & & & & &$ & & &$ /-9L)L M M Yt__rc(d|jd|jdS)Nz ShardingSpec(z, )r+r?)rFs r_sharding_spec_reprrVus > > >$*; > > >>r. Sequence[int]spec ShardingSpectuple[Index, ...]cPt||jS)aReturns numpy-style indices corresponding to a sharding spec. Each index describes a shard of the array. The order of the indices is the same as the device_buffers of a Array sharded using PmapSharding (i.e. the data is laid out row-major). Args: shape: The shape of the logical array being sharded. spec: Describes how the array is sharded and how the shards are assigned to the logical mesh. Returns: A tuple of length equal to the size of the mesh (inferred as the product of sharded dimension sizes and all replication factors). Each element is an int, a slice object with step=1, or a tuple thereof, to be treated as an index into the full logical array. )rAindicesflat)r rXs rspec_to_indicesr^s!& t||E""' ( ((r sharded_shapemap_axis int | Nonec  t||\}}|rJttgt|zd}|dkrdnt |f}|t d|jd|D d fd }tjj r*tj |j|t|g} n(tj |j|t|} t| tjt# g|t%||jSt|jt |f|z|jzS) azSharding spec for arguments or results of a pmap. Args: nrep: number of local XLA replicas (product of local axis sizes) axis_size: local axis size for outer pmap sharded_aval: the aval of the value inside the outer pmap, an instance of a ShapedArray. map_axis: the axis along which the value is mapped in the outer pmap Returns: A ShardingSpec. rrUrNc3BK|]}t|t VdSr!)rr-)rss rr$z%pmap_sharding_spec..s/[[AjJ777[[[[[[rrMeshDimAssignmentcvt|tr"|jkrt|jdzS|SNrr)rsharded_in_axiss rshift_sharded_axisz.pmap_sharding_spec..shift_sharded_axiss: A{ # #'/(A(A16A:&&& hr)rre)r6rY_UNSHARDED_INSTANCEr'r"sumr+rpmap_no_rank_reductionvaluer tuple_updater2 tuple_insertr/r<chainrmapr?) nreprJr_r`replication_factorrLpspecmaybe_replicaterir+rhs @rpmap_sharding_specrvs &dI66f  !4 5M8J8J J$& ( ( (%,11BB CU8V8V7X/ [[PYQYPYAZ[[[[[O $*:" .(GYK$8$8::hh" .(Ii$8$8::h ?  ' ' (/  %"4 5 577 8 8 88 ~y))+o=@RR T T TTrr*intsharded_dim_sizec|Htjjrtj||d}ntj||}|||}n|J|}t ||||Srg)rrlrmrrn tuple_deleterv)r r*rxr_s rcreate_pmap_sharding_specr{s~ $*<'{A>>mm'{;;m{+  ' ' 'M ,.> ' ) ))r)r r r r )r rWrXrYr rZ)r_rWr`rar rY)rN)r r r*rwrxra)% __future__rcollections.abcrr<r3typingrnumpyr9jax._srcrr jax._src.librrqsafe_map unsafe_mapr-r2r/rjrr"rerYrRrVr\__repr__rwrrAIndexr^rvr{rrrrs<#"""""$$$$$$ !!!!!!t} C       jll"  +z12$ 9999v???. +  c5%c5j 13 6778)))),'T'T'T'TTJK=A)))))))r