Vpf*ddlmZddlmZddlmZmZddlZddlZddl Z ddl Z ddl m Z mZmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddl m!Z!m"Z"m#Z#ddl$m%Z&ddl$m'Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3m4Z4ddl5m6Z6m7Z7m8Z8m9Z9m:Z:ddl;ZdfZ?e&j@Z@e=eAdfZBe ZCdSd ZDGd!d"ZEd#ZFe:d$d%&d'ZGe:d$d%&d(ZHdTd1ZIGd2dejJZKese8e&jKeKZKdUd6ZLeMeKd7deMeKd8d9dVd=ZN dWdXdBZOdYdDZPejQejReK<e jSd-ejReK<ejTejUeK<e jSd-ejVeK<ejJWeKdEZXejYeKeXdZdJZZdKZ[dLZ\e:d$d%&dMZ]dNZ^e^ej_eK<dOZ`e`ejaejb<e`ejaejc<dPZdedejeejb<edejeejc<dQZfefej_ejg<dRZhehejaeji<dS)[) annotations) defaultdict)CallableSequenceN)Any TYPE_CHECKINGcast)abstract_arrays)api)api_util) basearray)config)core)dispatch)dtypes)errors)profiler) xla_bridge)mlir)pxla)xla) AutoLayoutDeviceLocalLayoutLayout) xla_client) xla_extension)Sharding) PmapShardingSingleDeviceShardingdevice_replica_id_map hashed_indexnum_addressable_indiceslocal_to_global_shape) ArrayLike DLDeviceType)safe_zipunzip3 use_cpp_classuse_cpp_methodcache.a ArrayImplreturnDevicecT|jj}t|dksJ|dS)Nr)sharding_internal_device_listlen)r+devicess N/var/www/html/nettyfy-visnx/env/lib/python3.11/site-packages/jax/_src/array.py _get_devicer68s- J ,' W     ceZdZdZ ddd Zd ZejddZejddZ e dZ e dZ dS)SharduA single data shard of an Array. Attributes: device : Which device this shard resides on. index : The index into the global array of this shard. replica_id : Integer id indicating which replica of the global array this shard is part of. Always 0 for fully sharded data (i.e. when there’s only 1 replica). data : The data of this shard. None if ``device`` is non-local. Ndevicer.r1r global_shapeShapedataNone | ArrayImpl | PRNGKeyArrayc>||_||_||_||_dSN)_device _sharding _global_shape_data)selfr:r1r;r=s r5__init__zShard.__init__Js#DLDN%DDJJJr7c  d|jd|jd|jd|jd S#t$rd|jd|jdcYSwxYw)Nz Shard(device=z, index=z , replica_id=z, data=))r:index replica_idr= ValueErrorrEs r5__repr__zShard.__repr__Qs@AdkAATZAA OAA48IAAAB @@@ ?T[ ? ?49 ? ? ????@s"%AAr-Indexc |jj}n'#t$rtd|jddwxYw||j|j}|J|S)Nz(Cannot calculate indices from sharding: z<. Please create a device to index mapping for your sharding.)rBdevices_indices_mapAttributeErrorrKrCr:)rEdevice_indices_map_fnrIs r5rIz Shard.indexXs?"n@ ??? 4.444 5 5:>?? " !$"4 5 5dk BE     Ls $3intcLt|j|j|jSr@)r rBrCr:rLs r5rJzShard.replica_idds 1C D DT[ QQr7c|jSr@)rArLs r5r:z Shard.devicehs <r7c|jSr@)rDrLs r5r=z Shard.datals :r7r@)r:r.r1rr;r<r=r>)r-rN)r-rS) __name__ __module__ __qualname____doc__rFrM functoolscached_propertyrIrJpropertyr:r=r7r5r9r9>s  8<@@@      RRRR  8  8r7r9c||}||tj|}|jjdi||_|S)z=Method to reconstruct a device array from a serialized state.r^) __setstate__r device_putavalupdate)funargs arr_state aval_statenp_value jnp_values r5_reconstruct_arrayrjqsR S$Z(  """nX&&)(9>(66:66). r7F)max_sizetrace_context_in_keyc||}t}g}t|D]D\}}t |}||vr,|||||fE|Sr@)addressable_devices_indices_mapset enumeratevaluesr!addappend)sshapemap_seen_h_indicesl array_indexrIh_indexs r5_cached_index_calcr|zs * *5 1 1$55.!%dkkmm44%%k55!!Gn$$!!!hh U#$$$ (r7c8tjdkrdStd||D}td||D}||kS)Nr0Fc,h|]}t|Sr^r!.0vs r5 z3_process_has_full_value_in_mcjax..sFFF1|AFFFr7c,h|]}t|Sr^rrs r5rz3_process_has_full_value_in_mcjax..sRRR1|ARRRr7)r process_countr3rPrrro)rurvnum_unique_indicesnum_addressable_unique_indicess r5 _process_has_full_value_in_mcjaxrs1$$ 5FF 5 5e < < C C E EFFFHH#&RR A A% H H O O Q QRRR$T$T = ==r7arrays Sequence[ArrayImpl | np.ndarray]r1rrbcore.ShapedArrayexpected_shaper<c |j}|D]g}|j|krtd|jd|d||j|kr2td|d|jd|d|hdS) z:Validates that per-device arrays are valid and consistent.z8Input buffers to `Array` must have matching dtypes. Got z , expected z for buffer: zExpected shard shape z- doesn't match the single device array shape z. Shape of Array is z with sharding N)dtyperKrv str_short)rr1rbrexpected_dtypedbs r5/_validate_shape_and_dtype_for_per_device_arraysrs:.   b x>!!  H H H&4 H HCE H H   x>!!  9. 9 9 " 9 9 ^^   9 9.6 9 9  "  r7ceZdZUded<ded<ded<ded<ded <d ed <e dVdWdZdZedXdZedZ edZ edZ edZ edZ edZdZdZdZdZd Zd!Zd"Zd#Zd$ZdYd&Zd'Zd(Zd)Zd*ZedZd+Zd,ZedZd-Zd[d/Z d.d.d.d.d0d\d9Z!d]d;Z"d<Z#ed=Z$eed>Z%ed?Z&d^dAZ'edBZ(edCZ)d_dFZ*e+j,d`dHZ-edIZ.ed`dJZ/edKZ0edLZ1dMZ2edNZ3edOZ4edPZ5e6j7dQZ8ee+j9e6j7dRSdadUZ:d.S)br,rrbrrBzlist[ArrayImpl]_arraysbool _committed _skip_checksznp.ndarray | None _npy_valueFr1rSequence[ArrayImpl] committedc||_||_d|D|_||_d|_|rt jjr|dSdS)Nc(g|]}|jdS)r)rrr+s r5 z&ArrayImpl.__init__..s111QAIaL111r7) rbrBrrrr enable_checksvalue_check_and_rearrange)rErbr1rrrs r5rFzArrayImpl.__init__sn DIDN11&111DLDODO "6/5" !!!!!""r7cz d|jD |jj}t|jt|kr4t dt|dt|jt }d|D}||z }|r1||z }||z }d}|r |d|dz }|r |d|d z }t |t|j|j|j|j |j  |jj } fd |D|_dS) Nc8i|]}t|j|Sr^r6id)rrs r5 z2ArrayImpl._check_and_rearrange..s#IIIb;r??-rIIIr7z Expected zW per-device arrays (this is how many devices are addressable by the sharding), but got ch|] }|j Sr^r)rds r5rz1ArrayImpl._check_and_rearrange..s<<.s!PPPv' 2PPPr7) rr1addressable_devicesr3rKrpkeysrrb shard_shaperv_addressable_device_assignment) rEaddressable_devarray_device_idsaddressable_device_idsdiffdev_in_sharding_not_in_arraysdev_in_arrays_not_in_shardingerr_msgaddressable_dars @r5rzArrayImpl._check_and_rearrangesIIDLIIIm7O 4<C0000  %c/** % %T\"" % % & && .335566<r7c|t|jtrt |jjdS|jSNr)_check_if_deleted isinstancer1rlist device_setrLs r5r:zArrayImpl.devicesI$-!566/ $-* + +A .. =r7c|jjSr@)rb weak_typerLs r5rzArrayImpl.weak_types 9 r7c*t|jSr@)str_valuerLs r5__str__zArrayImpl.__str__ s t{  r7cb |jdS#t$r}td|d}~wwxYw)Nrzlen() of unsized object)rv IndexError TypeError)rEerrs r5__len__zArrayImpl.__len__sC: Z] ::: / 0 0c9:s  .).cRtj|t|jSr@)rcheck_bool_conversionrrrLs r5__bool__zArrayImpl.__bool__s#t$$$   r7c\tj||jSr@)rcheck_scalar_conversionr __float__rLs r5rzArrayImpl.__float__s' &&& ; " ""r7c\tj||jSr@)rrr__int__rLs r5rzArrayImpl.__int__s' &&& ;    r7c\tj||jSr@)rrr __complex__rLs r5rzArrayImpl.__complex__!s' &&& ; " " $ $$r7cRtj|t|jSr@)rcheck_integer_conversionhexrrLs r5__hex__zArrayImpl.__hex__%#!$''' t{  r7cRtj|t|jSr@)rroctrrLs r5__oct__zArrayImpl.__oct__)rr7c\tj|tj|jSr@)rroprIrrLs r5 __index__zArrayImpl.__index__-s%!$''' 8DK  r7Cc6|j|Sr@)rtobytes)rEorders r5rzArrayImpl.tobytes1s ;  u % %%r7c4|jSr@)rtolistrLs r5rzArrayImpl.tolist4s ;    r7cx|jdkrt|jd|St|j|S)Nrr^)rformatr)rE format_specs r5 __format__zArrayImpl.__format__7s5 yA~~ DKO[ 1 11 DK - --r7cddlm}ddlm}|t |jtrtj j rjt |tr|n|f}td|Dtdft|jt|z zz}nzt |ts-|ftdft|jdz zz}n8|tdft|jt|z zz}t|j|j} ||}n#t$$rd}YnwxYw||j|}t)|jt-t/||gdd} tj j r=td t1|D} || | S| S|||S) Nr)lax) lax_numpyc3lK|]/}t|trt||dzdn|V0dS)r0N)rrSslice)ris r5 z(ArrayImpl.__getitem__..GsW  CDZ3%7%7 >E!QUD ! ! !Q      r7r0FTrrc3JK|]\}}t|t|VdSr@)rrS)rrxs r5rz(ArrayImpl.__getitem__..^s5KKTQ 1c8J8JKqKKKKKKr7) dimensions) jax._src.laxrjax._src.numpyrrrr1rrpmap_no_rank_reductionrtuplerr3rvrPrrrIrKrr,rbrr6rqsqueeze_rewriting_take) rEidxrrcidx padded_cidxindicesarr_idxr+outdimss r5 __getitem__zArrayImpl.__getitem__>sB      (((((($-..  & , L e,,8ss3&  HL     4[[Nc$*ooD 9 :; #u%% L%++3tz??Q3F!GG++uT{{nDJ#c((0JKK+dm77 CCJJLLMMg-- ,,    L ! F(Q881#  ( . KKYt__KKKKK$ST22 2*  $ $T3 / //s0F FFcjdkrtdjs jsJt jjsjrddDStjtr&fdtj dDSfdtj dDS)Nrziteration over a 0-d arrayc3HK|]}|D]}|VdSr@)_unstack)rchunksls r5rz%ArrayImpl.__iter__..ks9QQu@P@PQQ"QQQQQQQr7dc3(K|] }|V dSr@r^rrrEs r5rz%ArrayImpl.__iter__..ms'66AQ666666r7c3VK|]#}tjj|V$dSr@)r rarrs r5rz%ArrayImpl.__iter__..qs3MM1t{1~..MMMMMMr7) rris_fully_replicatedis_fully_addressableris_single_device_shardingr1 _chunk_iterrrrangervrLs`r5__iter__zArrayImpl.__iter__es yA~~ 2 3 33  %B)BBBB  +DM : :Nd>VNQQ 0 0 5 5QQQQ dm\ 2 2N6666tz!}!5!56666NMMMdjm8L8LMMMMr7c|jjSr@)r1rrLs r5rzArrayImpl.is_fully_replicatedss = ,,r7cd}|j|jjrd|jjd}nd|jjd}|js|jrt jd}|jdkr d|j }nt j |j |dd | }t|| d z d z}d }|t|zd z|krd t|z}||d||S||j d |S)NzArray(zdtype=z, weak_type=True)rH linewidthrz [], shape=,z, )prefixsuffix separatormax_line_width r0 )rbrrnamerrnpget_printoptionsrrv array2stringrr3rfind)rEr dtype_str line_widthru last_line_lenseps r5rMzArrayImpl.__repr__ws5 F y!4=4:?===ii-4:?---i  2D$< 2&((5j a % % % ODKs&*: G G G!ffqwwt}},q0m c Y '! +j 8 8CKK , , ,C , , ,, 1 1 1i 1 11r7c|jjS)aIs this Array fully addressable? A jax.Array is fully addressable if the current process can address all of the devices named in the :class:`Sharding`. ``is_fully_addressable`` is equivalent to "is_local" in multi-process JAX. Note that fully replicated is not equal to fully addressable i.e. a jax.Array which is fully replicated can span across multiple hosts and is not fully addressable. )r1rrLs r5rzArrayImpl.is_fully_addressables = --r7NcD|ind|i}tj|jfd|i|S)Ncopyr)r%asarrayr)rErcontextr/kwdss r5 __array__zArrayImpl.__array__s222FD>D :dk 7 7 7$ 7 77r7)stream max_version dl_devicer/r4int | Any | Noner5tuple[int, int] | Noner6tuple[DLDeviceType, int] | Noner/ bool | Nonecddlm}|jj}t |dkr t dt |d|\}|||||||S)Nr) to_dlpackr0znto_dlpack can only pack a dlpack tensor from an array on a singular device, but an array with a Sharding over z devices was provided.)r4r5 src_devicer6r/)jax._src.dlpackr<r1rr3 BufferError)rEr4r5r6r/r<rr:s r5 __dlpack__zArrayImpl.__dlpack__s*))))))J :  58__      GF 9T&!, &(     r7tuple[enum.Enum, int]ct|jdkrtdddlm}|dkr |jdfS|dkrlt|jj }d|vr|j }nd|vr|j }ntd |t|j }|td ||fStd |) Nr0z/__dlpack__ only supported for unsharded arrays.r)r%cpugpucudarocmz%Unknown GPU platform for __dlpack__: z-Couldn't get local_hardware_id for __dlpack__z@__dlpack__ device only supported for CPU and GPU, got platform: ) r3rr?r>r%platformkDLCPUr6clientplatform_versionkDLCUDAkDLROCMlocal_hardware_id)rEr%rJdl_device_typerMs r5__dlpack_device__zArrayImpl.__dlpack_device__s2 4<A I J JJ,,,,,, }}%  ! ## E ! !$T**1B # # #%- % % %%-/,//00 0&d++=  "IJJJ . ..   ]]__    r7cr|j\}}}d|jji}t||||ffS)Nr)r __reduce__rbrrj)rErdrerfrgs r5rQzArrayImpl.__reduce__s?;1133Cyty23J dIz B CCr7ct|jdkrtd|jdS)Nr0z?unsafe_buffer_pointer() is supported only for unsharded arrays.r)r3rrKunsafe_buffer_pointerrLs r5rSzArrayImpl.unsafe_buffer_pointersF 4<A " # ## <? 0 0 2 22r7ctt|jdkrtd|jdjS)Nr0zB__cuda_array_interface__() is supported only for unsharded arrays.r)r3rrK__cuda_array_interface__rLs r5rUz"ArrayImpl.__cuda_array_interface__s? 4<A + , ,, <? 33r7c||jd}|}|t|jjzS)z>Returns the total global on-device size of the array in bytes.r)ron_device_size_in_bytesr3r1r)rEarrper_shard_sizes r5rWz!ArrayImpl.on_device_size_in_bytess8 ,q/C0022N C 899 99r7 set[Device]cB||jjSr@)rr1rrLs r5r4zArrayImpl.devicess = ##r7c td)NzBarr.device_buffer has been deprecated. Use arr.addressable_data(0)rQrLs r5 device_bufferzArrayImpl.device_buffers J L LLr7c td)NzTarr.device_buffers has been deprecated. Use [x.data for x in arr.addressable_shards]r]rLs r5device_bufferszArrayImpl.device_bufferss \ ^ ^^r7rIrScz||jr|S|j|Sr@)rr_fully_replicated_shardr)rErIs r5addressable_datazArrayImpl.addressable_datas> ,  ) ) + ++ < r7Sequence[Shard]c |g}|jD]>}|tt ||j|j|?|Sr@)rrrtr9r6r1rv)rErr+s r5addressable_shardszArrayImpl.addressable_shardss^ C \FF jj{1~~t}dj!DDEEEE Jr7cz|rtd|jS ttj|j|jS#t j$rT}|j^}}t|tur/| drtd|jcYd}~Sd}~wwxYw)N UNIMPLEMENTED) is_deletedrr1rfrom_pjrt_layout _pjrt_layoutxeXlaRuntimeErrorretyper startswith)rEemsg_s r5layoutzArrayImpl.layout s ) D$- ( (( %6t7HIIM### gcA cc  cnn_== dDM******** s$+AB:&AB5.B:4B55B:c B||jr|jSg}d|jD}|jjD]\}||jd||j}nd}|t||j|j |]|S)zReturns list of all `Shard`s of the Array across all devices. The result includes shards that are not addressable by the current process. If a `Shard` is not addressable, then its `data` will be `None`. c8i|]}t|j|Sr^rrs r5rz+ArrayImpl.global_shards..%s#FFFA;q>>,aFFFr7N) rrrfrr1rgetrrtr9rv)rErrglobal_darrays r5 global_shardszArrayImpl.global_shardss   %  $$ CFFFFFM,DD  d 3 3 ?#HK0 jjx EBBCCCC Jr7cp|jdS|jD]}|d|_d|_dSr@)rdeleter)rEbufs r5r{zArrayImpl.delete.sA | f| jjllllDLDOOOr7cP|jdStd|jDS)NTc3>K|]}|VdSr@)rirr|s r5rz'ArrayImpl.is_deleted..>s,88Cs~~888888r7)ranyrLs r5rizArrayImpl.is_deleted7s0 | T 884<888 8 88r7c|r*td|jddS)Nz"Array has been deleted with shape=.)ri RuntimeErrorrbrrLs r5rzArrayImpl._check_if_deleted@sR I  Gty/B/B/D/D G G G I IIIIr7cj||jD]}||Sr@)rrblock_until_ready)rErs r5rzArrayImpl.block_until_readyEs?l Kr7c@tj|jdSr)r%r0rrLs r5 _single_device_array_to_np_arrayz*ArrayImpl._single_device_array_to_np_arrayLs :dl1o & &&r7cD|jddSr)rcopy_to_host_asyncrLs r5'_copy_single_device_array_to_host_asyncz1ArrayImpl._copy_single_device_array_to_host_asyncPs LO&&(((((r7c||j\|jr|dSt |j|jD]&\}}|j|%dSdSr@)rrrrr|r1rvr)rErrrs r5rzArrayImpl.copy_to_host_asyncTs  ! 44666$T]DJ??BB$!Q Q??AAAA BBr7znp.asarray(jax.Array))r$ np.ndarrayc||j8|jrI||_d|jj_t tj|jS|j s)t|j |j stdt|j |j D]$\}}|j|%tj|j |j}t|j |j D]'\}}|j|||<(||_d|jj_|jS)NFa%Fetching value for `jax.Array` that spans non-addressable (non process local) devices is not possible. You can use `jax.experimental.multihost_utils.process_allgather` to print the global array or use `.addressable_shards` method of jax.Array to inspect the addressable (process local) shards.)rrrrflags writeabler r%ndarrayrrr1rvrr|rremptyr)rErrr npy_valueinds r5rzArrayImpl._value^sH   !1??AA*/'BJ000' .t}djII  ?   %T]DJ??BB$!Q Q??AAAA(4:tz22i&t}djAALL&!SaIIKK #!do(-do% ?r7)F) rbrr1rrrrrrr)r-r<)r)r-r)NNN)r4r7r5r8r6r9r/r:)r-rA)r-rZ)rIrSr-r,)r-rd)r-r);rWrXrY__annotations__r)rFrr]rvrrrr1r:rrrrrrrrrrrrrr rrrMrr3r@rOrQrSrUrWr4r^r`rcr[r\rfrsryr{rirrrrrannotate_functionrpartialrr^r7r5r,r,sY>6;"""""$$Q$Q$QL  8  8  8 !! 8!  8  8   8::: ###!!!%%%!!!&&&&   ...%0%0%0N N N N --- 8-222,  . . . 8 .8888 6:7;>B%)      (>DDD >333  >44 84 >::: $$$$ LL 8L ^^ 8^      8   8(>>999III > >'''>))) BBB 9X/6MNNNON 8r7slcrNrvcPtdt||DS)Nc3lK|]/\}}t|t|jp||jpdz V0dS)rN)rrstopstart)rrudims r5rz(_get_shape_from_index..s[ !S Au  v}A&r7)rr&)rrvs r5_get_shape_from_indexrs: S%((  r7__hash____array_priority__rSharding | Layout data_callback#Callable[[Index | None], ArrayLike]c t|tr|jnd t trt d|t|tr|jn|}t|t s"t d|dt|dfd |jrTt|j j } tdft|zgt|z}nE|| t } fd |D}|d }|j}||}t%j||}t)|||| t|t*rg|jr`|jrY|jrR|jjt1|kr5|jjt5jt |d||kr|St9j|jt8jr8t?j ||} |jj!"||| d S 6 fd|D}t?j ||} t+||| d St|t*r:t|#dkrt?j ||}t5j$||||S)aReturns a ``jax.Array`` via data fetched from ``data_callback``. ``data_callback`` is used to fetch the data for each addressable shard of the returned ``jax.Array``. This function must return concrete arrays, meaning that ``make_array_from_callback`` has limited compatibility with JAX transformations like :func:`jit` or :func:`vmap`. Args: shape : Shape of the ``jax.Array``. sharding: A ``Sharding`` instance which describes how the ``jax.Array`` is laid out across devices. data_callback : Callback that takes indices into the global array value as input and returns the corresponding data of the global array value. The data can be returned as any array-like object, e.g. a ``numpy.ndarray``. Returns: A ``jax.Array`` via data fetched from ``data_callback``. Examples: >>> import math >>> from jax.sharding import Mesh >>> from jax.sharding import PartitionSpec as P >>> import numpy as np ... >>> input_shape = (8, 8) >>> global_input_data = np.arange(math.prod(input_shape)).reshape(input_shape) >>> global_mesh = Mesh(np.array(jax.devices()).reshape(2, 4), ('x', 'y')) >>> inp_sharding = jax.sharding.NamedSharding(global_mesh, P('x', 'y')) ... >>> def cb(index): ... return global_input_data[index] ... >>> arr = jax.make_array_from_callback(input_shape, inp_sharding, cb) >>> arr.addressable_data(0).shape (4, 2) Nz{`DeviceLocalLayout.AUTO` cannot be used in place of a device-local layout when calling `jax.make_array_from_callback`. Got z6sharding should be an instance of `jax.sharding`. Got z of type rI Index | Noner-ArrayImpl | np.ndarrayc|J|}t|tjrtjdt j|S)NzFjax.make_array_from_callback cannot be called within a traced context.)rrTracerrUnexpectedTracerErrorrcanonicalize_dtype)rIrrs r5get_dataz*make_array_from_callback..get_datasa     eA!T[!!  (     !! $ $$r7c2g|]}|Sr^r^)rr:device_to_index_maprs r5rz,make_array_from_callback..s628$V,--r7r)rrbr1TrcJg|]}tt| Sr^)rr)rrdlls r5rz,make_array_from_callback..s,EEEvc/2233EEEr7r0)rIrr-r)%rrdevice_local_layoutrrr1rrnrrr2addressable_device_listrr3rorrrr ShapedArrayrr,r_device_assignmentrrsr_maybe_get_default_layoutr issubdtypeextendedr ra_rulesmake_sharded_arrayr4batched_device_put) rvr1rr4per_device_values first_valuerrrbrrrrs ` @@@r5make_array_from_callbackrsyT)38V(D(DN$$$#Z  P  ODL O O P PP#-Xv">"> LX  H( Hh ' '#  " " "h " " # ## % % % % % %! 81IJJG!5;;.3u::"=>>?#g,,N"BB5II&++--..Gtt|D]'}tt!||<(t#jd fdd fd } t'||| S)a/Creates distributed tensor using the data available in process. This function is a common special case of `make_array_from_callback`. It assumes that the data is available in the process and takes care of the index wrangling. The most common case is when the sharding is sharded across the batch dimension and each host just loads its corresponding sub-batch. This function supports more general cases as well, such as mixed multi-host and multi-axis replication and sharding but you would need to compute the size and the contents of process-local data correctly to satisfy the sharding constraints. In particular, if any two hosts are replicas, host_local_data should be identical as well. The global_shape is optional. If not provided it will be be inferred from the local_data and sharding, under the assumption that each host represents only their own data for uniform sharding. If sharding is non-uniform, (see note below) an exception will be raised. Setting global_shape explicitly allows for finer grain control and works with non-uniform shardings. Each dimension of global_shape must either match host_local_data, or match the inferred global shape of the sharding (in which case it is equivalent to setting it to None, but is more explicit). For example if dimension `i` is fully sharded then this size would be `per_device_shape[i] * jax.local_device_count()`. Each device will be mapped into local slice of `local_data` array. For example, if given process addresses slices (8, 12) and (24, 28), then these slices will be mapped into (0, 4) and (4, 8) of the `local_data`. For each dimension where global_shapes matches local_shape, each device will lookup the slice in the local_data. For example if global_shape == local_data.shape, the local data is assumed to be the actual target array that will be sharded into device. If global_shape is the same as local_data.shape, then the data must be the same across all hosts. Examples: >>> from jax.sharding import PartitionSpec as P >>> mesh_rows = 2 >>> mesh_cols = jax.device_count() // 2 ... >>> mesh = jax.sharding.Mesh(np.array(jax.devices()).reshape(mesh_rows, mesh_cols), ('x', 'y')) >>> sharding = jax.sharding.NamedSharding(mesh, P(('x', 'y'),)) >>> rows_per_device = 2 >>> feature_length = 32 >>> per_device_shape = (rows_per_device, feature_length) >>> per_host_shape = (rows_per_device * len(mesh.local_devices), feature_length) >>> per_host_generator = lambda : np.arange(np.prod(per_host_shape)).reshape(per_host_shape) >>> per_host_data = per_host_generator() # replace with your own per-host data pipeline that outputs numpy arrays >>> global_shape = (rows_per_device * len(sharding.device_set), ) + per_device_shape[1:] >>> output_global_array = jax.make_array_from_process_local_data(sharding, per_host_data, global_shape) ... >>> assert output_global_array.addressable_data(0).shape == per_device_shape >>> assert output_global_array.shape == global_shape NB: While most shardings are uniform, It is possible to design am exotic sharding mesh where each process's devices will be arranged in a non-grid like pattern in some dimensions, or for indices to overlap non-trivially. Such sharding is called "non-uniform" in those dimensions. In that case, the global shape along those directions must match local shape as there is no meaningful way to represent all needed per-process data in non-overlapping fashion. For example for global_shape 4x4 if sharding looks like this: 0123 2103 4675 4567 with 4 processes, containing devices (0,1), (2, 3), (4, 5), (6, 7) respectively. Then the data for each host look like xx.. ..xx .... .... .xx. x..x .... .... .... .... x..x .xx. .... .... xx.. ..xx the sharding is uniform on rows (each host requires either rows 1-2, or rows 3-4) and non-uniform on columns (hosts require overlapping but not matching set of columns). Thus local data must have the shape 2x4 or 4x4 for all hosts, even though each host can potentially fit into 2x2 shape. In this case user must provide global_shape explicitly and for local_shape=(2, 4), potentially valid global shapes are (2, 4) and (4, 4). On the other hand for sharding: 0213 x.x. .x.x. .... .... 0213 x.x. .x.x. .... .... 4657 .... .... .x.x x.x. 4657 .... .... .x.x x.x. for local_shape=(2, 2) this function can accept a choice of 2x2, 2x4, 4x2 and 4x4 global shapes. Setting global_shape to None, is equivalent to setting it to (4, 4) in this case. Args: sharding: sharding of the global tensor. host_local_data: data on the host to be placed on local devices. Each dimension should either match global_shape, or match num_addressable_indices(dim). global_shape: the target shape of the global tensor. If None, will infer from host_local_data and sharding. Returns: Tensor that will have sharding=sharding and of shape global_shape. Nz{Unable to compute global_shape due to non-uniform sharding. Specify global shape directly. Partially computed global_shape=rz global_shape=z" has Nones. This is not supported.z^Invalid host data, each dimension should match either global or process shape. In dimension i=z, the process data has zelements. Process addresses z elements and global_shape=cg|]}gSr^r^)rrrs r5rz6make_array_from_process_local_data..s)C)C)C")C)C)Cr7rrk)maxsizec||pd|z}||z}t||Sr)rIr)rrendrslices_for_each_dims r5 local_slicez7make_array_from_process_local_data..local_slicesH  " ( (! 4 4{1~ EE +a. C   r7rIrr-r$cn|Jfdt|D}t|S)Nc3TK|]"\}}|r|n||jV#dSr@)r)rrrfull_dimrs r5rzAmake_array_from_process_local_data..cb..sV As{9 Asy 9 9r7)rqr)rI data_slicerrrs r5cbz.make_array_from_process_local_data..cbsY    &&J eJ'' ((r7)rIrr-r$)rvr#rKrqziprtr"ronextiterrrrrrr3sortedrpr[ lru_cacher)r1rr; local_shaperdata_dim global_dim process_slicerfshard shard_indexrrrrrrs ` @@@@r5"make_array_from_process_local_datar sn +(;??L  # # # |  Q@L Q Q Q    | I III J JJ (#, * L))$$   a (J OOH *+++:-h<HHm ( " " +,  FN  +8         ?? MM t&--//00 1 1%    %e\::+)C)Cl)C)C)C'..0044k  " " "K((443!##CIN33334 \"" # #AAa#C(;A(>$?$?@@ t$$$%$)))))))) "," = ==r7Sequence[basearray.Array]c td|Dr2tdttt|t j||djd}tj |jtj r#|jj |||dSt||ttt|dS)aReturns a ``jax.Array`` from a sequence of ``jax.Array``\s each on a single device. Every device in input ``sharding``\'s mesh must have an array in ``arrays``\s. Args: shape : Shape of the output ``jax.Array``. This conveys information already included with ``sharding`` and ``arrays`` and serves as a double check. sharding: Sharding: A global Sharding instance which describes how the output jax.Array is laid out across devices. arrays: Sequence of ``jax.Array``\s that are each single device addressable. ``len(arrays)`` must equal ``len(sharding.addressable_devices)`` and the shape of each array must be the same. For multiprocess code, each process will call with a different ``arrays`` argument that corresponds to that processes' data. These arrays are commonly created via ``jax.device_put``. Returns: A global ``jax.Array``, sharded as ``sharding``, with shape equal to ``shape``, and with per-device contents matching ``arrays``. Examples: >>> import math >>> from jax.sharding import Mesh >>> from jax.sharding import PartitionSpec as P >>> import numpy as np ... >>> mesh_rows = 2 >>> mesh_cols = jax.device_count() // 2 ... >>> global_shape = (8, 8) >>> mesh = Mesh(np.array(jax.devices()).reshape(mesh_rows, mesh_cols), ('x', 'y')) >>> sharding = jax.sharding.NamedSharding(mesh, P('x', 'y')) >>> inp_data = np.arange(math.prod(global_shape)).reshape(global_shape) ... >>> arrays = [ ... jax.device_put(inp_data[index], d) ... for d, index in sharding.addressable_devices_indices_map(global_shape).items()] ... >>> arr = jax.make_array_from_single_device_arrays(global_shape, sharding, arrays) >>> assert arr.shape == (8,8) # arr.shape is (8,8) regardless of jax.device_count() For cases where you have a local array and want to convert it to a global jax.Array, use ``jax.make_array_from_process_local_data``. c3JK|]}t|tjVdSr@)rrr)rrXs r5rz7make_array_from_single_device_arrays..s.88#C % %888888r7z`jax.make_array_from_single_device_arrays requires a list of concrete arrays as input. got types rF)rTr)rrKrpmaprnrrrrrrrrr,r r)rvr1rrbs r5$make_array_from_single_device_arraysrs\ 8888888A  @'*3tV+<+<'='= @ @ A AA  %E B B B$ tz6?33@ :  / /h:> 0 @ @@ 44(;V#D#D! # # ##r7c tj|jS#t$rA}dt |vr*td|j|d}~wwxYw)Nz9Fetching value for `jax.Array` that spans non-addressablezClosing over jax.Array that spans non-addressable (non process local) devices is not allowed. Please pass such arrays as arguments to the function. Got jax.Array: )r ir_constantrrrrbr)valrps r5_array_mlir_constant_handlerrs   CJ ' ''    Cc!ffLL  E.1h.@.@.B.B E E F FLMM  s A&.+s?6363#&q#636363636363r7)r'rr3 _multi_slicer shaped_abstractifyrr) rr4rr1rrrshardsrbs ` r5shard_device_arrayr*s/563636363*16363630303,-  !HS3w<< FF ^^M=, G GF  $Q ' '$ x A AAr7ctt}d|jD}t|j|j}t||D]-\}}|t| |.g}t||D]\}} |t|} | s&tj |g|j gddcS| D]2}|| hkr| |n3| |tj|j|||S)Ncg|] }|j Sr^)r=rs r5rz8shard_sharded_device_array_slow_path..7s 3 3 3s#( 3 3 3r7F) canonicalizer)rrrfrr1rPrvrrr&r!rtr shard_argsrr4rrb) rr4rr1 candidatesbufs arr_indicesr|rr:candidates_lists r5$shard_sharded_device_array_slow_pathr5sh4  * 3 3a2 3 3 3$aj44QW==DDFFGG+4--..hc3|C  !((---- $gw//kc6 c!2!23O L_hZ!(% H H H KKKK 6( " " C  # kk# 4 A AAr7c||}||}|t|t|kfSr@)rorrr) src_shardingrv dst_sharding src_indices dst_indicess r5_sharding_indices_and_eqrPs`<,>> >>r7c pg}ggggf\}}}}tt||D]N\}\}} |t|j|j| \} } |js'| r||Ztd| j } | rw|d|||t| || ||tj |jr'|t|| | | )|t|| | | Ptj|||} t|| D]\}}||J|||<|S)Nz5Cannot reshard an input that is not fully addressable)rqr&rrr1rvrrtNotImplementedErrorrrrrrrxc+batched_copy_array_to_devices_with_sharding)xs shardingsresultsbatch_xs batch_devsbatch_shardings batch_indicesrrr1r same_indicesr4 copy_outscopy_outs r5_array_shard_argr Ws '9;RR6(J#HR$;$;<<QQa!X4 AGX''G\ !Q Eq! CEE E7g  Qt$w--(((x(((Q  -aj 9 9Q)!WgxHHIIII 0GWh O O Q Q Q Q< O--)mY77ka 1:   GAJJ .r7cjtjkrfdStjjtjr!jj||Stj||dS)NcLtjjtjSr@r%zerosrvrfloat0)rr global_avals r5z._array_global_result_handler..sRXk/??r7Tr) rrrrrrglobal_sharded_result_handlerrarray_result_handler)r out_shardingrs` r5_array_global_result_handlerrs&-'' ? ? ? ?? {(&/::.   # A A\9 . .. <94   r7cjtjkrfdStjjtjr!jj||Stj|ddS)NcLtjjtjSr@r)rrrbs r5rz-_array_local_result_handler..sRXdj&-88r7Tr) rrrrrrlocal_sharded_result_handlerrr)rbr1rs` r5_array_local_result_handlerrs Z6=  8 8 8 88 tz6?33! :  9 9 h ! !!  H4   r7c6td|D|S)Ncg|] }|j Sr^)_buf)rrs r5rz$_token_shard_arg..s...a16...r7)r )rrs r5_token_shard_argrs! ..2... : ::r7cHttj||fd}|S)Nc<|i|}tj|Sr@)rToken)rekwargsout_buf array_handlers r5wrapperz-_token_global_result_handler..wrappers'mT,V,,G :g  r7)rrtoken_shaped_array)rrrr&r%s @r5_token_global_result_handlerr(s<. |Y88- .r7)r+r,r-r.)rrr1rrbrrr<)rrNrvr<r-r<)rvr<r1rrrr-r,r@)r1rrrr;rr-r,)rvr<r1rrrr-r,)rXrrrNr-r)j __future__r collectionsrcollections.abcrrenumr[roperatorrtypingrrr jax._srcr r r r rrrrrrrjax._src.interpretersrrrjax._src.layoutrrr jax._src.librrrrljax._src.shardingrjax._src.sharding_implsrrr r!r"r#jax._src.typingr$r% jax._src.utilr&r'r(r)r*numpyr%rrSr<r.rrN PRNGKeyArrayr6r9rjr|rrArrayr,rsetattrrrrcanonical_concrete_avalpytype_aval_mappings attrgetteridentitycanonicalize_dtype_handlers_shaped_abstractify_handlersregisterrregister_constant_handlerrrrrr shard_arg_handlersrglobal_result_handlersr ConcreteArrayrlocal_result_handlersrr"r( AbstractTokenr^r7r5rHs#"""""######........  ++++++++++$$$$$$&&&&&&&&&&&&%%%%%%AAAAAAAAAA)))))),,,,,,&&&&&&YYYYYYYYYYYYYYYY43333333PPPPPPPPPPPPPP c3h  eSj  00000000f5111   21  5111 > >21 >,SSSSS SSSr5)mmBL)))44) :t$$$ '---sMsMsMsMr"&l>l>l>l>l>^8#8#8#8#v(7'N)$&3bmF&;&;#-1] *3@2=3H3H%i0 ###    y*FGGGIIII2BBBBBB65111??21? $$$N&6 "1MD,-2ND./0K4+,1L4-. ;;;&6 #3OD.///r7