VpfqUdZddlmZddlZddlmZmZmZmZddl Z ddl Z ddl Z ddl m Z ddlmZddlZe jeZdZdZd Zd Zd d id d iddidddddiddddZded<dZdZdZdGdZdHd!Z eeeeee iZ!d"ed#<d$d%dId*Z"dJd+Z#dKd/Z$dLd4Z%dMd6Z&dNd8Z'dOd9Z(dPd;Z)dQd<Z* dRd$d$d=dSd@Z+ dRd$dAd$dBdTdFZ,dS)Uz!Utils for building a device mesh.) annotationsN)Callable GeneratorMutableMappingSequence)Any) xla_bridgezTPU v2zTPU v3zTPU v4z TPU v5 lite)r )rr )r )r )rr r ))@r )r r)r))r)rr))r r r )r r r )r r r )r rr)rrr)rr r z=dict[tuple[int, ...], dict[tuple[int, ...], tuple[int, ...]]]_TRANSPOSE_TRICKS)rr r r )rr rr )rr r rrrr  rr mesh_shape Sequence[int]devices Sequence[Any]return np.ndarrayc "t|dkrdtdtj|}|tjt }||}|S|ddkrftj||}tdtjt }|d|f}|Stj||S)Nrz@Reordering mesh to physical ring order on single-tray TPU v2/v3.z>Reordering mesh to physical ring order on each TPU v2/v3 tray..)lenloggerinfonpasarrayarray_TRAY_RING_ORDERreshape)rr unused_kwargs device_meshperms [/var/www/html/nettyfy-visnx/env/lib/python3.11/site-packages/jax/experimental/mesh_utils.py_tpu_v2_v3_create_device_meshr1Gs  \\Q KKJ*W%%Kbh'7889K%%j11K "~*W%%--j99K KKH 8$ % %Dc4i(K  :g   & &z 2 22np.ndarray | Nonec td|D\}}}|dz|dz|dz}}}t|d} ||cxkrdkrfnnc|dkr]t|dkrJtj| } | tjt } | |} | S||cxkrdkrnn|dkr}t|dkrjt|t|krJtj| } | tjt} | |} | SdS) zCreates rotated pincer device assignment for selected topologies. Args: mesh_shape: Logical mesh shape used by the model. devices: TPU devices. **unused_kwargs: ... Returns: None or reordered devices reshaped as `mesh_shape`. c38K|]}t|ddVdS)coordsrrrN)getattr.0ds r0 z*_vlc_create_device_mesh..ps.MMGAx;;MMMMMMr2r c Xttt|ddS)Nr6r7)tuplereversedr8)r;s r0z)_vlc_create_device_mesh..ws"E(71h #B#BCCDDr2)keyr r r N) maxsortedr%r(r)r*_TRAY_2x2_RING_ORDERr,_TRAY_4x4_RING_ORDER) rrr-max_xmax_ymax_zbound_xbound_ybound_zsequential_devicesr.s r0_vlc_create_device_meshrMcsqMMWMMMMM%#aiEAIG7' D DFFF 1A#g,,!2C2C*/00Kbh';<<=K%%j11K  1A#g,,"2D2D :#g,,&&J122k)= > >?k'' 33k  r2z+dict[str, Callable[..., np.ndarray | None]]device_kind_handler_dictFallow_split_physical_axes physical_meshrPbooltuple[np.ndarray, np.ndarray]c t|j}d|D}ttt|D]\}}t dddD]y}t jt||}|D]H} t| \} tj | |kr"||<fdt|D}nI||rn4z|dkr-|std|d|d |d t||cSg} tj t|jt|gtj } t|D]@\} }|D]8}t|}|j|| || f<| |9A|| || fS) aAssigns logical parallelism axes to physical axes of an N-D torus network. Given logical parallelism axes with sizes in `mesh_shape` and devices in an N-dimensional torus network represented by `physical_mesh`, maps each logical axis to one or more physical axes. Prefer to map more-performance-sensitive logical axes to larger numbers of physical axes to maximize the bandwidth available to them. Also prefer to assign logical axes to multiple physical axes of the same size (e.g., a 2D square) rather than multiple physical axes of different sizes when possible. If allow_split_physical_axes = False (default), this routine will error out instead of splitting a physical axis over more than one logical axis (which would reduce total usable bandwidth). Let's use a concrete example to explain the concepts and considerations. As an example, suppose the logical mesh is [data, model], for data and model parallelism respectively. Also suppose that data parallelism is less performance sensitive than model parallelism. Consider a 3D TPU pod slice of shape 4x4x16, represented by a physical mesh of shape (4, 4, 16). A TPU pod slice has equal bandwidth along all axes with wraparound links, but a 2D plane of size 4x4 may have faster XLA collective implementations than a non-square plane or a 1D subgroup. If the mesh_shape is [16, 16], we may want the more performance sensitive `model` axis to be mapped to the 4x4 XY plane. Args: physical_mesh: a np.ndarray of devices in the shape of the N-D torus physical topology. mesh_shape: shape of the logical mesh (size of the various logical parallelism axes), with axes ordered by increasing network intensity. allow_split_physical_axes: If True, we would split physical axes if necessary to fit the desired mesh shape. Returns: An np.ndarray of devices in the shape of the logical mesh (mesh_shape), with each logical parallelism axis mapped to one or more physical mesh axes. The axis assignment matrix, which is a 2-d array mapping from (physical_axis, logical_axis) to the size assigned, with the invariant np.prod(assignment, axis=1) = physical_mesh_shape, and np.prod(assignment, axis=0) = mesh_shape. cg|]}dS)rV)r:_s r0 z4_create_device_mesh_for_nd_torus..s&>&>&>ar&>&>&>r2rrr$c&g|] \}}|vrdn|S)rrV)r:iv c_indicess r0rXz4_create_device_mesh_for_nd_torus..s9& & & !Q >>aaq& & & r2r z1Failed to find assignment for logical_axis_index z of size z with remaining assignable mesh a. The size of each axis in your logical mesh must be equal to the product of some subset of the physical mesh axis sizes. E.g. logical mesh (4, 16) is compatible with physical mesh 4x4x4 since 4=4 and 16=4x4. If you want to split physical axes, set allow_split_physical_axes to True.dtype)listshaper? enumeraterange itertools combinationszipr(prodNotImplementedError/_create_device_mesh_for_nd_torus_splitting_axesonesr%int64intappend transposer,)rQrrPassignable_physical_mesh assignmentlogical_axis_indexlogical_axis_sizenum_axesindices_and_axeselemc_axesrmassignment_arrayrZxyphysical_mesh_axisr\s @r0 _create_device_mesh_for_nd_torusrzsb"-"566&>&>:&>&>&>* 08 9Z !!002 2 ++!QOO. . "/ , - -x#$J 6 76??/ / /+4*' (& & & & #$<==& & & " % 0 & '  Q  ( $4$44/@44,D444   AZ     )W = Z1 ##++da ++q660=0C 1)1,-)**** +i((00<< r2c.tj|jtj|krtd|jd|d|j}t |}tjt |t |gtj}ttt|D];\}}d}t|||D]}|t||||r|}||dd|f<rir%rjr?r_ra,_enumerate_feasible_logical_axis_assignments%_prefer_first_logical_axis_assignment_generate_logical_mesh) rQrr}logical_mesh_shapero logical_axisrqbest_logical_axis_assignmentlogical_axis_assignment logical_meshs r0rhrhs|8W] !!RWZ%8%888  *   * *& * * *   &+Z((w %7!8!89**2 9' ( ())**??%l% $( #OZ):$$?? ' . 2%*"5#     /(?$">Jqqq,(', z !!r2rwrk list[int]c|dksJg}tdtj|dzD]8}||zdkr#||||z}||zdk#|dkr|cS9|gS)z"?"?27rDDD#d##,5#,,<<'m'%%788<< + + +f%,,];;;;< '288::fl NN6    "'/    #*,ABo Wc-../rx @ @ @F/8/I/I77+ +077-}!667 LLLL r2rxc tjfdt|D}tjfdt|D}||kr||kStfdt|D}tfdt|D}||kr||kStj|d tj fdt|D}tj fdt|D} || kr|| kSt |t |kS) afReturns True if the first axis assignment is preferred over the second. For now, this is implemented with some very simple heuristics. However, it is possible to introduce e.g., a value function here based on a more precise model of the underlying hardware. TODO(rosun): Use a proxy of network capacity to select the partitions. Args: x: Logical axis assignment as [len(physical_mesh_shape)] array. y: Logical axis assignment as [len(physical_mesh_shape)] array. physical_mesh_shape: Physical mesh shape. assignment: Assignment matrix. Returns: True if x is preferred over y. c2g|]\}}||k|SrVrVr:rZsr}s r0rXz9_prefer_first_logical_axis_assignment..-AAATQQ*=a*@%@%@q%@%@%@r2c2g|]\}}||k|SrVrVrs r0rXz9_prefer_first_logical_axis_assignment..rr2c>g|]\}}||k|dkdSr rVrs r0rXz9_prefer_first_logical_axis_assignment..2KKKTQQ*=a*@%@%@QUUqUUUr2c>g|]\}}||k|dkdSrrVrs r0rXz9_prefer_first_logical_axis_assignment..rr2r$rc2g|]\}}|dk|SrrVr:rZrassigned_physical_mesh_shapes r0rXz9_prefer_first_logical_axis_assignment..-IIITQ%A!%Dq%H%Hq%H%H%Hr2c2g|]\}}|dk|SrrVrs r0rXz9_prefer_first_logical_axis_assignment..rr2)r(rfrar%r>) rwrxr}rox_whole_axis_sizey_whole_axis_sizex_num_whole_axesy_num_whole_axesx_non_overlapping_axis_sizey_non_overlapping_axis_sizers ` @r0rrs:gAAAAYq\\AAAgAAAAYq\\AAA+++ 0 00 KKKKYq\\KKKKKKKYq\\KKK))) . .."$"!=!=!= "IIIIYq\\III!!!#IIIIYq\\III!!!$??? &)D DD qE!HH r2rc tjtjtjt |jtjd|jdg}tjtjtjt |tjd|jdg}tj||dg}ttt||tt |\}}}tj ||}tj||}|S)aCompute the logical mesh from assignment map. Args: physical_mesh: Physical device mesh. logical_mesh_shape: Logical mesh shape. assignment: 2-d assignment matrix shape [physical_dims, logical_dims]. Returns: Logical mesh reshaped from physical mesh. r]r$rr) r( broadcast_to expand_dimsaranger%r`rjr,rerCrbrm)rQrrophysical_indiceslogical_indicesrrWtranspose_axess r0rrsT_n )C +,,BH = = =B   GRDMM On )C*++28 < < <1   GRDMM  M:+=+=rd+C+CDD, o/s?7K7K1L1L M M!Q lN;;,L*<==, r2czt|ds Jd|j\}}}|dz|dz|dz|jdzfS)z*Gets the bound from the given last device.r6zOnly TPU supportedr )hasattrr6 core_on_chip) last_devicerwrxzs r0_bounds_from_last_devicer3sT h ' '==)====  '!Q QAq1uk6: ::r2 jax_devicesc |dj}d|D}tdt|D}t|dks J||tt fvrtd|Ddz}t j|dd|fzt }t||D]4\}}|ddks J||||d|d|j f<5ntt j|t }t||D]H\}}|j dkrtd |j d |d |d |||d|d|df<I|S)aVRearrange TPU devices in a slice into a physical mesh. Args: jax_devices: A list of JAX devices in a TPU slice in process-tiled z, y, x, core order, e.g. from jax.devices(). Returns: A np.ndarray of JAX devices with shape [global_x, global_y, global_z]. On v2 and v3, global_z is instead cores_per_chip (i.e., 2). rcg|] }|j SrV)r6r9s r0rXz*_get_physical_tpu_mesh..Is11118111r2c3 K|] }|dzV dS)r NrVr9s r0r<z)_get_physical_tpu_mesh..Js&11q1u111111r2rc3$K|] }|jV dSN)rr9s r0r<z)_get_physical_tpu_mesh..Ms$==A======r2r Nr r]zZCreating meshes for TPU >v3 requires one device per chip ("megacore" mode). Got device id z for a device of kind z: r|) device_kindr>rBr%_TPU_V2_TPU_V3r(emptyobjectrerAssertionError)rr device_coordsdimscores_per_chipoutr6r;s r0_get_physical_tpu_meshr=sA*+11[111- 11c-00111 1 1$ TaWg&&&=======AN (48~//v > > >C 4444  AY!^^^Q^^^23c&)VAY .//4 (4v & & &C 44//  1   ,12 , ,# , ,'( , , ,   ./c&)VAYq )** *r2c bt|}|j}|tvrtd|d}|D]}|dkr||fz }|t|vrBtd|d|dt t||jt||S)NzQcreate_device_mesh cannot create contiguous submeshes for physical mesh topology rVr zEcreate_device_mesh cannot create contiguous submeshes for mesh_shape z and physical mesh topology z. Available mesh_shapes: )r>r`rr~r_keysrm)rQrtopologymesh_shape_no_trivial_dimsdim_sizes r0_transpose_trickr`s Z  *  ( &&&  -"* - -   1300h1}} XK/ '8'BBB  M  M M>F M M"&'8'B'G'G'I'I"J"J M M   ! "#=> r2)contiguous_submeshesrPSequence[Any] | Nonerc |tj}tj|t |kr"t dt |d||d}t |jd}|||||}||S|j dkr8t|}|rt||}t|||\}} |Stj ||}|S)ahCreates a performant device mesh for jax.sharding.Mesh. Args: mesh_shape: shape of logical mesh, ordered by increasing network-intensity e.g. [replica, data, mdl] where mdl has the most network communication requirements. devices: optionally, the devices to construct a mesh for. Defaults to jax.devices(). contiguous_submeshes: if True, this function will attempt to create a mesh where each process's local devices form a contiguous submesh. A ValueError will be raised if this function can't produce a suitable mesh. This setting was sometimes necessary before the introduction of jax.Array to ensure non-ragged local arrays; if using jax.Arrays, it's better to keep this set to False. allow_split_physical_axes: If True, we will split physical axes if necessary to produce the desired device mesh. Raises: ValueError: if the number of devices doesn't equal the product of `mesh_shape`. Returns: A np.ndarray of JAX devices with mesh_shape as its shape that can be fed into jax.sharding.Mesh with good collective performance. NzNumber of devices z& must equal the product of mesh_shape r$)rtpurO)xbrr(rfr%r~rNgetrplatformrrrzr)r,) rrrrPrhandlerrrQr.rWs r0create_device_meshr|s9@ _jllGWZCLL((  &S\\ & &# & &   + $ ( ()@$ G G'  WG2FF mU""*733MB&}jAAm5";NK *W%%--j99K r2T)process_is_granuleshould_sort_granules_by_keyrPdcn_mesh_shaperrc8 |tj}|rdnd}t|d|sJtjt  |D]+} t |||,|r- fdt Dn }tj |t|kr"tdt|d|fd|D tjt||} tj fd t$g | } tj| } | S) aCreates a device mesh for hybrid (e.g., ICI and DCN) parallelism. Args: mesh_shape: shape of the logical mesh for the faster/inner network, ordered by increasing network intensity, e.g. [replica, data, mdl] where mdl has the most network communication requirements. dcn_mesh_shape: shape of the logical mesh for the slower/outer network, in the same order as mesh_shape. devices: optionally, the devices to construct a mesh for. Defaults to jax.devices(). process_is_granule: if True, this function will treat processes as the units of the slower/outer network. Otherwise it will look for slice_index attributes on devices and use slices as the units. Enabling this is meant as a fallback for platforms that don't set slice_index. should_sort_granules_by_key: Whether device granules should be sorted by the granule key, either slice or process index, depending on process_is_granule. allow_split_physical_axes: If True, we will split physical axes if necessary to produce the desired device mesh. Raises: ValueError: if the number of slices to which the `devices` belong doesn't equal the product of `dcn_mesh_shape`, or if the number of devices belonging to any single slice does not equal the product of `mesh_shape`. Returns: A np.ndarray of JAX devices with mesh_shape * dcn_mesh_shape as its shape that can be fed into jax.sharding.Mesh for hybrid parallelism. N process_index slice_indexrc g|] }| SrVrV)r:rA granule_dicts r0rXz-create_hybrid_device_mesh..s@@@S|C@@@r2zNumber of slices z* must equal the product of dcn_mesh_shape c4g|]}t|S)rO)r)r:granulerPrs r0rXz-create_hybrid_device_mesh..sC    $=r2c|SrrV)rZper_granule_meshess r0r@z+create_hybrid_device_mesh..s "4Q"7r2)otypes)rrrrrr_r8rlrCrvaluesr(rfr%r~rr, vectorizerblocktolist)rrrrrrPattrdevgranules granule_meshblocksr.rrs` ` @@r0create_hybrid_device_meshrsL _jllG. AM$ T " """"(.., 11cd##$++C0000 %!@@@@F<+<+<+>+>$?$?@@@@     W^H --  +CMM + +( + +    3x==))11.AA, I2<7777 I I I  &))+ r2)rrrr r!r")rrrr r!r3)rQr"rrrPrRr!rS)rQr"rrr!rS)rwrkr!r)r}rror"rqrkr!r) rwr"rxr"r}rror"r!rR)rQr"rrror"r!r")r!r)rr r!r")rQr"rrr!r"r) rrrrrrRrPrRr!r")rrrrrrrrRrrRrPrRr!r")-__doc__ __future__rrcollections.abcrrrrrcloggingrtypingrjax._srcr rnumpyr( getLogger__name__r&rr_TPU_V4 _TPU_V5_LITEr__annotations__r+rDrEr1rMrNrzrhrrrrrrrrrrVr2r0rs(''""""""IIIIIIIIIIII %%%%%%  8 $ $          #4,#M33338$$$$\ * *)', {{{{{{|J"J"J"J"Z    @@@@FGGGGT3333l;;;;    F<%)="'&+ ======F%)E %(,&+EEEEEEEEr2