WpfdZddlmZddlmZddlmZddlZddlmZeZ dZ dZ d Z d Z d Zd Zd ZdZdZdZdZdZdZdZdZdZe de ee de ee de ee deee deee deee deee deee d eee d!eee d"eee d#eee d$eee d%eee d&eed'Z d(Z!d)Z"d*Z#d+Z$d,Z%d-Z&e d.e%ee d/e ee d0e!ee d1e"ee d2e#ee d3e$ee d4e&ed5Z'd6Z(d7Z)d8Z*d9Z+d:Z,d;Z-dZ0d?Z1d@Z2dAZ3dBZ4dCZ5dDZ6dEZ7dFZ8dGZ9dHZ:dIZ;dJZdMZ?dNZ@dOZAdPZBdQZCdRZDdSZEdTZFdUZGeFdVdWeFdXeGdYZHdZZId[ZJd\ZKd]ZLd^ZMd_ZNd`ZOdaZPdbZQdcZRddZSdeZTdfZUdgZVdhZWdiZXdjZYdkZZdlZ[dmZ\eYdndWeYdoeZeYdpe[eYdqe\drZ]dsZ^dtZ_duZ`dvZadwZbdxZcdyZddzZed{Zfe d|e]ee d}e^ee d~e_ee de`ee gdeaee hdebee hdecee idedee jdeeee hdefedZkdZldZmdZndZodZpdZqdZrdZsdZtdZudZvdZwdZxdZydZzdZ{e dekee |delee demee |denee deoee depee |deqee |deree jdesee }detee ~deuee hdevee ~dewee jdexee deyee dezee de{edS)aECopyright 2019 The MediaPipe Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. This script defines a large number of getters and setters for storing multimedia, such as video or audio, and related machine learning data in tf.train.SequenceExamples. These getters and setters simplify sharing data by enforcing common patterns for storing data in SequenceExample key-value pairs. The constants, macros, and functions are organized into 6 groups: clip metadata, clip label related, segment related, bounding-box related, image related, feature list related, and keyframe related. The following examples will walk through common task structures, but the relevant data to store can vary by task. The clip metadata group is generally data about the media and stored in the SequenceExample.context. Specifying the metadata enables media pipelines, such as MediaPipe, to retrieve that data. Typically, set_clip_data_path, set_clip_start_timestamp, and set_clip_end_timestamp define which data to use without storing the data itself. Example: tensorflow.train.SequenceExample sequence set_clip_data_path("/relative/path/to/data.mp4", sequence) set_clip_start_timestamp(0, sequence) set_clip_end_timestamp(10000000, sequence) # 10 seconds in microseconds. The clip label group adds labels that apply to the entire media clip. To annotate that a video clip has a particular label, set the clip metadata above and also set the set_clip_label_index and set_clip_label_string. Most training pipelines will only use the label index or string, but we recommend storing both to improve readability while maintaining ease of use. Example: set_clip_label_string(("run", "jump"), sequence) set_Clip_label_index((35, 47), sequence) The segment group is generally data about time spans within the media clip and stored in the SequenceExample.context. In this code, continuous lengths of media are called clips, and each clip may have subregions of interest that are called segments. To annotate that a video clip has time spans with labels set the clip metadata above and use the functions set_segment_start_timestamp, set_segment_end_timestamp, set_segment_label_index, and set_segment_label_string. Most training pipelines will only use the label index or string, but we recommend storing both to improve readability while maintaining ease of use. By listing segments as times, the frame rate or other properties can change without affecting the labels. Example: set_segment_start_timestamp((500000, 1000000), sequence) # in microseconds set_segment_end_timestamp((2000000, 6000000), sequence) set_segment_label_index((35, 47), sequence) set_segment_label_string(("run", "jump"), sequence) The bounding box group is useful for identifying spatio-temporal annotations for detection, tracking, or action recognition. The exact keys that are needed can vary by task, but to annotate a video clip for detection set the clip metadata above and use repeatedly call add_bbox, add_bbox_timestamp, add_bbox_label_index, and add_bbox_label_string. Most training pipelines will only use the label index or string, but we recommend storing both to improve readability while maintaining ease of use. Because bounding boxes are assigned to timepoints in a video, changing the image frame rate can can change the alignment. The media_sequence.h's ReconcileMetadata function can align bounding boxes to the nearest image. The image group is useful for storing data as sequential 2D arrays, typically encoded as bytes. Images can be RGB images stored as JPEG, discrete masks stored as PNG, or some other format. Parameters that are static over time are set in the context using set_image_width, set_image_height, set_image_format, etc. The series of frames and timestamps are then added with add_image_encoded and add_image_timestamp. For discrete masks, the class or instance indices can be mapped to labels or classes using set_class_segmentation_class_label_{index,string} and set_instance_segmentation_object_class_index. The feature list group is useful for storing audio and extracted features, such as per-frame embeddings. SequenceExamples only store lists of floats per timestep, so the dimensions are stored in the context to enable reshaping. For example, set_feature_dimensions and repeatedly calling add_feature_floats and add_feature_timestamp adds per-frame embeddings. The feature methods also support audio features. Macros for common patterns are created in media_sequence_util.py and are used here extensively. Because these macros are formulaic, I will only include a usage example here in the code rather than repeating documentation for every instance. This header defines additional functions to simplify working with MediaPipe types. Each msu.create_{TYPE}_context_feature takes a NAME and a KEY. It provides setters and getters for SequenceExamples and stores a single value under KEY in the context field. The provided functions are has_${NAME}, get_${NAME}, set_${Name}, and clear_${NAME}. Eg. tf.train.SequenceExample example set_data_path("data_path", example) if has_data_path(example): data_path = get_data_path(example) clear_data_path(example) Each msu.create_{TYPE}_list_context_feature takes a NAME and a KEY. It provides setters and getters for SequenceExamples and stores a sequence of values under KEY in the context field. The provided functions are has_${NAME}, get_${NAME}, set_${Name}, clear_${NAME}, get_${NAME}_at, and add_${NAME}. Eg. tf.train.SequenceExample example set_clip_label_string(("run", "jump"), example) if has_clip_label_string(example): values = get_clip_label_string(example) clear_clip_label_string(example) Each msu.create_{TYPE}_feature_list takes a NAME and a KEY. It provides setters and getters for SequenceExamples and stores a single value in each feature field under KEY of the feature_lists field. The provided functions are has_${NAME}, get_${NAME}, clear_${NAME}, get_${NAME}_size, get_${NAME}_at, and add_${NAME}. tf.train.SequenceExample example add_image_timestamp(1000000, example) add_image_timestamp(2000000, example) if has_image_timestamp(example): for i in range(get_image_timestamp_size()): timestamp = get_image_timestamp_at(example, i) clear_image_timestamp(example) Each VECTOR_{TYPE}_FEATURE_LIST takes a NAME and a KEY. It provides setters and getters for SequenceExamples and stores a sequence of values in each feature field under KEY of the feature_lists field. The provided functions are Has${NAME}, Get${NAME}, Clear${NAME}, Get${NAME}Size, Get${NAME}At, and Add${NAME}. tf.train.SequenceExample example add_bbox_label_string(("run", "jump"), example) add_bbox_label_string(("run", "fall"), example) if has_bbox_label_string(example): for i in range(get_bbox_label_string_size(example)): labels = get_bbox_label_string_at(example, i) clear_bbox_label_string(example) As described in media_sequence_util.h, each of these functions can take an additional string prefix argument as their first argument. The prefix can be fixed with a new NAME by using functools.partial. Prefixes are used to identify common storage patterns (e.g. storing an image along with the height and width) under different names (e.g. storing a left and right image in a stereo pair.) An example creating functions such as add_left_image_encoded that adds a string under the key "LEFT/image/encoded" add_left_image_encoded = msu.function_with_default(add_image_encoded, "LEFT") )absolute_import)division)print_functionN)media_sequence_utilTz example/idzexample/dataset_namezexample/dataset/flag/stringzclip/data_pathz clip/media_idzclip/alternative_media_idzclip/encoded_media_bytesz"clip/encoded_media_start_timestampzclip/start/timestampzclip/end/timestampzclip/label/indexzclip/label/stringzclip/label/confidencezclip/label/start/timestampzclip/label/end/timestamp example_id module_dictexample_dataset_nameexample_dataset_flag_string clip_media_idclip_alternative_media_idclip_encoded_media_bytesclip_data_path"clip_encoded_media_start_timestampclip_start_timestampclip_end_timestampclip_label_stringclip_label_indexclip_label_confidenceclip_label_start_timestampclip_label_end_timestampzsegment/start/timestampzsegment/start/indexzsegment/end/timestampzsegment/end/indexzsegment/label/indexzsegment/label/stringzsegment/label/confidencesegment_label_stringsegment_start_timestampsegment_start_indexsegment_end_timestampsegment_end_indexsegment_label_indexsegment_label_confidencezregion/bbox/yminzregion/bbox/xminzregion/bbox/ymaxzregion/bbox/xmaxzregion/point/xzregion/point/yz region/radiuszregion/3d_point/xzregion/3d_point/yzregion/3d_point/zzregion/num_regionszregion/is_annotatedzregion/is_generatedzregion/is_occludedzregion/label/indexzregion/label/stringzregion/label/confidencezregion/track/indexzregion/track/stringzregion/track/confidencezregion/class/indexzregion/class/stringzregion/class/confidencezregion/timestampzregion/unmodified_timestampz region/partsz®ion/embedding/dimensions_per_regionzregion/embedding/formatzregion/embedding/floatzregion/embedding/encodedzregion/embedding/confidencect|dzt|tt|dzt|tt|dzt |tt|dzt|tt|dzt|tt|dzt|tt |dzt|tt|d zt|tt |d zt|tt |d zt|tt|d zt |tt |d zt"|tt |dzt$|tt|dzt&|tt |dzt(|tt |dzt*|tt |dzt,|tt |dzt.|tt |dzt0|tt |dzt2|tt |dzt4|tt |dzt6|tt |dzt8|tt |dzt:|tt|dzt>|tt |dztB|tt"|dztF|tt |dztH|tt |dztJ|tt |dztL|td }d!}d"}d#}d$}d%}d&}d'} d(} d)} d*} d+} d,}d-}d.}t'd/|zd0zt(||d1|zt(||d/|zd2zt(||d3|zt(||d4|zt(||it5t'd/|zd6zt(||d1|zd7zt(||d/|zd8zt(| |d3|zd7zt(| |d4|zd7zt(| |it5t'd/|zd9zt(| |d1|zd:zt(| |d/|zd;zt(||d3|zd:zt(||d4|zd:zt(||it5d.get_prefixed_bbox_atsm 8 0@@@ 0@@@ 0@@@ 0@@@ B    ctj|}|jdkrJtg||t g||t g||t g||dSt|dddf||t |dddf||t |dddf||t |dddf||dS)NrrBrC)rDarraysize add_bbox_ymin add_bbox_xmin add_bbox_ymax add_bbox_xmaxvaluesrLr"s rMadd_prefixed_bboxz5_create_region_with_prefix..add_prefixed_bboxs  Xf  F {aB(8888B(8888B(8888B(888888F111a4L"26BBBBF111a4L"26BBBBF111a4L"26BBBBF111a4L"26BBBBBBrOc$t||SNrB)get_bbox_ymin_sizerLr"s rMget_prefixed_bbox_sizez:_create_region_with_prefix..get_prefixed_bbox_sizes .v > > >>rOc$t||Sr]) has_bbox_yminr_s rMhas_prefixed_bboxz5_create_region_with_prefix..has_prefixed_bboxs )& 9 9 99rOct||t||t||t||dSr])clear_bbox_yminclear_bbox_xminclear_bbox_ymaxclear_bbox_xmaxr_s rMclear_prefixed_bboxz7_create_region_with_prefix..clear_prefixed_bboxsX$V4444$V4444$V4444$V444444rOcptjt|||t|||fdSrA)rDrEget_bbox_point_y_atget_bbox_point_x_atrJs rMget_prefixed_point_atz9_create_region_with_prefix..get_prefixed_point_atsG 8E#3FCCCE#3FCCCE  rOcvt|dddf||t|dddf||dS)NrrBrC)add_bbox_point_yadd_bbox_point_xrYs rMadd_prefixed_pointz6_create_region_with_prefix..add_prefixed_pointsLVAAAqD\#3FCCCCVAAAqD\#3FCCCCCCrOc$t||Sr])get_bbox_point_y_sizer_s rMget_prefixed_point_sizez;_create_region_with_prefix..get_prefixed_point_sizes !1& A A AArOc$t||Sr])has_bbox_point_yr_s rMhas_prefixed_pointz6_create_region_with_prefix..has_prefixed_points ,V < < <.clear_prefixed_points0'7777'777777rOc tjt|||t|||t |||fdSrA)rDrEget_bbox_3d_point_x_atget_bbox_3d_point_y_atget_bbox_3d_point_z_atrJs rMget_prefixed_3d_point_atz<_create_region_with_prefix..get_prefixed_3d_point_atsZ 8u&6vFFFu&6vFFFu&6vFFFH   rOct|dddf||t|dddf||t|dddf||dS)NrrBrCrQ)add_bbox_3d_point_xadd_bbox_3d_point_yadd_bbox_3d_point_zrYs rMadd_prefixed_3d_pointz9_create_region_with_prefix..add_prefixed_3d_pointsnqqq!t &6vFFFFqqq!t &6vFFFFqqq!t &6vFFFFFFrOc$t||Sr])get_bbox_3d_point_x_sizer_s rMget_prefixed_3d_point_sizez>_create_region_with_prefix..get_prefixed_3d_point_sizes #$4V D D DDrOc$t||Sr])has_bbox_3d_point_xr_s rMhas_prefixed_3d_pointz9_create_region_with_prefix..has_prefixed_3d_points / ? ? ??rOclt||t||t||dSr])clear_bbox_3d_point_xclear_bbox_3d_point_yclear_bbox_3d_point_zr_s rMclear_prefixed_3d_pointz;_create_region_with_prefix..clear_prefixed_3d_pointsD*6::::*6::::*6::::::rOget__atadd__sizehas_clear_r _point_at_point _point_size _3d_point_at _3d_point_3d_point_sizeN))msucreate_int_feature_listREGION_NUM_REGIONS_KEYglobalsREGION_IS_ANNOTATED_KEYcreate_int_list_feature_listREGION_IS_OCCLUDED_KEYREGION_IS_GENERATED_KEYREGION_TIMESTAMP_KEYREGION_UNMODIFIED_TIMESTAMP_KEYcreate_bytes_list_feature_listREGION_LABEL_STRING_KEYREGION_LABEL_INDEX_KEYcreate_float_list_feature_listREGION_LABEL_CONFIDENCE_KEYREGION_CLASS_STRING_KEYREGION_CLASS_INDEX_KEYREGION_CLASS_CONFIDENCE_KEYREGION_TRACK_STRING_KEYREGION_TRACK_INDEX_KEYREGION_TRACK_CONFIDENCE_KEYREGION_BBOX_YMIN_KEYREGION_BBOX_XMIN_KEYREGION_BBOX_YMAX_KEYREGION_BBOX_XMAX_KEYREGION_POINT_X_KEYREGION_POINT_Y_KEYREGION_3D_POINT_X_KEYREGION_3D_POINT_Y_KEYREGION_3D_POINT_Z_KEY!create_bytes_list_context_featureREGION_PARTS_KEY!create_float_list_context_feature*REGION_EMBEDDING_DIMENSIONS_PER_REGION_KEYcreate_bytes_context_featureREGION_EMBEDDING_FORMAT_KEYREGION_EMBEDDING_FLOAT_KEYREGION_EMBEDDING_ENCODED_KEYREGION_EMBEDDING_CONFIDENCE_KEYadd_functions_to_modulefunction_with_default)namer"rNr[r`rcrirmrqrtrwr{rrrrrs rM_create_region_with_prefixrMs^ d^35K%+DDDd_46M%+DDD"" ^3 #,,,"" _5 #,,,d\13G%+DDD $$&E ,,,$$ _5 %,,,"" ^3 #,,,$$  "= %,,,$$ _5 %,,,"" ^3 #,,,$$  "= %,,,$$ _5 %,,,"" ^3 #,,,$$  "= %,,,$$TG^5I,2 %KKK$$TG^5I,2 %KKK$$TG^5I,2 %KKK$$TG^5I,2 %KKK$$TJ%68J,2 %KKK$$TJ%68J,2 %KKK$$ ]1 %,,,$$ ]1 %,,,$$ ]1 %,,,''x(8/5799(NNN'' //0 (,,,""4*=#=#>*0gii#III$$T,?%?%?,2 %KKK$$T,@%@%A,2 %KKK$$T,C%C%D,2 %KKK C C C???:::555  DDDBBB===888 GGGEEE@@@;;;  tme # #$8& A A tm # #$5v > > tmg # #$:F C C tm # #$5v > >o # #$7 @ @     tmk! # #$96 B B tmh # #$6 ? ? tmm# # #$;V D D tmh # #$6 ? ?o # #$8& A A     tmn$ # #$ G G tmk! # #$96 B Bo # # #$;V D D      rO PREDICTEDbboxpredicted_bboxz image/formatzimage/channelszimage/colorspacez image/heightz image/widthzimage/frame_ratezimage/saturationzimage/class/label/indexzimage/class/label/stringzimage/object/class/indexz image/encodedzimage/multi_encodedzimage/timestampzimage/label/indexzimage/label/stringzimage/label/confidencezimage/data_pathct|dzt|tt|dzt|tt|dzt |tt|dzt|tt|dzt|tt |dzt|tt |dzt|tt |d zt|tt|d zt |tt|d zt"|tt|d zt$|tt|d zt(|tt|dzt,|tdS)z/Create multiple accessors for image based data._formatr! _colorspace _channels_height_width_encoded _frame_rate_class_label_string_class_label_index_object_class_index _data_pathr&_multi_encodedN)rrIMAGE_FORMAT_KEYrIMAGE_COLORSPACE_KEYcreate_int_context_featureIMAGE_CHANNELS_KEYIMAGE_HEIGHT_KEYIMAGE_WIDTH_KEYcreate_bytes_feature_listIMAGE_ENCODED_KEYcreate_float_context_featureIMAGE_FRAME_RATE_KEYrIMAGE_CLASS_LABEL_STRING_KEYcreate_int_list_context_featureIMAGE_CLASS_LABEL_INDEX_KEYIMAGE_OBJECT_CLASS_INDEX_KEYIMAGE_DATA_PATH_KEYrIMAGE_TIMESTAMP_KEYrIMAGE_MULTI_ENCODED_KEY)rr"s rM_create_image_with_prefixr/s""4)#35E*0gii#III""4-#79M*0gii#III   !35G(.GII!GGG   !13C(.GII!GGG  /(.GII!GGGz 13D'-799 FFF""4-#79M*0gii#III'' ""$@ (,,,%% !!#> &,,,%% ""$@ &,,,""4,#68K*0gii#IIId\13F%+DDD$$T,<%<%rNsT XXt'&&&&&%%%%%%7777771"?%# ;9)M&1-)+3!=9 .ggii!999 4''))!MMM%%!#B & &GGII!??? !> !  < ! (ggii!AAA(* CCC4''))MMM0ggiiIII%%.GGII&GGG##,''))$EEE%%6GGII&OOO## " $##  $8/3+02 :%%4''))&MMM##WWYY$888##2 $KKK##6GGII$OOO##.GGII$GGG##2 $KKK%% ggii&999*)))%%#+++-0/-./7./7./7)#@!-+869"?vvvr62&&&+-=>>> "%)!))799#/',-5'<<<@%06'2&&&.*=>>>.0IJJJ13OPPP$135"%#'%# &GGII!??? 4''))!MMM##6GGII$OOO%%8ggii&QQQ$''))===(ggiiAAA&GGII???* CCC""(ggii#AAA&GGII??? .!1/1// ;%#!+*-531##0ggii$III $''))!=== 4''))!MMM 2 !KKK4''))MMM2 KKK 2 !KKK !> !""(ggii#AAA""&GGII#??? $''))!===.GGIIGGG ,''))!EEE""0ggii#III%%8ggii&QQQ%%6GGII&OOO##4''))$MMMMMrO