U diJ@sdZddlZddlmZddlZddlZddlm Z ddl m Z ddl m Z mZmZmZerfddlZeeZddZe rerdd lmZn dd lmZGd d d eZGd ddeZdaddZddZddZddZ ddZ!dddZ"d ddZ#dS)!z Integration with Deepspeed N) partialmethod)dep_version_check)is_accelerate_availableis_torch_availableis_torch_mlu_availableloggingcCsXtjddk }|rTz&tr,td}WdStd}WdStjk rRYdSXdS)N deepspeedz deepspeed-mluTF) importlibutil find_specrimportlib_metadatametadataZPackageNotFoundError)Zpackage_exists_rG/tmp/pip-unpacked-wheel-bm_b0l5e/transformers/integrations/deepspeed.pyis_deepspeed_available#s  r)HfDeepSpeedConfig)objectcs eZdZdZfddZZS)raJ This object contains a DeepSpeed configuration dictionary and can be quickly queried for things like zero stage. A `weakref` of this object is stored in the module's globals to be able to access the config from areas where things like the Trainer object is not available (e.g. `from_pretrained` and `_get_resized_embeddings`). Therefore it's important that this object remains alive while the program is still running. [`Trainer`] uses the `HfTrainerDeepSpeedConfig` subclass instead. That subclass has logic to sync the configuration with values of [`TrainingArguments`] by replacing special placeholder values: `"auto"`. Without this special logic the DeepSpeed configuration is not modified in any way. Args: config_file_or_dict (`Union[str, Dict]`): path to DeepSpeed config file or dict. cs(t|tdtdt|dS)NZ accelerater )set_hf_deepspeed_configrsuper__init__selfZconfig_file_or_dict __class__rrrLszHfDeepSpeedConfig.__init__)__name__ __module__ __qualname____doc__r __classcell__rrrrr;srcsXeZdZdZfddZddZddZdd d Zeed d Z dddZ ddZ Z S)HfTrainerDeepSpeedConfigz The `HfTrainerDeepSpeedConfig` object is meant to be created during `TrainingArguments` object creation and has the same lifespan as the latter. cst|d|_g|_dSN)rr_dtype mismatchesrrrrrZs z!HfTrainerDeepSpeedConfig.__init__cCs|jdkrtd|jS)Nz8trainer_config_process() wasn't called yet to tell dtype)r# ValueErrorrrrrdtype_s zHfTrainerDeepSpeedConfig.dtypecCs"||}|dkrdS|dkSdS)NFauto) get_value)r ds_key_longvalrrris_autods z HfTrainerDeepSpeedConfig.is_autoNTc Cs~||\}}|dkrdS||dkr4|||<dS|srrrtrainer_config_processs`       z/HfTrainerDeepSpeedConfig.trainer_config_processcsJdddg}fdd|D}t|dkrt|jdr>|jj}nzt|jdrXt|jj}n`t|jd r~t|jjdr~|jjj}n:t|jd rt|jjdrt|jjj}ntd |d d|| rdt d ||dd | d|d d| |dtj dkrFdj }td|ddS)z This stage is run after we have the model and know num_training_steps. Now we can complete the configuration process. z$zero_optimization.reduce_bucket_sizez-zero_optimization.stage3_prefetch_bucket_sizez4zero_optimization.stage3_param_persistence_thresholdcsg|]}|r|qSr)r,).0xr&rr s zDHfTrainerDeepSpeedConfig.trainer_config_finalize..r hidden_size hidden_sizes text_configzThe model's config file has neither `hidden_size` nor `hidden_sizes` entry, therefore it's not possible to automatically fill out the following `auto` entries in the DeepSpeed config file: zb. You can fix that by replacing `auto` values for these keys with an integer value of your choice.g? z scheduler.params.total_num_stepsznum_training_steps (calculated)z!scheduler.params.warmup_num_stepsZ warmup_steps z]Please correct the following DeepSpeed config values that mismatch TrainingArguments values: zF The easiest method is to set these DeepSpeed config values to 'auto'.N)lenhasattrr1rEmaxrFrGr%r=is_zero3intr2Zget_warmup_stepsr$join)rr@modelnum_training_stepsZhidden_size_based_keysZhidden_size_auto_keysrEr$rr&rtrainer_config_finalizesT         z0HfTrainerDeepSpeedConfig.trainer_config_finalize)NT)F) rrrrrr'r,r2rr=rArRr rrrrr!Ts    Jr!cCst|adSr")weakrefref_hf_deepspeed_config_weak_ref)Zhf_deepspeed_config_objrrrrsrcCsdadSr")rUrrrrunset_hf_deepspeed_config srVcCs$tdk rtdk rtSdSdS)NF)rUrMrrrris_deepspeed_zero3_enabled&s rWcCs"tdk rtdk rtjSdSdSr")rUr1rrrrdeepspeed_config-srXc sddlm}m}|j}d}d|kr<|jr0td||d}n"|rNtd }d|d <d} d |krt||} n4t ||rfd d } ||| d } nj |d} || fS)zY A convenience wrapper that deals with optimizer and lr scheduler configuration. r) DummyOptimDummySchedulerN optimizerz|--adafactor was passed, but also found `optimizer` configured in the DeepSpeed config. Only one optimizer can be configured.)paramszDetected ZeRO Offload and non-DeepSpeed optimizers: This combination should work as long as the custom optimizer has both CPU and GPU implementation (except LAMB)TZzero_allow_untested_optimizerZ schedulercs"t}d|_|j|d}|S)NrQr[)copy lr_schedulercreate_scheduler)r[Z trainer_copyr_rQtrainerrr_lr_scheduler_callable_s z5deepspeed_optim_sched.._lr_scheduler_callable)Zlr_scheduler_callabler]) Zaccelerate.utilsrYrZr1Z adafactorr%Z is_offloadloggerinfoZcreate_optimizer isinstancer`) rbhf_deepspeed_configr@rQmodel_parametersrYrZr1r[r_rcrrardeepspeed_optim_sched4s.     riFc Csddlm}|j}|j}|jjjj}||||| | |rv| sTt d| d| dd\}}d} n0d|_ttdd |} t||||| \}}||fS) a Init DeepSpeed, after updating the DeepSpeed configuration with any relevant Trainer's args. If `resume_from_checkpoint` was passed then an attempt to resume from a previously saved checkpoint will be made. Args: trainer: Trainer object num_training_steps: per single gpu resume_from_checkpoint: path to a checkpoint if to resume from after normal DeepSpeedEngine load inference: launch in inference mode (no optimizer and no lr scheduler) auto_find_batch_size: whether to ignore the `train_micro_batch_size_per_gpu` argument as it's being set automatically by the auto batch size finder Returns: optimizer, lr_scheduler We may use `deepspeed_init` more than once during the life of Trainer, when we do - it's a temp hack based on: https://github.com/microsoft/DeepSpeed/issues/1394#issuecomment-937405374 until Deepspeed fixes a bug where it can't resume from a checkpoint after it did some stepping https://github.com/microsoft/DeepSpeed/issues/1612 r)rdzMZeRO inference only makes sense with ZeRO Stage 3 - please adjust your configr[r_)NNNcSs|jSr")Z requires_grad)prrrz deepspeed_init..)Zdeepspeed.utilsrdrPr@Z acceleratorstateZdeepspeed_pluginZ hf_ds_configrRsetLevelZget_process_log_levelrMr%Zdel_config_sub_treer[listfilter parametersri) rbrQZ inferenceZ ds_loggerrPr@rgr[r_rhrrrdeepspeed_initqs.    rrTcCsxddl}t||d}t|dkrftd||j||ddd\}}|dkrttd|ntd|dS)Nrz /global_step*zAttempting to resume from T)load_module_strictZload_optimizer_statesZload_lr_scheduler_statesz-[deepspeed] failed to resume from checkpoint z!Can't find a valid checkpoint at )globsortedrJrdreZload_checkpointr%)Zdeepspeed_engineZcheckpoint_pathrsrtZdeepspeed_checkpoint_dirsZ load_pathrrrrdeepspeed_load_checkpoints  rv)F)T)$rr^importlib.metadatarr importlib.utilr rS functoolsrZdependency_versions_checkrutilsrrrrr?Z get_loggerrrdrZaccelerate.utils.deepspeedrZDeepSpeedConfigbuiltinsrr!rUrrVrWrXrirrrvrrrrs2      B= 9