U dip@sddlZddlmZmZddlmZddlmZmZe e Z erLddl Z d(eeedee edefdd d Zd)eeedee edefdd d Zd*eeedee edefdd dZd+edee edefdddZd,edee edefdddZd-edee edefdddZeeeeeedZd.eeeeeeedddZd/eeedddZd0eeedddZd1eeedddZd2eeedd d!Zd3eeedd"d#Zd4eeedd$d%ZeeeeeedZd5eeedd&d'Z dS)6N)OptionalTuple)PretrainedConfig)is_torch_availableloggingz torch.devicez torch.Tensor)configdeviceseq_lenreturnc Ks|dk r(t|dkr(td|d|t|dkrF|d}|d}nB|dk r|j}t|drd|jnd}t|d |j|j}t||}d}d|t j d|d t j d  ||} | |fS) a Computes the inverse frequencies according to the original RoPE implementation Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_default_rope_parameters`, got `rope_kwargs`= and `config`=basedimpartial_rotary_factor?head_dimdtype)len ValueError rope_thetahasattrrgetattr hidden_sizenum_attention_headsinttorcharangeint64floatto) rr r rope_kwargsr rrrattention_factorinv_freqr%D/tmp/pip-unpacked-wheel-bm_b0l5e/transformers/modeling_rope_utils.py _compute_default_rope_parameterss   *r'cKst|dk r(t|dkr(td|d|t|dkr>|d}n|dk rP|jd}t|||f|\}}||}||fS)a Computes the inverse frequencies with linear scaling. Credits to the Reddit user /u/kaiokendev Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_linear_scaling_rope_parameters`, got `rope_kwargs`=r factor)rr rope_scalingr')rr r r"r(r$r#r%r%r&'_compute_linear_scaling_rope_parametersGs   r*c Ks|dk r(t|dkr(td|d|t|dkrV|d}|d}|d}|d}nR|dk r|j}t|d rt|jnd }t|d |j|j} t| |}|j }|j d}d } |dk r||kr|n|}|||||d ||d }d |t j d|d t j d||} | | fS)a4 Computes the inverse frequencies with NTK scaling. Credits to the Reddit users /u/bloc97 and /u/emozilla Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length, used to update the dynamic RoPE at inference time. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin (unused in this type of RoPE). NrzUnexpected arguments: `**rope_kwargs` and `config` are mutually exclusive in `_compute_dynamic_ntk_parameters`, got `rope_kwargs`=r r rmax_position_embeddingsr(rrrrrr)rrrrrrrrrr+r)rrrr r!) rr r r"r rr+r(rrr#r$r%r%r&_compute_dynamic_ntk_parametersps*    $*r,c sHt|dkrtd||j}t|dr0|jnd}t|d|j|j}t||}|j }|j d} |j d} | dkrd t | d} |j d pd } |j d pd } ddfdd} dd}|td|d||}d|}d| |}| | | |||\}}d ||||d|}|d |||}|| fS)a Computes the inverse frequencies with NTK scaling. Please refer to the [original paper](https://arxiv.org/abs/2309.00071) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rzYUnexpected arguments: `**rope_kwargs` should be unset in `_compute_yarn_parameters`, got rrrr(r#Ng? beta_fast beta_slowrcSs*|t||dtjdt|S)zPInverse dimension formula to find the dimension based on the number of rotationsr)mathlogpi)Z num_rotationsrr r+r%r%r&find_correction_dimsz5_compute_yarn_parameters..find_correction_dimcs@t||||}t||||}t|dt||dfS)z.Find dimension range bounds based on rotationsrr)r0floorceilmaxmin)Zlow_rotZhigh_rotrr r+lowhighr3r%r&find_correction_rangesz7_compute_yarn_parameters..find_correction_rangecSs>||kr|d7}tj|tjd|||}t|dd}|S)NgMbP?rrr)rrfloat32clamp)r7r6rZ linear_funcZ ramp_funcr%r%r&linear_ramp_factors z4_compute_yarn_parameters..linear_ramp_factorr)rrrrrrrrrr+r)getr0r1rrr r!)rr r r"r rrrr+r(r#r-r/r;r>Z pos_freqsZinv_freq_extrapolationZinv_freq_interpolationr8r9Zinv_freq_extrapolation_factorr$r%r:r&_compute_yarn_parameterss8        r@cKs^t|dkrtd||j}t|dr0|jnd}t|d|j|j}t||}|j d}|j d} |j d} |j d } t|d r|r||j kr|j } n|j } |j } | | } n|j } | | } | d kr| dkrd} nt d t | t | } | | krtj|tj|d }ntj| tj|d }tjd|dtj|d |}d|||}|| fS)a  Computes the inverse frequencies with LongRoPE scaling. Please refer to the [original implementation](https://github.com/microsoft/LongRoPE) Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. rz]Unexpected arguments: `**rope_kwargs` should be unset in `_compute_longrope_parameters`, got rrr long_factor short_factorr(r# original_max_position_embeddingsNr)rr r)rrrrrrrrrr)r?rCr+r0sqrtr1rZtensorr<rrr )rr r r"r rrrrArBr(r#Z expanded_max_position_embeddingsr+Z ext_factorsZinv_freq_shaper$r%r%r&_compute_longrope_parameterss<         rEcKst|||f|\}}|jd}|jd}|jd}|jd} | |} | |} dtj|} t| | k|||} | | |||}d|| ||| }| | k| | k}t||| } | |fS)a Computes the inverse frequencies for llama 3.1. Args: config ([`~transformers.PretrainedConfig`]): The model configuration. device (`torch.device`): The device to use for initialization of the inverse frequencies. seq_len (`int`, *optional*): The current sequence length. Unused for this type of RoPE. rope_kwargs (`Dict`, *optional*): BC compatibility with the previous RoPE class instantiation, will be removed in v4.45. Returns: Tuple of (`torch.Tensor`, `float`), containing the inverse frequencies for the RoPE embeddings and the post-processing scaling factor applied to the computed cos/sin. r(low_freq_factorhigh_freq_factorrCrr)r'r)r0r2rwhere)rr r r"r$r#r(rFrGZold_context_lenZlow_freq_wavelenZhigh_freq_wavelenZwavelenZinv_freq_llamaZ smooth_factorZsmoothed_inv_freqZis_medium_freqr%r%r&_compute_llama3_parameters6s    rI)defaultZlinearZdynamicZyarnZlongropeZllama3) rope_type received_keys required_keys optional_keys ignore_keyscCsd|kr|dh8}|d|dk r,||8}||}|rLtd|d||dk rb|||}n||}|rtd|d|dS)zYCompare the received keys in `config.rope_scaling` against the expected and optional keystyperKNz9Missing required keys in `rope_scaling` for 'rope_type'='z': z5Unrecognized keys in `rope_scaling` for 'rope_type'=')addKeyErrorloggerwarning)rKrLrMrNrOZ missing_keysZ unused_keysr%r%r&_check_received_keysns   rU)rrOcCs@|j}|d|dd}dh}t|}t||||ddS)NrKrPrO)r)r?setkeysrU)rrOr)rKrMrLr%r%r&!_validate_default_rope_parameterss  rYcCst|j}|d|dd}ddh}t|}t||||d|d}|dks`t|tr`|dkrptd|dS)NrKrPr(rVr8`rope_scaling`'s factor field must be a float >= 1, got r)r?rWrXrU isinstancer rSrT)rrOr)rKrMrLr(r%r%r&(_validate_linear_scaling_rope_parameterss r]cCs||j}|d|dd}ddh}dh}t|}t|||||d|d}|dksht|trh|dkrxtd|dS)NrKrPr(rCrVrrZr[)rrOr)rKrMrNrLr(r%r%r&)_validate_dynamic_scaling_rope_parameterss r^c Cs>|j}|d|dd}ddh}dddh}t|}t|||||d|d}|dkslt|trl|dkr|td ||d}|dk rt|tr|d krtd ||d} | dk rt| tstd | |d} | dk r t| ts td | | pd| pdkr:td| d| ddS)NrKrPr(r#r-r/rVrrZrL`rope_scaling`'s attention_factor field must be a float greater than 0, got z6`rope_scaling`'s beta_fast field must be a float, got z6`rope_scaling`'s beta_slow field must be a float, got r.rzO`rope_scaling`'s beta_fast field must be greater than beta_slow, got beta_fast=z( (defaults to 32 if None) and beta_slow=z (defaults to 1 if None)r[) rrOr)rKrMrNrLr(r#r-r/r%r%r&_validate_yarn_parameterss0     r`cCs|j}|d|dd}dddh}dddh}t|}t|||||dt|d r\|jnd }t|d |j|j }t ||} |d} t | t st d d | Drtd| t| | dkstd| ddt| |d} t | t st dd | Drtd| t| | dksNtd| ddt| t|drftdn|d} | dkrtdn&t | tr| d krtd| |d} | dk rt | tr| dkrtd| dS)NrKrPrBrAr#r(rCrVrrrcss|]}t|ttfVqdSNr\rr .0xr%r%r& sz0_validate_longrope_parameters..zC`rope_scaling`'s short_factor field must be a list of numbers, got rz5`rope_scaling`'s short_factor field must have length z, got css|]}t|ttfVqdSrarbrcr%r%r&rfszB`rope_scaling`'s long_factor field must be a list of numbers, got z4`rope_scaling`'s long_factor field must have length aYThis model has set a `original_max_position_embeddings` field, to be used together with `max_position_embeddings` to determine a scaling factor. Please set the `factor` field of `rope_scaling`with this ratio instead -- we recommend the use of this field over `original_max_position_embeddings`, as it is compatible with most model architectures.z1Missing required keys in `rope_scaling`: 'factor'rZgr_)r)r?rWrXrUrrrrrrr\listallrSrTrZ warning_oncer )rrOr)rKrMrNrLrrrrBrAr(r#r%r%r&_validate_longrope_parameterssD             ric Cs>|j}|d|dd}dddddh}t|}t||||d|d}|dksft|trf|dkrvtd ||d}|d}|dkst|tstd ||dkst|tstd |||krtd |d ||d} | dkst| t std| | |j kr:td| d|j dS)NrKrPr(rCrFrGrVrrZz<`rope_scaling`'s low_freq_factor field must be a float, got z=`rope_scaling`'s high_freq_factor field must be a float, got zc`rope_scaling`'s high_freq_factor field must be greater than low_freq_factor, got high_freq_factor=z and low_freq_factor=zP`rope_scaling`'s original_max_position_embeddings field must be an integer, got zg`rope_scaling`'s original_max_position_embeddings field must be less than max_position_embeddings, got z and max_position_embeddings=) r)r?rWrXrUr\r rSrTrr+) rrOr)rKrMrLr(rFrGrCr%r%r&_validate_llama3_parameterss6  rjcCsbt|dd}|dkrdS|d|dd}t|}|dk rL|||dntd|ddS) zO Validate the RoPE config arguments, given a `PretrainedConfig` object r)NrKrPrJrVzTMissing validation function mapping in `ROPE_VALIDATION_FUNCTIONS` for 'rope_type'='')rr?ROPE_VALIDATION_FUNCTIONSrSrT)rrOr)rKZ validation_fnr%r%r&rope_config_validation-s   rm)NNN)NNN)NNN)N)N)N)NN)N)N)N)N)N)N)N)!r0typingrrZconfiguration_utilsrutilsrrZ get_logger__name__rSrrr r'r*r,r@rErIZROPE_INIT_FUNCTIONSstrrWrUrYr]r^r`rirjrlrmr%r%r%r&s   + * 4 P E /  2&