U didP @sdZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z ddl mZmZmZmZe rzddlZddlmZe eZd d d d gd ddgdddgddd d d d gdddgdddgdddd d d d gd ddgdddgddd d d d gd ddgdddgdddZdddddddddddddddddddddd ddd!Zd"d#Zd5ed$d%d&Zd'd(Zd)d*Zd+d,Zd-d.Z d/d0Z!d1d2Z"d3d4Z#dS)6z;AWQ (Activation aware Weight Quantization) integration fileN)version)ACT2FN)PreTrainedModel)is_auto_awq_availableis_ipex_availableis_torch_availablelogging)AwqBackendPackingMethod AwqConfigAWQLinearVersionExllamaVersionq_projk_projv_projo_proj gate_projup_proj down_projZinput_layernormZpost_attention_layernormZnormF) attentionmlp layernorm use_alibiZw1Zw3Zw2g.A)rrrr rope_theta)ZmistralZmixtralZllamaZllavaactZc_fc)rlayer_before_actZ dense_h_to_4hZfc_inZ gelu_impl)Z starcoder2ZRefinedWebModelZfalconZmptZgptjZgpt_neoxZ gpt_bigcodeZbloomc Csddlm}|tkr|S|D]n\}}t|d}t|d}||krt||rt|t|d}|j}t|} ||| |j |<t ||} q |S)Nr)ScaledActivationrr) Zawq.modules.actrAWQ_SCALES_MAPPINGSnamed_childrenhasattrgetattr out_featurestorchZones_modulesreplace_quantization_scales) model model_typernamemoduleZact_nameZlayer_before_act_namersizeZ scale_like_r+A/tmp/pip-unpacked-wheel-bm_b0l5e/transformers/integrations/awq.pyr$Ms     r$)returnc s|dkr g}|j}ts td|tjkr|jtjkrHddlm }|}n|jtj krfddl m }|}n|jtj kr|jdtjkrddlm} | }q|jdtjkrddlm} | }qtd |jdn.|jtjkrdd lm} | }ntd |jndd lm} | }|D]\} }dkr*g| t|tjr| |krt fd d|Ds|j!}|j"}||j#|j$|||j%dk |j&j'd|j(| <d}|j(| )dt*t+|,dkrt-||||d\}}.dq||fS)a Public method that recursively replaces the Linear layers of the given model with AWQ quantized layers. `accelerate` is needed to use this method. Returns the converted model and a boolean that indicates if the conversion has been successfull or not. During the module replacement, we also infer the backend to use through the `quantization_config` object. Args: model (`torch.nn.Module`): The model to convert, can be any `torch.nn.Module` instance. quantization_config (`AwqConfig`): The quantization config object that contains the quantization parameters. modules_to_not_convert (`list`, *optional*): A list of modules to not convert. If a module name is in the list (e.g. `lm_head`), it will not be converted. current_key_name (`list`, *optional*): A list that contains the current key name. This is used for recursion and should not be passed by the user. has_been_replaced (`bool`, *optional*): A boolean that indicates if the conversion has been successful or not. This is used for recursion and should not be passed by the user. NzAWQ (either `autoawq` or `llmawq`) is not available. Please install it with `pip install autoawq` or check out the installation guide in https://github.com/mit-han-lab/llm-awqr) WQLinear_GEMM) WQLinear_GEMVr)WQLinear_Exllama)WQLinear_ExllamaV2Unrecognized Exllama version:  WQLinear_IPEXzUnrecognized AWQ version: )WQLinearc3s|]}|dkVqdS).N)join).0keycurrent_key_namer+r, sz*replace_with_awq_linear..)w_bit group_size in_featuresr!biasdevTF)modules_to_not_convertr;quantization_confighas_been_replaced)/backendr ValueErrorr AUTOAWQrr ZGEMMZawq.modules.linear.gemmr.ZGEMVZawq.modules.linear.gemvr/ZEXLLAMAexllama_configr ONEawq.modules.linear.exllamar0TWOawq.modules.linear.exllamav2r1ZIPEXawq.modules.linear.gemm_ipexr4Zawq.quantize.qmoduler5rappend isinstancennZLinearanyr?r!bitsr>r@weightdevicer#Zrequires_grad_lenlistchildrenreplace_with_awq_linearpop)r%rBrCr;rDrFr. target_clsr/r0r1r4r5r'r(r?r!r*r+r:r,rY^sp               rYcCst|tstd|jj|jdk r8|j}|j|d<nj|jjt krt |jj}|jj dd}|j }|j }t |d|}||d<||d<||d<|j|d<ntd |S) af Returns the fusing mapping given the quantization config and the model Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`~transformers.quantization_config.AWQConfig`): The quantization configuration to use. z:The model should be an instance of `PreTrainedModel`, got N max_seq_lenTdecodernum_key_value_heads hidden_sizenum_attention_headsaFusing mapping not found either on the quantization config or the supported `AWQ_FUSED_MAPPINGS`. Please pass a `fused_mapping` argument in the `quantization_config` or raise an issue on transformers https://github.com/huggingface/transformers to add its support.)rPr TypeError __class____name__modules_to_fuseZfuse_max_seq_lenconfigr&AWQ_FUSED_MAPPINGSget_text_configr`rar rG)r%rCZcurrent_fused_mappingrfr`rar_r+r+r,get_modules_to_fuses&       ric sbt|trt|}|j}t||}t|dd}|tjkr`ddl m }ddl m }ddl m}ntdg|D]\}|dk rtfdd |Drqtt|d |||jd krt||d ||n td t||||} | rtddqttdkr^|D]F\}tfdd Drt|drt|jdrd|j_q|S)aJ Optionally fuse some modules in the model to speedup inference. Args: model (`~PreTrainedModel`): The model to fuse - note this model should have been converted into AWQ format beforehand. quantization_config (`Union[AwqConfig, dict]`): The quantization configuration to use. rBNr)QuantAttentionFused) QuantFusedMLP)FasterTransformerRMSNormz0Fusing is only supported for the AutoAWQ backendc3s|]}|kVqdSNr+)r8Zmodule_name_to_not_convert)r'r+r,r<sz#fuse_awq_modules..rZipexrz7The IPEX version AWQ does not support fuse mlp for now.r6c3s|]}kVqdSrmr+)r8Zfused_attention_parent_module)fused_attention_modules module_namer+r,r<(srf_attn_implementationZcustom)rPdictr from_dictrFrir r rHZawq.modules.fused.attnrjZawq.modules.fused.mlprkZawq.modules.fused.normrlrGZ named_modulesrR_fuse_awq_layernormr _fuse_awq_mlploggerinfo_fuse_awq_attention_layersrOsplitrVrrfrp) r%rCrFrerBrjrkrlr(Zattention_has_been_fusedr+)rnror'r,fuse_awq_modulessH           rycCsB|D]8}t||rt||}||j|j|jj|j|<~qdS)a Fuse the LayerNorm layers into a target class using autoawq Args: fuse_module_names (`List[str]`): The list of module names to fuse module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.FasterTransformerRMSNorm`): The `FasterTransformerRMSNorm` class as it only supports that class for now. N)rr rTZvariance_epsilontorUr#)fuse_module_namesr(r[roZ old_moduler+r+r,rs0s    rscCst|dkrdSt||drt||d}t||d}t||d}|jj}|jjdd} | j} t| } ||||| } | dd\} }| | }t ||| |~~~dS)a Fuse the MLP layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model current_module_name (`str`): The current submodule name fuse_module_names (`List[str]`): The list of module names to fuse. For the MLP layers it has to be an array of length 3 that consists of the 3 MLP layers in the order (gate (dense layer post-attention) / up / down layers) module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse target_cls (`~autoawq.QuantFusedMLP`): The `QuantFusedMLP` class as it only supports that class for now. rNrTr]r6) rVrr qweightrUrfrh hidden_actrrsplit get_submodulesetattrrz)r%current_module_namer{r(r[rrrprevious_devicerfr~Z activation_fnZ new_module parent_name child_nameparentr+r+r,rtGs  rtc CsLddlm}m}d}t|ddkr(|St||ddrHt||dd}t||rb|} d} n^t||rv|} d} nJtrt t j dt dkrddlm } t|| r| } d} nt d |jj} t||dd} t||dd }t||dd }|jd k r$tj|j| j|jgdd nd }| |j|j|j|j| j|j|jd k tt|j}tj|j| j|jg| d |_tj|j| j|jg| d |_tj|j| j|jg| d |_t||r|j|_||_||d|d|d||| |d|d|ddd }d|_| dd\}}|!|}t"|||#| ~~ ~~d}|S)a Fuse the Attention layers into a target class using autoawq Args: model (`~PreTrainedModel`): The input pretrained model module (`nn.Module`): The pytorch parent module that has layernorm modules to fuse modules_to_fuse (`List[str]`): The module fusing mapping. The dictionary has to contain a field `attention` with attention module names in the correct order: q, k, v, o layer current_module_name (`str`): The current submodule name target_cls (`~autoawq.QuantAttentionFused`): The `QuantAttentionFused` class as it only supports that class for now. r)r.r/Frr|Zautoawqz0.2.6r3z'Unsupported q_proj type: {type(q_proj)}rN)Zdimr`rar_r\rrg@)rrTr6)$Zawq.modules.linearr.r/rVrr rPrrparse importlibmetadatar4rGr}rUr@r"catr=r>r?r!nextiterZ state_dictvaluesZqzerosZscalesZ split_k_itersgetZis_hf_transformersrrrrz)r%r(rerr[r.r/Zmodule_has_been_fusedrZlinear_target_clsZcat_dimr4rrrrr@Z qkv_layerZfused_attention_layerrrrr+r+r,rwpsn  "  *     rwcCsl|dtjkr$ddlm}||}nD|dtjkrVddlm}|||d|dd}ntd|d|S) z Runs post init for Exllama layers which performs: - Weights unpacking, reordering and repacking - Devices scratch space allocation rr)exllama_post_init)exllamav2_post_init max_input_lenmax_batch_size)rrr2)r rJrKrrLrMrrG)r%rIrrr+r+r,post_init_awq_exllama_moduless   rcCsddlm}||}|S)zl Runs post init for IPEX layers which performs: - Weights packing, reordering and repacking r)ipex_post_init)rNr)r%rr+r+r,post_init_awq_ipex_moduless r)NNNF)$__doc__r packagingrZ activationsrZmodeling_utilsrutilsrrrr Zutils.quantization_configr r r r r"Ztorch.nnrQZ get_loggerrdrurgrr$boolrYriryrsrtrwrrr+r+r+r,sp           i)@)_