U di@s^dZddlZddlZddlZddlmZddlmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZd d lmZd d lmZd d l m!Z!d d l"m#Z#d dl$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.d dl/m0Z0m1Z1m2Z2e23e4Z5dZ6dZ7dZ8dZ9dZ:e%d7Z%eeeedZ;e6e9dZd?d@Z6e!eed"dAdBZ7dPe#ee!efe e edCdDdEZ8dQe#ee9j:feeee eeeedFdGdHZ;dRdIdJZ<Z=S)SPreTrainedTokenizerFastaQ Base class for all fast tokenizers (wrapping HuggingFace tokenizers library). Inherits from [`~tokenization_utils_base.PreTrainedTokenizerBase`]. Handles all the shared methods for tokenization and special tokens, as well as methods for downloading/caching/loading pretrained tokenizers, as well as adding tokens to the vocabulary. This class also contains the added tokens in a unified way on top of all tokenizers so we don't have to handle the specific vocabulary augmentation methods of the various underlying dictionary structures (BPE, sentencepiece...). Nslow_tokenizer_classcsz|dd}|dd}|dd}|dd}|dd}|di}|rf|dkrf|jdkrftd|dk rzt|} n|dk r|st|} n|rt|} n|dk rt| d } | d d } | d } | d } t | | \} }| | t |dkrn| |nh|jdk r2|dk r2|j||}t|} n<|sf| d d|_ | dg|_t|dd} d}ntd| |_|dk r| |jd|_|jj}|dk r|jjf||d|d|d|d|d|d|d|dn |j|jj}|dk rl|jjf||d|d|d|d|d|d|d|d|d|dtjf||j|j_dd |jDfd!d"t|d#d$d%Dt|j !d&d"Dfd'd"|j"D7t dkrvg}|j#}D]\}t$|t%r,|j&p6t'||kn t'||k}t$|t'rRt%||d(}n||_&|(|q|rv|)|dS))Ntokenizer_objectZ__slow_tokenizer gguf_filer& from_slowFadded_tokens_decoderzCannot instantiate this tokenizer from a slow version. If it's based on sentencepiece, make sure you have sentencepiece installed.r'configZ model_type tokenizertokenizer_configradditional_special_tokensT)Z from_tiktokena9Couldn't instantiate the backend tokenizer from one of: (1) a `tokenizers` library serialization file, (2) a slow tokenizer instance to convert or (3) an equivalent slow tokenizer class to instantiate and convert. You need to have sentencepiece or tiktoken installed to convert a slow tokenizer to a fast one. max_lengthtruncation_side directionstridetruncation_strategystrategy pad_tokenpad_token_type_id pad_type_id padding_sidelengthpad_to_multiple_ofcSsh|]}tt|qShashrepr.0tokenr>r>H/tmp/pip-unpacked-wheel-bm_b0l5e/transformers/tokenization_utils_fast.py sz3PreTrainedTokenizerFast.__init__..cs$g|]\}}tt|kr|qSr>r?)rCindexrD)added_tokens_decoder_hashr>rE sz4PreTrainedTokenizerFast.__init__..cSs|dSNrr>)xr>r>rEz2PreTrainedTokenizerFast.__init__..keycSsg|] }t|qSr>)strrBr>r>rErIscs g|]}|kr|kr|qSr>r>rB)encoder tokens_to_addr>rErIs)special)*popr) ValueErrorcopydeepcopy TokenizerFast from_filerrgetrupdatelenr'r1 _tokenizer init_kwargs_decode_use_source_tokenizer truncationenable_truncation setdefault no_truncationpaddingenable_paddingsuper__init__split_special_tokensencode_special_tokensr-sorteditemslistadded_tokens_encoderkeysZall_special_tokens_extendedZall_special_tokens isinstancerrSrPappend add_tokens)selfargskwargsr*Zslow_tokenizerr+Zfast_tokenizer_filer,r-Zfast_tokenizerZ gguf_param architectureZtokenizer_dictr0Zadditional_kwargs _truncation_paddingtokensspecial_tokensrDZ is_special __class__)rHrQrRrErgbs                          z PreTrainedTokenizerFast.__init__)returncCsdS)NTr>rrr>r>rEis_fastszPreTrainedTokenizerFast.is_fastcCsdS)z `bool`: Whether or not the slow tokenizer can be saved. Usually for sentencepiece based slow tokenizer, this can only be `True` if the original `"sentencepiece.model"` was not deleted. Tr>r}r>r>rEcan_save_slow_tokenizersz/PreTrainedTokenizerFast.can_save_slow_tokenizercCs|jjddS)zP `int`: Size of the base vocabulary (without the added tokens). FZwith_added_tokensr]Zget_vocab_sizer}r>r>rE vocab_sizesz"PreTrainedTokenizerFast.vocab_sizecCs|jjddS)NTr)r] get_vocabr}r>r>rErsz!PreTrainedTokenizerFast.get_vocabcCs|SN)rr}r>r>rEvocabszPreTrainedTokenizerFast.vocabcCs ddt|jdddDS)z Returns the sorted mapping from string to index. The added tokens encoder is cached for performance optimisation in `self._added_tokens_encoder` for the slow tokenizers. cSsi|]\}}|j|qSr>contentrCvkr>r>rE sz@PreTrainedTokenizerFast.added_tokens_encoder..cSs|dSrJr>itemr>r>rErLrMz>PreTrainedTokenizerFast.added_tokens_encoder..rNrjr-rkr}r>r>rErmsz,PreTrainedTokenizerFast.added_tokens_encodercCs |jS)z Returns the added tokens in the vocabulary as a dictionary of index to AddedToken. Returns: `Dict[str, int]`: The added tokens. )r]Zget_added_tokens_decoderr}r>r>rEr-sz,PreTrainedTokenizerFast.added_tokens_decodercCs ddt|jdddDS)z Returns the added tokens in the vocabulary as a dictionary of token to index. Returns: `Dict[str, int]`: The added tokens. cSsi|]\}}|j|qSr>rrr>r>rErsz;PreTrainedTokenizerFast.get_added_vocab..cSs|dSrJr>rr>r>rErLrMz9PreTrainedTokenizerFast.get_added_vocab..rNrr}r>r>rEget_added_vocabsz'PreTrainedTokenizerFast.get_added_vocabcCs|jjddS)zD Size of the full vocabulary with the added tokens. Trrr}r>r>rE__len__szPreTrainedTokenizerFast.__len__cCs|jS)zc `tokenizers.implementations.BaseTokenizer`: The Rust tokenizer used as a backend. )r]r}r>r>rEbackend_tokenizer sz)PreTrainedTokenizerFast.backend_tokenizercCs|jjS)zU `tokenizers.decoders.Decoder`: The Rust decoder for this tokenizer. )r]decoderr}r>r>rErszPreTrainedTokenizerFast.decoderFT) encodingreturn_token_type_idsreturn_attention_maskreturn_overflowing_tokensreturn_special_tokens_maskreturn_offsets_mapping return_lengthverboser|c Cs|dkrd|jk}|dkr$d|jk}|r@|jdk r@|g|j} n|g} tt} | D]|} | d| j|rz| d| j|r| d| j|r| d| j|r| d| j |rR| dt | jqR| | fS)a Convert the encoding representation (from low-level HuggingFace tokenizer output) to a python Dict and a list of encodings, take care of building a batch from overflowing tokens. Overflowing tokens are converted to additional examples (like batches) so the output values of the dict are lists (overflows) of lists (tokens). Output shape: (overflows, sequence length) NZtoken_type_idsattention_mask input_idsspecial_tokens_maskZoffset_mappingr<) Zmodel_input_namesZ overflowingrrlrpidsZtype_idsrroffsetsr\) rrrrrrrrrr encodingsZ encoding_dicter>r>rE_convert_encodings*  z)PreTrainedTokenizerFast._convert_encoding)rxr|cs2|dkr dSt|tr |Sfdd|DS)aT Converts a token string (or a sequence of tokens) in a single integer id (or a sequence of ids), using the vocabulary. Args: tokens (`str` or `List[str]`): One or several token(s) to convert to token id(s). Returns: `int` or `List[int]`: The token id or list of token ids. Ncsg|]}|qSr>)#_convert_token_to_id_with_added_vocrBr}r>rErIZszAPreTrainedTokenizerFast.convert_tokens_to_ids..)rorPrrrrxr>r}rEconvert_tokens_to_idsIs   z-PreTrainedTokenizerFast.convert_tokens_to_ids)rDr|cCs|j|}|dkr|jS|Sr)r] token_to_idZ unk_token_id)rrrDrGr>r>rEr\s z;PreTrainedTokenizerFast._convert_token_to_id_with_added_voc)rGr|cCs|jt|Sr)r] id_to_tokenint)rrrGr>r>rE_convert_id_to_tokenbsz,PreTrainedTokenizerFast._convert_id_to_token) new_tokensr|cCs|r|j|S|j|Sr)r]add_special_tokensrq)rrrryr>r>rE _add_tokenses z#PreTrainedTokenizerFast._add_tokens)pairr|cCs |j|S)aG Returns the number of added tokens when encoding a sequence with special tokens. This encodes a dummy input and checks the number of added tokens, and is therefore not efficient. Do not put this inside your training loop. Args: pair (`bool`, *optional*, defaults to `False`): Whether the number of added tokens should be computed in the case of a sequence pair or a single sequence. Returns: `int`: Number of special tokens added to sequences. )r]num_special_tokens_to_add)rrrr>r>rErksz1PreTrainedTokenizerFast.num_special_tokens_to_add)rskip_special_tokensr|cCsRt|tr|j|Sg}|D].}t|}|r:||jkr:q||j|q|S)a Converts a single index or a sequence of indices in a token or a sequence of tokens, using the vocabulary and added tokens. Args: ids (`int` or `List[int]`): The token id (or token ids) to convert to tokens. skip_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not to remove special tokens in the decoding. Returns: `str` or `List[str]`: The decoded token(s). )rorr]rZall_special_idsrp)rrrrrxrGr>r>rEconvert_ids_to_tokenss  z-PreTrainedTokenizerFast.convert_ids_to_tokens)textrrr|cKs|jf|||d|S)N)r text_pairr)Z encode_plusrx)rrrrrrtr>r>rEtokenizesz PreTrainedTokenizerFast.tokenizepadding_strategyr6r2r5r=r;c s|jj|jj}|tjkr.dk rv|jnH|||j|jd}dkrNd} nfdd|D} | |krv|jjf||t j kr|dk r|j nN|t j kr|nd} | |dk r|n|j |j|j|j|d}||kr|jjf|dS)a Define the truncation and the padding strategies for fast tokenizers (provided by HuggingFace tokenizers library) and restore the tokenizer settings afterwards. The provided tokenizer has no padding / truncation strategy before the managed section. If your tokenizer set a padding / truncation strategy before, then it will be reset to no padding / truncation when exiting the managed section. Args: padding_strategy ([`~utils.PaddingStrategy`]): The kind of padding that will be applied to the input truncation_strategy ([`~tokenization_utils_base.TruncationStrategy`]): The kind of truncation that will be applied to the input max_length (`int`): The maximum size of a sequence. stride (`int`): The stride to use when handling overflow. pad_to_multiple_of (`int`, *optional*): If set will pad the sequence to a multiple of the provided value. This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability `>= 7.5` (Volta). padding_side (`str`, *optional*): The side on which the model should have padding applied. Should be selected between ['right', 'left']. Default value is picked from the class attribute of the same name. N)r2r5r7r4csi|]}||dqSrrZ)rCrrvr>rErszFPreTrainedTokenizerFast.set_truncation_and_padding..)r<r4Zpad_idr8r:r=)r]r`rdrDO_NOT_TRUNCATErcvaluer3rar DO_NOT_PADZ no_paddingZ MAX_LENGTHr;Z pad_token_idr8r9re) rrrr6r2r5r=r;rwtargetcurrentr<r>rrEset_truncation_and_paddings8!     z2PreTrainedTokenizerFast.set_truncation_and_paddingr)batch_text_or_text_pairsrrr6r2r5is_split_into_wordsr=r;return_tensorsrrrrrrrrhr|cs(t|ttfs"tdt|dj|||||| djj|krL|j_jj|||d}fdd|D}i}|dd D]fdd|D}||<qd d|D}rg}t |D]"\}\}}||gt |d 7}q||d <|d D]} ||qt ||| d S) Nz:batch_text_or_text_pairs has to be a list or a tuple (got )r)rZis_pretokenizedc s&g|]}j|dqS))rrrrrrrr)r)rCr)rrrrrrrrrr>rErIs z>PreTrainedTokenizerFast._batch_encode_plus..rcs"g|]\}}|D]}|qqSr>r>)rCr_rrNr>rErI3s cSsg|]\}}|D]}|qqSr>r>)rCrrrr>r>rErI5sroverflow_to_sample_mapping)Z tensor_type)rotuplerl TypeErrortyperr]riZ encode_batchrn enumerater\&_eventual_warn_about_too_long_sequencer)rrrrrr6r2r5rr=r;rrrrrrrrrhrZtokens_and_encodingsZsanitized_tokensstackZsanitized_encodingsritoksrrr>) rOrrrrrrrrrrE_batch_encode_plussF     z*PreTrainedTokenizerFast._batch_encode_plus)rrrrr6r2r5rr=r;rrrrrrrrrhr|cKs|r||fgn|g}|j|f||||||| | | | | ||||||d|}| dkrp|sptdd|D|j}||d|||S)N)rrrr6r2r5r=r;rrrrrrrrrhcSs8i|]0\}}|t|dkr0t|dtr0|dn|qS)r)r\rorl)rCrOrr>r>rErusz8PreTrainedTokenizerFast._encode_plus..r)rrrkrr)rrrrrrr6r2r5rr=r;rrrrrrrrrhrtZ batched_inputZbatched_outputr>r>rE _encode_plusCsB z$PreTrainedTokenizerFast._encode_pluscCs|jj|Sr)rrdecoderr>r>rEconvert_tokens_to_stringsz0PreTrainedTokenizerFast.convert_tokens_to_string) token_idsrclean_up_tokenization_spacesr|cKsZ|dd|_t|tr|g}|jj||d}|dk r:|n|j}|rR||}|S|SdS)NZuse_source_tokenizerF)r)rTr_rorr]rrZclean_up_tokenization)rrrrrrtrZ clean_textr>r>rE_decodes  zPreTrainedTokenizerFast._decode)save_directory file_names legacy_formatfilename_prefixr|c s&t|}jdkr"|dkr"td|dks2|dko@jdk o@j}|dkpP|dk}|rtj||rj|dndt}fddj D}|rt |d d d &} t j |d ddd d} | | W5QRXj||d} || |f}|r"tj||r|dndt} j| || f}|S)z Save a tokenizer using the slow-tokenizer/legacy format: vocabulary + added tokens as well as in a unique JSON file containing {config + vocab + added-tokens}. NTzYour tokenizer does not have a legacy version defined and therefore cannot register this version. You might consider leaving the legacy_format at `None` or setting it to `False`.F-cs i|]\}}|jkr||qSr>)r)rCtokrGr}r>rErs z.wzutf-8)r)indent sort_keys ensure_ascii )r)rPr)rUrospathjoinADDED_TOKENS_FILErmrkopenjsondumpswriteZsave_vocabularyTOKENIZER_FILErsave) rrrrrrZ save_slowZ save_fastZadded_tokens_fileZ added_vocabfZout_strZ vocab_filesr&r>r}rE_save_pretraineds>   z(PreTrainedTokenizerFast._save_pretrainedc sBt|j}|d}|d} d} |dddkrRi|dd<g|dd<n|ddd kr|dd dk r|dd } |dd| d } dk r| kr| } d |dd <| d gg|dd<n6|ddd kri|dd<ntd|ddddk rBd|dkrB|ddkrB|dd|dd<tt|g} |D]r} | dd}| dd}|ddd kr|sqZdk r| dkr| d| d<| t f| qZ|dk r| ||dddkr d|kr |dddk r |dd|d<|dddkr^d|kr^|dddk r^|dd|d<|ddd kr| dk r| |d<|ddk r|dddks|dddkrd|dkrt dd|ddDrt j|d<t|dd}|f|| d|}j|||d| dk rRt}d| kr| dD]}| d|d }dk rzfd!d"|D}|| d|d <|D]"}|}|dkrtd#qfd$d"|D| d|d%<qJd&D]`}|| kr| |\}}dk r|kr|}|}|dkr*td#||g| |<q| |d<tt||j}tj}|d'|D]}t|d(|dk rtt||}dk r|kr|}t|d(|}t|t rt ||j|j|j|jd)d*||<n|||<qt|j}|dk r| |t|d kr.||d'<|j fd+i|S),uf Trains a tokenizer on a new corpus with the same defaults (in terms of special tokens or tokenization pipeline) as the current one. Args: text_iterator (generator of `List[str]`): The training corpus. Should be a generator of batches of texts, for instance a list of lists of texts if you have everything in memory. vocab_size (`int`): The size of the vocabulary you want for your tokenizer. length (`int`, *optional*): The total number of sequences in the iterator. This is used to provide meaningful progress tracking new_special_tokens (list of `str` or `AddedToken`, *optional*): A list of new special tokens to add to the tokenizer you are training. special_tokens_map (`Dict[str, str]`, *optional*): If you want to rename some of the special tokens this tokenizer uses, pass along a mapping old special token name to new special token name in this argument. kwargs (`Dict[str, Any]`, *optional*): Additional keyword arguments passed along to the trainer from the 🤗 Tokenizers library. Returns: [`PreTrainedTokenizerFast`]: A new tokenizer of the same type as the original one, trained on `text_iterator`. added_tokenspost_processorNmodelrr"rZmergesr#unk_idrg)r$r%z;This method does not support this type of tokenizer (found z-) only BPE, Unigram, WordLevel and WordPiece. unk_tokenrSidrZcontinuing_subword_prefixZend_of_word_suffixZ pre_tokenizer ByteLevelSequenceZ pretokenizerscss|]}|ddkVqdS)rrNr>)rCZ pretokenizerr>r>rE 4szBPreTrainedTokenizerFast.train_new_from_iterator..Zinitial_alphabet)rry)r<trainerryrxcsg|]}||qSr>rrB)special_tokens_mapr>rErIFszCPreTrainedTokenizerFast.train_new_from_iterator..zQAttempted to set a token in the post processor that does not exist in the mappingcsg|]}|qSr>)rrB)r/r>rErIOsr)clssepr1rT) single_wordlstriprstrip normalizedrSr*)!rloadsr]Zto_strrTrUrXZfrom_strrrprextendanypre_tokenizers_fastralphabetMODEL_TO_TRAINER_MAPPINGZtrain_from_iteratorrr^rVrZSPECIAL_TOKENS_ATTRIBUTESremovegetattrrorrrrr1r\r{)rrZ text_iteratorrr<Znew_special_tokensrrtZtokenizer_jsonrrrrryZ added_tokenrSrZ trainer_classrZtrained_tokenizer_jsonrOrxrDZtoken_idZ special_tokenZspecial_tokens_listZspecial_token_fullr1r>)rr/rEtrain_new_from_iterators"                    "            z/PreTrainedTokenizerFast.train_new_from_iterator)NNFFFFT)F)F)F)NF)FN)NN)NNN)>__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESZvocab_files_namesr)r__annotations__rgpropertyboolr~rrrrrPrrrmrr-rrrXr DecoderFastr EncodingFastrrrrrrrrrrrrrrrrrrrrrrrrrrrrPathLikerr __classcell__r>r>rzrEr(QsD    o    /(  P ^  =   5r()>r rVrr collectionsrtypingrrrrrrZtokenizers.pre_tokenizersZpre_tokenizersrZ tokenizersr rr rXZtokenizers.decodersr rZtokenizers.trainersr r rrrZintegrations.ggmlrZmodeling_gguf_pytorch_utilsrZtokenization_utilsrZtokenization_utils_baserrrrrrrrrrutilsrr r!Z get_loggerrloggerrZSPECIAL_TOKENS_MAP_FILEZTOKENIZER_CONFIG_FILEZTIKTOKEN_VOCAB_FILErrr r(r>r>r>rEs>          0