U di|B@sdZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZdd lmZd d lmZmZmZmZmZm Z m!Z!m"Z"d d l#m$Z$d d l%m&Z&m'Z'd dl(m)Z)erd dl*m+Z+e ,e-Z.e/dZ0dZ1dddZ2eGdddZ3e"d;ee3ee4e4e4ee4ee4eee4e4feee4eee5ee5ee4fd ddZ6GdddeZ7GdddeZ8de4dd-d.d/Z?ee3e>ee4d0d1d2Z@eee4e8d3d4d5ZAdee4e>eed6d7d8ZBdee4e>eed6d9d:ZCdS)=z.Git LFS related type definitions and utilitiesN) dataclass)ceil)getsize)Path) TYPE_CHECKINGBinaryIODictIterableListOptionalTuple TypedDict)unquote) constants)build_hf_headersfix_hf_endpoint_in_url get_sessionhf_raise_for_status http_backoffloggingtqdmvalidate_hf_hub_args) SliceFileObj)sha256 sha_fileobj)is_tqdm_disabled)CommitOperationAddz^[0-9a-f]{40}$zlfs-multipart-uploadzapplication/vnd.git-lfs+json)Acceptz Content-Typec@s`eZdZUdZeed<eed<eed<eedddZ eedd d Z ee d d d Z dS) UploadInfoad Dataclass holding required information to determine whether a blob should be uploaded to the hub using the LFS protocol or the regular protocol Args: sha256 (`bytes`): SHA256 hash of the blob size (`int`): Size in bytes of the blob sample (`bytes`): First 512 bytes of the blob rsizesample)pathc CsHt|}t|d }|ddd}t|}W5QRX||||dS)Nrbr rr!)rioopenpeekr)clsr"r filer!shar,7/tmp/pip-unpacked-wheel-ohruig0i/huggingface_hub/lfs.py from_pathOs zUploadInfo.from_pathdatacCs&t|}|t||dd|dS)Nr$)r r!r)rdigestlen)r)r0r+r,r,r- from_bytesWs zUploadInfo.from_bytes)fileobjcCsD|d}|dtjt|}|}|dtj||||dS)Nr$rr%)readseekr&SEEK_SETrtell)r)r4r!r+r r,r,r- from_fileobj\s  zUploadInfo.from_fileobjN) __name__ __module__ __qualname____doc__bytes__annotations__int classmethodstrr.r3rr9r,r,r,r-r<s  r) upload_infostoken repo_typerepo_idrevisionendpointheaders transfersreturncCs |dk r |ntj}d}|tjkr*tj|}|d||d} d|dk rL|nddgdd |Dd d } |dk r~d t|i| d <tt|d|pi}tj| || d} t| | } | dd} t | t st d| d}t |tr|nd}dd | Ddd | D|fS)a Requests the LFS batch endpoint to retrieve upload instructions Learn more: https://github.com/git-lfs/git-lfs/blob/main/docs/api/batch.md Args: upload_infos (`Iterable` of `UploadInfo`): `UploadInfo` for the files that are being uploaded, typically obtained from `CommitOperationAdd.upload_info` repo_type (`str`): Type of the repo to upload to: `"model"`, `"dataset"` or `"space"`. repo_id (`str`): A namespace (user or an organization) and a repo name separated by a `/`. revision (`str`, *optional*): The git revision to upload to. headers (`dict`, *optional*): Additional headers to include in the request transfers (`list`, *optional*): List of transfer methods to use. Defaults to ["basic", "multipart"]. Returns: `LfsBatchInfo`: 3-tuple: - First element is the list of upload instructions from the server - Second element is a list of errors, if any - Third element is the chosen transfer adapter if provided by the server (e.g. "basic", "multipart", "xet") Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If an argument is invalid or the server response is malformed. [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the server returned an error. N/z.git/info/lfs/objects/batchuploadbasic multipartcSsg|]}|j|jdqS)oidr )rhexr ).0rNr,r,r- sz'post_lfs_batch_info..r) operationrJobjectsZ hash_algonameref)rDrIjsonrWzMalformed response from serverZtransfercSsg|]}d|krt|qSerror)_validate_batch_actionsrTobjr,r,r-rUscSsg|]}d|krt|qSr\)_validate_batch_errorr_r,r,r-rUs)rZENDPOINTZREPO_TYPES_URL_PREFIXESr LFS_HEADERSrrpostrr[get isinstancelist ValueErrorrB)rCrDrErFrGrHrIrJZ url_prefixZ batch_urlpayloadrespZ batch_inforWZchosen_transferr,r,r-post_lfs_batch_infofs<,        rjc@seZdZUeed<eed<dS) PayloadPartT partNumberetagN)r:r;r<r@r?rBr,r,r,r-rks rkc@s&eZdZUdZeed<eeed<dS)CompletionPayloadTz?Payload that will be sent to the Hub when uploading multi-part.rRpartsN)r:r;r<r=rBr?r rkr,r,r,r-rns rnr)rVlfs_batch_actionrDrIrHrKc CsPt||d}|dkr2td|jddS|dd}t||dd}|dk rdt||di}|d} t|d |d } | dk rz t| } Wn(tt fk rtd | d YnXt ||| | d n t || d|dk r:t|t|d |} t j | t||d|jj|jjdd} t| t|jddS)aw Handles uploading a given object to the Hub with the LFS protocol. Can be a No-op if the content of the file is already present on the hub large file storage. Args: operation (`CommitOperationAdd`): The add operation triggering this upload. lfs_batch_action (`dict`): Upload instructions from the LFS batch endpoint for this object. See [`~utils.lfs.post_lfs_batch_info`] for more details. headers (`dict`, *optional*): Headers to include in the request, including authentication and user agent headers. Raises: [`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError) If `lfs_batch_action` is improperly formatted [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error actionsNzContent of file z. is already present upstream - skipping uploadrNverifyheader chunk_sizehref)rHzTMalformed response from LFS batch endpoint: `chunk_size` should be an integer. Got 'z'.)rVrsrt upload_url)rVrv)rDrIrQrZz: Upload successful)r^rdloggerdebug path_in_repo_validate_lfs_actionrr@rg TypeError_upload_multi_part_upload_single_partrrcr upload_inforrSr r) rVrprDrIrHrq upload_action verify_actionrsrtrvZ verify_urlZ verify_respr,r,r- lfs_uploads@         rZ lfs_actioncCs:t|dtr.|ddks6t|dts6td|S).validates response from the LFS batch endpointrursNz"lfs_action is improperly formatted)rerdrBdictrgrr,r,r-rzs rz)lfs_batch_actionscCspt|dtr t|dts(td|did}|did}|dk r\t||dk rlt||S)rrRr z)lfs_batch_actions is improperly formattedrqrNrrN)rerdrBr@rgrz)rrrr,r,r-r^s r^)lfs_batch_errorcCsht|dtr t|dts(td|d}t|tr\t|dtr\t|dtsdtd|S)rrRr z'lfs_batch_error is improperly formattedr]messagecode)rerdrBr@rgr)r error_infor,r,r-ra's  ra)rVrvrKc Cs2|jdd}td||d}t|W5QRXdS)a Uploads `fileobj` as a single PUT HTTP request (basic LFS transfer protocol) Args: upload_url (`str`): The URL to PUT the file to. fileobj: The file-like object holding the data to upload. Returns: `requests.Response` Raises: [`HTTPError`](https://requests.readthedocs.io/en/latest/api/#requests.HTTPError) If the upload resulted in an error. TZ with_tqdmPUTr/N)as_filerr)rVrvr4responser,r,r-r}5sr})rVrsrtrvrKcCst||j|d}tj}tjrBt|jtsBt|jtsBt dd}|rTt |||dn t |||d}t j |t||jjtd}t|dS)z@ Uploads file using HF multipart LFS transfer protocol. )rsr~rtzlhf_transfer is enabled but does not support uploading from bytes or BinaryIO, falling back to regular uploadF)rVsorted_parts_urlsrt)r[rIN)_get_sorted_parts_urlsr~rZHF_HUB_ENABLE_HF_TRANSFERrepath_or_fileobjrBrwarningswarn_upload_parts_hf_transfer_upload_parts_iterativelyrrc_get_completion_payloadrrSrbr)rVrsrtrvrZuse_hf_transferresponse_headersZcompletion_resr,r,r-r|Ks,   r|)rsr~rtrKcCsNddtdd|DdddD}t|}|t|j|krJtd|S)NcSsg|] \}}|qSr,r,)rT_rvr,r,r-rUosz*_get_sorted_parts_urls..cSs2g|]*\}}|rt|dkrt|d|fqS)r )isdigitr2r@)rTZpart_numrvr,r,r-rUrs cSs|dS)Nrr,)tr,r,r-wz(_get_sorted_parts_urls..)keyz0Invalid server response to upload large LFS file)sorteditemsr2rr rg)rsr~rtZsorted_part_upload_urlsZ num_partsr,r,r-rns r)rrRrKcCsfg}t|D]N\}}|d}|dks.|dkrFtd|d|d||d|dq ||dS)NrmrLzInvalid etag (`z`) returned for part r)rlrm)rRro) enumeraterdrgappend)rrRroZ part_numberrsrmr,r,r-rs r)rVrrtrKc Csrg}|jddX}t|D]H\}}t||||d(}td||d}t|||jW5QRXqW5QRX|S)NTr)Z seek_fromZ read_limitrr/)rrrrrrrI) rVrrtrIr4Zpart_idxZpart_upload_urlZ fileobj_sliceZpart_upload_resr,r,r-rs"rc Cs zddlm}Wntk r,tdYnXdt|jk}|sLtd|j j }|j }t |dkrxd|dd}t d d |d|ttd d }z0|f|j||d ddd|rd|jini}Wn,tk r} ztd| W5d} ~ XYnX|s|||W5QRSQRXdS)Nr)multipart_uploadzFast uploading using 'hf_transfer' is enabled (HF_HUB_ENABLE_HF_TRANSFER=1) but 'hf_transfer' package is not available in your environment. Try `pip install hf_transfer`.callbackzYou are using an outdated version of `hf_transfer`. Consider upgrading to latest version to enable progress bars using `pip install -U hf_transfer`.(u(…)iBTzhuggingface_hub.lfs_upload)unitZ unit_scaletotalinitialdescdisablerX) file_pathZ parts_urlsrtZ max_filesZparallel_failures max_retriesz~An error occurred while uploading using `hf_transfer`. Consider disabling HF_HUB_ENABLE_HF_TRANSFER for better error handling.)Z hf_transferr ImportErrorrginspect signature parametersrrr~r ryr2rrrwgetEffectiveLevelrupdate Exception RuntimeError) rVrrtrZsupports_callbackrrprogressoutputer,r,r-rsZ     r)NNNN)NNN)Dr=rr&rerZ dataclassesrmathros.pathrpathlibrtypingrrrr r r r r urllib.parserZhuggingface_hubrutilsrrrrrrrrZ utils._lfsrZ utils.sharrZ utils.tqdmrZ _commit_apirZ get_loggerr:rwcompileZ OID_REGEXZLFS_MULTIPART_UPLOAD_COMMANDrbrrBrrjrkrnrrzr^rar}r@r|rrrrr,r,r,r-s    (  (     ) V  F #