U
dix @ s d Z ddlZddlZddlmZ ddlmZmZmZmZ ddl Z
ddl m Z ddl
mZmZ ddlmZ dd lmZ eeZeG d
d d ZG d d
d
ZeG dd deZG dd dZG dd deZG dd deZ G dd deZ G dd deZ G dd deeZ dS ) zJ
Callbacks to use with the Trainer class and customize the training loop.
N) dataclass)DictListOptionalUnion)tqdm )IntervalStrategy
has_length)TrainingArguments)loggingc @ sN e Zd ZU dZdZee ed< dZe ed< dZ
e ed< dZ e ed< dZ e ed < dZ
e ed
< dZe ed < dZe ed < dZe ed
< dZeed< dZeeeef ed< dZee ed< dZee ed< dZeed< dZeed< dZeed< dZ eed< dZ eee eee ef f ed< dZ ed ed< dd Z ed d d Z!e"ed d d!Z#dS )" TrainerStatea
A class containing the [`Trainer`] inner state that will be saved along the model and optimizer when checkpointing
and passed to the [`TrainerCallback`].
In all this class, one step is to be understood as one update step. When using gradient accumulation, one update
step may require several forward and backward passes: if you use `gradient_accumulation_steps=n`, then one update
step requires going through *n* batches.
Args:
epoch (`float`, *optional*):
Only set during training, will represent the epoch the training is at (the decimal part being the
percentage of the current epoch completed).
global_step (`int`, *optional*, defaults to 0):
During training, represents the number of update steps completed.
max_steps (`int`, *optional*, defaults to 0):
The number of update steps to do during the current training.
logging_steps (`int`, *optional*, defaults to 500):
Log every X updates steps
eval_steps (`int`, *optional*):
Run an evaluation every X steps.
save_steps (`int`, *optional*, defaults to 500):
Save checkpoint every X updates steps.
train_batch_size (`int`, *optional*):
The batch size for the training dataloader. Only needed when
`auto_find_batch_size` has been used.
num_input_tokens_seen (`int`, *optional*, defaults to 0):
The number of tokens seen during training (number of input tokens, not the number of prediction tokens).
total_flos (`float`, *optional*, defaults to 0):
The total number of floating operations done by the model since the beginning of training (stored as floats
to avoid overflow).
log_history (`List[Dict[str, float]]`, *optional*):
The list of logs done since the beginning of training.
best_metric (`float`, *optional*):
When tracking the best model, the value of the best metric encountered so far.
best_model_checkpoint (`str`, *optional*):
When tracking the best model, the value of the name of the checkpoint for the best model encountered so
far.
is_local_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on
several machines) main process.
is_world_process_zero (`bool`, *optional*, defaults to `True`):
Whether or not this process is the global main process (when training in a distributed fashion on several
machines, this is only going to be `True` for one process).
is_hyper_param_search (`bool`, *optional*, defaults to `False`):
Whether we are in the process of a hyper parameter search using Trainer.hyperparameter_search. This will
impact the way data will be logged in TensorBoard.
stateful_callbacks (`List[StatefulTrainerCallback]`, *optional*):
Callbacks attached to the `Trainer` that should have their states be saved or restored.
Relevent callbacks should implement a `state` and `from_state` function.
Nepochr global_step max_stepsi
logging_steps
eval_steps
save_stepstrain_batch_sizenum_train_epochsnum_input_tokens_seen
total_flos log_history best_metricbest_model_checkpointTis_local_process_zerois_world_process_zeroFis_hyper_param_search
trial_name trial_paramsTrainerCallbackstateful_callbacksc C s | j d krg | _ | jd kr"i | _nt| jtr0n~i }| jD ]l}t|tsZtdt| |jj}||krt|| t s|| g||< ||
| q:| ||< q:|| _d S )NzNAll callbacks passed to be saved must inherit `ExportableState`, but received ) r r!
isinstancedictExportableState TypeErrortype __class____name__listappendstate)selfr! callbackname r/ A/tmp/pip-unpacked-wheel-bm_b0l5e/transformers/trainer_callback.py
__post_init__p s&
zTrainerState.__post_init__) json_pathc C sB t jt| dddd }t|ddd}|| W 5 Q R X dS ) zDSave the content of this instance in JSON format inside `json_path`. T)indent sort_keys
wutf-8encodingN)jsondumps dataclassesZasdictopenwrite)r, r2 Z json_stringfr/ r/ r0 save_to_json s zTrainerState.save_to_jsonc C s2 t |ddd}| }W 5 Q R X | f t|S )z3Create an instance from the content of `json_path`.rr8 r9 )r> readr; loads)clsr2 r@ textr/ r/ r0 load_from_json s zTrainerState.load_from_json)$r(
__module__ __qualname____doc__r r float__annotations__r intr r r r r r r r r r r strr r r boolr r r r r r! r1 rA classmethodrG r/ r/ r/ r0 r
# s0
7 r
c @ s* e Zd ZdZedddZedd ZdS )r$ aj
A class for objects that include the ability to have its state
be saved during `Trainer._save_checkpoint` and loaded back in during
`Trainer._load_from_checkpoint`.
These must implement a `state` function that gets called during the respective
Trainer function call. It should only include parameters and attributes needed to
recreate the state at a particular time, to avoid utilizing pickle/maintain standard
file IO writing.
Example:
```python
class EarlyStoppingCallback(TrainerCallback, ExportableState):
def __init__(self, early_stopping_patience: int = 1, early_stopping_threshold: Optional[float] = 0.0):
self.early_stopping_patience = early_stopping_patience
self.early_stopping_threshold = early_stopping_threshold
# early_stopping_patience_counter denotes the number of times validation metrics failed to improve.
self.early_stopping_patience_counter = 0
def state(self) -> dict:
return {
"args": {
"early_stopping_patience": self.early_stopping_patience,
"early_stopping_threshold": self.early_stopping_threshold,
},
"attributes": {
"early_stopping_patience_counter": self.early_stopping_patience_counter,
}
}
```returnc C s t dd S )Nz s z TrainerCallback.on_train_beginc K s dS )z6
Event called at the end of training.
Nr/ rj r/ r/ r0 on_train_endD s z TrainerCallback.on_train_endc K s dS )z<
Event called at the beginning of an epoch.
Nr/ rj r/ r/ r0 on_epoch_beginJ s z TrainerCallback.on_epoch_beginc K s dS )z6
Event called at the end of an epoch.
Nr/ rj r/ r/ r0 on_epoch_endP s z TrainerCallback.on_epoch_endc K s dS )z
Event called at the beginning of a training step. If using gradient accumulation, one training step might take
several inputs.
Nr/ rj r/ r/ r0
on_step_beginV s z TrainerCallback.on_step_beginc K s dS )zv
Event called before the optimizer step but after gradient clipping. Useful for monitoring gradients.
Nr/ rj r/ r/ r0 on_pre_optimizer_step] s z%TrainerCallback.on_pre_optimizer_stepc K s dS )z}
Event called after the optimizer step but before gradients are zeroed out. Useful for monitoring gradients.
Nr/ rj r/ r/ r0 on_optimizer_stepc s z!TrainerCallback.on_optimizer_stepc K s dS )zU
Event called at the end of an substep during gradient accumulation.
Nr/ rj r/ r/ r0 on_substep_endi s z TrainerCallback.on_substep_endc K s dS )z
Event called at the end of a training step. If using gradient accumulation, one training step might take
several inputs.
Nr/ rj r/ r/ r0 on_step_endo s zTrainerCallback.on_step_endc K s dS )z9
Event called after an evaluation phase.
Nr/ rj r/ r/ r0 on_evaluatev s zTrainerCallback.on_evaluatec K s dS )z=
Event called after a successful prediction.
Nr/ )r, rU r+ ri metricsrk r/ r/ r0
on_predict| s zTrainerCallback.on_predictc K s dS )z7
Event called after a checkpoint save.
Nr/ rj r/ r/ r0 on_save s zTrainerCallback.on_savec K s dS )z;
Event called after logging the last logs.
Nr/ rj r/ r/ r0 on_log s zTrainerCallback.on_logc K s dS )z7
Event called after a prediction step.
Nr/ rj r/ r/ r0 on_prediction_step s z"TrainerCallback.on_prediction_stepN)r( rH rI rJ r r
r] rl rm rn ro rp rq rr rs rt ru rv rx ry rz r{ r/ r/ r/ r0 r s 3r c @ sR e Zd ZdZdd Zdd Zdd Zdd Zed
d Z e
e e d d
dZ
e
e e d ddZe
e e d ddZe
e e d ddZe
e e d ddZe
e e d ddZe
e e d ddZe
e e d dd Ze
e e d d d Ze
e e d d d Ze
e e d d!d"Ze
e e d d#d$Ze
e e d d%d&Ze
e e d d'd(Ze
e e d d)d*Zd+d, Z d-S ).CallbackHandlerz>Internal class that just calls the list of callbacks in order.c C sf g | _ |D ]}| | q
|| _|| _|| _|| _d | _d | _tdd | j D sbt
d| j d S )Nc s s | ]}t |tV qd S N)r" DefaultFlowCallback.0cbr/ r/ r0 s z+CallbackHandler.__init__..zThe Trainer will not work properly if you don't have a `DefaultFlowCallback` in its callbacks. You
should add one before training with `trainer.add_callback(DefaultFlowCallback). The current list ofcallbacks is
:) callbacks add_callbackmodelprocessing_class optimizer lr_schedulertrain_dataloadereval_dataloaderanyloggerwarning
callback_list)r, r r r r r r r/ r/ r0 __init__ s zCallbackHandler.__init__c C sh t |tr| n|}t |tr"|n|j}|dd | jD krXtd| dd | j | j| d S )Nc S s g | ]
}|j qS r/ )r' )r cr/ r/ r0
s z0CallbackHandler.add_callback..zYou are adding a zH to the callbacks of this Trainer, but there is already one. The currentzlist of callbacks is
:)r" r& r' r r r r r* )r, r- r Zcb_classr/ r/ r0 r s
z CallbackHandler.add_callbackc C sb t |tr6| jD ]"}t ||r| j| | S qn(| jD ] }||kr<| j| | S q.)joinr rT r/ r/ r0 r s z CallbackHandler.callback_listrh c C s | d|||S )Nrl
call_eventr, rU r+ ri r/ r/ r0 rl s zCallbackHandler.on_init_endc C s d|_ | d|||S )NFrm )r^ r r r/ r/ r0 rm s z CallbackHandler.on_train_beginc C s | d|||S )Nrn r r r/ r/ r0 rn s z CallbackHandler.on_train_endc C s d|_ | d|||S )NFro )r_ r r r/ r/ r0 ro s z CallbackHandler.on_epoch_beginc C s | d|||S )Nrp r r r/ r/ r0 rp s z CallbackHandler.on_epoch_endc C s" d|_ d|_d|_| d|||S )NFrq )rb ra r` r r r/ r/ r0 rq s z CallbackHandler.on_step_beginc C s | d|||S )Nrr r r r/ r/ r0 rr s z%CallbackHandler.on_pre_optimizer_stepc C s | d|||S )Nrs r r r/ r/ r0 rs s z!CallbackHandler.on_optimizer_stepc C s | d|||S )Nrt r r r/ r/ r0 rt s z CallbackHandler.on_substep_endc C s | d|||S )Nru r r r/ r/ r0 ru s zCallbackHandler.on_step_endc C s d|_ | jd||||dS )NFrv rw )ra r r, rU r+ ri rw r/ r/ r0 rv s zCallbackHandler.on_evaluatec C s | j d||||dS )Nrx r r r r/ r/ r0 rx s zCallbackHandler.on_predictc C s d|_ | d|||S )NFry )r` r r r/ r/ r0 ry s zCallbackHandler.on_savec C s d|_ | jd||||dS )NFrz )logs)rb r )r, rU r+ ri r r/ r/ r0 rz s zCallbackHandler.on_logc C s | d|||S )Nr{ r r r/ r/ r0 r{ s z"CallbackHandler.on_prediction_stepc
K sP | j D ]D}t|||||f| j| j| j| j| j| jd|}|d k r|}q|S )N)r r r r r r )r getattrr r r r r r )r, eventrU r+ ri rk r- resultr/ r/ r0 r s$
zCallbackHandler.call_eventN) r( rH rI rJ r r r r propertyr r r
r] rl rm rn ro rp rq rr rs rt ru rv rx ry rz r{ r r/ r/ r/ r0 r| s.
r| c @ s4 e Zd ZdZeeedddZeeedddZdS )r~ zx
A [`TrainerCallback`] that handles the default flow of the training loop for logs, evaluation and checkpoints.
rh c K s |j dkr|jrd|_|jtjkr8|j |j dkr8d|_|jtjkrf|j |j dkrf|j |j krfd|_
|j tjkr|j dkr|j |j dkrd|_
|j |jkrd|_|j tjkrd|_
|S )Nr Tr )r Zlogging_first_steprb logging_strategyr ZSTEPSr
eval_strategyr
eval_delayra
save_strategyr r` r r^ NOrj r/ r/ r0 ru s.
z DefaultFlowCallback.on_step_endc K sF |j tjkrd|_|jtjkr0|j|jkr0d|_|jtjkrBd|_ |S )NT)
r r EPOCHrb r r r ra r r` rj r/ r/ r0 rp = s z DefaultFlowCallback.on_epoch_endN) r( rH rI rJ r r
r] ru rp r/ r/ r/ r0 r~ s r~ c @ sT e Zd ZdZdd Zdd Zdd Zdd d
Zd d Zd
d Z dddZ
dd Z dS )ProgressCallbackzU
A [`TrainerCallback`] that displays the progress of training or evaluation.
c C s d | _ d | _d S r} ) training_barprediction_barrT r/ r/ r0 r R s zProgressCallback.__init__c K s |j rt|jdd| _d| _d S )NT)total
dynamic_ncolsr )r r r r current_steprj r/ r/ r0 rm V s z ProgressCallback.on_train_beginc K s& |j r"| j|j| j |j| _d S r} )r r updater r rj r/ r/ r0 ru [ s z ProgressCallback.on_step_endNc K sB |j r>t|r>| jd kr2tt|| jd kdd| _| jd d S )NT)r Zleaver r )r r
r r lenr r )r, rU r+ ri r rk r/ r/ r0 r{ ` s
z#ProgressCallback.on_prediction_stepc K s$ |j r | jd k r| j d | _d S r} r r closerj r/ r/ r0 rv h s
z ProgressCallback.on_evaluatec K s$ |j r | jd k r| j d | _d S r} r rj r/ r/ r0 rx n s
zProgressCallback.on_predictc
K sh |j rd| jd k rdi }| D ]\}}|||< q |dd } d|krTt|d d|d< | jt| d S )Nr r r3 )r r rW poproundr? rN )
r, rU r+ ri r rk Z shallow_logsrZ r[ _r/ r/ r0 rz t s
zProgressCallback.on_logc K s |j r| j d | _d S r} )r r r rj r/ r/ r0 rn s
z ProgressCallback.on_train_end)N)N) r( rH rI rJ r rm ru r{ rv rx rz rn r/ r/ r/ r0 r M s
r c @ s e Zd ZdZdddZdS )PrinterCallbackz?
A bare [`TrainerCallback`] that just prints the logs.
Nc K s | dd }|jrt| d S )Nr )r r print)r, rU r+ ri r rk r r/ r/ r0 rz s zPrinterCallback.on_log)N)r( rH rI rJ rz r/ r/ r/ r0 r s r c @ sL e Zd ZdZdeee dddZdd Zd d
Z d d Z
e d
ddZ dS )EarlyStoppingCallbacka1
A [`TrainerCallback`] that handles early stopping.
Args:
early_stopping_patience (`int`):
Use with `metric_for_best_model` to stop training when the specified metric worsens for
`early_stopping_patience` evaluation calls.
early_stopping_threshold(`float`, *optional*):
Use with TrainingArguments `metric_for_best_model` and `early_stopping_patience` to denote how much the
specified metric must improve to satisfy early stopping conditions. `
This callback depends on [`TrainingArguments`] argument *load_best_model_at_end* functionality to set best_metric
in [`TrainerState`]. Note that if the [`TrainingArguments`] argument *save_steps* differs from *eval_steps*, the
early stopping will not occur until the next save step.
r early_stopping_patienceearly_stopping_thresholdc C s || _ || _d| _d S )Nr r r early_stopping_patience_counter)r, r r r/ r/ r0 r s z EarlyStoppingCallback.__init__c C sV |j r tjntj}|jd ks<|||jrDt||j | jkrDd| _n| jd7 _d S )Nr r )Zgreater_is_betternpZgreaterZlessr absr r )r, rU r+ ri metric_valueoperatorr/ r/ r0 check_metric_value s
z(EarlyStoppingCallback.check_metric_valuec K s8 |j std|jd k s td|jtjks4tdd S )Nz s.
u,= 5: