transformers.PreTrainedTokenizer
https://huggingface.co/docs/transformers/v4.18.0/en/main_classes/tokenizer#transformers.PreTrainedTokenizer
Base class for all slow tokenizers.
transformers.PreTrainedTokenizerBase
を継承
https://huggingface.co/docs/transformers/main_classes/tokenizer
return_overflowing_tokens (bool, optional, defaults to False) — Whether or not to return overflowing token sequences.