菊田遥平「原論文から解き明かす生成 AI」2025/8/23

https://gyazo.com/f5bd97a8436e2038c079c6298587535a

2.「入力 data の特徵量化」

Herbert Rubenstein, John B. Goodenough "Contextual correlates of synonymy" 1965/10/1

Experimentol corroboration was obtained for the hypothesis that the proportion of words common to the contexts of word A and to the contexts of word B is a function of the degree to which A and B are similar in meaning. The tests were carried out for variously defined contexts. The shapes of the functions, however, indicate that similarity of context is reliable as criterion only for detecting pairs of words that are very similar in meaning.

實驗的檢證により、單語 A の文脈と單語 B の文脈に共通する單語の割合が、A と B の意味的類似度の度合ひに依存するといふ假說が支持された。この檢證は樣々な定義に基づく文脈に對して實施された。ただし、得られた函數の形狀から判斷すると、文脈の類似性は意味的に極めて類似した單語の pair を檢出するための基準としてのみ信賴できることが明らかになった。

Rico Sennrich, Barry Haddow, Alexandra Birch "Neural Machine Translation of Rare Words with Subword Units" 2015/8/31

Neural machine translation (NMT) models typically operate with a fixed vocabulary, but translation is an open-vocabulary problem. Previous work addresses the translation of out-of-vocabulary words by backing off to a dictionary. In this paper, we introduce a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units. This is based on the intuition that various word classes are translatable via smaller units than words, for instance names (via character copying or transliteration), compounds (via compositional translation), and cognates and loanwords (via phonological and morphological transformations). We discuss the suitability of different word segmentation techniques, including simple character n-gram models and a segmentation based on the byte pair encoding compression algorithm, and empirically show that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.1 and 1.3 BLEU, respectively.

neural 機械飜譯 (NMT) model は通常、固定された語彙 set で動作するが、飜譯問題自體は open 語彙問題である。從來の硏究では、辭書を參照することで語彙外單語の飜譯に對處してきた。本論文では、より簡潔で效果的な approach を提案する。具體的には、NMT model が稀少語や未知語を subword 單位の連續として符號化できるようにすることで、open 語彙飜譯を實現する。この手法の根據は、樣々な語種が單語單位よりも小さな單位によって飜譯可能であるといふ直觀に基づいてゐる。例へば、人名は文字の copy や轉寫によって、複合語は構成要素ごとの飜譯によって、同義語や借用語は音韻的・形態的變換によってそれぞれ飜譯可能である。本硏究では、單純な文字 n-gram model や byte pair 符號化壓縮 algorithm に基づく segmentation 手法を含む、樣々な單語分割技術の適用可能性について考察する。實驗結果から、subword model が WMT 15 の英語-Deutsche 語および英語-русский 語飜譯 task において、それぞれ 1.1 BLEU point および 1.3 BLEU point の改善をもたらし、辭書參照 baseline を上囘る性能を示すことを實證した。

Philip Gage "A new algorithm for data compression" 1994/2/1

This article describes a simple general-purpose data compression algo-rithm, called Byte Pair Encoding (BPE), which provides almost as much compression as the popular Lempel, Ziv compression.

Data compression is becoming increasingly important as a way to stretch disk space and speed up data transfers. This article describes a simple general-purpose data compression algo-rithm, called Byte Pair Encoding (BPE), which provides almost as much compression as the popular Lempel, Ziv.

本稿では、一般的な data 壓縮 algorithm である「byte pair 符號化 (byte pair encoding: BPE)」について解說する。この algorithm は、廣く利用されてゐる Lempel-Ziv 壓縮法とほぼ同等の壓縮率を實現する simple な汎用 data 壓縮手法である。

disk 容量の有效活用や data 轉送速度の向上において、data 壓縮の重要性はますます高まってゐる。本稿では、一般的な data 壓縮 algorithm である「byte pair 符號化 (byte pair encoding: BPE)」について說明する。この手法は、廣く普及してゐる Lempel-Ziv 壓縮法とほぼ同等の壓縮性能を発揮する simple な汎用 algorithm である。

Taku Kudo "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates" 2018/4/29

Subword units are an effective way to alleviate the open vocabulary problems in neural machine translation (NMT). While sentences are usually converted into unique subword sequences, subword segmentation is potentially ambiguous and multiple segmentations are possible even with the same vocabulary. The question addressed in this paper is whether it is possible to harness the segmentation ambiguity as a noise to improve the robustness of NMT. We present a simple regularization method, subword regularization, which trains the model with multiple subword segmentations probabilistically sampled during training. In addition, for better subword sampling, we propose a new subword segmentation algorithm based on a unigram language model. We experiment with multiple corpora and report consistent improvements especially on low resource and out-of-domain settings.

subword 單位は、neural 機械飜譯 (NMT) における open 語彙問題を緩和する效果的な手法である。通常、文は固有の subword 列に變換されるが、subword 分割には潛在的な曖昧性が存在し、同一の語彙 set であっても複數の分割可能性が生じる場合がある。本論文で取り組む課題は、かうした分割の曖昧性を noise として積極的に活用することで、NMT system の頑健性を向上させ得るか否かといふ點である。本硏究では、訓練時に確率的に sampling された複數の subword 分割を用ゐて model を訓練する、simple な正則化手法「subword 正則化」を提案する。さらに優れた subword sampling を實現するために、unigram 言語 model に基づく新たな subword 分割 algorithm を提案する。複數の corpus を用ゐた實驗により、特に resource 制約下や domain 外の設定において、一貫した性能向上が確認された。

Taku Kudo, John Richardson "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing" 2018/8/19

SentencePiece is an unsupervised text tokenizer and detokenizer mainly for Neural Network-based text generation systems where the vocabulary size is predetermined prior to the neural model training. SentencePiece implements subword units (e.g., byte-pair-encoding (BPE)［Sennrich et al.］) and unigram language model［Kudo.］) with the extension of direct training from raw sentences. SentencePiece allows us to make a purely end-to-end system that does not depend on language-specific pre/postprocessing.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever "Language Models are Unsupervised Multitask Learners" 2019/2/14

大規模で多樣性に富んだ dataset で言語 model を學習させると、model は廣範な domain や dataset において優れた性能を発揮できるようになる。GPT-2は、8種類の言語モデリング dataset 中7つにおいて最先端の性能をゼロショットで達成した。ゼロショット設定で model が實行可能な task の多樣性は、十分な多樣性を持つ text corpus の尤度を最大化するように訓練された高能力 model が、明示的な敎師信號を必要とせずに驚くほど多くの task を實行する方法を學習し始めることを示唆してゐる。

Nived Rajaraman, Jiantao Jiao, Kannan Ramchandran "Towards a Theory of Tokenization in LLMs" 2024/4/12

While there has been a large body of research attempting to circumvent tokenization for language modeling (Clark et al., 2022; Xue et al., 2022), the current consensus is that it is a necessary initial step for designing state-of-the-art performant language models. In this paper, we investigate tokenization from a theoretical point of view by studying the behavior of transformers on simple data generating processes. When trained on data drawn from certain simple kth-order Markov processes for k>1, transformers exhibit a surprising phenomenon - in the absence of tokenization, they empirically fail to learn the right distribution and predict characters according to a unigram model (Makkuva et al., 2024). With the addition of tokenization, however, we empirically observe that transformers break through this barrier and are able to model the probabilities of sequences drawn from the source near-optimally, achieving small cross-entropy loss. With this observation as starting point, we study the end-to-end cross-entropy loss achieved by transformers with and without tokenization. With the appropriate tokenization, we show that even the simplest unigram models (over tokens) learnt by transformers are able to model the probability of sequences drawn from kth-order Markov sources near optimally. Our analysis provides a justification for the use of tokenization in practice through studying the behavior of transformers on Markovian data.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin "Attention is All You Need" 2017/6/12

4.「Generative Pre-trained Transformer と text 生成」

Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever "Improving language understanding by generative pre-training" 2018/6/11

Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).

自然言語理解には、text 含意関係判定、質問応答、意味的類似性評価、文書分類など、多岐にわたる多樣な task が含まれる。大規模な未ラベル text corpus は豊富に存在するものの、これらの特定 task を學習するためのラベル付き data は依然として不足してをり、識別的學習手法による model の性能向上は困難である。本硏究では、多樣な未ラベル text corpus を用ゐて言語 model を生成的事前學習した後、各個別 task に對して識別的 fine-tuning を施すことで、これらの task において大幅な性能向上が可能であることを實證する。從來の手法とは異なり、本手法では fine-tuning 時に task 特性を考慮した入力變換を適用することで、model アーキテクチャへの最小限の變更で效果的な知識転移を實現する。本手法の有效性を、自然言語理解分野における廣範なベンチマーク dataset で實證した。task 非依存型の汎用 model は、各 task 専用に設計されたアーキテクチャを採用する識別的學習 model を凌駕する性能を示し、硏究對象とした 12 task 中 9 task において最先端性能を大幅に更新した。具體的には、常識推論 task (Stories Cloze Test) で8.9%、質問応答 task (RACE) で5.7%、text 含意関係判定 task (MultiNLI) で1.5%といふ絶對的な性能向上を達成した。

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever "Language Models are Unsupervised Multitask Learners" 2019/2/14

上記

Tom B. Brown, OpenAI "Language Models are Few-Shot Learners" 2020/5/28

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

We presented a 175 billion parameter language model which shows strong performance on many NLP tasks and benchmarks in the zero-shot, one-shot, and few-shot settings, in some cases nearly matching the performance of state-of-the-art fine-tuned systems, as well as generating high-quality samples and strong qualitative performance at tasks defined on-the-fly. We documented roughly predictable trends of scaling in performance without using fine-tuning.We also discussed the social impacts of this class of model. Despite many limitations and weaknesses, these results suggest that very large language models may be an important ingredient in the development of adaptable, general language systems.

近年の硏究では、大規模な text corpus を用ゐた事前學習後に特定 task 向けに微調整を行ふことで、自然言語處理（NLP）の多岐にわたる task やベンチマークにおいて顕著な性能向上が確認されてゐる。この手法はアーキテクチャ的には task 非依存であるものの、依然として數千から數万例規模の task 特化型微調整用 dataset を必要とする。これに對し、人閒はわずか數例の事例や單純な指示文から新たな言語 task を一般的に習得できる能力を有してをり、現在のNLP system が依然として大きく苦戦してゐる課題である。本硏究では、言語 model の大規模化が task 非依存の少數事例學習性能を大幅に向上させることを示し、場合によっては從來の最先端微調整手法と遜色ない性能を達成できることを實證した。具體的には、1750億パラメータを有する自己囘帰型言語 model GPT-3を訓練し、これまでで最もパラメータ數の多い非スパース言語 model の10倍の規模を實現した。そして、この model を少數事例設定下で評価した。全ての task において、GPT-3は勾配更新や微調整を一切行わずに適用され、task 定義と少數事例のデモンストレーションは完全に text based の model との相互作用によって行われる。GPT-3は、飜譯、質問応答、穴埋め問題といった多くのNLP dataset において優れた性能を示すほか、單語の並べ替え、新規單語の文中使用、3桁の加減算など、オンザフライでの推論や domain 適応を必要とする task においても良好な結果を得た。同時に、GPT-3の少數事例學習が依然として困難を示す dataset や、大規模ウェブ corpus を用ゐた學習に伴う方法論的課題が顕在化する dataset も特定した。さらに、GPT-3が生成するニュース記事サンプルは、人閒の評価者が人閒が書いた記事と区別するのに困難を感じるレベルに達してゐることも確認した。本硏究成果が持つ社会的影響と、GPT-3 model 全體に関する廣範な議論を展開する。

我々は、ゼロショット、ワンショット、少數事例設定において多くの NLP task やベンチマークで優れた性能を示す1750億パラメータ規模の言語 model を提示した。場合によっては最先端の微調整済み system とほぼ同等の性能を達成するとともに、オンザフライで定義された task においても高品質なサンプル生成と優れた定性的性能を示した。微調整を使用せずに性能がスケールする傾向はほぼ予測通りの結果であった。また、この種の model が社会に与える影響についても考察した。多くの制約や弱點が存在するものの、これらの結果は、極めて大規模な言語 model が適応可能で汎用的な言語 system の開発において重要な要素となり得ることを示唆してゐる。

Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever "Generating Long Sequences with Sparse Transformers" 2019/4/23

OpenAI, et al. "GPT-4 Technical Report" 2023/3/15

We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, Baobao Chang, Xu Sun, Lei Li, Zhifang Sui "A Survey on In-context Learning" 2022/12/31

With the increasing capabilities of large language models (LLMs), in-context learning (ICL) has emerged as a new paradigm for natural language processing (NLP), where LLMs make predictions based on contexts augmented with a few examples. It has been a significant trend to explore ICL to evaluate and extrapolate the ability of LLMs. In this paper, we aim to survey and summarize the progress and challenges of ICL. We first present a formal definition of ICL and clarify its correlation to related studies. Then, we organize and discuss advanced techniques, including training strategies, prompt designing strategies, and related analysis. Additionally, we explore various ICL application scenarios, such as data engineering and knowledge updating. Finally, we addreƒss the challenges of ICL and suggest potential directions for further research. We hope that our work can encourage more research on uncovering how ICL works and improving ICL.

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, Furu Wei "Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers" 2022/12/20

This repository contains the implementation of ACL 2023 Findings paper "Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers".

本リポジトリには、ACL 2023で発表された論文「Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers」（GPTはなぜ文脈内學習が可能なのか？言語 model は暗黙的に勾配降下法をメタ最適化器として機能させる）の實裝コードが含まれています。

Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei "Deep reinforcement learning from human preferences" 2017/6/12

For sophisticated reinforcement learning (RL) systems to interact usefully with real-world environments, we need to communicate complex goals to these systems. In this work, we explore goals defined in terms of (non-expert) human preferences between pairs of trajectory segments. We show that this approach can effectively solve complex RL tasks without access to the reward function, including Atari games and simulated robot locomotion, while providing feedback on less than one percent of our agent's interactions with the environment. This reduces the cost of human oversight far enough that it can be practically applied to state-of-the-art RL systems. To demonstrate the flexibility of our approach, we show that we can successfully train complex novel behaviors with about an hour of human time. These behaviors and environments are considerably more complex than any that have been previously learned from human feedback.

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, Ryan Lowe "Training language models to follow instructions with human feedback" 2022/3/4

Making language models bigger does not inherently make them better at following a user's intent. For example, large language models can generate outputs that are untruthful, toxic, or simply not helpful to the user. In other words, these models are not aligned with their users. In this paper, we show an avenue for aligning language models with user intent on a wide range of tasks by fine-tuning with human feedback. Starting with a set of labeler-written prompts and prompts submitted through the OpenAI API, we collect a dataset of labeler demonstrations of the desired model behavior, which we use to fine-tune GPT-3 using supervised learning. We then collect a dataset of rankings of model outputs, which we use to further fine-tune this supervised model using reinforcement learning from human feedback. We call the resulting models InstructGPT. In human evaluations on our prompt distribution, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having 100x fewer parameters. Moreover, InstructGPT models show improvements in truthfulness and reductions in toxic output generation while having minimal performance regressions on public NLP datasets. Even though InstructGPT still makes simple mistakes, our results show that fine-tuning with human feedback is a promising direction for aligning language models with human intent.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale" 2020/10/22

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli "Deep Unsupervised Learning using Nonequilibrium Thermodynamics" 2015/3/12

A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and tractability. The essential idea, inspired by non-equilibrium statistical physics, is to systematically and slowly destroy structure in a data distribution through an iterative forward diffusion process. We then learn a reverse diffusion process that restores structure in data, yielding a highly flexible and tractable generative model of the data. This approach allows us to rapidly learn, sample from, and evaluate probabilities in deep generative models with thousands of layers or time steps, as well as to compute conditional and posterior probabilities under the learned model. We additionally release an open source reference implementation of the algorithm.

機械學習における中核的な課題の一つは、學習、sampling、推論、評価の各プロセスが解析的あるいは計算的に扱い可能な状態を維持しつつ、高度に柔軟な確率分布族を用ゐて複雑な dataset を model 化する手法の開発である。本硏究では、柔軟性と扱いやすさの兩立を同時に實現する新たな approach を提案する。その核心的な着想は、非平衡統計物理學の知見に着想を得たもので、反復的な順方向擴散プロセスを通じて data 分布の構造を系統的かつ緩やかに破壊するといふ手法である。その後、data に構造を再構築する逆方向擴散プロセスを學習することで、高度に柔軟でありながら計算效率に優れた data 生成 model を構築する。この手法により、數千層あるいは時閒ステップを有する深層生成 model においても、迅速な學習、sampling、確率評価が可能となる。さらに、學習済み model に基づく條件付き確率や事後確率の計算も實現する。加えて、本 algorithm の open source 參照實裝を公開する。

W. Feller "On the Theory of Stochastic Processes, With Particular Reference to Applications" 1949

Jonathan Ho, Ajay Jain, Pieter Abbeel "Denoising Diffusion Probabilistic Models" 2020/6/19

本硏究では、非平衡熱力學の原理に着想を得た潛在變數 model の一種である擴散確率 model を用ゐて、高品質な画像生成結果を提示する。最良の結果は、擴散確率 model とランジュバン力學を用ゐた noise 除去スコアマッチングとの新たな関連性に基づいて設計された重み付き變分下界を用ゐて學習を行ふことで得られた。本 model は自然に、自己囘帰型復號化の一般化と解釈可能な漸進的損失壓縮スキームを許容する。無條件のCIFAR10 dataset において、我々はInceptionスコア9.46および最先端レベルのFIDスコア3.17を達成した。256×256解像度のLSUN dataset では、ProgressiveGANと同等のサンプル品質を得てゐる。本實裝は以下のHTTPS URLで利用可能である

William Peebles, Saining Xie "Scalable Diffusion Models with Transformers" 2022/12/19

We explore a new class of diffusion models based on the transformer architecture. We train latent diffusion models of images, replacing the commonly-used U-Net backbone with a transformer that operates on latent patches. We analyze the scalability of our Diffusion Transformers (DiTs) through the lens of forward pass complexity as measured by Gflops. We find that DiTs with higher Gflops -- through increased transformer depth/width or increased number of input tokens -- consistently have lower FID. In addition to possessing good scalability properties, our largest DiT-XL/2 models outperform all prior diffusion models on the class-conditional ImageNet 512x512 and 256x256 benchmarks, achieving a state-of-the-art FID of 2.27 on the latter.

6.「text と畫像の融合」

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever "Learning Transferable Visual Models From Natural Language Supervision" 2021/2/26

最先端のコンピュータビジョン system は、事前に定義された固定の物體カテゴリを予測するように訓練されてゐる。この制約された敎師あり學習の形態は、他の視覚的概念を指定するために追加のラベル付き data が必要となるため、system の汎用性と實用性を制限する要因となってゐる。画像に関する生の text から直接學習することは、はるかに廣範な敎師情報を利用できる有望な代替手法である。本硏究では、4億組の (画像・text) dataset（インターネットから収集）において、「どのキャプションがどの画像に對応するか」を予測するといふ單純な事前學習 task が、SOTA（最先端）レベルの画像表現をゼロから學習するための效率的かつ擴張可能な手法であることを實證する。事前學習後、自然言語を用ゐて學習済みの視覚概念を參照したり（あるいは新たな概念を記述したり）することが可能となり、これにより model を下流 task へゼロショットで轉移させることができる。本手法の性能を評価するため、OCR、動画内動作認識、地理位置情報、細粒度物體分類など、30種類以上の既存コンピュータビジョン dataset を用ゐてベンチマークを實施した。この approach はほとんどの task に對して非自明な轉移性を示し、dataset 固有の訓練を必要とせずに完全敎師あり baseline と競合する性能を発揮する場合が多い。例へば、我々は訓練に使用された128万の學習例を一切用ゐることなく、ImageNetにおけるオリジナルのResNet-50と同等の精度をゼロショットで達成した。本硏究で使用したコードと事前學習済み model の重みは、このhttps URLから公開してゐる。

CLIP (Contrastive Language-Image Pre-Training) is a neural network trained on a variety of (image, text) pairs. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task, similarly to the zero-shot capabilities of GPT-2 and 3. We found CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, Mark Chen "Hierarchical Text-Conditional Image Generation with CLIP Latents" 2022/4/13

Contrastive models like CLIP have been shown to learn robust representations of images that capture both semantics and style. To leverage these representations for image generation, we propose a two-stage model: a prior that generates a CLIP image embedding given a text caption, and a decoder that generates an image conditioned on the image embedding. We show that explicitly generating image representations improves image diversity with minimal loss in photorealism and caption similarity. Our decoders conditioned on image representations can also produce variations of an image that preserve both its semantics and style, while varying the non-essential details absent from the image representation. Moreover, the joint embedding space of CLIP enables language-guided image manipulations in a zero-shot fashion. We use diffusion models for the decoder and experiment with both autoregressive and diffusion models for the prior, finding that the latter are computationally more efficient and produce higher-quality samples.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, Michal Irani "Imagic: Text-Based Real Image Editing with Diffusion Models" 2022/10/17

Text-conditioned image editing has recently attracted considerable interest. However, most methods are currently either limited to specific editing types (e.g., object overlay, style transfer), or apply to synthetically generated images, or require multiple input images of a common object. In this paper we demonstrate, for the very first time, the ability to apply complex (e.g., non-rigid) text-guided semantic edits to a single real image. For example, we can change the posture and composition of one or multiple objects inside an image, while preserving its original characteristics. Our method can make a standing dog sit down or jump, cause a bird to spread its wings, etc. -- each within its single high-resolution natural image provided by the user. Contrary to previous work, our proposed method requires only a single input image and a target text (the desired edit). It operates on real images, and does not require any additional inputs (such as image masks or additional views of the object). Our method, which we call "Imagic", leverages a pre-trained text-to-image diffusion model for this task. It produces a text embedding that aligns with both the input image and the target text, while fine-tuning the diffusion model to capture the image-specific appearance. We demonstrate the quality and versatility of our method on numerous inputs from various domains, showcasing a plethora of high quality complex semantic image edits, all within a single unified framework.

7.「生成 AI model の scaling 則」

Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei "Scaling Laws for Neural Language Models" 2020/1/23

本硏究では、言語 model の性能と交差エントロピー損失の関係について、経驗的なスケーリング則を調査した。損失関數の値は model size、dataset size、および學習に使用した計算資源量に對して、べき乗則に従ってスケールすることが明らかとなった。このスケーリング関係は、7桁以上の範囲にわたって一貫した傾向を示してゐる。ネットワーク幅や深さといった他のアーキテクチャ的詳細要素は、廣範な範囲において影響が極めて小さいことが判明した。過學習の度合いが model size および dataset size に依存する関係、および學習速度が model size に依存する関係は、單純な數式で記述可能である。これらの関係性を活用することで、固定された計算資源予算の最適な配分を決定することが可能となる。特に、大規模 model はサンプル效率が著しく高く、計算效率を最大化するためには、比較的小規模な dataset を用ゐて非常に大規模な model を學習させ、収束點を大幅に手前で學習を停止させる手法が有效であることが示された。

Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B. Brown, Prafulla Dhariwal, Scott Gray, Chris Hallacy, Benjamin Mann, Alec Radford, Aditya Ramesh, Nick Ryder, Daniel M. Ziegler, John Schulman, Dario Amodei, Sam McCandlish "Scaling Laws for Autoregressive Generative Modeling" 2020/10/28

We identify empirical scaling laws for the cross-entropy loss in four domains: generative image modeling, video modeling, multimodal image↔text models, and mathematical problem solving. In all cases autoregressive Transformers smoothly improve in performance as model size and compute budgets increase, following a power-law plus constant scaling law. The optimal model size also depends on the compute budget through a power-law, with exponents that are nearly universal across all data domains.

We find a number of additional scaling laws in specific domains: (a) we identify a scaling relation for the mutual information between captions and images in multimodal models, and show how to answer the question "Is a picture worth a thousand words?"; (b) in the case of mathematical problem solving, we identify scaling laws for model performance when extrapolating beyond the training distribution; (c) we finetune generative image models for ImageNet classification and find smooth scaling of the classification loss and error rate, even as the generative loss levels off. Taken together, these results strengthen the case that scaling laws have important implications for neural network performance, including on downstream tasks.

本硏究では、生成的画像モデリング、動画モデリング、マルチモーダル画像↔text model、および數學的問題解決といふ 4 つの分野において、交差エントロピー損失関數に関する経驗的なスケーリング法則を明らかにした。いずれの場合においても、自己囘帰型Transformer model は model size と計算資源が増加するにつれて性能が滑らかに向上し、その傾向はべき乗則に定數項を加えたスケーリング法則に従うことが明らかとなった。最適な model size もまた、べき乗則に従って計算資源に依存し、その指數値は全 data 領域でほぼ普遍的な値を示す。

DeepSeek-AI, et al. "DeepSeek-V3 Technical Report" 2024/12/27

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" 2022/1/28

We explore how generating a chain of thought -- a series of intermediate reasoning steps -- significantly improves the ability of large language models to perform complex reasoning. In particular, we show how such reasoning abilities emerge naturally in sufficiently large language models via a simple method called chain of thought prompting, where a few chain of thought demonstrations are provided as exemplars in prompting. Experiments on three large language models show that chain of thought prompting improves performance on a range of arithmetic, commonsense, and symbolic reasoning tasks. The empirical gains can be striking. For instance, prompting a 540B-parameter language model with just eight chain of thought exemplars achieves state of the art accuracy on the GSM8K benchmark of math word problems, surpassing even finetuned GPT-3 with a verifier.

DeepSeek-AI, et al. "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" 2025/1/22

We introduce our first-generation reasoning models, DeepSeek-R1-Zero and DeepSeek-R1. DeepSeek-R1-Zero, a model trained via large-scale reinforcement learning (RL) without supervised fine-tuning (SFT) as a preliminary step, demonstrates remarkable reasoning capabilities. Through RL, DeepSeek-R1-Zero naturally emerges with numerous powerful and intriguing reasoning behaviors. However, it encounters challenges such as poor readability, and language mixing. To address these issues and further enhance reasoning performance, we introduce DeepSeek-R1, which incorporates multi-stage training and cold-start data before RL. DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks. To support the research community, we open-source DeepSeek-R1-Zero, DeepSeek-R1, and six dense models (1.5B, 7B, 8B, 14B, 32B, 70B) distilled from DeepSeek-R1 based on Qwen and Llama.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, Daya Guo "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" 2024/2/5

Mathematical reasoning poses a significant challenge for language models due to its complex and structured nature. In this paper, we introduce DeepSeekMath 7B, which continues pre-training DeepSeek-Coder-Base-v1.5 7B with 120B math-related tokens sourced from Common Crawl, together with natural language and code data. DeepSeekMath 7B has achieved an impressive score of 51.7% on the competition-level MATH benchmark without relying on external toolkits and voting techniques, approaching the performance level of Gemini-Ultra and GPT-4. Self-consistency over 64 samples from DeepSeekMath 7B achieves 60.9% on MATH. The mathematical reasoning capability of DeepSeekMath is attributed to two key factors: First, we harness the significant potential of publicly available web data through a meticulously engineered data selection pipeline. Second, we introduce Group Relative Policy Optimization (GRPO), a variant of Proximal Policy Optimization (PPO), that enhances mathematical reasoning abilities while concurrently optimizing the memory usage of PPO.

數學的推論は、その複雑かつ構造化された性質ゆえに、言語 model にとって重大な課題となる。本論文では、Common Crawlから収集した1200億語に及ぶ數學関連 token と自然言語・コード data を追加學習したDeepSeekMath 7Bを提案する。DeepSeekMath 7Bは、外部 toolkit や投票手法に依存することなく、競技レベルのMATHベンチマークにおいて51.7%といふ優れたスコアを達成してをり、Gemini-UltraやGPT-4に匹敵する性能を示してゐる。DeepSeekMath 7Bから64サンプルについて自己整合性評価を行った結果、MATHベンチマークで60.9%の精度を達成した。DeepSeekMathの數學的推論能力は、主に以下の2つの要因に起因する：第一に、入念に設計された data 選択パイプラインを通じて、公開されてゐるウェブ data が持つ膨大な潛在能力を最大限に活用してゐる點である。第二に、Proximal Policy Optimization（PPO）の變種であるGroup Relative Policy Optimization（GRPO）を新たに導入したことで、數學的推論能力を向上させるとともに、PPOのメモリ使用量の最適化を同時に實現してゐる點である。

8.「生成 AI model の評價」

Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, Ion Stoica "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference" 2024/3/7

Long Phan, et al. "Humanity’s Last Exam" 2025/1/24