GeminiとTypeScriptでテキスト類似度を計算する
https://www.kaggle.com/code/markishere/day-2-embeddings-and-similarity-scores をTypeScriptで書いてみる
npm:@google/generative-aiを使う
APIは https://ai.google.dev/api/embeddings#method:-models.embedcontent を参照
実行方法
1. .envを作る
code:.env
GOOGLE_API_KEY=<your api key>
2. .envと同じdirectoryで↓を実行する
$ deno run --allow-env=GOOGLE_API_KEY --allow-read=.env --allow-net=generativelanguage.googleapis.com https://scrapbox.io/api/code/work4ai/GeminiとTypeScriptでテキスト類似度を計算する/similarity.ts
code:similarity.ts
//Copyright 2024 Google LLC. under the Apache License, Version 2.0
import "jsr:@std/dotenv@0.225/load";
import { GoogleGenerativeAI } from "npm:@google/generative-ai@0.21.0";
import { sumOf } from "jsr:@std/collections@1/sum-of";
const texts = [
"The quick brown fox jumps over the lazy dog.",
"The quick rbown fox jumps over the lazy dog.",
"teh fast fox jumps over the slow woofer.",
"a quick brown fox jmps over lazy dog.",
"brown fox jumping over dog",
"fox > dog",
// Alternative pangram for comparison:
"The five boxing wizards jump quickly.",
// Unrelated text, also for comparison:
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus et hendrerit massa. Sed pulvinar, nisi a lobortis sagittis, neque risus gravida dolor, in porta dui odio vel purus.",
];
const ai = new GoogleGenerativeAI(Deno.env.get("GOOGLE_API_KEY")!);
const model = ai.getGenerativeModel({ model: "models/text-embedding-004" });
const response = await model.batchEmbedContents({
requests: texts.map((text) => ({
content: { role: "user", parts: text } },
})),
});
embedContentだと、1つのテキストデータしか文章ベクトルに変換できない
batchEmbedContentsを使うことで、複数のデータをまとめてベクトルにできる
これって学習済みのモデルから文章ベクトルを作っているということか？takker.icon
text-embedding-004が、今回使った学習モデルということ
関係ないけどThe quick brown fox jumps over the lazy dogだwogikaze.icon
アルファベットのいろは歌（重なりの少ないすべての文字を使った作文）wogikaze.icon
へ～takker.icon
code:similarity.ts
const matrix = response.embeddings.map((embedding) => embedding.values);
const similarities = matrix.map((row) =>
matrix.map((col) => sumOf(row.map((v, j) => v * colj), (i) => i))
);
console.table(similarities.map((row) => row.map((v) => v.toPrecision(3))));
code:output example
┌───────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┬─────────┐
│ (idx) │ 0 │ 1 │ 2 │ 3 │ 4 │ 5 │ 6 │ 7 │
├───────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┼─────────┤
│ 0 │ "1.00" │ "0.894" │ "0.669" │ "0.832" │ "0.690" │ "0.605" │ "0.529" │ "0.405" │
│ 1 │ "0.894" │ "1.00" │ "0.700" │ "0.800" │ "0.691" │ "0.634" │ "0.512" │ "0.403" │
│ 2 │ "0.669" │ "0.700" │ "1.00" │ "0.684" │ "0.613" │ "0.648" │ "0.459" │ "0.398" │
│ 3 │ "0.832" │ "0.800" │ "0.684" │ "1.00" │ "0.667" │ "0.663" │ "0.478" │ "0.429" │
│ 4 │ "0.690" │ "0.691" │ "0.613" │ "0.667" │ "1.00" │ "0.692" │ "0.406" │ "0.321" │
│ 5 │ "0.605" │ "0.634" │ "0.648" │ "0.663" │ "0.692" │ "1.00" │ "0.307" │ "0.358" │
│ 6 │ "0.529" │ "0.512" │ "0.459" │ "0.478" │ "0.406" │ "0.307" │ "1.00" │ "0.339" │
│ 7 │ "0.405" │ "0.403" │ "0.398" │ "0.429" │ "0.321" │ "0.358" │ "0.339" │ "1.00" │
└───────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┴─────────┘
コサイン類似度は0~1の値になるよう、あらかじめvectorが正規化されている
コサイン類似度は負にならない
jsr:@openai/openaiで書いてみる
Gemini APIをOpenAIライブラリで動かす方法を用いる
OpenAI形式のbatch処理に対応しているか不明だったため、1データずつembedした
$ deno run --allow-env=GOOGLE_API_KEY,OPENAI_ORG_ID,OPENAI_PROJECT_ID,DEBUG --allow-read=.env --allow-net=generativelanguage.googleapis.com https://scrapbox.io/api/code/work4ai/GeminiとTypeScriptでテキスト類似度を計算する/similarity-openai-compability.ts
code:similarity-openai-compability.ts
import "jsr:@std/dotenv@0.225/load";
import { OpenAI } from "jsr:@openai/openai@4";
import { sumOf } from "jsr:@std/collections@1/sum-of";
const texts = [
"The quick brown fox jumps over the lazy dog.",
"The quick rbown fox jumps over the lazy dog.",
"teh fast fox jumps over the slow woofer.",
"a quick brown fox jmps over lazy dog.",
"brown fox jumping over dog",
"fox > dog",
// Alternative pangram for comparison:
"The five boxing wizards jump quickly.",
// Unrelated text, also for comparison:
"Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vivamus et hendrerit massa. Sed pulvinar, nisi a lobortis sagittis, neque risus gravida dolor, in porta dui odio vel purus.",
];
const openai = new OpenAI({
apiKey: Deno.env.get("GOOGLE_API_KEY")!,
baseURL: "https://generativelanguage.googleapis.com/v1beta/openai/",
});
const result = await Promise.all(texts.map((input) =>
openai.embeddings.create({
model: "text-embedding-004",
input,
})
));
const matrix = result.map((response) => response.data0.embedding);
const similarities = matrix.map((row) =>
matrix.map((col) => sumOf(row.map((v, j) => v * colj), (i) => i))
);
console.table(similarities.map((row) => row.map((v) => v.toPrecision(3))));