AgentOps
LLMOps
MLOps
DataOps
マルチエージェントシステム
AIエージェント
GUIエージェント(Computer use)
LLMの評価
AI safety
MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
https://arxiv.org/abs/2410.07095
https://openai.com/index/mle-bench/
How Rexera’s AI agents drive quality control with LangGraph
https://blog.langchain.dev/customers-rexera/
SREが投資するAIOps ~ペアーズにおけるLLM for Developerへの取り組み~
https://speakerdeck.com/takumiogawa/sregatou-zi-suruaiops-peazuniokerullm-for-developerhenoqu-rizu-mi?slide=4
AIエージェントを実運用に乗せるステップはこんな感じじゃなかろうか
https://speakerdeck.com/naotoota/aiezientowoshi-yun-yong-nicheng-serusutetupuhakonnagan-ziziyanakarouka
The Shift from Models to Compound AI Systems
https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/
Weaveを用いた生成AIアプリケーションの評価_モニタリンングと実践例.pdf
https://speakerdeck.com/olachinkei/weavewoyong-itasheng-cheng-aiahurikesiyonnoping-jia-monitarinnkutoshi-jian-li
Announcing the OWASP LLM and Gen AI Security Project Initiative for Securing Agentic Applications
https://genai.owasp.org/2024/12/15/announcing-the-owasp-llm-and-gen-ai-security-project-initiative-for-securing-agentic-applications/