Automated Paper Reviewing

Review Process

The AI Scientist generates an automated review using a GPT-4o-based agent, following guidelines from the Neural Information Processing Systems (NeurIPS) conference

Review Output

The output includes numerical scores (soundness, presentation, contribution, overall, confidence), lists of weaknesses and strengths, and a preliminary decision (accept or reject)

Calibration

Reviews can be calibrated by thresholding using the reviewer score

Additional processes like self-reflection, few-shot examples, and review ensembling are used to improve accuracy