論文の信頼度
「論文が1つでた」というだけではその情報の精度はよくわからない
根拠の信頼度は研究の比較の精細さによって変わってくる。例えば疫学など実験的にわかる研究においては次のような順に信頼度が高い
https://gyazo.com/05e2924d1d2decdcf98fad06f8b5d0b0
Schematic of proposed hierarchy of translational potential in basic research.
Validity of Evidence in the Basic Sciences
To evaluate the translational potential of basic research, the validity of evidence must first be assessed, usually by examining the approach taken to collect and evaluate the data.
Studies in the basic sciences are broadly grouped as
1. hypothesis-generating
2. hypothesis-driven.
The former tend to be small-sampled proof-of-principle studies and are typically exploratory and less valid than the latter. An argument can even be made that studies that report novel findings fall into this group as well, since their findings remain subject to external validation prior to being accepted by the broader scientific community
Alternatively, hypothesis-driven studies build upon what is known or strongly suggested by earlier work. These studies can also validate prior experimental findings with incremental contributions. Although such studies are often overlooked and even dismissed due to a lack of substantial novelty, their role in external validation of prior work is critical for establishing the translational potential of findings.
Another dimension to the validity of evidence in the basic sciences is the selection of experimental model.
The human condition is near-impossible to recapitulate in a laboratory setting, therefore experimental models (e.g., cell lines, primary cells, animal models) are used to mimic the phenomenon of interest, albeit imperfectly. For these reasons, the best quality evidence comes from evaluating the performance of several independent experimental models. This is accomplished through systematic approaches that consolidate evidence from multiple studies, thereby filtering the signal from the noise and allowing for side-by-side comparison. While systematic reviews can be conducted to accomplish a qualitative comparison, meta-analytic approaches employ statistical methods which enable hypothesis generation and testing. When a meta-analysis in the basic sciences is hypothesis-driven, it can be used to evaluate the translational potential of a given outcome and provide recommendations for subsequent translational- and clinical-studies. Alternatively, if meta-analytic hypothesis testing is inconclusive, or exploratory analyses are conducted to examine sources of inconsistency between studies, novel hypotheses can be generated, and subsequently tested experimentally. Figure 2 summarizes this proposed framework.
複数の過去の研究をメタ解析したもの
出版バイアスの排除
もちろん対象の論文が粗雑だとこの結果も粗雑になる
嘘八百の論文をメタ解析しても嘘しか出ない
https://cdn-ak.f.st-hatena.com/images/fotolife/h/honeshabri/20210101/20210101222104.png https://honeshabri.hatenablog.com/entry/books-2020-2#%E8%B2%A7%E4%B9%8F%E4%BA%BA%E3%81%AE%E7%B5%8C%E6%B8%88%E5%AD%A6%E3%82%82%E3%81%86%E3%81%84%E3%81%A1%E3%81%A9%E8%B2%A7%E5%9B%B0%E5%95%8F%E9%A1%8C%E3%82%92%E6%A0%B9%E3%81%A3%E3%81%93%E3%81%8B%E3%82%89%E8%80%83%E3%81%88%E3%82%8B
制約
ランダム化比較実験の結果と同じ効果が得られると考えるには「二つの仮定」が存在する。
処置に対する反応は個人の処置によって決まり、母集団の他のメンバーが受けた処置の影響は受けないとする。
処置群の結果の分布は、母集団全員に処置を施した場合と同じになるとする
政策で得られたい効果は長期的なものであるのに対して、実験で観察できるのは多くは短期的な結果である。ここには短期的な結果は長期的な結果と同じ（もしくは近い）という「仮定」が存在している。他にも様々な仮定があるが、全てをここでは述べない。ここで述べたいのは、ランダム化比較実験の結果から物事を判断するためには、こうした仮定が必要ということだ。
対象が無作為ではない
被験者を対象群と介入群に分けない
総説（専門家の意見）
詳しい人が実験なしに過去の知識から思ったことをいうだけ
何も知らない人の意見よりはマシだが、間違うこともよくある