We are announcing PaperQA2 (https://github.com/Future-House/paper-qa), the first AI agent to achieve superhuman performance on a variety of different scientific literature search tasks. PaperQA2 is an agent optimized for retrieving and summarizing information over the scientific literature. PaperQA2 has access to a variety of tools that allow it to find papers, extract useful information from those papers, explore the citation graph, and formulate answers. PaperQA2 achieves higher accuracy than PhD and postdoc-level biology researchers at retrieving information from the scientific literature, as measured using LitQA2, a piece of the LAB-Bench evals set that we released earlier this summer. In addition, when applied to produce wikipedia-style summaries of scientific information, WikiCrow, an agent built on top of PaperQA2, produces summaries that are more accurate on average than actual articles on Wikipedia that have been written and curated by humans, as judged by blinded PhD and postdoc-level biology researchers.
PaperQA2 allows us to perform analyses over the literature at a scale that are currently unavailable to scientists. At FutureHouse, we previously showed that we could use an older version (PaperQA) to generate a Wikipedia article for all 20,000 genes in the human genome, by combining information from 1 million distinct scientific papers. However, those articles were less accurate on average than existing articles on Wikipedia. Now that the articles we can generate are significantly more accurate than Wikipedia articles, one can imagine generating Wikipedia-style summaries on demand, or even regenerating Wikipeda from scratch with more comprehensive and recent information. In the coming weeks, we will use WikiCrow to generate Wikipedia articles for all 20,000 genes in the human genome, and will release them at wikicrow.ai. In the meantime, wikicrow.ai contains a preview of 240 articles used in the paper.
In addition, we are very interested in how PaperQA2 could allow us to generate new hypotheses. One approach to that problem is to identify contradictions between published scientific papers, which can point the way to new discoveries. In our paper, we describe how ContraCrow, an agent built on top of PaperQA2, can evaluate every claim in a scientific paper to identify any other papers in the literature that disagree with it. We can grade these contradictions on a Likert scale to remove trivial contradictions. We find 2.34 statements per paper on average in a random subset of biology papers that are contradicted by other papers from anywhere else in the literature. Exploring these contradictions in detail may allow agents like PaperQA2 and ContraCrow to generate new hypotheses and propose new pivotal experiments.
Is it possible for you to post the full text of the Alzheimer article that PaperQA2 actually generated in the Youtube video demo? It would be interesting to compare the PaperQA2 output to what other similar systems can produce.
The understanding of amyloid-beta's role in Alzheimer's disease has evolved significantly over time:
Initially, amyloid plaques were considered the primary cause of Alzheimer's, leading to the amyloid cascade hypothesis (Fantini2020Progress pages 4-7).
Research shifted focus from plaques to small oligomeric structures of amyloid-beta as the neurotoxic culprits (Fantini2020Progress pages 4-7).
It was discovered that physiological (low) levels of amyloid-beta enhance memory, while only pathological (high) levels are harmful (Morley2014The pages 1-6).
The view changed from Aβ peptides being purely harmful to recognizing their important physiological functions in the brain (Fantini2020Progress pages 1-4).
Clinical trials targeting amyloid have had mixed results, leading to questioning of the amyloid hypothesis (Hardy2009The pages 2-3).
Some researchers now propose rejecting the amyloid cascade hypothesis as insufficient to explain Alzheimer's complex biology (Herrup2015The pages 1-1).
There's growing recognition that tau pathology may correlate better with cognitive impairment than amyloid accumulation (Mullane2020Alzheimer's pages 16-16).
Current approaches are shifting towards considering amyloid-beta as one factor among many in the aging process, rather than the central cause of Alzheimer's (Josepha2001Copernicus pages 1-2).
References:
Fantini2020Progress: Jacques Fantini, Henri Chahinian, and Nouara Yahi. Progress toward alzheimer’s disease treatment: leveraging the achilles’ heel of aβ oligomers? Protein Science, 29(8):1748–1759, July 2020. URL: http://dx.doi.org/10.1002/pro.3906, doi:10.1002/pro.3906. This article has 48 citations and is from a peer-reviewed journal.
Morley2014The: John E. Morley and Susan A. Farr. The role of amyloid-beta in the regulation of memory. Biochemical Pharmacology, 88(4):479–485, April 2014. URL: http://dx.doi.org/10.1016/j.bcp.2013.12.018, doi:10.1016/j.bcp.2013.12.018. This article has 96 citations and is from a domain leading peer-reviewed journal.
Hardy2009The: John Hardy. The amyloid hypothesis for alzheimer’s disease: a critical reappraisal. Journal of Neurochemistry, 110(4):1129–1134, July 2009. URL: http://dx.doi.org/10.1111/j.1471-4159.2009.06181.x, doi:10.1111/j.1471-4159.2009.06181.x. This article has 615 citations and is from a domain leading peer-reviewed journal.
Josepha2001Copernicus: J Josepha. Copernicus revisited: amyloid beta in alzheimer’s disease. Neurobiology of Aging, 22(1):131–146, January 2001. URL: http://dx.doi.org/10.1016/s0197-4580(00)00211-6, doi:10.1016/s0197-4580(00)00211-6. This article has 146 citations and is from a domain leading peer-reviewed journal.
Hamley2012The: I. W. Hamley. The amyloid beta peptide: a chemist’s perspective. role in alzheimer’s and fibrillization. Chemical Reviews, 112(10):5147–5192, July 2012. URL: http://dx.doi.org/10.1021/cr3000994, doi:10.1021/cr3000994. This article has 775 citations and is from a highest quality peer-reviewed journal.
Herrup2015The: Karl Herrup. The case for rejecting the amyloid cascade hypothesis. Nature Neuroscience, 18(6):794–799, May 2015. URL: http://dx.doi.org/10.1038/nn.4017, doi:10.1038/nn.4017. This article has 593 citations and is from a highest quality peer-reviewed journal.
Jacobs2022It’s: Noortje Jacobs and Bert Theunissen. It’s groundhog day! what can the history of science say about the crisis in alzheimer’s disease research? Journal of Alzheimer’s Disease, 90(4):1401–1415, December 2022. URL: http://dx.doi.org/10.3233/jad-220569, doi:10.3233/jad-220569. This article has 4 citations.
Mullane2020Alzheimer’s: Kevin Mullane and Michael Williams. Alzheimer’s disease beyond amyloid: can the repetitive failures of amyloid-targeted therapeutics inform future approaches to dementia drug discovery? Biochemical Pharmacology, 177:113945, July 2020. URL: http://dx.doi.org/10.1016/j.bcp.2020.113945, doi:10.1016/j.bcp.2020.113945. This article has 68 citations and is from a domain leading peer-reviewed journal.
To get a better feel for how it works, try out the repo or check this tweet thread here (https://x.com/SGRodriques/status/1833908643856818443). It's got some videos of the workflow live.
PaperQA2 allows us to perform analyses over the literature at a scale that are currently unavailable to scientists. At FutureHouse, we previously showed that we could use an older version (PaperQA) to generate a Wikipedia article for all 20,000 genes in the human genome, by combining information from 1 million distinct scientific papers. However, those articles were less accurate on average than existing articles on Wikipedia. Now that the articles we can generate are significantly more accurate than Wikipedia articles, one can imagine generating Wikipedia-style summaries on demand, or even regenerating Wikipeda from scratch with more comprehensive and recent information. In the coming weeks, we will use WikiCrow to generate Wikipedia articles for all 20,000 genes in the human genome, and will release them at wikicrow.ai. In the meantime, wikicrow.ai contains a preview of 240 articles used in the paper.
In addition, we are very interested in how PaperQA2 could allow us to generate new hypotheses. One approach to that problem is to identify contradictions between published scientific papers, which can point the way to new discoveries. In our paper, we describe how ContraCrow, an agent built on top of PaperQA2, can evaluate every claim in a scientific paper to identify any other papers in the literature that disagree with it. We can grade these contradictions on a Likert scale to remove trivial contradictions. We find 2.34 statements per paper on average in a random subset of biology papers that are contradicted by other papers from anywhere else in the literature. Exploring these contradictions in detail may allow agents like PaperQA2 and ContraCrow to generate new hypotheses and propose new pivotal experiments.