GeneTuring tests GPT models in genomics

Paper

GeneTuring tests GPT models in genomics

Published 2023 · Wenpin Hou, Zhicheng Ji

UNKNOWN SJR score

30

Citations

2

Influential Citations

Semantic Scholar

Abstract

Generative Pre-trained Transformers (GPT) are powerful language models that have great potential to transform biomedical research. However, they are known to suffer from artificial hallucinations and provide false answers that are seemingly correct in some situations. We developed GeneTuring, a comprehensive QA database with 600 questions in genomics, and manually scored 10,800 answers returned by six GPT models, including GPT-3, ChatGPT, and New Bing. New Bing has the best overall performance and significantly reduces the level of AI hallucination compared to other models, thanks to its ability to recognize its incapacity in answering questions. We argue that improving incapacity awareness is equally important as improving model accuracy to address AI hallucination.

Study Snapshot

Key takeawayNew Bing is the best GPT model in genomics for reducing AI hallucination due to its ability to recognize incapacity in answering questions.

PopulationOlder adults (50-71 years)

Sample size24

MethodsObservational

OutcomesBody Mass Index projections

ResultsSocial networks mitigate obesity in older groups.

Sign up to use Study Snapshot

Consensus is limited without an account. Create an account or sign in to get more searches and use the Study Snapshot.

Create an account

Paper

GeneTuring tests GPT models in genomics

Sign up to use Study Snapshot

References

The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers

ChatGPT shows potential in medical education by generating virtual patient simulations, quizzes, and curriculums, but requires proper prompting and awareness of potential errors.

The Utility of ChatGPT as an Example of Large Language Models in Healthcare Education, Research and Practice: Systematic Review on the Future Perspectives and Potential Limitations

ChatGPT shows promise in healthcare education, research, and practice, but its adoption should be cautious due to ethical concerns and potential risks.

A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity

ChatGPT outperforms other LLMs in multitask, multilingual, and multimodal tasks, but is unreliable and suffers from hallucination issues.

Artificial Hallucinations in ChatGPT: Implications in Scientific Writing

ChatGPT's performance in scientific writing on homocystinuria-associated osteoporosis and late-onset Pompe disease cases showed positive, negative, and troubling aspects, with potential implications for medical education and research.

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining

BioGPT outperforms previous models in biomedical natural language processing tasks, achieving higher accuracy in relation extraction and text generation.

Language Models are Few-Shot Learners

Scaling up language models significantly improves task-agnostic, few-shot performance, reaching competitiveness with prior fine-tuning approaches.

PubMedQA: A Dataset for Biomedical Research Question Answering

PubMedQA is a biomedical question-answering dataset that uses PubMed abstracts to provide no/maybe/maybe answers to research questions, with a best performance of 68.1%.

OMIM.org: leveraging knowledge across phenotype–gene relationships

OMIM.org is a comprehensive knowledge repository for medical genetics, providing interactive access to gene and phenotype data, and linking to other genomic resources.

Citations

Enhancing Large Language Models with Domain-specific Retrieval Augment Generation: A Case Study on Long-form Consumer Health Question Answering in Ophthalmology

Large Language Models with Retrieval Augment Generation (RAG) significantly improve accuracy and reduce error rates in long-form consumer health questions, but face challenges in attribution and completeness.

GP-GPT: Large Language Model for Gene-Phenotype Mapping

GP-GPT is a specialized large language model for genetic-phenotype knowledge representation and genomics relation analysis, outperforming state-of-the-art models in genomics and medical genetics.

Assessing and Enhancing Large Language Models in Rare Disease Question-answering

Large Language Models (LLMs) can improve their diagnostic performance in rare diseases by an average of 8% when augmented with the rare diseases corpus (ReCOP).

BioRAG: A RAG-LLM Framework for Biological Question Reasoning

BioRAG, a RAG-LLM framework, outperforms other scientific RAG and LLM frameworks in biological question-answering tasks due to its extensive knowledge base and domain-specific knowledge hierarchy.

LAB-Bench: Measuring Capabilities of Language Models for Biology Research

The LAB-Bench dataset evaluates AI systems' performance on practical biology research tasks, aiming to help researchers in areas like literature search and molecular cloning.

Bioinformatics and biomedical informatics with ChatGPT: Year one review

ChatGPT chatbot has shown promise in bioinformatics, with strengths and limitations, and offers insights into potential future developments.