Automating Scientific Knowledge Retrieval with AI in Python
End-to-End Guide for Developing a Research Chatbot with OpenAI functions Capable of Semantic Search Across Arxiv
The vast amount of scientific publications today challenges effective knowledge retrieval. Researchers, academics, and professionals need innovative methods to stay updated.
AI and semantic search technologies have drastically changed information access. Leading these innovations, OpenAI functions turn natural language into structured outputs.
For instance, when queried about renewable energy advancements, OpenAI’s models quickly sift through publications, identify crucial papers, and summarize trends.
This method speeds up research and uncovers insights not easily seen through traditional methods.
This article aims to provide Python code for searching and processing scientific literature. Using OpenAI functions and the arXiv API, it simplifies the retrieval, summarization, and presentation of findings.
Here’s the guide’s structure:
This guide is structured as follows:
- Solution Architecture
- Core Python Functions
- Interacting with the Research Chatbot
- Challenges and Improvements
1. Solution Architecture
The architecture of the Scientific Knowledge Retrieval solution outlines a multi-layered approach for processing and delivering scientific knowledge to users.
First, the workflow actively manages complex user queries, interacts with external APIs, and delivers informative responses.
Next, the architecture integrates various components that ensure a smooth flow of information from the initial user input to the final response.
Figure. 1: Solution Architecture for Automatic Scientific Knowledge Retrieval with OpenAI Functions and the arXiv API.
1. User Interface (UI): The user submits queries through this interface. In this case from a jupyter notebook
2. Conversation Management: This module handles the dialogue, ensuring context is maintained throughout the user interaction.
3. Query Processing: The user’s query is interpreted here, which involves understanding the intent and preparing it for subsequent actions.
4. OpenAI API Integration (Embedding & Completion):
- The Completion part directly processes the query to generate an immediate response for some queries.
- The Embedding Request is used for queries that need academic paper retrieval, generating a vector to find relevant documents.
5. External APIs (arXiv): This is where the chatbot interacts with external databases like arXiv to fetch scientific papers based on the query.
6. Get Articles & Summarize: This function retrieves articles and then uses the embeddings to prioritize which articles to summarize based on the query’s context.
7. PDF Processing, Text Extraction & Chunking: If detailed information is needed, the system processes the PDFs, extracts text, and chunks it into smaller pieces, preparing for summarization.
8. Response Generation:
- It integrates responses from the OpenAI API Completion service.
- It includes summaries of articles retrieved and processed from the arXiv API, which are based on the embeddings generated earlier.
9. Presentation to User: The final step where a cohesive response, combining AI-generated answers and summaries of articles, is presented to the user.
2. Getting Started in Python
2.1 Installation of Necessary Libraries
We utilize a variety of Python libraries, each serving a specific function to facilitate the retrieval and processing of scientific knowledge. Here is an overview of each library and its role:
scipy: Essential for scientific computing,scipyoffers modules for optimization, linear algebra, integration, and moretenacity: Facilitates retrying of failed operations, particularly useful for reliable requests to external APIs or databases.tiktoken: is a fast BPE tokenizer designed for use with OpenAI’s models, facilitating the efficient tokenization of text for processing with AI models like GPT-4.termcolor: Enables colored terminal output, useful for differentiating log messages or outputs for easier debugging.openai: Official library for interacting with OpenAI’s APIs like GPT-3, crucial for querying and receiving AI model responses.requests: For making HTTP requests to web services or APIs, likely used for data retrieval or interaction with scientific resources.arxiv: Simplifies searching, fetching, and managing scientific papers from arXiv.org.pandas: Key for data manipulation and analysis, offering structures and functions for handling large datasets.PyPDF2: Enables text extraction from PDF files, vital for processing scientific papers in PDF format.tqdm: Generates progress bars for loops or long-running processes, improving the user experience.
2.2 Setting Up the Enviroment
First, you’ll need to create an account on OpenAI’s platform and obtain an API key from the API section of your account settings.
openai.api_key = "API_KEY"
GPT_MODEL = "gpt-3.5-turbo-0613"
EMBEDDING_MODEL = "text-embedding-ada-002"
2.3 Project Setup
Creating a structured directory for managing downloaded papers or data is crucial for organization and easy access. Here’s how you can set up the necessary directories:
- Create Directory Structure: Decide on a structure that suits your project’s needs. For managing downloaded papers, a
./data/papersdirectory is suggested. - Implementation: Use Python’s
oslibrary to check for the existence of these directories and create them if they don’t exist:
import os
directory = './data/papers'
if not os.path.exists(directory):
os.makedirs(directory)
This snippet ensures that your script can run on any system without manual directory setup, making your project more portable and user-friendly.
3. Core Functionalities
The research chatbot aims to streamline scientific knowledge retrieval and includes several key functionalities.
These focus on processing natural language queries, retrieving and summarizing academic content, and improving user interactions with sophisticated NLP techniques.
Below, we explore these functionalities, highlighted by specific code examples that demonstrate how they are implemented.
3.1 Embedding Generation
To effectively understand and process user queries, the chatbot uses embeddings—a numerical representation of text that encapsulates semantic meanings. This capability is vital for tasks like assessing the relevance of scientific papers to a query.
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def embedding_request(text):
response = openai.Embedding.create(input=text, model=EMBEDDING_MODEL)
return response['data']['embeddings']
This function, using a retry mechanism, requests embeddings from OpenAI’s API, ensuring robustness in face of potential API errors or rate limits.
3.2 Retrieving Academic Papers
Upon understanding a query, the chatbot fetches relevant academic papers, demonstrating its ability to interface directly with external databases like arXiv.
# Function to get articles from arXiv
def get_articles(query, library=paper_dir_filepath, top_k=5):
"""
Searches for and retrieves the top 'k' academic papers related to a user's query from the arXiv database.
The function uses the arXiv API to search for papers, with the search criteria being the user's query and the number of results limited to 'top_k'.
For each article found, it stores relevant information such as the title, summary, and URLs in a list.
It also downloads the PDF of each paper and stores references, including the title, download path, and embedding of the paper title, in a CSV file specified by 'library'.
This is useful for keeping a record of the papers and their embeddings for later retrieval and analysis.
This function will be used by the read_article_and_summarize
"""
search = arxiv.Search(
query=query, max_results=top_k, sort_by=arxiv.SortCriterion.Relevance
)
result_list = []
for result in search.results():
result_dict = {}
result_dict.update({"title": result.title})
result_dict.update({"summary": result.summary})
# Taking the first url provided
result_dict.update({"article_url": [x.href for x in result.links][0]})
result_dict.update({"pdf_url": [x.href for x in result.links][1]})
result_list.append(result_dict)
# Store references in library file
response = embedding_request(text=result.title)
file_reference = [
result.title,
result.download_pdf(data_dir),
response["data"][0]["embedding"],
]
# Write to file
with open(library, "a") as f_object:
writer_object = writer(f_object)
writer_object.writerow(file_reference)
f_object.close()
return result_list
3.3 Ranking and Summarization
With relevant papers at hand, the system ranks them based on their relatedness to the query and summarizes the content to provide concise, insightful information back to the user.
# Function to rank strings by relatedness to a query string
def strings_ranked_by_relatedness(
query: str,
df: pd.DataFrame,
relatedness_fn=lambda x, y: 1 - spatial.distance.cosine(x, y),
top_n: int = 100,
) -> list[str]:
"""
Ranks and returns a list of strings from a DataFrame based on their relatedness to a given query string.
The function first obtains an embedding for the query string. Then, it calculates the relatedness of each string in the DataFrame to the query,
using the provided 'relatedness_fn', which defaults to computing the cosine similarity between their embeddings.
It sorts these strings in descending order of relatedness and returns the top 'n' strings.
"""
query_embedding_response = embedding_request(query)
query_embedding = query_embedding_response["data"][0]["embedding"]
strings_and_relatednesses = [
(row["filepath"], relatedness_fn(query_embedding, row["embedding"]))
for i, row in df.iterrows()
]
strings_and_relatednesses.sort(key=lambda x: x[1], reverse=True)
strings, relatednesses = zip(*strings_and_relatednesses)
return strings[:top_n]
3.4 Summarizing Academic Papers
Following the identification of relevant papers, the chatbot employs a summarization process to distill the essence of scientific documents.
Other Articles Worth Reading:
Narrating Videos With OpenAI Vision And Whisperer Automatically
Newsletter