Automated doc processing is likely one of the best achievements of the ChatGPT revolution. It is because LLM can sort out a variety of topics and duties in a zero-shot setting, i.e. with out coaching information labeled throughout the area. This makes it a lot simpler to construct AI-powered purposes to course of, parse, and routinely perceive any doc. Easy approaches utilizing LLM are nonetheless hampered by non-textual contexts corresponding to figures, pictures, and tables, however this weblog publish makes an attempt to deal with this by focusing particularly on PDFs.
At a fundamental degree, a PDF is only a assortment of textual content, pictures, strains and their exact coordinates. They haven’t any inherent “textual content” construction and aren’t constructed to be processed as textual content, however solely to be displayed as such. This makes these paperwork troublesome to make use of, as text-only approaches fail to seize all of the format and visible components inside a majority of these paperwork, leading to a big lack of context and knowledge.
One technique to get round this “text-only” limitation is to carry out superior pre-processing of the doc by detecting tables, pictures, and layouts earlier than sending them to LLM. Tables will be parsed into Markdown or JSON, pictures and figures will be represented with captions, and textual content will be entered verbatim. Nonetheless, this method requires a customized mannequin and a few info should still be misplaced, so may it’s higher?
Most fashionable large-scale fashions are multimodal and might deal with a number of modalities corresponding to textual content, code, and pictures. This paves the way in which to an easier resolution to the issue of 1 mannequin doing every part without delay. So as an alternative of captioning pictures or parsing tables, you may feed the web page as a picture and course of it as is. Our pipeline can load a PDF, extract every web page as a picture, break up it into chunks (utilizing LLM), and index every chunk. As soon as a piece is retrieved, the whole web page is introduced into the LLM context to carry out the duty. Under we element how this may be carried out in apply.
The pipeline we’re implementing is a two-step course of. First, divide every web page into vital elements and summarize every one. It then indexes the chunks as soon as, appears to be like up the chunks each time it receives a request, and contains the whole context of every retrieved chunk within the LLM context.
Step 1: Break up and summarize pages
Extract the pages as pictures and move every one to a multimodal LLM for segmentation. Fashions like Gemini can simply perceive and deal with web page layouts.
- desk Recognized as one chunk.
- numbers Kind one other lump.
- textual content block Break up into particular person chunks.
- …
LLM generates a abstract for every ingredient, which will be embedded and listed right into a vector database.
Step 2: Embedding and context retrieval
This tutorial makes use of textual content embeddings just for simplicity, however one enchancment is to make use of imaginative and prescient embeddings straight.
Every entry within the database accommodates:
- Chunk overview.
- Discovered web page quantity.
- A hyperlink to a picture illustration of the whole web page so as to add context.
This schema permits Native degree search (at chunk degree) Monitor context (Hyperlinks again to finish web page). For instance, when a search question retrieves an merchandise, brokers can embrace the whole web page picture to offer the whole format and extra context to LLM to maximise response high quality.
Offering an entire picture permits LLM to make the most of all visible cues and vital format info (pictures, titles, bullet factors, and many others.) and adjoining objects (tables, paragraphs, and many others.) when producing a response. It is going to be.
Implement every step as a separate reusable agent.
The primary agent is for parsing, chunking, and summarizing. This includes dividing the doc into vital chunks after which producing a abstract for every chunk. This agent solely must be run as soon as per PDF to preprocess the doc.
The second agent manages indexing, looking out, and retrieval. This includes embedding chunks in a vector database for environment friendly retrieval. Indexing is carried out as soon as per doc, however searches will be repeated as many instances as wanted for various queries.
For each brokers, use: geminia multimodal LLM with robust visible understanding talents.
Parsing and chunking agent
The primary agent is chargeable for dividing every web page into significant chunks and summarizing every one by following these steps:
Step 1: Extract PDF pages as pictures
What we use is pdf2image library. The picture is then encoded in Base64 format, making it straightforward so as to add it to LLM requests.
The implementation is as follows:
from document_ai_agents.document_utils import extract_images_from_pdf
from document_ai_agents.image_utils import pil_image_to_base64_jpeg
from pathlib import Pathclass DocumentParsingAgent:
@classmethod
def get_images(cls, state):
"""
Extract pages of a PDF as Base64-encoded JPEG pictures.
"""
assert Path(state.document_path).is_file(), "File doesn't exist"
# Extract pictures from PDF
pictures = extract_images_from_pdf(state.document_path)
assert pictures, "No pictures extracted"
# Convert pictures to Base64-encoded JPEG
pages_as_base64_jpeg_images = [pil_image_to_base64_jpeg(x) for x in images]
return {"pages_as_base64_jpeg_images": pages_as_base64_jpeg_images}
extract_images_from_pdf: Extract every web page of the PDF as a PIL picture.
pil_image_to_base64_jpeg: Converts the picture to Base64 encoded JPEG format.
Step 2: Chunking and summarizing
Every picture is then despatched to LLM for segmentation and summarization. Use structured output to make sure you get predictions within the anticipated format.
from pydantic import BaseModel, Discipline
from typing import Literal
import json
import google.generativeai as genai
from langchain_core.paperwork import Docclass DetectedLayoutItem(BaseModel):
"""
Schema for every detected format ingredient on a web page.
"""
element_type: Literal["Table", "Figure", "Image", "Text-block"] = Discipline(
...,
description="Sort of detected merchandise. Examples: Desk, Determine, Picture, Textual content-block."
)
abstract: str = Discipline(..., description="An in depth description of the format merchandise.")
class LayoutElements(BaseModel):
"""
Schema for the listing of format components on a web page.
"""
layout_items: listing[DetectedLayoutItem] = []
class FindLayoutItemsInput(BaseModel):
"""
Enter schema for processing a single web page.
"""
document_path: str
base64_jpeg: str
page_number: int
class DocumentParsingAgent:
def __init__(self, model_name="gemini-1.5-flash-002"):
"""
Initialize the LLM with the suitable schema.
"""
layout_elements_schema = prepare_schema_for_gemini(LayoutElements)
self.model_name = model_name
self.mannequin = genai.GenerativeModel(
self.model_name,
generation_config={
"response_mime_type": "software/json",
"response_schema": layout_elements_schema,
},
)
def find_layout_items(self, state: FindLayoutItemsInput):
"""
Ship a web page picture to the LLM for segmentation and summarization.
"""
messages = [
f"Find and summarize all the relevant layout elements in this PDF page in the following format: "
f"{LayoutElements.schema_json()}. "
f"Tables should have at least two columns and at least two rows. "
f"The coordinates should overlap with each layout item.",
{"mime_type": "image/jpeg", "data": state.base64_jpeg},
]
# Ship the immediate to the LLM
end result = self.mannequin.generate_content(messages)
information = json.hundreds(end result.textual content)
# Convert the JSON output into paperwork
paperwork = [
Document(
page_content=item["summary"],
metadata={
"page_number": state.page_number,
"element_type": merchandise["element_type"],
"document_path": state.document_path,
},
)
for merchandise in information["layout_items"]
]
return {"paperwork": paperwork}
of LayoutElements The schema defines the construction of the output, together with every format merchandise kind (desk, diagram,…) and its abstract.
Step 3: Web page parallelism
Pages are processed in parallel to extend velocity. The next methodology creates an inventory of duties to course of all web page pictures without delay, for the reason that processing is io sure.
from langgraph.sorts import Shipclass DocumentParsingAgent:
@classmethod
def continue_to_find_layout_items(cls, state):
"""
Generate duties to course of every web page in parallel.
"""
return [
Send(
"find_layout_items",
FindLayoutItemsInput(
base64_jpeg=base64_jpeg,
page_number=i,
document_path=state.document_path,
),
)
for i, base64_jpeg in enumerate(state.pages_as_base64_jpeg_images)
]
Every web page is shipped to: find_layout_items Acts as an impartial activity.
full workflow
The agent workflow is StateGraphhyperlink the picture extraction and format detection steps into an built-in pipeline ->
from langgraph.graph import StateGraph, START, ENDclass DocumentParsingAgent:
def build_agent(self):
"""
Construct the agent workflow utilizing a state graph.
"""
builder = StateGraph(DocumentLayoutParsingState)
# Add nodes for picture extraction and format merchandise detection
builder.add_node("get_images", self.get_images)
builder.add_node("find_layout_items", self.find_layout_items)
# Outline the circulation of the graph
builder.add_edge(START, "get_images")
builder.add_conditional_edges("get_images", self.continue_to_find_layout_items)
builder.add_edge("find_layout_items", END)
self.graph = builder.compile()
To run the agent on the pattern PDF:
if __name__ == "__main__":
_state = DocumentLayoutParsingState(
document_path="path/to/doc.pdf"
)
agent = DocumentParsingAgent()# Step 1: Extract pictures from PDF
result_images = agent.get_images(_state)
_state.pages_as_base64_jpeg_images = result_images["pages_as_base64_jpeg_images"]
# Step 2: Course of the primary web page (for example)
result_layout = agent.find_layout_items(
FindLayoutItemsInput(
base64_jpeg=_state.pages_as_base64_jpeg_images[0],
page_number=0,
document_path=_state.document_path,
)
)
# Show the outcomes
for merchandise in result_layout["documents"]:
print(merchandise.page_content)
print(merchandise.metadata["element_type"])
This ends in a parsed, segmented, and summarized PDF illustration. This would be the enter for the second agent that we’ll construct subsequent.
RAG agent
This second agent handles the indexing and retrieval half. Retailer earlier agent paperwork in a vector database and use the outcomes for searches. This may be damaged down into two separate steps: indexing and retrieval.
Step 1: Indexing break up paperwork
Use the generated abstract to vectorize and put it aside to the ChromaDB database.
class DocumentRAGAgent:
def index_documents(self, state: DocumentRAGState):
"""
Index the parsed paperwork into the vector retailer.
"""
assert state.paperwork, "Paperwork ought to have at the very least one ingredient"
# Verify if the doc is already listed
if self.vector_store.get(the place={"document_path": state.document_path})["ids"]:
logger.data(
"Paperwork for this file are already listed, exiting this node"
)
return # Skip indexing if already carried out
# Add parsed paperwork to the vector retailer
self.vector_store.add_documents(state.paperwork)
logger.data(f"Listed {len(state.paperwork)} paperwork for {state.document_path}")
of index_documents The tactic embeds the chunk abstract right into a vector retailer. Metadata corresponding to doc path and web page numbers are saved for later use.
Step 2: Reply to questions
When a person asks a query, the agent searches the vector retailer for probably the most related chunk. Get an summary and corresponding web page pictures to know the context.
class DocumentRAGAgent:
def answer_question(self, state: DocumentRAGState):
"""
Retrieve related chunks and generate a response to the person's query.
"""
# Retrieve the top-k related paperwork primarily based on the question
relevant_documents: listing[Document] = self.retriever.invoke(state.query)# Retrieve corresponding web page pictures (keep away from duplicates)
pictures = listing(
set(
[
state.pages_as_base64_jpeg_images[doc.metadata["page_number"]]
for doc in relevant_documents
]
)
)
logger.data(f"Responding to query: {state.query}")
# Assemble the immediate: Mix pictures, related summaries, and the query
messages = (
[{"mime_type": "image/jpeg", "data": base64_jpeg} for base64_jpeg in images]
+ [doc.page_content for doc in relevant_documents]
+ [
f"Answer this question using the context images and text elements only: {state.question}",
]
)
# Generate the response utilizing the LLM
response = self.mannequin.generate_content(messages)
return {"response": response.textual content, "relevant_documents": relevant_documents}
The retriever queries the vector retailer to seek out the chunks most related to the person’s query. Subsequent, we construct a context for LLM (Gemini) that mixes textual content chunks and pictures to generate a response.
Full agent workflow
The agent workflow has two levels: an indexing stage and a query answering stage.
class DocumentRAGAgent:
def build_agent(self):
"""
Construct the RAG agent workflow.
"""
builder = StateGraph(DocumentRAGState)
# Add nodes for indexing and answering questions
builder.add_node("index_documents", self.index_documents)
builder.add_node("answer_question", self.answer_question)
# Outline the workflow
builder.add_edge(START, "index_documents")
builder.add_edge("index_documents", "answer_question")
builder.add_edge("answer_question", END)
self.graph = builder.compile()
Execution instance
if __name__ == "__main__":
from pathlib import Path# Import the primary agent to parse the doc
from document_ai_agents.document_parsing_agent import (
DocumentLayoutParsingState,
DocumentParsingAgent,
)
# Step 1: Parse the doc utilizing the primary agent
state1 = DocumentLayoutParsingState(
document_path=str(Path(__file__).dad and mom[1] / "information" / "docs.pdf")
)
agent1 = DocumentParsingAgent()
result1 = agent1.graph.invoke(state1)
# Step 2: Arrange the second agent for retrieval and answering
state2 = DocumentRAGState(
query="Who was acknowledged on this paper?",
document_path=str(Path(__file__).dad and mom[1] / "information" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
agent2 = DocumentRAGAgent()
# Index the paperwork
agent2.graph.invoke(state2)
# Reply the primary query
result2 = agent2.graph.invoke(state2)
print(result2["response"])
# Reply a second query
state3 = DocumentRAGState(
query="What's the macro common when fine-tuning on PubLayNet utilizing M-RCNN?",
document_path=str(Path(__file__).dad and mom[1] / "information" / "docs.pdf"),
pages_as_base64_jpeg_images=result1["pages_as_base64_jpeg_images"],
paperwork=result1["documents"],
)
result3 = agent2.graph.invoke(state3)
print(result3["response"])
This implementation completes the pipeline for doc processing, search, and query answering.
Let’s examine a working instance utilizing the documentation LLM and adaptation.pdf a set of 39 slides with textual content, equations, and figures (CC BY 4.0).
Step 1: Parse and summarize paperwork (Agent 1)
- execution time: It took a very long time to parse the 39 web page doc. 29 seconds.
- end result: Agent 1 creates an listed doc consisting of a piece abstract of every web page and a Base64 encoded JPEG picture.
Step 2: Query the doc (Agent 2)
We ask questions corresponding to:
“Explains LoRA and exhibits associated equations”
end result:
Retrieved web page:
