01. Structure of Document

Document & Document Loaders

Reference

• Main loader used in LangChain

• List of loaders used in LangChain

Documents utilized for practice

Software Policy Institute (SPRi)-December 2023

• Author: Jaeheung Lee (AI Policy Institute Office Liability Institute), Lee Ji-soo (AI Policy Lab Yi Phyang Institute)

• link: https://spri.kr/posts/view/23669

• File name: SPRI_AI_Brief_2023년12월호_F.pdf

Document

This is the basic document object of LangChain.

property - page_content : A string representing the content of the document. - metadata : A dictionary representing the document's metadata.

from langchain_core.documents import Document

document = Document("Hello, this is Langchain's document.")

# Check the properties of a document
document.__dict__

{'id': None,'metadata': {},'page_content':'Hello? This is Langchain's Tokemand','type':'Document' } 

Add properties to metadata

# Add metadata
document.metadata["source"] = "TeddyNote"
document.metadata["page"] = 1
document.metadata["author"] = "Teddy"

# Check the properties of a document
document.metadata

 {'source':'TeddyNote','page': 1,'author':'Teddy'} 

Document Loader

It serves to convert content from various file formats to Document objects.

Main Loader

• PyPDFLoader: A loader that loads PDF files.

• CSVLoader: A loader that loads CSV files.

• UnstructuredHTMLLoader: A loader that loads HTML files.

• JSONLoader: A loader that loads JSON files.

• TextLoader: A loader that loads text files.

• DirectoryLoader: A loader that loads directories.

# Example file path
FILE_PATH = "./data/SPRI_AI_Brief_2023년12월호_F.pdf"

from langchain_community.document_loaders import PyPDFLoader

# Loader Settings
loader = PyPDFLoader(FILE_PATH)

load()

• Load and return documents.

• Returned results List[Document] Form.

# PDF loader
docs = loader.load()

# Check the number of loaded documents
len(docs)

23

# Check the first document
docs[0]

load_and_split()

• Split and return documents using splitter.

• Returned results List[Document] Form.

from langchain_text_splitters import CharacterTextSplitter

# Setting the text divider
text_splitter = CharacterTextSplitter(chunk_size=200, chunk_overlap=0)
# Split document
docs = loader.load_and_split(text_splitter=text_splitter)

# Check the number of loaded documents
len(docs)
# Check the first document
docs[0]

 Document (metadata={'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 0}, page_content='December 2023')

lazy_load()

• Load documents in a generator way.

# generator Load document in this way
for doc in loader.lazy_load():
    print(doc.metadata)

 {'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 0} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 1} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 2} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 3} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 4} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 5} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 6} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 7} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 8} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 9} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 10} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 11} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 12} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 13} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 14} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 15} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 16} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 17} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 18} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 19} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 20} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 21} 
{'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 22} 

aload()

• Loading documents in asynchronous (Async)

# The document async Load in a manner
adocs = loader.aload()

# load documentation
await adocs

 [Document (metadata={'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 0}, page_content=' 12/2023 issue'), Document (metadata<TA Artificial Industry Trend Brief\n 1. United States ▹ United States, safe and reliable AI development and use executive order · · · · · · · · · · · · · · · · · · · · · G7, Hiroshima AI process to AI company target international action force · · Submission of AI comments in terms of consumer protection and competition to the Copyright Office ················ 5\n ▹ EU AI law 3rd party negotiation, based model regulation related views, ovulation,································ Corporate/Industry \n ▹ American Frontier Model Forum, 1,  $0 million AI Safety Fundraising ································ 7\n ▹ Cohir, Data Sources to Ensure Data Transparency Explorer Disclosure ············  
... 
(meditation) 
... 
Conference \non Artificial \nIntelligence\n-AI Development Association Conference (AAAI) promotes AI research, provides opportunities for exchanges between AI fields \n researchers, practitioners, scientists, students and engineers \N-Conference announces AI-related skills, special tracks, Invited speakers, \nworkshop, tutorial, poster session, topicn, exhibition Document (metadata={'source':'./data/SPRI_AI_Brief_2023 December issue_F.pdf','page': 22}, page_content='Homepage: https://spri.kr/\n보고서와 Inquiries related to AI Policy Lab (jayoo@spri.