# 10. Hangeulocyte analyzer (Kiwi, Kkma, Okt) + BM25 finder

## Combination of Hange Morphology Analyzer and BM25Retriever <a href="#bm25retriever" id="bm25retriever"></a>

Define the function to neatly check the output result.

```python
def pretty_print(docs):
    for i, doc in enumerate(docs):
        if "score" in doc.metadata:
            print(f"[{i+1}] {doc.page_content} ({doc.metadata['score']:.4f})")
        else:
            print(f"[{i+1}] {doc.page_content}")
```

### BM25Retriever with Kiwi talkizer <a href="#kiwi-bm25retriever" id="kiwi-bm25retriever"></a>

```python
# Install required libraries
# !pip install -qU kiwipiepy konlpy langchain-teddynote
```

```python
# For comparison BM25Retriever
from langchain_community.retrievers import BM25Retriever

# BM25Retriever using a custom-implemented Korean morphological analyzer (Kiwi)
from langchain_teddynote.retrievers import KiwiBM25Retriever

sample_texts = [
    "Financial insurance is a financial product designed for long-term asset management and risk management.",
    "Financial savings product insurance is a special financial product that has a long-term savings purpose as well as a livestock product provision function.",
    "Financial savings product insurance is a special financial product that has a long-term savings purpose as well as a livestock product provision function.",
    "Financial group bombing insurance is a product that focuses on risk management rather than savings. It is suitable for customers who are willing to take high risks.",
]
```

```python
# create a retriever.
kiwi = KiwiBM25Retriever.from_texts(sample_texts)
```

```python
# Perform a similarity search.
pretty_print(kiwi.invoke("fiannce and insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. 
```

Calculate similarity scores by personal needs and on metadata `score` Added the ability to add

```python
# perform a similarity search.
pretty_print(kiwi.search_with_score("financial insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2673) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2616) 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. (0.2463) 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. (0.2248) 
```

```python
bm25 = BM25Retriever.from_texts(sample_texts)

# BM25 KiwiBM25 비교
print(f'Kiwi:  \t {kiwi.invoke("finance and insurance")[0].page_content}')
print(f'BM25:  \t {bm25.invoke("finance and insurance")[0].page_content}')
```

```
 Kiwi: Financial insurance is a financial product designed for long-term asset management and risk handling. 
BM25: Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 
```

k value setting

```python
# create a retriever.
kiwi = KiwiBM25Retriever.from_texts(sample_texts)
kiwi.k = 2
# perform a similarity search.
pretty_print(kiwi.search_with_score("financial insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2673) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2616) 
```

### BM25Retriever used KonlPy (Kkma, Okt) <a href="#konlpykkma-okt-bm25retriever" id="konlpykkma-okt-bm25retriever"></a>

```python
# Using a custom implemented Korean morphological analyzer BM25Retriever
from langchain_teddynote.retrievers import (
    KkmaBM25Retriever,
    OktBM25Retriever,
)
```

```python
# create a retriever.
kkma = KkmaBM25Retriever.from_texts(sample_texts)
okt = OktBM25Retriever.from_texts(sample_texts)
```

```python
# kkma + BM25
pretty_print(kkma.invoke("finance and insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. 
[3] Mr. Financial Services, don't say anything, and do some savings or something. I don't know what's so urgent. 
[4] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 
```

```python
# perform a similarity search.
pretty_print(kkma.search_with_score("finance and insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2770) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2701) 
[3] Mr. Financial Services, don't say anything, and do some savings or something. I don't know what's so urgent. (0.2268) 
[4] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. (0.2260) 
```

```python
# okt + BM25
pretty_print(okt.invoke("finance and insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. 
```

```python
# perform a similarity search.
pretty_print(okt.search_with_score("finance and insurance"))
```

```
 [1] Financial insurance is a financial product designed for long-term asset management and risk handling. (0.2672) 
[2] Financial savings insurance is a special financial product with a long-term savings purpose, as well as the ability to provide livestock products. (0.2620) 
[3] Financial bombardment insurance is a product focused on risk preparedness for savings. Suitable for customers who want to take high risks. (0.2460) 
[4] Mr. Financial Services, don't say anything rough, and do some savings or something. I don't know what's so urgent. (0.2248) 
```