14. Automating evaluation using online evaluation

Online Evaluators

Sometimes you want to evaluate the products recorded in your project.

# 설치
# !pip install -qU langsmith langchain-teddynote

# API KEY를 환경변수로 관리하기 위한 설정 파일
from dotenv import load_dotenv

# API KEY 정보로드
load_dotenv()

 True 

# LangSmith 추적을 설정합니다. https://smith.langchain.com
# !pip install -qU langchain-teddynote
from langchain_teddynote import logging

# 프로젝트 이름을 입력합니다.
logging.langsmith("CH16-Auto-Evaluation-Test")

 Start tracking LangSmith. 
[Project name] 
CH16-Auto-Evaluation-Test 

Chain setting for online Evaluation

from myrag import PDFRAG
from langchain_openai import ChatOpenAI

# PDFRAG 객체 생성
rag = PDFRAG(
    "data/SPRI_AI_Brief_2023년12월호_F.pdf",
    ChatOpenAI(model="gpt-4o-mini", temperature=0),
)

# 검색기(retriever) 생성
retriever = rag.create_retriever()

# 체인(chain) 생성
rag_chain = rag.create_chain(retriever)

from langchain_core.runnables import RunnableParallel, RunnablePassthrough

# RunnableParallel 객체 생성합니다.
evaluation_runnable = RunnableParallel(
    {
        "context": retriever,
        "answer": rag_chain,
        "question": RunnablePassthrough(),
    }
)

Run the test chain to see if Runs reflects the results.

# 체인 실행
_ = evaluation_runnable.invoke("삼성전자가 자체 개발한 생성형 AI의 이름은 무엇인가요?")

Online LLM-as-judge creation

Online LLM-as-judge creation

Secrets & API Keys designation (OpenAI API Key)

Provider, Model, Prompt Settings

Provider, Model, Prompt Settings

For facts, specify output.context (change to suit your settings)

answer designates output.answer (changed to suit your settings)

Preview ensures that data is entered in the correct place