
giskard-oss giskard-checks/v1.0.4
ð¢ LLMãšãŒãžã§ã³ãåãã®ãªãŒãã³ãœãŒã¹è©äŸ¡ã»ãã¹ãã©ã€ãã©ãª
ãšãŒãžã§ã³ãåã·ã¹ãã ã®ããã®è©äŸ¡ãã¬ããããŒãã³ã°ããã¹ãçæ
ã¢ãžã¥ãŒã«åŒã軜éãåçãéåæãã¡ãŒã¹ã
ããã¥ã¡ã³ã ⢠ãŠã§ããµã€ã ⢠ã³ãã¥ããã£
[!IMPORTANT] Giskard v3 ã¯ãAIãšãŒãžã§ã³ãã®åçãªãã«ãã¿ãŒã³ãã¹ãã®ããã«èšèšããããæ°ããªæžãçŽãçã§ãããã®ãªãªãŒã¹ã§ã¯ãããé«ãå¹çæ§ã®ããã«ééçŽã®äŸåé¢ä¿ãåé€ãã€ã€ããã匷åãªAIè匱æ§ã¹ãã£ããŒãšåŒ·åãããRAGè©äŸ¡ãå°å ¥ããŠããŸãããããã¯çŸåšãv2ãžã®äŸåãªãã«
giskard-scanã«ãã€ãã£ãã«æèŒãããŠããŸãã衚圢åŒ/MLã¢ãã«åãã®ã¬ã¬ã·ãŒã¹ãã£ã³ã®ã¿ãv2å°çšãšããŠæ®ã£ãŠããŸãã Giskard v2 ã¯åŒãç¶ãå©çšå¯èœã§ããããã¯ãç©æ¥µçã«ã¡ã³ããã³ã¹ãããŠããŸããã 鲿ããã©ã㌠â v3 çºè¡šãèªã · ããŒãããã
ã€ã³ã¹ããŒã«
pip install giskard # checks (+ agents, llm, core)
pip install "giskard[scan]" # + vulnerability / quality scan
pip install "giskard[openai]" # provider SDK for LLM judges / generators
Python 3.12+ ãå¿ èŠã§ãã
| Extra | 远å ããããã® |
|---|---|
| (ãªã) | giskard-checks ãšäŸåé¢ä¿ |
scan | giskard-scan |
openai / anthropic / ⊠| ãããã€ã㌠SDKïŒpyproject.toml ã®ãªãã·ã§ã³äŸåé¢ä¿ãåç
§ïŒ |
ãã¬ã¡ããªãŒ: giskard-core ã«ããä»»æã®éèšåæãããã³ãããåºåã¯éä¿¡ãããŸããã
export DO_NOT_TRACK=1 ãŸã㯠export GISKARD_TELEMETRY_DISABLED=1ïŒãããã¯äœæ¥ãã£ã¬ã¯ããªã® .env ãã¡ã€ã«ã«åãããŒãèšå®ïŒã§ãªããã¢ãŠãã§ããŸãã~/.giskard/id ã®äœæãã¹ãããããã«ã¯ã€ã³ããŒãåã«èšå®ããŠãã ãããåŸããèšå®ããå Žåã§ã以éã®éä¿¡ã¯åæ¢ããŸãã
詳现: giskard-core READMEã
Giskard 㯠ãšãŒãžã§ã³ãåã·ã¹ãã ã®ãã¹ããšè©äŸ¡ ãè¡ããªãŒãã³ãœãŒã¹ã® Python ã©ã€ãã©ãªã§ããv3 ã¢ãŒããã¯ãã£ã¯ããããããå¿ èŠãªäŸåé¢ä¿ã®ã¿ãæã€ãçŠç¹ãçµã£ãã¢ãžã¥ãŒã«åŒããã±ãŒãžçŸ€ã§ãããLLMããã©ãã¯ããã¯ã¹ãšãŒãžã§ã³ãã倿®µéãã€ãã©ã€ã³ãªã©ãããããã®ãã©ããã§ãããããŒãããæ§ç¯ãããŠããŸãã
| ã¹ããŒã¿ã¹ | ããã±ãŒãž | 説æ |
|---|---|---|
| â å®å® | giskard-checks | ãã¹ããšè©äŸ¡ â ã·ããªãª APIãçµã¿èŸŒã¿ãã§ãã¯ãLLM-as-judge |
| â å®å® | giskard-scan | ãšãŒãžã§ã³ãè匱æ§ã¹ãã£ã㌠+ RAG/å質è©äŸ¡ â ã¬ããããŒãã³ã°ãããã³ããã€ã³ãžã§ã¯ã·ã§ã³ããžã§ã€ã«ãã¬ã€ã¯ãæå®³ã³ã³ãã³ãïŒvulnerability_scanãv2 Scan ã®åŸç¶ïŒãããã³ãã¬ããžããŒã¹å質è©äŸ¡ïŒquality_scanãv2 RAGET ã®åŸç¶ïŒ |
ããã㯠3 ã€ã®åºç€ã©ã€ãã©ãª â giskard-coreïŒå
±æãŠãŒãã£ãªãã£ãšãã¬ã¡ããªãŒïŒãgiskard-llmïŒãããã€ããŒéäŸåã® LLM ã«ãŒãã£ã³ã°ïŒãgiskard-agentsïŒãšãŒãžã§ã³ããšã¯ãŒã¯ãããŒã®ãªãŒã±ã¹ãã¬ãŒã·ã§ã³ïŒâ ã®äžã«æ§ç¯ãããŠããããããã¯èªåçã«åã蟌ãŸããçŽæ¥äœ¿çšãããããšã¯ã»ãšãã©ãããŸããã
Giskard Checks â ãšãŒãžã§ã³ãã®ãã¹ãçš eval ã®äœæãšé©çš
pip install giskard-checks
Giskard Checks ã¯ãLLMããŒã¹ã®ã·ã¹ãã ããã¹ãããè©äŸ¡ïŒevalsïŒãäœæããããã®è»œéã©ã€ãã©ãªã§ã â åçŽãªã¢ãµãŒã·ã§ã³ããLLM-as-judgeã«ããè©äŸ¡ãŸã§å¯Ÿå¿ããŸããåŸæ¥ã®ãŠããããã¹ããšã¯ç°ãªããevalsã¯é決å®çãªåºåïŒåãå ¥åã§ãç°ãªãæå¹ãªå¿çãçæããããïŒåãã«èšèšãããŠããŸãã
Giskard Checks ã®çšé:
- ãªã°ã¬ãã·ã§ã³ã®æ€åº â 倿ŽåŸãã·ã¹ãã ãæ£ããåäœããããšãæ€èšŒ
- RAGåè³ªã®æ€èšŒ â åçãååŸããã³ã³ããã¹ãã«åºã¥ããŠãããã確èª
- å®å šæ§ã«ãŒã«ã®é©çš â åºåãã³ã³ãã³ãããªã·ãŒã«æºæ ããŠããããšãä¿èšŒ
- ãã«ãã¿ãŒã³ãšãŒãžã§ã³ãã®è©äŸ¡ â åäžã®ããåãã ãã§ãªããäŒè©±å šäœããã¹ã
çµã¿èŸŒã¿ã®evalsã«ã¯ãæååãããã³ã°ãæ¯èŒãæ£èŠè¡šçŸãæå³çé¡äŒŒåºŠãLLM-as-judgeãã§ãã¯ïŒGroundednessãConformityãLLMJudgeïŒãå«ãŸããŸãã
æŠå¿µ
- Target â ãã¹ã察象ã®ã·ã¹ãã : ä»»æã®åæ/éåæåŒã³åºãå¯èœãªããžã§ã¯ã
(inputs) -> outputsïŒãªãã·ã§ã³ã§traceä»ãïŒ - Scenario â 1ã€ã®eval: ã€ã³ã¿ã©ã¯ã·ã§ã³ + ãã§ãã¯
- Check â ãã¬ãŒã¹ã«å¯Ÿããã¢ãµãŒã·ã§ã³ãŸãã¯LLMãžã£ããž
- Suite â è€æ°ã®ã·ããªãªããŸãšããŠå®è¡
giskard.agents.Generator ã¯ã¯ãŒã¯ãããŒ/ãžã£ããžçšã®LLMã¯ã©ã€ã¢ã³ãã§ããã
ãŠãŒã¶ãŒã¡ãã»ãŒãžãåæãã giskard.checks ã®å
¥åãžã§ãã¬ãŒã¿ãŒïŒLLMGeneratorïŒãšã¯ç°ãªããŸãã
ã¯ã€ãã¯ã¹ã¿ãŒã
ã€ãã³ãã«ãŒããå®è¡ãããŠããªãã¹ã¯ãªããå ã§:
from giskard.checks import Scenario, Groundedness
def get_answer(inputs: str) -> str:
return "Paris" # replace with your model / agent
scenario = (
Scenario("test_france_capital")
.interact(inputs="What is the capital of France?", outputs=get_answer)
.check(
Groundedness(
name="answer is grounded",
context="France is in Western Europe. Its capital is Paris.",
)
)
)
result = scenario.run_sync()
result.print_report()
ããŒãããã¯ã®ã»ã«å ããŸãã¯éåæé¢æ°å ã§ã¯ã代ããã«ã·ããªãªã await ããŠãã ãã:
result = await scenario.run()
result.print_report()
Groundedness 㯠LLM ãžã£ããžã§ã â ãããã€ããŒã® extra ãã€ã³ã¹ããŒã«ãïŒäŸ: pip install "giskard[openai]"ïŒã察å¿ãã API ããŒãèšå®ããŠãã ãããããã©ã«ãã¢ãã«: openai/gpt-4o-miniã
SuitesãLLMJudgeããã«ãã¿ãŒã³ã·ããªãªãªã©ã®è©³çްã«ã€ããŠã¯ãå®å
šãªããã¥ã¡ã³ããåç
§ããŠãã ããã
Giskard Scan â AI ãšãŒãžã§ã³ãåãè匱æ§ã¹ãã£ããŒ
pip install "giskard[scan]" # or: pip install giskard-scan
Giskard Scan ã¯ããšãŒãžã§ã³ãåã·ã¹ãã åãã®ã¬ããããŒãã³ã°ããã³è匱æ§ã¹ãã£ã³ã®ã¬ã€ã€ãŒã§ãããšãŒãžã§ã³ãã®å¹³æãªèšèã«ãã説æãããããã³ããã€ã³ãžã§ã¯ã·ã§ã³ãæå®³ã³ã³ãã³ããã¹ãã¬ãªã¿ã€ãã誀æ å ±ãªã©ãã«ããŒããæµå¯Ÿçãã¹ãã¹ã€ãŒããèªåçã«çæããŸãã
Giskard Scan ã¯æ¬¡ã®çšéã«äœ¿çšããŸãïŒ
- ãšãŒãžã§ã³ãã®ã¬ããããŒãã³ã° â OWASP LLM Top-10 ã®è åšã«ããŽãªå šäœã«ãããæµå¯Ÿçå ¥åãèªåçæ
- ããã³ããã€ã³ãžã§ã¯ã·ã§ã³ãããŒãã®å®è¡ â ããã«äœ¿ããã€ã³ãžã§ã¯ã·ã§ã³ãã€ããŒãã®çµã¿èŸŒã¿ããŒã¿ã»ãã
- ã«ã¹ã¿ã ãžã§ãã¬ãŒã¿ãŒã«ããæ¡åŒµ â ç¬èªã®
ScenarioGeneratorã€ã³ã¹ã¿ã³ã¹ãgenerate_suiteã«æž¡ãããvulnerability_suite_generator_registryã«ç»é²
ã¯ã€ãã¯ã¹ã¿ãŒã
import asyncio
from giskard.scan import vulnerability_scan
async def my_agent(inputs: str) -> str:
# Replace with your agent / model call
return f"Echo: {inputs}"
async def main() -> None:
await vulnerability_scan(
target=my_agent,
description="A customer support chatbot for an e-commerce platform.",
languages=["en"],
)
asyncio.run(main())
ã¹ãã£ã³ãžã§ãã¬ãŒã¿ãŒãåæ§ã«ãLLMãããã€ããŒã®extraãšAPIããŒãå¿ èŠã§ãïŒäžèšã®Checksã®ãžã£ããžãšåãã§ãïŒã
Giskard v2ããæ¢ãã§ããïŒ
Giskard v2ã«ã¯ScanïŒèªåèåŒ±æ§æ€åºïŒãšRAGETïŒRAGè©äŸ¡ãã¹ãã»ããçæïŒãå«ãŸããŠããŸããã
LLMãšãŒãžã§ã³ãã«ã€ããŠã¯ãv3ã§ã¯äž¡æ¹ãšãgiskard-scanã«åã£ãŠä»£ããããŠããŸããv2ã®LLMã¹ãã£ã³ã®ä»£ããã«vulnerability_scanããRAGETã®ä»£ããã«quality_scanïŒKnowledgeBaseãšäœµçšïŒã䜿çšããŠãã ããã
v3ã¯MLã¢ãã«ã§ãåäœããŸããã¢ãã«ãã¿ãŒã²ãããšããŠã©ããããgiskard-checksãŸãã¯giskard-scanã§è©äŸ¡ã§ããŸãã以äžã®äŸã§æ±ã£ãŠããã®ã¯ãv2å°çšã®èªå衚圢åŒã¹ãã£ã³ïŒgiskard.Model + giskard.Datasetãå
çããŠããã©ãŒãã³ã¹ããã€ã¢ã¹ãããã¹ããã¹ã®åé¡ãèªåæ€åºããæ€åºåšã¹ã€ãŒãïŒãšãgiskard.testing MLãã¹ãã¹ã€ãŒãããããŠGiskard Hubã§ãããããã¯v3ã§ã®æäŸã¯äºå®ãããŠããŸããã
pip install "giskard[llm]>2,<3"
Scan â ããã©ãŒãã³ã¹ããã€ã¢ã¹ãã»ãã¥ãªãã£ã®åé¡ãèªåæ€åº
ã¢ãã«ãã©ããããŠã¹ãã£ã³ãå®è¡ããŸã:
import giskard
import pandas as pd