nameDataYkhmBOctzWE
titleData
You Can Learn AI Agent Observability & Eval In 20 Min | LLM Ops, Traces, Treg.to, Waku.one embedTexthttps://www.youtube.com/embed/YkhmBOctzWE
youtube_urlDatahttps://www.youtube.com/watch?v=YkhmBOctzWE
descriptionLong TextTry https://treg.to, Memory https://waku.one, Me https://seanchen.io, Open Source https://github.com/shenseanchen/waku-agent | Your agent just spent your money and you probably cannot say where it went. Observability is the camera that records what ran, how long and what it cost. Eval is the referee that decides whether the run was good enough to ship. I draw both out on the whiteboard, step by step, so you can follow it whether or not you come from a technical background, then I run two real research tasks live and trace every tool call, every token and every cent inside my own dashboard. On the second run, 53 of the 54 cents went to the model and 1 cent covered all seven tool calls.
Here's what we cover:
- Observability is a camera: what ran, how long, what it cost
- Eval is a referee: AI, code or a human judging whether the run was good
- The three things every call passes through: model, tools and memory
- The gateway and metering proxy, and why every step leaves a trace
- Reading a real trace: 5 agent loops, 10 tool calls with Treg.to, 70 seconds, 29 cents
- Six eval checks, including one that catches the agent lying about its own spend
- A second run at 54 cents, and the one tool call that failed
You Can Build Anything. You Can Learn Anything. 💪
treg (OpenRouter for agent tools): https://treg.to
Waku Agent (open source): https://github.com/ShenSeanChen/waku-agent
Waku portable memory: https://waku.one
Community, private Discord, twice-monthly live sessions, and the original .excalidraw files: https://join-us.seanchen.io
☕️ Buy me a coffee: https://buy.stripe.com/5kA176bA895ggog4gh
- Website & Community: https://seanchen.io
- X: https://x.com/ShenSeanChen
- Sean's Discord: https://discord.gg/tvECErKcFr
- Waku Agent Discord: https://discord.gg/tGsJ2nJH69
- LinkedIn: https://linkedin.com/in/shen-sean-chen
- Github Link: https://github.com/ShenSeanChen
- Instagram: https://www.instagram.com/sean_ai_stories
- TikTok: https://www.tiktok.com/@sean_ai_stories
Sub Count when launched: 38,677
Previous episodes:
Jev vs LLM, a new path to AGI: https://www.youtube.com/watch?v=RMQI_y1AvvA
GrokBot vs Meta Muse: https://www.youtube.com/watch?v=DFEF1mFDacU
AI Agent Harness & Loop Engineering: https://www.youtube.com/watch?v=GrNbuWWJYiI&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR
AI Agent Memory System: https://www.youtube.com/watch?v=mY3bR9qjZr4&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR
Waku Agent launch: https://www.youtube.com/watch?v=rvRyBhILrls&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR
Pi Minimal Coding Agent Harness: https://www.youtube.com/watch?v=0sI0MbCt4f4&list=PLE9hy4A7ZTmpGq7GHf5tgGFWh2277AeDR
Top Github Repos
- WakuAgent LocalAI Assistant ([update], was 19,09 stars): https://github.com/ShenSeanChen/waku-agent
- LaunchMVP NextJS,Supabase,Stripe (1,099 stars): https://github.com/ShenSeanChen/launch-mvp-stripe-nextjs-supabase
- ReactNextJSTips (133 stars): https://github.com/ShenSeanChen/yt-react-nextjs-tips
- RAGVectorDB (112 stars): https://github.com/ShenSeanChen/launch-rag
- DeepResearchAgentBackend (91 stars): https://github.com/ShenSeanChen/yt-DeepResearch-Backend
- SupabaseTipsKanban (69 stars): https://github.com/ShenSeanChen/yt-supabase-tips-kanban
- AgenticRAG (43 stars): https://github.com/ShenSeanChen/yt-agentic-rag
- DeepResearchAgentFrontend (37 stars): https://github.com/ShenSeanChen/yt-DeepResearch-Frontend
Chapters:
0:00 Why observability and eval matter
0:32 The two tools: treg and Waku Memory
0:55 Live run: one question, one API key
1:37 3,800 endpoints, 108 providers
2:49 The system design, harness call to eval
3:04 Your agents are spending your money
3:30 Observability is a camera
3:55 Eval is a referee
4:20 Breaking down what actually happens
4:54 Model calls, tool calls, memory
5:30 Memory as a shared brain across harnesses
6:00 Test: Claude Code recalls the research over MCP
9:23 Back to the product, reading the trace
10:27 What one agent turn looks like
11:09 This is observability
11:48 Loop by loop: 1.2s, 0.2s, 34.8s
12:40 The whole run: 70 seconds, 29 cents
14:08 The six eval checks, one by one
16:01 Second research, and a tool error
17:23 129 seconds, 54 cents, loop 6 the slowest
20:19 Final take: get your hands dirty
published_dateDate2026-10-06 02:57:06
is_publishedCheck0