This hands-on workshop is designed for participants who already develop solutions using large language models (LLMs), such as RAG systems and AI agents, and want to make them reliable for production use. We will explore how to systematically evaluate the quality of LLM applications, monitor their performance, protect them against misuse, and optimise costs and response times. We will also try out local open-source models and model adaptation through fine-tuning (LoRA). In the practical sessions, we will enhance a RAG chatbot with evaluations, tracing, guardrails and cost monitoring. We will use the Python programming language throughout the workshop.
Content
- Evaluating LLM applications: gold-standard datasets, LLM-as-a-judge and comparison with human assessments, RAG evaluation metrics (RAGAS), and regression testing
Tracing and monitoring with Langfuse, Arize Phoenix and OpenTelemetry, and analysing errors using execution traces - Misleading inputs, prompt injection and tool failures; verifying improvements on a separate test set
Security and guardrails: red-teaming your own agent, input and output filtering, and personal data detection - Cost and speed optimisation: model routing, prompt caching, semantic caching and batch processing
Local open-source models: quantisation, Ollama and vLLM, and comparison with cloud models using your own evaluation dataset - Choosing between prompting, RAG and fine-tuning; preparing and validating training data (2 hours)
- Guided demonstration of SFT and LoRA using Hugging Face TRL/PEFT; comparison of base and fine-tuned models
- Final project: enhancing a RAG chatbot to create a reliable system with evaluations, tracing, guardrails and cost monitoring
- Using Python to develop reliable LLM solutions
Recommended prior knowledge
Suitable for developers and individuals who already use LLM APIs, RAG and AI agents and want to deploy them reliably in production.
Learning objectives
- Build an evaluation dataset and implement automated assessment of LLM applications
- Prepare development data and a separate test set covering typical cases, edge cases and adversarial examples
- Set up tracing and monitoring (observability) for LLM systems
- Identify and mitigate security risks, including prompt injection and data leakage
- Optimise the costs and response times of LLM solutions
- Use local open-source models and compare them with cloud models
- Assess when to use RAG or fine-tuning, and become familiar with SFT and LoRA
- Set up repeatable regression tests and prepare a system quality report
Target group
Developers and technical professionals who already use LLM APIs, RAG and AI agents and want to deploy them reliably in production.