AI Automation & Testing with Generative and Agentic AI
Instructor Led Online and In-Person Training Program
Course Program
AI Automation & Testing with Generative and Agentic AI
- Prerequisites: None — open to all professionals with an interest in testing and automation of AI applications
- Mode of Delivery: In-Person, Online (Live Virtual), and Classroom options available
- Duration:
- 6 weeks (36 hours) of guided instruction and resume preparation,
- 2 additional weeks dedicated to a hands-on Final Project
🔍 Course Overview
This instructor-led professional program offers a practical, end-to-end journey into automation and testing of Generative and Agentic AI systems using Python.
Across 12+ sessions, you’ll gain deep, hands-on experience with industry-standard tools and frameworks — including OpenAI APIs, Local LLMs, RAG applications, Testing/Evaluation with RAGAS and LangSmith, and end to end testing with Playwright — to automate, evaluate, and ensure the quality of AI-driven applications.
The curriculum progresses from foundational Python and AI concepts to designing and executing automated test frameworks for LLMs, RAG pipelines, and Agentic AI workflows, culminating in a portfolio-grade capstone project that demonstrates your ability to validate and optimize complex AI systems.
✅ What You’ll Gain
🚀 Master AI Testing & Automation
Learn how to design and automate tests for Generative and Agentic AI systems — from validating LLM responses and RAG precision to monitoring performance and safety in real-time.
💡 Evaluate AI Systems with Confidence
Use frameworks like LangSmith, RAGAS, and DeepEval to measure model accuracy, context relevance, and faithfulness — ensuring every AI output meets business and compliance standards.
🧠 Test and Orchestrate Agent Workflows
Validate multi-step reasoning, tool use, and decision-making across LangGraph-based agents to ensure reliability, consistency, and predictability in AI behavior.
📊 Automate End-to-End QA Pipelines
Integrate AI evaluation into CI/CD pipelines, track performance metrics, detect regressions, and apply Responsible AI principles in continuous testing environments.
🧰 Test Real-World AI Applications
Use Playwright to perform end-to-end testing of AI chatbots and copilots, simulating user interactions while capturing and scoring responses for quality assurance.
🏆 Build a Portfolio-Ready Project
Develop a complete AI Quality Automation Framework — combining test design, evaluation metrics, dashboards, and monitoring — supported by resume and interview preparation for QA, SDET, and AI Engineering roles.
👥 Who Should Enroll
QA Engineers & SDETs looking to upskill in testing AI applications
Software Developers exploring automation, validation, and performance optimization of LLM-based systems
AI/ML Practitioners who want to add testing and evaluation frameworks to their skillset
Product Managers & Technical Leads ensuring reliability, compliance, and quality in enterprise AI solutions
Professionals eager to transition into AI Quality Engineering, combining testing discipline with Generative AI innovation
“Growth begins at the edge of your comfort zone—keep pushing, keep evolving.”
What will be covered?
Course is divided in three sections.
Session 1: Python Intro + OpenAI API Setup
- Python fundamentals: syntax, variables, conditionals, loops, and functions
- Exception handling basics
- OpenAI API setup and environment configuration
- Installing and using the OpenAI Python package
- Lab: Write your first GPT chatbot using the OpenAI API
Session 2: Python for GenAI
- Python working with Data types: strings, lists, dictionaries, and sets
- Working with JSON, and API requests
- Key libraries: pandas, numpy, requests, re, os, dotenv
- Prompt engineering concepts with f-strings
- OpenAI moderation API
- Lab: Build a mini text-to-text transformer using OpenAI API including inbound moderation
Session 3: Working with Local LLMs
- Overview of open-source LLMs such as LLaMA, Mistral, Phi, Gemma, GPT-OSS
- Running models locally using Ollama
- Hands-on: Prompting, inference, and comparison with OpenAI models
- Lab: Run a local LLM and generate accounting or HR responses
Session 4: Retrieval-Augmented Generation (RAG)
- Introduction to embeddings and vector search
- Document loaders and chunking strategies
- RAG pipeline setup
- Lab: Build a document Q&A bot using your own PDFs or Document Chunks
Session 5: Vector Databases (Milvus)
- Understanding embeddings and cosine similarity
- Setting up Milvus locally
- Performing insertions, search, and similarity queries
- Lab: Index and retrieve text data using Milvus
Session 6: Fine-Tuning LLMs
- When and why to fine-tune a model
- Dataset preparation and JSONL format
- Fine-tuning with OpenAI
- Lab: Fine-tune OpenAI for classification or structured output
Session 7: Agentic AI Concepts
- What are agents and how they differ from chatbots
- Core agent loop: plan → act → observe → refine
- Introduction to LangGraph and multi-agent collaboration
- Lab: Create a two-agent conversation workflow
Session 8: MCP Protocol and Server Hosting
- Understanding Model Context Protocol (MCP)
- Hosting your own MCP server for custom AI tools
- Integrating MCP with OpenAI ecosystem
- Lab: Deploy a simple MCP service for external access
Session 9: Evaluation Frameworks
- Introduction to model evaluation and RAGAS metrics
- Context precision, recall, faithfulness, and answer correctness
- Evaluation measurement using Ragas
- Lab: Evaluate chatbot accuracy using LangSmith or RAGAS
Session 10: LangSmith for Evaluation and Observability
- Purpose of LangSmith and integration with LangChain/LangGraph
- Tracing model calls, tracking metrics, and analyzing responses
- Comparing different prompt or model versions
- Setting up datasets for structured evaluation
- Best practices for reproducibility and performance monitoring
- Lab: Evaluate chatbot accuracy and trace agent performance using LangSmith
Session 11: End-to-End Testing of AI Applications with Playwright
Introduction to Playwright for browser-based automation and testing
Setting up Playwright with Python or Node.js
Writing automated tests to simulate user interactions in AI-driven UIs (e.g., chatbots, copilots, dashboards)
Handling asynchronous responses, dynamic rendering, and streaming outputs
Capturing screenshots, console logs, and network responses for debugging
Lab: Build an end-to-end Playwright test that interacts with an AI chatbot interface, validates its response, and logs evaluation results
Session 12: Integrating Playwright with AI Evaluation Frameworks
Connecting Playwright test outputs to evaluation frameworks like LangSmith or RAGAS
Automating response scoring (accuracy, faithfulness, relevance) from captured UI results
Building regression tests for multi-turn AI conversations
Using Playwright in CI/CD pipelines for continuous validation of AI models and front-end behavior
Generating reports and dashboards for AI quality tracking
Lab: Create a hybrid automation pipeline — Playwright + LangSmith — to perform full-stack testing of an AI application (UI + backend + evaluation)
Session 13: Capstone Project — AI Automation & Testing
Work on a final end-to-end automation project of your choice with mentor guidance.
Students will plan, design, and implement a complete AI quality solution that demonstrates mastery of testing, evaluation, and deployment.
Example Project Options
AI Quality Automation Framework: Build a reusable test harness that evaluates model accuracy, hallucination rate, and latency using LangSmith or RAGAS.
Chatbot E2E Testing Suite: Create a Playwright-based workflow that simulates user conversations, validates responses, and generates reports.
Responsible AI Validator: Design an automated pipeline that detects bias, toxicity, and unsafe content using moderation and guardrail APIs.
RAG Validation System: Test a retrieval-augmented app’s grounding precision and faithfulness with automated regression runs.
Full-Stack AI App: Combine a FastAPI backend and React front end, connect the conversational UI with your AI services, and containerize the solution in Docker for deployment and testing.
Session 14: Capstone Presentation + Career Guidance
Final project presentations and peer feedback
Code reviews and deployment validation
GitHub portfolio optimization and presentation tips
Resume enhancements and positioning for AI/ML roles
Career paths and next steps in the AI/ML journey
Additional Topics
GitHub Portfolio & Open-Source Contributions
- How to set up a GitHub portfolio for ML projects
- Writing README files and project documentation
- Contributing to open-source ML projects
- Using Git and GitHub for version control
Creating AI Applications using Fast API Backend and React Frontend
- Creating a backend (FastAPI) and frontend(React) app
- Connecting the Conversational UI with the back end services
- Running it locally in a Docker container
Ready to upskill and lead in the era of AI?
Get hands-on with AI through expert-led, project-based training. Enroll today to transform your skills and accelerate your career in Artificial Intelligence!
If you have questions, please give us a call to talk to an advisor!