Advanced AI SecureOps: Attacking & Defending Enterprise AI Applications & Agents - Abhinav Singh - DCTSG2027
Description
Name of Training: Advanced AI SecureOps: Attacking & Defending Enterprise AI Applications & Agents
Trainer(s): Abhinav Singh
Skill Level: Intermediate/Advanced
Dates: March 5 – March 6, 2027
Time: 9 am to 5 pm
Venue: TBD
Early Bird: S$3,300 (incl. GST, until Dec 31, 2026)
Regular: S$3,600 (incl. GST)
Short Summary
Can a prompt injection turn into RCE, backend compromise, or even an infrastructure takeover? In this hands-on, CTF-style training, you’ll attack and defend real AI applications, agents, and MCP-connected systems exploiting prompt injection, poisoned context, tool abuse, unsafe orchestration, and trust failures. Then build the defenses: guardrails, red/blue team workflows, monitoring, incident response, and enterprise security controls for modern AI systems.
Top 3 Takeaways
- Learn how to identify, exploit, and defend against real-world attacks on AI applications, agents, and tool-connected systems, including prompt injection, jailbreaks, agent abuse, and chained compromise paths.
- Build practical defensive capabilities for enterprise AI, including guardrails, security scanners, monitoring, and response patterns for public, private, and MCP-enabled AI services.
- Gain hands-on experience using modern AI techniques for security testing, validation, and red/blue teaming, including judge-LLM workflows, attack automation, and securing agentic AI supply chains.
Course Description
Can prompt injections lead to complete infrastructure takeovers? Could AI agents, MCP-connected tools, or poisoned external context be abused to compromise backend services? Can data poisoning in AI copilots impact a company’s stock? Can jailbreaks create false crisis alerts in security systems? This immersive, CTF-styled training in AI, LLM, agent, and MCP security dives into these pressing questions. Engage in realistic attack-and-defense scenarios focused on real-world threats, from prompt injection and remote code execution to backend compromise, tool abuse, unsafe agent orchestration, and MCP-specific trust and authorization failures. Tackle hands-on challenges with live AI applications to understand vulnerabilities and build robust defenses. Learn how to build a comprehensive security pipeline, master AI red and blue team strategies, secure tool-connected and agentic systems, implement resilient guardrails for LLMs, and handle incident response for AI-based threats. You will also explore governance, Responsible AI, and enterprise security patterns for modern AI ecosystems.
By the end of this training, you will be able to:
- Exploit vulnerabilities in AI applications to achieve code and command execution, uncovering scenarios such as instruction injection, agent control bypass, remote code execution for infrastructure takeover, as well as chaining multiple agents for goal hijacking.
- Conduct AI red-teaming using adversary simulation, OWASP LLM Top 10, and MITRE ATLAS frameworks, while applying AI security and ethical principles in real-world scenarios.
- Execute and defend against adversarial attacks, including prompt injection, data poisoning, jailbreaks, agentic attacks, and insecure tool-connected workflows.
- Perform advanced AI red and blue teaming through multi-agent auto-prompting attacks, implementing a 3-way autonomous system consisting of attack, defend, and judge models.
- Build and deploy enterprise-grade LLM defenses, including custom guardrails for input/output protection, security benchmarking, penetration testing of LLM agents, and defensive controls for MCP-enabled integrations.
- Understand MCP fundamentals and assess how they expand the attack surface of modern AI systems.
- Establish a comprehensive LLM SecOps process to secure the supply chain from adversarial attacks and create a robust threat model for enterprise applications, including AI systems connected to external tools and data sources through MCP-like architectures.
- Implement an incident response and risk management plan for enterprises developing or using AI services.
Course Outline
View full outline
Hide outline
Day 1 - Morning
Introduction
- Introduction to LLM and AI.
- Terminologies and architecture.
- Transformers, Attention & their security implications (hallucinations, jailbreaks etc).
- Agents, multi-agents and multi-modal models.
- Introduction to tool-connected AI systems and MCP as an emerging standard for connecting agents to external tools, data, and workflows.
Elements of AI Security (1 lab)
- Understanding AI vulnerabilities with case studies on AI security breaches.
- OWASP LLM Top 10 and MITRE mapping of attacks on AI supply chain.
- Threat modeling of AI Applications, tool-connection and MCP-enabled architectures, including trust boundaries across hosts, clients, servers, tools, resources, and external systems.
Adversarial LLM Attacks and Defenses (6 labs)
- (What, Why & how’s) Direct and indirect prompt injection attacks and their subtypes.
- Advanced prompt injections through obfuscation and cross-model injections.
- Breaking system prompts and their trust criteria.
- Indirect prompt injections through external input sources.
Day 1 - Afternoon
Responsible AI & Jailbreaking (6 labs)
- Jailbreaking public LLMs covering adversarial AI, offensive security, and CBRN use-cases.
- Responsible use and governance implications of increasingly autonomous, tool-connected AI systems.
- Model alignment, system prompt optimization, and defense.
Building Enterprise-grade LLM Defenses (2 labs)
- Deploying LLM security scanner, adding custom rules, prompt block-lists, and guardrails.
- Writing custom detection logic, trustworthiness checks, and filters.
- Building security log monitoring and alerting for models using open-source tools.
- LLM security benchmarking and continuous reporting.
Day 2 - Morning
Red & Blue Teaming of Enterprise AI applications & Agents(4 labs)
- Business control flow testing for risky responses & misaligned behavior of applications.
- Using Colab notebooks for automation of API calls and reporting
- Vector database and model-weight tracing for root-cause investigation.
- Rainbow teaming through a 3-way LLM implementation: target, attacker, and judge with self-improving attack prompts.
Attacking & Defending Agentic Systems (5 labs)
- Threat modeling of agentic and multi-agent systems, including planning loops, memory, tool invocation, delegation, trust - boundaries, and escalation paths.
- Attacking LLM agents for task manipulation, risky behavior and PII disclosure in RAG.
- Injection attacks on AI agents for code and command execution.
- Compromising backend infrastructure by abusing over-permissioning and tool usage in agentic systems.
- Multi-agent attacks causing privilege too calls, goal manipulation & chained escalations.
- Defense patterns for agentic systems, including observability, approval gates, scoped permissions, secure delegation, and runtime tracing for high-risk actions.
Day 2 - Afternoon
MCP Security & Defensive Architecture (1 lab).
- MCP fundamentals & security for agentic systems: protocol basics, trust-boundary changes, key risks like malicious servers and over-broad permissions, plus a browser-based exploit-and-defend lab.
- Defense patterns for MCP-enabled systems with protection architectures.
Building AI SecOps Process
- Summarizing the learnings into a SecOps workflow.
- Monitoring trustworthiness, safety and security of enterprise AI applications.
- Implementing NIST AI Risk Management Framework (RMF) for security monitoring.
Difficulty Level
Intermediate - The student has education and some experience in the field and familiarity with the topic being presented. The student has foundational knowledge that the course will leverage to provide practical skills on the topic.
Suggested Prerequisites
General security knowledge and comfort with technical concepts.
Familiarity with AI and machine learning concepts is helpful, but not required.
Basic familiarity with Python, APIs, or notebooks is useful for some exercises.
Familiarity with common GenAI platforms and workflows is recommended.
What Students Should Bring
A laptop with a modern browser and reliable internet access.
An OpenAI API key.
A Google Colab account.
Completion of the pre-training setup before Day 1
What the Trainer Will Provide
One year of access to a live, interactive playground with exercises covering attack and defense scenarios for GenAI, LLM, and agentic applications. An "AI SecureOps" metal coin for CTF players. A complete course guide in PDF format with step-by-step instructions for labs, exercises, and concepts covered in the training. PDF copies of the slides used during the course. Access to a dedicated Slack channel for continued discussion, support, and collaboration. Access to a GitHub repository with custom-built source code and tools. Access to Hugging Face models, datasets, and transformers used in the training environment.
Trainer(s) Bio
Abhinav Singh is a security leader, founder of Wingback Security, and a globally recognized speaker and trainer focused on securing enterprise AI systems. He has been involved with AI fellowship and research communities including MATS, PIBBSS, CSA, AIUC, and the Foresight Institute. He is a frequent speaker and trainer at Black Hat, RSAC, DEF CON, OWASP, BruCon, NeurIPS, and other major conferences worldwide. He is also the author of “Metasploit Penetration Testing Cookbook” and “Instant Wireshark Starter”. His work spans patents, open-source tools, academic & research publications, and bridges advanced AI security research with practical enterprise defense, with a focus on securing AI applications, agents, and enterprise AI infrastructure.
Proficiency Exam
Students are required to score more than 1400 out of 2400 in the CTF. Once day-2 ends, a bit of grace period is given after which attendees have to share their CTF user name, their registration email and final score for validation.
Registration Terms and Conditions
Trainings are refundable before January 29, 2027, minus a non-refundable processing fee of $350.
Between January 30, 2027 and February 28, 2027 partial refunds will be granted, equal to 50% of the course fee minus a processing fee of $350.
All trainings are non-refundable after Feb 28, 2027.
Training tickets may be transferred to another student. Please email us at training@defcon.org for specifics.
If a training does not reach the minimum registration requirement, it may be cancelled. In the event the training you choose is cancelled, you will be provided the option of receiving a full refund or transferring to another training (subject to availability).
Failure to attend the training without prior written notification will be considered a no-show. No refund will be given.
DEF CON Training may share student contact information, including names and emails, with the course instructor(s) to facilitate sharing of pre-work and course instructions. Instructors are required to safeguard this information and provide appropriate protection so that it is kept private. Instructors may not use student information outside the delivery of this course without the permission of the student.
By purchasing this ticket you agree to abide by the DEF CON Training Code of Conduct and the registration terms and conditions listed above.
Several breaks will be included throughout the day. Please note that food is not included.
All courses come with a certificate, contingent on attendance at 75% of course sessions. Students who pass the end-of-course evaluation will be issued a certificate of proficiency. All other students with sufficient course attendance will be issued a certificate of completion. Students are responsible for making sure their attendance is recorded at the start of the day and after returning from lunch.
