HTB AI Security Lab Series

Overview

This project documents my public Hack The Box Academy study across AI security and adversarial machine learning topics. It is designed to show the concepts I studied while preparing for and building skills around AI application security, offensive AI testing, adversarial evasion, privacy, and defense.

This page does not contain HTB Certified Offensive AI Expert (COAE) exam questions, exam solutions, flags, proprietary exam scenarios, or restricted certification content. It only summarizes public Academy subject areas and the security concepts I practiced.

Public Academy Topics Completed

My AI security study included the following public HTB Academy modules:

Area	Public Academy module
AI foundations	Fundamentals of AI
AI in security	Applications of AI in InfoSec
Offensive AI	Introduction to Red Teaming AI
Prompt security	Prompt Injection Attacks
Data security	AI Data Attacks
LLM behavior	LLM Output Attacks
Application security	Attacking AI - Application and System
Adversarial ML	AI Evasion - Foundations
Adversarial ML	AI Evasion - First-Order Attacks
Adversarial ML	AI Evasion - Sparsity Attacks
Privacy	AI Privacy
Defense	AI Defense

Skills Demonstrated

• AI security fundamentals
• LLM application threat modeling
• Prompt injection awareness
• Data poisoning and data integrity concepts
• Unsafe output and downstream risk analysis
• AI application attack surface review
• Adversarial example and evasion concepts
• Privacy risk analysis
• Defensive control thinking for AI systems
• Security testing methodology
• Risk communication
• Translating offensive techniques into defensive recommendations

Security Areas Explored

Prompt Injection

Prompt injection testing focuses on how untrusted input can manipulate model behavior, override intended instructions, or influence how an AI enabled application uses connected tools and data.

From a defensive perspective, important control ideas include:

• Treating model input as untrusted data
• Separating system instructions from user supplied content
• Applying least privilege to tools and integrations
• Validating model initiated actions outside the model
• Limiting access to sensitive context
• Monitoring suspicious or repeated instruction manipulation attempts

AI Data Attacks

AI systems depend heavily on the quality and integrity of their data. Data attacks can target training data, retrieval data, fine tuning data, or other information that influences model behavior.

Defensive considerations include:

• Data provenance
• Integrity validation
• Access control around datasets
• Change monitoring
• Dataset review and versioning
• Separating trusted and untrusted data sources

LLM Output Attacks

AI generated output can become dangerous when another application interprets it as trusted code, markup, commands, or instructions.

The defensive lesson is that model output should not automatically be trusted simply because it was generated by the model. Output validation, encoding, sandboxing, and application level authorization remain necessary.

AI Application and System Security

AI risk is not limited to the model itself. The surrounding application can include APIs, plugins, retrieval systems, authentication, authorization, data stores, and automation components.

A practical assessment therefore considers:

• Model behavior
• Application logic
• API security
• Identity and authorization
• Data exposure
• Tool permissions
• Retrieval and context sources
• Logging and monitoring
• Failure handling

Adversarial Evasion

Adversarial machine learning explores how inputs can be modified to influence model predictions or bypass expected detection behavior.

My public Academy study covered foundational evasion concepts, first order approaches, and sparsity focused attacks. From a defensive perspective, this reinforced the importance of layered controls, robust validation, monitoring, and understanding that model confidence does not automatically equal trustworthiness.

AI Privacy

AI systems can create privacy risks through training data exposure, model memorization, excessive context access, or poorly controlled integrations.

Useful defensive principles include:

• Data minimization
• Access control
• Sensitive data filtering
• Retention limits
• Privacy aware logging
• Segmentation of confidential context
• Reviewing what information an AI system can retrieve or expose

AI Defense

AI defense requires combining traditional application security with AI specific safeguards. Controls should be placed around the model rather than assuming the model can reliably police itself.

Defensive areas include:

• Input validation
• Output validation
• Authentication and authorization
• Least privilege tool access
• Monitoring and logging
• Rate limiting
• Data governance
• Human approval for high impact actions
• Secure architecture around model integrations

Defensive Takeaways

The most important lesson from this study is that AI security is still security engineering. Strong controls around identity, data, application logic, monitoring, least privilege, and validation remain essential even when a large language model or machine learning system is involved.

Offensive AI testing is useful because it exposes where assumptions break. Those observations can then be translated into practical defensive controls, detection opportunities, and safer application design.

Related Credential

I earned the HTB Certified Offensive AI Expert (COAE) credential after completing the associated learning path and certification process.

View my verified COAE credential

Disclosure

This writeup intentionally stays at the public training and conceptual level. It does not reproduce certification exam tasks, proprietary datasets, flags, answers, or restricted Hack The Box content.