What to study for each CAITP-001 domain
Introduction to AI Testing
Covers how AI testing differs from traditional software testing, including probabilistic outputs, data dependence, ground truth, the test oracle problem, concept drift, representative test data, and testing across the AI lifecycle.
Key skills
AI testing fundamentals, probabilistic behavior, ground truth, test oracle problem, concept drift, risk-based AI testing
AI Model Evaluation and Performance Testing
Covers AI model evaluation and performance testing, including accuracy, precision, recall, F1, confusion matrices, ROC-AUC, regression error, cross-validation, calibration, overfitting, and latency and throughput under load.
Key skills
classification metrics, confusion matrix, ROC-AUC, cross-validation, calibration, overfitting, performance testing
Bias and Fairness Testing in AI
Covers bias and fairness testing in AI, including historical and sampling bias, label bias, proxy features, demographic parity, equal opportunity, intersectional subgroup analysis, mitigation, and production fairness monitoring.
Key skills
bias sources, fairness metrics, disparate impact, proxy features, intersectional analysis, bias mitigation
Explainability and Interpretability Testing
Covers explainability and interpretability testing, including global and local explanations, SHAP, LIME, counterfactuals, surrogate models, saliency maps, and testing explanation fidelity and stability for different audiences.
Key skills
SHAP, LIME, counterfactual explanations, surrogate models, explanation fidelity, explanation stability
Adversarial Testing and AI Security
Covers adversarial testing and AI security, including adversarial examples, evasion, data poisoning, model extraction, model inversion, membership inference, prompt injection, robustness testing, and layered defenses such as adversarial training.
Key skills
adversarial examples, evasion, data poisoning, model extraction, prompt injection, adversarial training, robustness testing
AI Testing for Compliance and Ethics
Covers AI testing for compliance and ethics, including lawful data use, model cards, audit traceability, human oversight, impact assessment, data minimization, lifecycle risk management, accessibility, third-party due diligence, and incident handling.
Key skills
privacy compliance, model cards, audit trails, human oversight, impact assessment, AI risk management, incident reporting