Loading demo links...
What to study for each CT-AI-V2 domain
Introduction to Artificial Intelligence
This area introduces what artificial intelligence is, how AI-based systems differ from conventional software, and the forms AI can take. It also covers intelligent agents, probabilistic behaviour, and why learned behaviour creates new challenges for testers.
Key skills
AI definitions, narrow and general AI, symbolic and data-driven AI, intelligent agents, probabilistic systems, human oversight
Quality Characteristics for AI-Based Systems
This area covers the quality characteristics that matter for AI-based systems, including accuracy, robustness, fairness, transparency, and safety. It also addresses explainability, privacy, adaptability, autonomy, and the trade-offs between competing characteristics.
Key skills
accuracy, robustness, fairness and bias, transparency, explainability, reliability, safety, privacy, adaptability, autonomy levels
Machine Learning
This area explains how machine learning works, covering supervised, unsupervised, and reinforcement learning and the difference between classification and regression. It also covers training and test data, features and labels, overfitting and underfitting, and basic performance metrics.
Key skills
supervised learning, unsupervised learning, reinforcement learning, classification, regression, overfitting, underfitting, precision, recall, data splitting
Testing AI-Based Systems
This area focuses on how testing changes for AI-based systems, including the test oracle problem, non-determinism, and measurable acceptance criteria. It also covers test levels and approaches such as exploratory testing, A/B testing, shadow mode, and regression testing after retraining.
Key skills
test oracle problem, non-determinism, acceptance criteria, exploratory testing, A/B testing, shadow mode, regression testing, boundary testing
Input Data Testing
This area covers testing the data fed into AI-based systems, including completeness, consistency, accuracy of labels, and value ranges and formats. It also addresses bias, class imbalance, duplicates, outliers, representativeness, data drift, and privacy in test data.
Key skills
data completeness, data consistency, labelling errors, bias in data, class imbalance, outliers, representativeness, data drift, duplicates, privacy
Model Testing
This area covers evaluating the trained model itself, using tools such as the confusion matrix and metrics including precision, recall, and the F1 score. It also addresses threshold tuning, cross-validation, metamorphic testing, back-to-back testing, and adversarial examples.
Key skills
confusion matrix, precision, recall, F1 score, threshold tuning, cross-validation, metamorphic testing, back-to-back testing, adversarial testing
Machine Learning Development Testing
This area covers testing across the machine learning development lifecycle, from data pipelines and component unit tests to integration and deployment checks. It also addresses versioning, reproducibility, experiment tracking, training-serving skew, monitoring, and retraining triggers.
Key skills
pipeline testing, unit testing, integration testing, data and model versioning, reproducibility, experiment tracking, training-serving skew, monitoring, retraining