Loading demo links...
What to study for each Data-Engineering-10.2-Developer domain
Big Data Integration Course Introduction & Big Data Basics
Introduction and big data basics: HDFS, YARN and MapReduce, data locality and how big data integration differs from traditional ETL.
Key skills
HDFS blocks and replication, YARN, MapReduce
Informatica on Hadoop Architecture
Informatica on Hadoop architecture: how mappings are submitted through the Hadoop connection and executed by cluster engines, and the services involved.
Key skills
Hadoop connection, Data Integration Service, engine execution
Data Warehouse Offloading & Code Migration and Ingestion
Data warehouse offloading, code migration and ingestion: offloading workloads to Hadoop, SQL-to-Mapping conversion, Sqoop transfers and ingestion design.
Key skills
Offloading, SQL-to-Mapping, Sqoop, ingestion
Informatica Polyglot Computing in Hadoop
Polyglot computing: running one mapping design on different engines such as Blaze and Spark, and the Smart Executor concept.
Key skills
Blaze, Spark, engine selection
Monitoring, Logs, and Troubleshooting
Monitoring, logs and troubleshooting for Hadoop executions: Informatica monitoring views and cluster-side job logs and resource errors.
Key skills
Job monitoring, YARN resource diagnosis
Hadoop Data Integration Challenges and Performance Tuning
Hadoop integration challenges and tuning: the small-files problem, data skew and parallelism matched to cluster capacity.
Key skills
Small files, skew handling, partitioning
Complex File Parsing
Complex file parsing with the Data Processor transformation: parsers for hierarchical and industry formats defined from examples.
Key skills
Data Processor, example-based parsers
NoSQL Databases
NoSQL databases in Data Engineering: integrating stores such as HBase as mapping sources and targets.
Key skills
HBase access patterns
Developer Tool Fundamentals
Developer tool fundamentals: the design client, the Model repository and mapping validation.
Key skills
Developer tool, Model Repository Service, validation
Developing Physical Data Objects
Developing physical data objects: objects bound to real tables, files and Hadoop data, logical data objects and metadata import.
Key skills
Physical and logical data objects, metadata import
Viewing Data
Viewing data: the Data Viewer for row-level inspection through transformations and preview for understanding source content.
Key skills
Data Viewer, data preview
Developing Mappings and Transformations
Developing mappings and transformations: data-flow structure and core transformations such as Joiner and Lookup on cluster engines.
Key skills
Mapping structure, Joiner, Lookup
Working with Dynamic Schema and Dynamic Mappings
Dynamic schema and dynamic mappings: parameters, run-time links and port rules that absorb schema change and source proliferation.
Key skills
Dynamic mappings, run-time links
Deploying Applications
Deploying applications: packaging objects into applications deployed to the Data Integration Service and updating them.
Key skills
Applications, deployment and updates
Parameters
Parameters: mapping parameters and parameter sets that make mappings and deployed applications reusable.
Key skills
Parameters, parameter sets
Workflows
Workflows: sequencing mapping tasks and other tasks with gateways and conditional flows, deployed within applications.
Key skills
Workflow tasks, gateways, sequence flows
Edge Data Streaming (EDS) Overview
Edge Data Streaming overview: collecting data at the edge with agents and reliable buffered forwarding into the streaming platform.
Key skills
EDS agents, edge buffering
Big Data Streaming Overview
Big Data Streaming overview: continuous processing of streaming sources with streaming mappings on engines such as Spark Streaming.
Key skills
Streaming concepts
Kafka Overview
Kafka overview: topics, partitions, producers, consumers, consumer groups and offsets.
Key skills
Kafka topics and partitions, offsets
Streaming Mappings
Streaming mappings: continuous mappings over Kafka and other streams, windows, checkpointing, enrichment, error routing and streaming health indicators.
Key skills
Windows, checkpointing, offset handling, lag
Monitoring Logs and Troubleshooting
Monitoring logs and troubleshooting for streaming: diagnosing growing lag and correlating Informatica job logs with cluster engine logs.
Key skills
Lag diagnosis, log correlation
Performance Tuning and Best Practices
Performance tuning and best practices: parallelism and partitioning, minimising shuffles and pushing filters to the source.
Key skills
Shuffle reduction, pushdown, evidence-led tuning
End-to-End Use Case
The end-to-end use case: ingesting, developing, deploying and monitoring a complete Data Engineering solution including streaming.
Key skills
Capstone flow, deployment, verification