Big Data — Session 3

Overview

This course session provides a hands-on demonstration of building a data processing job using Talend Studio for Hadoop environments, focusing on ETL (Extract, Transform, Load) workflows with HDFS components. The instructor walks through setting up a job to clean a CSV file stored in Hadoop by removing leading/trailing whitespace, leveraging metadata schemas, logging execution traces, and optimizing connection reuse to improve performance. The session emphasizes architectural principles of Talend’s code-generation engine—how components like THDFS Input/Output trigger automatic MapReduce script generation—and best practices for naming conventions, error handling, and environment portability across Windows and Linux systems.

Topic (Timeline)

1. Environment Setup and Job Initialization [00:00:00 - 00:03:02]

The session begins with the instructor troubleshooting slow virtual machine performance due to high memory allocation (6 GB). After confirming UI load completion, a folder named “Procesamiento” is created to organize job artifacts. A new job titled “Job Limpieza Archivo Producción” is initiated, establishing the foundational structure for the data cleaning workflow.

2. Logging and Connection Configuration [00:03:02 - 00:06:37]

The instructor introduces TPreJob and TPosJob components for job orchestration. A THDFS Connection component is configured using static metadata from a previously defined repository to ensure consistency. A TWAR (Log) component is added to the TPreJob to record a trace message: “conexión exitosa Cluster Hadoop” with error code 42 and trace level. This establishes a standardized logging practice for monitoring job execution.

3. Log Capture and Output File Setup [00:06:37 - 00:09:40]

A T-LOG Catcher component is added to automatically capture Java errors, TDAI events, and component-level failures. It is connected to a T-LOG RAW component (for console output) and a T-FILE Output Delimited component to write logs to a persistent file. The output file is configured to append (not overwrite) entries, with path C:\ten\logs.txt. The instructor emphasizes standardizing file paths and naming conventions (all lowercase) to ensure cross-platform compatibility between Windows and Linux environments.

4. HDFS Input/Output Architecture and Connection Optimization [00:09:40 - 00:13:27]

Two HDFS components—THDFS Input and THDFS Output—are introduced. The instructor explains the performance penalty of multiple Hadoop connections and demonstrates reusing a single THDFS Connection across both components to reduce latency. The input component is configured to read produccion.csv from HDFS, with delimiter set to comma (verified via external file inspection), header enabled, and encoding set to ISO-8859-1. Compression settings are noted but not configured.

5. Data Transformation with T-Map and Code Generation Principles [00:13:27 - 00:19:48]

A T-Map component is inserted between HDFS Input and Output to transform data. The instructor explains that Talend automatically generates MapReduce (or Spark) code when components like THDFS Input or T-Spark are used—eliminating the need for manual scripting. The connection from T-Map to THDFS Output is made via Component OK, illustrating Talend’s trigger-based connectivity model. A TWAR log is added after output to confirm successful file cleanup: “Archivo Curado de forma correcta”.

6. Schema Reuse via Metadata and Final T-Map Configuration [00:19:48 - 00:25:52]

The instructor revisits the THDFS Input to apply a schema from metadata (file delimited producción), avoiding manual column definition. The schema is loaded via the repository dropdown, ensuring consistency with prior definitions. In T-Map, input columns (nombre, area, reducción) are mapped directly to output. Two variables are created: season and clock, each transformed using .trim() to remove leading/trailing whitespace. The output schema is finalized with all fields correctly mapped, completing the data cleaning job.

Appendix

Key Principles

• Code Generation: Talend automatically generates MapReduce/Spark code when using HDFS or Spark components—users design visually, not programmatically.
• Connection Reuse: Reuse a single HDFS connection across multiple components to minimize cluster latency and improve performance.
• Cross-Platform Consistency: Use lowercase for folder and file names to avoid case-sensitivity issues when moving jobs from Windows to Linux/Unix systems.
• Logging Best Practices: Use TWAR for trace logging and T-LOG Catcher to capture runtime errors automatically; always append logs to avoid data loss.

Tools Used

• Talend Studio (version with Hadoop support)
• THDFS Input / Output
• T-Map
• TPreJob / TPosJob
• TWAR (Log)
• T-LOG Catcher
• T-LOG RAW
• T-FILE Output Delimited

Common Pitfalls

• Memory Overload: Running jobs on VMs with insufficient or misconfigured memory causes delays.
• Case Sensitivity: Uppercase folder names (e.g., “Procesamiento”) may break jobs on Linux systems if the job expects lowercase.
• Schema Mismatch: Manually defining schemas instead of reusing metadata leads to inconsistencies and errors.
• Multiple Connections: Configuring separate HDFS connections per component increases execution time and resource usage.

Practice Suggestions

• Recreate this job using a different dataset and validate log output.
• Experiment with switching execution mode from MapReduce to Spark in RuneJob settings (if available).
• Test job portability by exporting and importing the job into a Linux-based Talend environment.
• Add a T-File Input component to read from local disk and compare behavior with HDFS input.