DS401Semester 53 (2-0-2)Major

Big Data Technologies

Syllabus

Unit 1: Big Data Fundamentals and Hadoop Ecosystem

3V's characteristics (Volume, Velocity, Variety), CAP theorem and BASE consistency, Hadoop Distributed File System (HDFS) architecture (NameNode, DataNode, federation, high availability), HDFS data replication and fault tolerance, MapReduce programming model (mapper, reducer, combiner, partitioner), YARN resource management and application lifecycle.

Unit 2: Advanced Hadoop Components

Hive data warehousing ( metastore, SerDe, partitioning, bucketing), HiveQL optimization and execution engine, Pig Latin scripting for ETL pipelines, HBase NoSQL database (column-family model, coprocessors), Sqoop data transfer between RDBMS and HDFS, Flume streaming data ingestion, Oozie workflow orchestration.

Unit 3: Apache Spark Core and RDDs

Spark architecture (driver, executors, cluster manager), Resilient Distributed Datasets (RDDs) lineage and fault tolerance, Spark transformations (map, flatMap, filter, join) and actions (collect, count, reduce), Spark SQL and DataFrames, Catalyst optimizer and Tungsten execution engine, Delta Lake for ACID transactions on data lakes.

Unit 4: Spark Streaming, MLlib, and GraphX

Structured Streaming (micro-batch vs. continuous processing), Kafka integration and exactly-once semantics, Spark MLlib pipelines (transformers, estimators, cross-validation), MLflow integration for experiment tracking, GraphX graph-parallel computation (Pregel API, PageRank, connected components), GraphFrames for DataFrame-based graph analytics.

Unit 5: Big Data Platforms and Cloud Integration

Cloud data platforms (AWS EMR, Azure HDInsight, Databricks), Managed Spark services and autoscaling, Data lake architectures (lakehouse pattern, medallion architecture), Apache Kafka fundamentals (topics, partitions, consumer groups, exactly-once), Apache Airflow DAG orchestration, Cost optimization strategies and spot instance utilization.

Skills

SQLNoSQLData StructuresMachine LearningBig DataCloud ComputingDatabasesRobotics

Structure

Semester: 5

Credits: 3 (2-0-2)

Category: Major

Practicals

Lab 1: HDFS Operations and MapReduce: Focuses on HDFS shell commands, managing files/directories, implementing MapReduce jobs for word count and log analysis.
Lab 2: Hive Data Warehousing: Focuses on creating partitioned tables, HiveQL queries, UDF development, and analyzing query execution plans.
Lab 3: Spark RDD Programming: Focuses on RDD transformations/actions, custom partitioners, and broadcast variables for performance optimization.
Lab 4: Spark SQL and DataFrames: Focuses on DataFrame operations, Catalyst optimizer analysis, window functions, and Delta Lake transactions.
Lab 5: Spark Streaming with Kafka: Focuses on real-time data ingestion, stateful aggregations, and checkpointing for fault tolerance.
Lab 6: MLlib Machine Learning Pipelines: Focuses on end-to-end ML pipelines with feature engineering, hyperparameter tuning, and model persistence.
Lab 7: HBase NoSQL Operations: Focuses on column-family design, bulk loading, coprocessor implementation, and Phoenix SQL layer.
Lab 8: Data Pipeline Orchestration: Focuses on Apache Airflow DAGs integrating Spark, Hive, and HDFS for complete ETL workflows.
Lab 9: Cloud Big Data Services: Focuses on AWS EMR/S3 or Azure HDInsight deployments with autoscaling and cost monitoring.
Lab 10: Lakehouse Architecture Project: Focuses on complete data lake implementation with ingestion, transformation, ML training, and dashboarding.