Add to Wishlist

Big Data Masters Program

Get course
One Time Payment Enroll with Membership

Request more information

    Introduction to Big Data

    1
    Introduction to Big Data
    2
    Big Data System Requirements
    3
    Monolithic vs Distributed System
    4
    Distributed System Architecture
    5
    What is Hadoop And Evolution of Hadoop
    6
    Core Components of Hadoop
    7
    HDFS Architecture:
    8
    What is Node And What is Cluster
    9
    Data Block & Block Size
    10
    Slave Node, Master Node, Data Node & Name Node
    11
    Metadata And Replication Factor
    12
    Heart Beat & Fault Tolerance
    13
    Handling Namenode Failure
    14
    What is SPOF
    15
    FSimage & Edit Logs
    16
    Secondary Namenode
    17
    Name Node Recovery
    18
    Check Pointing
    19
    Understanding Replication Factor
    20
    What is Rack And Rack Failure
    21
    Rack Awareness Mechanism
    22
    Block Report
    23
    Namenode High Availability
    24
    Quorum Journal Manager & Quorum Journal Node
    25
    Understanding Linux File System
    26
    List & Parameters of List Command
    27
    Touch, Mkdir, Rmdir & Other Linux Commands
    28
    HDFS Commands:
    29
    List Files & Directories
    30
    How HDFS Commands Work
    31
    ‘ls’ Command With Various Parameters
    32
    Create, Remove File/Directory
    33
    Copy & Get Files/Folders From Local to HDFS & Vice Versa
    34
    Move Files/Folders From HDFS to HDFS
    35
    Change Replication Factor Dynamically
    36
    View File Metadata Information

    MapReduce - Distributed Computing Framework

    1
    Introduction to MapReduce
    2
    Stages in MapReduce
    3
    What is Key-Value
    4
    What is Map & What is Reduce
    5
    Example to Undestand Map&Reduce
    6
    Word Count Example in MapREduce
    7
    Record Reader
    8
    Mapper Phase
    9
    Reducer Phase
    10
    MapReduce Shuffle & Sort
    11
    Inside Map & Reduce Phase
    12
    Wordcount Example in MapReduce
    13
    Typical MapReduce Flow
    14
    Blocks in MapReduce
    15
    Default Number of Mappers & Reducers
    16
    Understanding Number of Mappers/Reducers
    17
    MapReduce Framework Behind the Scenes
    18
    Role of Hash Function in MapReduce
    19
    Partitioning in MapReduce
    20
    How to Choose Number of Reducers
    21
    How Hash Function Works
    22
    Understanding Shuffle & Sort
    23
    Example: Calculating Max Temperature in a Day
    24
    Combiner Function in MapReduce
    25
    Advantages of Combiners
    26
    When to Use or Not to Use Combiner
    27
    Example1: Filtering Data using MapReduce
    28
    Example2: Finding Distinct Values
    29
    Example3: Finding Top 3 Most Influential users
    30
    Realtime Use Case: Google Web Search
    31
    MapReduce Programming
    32
    MR Code Explanation
    33
    How to Write Map Reduce Code
    34
    Mapper Code
    35
    Reducer Code
    36
    Main Code
    37
    Finding the Frequency of Each Word in a File
    38
    Mapreduce Jars
    39
    MapReduce Practical Sessions
    40
    Word Count Program – Practical Session1
    41
    Jar Creation & Execution – Practical Session2:
    42
    How to Create a Jar
    43
    How to Execute the Jar
    44
    How to Track a Job
    45
    How to Track All Previous Jobs
    46
    MR Program Variations – Practical Session3:
    47
    How to Change Number of Reducers
    48
    Writing Custom Partitioner Logic
    49
    Changing Number of Reducers to Zero
    50
    Introducing Combiner
    51
    Writing Custom Combiner logic
    52
    Introduction to Partitioners
    53
    Partitioners Code Example

    Apache Sqoop - Data Ingestion to Hadoop

    1
    Sqoop Fundamentals
    2
    Sqoop Basics
    3
    What is sqoop
    4
    Sqoop Workflow
    5
    Key Features of Sqoop
    6
    Sqoop Import
    7
    Sqoop Export
    8
    Connecting to MySQL
    9
    Acessing MySQL Databases from Hadoop
    10
    Acessing MySQL Tables from Hadoop
    11
    Sqoop Import Practicals
    12
    Sqoop Export Practicals
    13
    Sqoop Job
    14
    Sqoop Incremental Load
    15
    Sqoop Default Import
    16
    Sqoop Free-From Query Import
    17
    Sqoop Direct import
    18
    Importing Data Into Hive
    19
    Importing Data Into HBase
    20
    Sqoop Validate
    21
    When a Sqoop Export May Fail

    Apache Hive

    1
    Hive Overview:
    2
    Transactional System and Analytical System
    3
    Examples of Transactional Systems
    4
    Examples of Analytical Systems
    5
    What is Hive
    6
    Hive Query Language (HQL)
    7
    Understanding Hive Table
    8
    Introduction to Hive Metadata
    9
    Why Hive over traditional databases
    10
    Transactional and Analytical Processing
    11
    What is Data Warehouse
    12
    Hive Architecture
    13
    Hive on top of Hadoop
    14
    How Hive Works
    15
    Transactional vs Analytical Processing
    16
    Data Warehouse Concept
    17
    The Hive Metastore
    18
    Hive vs RDBMS
    19
    HQL vs SQL
    20
    Hive Subqueries Views & Index
    21
    Transactional and Analytical Processing
    22
    What is Data Warehouse
    23
    Hive Architecture
    24
    Hive on Hadoop
    25
    Hive Metastore
    26
    Hive vs. RDBMS
    27
    Hive Complex Data Types
    28
    Hive Array, Map & Struct
    29
    Hive Built-in Functions
    30
    Hive UDF, UDAF & UDTF
    31
    Hive Lateral Views
    32
    Hive Subqueries
    33
    Hive Views
    34
    Hive Normalization vs Denormalization

    Apache Hive Advance

    1
    Hive Structure Level Optimizations:
    2
    Hive Partitioning
    3
    Hive Partitioning With 2 Columns
    4
    Hive Bucketing
    5
    Hive Partitioning With Bucketing
    6
    Hive Query Level Optimizations:
    7
    Hive Join Optimizations
    8
    Hive Bucket Map Join Optimizations
    9
    Hive Window Functions
    10
    Hive Ranking
    11
    Hive Sorting
    12
    Hive File Format
    13
    Row vs Column File Formats
    14
    Specialized File Formats
    15
    Internals of ORC File Formats
    16
    Internals of Parquet File Formats
    17
    ORC vs Parquet File Formats
    18
    Hive Compression Techniques
    19
    Hive Vectorization
    20
    Changing the Hive Engine
    21
    Hive Thrift Server

    NoSQL Databases - HBase

    1
    Hbase Basics
    2
    Key requirements of database
    3
    Limitations of Hadoop
    4
    Google Bigtable concept for quick searching
    5
    Implementation of Bigtable as Hbase
    6
    Properties of Hbase
    7
    What Hbase can offer
    8
    Row based storage vs Columnar storage
    9
    Advantages of columnar storage
    10
    Normalization vs Denormalization
    11
    CRUD Operation
    12
    RDBMS vs Hbase
    13
    Hbase data model
    14
    4-Dimensional data model
    15
    CAP Theorem
    16
    Hbase Architecture
    17
    Hbase Region Server
    18
    Region, Memstore, Wal & Block Cache
    19
    Hfile
    20
    Zookeeper
    21
    Hmaster & Meta Table
    22
    Hbase Architecture components in details
    23
    Hbase Read/Write operations
    24
    Compaction
    25
    Hbase Data Update
    26
    Hbase Data Deletion
    27
    Handling Server Failures
    28
    Hbase Practicals
    29
    Handling Hbase Failure Services
    30
    Create & List Table
    31
    Insert Records in Table
    32
    Scan(view) & Get records from table
    33
    Delete a column
    34
    Describe a table
    35
    Check table exists or not
    36
    Drop table – Understanding how it works
    37
    Parameters of get command
    38
    Parameters of scan command
    39
    Hbase files structure in HDFS
    40
    How to disable/enable a table
    41
    Various filters in Hbase
    42
    Count Records

    NO SQL Database --Cassandra Overview

    1
    What is Cassandra
    2
    How Cassandra Cluster Look Like
    3
    Tunable read/write Consistency
    4
    Hbase vs Cassandra
    5
    Integration with Hadoop (Mini Project)
    6
    Hbase-Hive Integration

    Learning Scala - A Guide to Functional Programming

    1
    Why Scala
    2
    Where to Run Scala Code
    3
    Scala Code Using IDE
    4
    Scala Basics
    5
    Var vs val
    6
    Type inference
    7
    Data types in Scala
    8
    String Interpolation
    9
    String Comparison
    10
    Flow control: If else
    11
    Match Case
    12
    For Loop
    13
    While loop
    14
    Scala Functional Programming
    15
    How to define a function
    16
    Higher order function
    17
    Anonymous function
    18
    Scala Collections
    19
    Array
    20
    List
    21
    Tuple
    22
    Range
    23
    Set
    24
    Map
    25
    Scala Functional Programming:
    26
    Why Scala
    27
    Modes of writing Scala code
    28
    What is a functional programming
    29
    What is a function
    30
    What is a pure function?
    31
    First class function
    32
    Higher order function
    33
    Anonymous function
    34
    Immutability
    35
    Loop
    36
    Recursion
    37
    Tail recursion
    38
    Statement vs Expression
    39
    Closure
    40
    Scala type system
    41
    Scala operators
    42
    Anonymous function
    43
    Placeholder syntax
    44
    Partially applied functions
    45
    Function currying

    Apache Spark - General Purpose Cluster Computing Framework

    1
    What is App class in Scala
    2
    Default args, named args & variable args
    3
    Difference between nil, null, none & nothing
    4
    What is option in Scala
    5
    What is unit in Scala
    6
    Dealing with nulls in Scala
    7
    What is yield
    8
    What is vector
    9
    Scala if guards & pattern guards
    10
    What is “for comprehensions”
    11
    Difference between “==” in java and Scala
    12
    Difference between strict val vs lazy val
    13
    What are default packages in Scala
    14
    What is Scala apply method
    15
    What is a diamond problem in Scala
    16
    What is a trait
    17
    Why Scala is the top most choice for a big data
    18
    What is Apache Spark

    Apache Spark Introduction

    1
    What is Apache Spark
    2
    Understanding Spark cluster
    3
    Is Spark a replacement to Hadoop
    4
    Why Spark is faster than MapReduce
    5
    How data store in Spark
    6
    What is RDD
    7
    What is DAG
    8
    RDD Lineage
    9
    Resiliency
    10
    Immutability
    11
    Transformation & Action
    12
    Lazy Evaluation
    13
    Word count program in Spark
    14
    Word count program in PySpark
    15
    Word count problem real-time example

    Apache Spark --ADVANCE

    1
    Spark Real-Time Example
    2
    Broadcast Variable
    3
    Accumulators
    4
    How Spark Executes Program on the Cluster
    5
    Spark Driver and Executors
    6
    Client Mode, Cluster Mode and Local Mode Analyzing Log Messages – Hands on
    7
    Narrow vs Wide Transformations
    8
    Stages in Spark
    9
    Difference Between reduceByKey & reduce
    10
    Difference Between groupByKey & reduceByKey
    11
    Pair RDD
    12
    Pair RDD vs Map
    13
    Understanding Default Parallelism
    14
    Difference Between repartition & coalesce
    15
    When to Increase/Decrease Partitions
    16
    Spark on YARN Architecture
    17
    YARN – Yet Another Resource Negotiator
    18
    Application Master
    19
    Containers

    Apache Spark - Structured API Part-1

    1
    Cache vs Persist
    2
    Spark Storage Levels
    3
    Difference Between DAG & Lineage
    4
    How to Submit a Spark Job
    5
    Real-time example – Finding top movies based on ratings
    6
    Spark Ecosystem
    7
    Map vs Map Partitions
    8
    Introduction to Spark Structured API
    9
    Spark DataFrame
    10
    Understanding SparkSession
    11
    SparkSession vs SparkContext
    12
    Dataframe with Various Transformations
    13
    RDD vs DataFrame vs Datasets
    14
    Challenges with DataFrame
    15
    Spark Dataset API
    16
    Difference Between DataFrame and Dataset
    17
    Benefits of Dataset
    18
    Creating Dataframe/Datasets from Various File Formats
    19
    Read Modes & Schema
    20
    Ways to Define the Schema
    21
    Defining a Explicit Schema

    Apache Spark - Structured API Part-2

    1
    Writing Output to Sink (spark.write)
    2
    Spark File Layout
    3
    Benefits of Repartitions
    4
    partitionBy & bucketBy
    5
    Saving file in Various file format
    6
    Introduction to SparkSql
    7
    Storing Data in Persistent Manner
    8
    Handling Spark Metadata
    9
    Low & High level Transformations
    10
    Refering to a Column in Dataframe/Dataset
    11
    Column String
    12
    Column Object
    13
    Column Expression
    14
    Spark UDF using Structured API
    15
    Adding Column in Dataframe
    16
    Dataframe to Dataset Using Case Class.
    17
    Dataset to DataFrame Conversion
    18
    Spark Catalog
    19
    Registring UDF with Driver
    20
    Transformations Hands on Examples
    21
    Aggregate Transformations
    22
    Simple Aggregations
    23
    Grouping Aggregations
    24
    Window Aggregations
    25
    Joins on DataFrame
    26
    Simple Join (Shuffle Sort Merge Join)
    27
    Broadcast Join
    28
    Dealing With Ambiguoes Column Names
    29
    Dealing With Null’s
    30
    Internals of Join Operations
    31
    When to Use Simple Join When Use Broadcast Join
    32
    Grouping Aggregation Real-time Example
    33
    Infering Data in SparkSQL
    34
    Quiz
    35
    Assignment
    36
    Assignment Solution

    Apache Spark - Optimization Part-1

    1
    Level of Optimizations
    2
    Resource level optimizations
    3
    Application level optimizations
    4
    Cluster level optimizations
    5
    How to calculate no of Executors
    6
    Thin Executor
    7
    Fat Executor
    8
    How to calculate no of Executors
    9
    How to Calculate Memory allacation
    10
    How to Calculate No of Cores
    11
    Heap Memory
    12
    Off-Heap Memory
    13
    Hands on With Real-time cluster
    14
    Understanding Cluster Configuarations
    15
    Realtime Example:Moving ata to HDFS using a Edge node and work around it in a realtime cluster
    16
    Static Resource allocation
    17
    Dynamic Resource allocation
    18
    Understanding Memory Usage in Spark
    19
    Execution Memory
    20
    Storage Memory
    21
    Practical Demonstration: Cache & Persist
    22
    Java Serializer vs Kryo Serializer
    23
    Quiz
    24
    Assignment
    25
    Assignment Solution

    Apache Spark - Optimization Part-2

    1
    Broadcast Join Practical Demonstartions
    2
    Broadcast Join Using RDD
    3
    When to Use Broadcast Join
    4
    Broadcast Join Using Dataframe
    5
    Visualizing Broadcast Join with Structured API
    6
    Practical Demo on Repartition vs Coalesce
    7
    Client Mode vs Cluster Mode When using Spark submit
    8
    Spark Join Optimizations
    9
    Spark Advance Optimizations: Sort Aggregate vs Hash Aggregate
    10
    Spark Catalyst Optimizer
    11
    Quiz
    12
    Assignment
    13
    Assignment Solution

    Apache Spark - Streaming

    1
    What is Real-time Processing
    2
    The Importance of Real-time Processing
    3
    Batch processing vs Real-time Stream Processing
    4
    Spark Streaming Data
    5
    Spark discretized stream or DStream
    6
    Batch & Batch Interval
    7
    Do Spark is a real-time streaming engine
    8
    Stream Processing in Spark
    9
    Transformed DStream
    10
    Understanding Producer & Consumer
    11
    Practical on Real-time Processing
    12
    Stream Transformations
    13
    Stateless Transformations
    14
    Stateful Transformations
    15
    Window Operations
    16
    Batch Interval
    17
    Window Size
    18
    Sliding Interval
    19
    Practical on Stateless Transformation
    20
    Practical on Stateful Transformation
    21
    reduceByKey vs updateStateByKey
    22
    Working With Sliding Window
    23
    reduceByKeyAndWindow Transformation
    24
    reduceByWindow Transformation
    25
    countByWindow Transformation
    26
    Quiz
    27
    Assignment
    28
    Assignment Solution

    Apache Spark - Streaming Part-2

    1
    What Is Structured Streaming
    2
    Requirement Of Structure Streaming
    3
    Limitations Of Spark Streaming
    4
    Benefits Of Spark Structure Streaming
    5
    Practical – Wordcount Example On Structured Streaming
    6
    Dynamically Setting The Shuffle Partitions
    7
    Data Stream Writer Output Modes
    8
    Datastream Output Modes – append, update & complete
    9
    Spark Streaming Graceful Shutdown
    10
    How Does Spark Streaming Code Executes Internally
    11
    How a Job Converted to Micro batches
    12
    Trigger Point For Micro Batches
    13
    Types of Triggers – unspecified, time interval,one time, continuous
    14
    Types of Data Sources – Socket Source, Rate ,Source, File Source, Kafka Source
    15
    Limitations of socket source
    16
    Practical on File Data Source
    17
    Types of Spark Streaming Output Data Options
    18
    Fault Tolerance and Exactly Once Guarantee
    19
    Understanding Checkpoint Location
    20
    Stateful vs Stateless Transformations
    21
    Managed Stateful Operations vs UnManaged Stateful Operations
    22
    Types of Aggregations – Continuous Aggregations vs Time Bound Aggregations
    23
    Window Tranformations
    24
    updateStateByKey, reduceByKeyAndWindow,reduceByWindow, countByWindow
    25
    Types of windows – Tumbling Time Window,Sliding Time Window
    26
    Streaming Joins
    27
    Streaming Dataframe to static dataframe
    28
    Streaming Dataframe With Another Streaming Dataframes
    29
    Quiz
    30
    Assignment
    31
    Assignment Solution

    Apache Kafka - Distributed Event Streaming Platform

    1
    Introduction To Kafka
    2
    Kakfa Architecture
    3
    Kafka Key Concepts/Fundamentals
    4
    Overview Of Zookeeper And It’s Role In Kafka Cluster
    5
    Cluster, Nodes, Brokers, Topics
    6
    Consumer, Producers, Logs, Partitions
    7
    Concept Of Consumer Groups
    8
    Leader & Follower Partition
    9
    Installing One Node Kafka Cluster On Local
    10
    Installing Multi Broker Kafka Cluster On Local
    11
    Command Line Producer And Consumer
    12
    Replication Concept For Fault Tolerance
    13
    How Data Is Stored In Brokers
    14
    Log Segments, Message Offsets, Message Index
    15
    Isr List / Minimum Isr
    16
    Committed Vs Uncommited Messages
    17
    Writing A Kafka Producer In Java
    18
    Writing A Kafka Consumer In Java
    19
    Achieving Exactly Once Semantics
    20
    Integrating Kafka With Spark Structured Streaming.
    21
    Quiz
    22
    Assignment
    23
    Assignment Solution

    Big Data on Cloud

    1
    AWS EMR (Elastic MapReduce)
    2
    What is a VM (Virtual Machine)
    3
    On-Premise vs Cloud Setup
    4
    Major Vendors of Hadoop Distribution
    5
    Why Cloud & Big Data on Cloud
    6
    Major Cloud Providers of Bigdata
    7
    What is EMR
    8
    Hdfs vs S3
    9
    What Is S3
    10
    Important Instances in AWS
    11
    Kinds of Nodes in Cluster
    12
    Transient vs Long Running Cluster
    13
    Running Spark Code on Emr
    14
    How to Track Your Job
    15
    Copy File From S3 to Local
    16
    Zeppelin Notebook
    17
    Types of EC2 Instances
    18
    How to Create a VM
    19
    What is a Keypair
    20
    Elastic IP
    21
    AWS Storage, Networking & CLI
    22
    Instance Store
    23
    S3 & EBS
    24
    Public Ip Vs Private Ip
    25
    Network Switches
    26
    Security Group
    27
    Aws Command Line Interface
    28
    Launch A Emr Cluster Using Advanced Options

    AWS Athena:

    1
    What is Athena
    2
    When do we require Athena
    3
    What problem Athena Solve
    4
    How Athena Works
    5
    Athena Practical Demonstration
    6
    How to create a normal table manually on csv data residing in s3
    7
    How to minimize data scanning in Athena
    8
    How to create partition table on Parquet file
    9
    Infering Schema automatically using AWS Glue
    10
    Glue Catalog
    11
    Quiz
    12
    Assignment
    13
    Assignment Solution

    Final Project

    1
    One end-to-end pipeline PROJECT involving all Major components like Sqoop, Hdfs, Hive, Hbase, Spark… etc.
    2
    Interview Preparation Tips:
    3
    Resume Building
    4
    15+ Mock Interview Recordings
    5
    Mock Interview
    6
    Interview Questions
    7
    How to Handle Managerial Round Qs