Talk to us

Data Engineering

Big Data Engineering

US Online

Big Data is a very popular data engineering technology in today’s enterprises. In this course, you will learn some basic concepts of Big Data. You will then primarily study Spark and use PySpark on the Databricks platform to import, export, process, and store large-scale data, as well as implement some machine learning models using Big Data. Throughout the course, you will practice your acquired knowledge through several small projects and finally learn how to optimize Big Data infrastructures.

After the course, you will be able to use PySpark to store, read, write, and process various types and scales of data, and successfully complete tasks assigned by your company.

Intermediate
Talk to our Advisor
Part-time
Self-paced
8 weeks
80 hours
View Program Package
Register

About the Course

This course can be taken as a part of WeCloudData’s Data Engineering Bootcamp. It teaches you how to use the modern big data tools to scale the data processing pipelines for enterprises. It focuses on case-based learning and project building.

What you learn:

  • NoSQL databases
  • Data ingestion with Azure and Azure Data Factory (AWS material also available for self-paced learning)
  • Distributed data processing with Databricks and Apache Spark
  • Data pipeline orchestration with Apache Airflow (or Mage.ai)
  • Big data best practices in the industry
  • How to build end-to-end big data projects

WeCloudData is the perfect place to grow your career

5/5

“The BEST career decision I have made enrolling with WeCloudData’s Data Engineer program.”

Pros

  • Pre-bootcamp preparation for Python and SQL to help newbies warmed up since the program needs you to be relatively comfortable
  • Very wide variety of content touching aspects of ETL, Big Data technologies, and AWS
  • Instructors are great at delivering the material; always available for assistance
  • TA’s were amazing and reachable at flexible hours; I always felt comfortable to bounce ideas with them; and to have their feedback
  • Really helpful TA sessions to go over problems assignments
  • Last couple weeks of the program, were assigned client projects that were very helpful in keeping sharp and continued to build industry skills for future potential employers
  • WeCloudData was very generous to offer additional resources not part of the program to help us be job ready

Cons: (Not really cons, but I have nothing else to say)

  • The pace of the course is heavy, material is coming fast and in abundance, so be ready to work super hard and spending many hours working through challenging assignments.

Ranked #1 Data Training Program

4.9/5
4.96/5
4.95/5
4.95/5
CURRICULUM

Be ready for the new economy

WeCloudData programs are designed to be project-based. We not only cover essential theories, but also teach how to apply tools and platforms that are in high demand today. Our program curriculum is also highly adaptive to the latest market trends. 

Module 1
NoSQL Databases
The big data engineer course begins with a module covering NoSQL databases which is an important concept to grasp for data engineers whose responsibilities extends beyond data warehouses. In this module, we will discuss the pros and cons of RDBMS and why NoSQL databases can be better fit for many use cases.
LEARNING OUTCOMES
  • Understand the differences between RDBMS and NoSQL
  • Understand the CAP theorem and various types of NoSQL databases
  • Learn different types of NoSQL engines such as key/value store, search engine database, caching database, graph database, as well as the new vector database used by many generative AI applications
  • Get hands-on experience with Azure Cosmos DB and Elasticsearch
KEY SKILLS
  • NoSQL
  • CAP Theorem
  • Azure Cosmos DB
  • Cassandra
  • ScyllaDB
  • Elasticsearch
  • ELK
Tools:
Elasticsearch
Cosmos DB
Cassandra
ScyllaDB
Tools:
Spark
Databricks
Parquet
MapReduce
Module 2
Apache Spark
This module introduces students to the world of distributed system, MapReduce, and Spark. Apache Spark is the workhorse of big data. It has evolved to a mature state and many enterprise big data applications depend on it.
LEARNING OUTCOMES
  • Understand the ecosystem of Spark
  • Understand how Spark engine works under the hood
  • Develop Spark applications to process big data
  • Learn how to tune Spark jobs for optimized performance
KEY SKILLS
  • Apache Spark
  • MapReduce
  • DataFrame
  • Distributed Shuffling
  • Partitioning
  • RDD
  • Parquet
  • Predicate Pushdown
  • Spark Job Tuning
  • Databricks
  • EMR
Module 3
ETL on Microsoft Azure
Each cloud provider has its own managed services for data engineering. Some are purposefully built for big data, others are more generic tools. In this module, we will teach learners the Azure for building platforms for building big data pipelines.
LEARNING OUTCOMES
  • Learn how the Azure basics such as Active Directory, Blob Storage, VM
  • Learn how to ingest data using low-code tool such as Azure Data Factory
  • Ingest data using Azure Functions
  • Understand the zoned separation in data lakes
  • Learn how to ingest data into Azure Synapse Analytics using Serverless Spark Pool
  • Build a simple data lake solution using Azure Databricks
KEY SKILLS
  • Azure Data Factory
  • Azure Synapse Analytics
  • Spark ETL
  • Blob Storage
  • Data Lake
  • Landing Zone
  • Azure Function
Tools:
Azure
ADF
Synapse
Databricks
Spark
Tools:
Airflow
Mage
Dagster
Prefect
Module 4
Data Pipelines with Airflow
In this module, learners will experience how to work with Apache Airflow to orchestrate complex ETL pipelines.
LEARNING OUTCOMES
  • Learn to create DAGs and express job dependency graphs
  • Learn how to use Airflow to set up data polling and run scheduler
  • Use airflow to spin up or tear down infrastructure
  • Use Airflow to trigger DBT or ADF steps
  • Explore Airflow alternatives such as Prefect, Dagster, and Mage.ai
KEY SKILLS
  • Data Pipeline
  • Pipeline Orchestration
  • Airflow Deployment
  • Dagster
  • Prefect
  • Mage.ai
Module 5
Data Governance
In this module, we will cover data governance topics such as data privacy, catalogs, metadata, data quality, and governance
LEARNING OUTCOMES
  • Understand data privacy and protection laws such as CCPA and GDPR
  • Learn how to build data catalogs using Azure Purview, Data Hub, or Amundsen
  • Practice data quality checking using great expectations
  • Explore the people. process, and technology pillars of data governance
KEY SKILLS
  • Data Catalogs
  • Azure Purview
  • Data Quality
  • Great Expectation
  • Data Hub
  • Amundsen
  • Data Governance
  • First Party Data
  • 3rd party Data
  • GDPR
  • CCPA
Tools:
Purview
DataHub
Amundsen
FACULTY TEAM

Learn from the best

We’ve brought together a team of highly skilled and experienced instructors to help you learn effectively. Our instructors have a passion for teaching and a wealth of real-world experiences in their respective fields, so you can be confident that you’re learning from the best.

Stan Taov
Data Engineer @ Telus
Stan Taov is a professional data engineer with hands-on experience in building, validating and deploying machine learning models and creating end-to-end data pipelines. He is passionate about all things data. Stan has strong knowledge of converting business problems into analytical use cases and enjoys solving problems and building data-driven architecture. Stan has a passion for big data technologies and machine learning. Stan finished the Applied Data Engineering diploma program from WeCloudData and York University’s machine learning and predictive analytics certification programs. Stan switched his career multiple times, he moved from civil engineering to business and marketing and from the business field, he moved to data engineering. Currently, Stan works for Telus where he foresees and solves any operational issues with the on-prem cluster. He builds ETL pipelines and deploys spark analytical workloads.
Indrani Gorti
Director of Platform and ML engineering, Loblaw Digital
Indrani Gorti is a polyglot with experience in data engineering, data science, and solving business problems using ML and AI. She has worked in major industries like Finance, Telecommunications, Technology, Retail, and Startups. Her work has involved end-to-end projects from data ingestion, data analysis to the deployment of models in various setups using user transaction and interaction logs. Indrani is a Director of Platform and ML engineering at Loblaw Digital. She holds a BS and MS in Computer Science.
Shaohua Zhang
Chief Instructor (WeCloudData) | Head of Data Science (BeamData) | Startup Advisor and Data Coach
Shaohua is the Co-founder and Chief Instructor at WeCloudData. In the past few years, he has trained hundreds of students and helped many of them launch their data science careers. He is also the CEO and Head of Data Science at Beam Data where he works closely with the industry partners on implementing data science projects. Prior to co-founding the Toronto Institute of Data Science and Technology (WeCloudData), he led the data science team at BlackBerry and helped build the data science practice at Kik. He has also worked with big companies such as TD, Canadian Tire, Sunlife, CIBC, Communitech, and MaRS on upskilling their data teams.

Project

Portfolio Experience Building

This is what you will build

  • Create the big data pipeline project plan
  • Design the end to end data flow
  • Create the system design diagram
  • Set up data ingestion pipeline on Azure
  • Develop the ETL and data transformation strategy
  • Use Apache Airflow or Mage.ai to orchestrate the ETL pipelines
  • Use Apache Spark to transform the data
  • Set up pipeline testing and quality checks
  • Code review with project mentors
  • Present the project and publish your portfolio work
SCHEDULE

Upcoming Start Dates

Enroll Anytime
  • TUE, THU
  • 7:30PM-10:30PM ET
$3,800.00
Register
Enroll Anytime
  • TUE, THU
  • 7:30PM-10:30PM ET
$3,800.00
Register

Explore your personalized learning path

Big Data Engineering
$3,800 USD
  • Case-based learning
  • Portfolio project mentoring
  • Flexible payment plan
Register Now
Recommended Short Courses
$3,800 USD
  • Up-skill your DE skills by taking the next-level DE courses
  • Get alumni discount for other DE, DS, MLE courses
  • Short courses to consider after completing this course ⇩

No results found.

Upgrade to Bootcamp
$6,000 - $11,000 USD
  • Upgrade to the data engineering bootcamp and save $5,000
  • Get extensive 1-1 career mentoring and job support
  • Get the flexibility to create your own bootcamp
Have Questions?
Contact our Advisor
OUR ALUMNI ARE WORKING AT
OPEN RESOURCES

Start Learning With WeCloudOpen

WeCloudOpen is here to help you unlock your full potential in tech, with our free courses and workshop. Learn the fundamentals of coding and data, and become a proficient tech professional in no time!

WeCloudOpen Course

Our comprehensive courses on Python and SQL are the perfect way to start your journey into the world of tech. WeCloudOpen ensures you learn the basics without any hassles

Learn More

WeCloudOpen Workshop

Our free workshops offer topics like Business Intelligence, Data Science, Data Engineering, DevOps, Machine Learning – allowing you to get a head start in tech career

Learn More
student success

What our graduates are saying

5/5

Alumni Review, Switchup

As someone who comes from a non programming background, I found this program to be very well structured and taught you the core technologies that are being used in the data engineering field. The program goes over case studies and problems that are as being used in the industry which gives you an insight on what potential challenges you can face when you start your career as DE. The TA’s and support team will provide support even after graduation which definitely is a big plus point. Overall it was a great experience and would recommend it to anyone looking to break into the data engineering field.

5/5

Alumni Review, Switchup

I took WeCloudData’s Data Engineering Diploma program as I wanted to move my career into a more Data Engineering oriented role. The instructors were great in teaching us everything from the tools, such as Spark and AirFlow, to the languages involved, like Python and Scala. The 2 personal projects that we did required us to build a batch and a real time pipeline and were excellent at applying and cementing what was learned. The course also did a great job in helping with resume preparation, job hunting and interview preparation. Overall, I thoroughly enjoyed the program and would gladly recommend it to anyone interested in making that switch to Data Engineering.

Let WeCloud Accelerate Your Career in Tech

Have questions?

Want more details about this course? Unsure about which path to take? Apply now to reserve a spot or make an appointment with our learning advisor. 

Speak to our Advisors
Start learning with WeCloud Open

Join WeCloud Open and start learning today! We provide open courses, career guide, and learning resources. It’s a great way to start your career in tech!

Sign Up

FAQ

Frequently asked questions about the bootcamp
This is an intermediate to advanced level data engineering course.
Yes. Ideally the learners joining this course already have experiences with AWS, Python (or Scala), SQL, and Databases.
Students taking this course are either career switchers going through the Data Engineering Bootcamp (they take this as a part of the graduation criteria), or data scientists, developers, and IT professionals who would like to up-skill themselves and stay competitive in the market.
This course prepares you for the skills required for a data engineer role. We provide project mentoring and students will graduate with a solid capstone project. If you want career help, job referrals, and career mentorship, you will need to purchase different packages. Career support comes by default with the bootcamp programs, but not the short courses.
You will be eligible for big data engineer, data engineer, and big data developer related positions. Past experience also matters when it comes to the seniority of the offer. You can inquire about WeCloudData’s real project course or career mentoring programs if you want job support.
This course primarily focuses on Microsoft Azure.
This course takes quite a bit programming skills. You will need to be good at Python programming and SQL. If you are a career switcher and don’t know much about coding, we recommend you take WeClouddata’s Data Fundamentals course.
Yes, if you take the Data Engineer Bootcamp, you will get $5,000 scholarship. You can basically take the third course in the data engineering curriculum for free.
Yes. You can fill out the form on this course page to see the tuition and financing options.
Some students have their employers cover the tuition. You can always ask your employers about it. We’re happy to provide the curriculum and enrolment letter.
WeCloudData

WeCloudData is the leading data science and AI academy. Our blended learning courses have helped thousands of learners and many enterprises make successful leaps in their data journeys.

Sign up for newsletter

"*" indicates required fields

This field is for validation purposes and should be left unchanged.
Programs
Corporate Services
Resources
Company
Let’s Connect!
Facebook-f Twitter Linkedin Instagram Meetup

Canada:

180 Bloor St W #1003

Toronto, ON, Canada M5S 2V6

US:

16192 Coastal Hwy

Lewes, DE 19958, USA

info@weclouddata.com

View our Big Data Engineering course package
View our Big Data Engineering course package