Data Engineer Portfolio Project

Inquire about Data Engineering programs
Become a Data Engineer

Contact our advisors now to learn more about our programs and courses. They are here to answer all your questions and help you embark on a successful journey.

"*" indicates required fields

Name*
This field is for validation purposes and should be left unchanged.

What are portfolio projects?

A portfolio is a collection of projects, code, document, and other things that can help you showcase your skills. These usually go beyond degrees and certifications and show practical skills that the candidates have obtained.

A decent portfolio project usually take more effort than most learners would expect. It’s more than just uploading your code to github. Data engineering projects are not as easy to demonstrate as compared to a data science project that has more visual components.

Below is a list of things aspiring data engineers can work on to strength their profile.

  • Personal Projects
    • Project summary
    • Data pipeline architecture diagrams
    • Data infrastructure configuration templates/scripts
    • Working code and scripts (ETL/ELT scripts, pipeline DAGs, etc.)
  • Github
    • Code committed and pushed to your github branches that employers can look into
  • Blogs
    • Blog posts that document your learning journey and your views on the data engineering world. We suggest Medium/Substack as the blogging platform or WordPress. You can also set up your own Github pages.

WeCloudData’s suggestion

  • Portfolio is extremely important because job search is usually a chicken and egg problem. It’s somewhat hard for applicants to show work related experiences without actually working in a DE role. Therefore, one of the best ways to showcase a career switcher’s background and skills is the project!
  • Blog posts and Github can be used as great tools for networking and LinkedIn cold calling
  • Portfolio projects not only demonstrate your skills but also show the hiring team how serious you’re and the level of effort you’ve put in

Building End-to-End DE Projects

Traditional data engineers in big companies may just need to focus on very specific tasks. For example, as an ETL specialist the DE may work with Informatica to transform and prepare data in the Informatica environment. The job market is increasingly looking for candidates who know more tools and design patterns. Especially in startups and tech companies that build their infrastructure on public clouds, data engineers have access to an abundant amount of tools and they either need to know how to work with a lot of those tools, or have to learn very quickly.

Therefore, WeCloudData suggests that when learning data engineering and building your portfolio projects, it’s very important to build projects that are end-to-end.

What does end-to-end project mean? An end-to-end project covers the entire data pipeline, from data collection, ingestion, to cleaning/transformation (ETL/ELT), loading into data lake/warehouse, and reverse ETL. It will be challenging to build something of this scope for a learner but try to build a pipeline that’s as complex and complete as possible.

Only showing code/scripts of processing data for ETL is not going to give candidates a huge advantage in the job search process.

Keep in mind that hiring managers always want to know the following:

  • What problem are you trying to solve?
  • What tools and platforms did you use?
  • Why and how did you make some of the decisions on some tools and approaches
  • What benefits did you approach have? Was it trying to optimizing the speed, or lowering the infrastructure cost, or improving the reliability?
End-to-end AWS-based big data pipeline - created by WeCloudData
End-to-end AWS-based big data pipeline – created by WeCloudData

Importance of Data Architecture and Demo

It’s very hard to explain the data pipeline and the best way is to create your pipeline diagram. Data engineers need to know how to work with diagraming tools such as draw.io or lucidchart to design pipeline/data architectures. It doesn’t have to be the most professional looking diagram but some key components need to be articulated. This will help the hiring managers understand what you did and leave a very strong impression. It can also be used as a potential tool during an interview for you to walk the hiring managers and senior data engineers through your project.

Things to include on a DE project diagram
  • Data sources
  • Data destinations
  • Data store (databases, data warehouse, or data lake)
  • Data ingestion process (CDC, Kafka, or airbyte/fivetran, etc.)
  • Staging vs production zones
  • Critical logics in your pipelines
  • Automation workflow
  • Environment/Infrastructure (AWS, Kubernetes, Spark Clusters, etc.)
  • Analytics output if possible (BI dashboards)
Data engineering pipeline diagram project
Here’s a pipeline diagram of one of the real projects WeCloudData students work on.
Real-time Twitter Sentiment Dashboard Data Engineering Project
Real-time Twitter Sentiment Dashboard (pipeline built with streaming data on AWS and Spark)

Importance of real project experience

Nothing beats real hands-on experience! Hiring managers will always prefer to hire someone who has real experiences. Real experience doesn’t mean work experience, it could be self-guided projects, freelance projects, or some kind of real project experience working with clients.

In this section, we will discuss why having real project experience can help you stand out easily. Here’re some of the benefits of real client projects:

  • You will gain experience working with real companies and clients so your work can be articulated in interviews and helps you build more credibility
  • You will get more interviews because of the real experience
  • You will be able to negotiate for higher starting salary
  • For career switchers, real projects help you close the experience gap.

Let’s try to understand the important factors in hiring decisions:

  • Hiring managers want to hire someone who has the right skills
  • Hiring managers want to hire someone who has good track record so they have more certainty in the hiring decision
  • Hiring managers want to hire someone who is at lower risks
  • It’s not about how MANY tools a candidate knows
  • It’s not about how MANY years of experience a candidate has
  • It’s more about experience which is related to problem solving skills

For career switchers coming from non-data and non-tech background, your past work experience doesn’t carry much weight and often times creates disadvantages. It’s pretty easy for hiring managers to make biased assumptions that you’re not a good fit without even looking into your skills and capabilities. Working on real client projects means the work you do will carry more weight on your resume and help build more trust. Employers will have more confidence when they are deciding among different candidates. You will also learn things that you don’t have the opportunities to practice yourself:

  • Project scoping and requirements gathering
  • Business communications
  • Task prioritization
  • Debugging, solving technical challenges, handling messy situations, and learning from mistakes
  • Associating your work with measurable business values

Understanding how real-life projects work

Here’s a sample data warehousing project workflow.

A project team is tasked with creating a modern data warehouse in the Azure cloud. The project managers, data engineers, IT/platform engineers need to work together to make the following things happen for the company/client:

An example project workflow and what you will learn in real projects
An example project workflow and what you will learn in real projects

  • Understand the business needs of a data warehouse
  • Get IT infrastructure access sorted out
  • Perform data profiling and assess the feasibility/scope of the work involved
  • Decide the tools/platforms to use for the data warehouse project
  • Work with compliance team to make sure best practices are followed
  • Design the overarching data architecture
  • Create the data models (dimension models, wide tables, etc.)
  • Ingest the data from different sources to the data lake or staging database
  • Write ELT/ETL scripts to materialize the tables in data warehouse
  • Run UAT sessions with the business users along the way
  • Create wide tables and data marts and build dashboards for final UAT and approval
  • Document the entire process for the company/client

Sample Data Engineering Portfolio Projects

Do know where to get started from? Get some inspiration from WeCloudData students/alumni. You can find some project demos on our youtube channel: https://youtube.com/@weclouddata

Check out other career guides

Our Career Guide provides all the resources you will need to help you get started in navigating data careers. We include free resources, guides, and tools that will help you get started.