Data Engineer Career Path

The concept of data engineering covers a wide range. Whether building a data warehouse or displaying big data analysis result on the front end of a mobile app, it is the work of a data engineer. Basically, data engineering has two broad directions: Data Warehousing direction, and Big Data direction.

Data Warehousing: This direction is mainly to work in the data warehouse, perform data modeling in the data warehouse, and perform various data processing and ETL process. The main skills required for this kind of data engineering are SQL, Python and Linux. More senior data engineers in this area will become data architects, responsible for designing and optimizing the data architecture of the entire enterprise.

Big Data: Data engineering in the direction of big data mainly uses big data technologies, such as Spark, to perform batch or stream processing of multiple data sources. This kind of data engineering is more inclined to software engineering, and the system structure is more complex. In this regard, data engineers need to better understand the architecture of cloud platforms and use more complex system tools, such as Kubernetes, Kafka, etc. Senior data engineers in this regard will become data system architects.

Of course, each type of data engineers is not independent of each other, and many companies require data engineers to have both capabilities.

If you want to know if data engineer is the right path for you, please watch the following video:

If data engineer is the right path for me?