A guide to data engineering resources and communities for beginners and practitioners
When entering a new field and unsure where to start, this repository organizes the necessary materials systematically. It is a collection of links covering data engineering books, courses, communities, and industry trends.
Learning roadmap for beginners
Those new to data engineering can follow the 2024 entry roadmap. Introductions and required software lists are provided for a 4-week introductory bootcamp and a 6-week intermediate bootcamp. For practical application, it is recommended to refer to the practice examples in the projects section and the interview tips in the interviews section. The books, communities, and newsletter sections are also categorized as important resources for learning.
Essential books and key communities
From a list of more than 25 books, 'Fundamentals of Data Engineering', 'Designing Data-Intensive Applications', and 'Designing Machine Learning Systems' are recommended as essential reading. Key data engineering communities include DataExpert.io Discord, Data Talks Club Slack, and Data Engineer Things Community, while AdalFlow Discord and Chip Huyen MLOps Discord are highlighted for the machine learning field. These communities are useful for resolving questions that arise during the learning process and sharing trends.
Tools categorized by use case
Tools used in data engineering are clearly distinguished by purpose. Orchestration includes Airflow, Dagster, and Prefect, while data lakes and cloud services include Databricks, Delta Lake, and Apache Iceberg. Snowflake, Firebolt, and Databend are listed as data warehouses, and dbt, Great Expectations, and Soda are listed for data quality management. Additionally, tool lists are provided for specific fields such as analytics visualization, data integration, semantic layers, modern OLAP, LLM application libraries, real-time data aggregation, and data lineage tracking.
Industry blogs and content sources
Links to data engineering blogs from major companies such as Netflix, Uber, Databricks, Airbnb, Amazon AWS, Microsoft, Oracle, and Meta are collected, allowing you to review practical cases. A list of whitepapers that formed the foundation of data engineering, such as 'Lakehouse', 'Big Data Quality', 'Spark', and 'MapReduce', is also included. Creator lists and follower counts are organized by social media platform, including YouTube, LinkedIn, X/Twitter, Instagram, and TikTok, making it easy to find channels of interest. More than 20 podcasts and newsletters are provided, enabling learning through various media.
Courses, certifications, and pre-use checks
The list includes courses from various platforms such as DataExpert.io, LearnDataEngineering.com, IBM, DataCamp, and Udemy, as well as specialized certification programs from DeepLearning.AI. Information on data engineering certification exams from Google Cloud, Databricks, Microsoft, and AWS is also available. Since this repository is a collection of links, the validity and current status of each link must be verified directly. Access rights or costs for specific courses or communities vary according to each service's policy, so individual confirmation is required before participating. The README does not provide explicit information on license or maintenance cycles, so the continuous updating of materials must be judged directly at the time of visit.