One of the biggest hurdles when building a Data Engineering portfolio is finding good data. The classic “Titanic” or “Iris” datasets are fine for basic data science, but they don’t challenge a Data Engineer. You need messy, large, or streaming data to demonstrate that you can build robust pipelines.
Here is a curated list of where to find the best datasets for Data Engineering projects, and how to use them effectively.
1. Kaggle (The Go-To Source)
Kaggle is the most popular platform for finding datasets. While often used for machine learning, it hosts massive datasets perfect for DE pipelines.
- What to look for: Search for datasets that are frequently updated (daily/weekly) or have multiple related tables (relational data).
- Examples: The Olist Brazilian E-Commerce Dataset (multiple tables to practice joins and dimensional modeling) or Crypto Market Data (time-series).
- How to use it: Don’t just download a CSV. Use the Kaggle API to programmatically ingest the data into your Azure Data Lake on a schedule.
2. Public Cloud Datasets (AWS / Azure)
Major cloud providers host massive public datasets that are completely free to access.
- AWS Registry of Open Data: Hosts petabytes of data, including weather data (NOAA), satellite imagery, and the famous NYC Taxi Trip records.
- Azure Open Datasets: Includes curated public data like COVID-19 tracking, weather, and holidays.
- How to use it: These datasets are large. Practice building incremental loading pipelines (e.g., only pulling yesterday’s NYC taxi data) using Databricks or Azure Data Factory.
3. APIs for Streaming & JSON Data
Modern data pipelines frequently ingest data from APIs. You need to practice handling nested JSON and pagination.
- OpenWeatherMap API: Great for pulling live weather data to practice streaming or micro-batching.
- Twitter / X API (or Reddit API): Good for practicing real-time streaming (using tools like Kafka or Event Hubs).
- The Rick and Morty API: A fun API that forces you to deal with nested JSON and pagination.
4. Mockaroo (Generate Your Own)
If you want to practice specific edge cases (like handling nulls, bad formats, or slowly changing dimensions), sometimes the best data is fake data.
- Mockaroo lets you design custom schemas and generate millions of rows of realistic dummy data.
- How to use it: Design a schema that simulates a realistic business problem (e.g., users, transactions, and product catalogs) and deliberately introduce errors (e.g., 5% null values in a required field) to test your data quality checks.
Pro-Tip: Focus on the “Engineering”, Not the Data
A common mistake is spending weeks analyzing the data. As a Data Engineer, your job is to move, clean, and store the data reliably. Your portfolio project should highlight your Medallion architecture (Bronze/Silver/Gold), your idempotency (can the pipeline run twice safely?), and your orchestration, rather than the final dashboard.