Imagine: marketing asks for yesterday's funnel report, and it turns out the nightly ETL failed and the data for the last three days is incomplete. Analysts are already building charts, managers are making decisions — all based on leaky numbers. Sound familiar? This is a classic situation in teams where data pipelines are cobbled together. Data Engineering as a discipline exists precisely so that such stories end before they begin.
In this article, we'll break down what a data engineer's work consists of, show with a concrete example how a production pipeline is built with Apache Spark and dbt, and honestly explain where things most often break. We'll also compare learning paths — from digging through documentation on your own to structured courses like Data Engineering (Spark, dbt).
What Data Engineering Is and Why Business Needs It
Data Engineering is the design, construction, and maintenance of infrastructure for collecting, storing, and transforming data. If an analyst answers the question "what happened?", a data engineer builds the system that guarantees the answer is based on complete, up-to-date, and high-quality data.
The key task is building data pipelines: sequences of steps that take raw data from sources, clean it, enrich it, aggregate it, and load it into a warehouse or data mart for analytics and ML.
Why this is critical. As a company grows, the number of sources (CRM, billing, web analytics, application logs) grows avalanche-like. Manual data collection and preparation stop scaling: analysts spend hours on extracts, and the business gets conflicting numbers from different reports. A data engineer solves this problem systematically — through automation and quality control.
What a beginner needs to understand. Data Engineering is not about writing code for code's sake. It's about reliability. A pipeline that ran once on a laptop and a pipeline that runs in production 24/7 and withstands failures are two different things. The core skills lie in three areas: distributed computing (Spark), transformations and documentation (dbt), orchestration and monitoring (Airflow, Dagster, Great Expectations).
Breaking Down a Real Task: Building a Pipeline from Raw Logs to a Data Mart
To understand how this works, let's take a typical scenario: an online store wants to see daily revenue by product category, taking returns into account. Data comes from two sources: an order log (JSON files in S3, about 5 GB per day) and a returns table in PostgreSQL.
Step 1. Ingestion: Pulling Raw Data
First, you need to deliver the data to the data lake. For logs, we use Spark Structured Streaming or batch loading — depending on latency requirements. If the report is needed once a day, batch processing is sufficient.
# Example of reading JSON logs from S3 and writing to Delta Lake
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("orders_ingest") \
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension") \
.getOrCreate()
raw_df = spark.read.json("s3://bucket/orders/2026-11-07/*.json")
raw_df.write.format("delta").mode("append").save("s3://datalake/bronze/orders")
What goes wrong if you skip this step. Without the bronze layer (raw data as-is), you lose the ability to reprocess history when you discover a logic error. Always keep the source data unchanged — it's your insurance.
Step 2. Transform: Cleaning and Enriching with Spark
Now we turn the raw JSON into a table with the required fields, filter out garbage, and cast types.
from pyspark.sql.functions import col, to_date, when
orders = spark.read.format("delta").load("s3://datalake/bronze/orders")
cleaned = orders \
.filter(col("status").isin("paid", "shipped", "delivered")) \
.withColumn("order_date", to_date(col("created_at"))) \
.withColumn("amount", when(col("amount") < 0, 0).otherwise(col("amount"))) \
.select("order_id", "user_id", "product_id", "category", "amount", "order_date")
cleaned.write.format("delta").mode("overwrite").save("s3://datalake/silver/orders")
Typical mistake: writing overwrite without partitioning — every run rewrites the entire table, which is expensive and slow. Better to use partitionBy("order_date") and mode("overwrite").option("replaceWhere", "order_date = '2026-11-07'") for incremental processing.
Step 3. Modeling in dbt: Building the Data Mart
dbt lets you describe transformations as SQL models with dependencies and tests. Let's create a model that joins orders and returns and calculates revenue.
-- models/marts/revenue_by_category.sql
{{ config(materialized='table') }}
WITH orders AS (
SELECT * FROM {{ ref('silver_orders') }}
),
returns AS (
SELECT order_id, SUM(refund_amount) AS refunded
FROM {{ ref('stg_returns') }}
GROUP BY order_id
)
SELECT
o.category,
o.order_date,
SUM(o.amount) - COALESCE(SUM(r.refunded), 0) AS revenue
FROM orders o
LEFT JOIN returns r ON o.order_id = r.order_id
GROUP BY 1, 2
What's important: dbt automatically builds a dependency graph and runs models in the correct order. But if you don't set up tests, you can silently get negative revenue due to incorrect returns.
Step 4. Orchestration and Monitoring
The ready steps need to be run on a schedule and checked for quality. For orchestration we use Airflow or Dagster, for quality control — Great Expectations.
# Example check in Great Expectations: amount must not be negative
import great_expectations as ge
df = ge.read_delta("s3://datalake/silver/orders")
result = df.expect_column_values_to_be_between("amount", min_value=0)
assert result.success, "Negative order amounts detected"
Bottom line: a four-step pipeline turns raw logs into a ready data mart. Each step is isolated, data between layers (bronze → silver → gold) is preserved, and quality checks are in place. Such a pipeline can be restarted on failures and extended without fear of breaking something.
Where the Mechanism Breaks: 4 Failure Scenarios
Even a well-designed pipeline can fail. Here are the most common causes — not beginner mistakes, but systemic failures.
1. Source Schema Changed
What happens: a new field is added to the CRM or a column is renamed. Spark reads the JSON and fails with a schema mismatch error, or, worse, silently substitutes null. dbt models tied to old names break.
How to catch it early: enable schema validation at the ingestion stage. In Spark, use spark.read.schema(defined_schema) instead of automatic inference — then on mismatch you'll get an explicit error, not a silent failure. In dbt, add not_null and accepted_values tests on key columns.
What to do if it's too late: roll back to the last successful data version (Delta Lake and Iceberg support time travel), fix the schema, and restart the pipeline from the historical date. Set up alerts for job failures.
2. Rule Not Applied (Business Context Changed)
What happens: the revenue calculation logic is hardcoded in a SQL model, but the finance department changes the definition of "return" — now some adjustments are not considered returns. Your pipeline keeps calculating the old way, and the numbers diverge from accounting.
How to catch it early: document business rules in dbt (model and column descriptions) and maintain feedback with stakeholders. Regularly reconcile final numbers with a reference report.
What to do: move business logic into separate dbt models or macros so that a rule change is in one place. If a discrepancy is found, recalculate the data mart for the affected period.
3. Retention Expired or Data Deleted
What happens: a lifecycle policy was set in S3, and after 30 days the raw logs were automatically deleted. You can't reprocess history when you discover an error.
How to catch it early: check cloud retention policies and set up archiving to cold storage (Glacier, Coldline). Maintain a data catalog with retention periods.
What to do: if the data is already lost, the only option is to request restoration from the source (if it still stores it) or accept the loss. In the future, set up auditing and versioning.
4. Tool Stopped Working (Update, Failure)
What happens: after a Spark update, a function's behavior changed, or a dbt package became incompatible with the new version. The pipeline fails at the compilation stage.
How to catch it early: use lock files for dependencies (requirements.txt, packages.yml) and test updates in an isolated environment. Subscribe to tool releases.
What to do: pin versions in production, roll back to the previous working combination, and test the update on a copy of the data.
Learning Paths for Data Engineering: An Honest Trade-off
There are several ways to enter the profession. Each has its own cost and outcome.
Self-Study
Pros: free (or almost), flexible schedule, access to documentation and open courses.
Cons: no structure, easy to go off track or get stuck on irrelevant details. No feedback: you won't know if you designed the pipeline correctly until you hit a problem in real work. Architectural mistakes can be costly.
Time: from 6 months to several years depending on discipline.
On-the-Job Practice (If You're Lucky with the Team)
Pros: real tasks, mentorship, understanding of business context.
Cons: not always a mentor available; you can repeat the same operations for years without learning best practices. Risk of getting stuck at the level of "maintaining legacy scripts".
Structured Courses
Pros: sequence, proven materials, feedback on solutions, analysis of typical mistakes. Saves time by focusing on what matters.
Cons: cost (typically a few hundred dollars), need to allocate time on a schedule.
Comparison table:
| Criterion | Self-Study | Courses | On-the-Job Practice |
|---|---|---|---|
| Time to first pipeline | 6–12 months | 3–6 months | Depends on tasks |
| Feedback | No | Yes (via chat, assignment review) | Yes, but not always |
| Depth | Depends on motivation | Systematic | Fragmented |
| Cost | Low | Medium/High | Free (but time) |
Non-obvious conclusion: the fastest path is not courses or books, but a combination: a course provides a framework and feedback, while independent projects on real (even synthetic) data reinforce the skill. Without the second, the first fades within a month.
Who Data Engineering Suits and Who It Doesn't
Suits you if:
- you love systematicity and order, you enjoy building infrastructure, not just analyzing;
- you have basic experience in SQL and Python (or are ready to learn quickly);
- you're prepared for the fact that part of the work is debugging pipelines and monitoring, not just creative design;
- you're interested in how data lives in the cloud and how to ensure its quality.
Doesn't suit you if:
- you want to focus only on visualization and reports (that's closer to analytics);
- you find it hard to work with abstractions and distributed systems (Spark requires understanding how data is sharded and moved);
- you're not ready for routine: a significant portion of time goes into maintenance and bug fixing, not creating new things.
Important: Data Engineering is not "just SQL". It's an engineering discipline with an emphasis on reliability and scalability. If you like writing code and seeing a system run like clockwork, this is your path.
What's Next: How to Start
If you've read the breakdown and realized you want to build pipelines, not just observe, start small: try building a local pipeline with Spark and dbt on open data (e.g., from Kaggle). But keep in mind: without feedback, it's easy to reinforce incorrect patterns. The course Data Engineering (Spark, dbt) provides a structured path: from the basics of distributed computing to production monitoring and cost optimization. It covers exactly the failure scenarios we discussed and gives feedback on your solutions. If you want to save time and avoid typical pitfalls — check out the course page. See also , and Cryptography in Practice.
You can start here: Data Engineering. See also , and Shadowing, Not Translating: How ChatGPT Prompts Turn Vocabulary Lists Into Actual Speech.