Data Prep

Get raw data from operational systems (databases, SaaS, files, streams) into a place you can analyze it.

• Batch: nightly dumps via Fivetran / Airbyte / custom scripts.
• CDC (Change Data Capture): stream every row change in near-real-time (Debezium, Estuary).
• Streaming: Kafka / Kinesis for event firehoses.

import pandas as pd

df = pd.read_parquet("raw/orders.parquet")
df = (df
    .drop_duplicates(subset="order_id")
    .dropna(subset=["customer_id"])
    .assign(
        order_date = pd.to_datetime(df.order_date),
        revenue    = df.qty * df.unit_price,
    )
    .query("revenue > 0")
)

Modern teams increasingly do this in SQL via dbt, or in Polars / DuckDB for 10-100× speedups over pandas on large data.

• Encoding: categorical → one-hot, target, hashing.
• Scaling: standardize / normalize for distance-based models.
• Aggregations: rolling sums, group-by means, lags, recency.
• Datetime: hour-of-day, day-of-week, holiday flags.
• Text/Image: embeddings (often produced by a foundation model).

Fail loud, fail early. Tests run as part of the pipeline:

• Schema: column exists, type is right.
• Range: price > 0, percentage in [0,100].
• Cardinality: distinct values within bounds.
• Freshness: latest timestamp not older than X.
• Uniqueness & nulls.

• Manual labeling in tools like Label Studio (with QA & agreement scoring).
• Programmatic labeling (Snorkel) — write weak rules, train a label model.
• LLM-assisted labeling — use a strong model to pre-label; humans verify.
• Active learning — label only the examples the model is most uncertain about.

Beyond classical ML data prep, LLM apps need:

• Document chunking — split long documents into RAG-friendly pieces (semantic, fixed-size, recursive).
• Cleaning — strip nav/ads from HTML, OCR PDFs, normalize whitespace.
• Embedding generation — batch-call an embedding API; store in a vector DB.
• Metadata enrichment — tag chunks with source, date, ACL for filtering.
• Eval datasets — golden Q&A pairs to measure RAG quality over time.