feed-forge

A small, strict, dependency-free library for validating and normalizing e-commerce product feeds in the style of a Google Merchant Center or Meta catalog import.

Why

Product feeds are where clean data goes to die. The same catalog reaches me as a CSV export, a Shopify JSON payload, and a hand-maintained spreadsheet, and every source has its own idea of what "in stock" means, whether a price is "19.99", 19.99, or "19,99 USD", and which fields are optional this week. Merchant platforms then reject the whole upload over problems they report one at a time.

I built feed-forge to move that mess behind a single, predictable boundary: hand it raw rows, get back a list of clean, typed products plus a full report of everything wrong with the rest. It does not fail on the first bad row, it does not do any I/O, and it never stores money as a float.

Install

pip install -e '.[dev]'   # from a checkout

Requires Python 3.11+ and has zero runtime dependencies.

Quick start

from feed_forge import process_feed, Product

feed = [
    Product(
        id="SKU-1",
        title="Blue Cotton Shirt",
        price="19.99",
        currency="usd",
        availability="in stock",
        link="https://shop.example.com/p/1",
        image_link="https://cdn.example.com/1.jpg",
        gtin="4006381333931",
    ),
    {  # plain dicts work too, via Product.from_mapping
        "id": "SKU-2",
        "title": "Red Mug",
        "price": "-3",
        "currency": "ZZZ",
        "availability": "maybe",
        "link": "notaurl",
        "image_link": "https://cdn.example.com/2.jpg",
    },
]

result = process_feed(feed)

for product in result.products:
    print(product.id, product.price_minor, product.currency, product.availability)
# SKU-1 1999 USD in_stock

for issue in result.issues:
    print(issue.product_id, issue.field, issue.code.value, issue.severity.value)
# SKU-2 price non_positive_price error
# SKU-2 currency invalid_currency error
# SKU-2 availability invalid_availability error
# SKU-2 link invalid_url error

price is returned as price_minor: an integer in the currency's ISO 4217 minor units (cents for USD, whole yen for JPY, thousandths for KWD). currency is upper-cased, and availability is mapped to one of in_stock, out_of_stock, preorder, backorder.

Adding your own rule

A rule is just a function NormalizedDraft -> list[Issue]. Inject a custom rule set without touching the core:

from feed_forge import DEFAULT_RULES, Issue, IssueCode, Severity, process_feed


def require_brand_prefix(draft):
    if draft.id is not None and not draft.id.startswith("ACME-"):
        return [
            Issue(
                draft.id,
                "id",
                IssueCode.MISSING_FIELD,
                "id must start with ACME-",
                Severity.WARNING,
            )
        ]
    return []


result = process_feed(feed, rules=(*DEFAULT_RULES, require_brand_prefix))

Commands

ruff check .            # lint
ruff format --check .   # formatting
mypy src                # strict type check
pytest                  # tests + coverage
make check              # all of the above

Architecture

The core is pure: normalization and validation are separate passes, rules are independent functions, and nothing reads or writes I/O. That is what makes the whole thing trivially unit-testable and safe to embed anywhere.

flowchart TD
    A[Raw products: Product or dict] --> B[normalize_product]
    B --> C[NormalizedDraft: cleaned, best-effort typed]
    C --> D{Rule set}
    D --> R1[require_identity]
    D --> R2[validate_price]
    D --> R3[validate_currency]
    D --> R4[validate_availability]
    D --> R5[validate_link / image_link]
    D --> R6[validate_gtin - warning]
    D --> R7[check_title_length - warning]
    R1 & R2 & R3 & R4 & R5 & R6 & R7 --> E[Issues collected per product]
    E --> F{Any error-severity issue?}
    F -->|no| G[NormalizedProduct]
    F -->|yes| H[Product dropped, issues kept]
    G --> I[ValidationResult.products]
    H --> J[ValidationResult.issues]

Loading

Normalization is deliberately total: it never raises and never emits issues. It cleans what it can and leaves the rest as None. Judging whether a normalized value is acceptable belongs to the rules, which keeps "how do we clean this" and "is this valid" from bleeding into each other.

Design decisions (ADRs)

ADR	Decision	Why
0001	Money as integer minor units	No float rounding bugs in prices
0002	Rules as composable functions	Open/closed: add a rule without editing the core
0003	Pure core, no I/O	Deterministic, embeddable, trivial to test
0004	Collect all issues, no fail-fast	Fix a feed in one pass, not one error at a time

Lessons learned and future work

• Splitting normalization from validation paid off immediately. Once normalization was guaranteed total, every rule collapsed to a small pure function with an obvious test, and the double-reporting ("missing" vs "unparseable") disappeared because each concern owned one field.
• Keeping the raw Product on the draft was the detail that made good error messages possible: a rule can tell "field absent" from "field present but unusable" without re-parsing anything.
• Storing money as int minor units removed a whole class of == comparison and rounding tests that a float design would have needed.

Future work I would pick up next:

• Pluggable per-locale price parsing (comma decimals, embedded currency suffixes like "19,99 zl").
• A fuller ISO 4217 table plus opt-in country/currency cross-checks.
• CSV and Merchant Center XML adapters as a thin, separately tested I/O layer on top of the pure core.
• Optional GTIN warning -> error escalation for feeds where a barcode is mandatory.