[MNT] narwhals migration by solegalli · Pull Request #965 · feature-engine/feature_engine

solegalli · 2026-07-21T20:40:09Z

No description provided.

ojassharma7 · 2026-07-28T22:22:31Z

Hi @solegalli — I'd like to help with the narwhals migration.

If it is still free, I can take feature_engine/scaling first (small surface: mainly MeanNormalisationScaler) as a single-module PR, following the dataframe_checks pattern from #966.

Please let me know if that module is already spoken for — happy to pick another (e.g. a simpler preprocessing piece) instead.

solegalli · 2026-07-29T13:54:17Z

That is actually a good one to start with. The tests should pass with pandas. I am not sure they will pass with polars because we need to change the functions that select variables, on which I am working on right now and will soon make a PR.

ojassharma7 · 2026-07-29T15:06:16Z

Started on scaling as discussed — opened a PR against this branch: will link here once created (see latest open PR from @ojassharma7 titled migrate scaling module to narwhals).

Pandas tests for the module pass locally. As you said, polars may still need your variable-selection updates.

ojassharma7 · 2026-07-29T15:06:22Z

Scaling PR: #979

FBruzzesi

Hi @solegalli - Following up on my discord comment. I added a few comments, mostly focusing on the changes in feature_engine/dataframe_checks.py - I hope you find them helpful

I noticed that a lot of tests were refactored as well: if you want to test the same behavior for many dataframes, I would reference what I did for fairlearn (see their conftest file), namely create fixture dataframe constructor for all the dataframe types you want to test. Ideally I would like to move that into narwhals as well (see narwhals-dev/narwhals#3552), but that's still work-in-progress and under discussion 🙏🏼

FBruzzesi · 2026-08-23T20:17:09Z

-    elif isinstance(y, pd.DataFrame):
-        if y.isnull().any().any():
+        if nw_y.dtype.is_numeric():
+            if not np.isfinite(nw_y.to_numpy()).all():


Suggested change

if not np.isfinite(nw_y.to_numpy()).all():

if not nw_y.is_finite().all():

(see Series.is_finite())

Hi @FBruzzesi , thanks for the suggestion. It seems that using numpy is faster than using narwhals both for pandas and polars (mostly so for pandas). Is this a known issue?

For the polars case we run its native functionality polars.Series.is_finite. I am surprised that's faster than numpy, at least at scale

For pandas-like, we do (s > float("-inf")) & (s < float("inf")). IIRC that's to avoid using numpy with non-numpy backed series (e.g. pyarrow backed series, cudf series that live in the GPU, etc). If the delta is large at scale, we can take a look for a refactor with performance in mind.

For context: in general we tend to use the native dataframe libraries API/functionalities. pandas is a special kid as we need to do quite some gymnastic for null vs nan's, its datatype system, its multiple backends, etc..

So please keep reporting these kind of performance issues - we aim to keep overhead at the minimum

Thanks for replying so quickly. These are the values I've got (on pandas and polars, 200k rows × 20 cols):

Check pandas polars
null check (multi-col) narwhals-native 1.2x slower narwhals-native 4x slower
inf check (multi-col) narwhals-native 2.4x slower narwhals-native 1.3x slower
is_finite (single series) narwhals-native 10x slower ~same

is_finite is the same for polars, the inf and null checks make it a bit slower respect to numpy.

For pandas I just opened a PR to use numpy/cupy/pyarrow.compute native functionalities directly: see narwhals-dev/narwhals#3874

For polars, I cannot tell why numpy is faster than their native implementation - If interested, you can double check with them either in discord or in their repo

FBruzzesi · 2026-08-23T20:18:11Z

+
+    if nwd.is_into_dataframe(y):
+        nw_y = nw.from_native(y, eager_only=True)
+        if nw_y.select(nw.all().is_null().any()).to_numpy().any():


You can avoid casting to numpy:

Suggested change

if nw_y.select(nw.all().is_null().any()).to_numpy().any():

if nw_y.select(nw.any_horizontal(nw.all().is_null().any())).item():

FBruzzesi · 2026-08-23T20:20:11Z

+        "`missing_values='ignore'` when initialising this transformer."
+    )
+    nw_X = nw.from_native(X, eager_only=True)
+    if nw_X.select(nw.col(variables).is_null().any()).to_numpy().any():


Similar as above, you can use any_horizontal

* address review feedback on dataframe_checks.py Follow-up to FBruzzesi's review on PR #965: - Clarify docstrings for check_X, check_y, check_X_y in terms of which dataframe libraries are safe to pass in (pandas, polars, PyArrow, modin, cuDF), instead of narwhals-specific "eager" terminology. - Use narwhals' IntoDataFrameT instead of IntoDataFrame for check_X and check_X_y, since both return the same concrete dataframe type they receive. - Fix a null/NaN detection bug: in polars, is_null() does not catch an explicit float("nan") value (only None counts as null), so check_y and _check_contains_na could silently miss NaNs in polars data. Now also check is_nan() for numeric columns/series, keeping numpy for the finite/inf checks since it benchmarks as fast or faster there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * inline null/nan checks to match FBruzzesi's suggested one-liner Collapses the has_na/has_null/has_nan accumulator variables into a single short-circuiting if-condition, as suggested in review. This also avoids an unnecessary is_nan() call when is_null() already found a null value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * speed up numeric column detection in _check_contains_na The schema-based list comprehension rebuilt narwhals' full column schema on every single-column access, making it scale roughly quadratically with column count on pandas (benchmarked up to ~500x slower than necessary at 200 columns). Switch to the pandas fast-path / narwhals-selector pattern already used in variable_handling (find_numerical_variables, check_numerical_variables) for the same "which of these columns are numeric" problem. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* update dataframe checks * update dataframe checks take 2 * update dataframe checks take 3 * update docstrings * refactor dataframe checks * fix mypy error * add missing type hints * add missing matching error syntax * finalise tests for df checks'

The project already requires scikit-learn>=1.7.0 (pyproject.toml, tox.ini, .circleci/config.yml), so the sklearn<=1.6 branches of every check_estimator/tags conditional were dead code. This removes them, keeping only the >=1.6 branch (the one using check_estimator(expected_failed_checks=...)): - feature_engine/tags.py: collapse the sklearn_version > 1.6 check in _return_tags(), the shared helper used across ~20 estimator classes. - 11 tests/**/test_check_estimator_*.py files: collapse each if/else on sklearn_version vs 1.6, drop the now-unused sklearn/ parse_version imports and sklearn_version variables. - tests/test_prediction/test_check_estimator_prediction.py: this file had no >=1.6 branch, only the dead <1.6 one (its own TODO already flagged this). Removing it leaves the prediction module with no test_check_estimator_from_sklearn coverage - a pre-existing gap, not introduced by this change, left as a follow-up. - tests/test_creation/test_geo_features.py: __sklearn_tags__ always exists at sklearn>=1.7, so drop the hasattr() guard around it. - tests/test_wrappers/test_sklearn_wrapper.py: also collapse the _OneHotEncoder() test helper's sparse/sparse_output branch (sklearn <1.2 compat, dead for the same reason). The separate KBinsDiscretizer(quantile_method=...) branch (sklearn<1.7) is intentionally left as-is - different threshold, out of scope here. - tests/check_estimators_with_parametrize_tests.py: delete entirely. A standalone, non-CI reference file documenting the pre-1.6 parametrize_with_checks() call signature. _more_tags()/__sklearn_tags__() method definitions are untouched: _more_tags() is feature_engine's own internal metadata/xfail-checks store (read by tests/estimator_checks/*.py), not a legacy sklearn shim, and __sklearn_tags__() is the current sklearn API. Verified: identical test suite pass/fail counts before and after (2010 passed, 114 failed - all 114 are pre-existing narwhals-migration WIP failures unrelated to this change), flake8 and mypy clean (the one remaining mypy error is pre-existing in datetime_subtraction.py, unrelated to this PR).

* address review feedback on dataframe_checks.py Follow-up to FBruzzesi's review on PR #965: - Clarify docstrings for check_X, check_y, check_X_y in terms of which dataframe libraries are safe to pass in (pandas, polars, PyArrow, modin, cuDF), instead of narwhals-specific "eager" terminology. - Use narwhals' IntoDataFrameT instead of IntoDataFrame for check_X and check_X_y, since both return the same concrete dataframe type they receive. - Fix a null/NaN detection bug: in polars, is_null() does not catch an explicit float("nan") value (only None counts as null), so check_y and _check_contains_na could silently miss NaNs in polars data. Now also check is_nan() for numeric columns/series, keeping numpy for the finite/inf checks since it benchmarks as fast or faster there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * inline null/nan checks to match FBruzzesi's suggested one-liner Collapses the has_na/has_null/has_nan accumulator variables into a single short-circuiting if-condition, as suggested in review. This also avoids an unnecessary is_nan() call when is_null() already found a null value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * speed up numeric column detection in _check_contains_na The schema-based list comprehension rebuilt narwhals' full column schema on every single-column access, making it scale roughly quadratically with column count on pandas (benchmarked up to ~500x slower than necessary at 200 columns). Switch to the pandas fast-path / narwhals-selector pattern already used in variable_handling (find_numerical_variables, check_numerical_variables) for the same "which of these columns are numeric" problem. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* update variable handling module for narwahls * creating own datetime parser * Improve readability of narwhals date/type-check helpers, add missing tests Replace the double-parse-with-disagreeing-defaults trick in _looks_like_date_string with a direct call to dateutil's parser()._parse(), which exposes which date/time fields were actually found in a string without needing to approximate it - this also drops the now-unneeded sentinel default datetimes and the defensive str() coercion at its call site. Make truthiness checks and compound boolean returns explicit throughout the module, and restore the pre-narwhals function names that PR #978 had prefixed with _nw_ for no continuing reason. Rename test_fe_type_checks.py to test_variable_type_checks.py to match the module it tests, add docstrings, and add coverage for _looks_like_date_string and _is_categories_num, the two functions that previously had no direct tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Replace per-column schema access with bulk narwhals selectors for speed nw_X.schema is not cached - every access re-derives the full schema from the underlying native dataframe, so checking dtype-based conditions (is_numeric(), native Date/Datetime, categorical/enum/string) one column at a time inside a loop was quadratic instead of linear. Replace each such loop with a single nw_df.select(<selector>).columns call converted to a set, then a plain membership test per column - confirmed old vs new give identical results, and measured 8x-120x speedups depending on backend and column count. Also use by_dtype(Date, Datetime) to bulk-detect native datetime columns in one pass, only falling back to the expensive per-value _is_categorical_and_is_datetime check for columns that aren't already known to be numeric or natively datetime. Drop the now-unused _is_date_or_datetime import from both files. Simplify _looks_like_date_string's comment to link directly to the pandas source it mirrors, and instantiate dateutil's parser() per call instead of reusing a module-level instance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor find variables: * refactor check_variables * final refactor of find and check variables * finalise migration of variable handling module * update user guide * Trim backend-difference notes from docs, revert datetime.py out of scope Removes the trailing pandas/polars note blocks from the check/find categorical and datetime variable docs, keeping them focused on the walkthrough. Reverts feature_engine/datetime/datetime.py to main - the DatetimeFeatures index-datetime fix needed there for the narwhals migration belongs in a separate datetime-module PR. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@FBruzzesi

* Migrate creation/mixins shared base classes to narwhals, remove all pandas imports BaseCreation, BaseNumericalTransformer, and mixins.py (TransformXyMixin, FitFromDictMixin, GetFeatureNamesOutMixin) are used by every transformer in the creation module, so their remaining pandas-only code blocked a polars-only install regardless of which transformer was migrated. Adds pandas fast paths (benchmarked ~2-11x) alongside narwhals-generic branches, replaces y.loc[X.index] row alignment in TransformXyMixin with a narwhals with_row_index()-based mechanism for non-pandas backends, and adds test_base_creation.py plus polars coverage for transform_x_y. * Apply suggestion from @FBruzzesi * Fix test_get_feature_names_out_mixin.py after to_list() removal, add process rules to AGENTS.md The 48 failures here were pre-existing (unrelated to the to_list() fix, confirmed identical before/after): check_X no longer accepts raw numpy arrays, and most of this file's tests fit() on df_vartypes.to_numpy() or feed a raw-array-outputting sklearn transformer upstream. Fixes: - array-input tests converted to set feature_names_in_/n_features_in_ directly, since that's the only way left to reach the mixin's x0/x1/... naming branch (fit() rejects arrays outright now). - SimpleImputer/PolynomialFeatures steps get .set_output(transform="pandas") so they hand a dataframe to the next pipeline step instead of an array - this is also the fix any real user chaining sklearn + feature-engine transformers in a Pipeline now needs. - pure Mock-only tests (no sklearn transformer involved) parametrized over pandas and polars. Also adds two AGENTS.md rules: run a changed function/class's tests and resolve any failures, and keep user-guide docs in sync with new transformer functionality. * Remove dead array-input branch from GetFeatureNamesOutMixin This branch handled feature_names_in_ == ["x0", "x1", ...], the naming sklearn gives an estimator fit on a raw array. check_X no longer accepts arrays (dataframe-only input, per AGENTS.md), so fit() can never produce that pattern anymore - the branch, its indices=True path in _remove_feature_names, and get_support(indices=True) were all unreachable. It was also a latent correctness gap: a dataframe with columns genuinely named x0..xn would have hit this branch and skipped the usual input_features-must-match-feature_names_in_ validation. Verified via git history (#519, 2022) this was built for the old array-accepting check_X; confirmed no other code in the library still generates x0/x1/... names. Removed the branch, its now-single-path _remove_feature_names, and the tests that existed only to reach it - replaced by tests/test_base_transformers/test_get_feature_names_out_mixin.py's remaining pandas+polars dataframe coverage, which already exercises the same validation/renaming logic through the one reachable path.

* Migrate CyclicalFeatures to narwhals, add polars support fit(): unified across backends via .to_numpy().max(axis=0) instead of pandas' .max().to_dict() (~1.55x faster for pandas, ~1.28x for polars, benchmarked). .tolist() keeps the returned dict's values as plain Python int/float, matching the old .to_dict() dtype. transform(): kept as two branches rather than one narwhals-only path - benchmarked running narwhals expressions against a pandas-backed frame and it was consistently 1.24x-2.06x slower than the pandas-native loop across variable counts and row counts, worse at small scale. The pandas branch is therefore left as the original, unmodified loop (an earlier numpy-vectorized version of it was only a 1.0x-1.4x gain, not worth it once the branches stay separate anyway). The narwhals branch uses column expressions, the only approach that stayed competitive with pandas-native as variable count grows (a numpy-array round-trip loses to expressions on polars once there is more than 1 variable). Verified no legacy numpy-array-input code remains in this file or its base classes. Tests rewritten to parametrize pandas and polars via make_df; error-matching tightened per AGENTS.md except where the message legitimately differs by backend. Docstring and user-guide example gained a polars walkthrough per the new AGENTS.md doc-sync rule. * unify pandas/polars branches * Fix style/docs failures on top of the pandas/polars branch unification Style: removed the now-unused narwhals.dependencies import (flake8 F401) left over from dropping the is_pandas_dataframe branch. Also fixed 7 pre-existing flake8 issues (line length, unused variable) in test_get_feature_names_out_mixin.py that predate this branch. Docs: docs/user_guide/creation/CyclicalFeatures.rst's polars output block was under `.. code:: python`, and Sphinx's Pygments highlighter can't lex the box-drawing table as Python (misc.highlighting_failure), which -W promotes to a build error. Switched to `.. code:: text`, matching the convention already used elsewhere (PowerTransformer.rst, MeanImputer.rst) for output-only blocks. Pre-existing bug in my own doc addition, unrelated to the branch unification. Two correctness issues surfaced by testing the unification: - max_values_ lost its .tolist() call, so it held numpy scalars (np.int64) instead of plain Python int/float - restored. - narwhals' .select([]) collapses row count to 0 (not just columns), so routing pandas through the narwhals numpy path broke return_empty=True (empty variables_) with a "zero-size array to reduction operation maximum" error. Guarded for it explicitly, since return_empty=True is a real, designed-for case, not a hypothetical.

@solegalli

* Migrate GeoDistanceFeatures to narwhals, add polars support Six pandas-specific spots split into a pandas-native branch and a narwhals-generic branch, each decision benchmarked at 10k-50k rows and 0/1/6 extra columns (not assumed): - missing-columns check, feature_names_in_ extraction: narwhals-on-pandas is 13-22x slower (pure metadata overhead, row-count independent) - kept the pandas fast path established in Pass 1. - coordinate range validation: 6-8.6x slower on narwhals-on-pandas - new narwhals branch added (previously crashed outright on polars), pandas branch untouched. - numpy extraction of the 4 coordinate columns: 5-9x slower via narwhals on pandas; for the narwhals branch itself, .get_column().to_numpy() per column beats .select().to_numpy() by 5-7x on polars, so that's what it uses. - assign new column + optional drop: 1.7-2.9x slower on narwhals-on-pandas, consistent with the bar CyclicalFeatures used to keep branches separate. - column reorder is the one exception - narwhals-on-pandas is actually ~35% *faster* here at 10k rows - but stays a two-branch split per an explicit decision to keep the narwhals-everywhere pattern consistent with Pass 1/2, rather than special-case one operation. Verified end-to-end (not just isolated snippets): pandas output identical to the pre-migration code, polars value-identical to pandas, ~2% pandas speed delta (noise) at 10k rows/1 extra column, both backends' fit() error paths (missing columns, out-of-range coordinates) raise the same messages. Also fixed a pre-existing, unrelated inaccuracy in the class docstring's Examples section - the documented pandas output didn't match what the current (pre-migration) code actually produces. The same drift exists in the user guide's Python-implementation number tables (haversine, euclidean, manhattan, miles) but fixing those throughout is out of scope for this pass - flagged separately. Tests parametrized pandas+polars where a dataframe is involved; pure __init__/tag-validation tests (no dataframe) left as-is, already using match= throughout. * Apply suggestion from @solegalli * Fix stale example output throughout GeoDistanceFeatures user guide Every numeric output table in the "Python implementation" section (haversine, euclidean, manhattan, miles) had drifted from what the code actually produces - confirmed by running each documented example directly and comparing. Some differences are rounding-level, but euclidean trip 4 (1720.18 documented vs 1898.82 actual) and manhattan trip 2 (4684.16 vs 4266.82) are real gaps, and the pipeline predictions example was the furthest off: documented as the training targets exactly ([100, 150, 80, 200]), actual output is [116.67, 120.75, 88.48, 204.10]. Pre-existing, unrelated to the narwhals migration - verified the old, unmigrated code produces the same "actual" numbers used here.

* Migrate MathFeatures to narwhals, add polars support The numpy-reducer fast path (sum/mean/std/var/min/max/prod/median) is unified into a single narwhals-based code path rather than split by backend: benchmarked narwhals-on-pandas vs pandas-native at 10k rows/3 reducers and found only a 1.01x-1.27x difference, well under the bar that kept CyclicalFeatures/GeoDistanceFeatures split (1.7x+). Value extraction for the fast path stays a small pandas/narwhals split though - narwhals' select() doesn't accept integer column names the way pandas' own indexing does, and int-named variables is a real, tested, pandas-only feature (polars requires string columns). The custom-callable/uncommon-aggregation fallback can't be unified at all - narwhals has no row-wise apply. Pandas keeps .agg(func, axis=1); polars uses its native map_rows(), which passes each row as a plain tuple rather than a Series, so callables relying on Series methods (row.max()) need max(row) instead to work on both backends. Documented this explicitly. A non-callable func (e.g. an uncommon pandas aggregation string like "sem") now raises NotImplementedError for polars input rather than failing obscurely, since there's no way to resolve a pandas-specific aggregation name without pandas itself. Also fixed a real bug: the module-level `_PANDAS_LT_3 = int(pd.__version__...)` constant required pandas importable just to import this module at all, breaking every creation transformer for a polars-only install. Replaced with a lazy check using narwhals.dependencies.get_pandas() (returns the already-imported module without importing it), computed only once we already know X is pandas-backed. User guide had three separate pre-existing inaccuracies, unrelated to this migration (confirmed against the old, unmigrated code): a get_feature_names_out example listed 'amin_Age_Marks'/'amax_Age_Marks' for a transformer that was never passed np.min/np.max - it uses plain "min"/"max" strings, which have always produced "min_Age_Marks"/"max_Age_Marks"; and a std column's values matched pre-pandas-3 semantics (ddof=1) for a np.std example that runs under ddof=0 in the installed pandas 3.x, already reflected in this repo's own tests. Fixed both while verifying every table for the new "With polars" section. * Rewrite MathFeatures tests to run the same test against both backends Previously: the original pandas-only tests were left untouched and new, separate polars-only tests were added alongside them for the same behavior. That's not what dataframe-agnostic means - same input in, same values out, checked by the same test. Rewrote every test that touches a dataframe to build it via make_df and parametrize over [pd.DataFrame, pl.DataFrame], replacing pd.testing.assert_frame_equal with a cross-backend assert_df_equal (nw.from_native(...).to_dict() + a per column approx compare, handling None-vs-NaN as the same "missing" value on both sides). The one deliberately un-unified case: an uncommon aggregation string like "sem" succeeds on pandas (routes through its native .agg()) but raises NotImplementedError on polars (no way to resolve an arbitrary pandas-specific string without pandas) - that's a real, documented asymmetry, not an oversight, so it's one parametrized test with an explicit if/else on the expected outcome rather than two separate tests pretending it's the same behavior. Two genuinely pandas-only tests stay pandas-only, with a comment saying why: integer column names (polars requires string columns) and pandas' nullable Int64 dtype (no polars equivalent). Custom-callable fallback tests merged into one using max()/min()/sum() built-ins, which work identically whether the callable receives a pandas Series (pandas' agg(axis=1)) or a plain tuple (polars' map_rows) - no need for Series-specific vs tuple-specific callables in separate tests. Picked up narwhals.dependencies.is_pandas_dataframe(X) is True -> nwd.is_pandas_dataframe(X) and the _pandas_lt_3() -> _pandas_version() rename from upstream changes to the class file. * fix: correct _pandas_version() return type hint from bool to int The function returns int(pandas_version.split(".")[0]) and is used as _pandas_version() < 3, but its signature still said -> bool, failing type checking. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate RelativeFeatures to narwhals+numpy, add polars support Replaces the 8 near-identical _add/_sub/_mul/_div/_truediv/_floordiv/_mod/ _pow pandas methods (~90 lines) with a single numpy-ufunc-driven transform(), per request. Benchmarked at 10k rows, 3 variables, 2 references: the numpy version is not just "minimal loss" but actually faster than the current pandas .div(..., axis=0) approach (552.6us vs 637.0us, 0.87x) - so this is a single unified narwhals+numpy code path, no pandas/polars branch at all (re-verified against the final committed code: 635.9us pandas, down from 800.7us before this change; 262.4us polars, previously unsupported). One correctness fix during implementation: extracting all `variables` as one batched 2D array via select().to_numpy() upcasts every column to a common dtype, silently turning an int column's subtraction result into float and failing 3 existing tests. Fixed by extracting each variable as its own 1D array instead, preserving each column's own dtype promotion independently - matches pandas' per-column .sub()/.div()/etc. semantics, still a single vectorized numpy op per column (no Python-level row loop). Also matched a subtler pandas behavior: floordiv/mod on integer input stay integer-typed, and assigning a float fill_value at zero-denominator positions needs the result array explicitly widened to float first (numpy arrays don't auto-promote dtype on assignment the way pandas' DataFrame column assignment does) - verified this reproduces pandas' output exactly, including for negative numbers (floor-division sign conventions matched NumPy's floor_divide/mod exactly across int/float/negative cases, so no other adjustment was needed there). User guide's example tables verified accurate already (including the Age_pow_Age int64-overflow values, which are genuine hardware overflow behavior, not a doc error - confirmed identical between pandas and polars). Added "With polars" sections to docstring and user guide. * test: merge pandas/polars tests for RelativeFeatures into single parametrized suite Same treatment as the MathFeatures test rewrite: one test per behavior, parametrized over make_df=[pd.DataFrame, pl.DataFrame], checking identical values come out for identical input instead of separate pandas-only and polars-only test functions. Deletes the redundant separately-added polars section, keeps its 3 genuinely-new cases (mixed dtype preservation, float fill_value dtype widening, drop_original column list), and converts the pandas-specific .loc-based zero-fill assertion to a narwhals-based one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate DecisionTreeFeatures to narwhals, add polars support Follows the same pandas-native / narwhals-generic split established for GeoDistanceFeatures (this transformer also reimplements fit()/transform() directly, not via BaseCreation): .columns extraction, column reorder, prediction-column assignment, and drop_original all split by backend, consistent with every other operation in this module that's been benchmarked as a real (not minimal) loss when routed through narwhals on pandas. Confirmed empirically before designing: sklearn's DecisionTreeRegressor/ Classifier and GridSearchCV accept a polars DataFrame directly for both fit() and predict()/predict_proba(), so the actual tree training/inference calls are unchanged - only the surrounding column selection, extraction, and reassembly needed migrating. Fixed a pre-existing bug found while rewriting the exact code path it lived in: single-feature combos with an integer column name (e.g. DecisionTreeFeatures(features_to_combine=1) on a dataframe with columns 0, 1, ...) crashed, since the original `isinstance(features, str)` check missed the int case and fell through to plain X[features] indexing, which returns a 1D Series rather than the 2D input sklearn requires. Widened to isinstance(features, (str, int)); verified the same single-feature narwhals path (get_column().to_frame()) already handles both cleanly. Regression, binary classification, and multiclass classification paths all verified to produce identical predictions between pandas and polars input. return_empty=True + polars remains untestable here too (same nw.col([]) bug in dataframe_checks.py found during CyclicalFeatures, still tabled) - this is the second transformer it blocks. docs/user_guide/creation/DecisionTreeFeatures.rst is large (511 lines) and built around actual cross-validated tree fitting on the real California housing dataset across many sections - re-verified the cheap, deterministic parts (the raw data table) but did not re-run every tree-fitting example given the cost of repeated grid-search CV fits; unlike the other three creation-module docs this pass touched, the rest of this file's numbers are unverified. Added a self-contained "With polars" section using simple synthetic data instead, fully verified. * Apply suggestion from @solegalli * Apply suggestion from @solegalli * docs: clarify is True/is False and cross-backend test conventions in AGENTS.md Two rules made explicit based on recent work: the is True/is False comparison is for flow control only, not variable assignment (per Sole's own simplification of is_pandas = nwd.is_pandas_dataframe(X) is True to just nwd.is_pandas_dataframe(X) in decision_tree_features.py); and dataframe-agnostic transformers get one parametrized test per behavior covering both pandas and polars, never separate per-backend tests. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * feat: add n_jobs for parallel tree training, merge tests to single cross-backend suite Adds an n_jobs parameter to DecisionTreeFeatures that parallelizes tree training across feature combinations via joblib, using threads rather than processes since fitting a decision tree releases the GIL for the bulk of its computation - threads avoid the overhead of copying the whole dataframe to worker processes. Defaults to None (sequential), preserving current behavior. Benchmarked on the committed transformer (5000 rows, 10 vars, features_to_combine=3, 8-point param_grid, 175 trees): 12.17s sequential vs 5.15s at n_jobs=-1, ~2.4x. On small workloads (a handful of feature combinations, the shape of the existing unit tests) parallelizing is a net loss - thread-dispatch overhead outweighs the gain - which is why the default stays sequential. Parallelizing transform()'s predict loop the same way was also benchmarked and found to have no benefit (predict is too cheap per call), so only fit()'s tree training is parallelized. Correctness verified: identical trees/predictions regardless of n_jobs. Also rewrites test_decision_tree_features.py to the single cross-backend-parametrized-test convention used elsewhere in this migration: one test per behavior over make_df=[pd.DataFrame, pl.DataFrame], deleting the separately-added polars-only section that duplicated coverage already present once the original tests are parametrized. Adds n_jobs correctness coverage (parallel vs sequential training gives identical output, both backends). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: avoid pandas fragmentation warning in DecisionTreeFeatures.transform transform() assigned one new tree-prediction column at a time (X[col_name] = preds), which triggers pandas' "DataFrame is highly fragmented" PerformanceWarning once there are enough feature combinations - confirmed with 10 vars/features_to_combine=3 (175 new columns). .assign(**kwargs) does NOT fix this: it inserts columns one at a time internally too, same warning. The actual fix is building all new columns into one DataFrame and joining once (single insertion). Verified: output is byte-identical to the old behavior (pd.testing.assert_frame_equal on a 3000-row/9-var/129-tree case), drop_original still works, and a new regression test confirms the warning is gone (and fails against the old code, confirming it actually catches the regression). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * shorten docstring --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Pure elementwise math (1 / x), so followed the same precedent as ArcsinTransformer (same module, same shape of problem): extract the transform columns to a single numpy array via narwhals' to_numpy(), apply the division once, reassign via nw.new_series + with_columns. Benchmarked narwhals-on-pandas vs the old pandas-native .loc-assignment across 10k-100k rows and 1-10 columns: narwhals-on-pandas was 2-3x *faster* than the old code (0.34x-0.45x of old runtime), narwhals-on- polars faster still - a stronger case for merging into one path than even ArcsinTransformer's parity/faster numbers, so no pandas/polars branch was added. The zero-denominator check (raises ValueError "Some variables contain the value zero...") is preserved exactly in both fit() and transform(), just computed via a numpy comparison on the extracted values instead of a pandas boolean mask. inverse_transform() is unchanged - it still just calls transform(), since 1/(1/x) = x. Rewrote test_reciprocal_transformer.py to one parametrized test per behavior over pandas/polars input (previously pandas-only, relying on the global df_vartypes/df_na fixtures - replaced with local dict data, same pattern as test_arcsin_transformer.py, so both backends build from the same source). Added a verified "With polars" section to the docs; left the pre-existing Ames-housing walkthrough untouched (no network access in this environment to re-verify fetch_openml output, and it wasn't modified by this migration). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Pure elementwise math (arcsin(sqrt(x))), so followed the MathFeatures/ RelativeFeatures precedent: extract the transform columns to a single numpy array via narwhals' to_numpy(), apply np.arcsin(np.sqrt(...)) once, reassign via nw.new_series + with_columns. Benchmarked against the old pandas-native .loc assignment across 10k-100k rows and 1-10 columns: narwhals-on-pandas was consistently at parity or faster (0.4x-1.05x of old runtime, never a regression), so merged into one narwhals-generic path with no pandas/polars branch - same decision MathFeatures/RelativeFeatures landed on for the same shape of problem. fit() and transform() both extract the same numpy array for the range check (values must be in [0, 1]) and reuse it directly for the transform in transform(), avoiding a second backend round-trip. inverse_transform() follows the same pattern. Rewrote test_arcsin_transformer.py to one parametrized test per behavior over pandas/polars input (previously pandas-only, relying on the global df_vartypes/df_na fixtures - replaced with local dict data so both backends can build from the same source). Added a verified "With polars" section to the docs. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

Same elementwise-math shape as ArcsinTransformer: extract the transform columns to one numpy array via narwhals' to_numpy(), apply np.arcsinh((x - loc) / scale) once, reassign via nw.new_series + with_columns. Benchmarked against the old pandas-native .loc assignment across 10k-100k rows and 1-10 columns: narwhals-on-pandas was consistently faster than the old code (0.48x-0.83x of old runtime), so merged into one narwhals-generic path with no backend branch. Found a pre-existing stale docstring while verifying output against the old code: the class docstring's example table (arcsinh of np.random.randn(100) * 1000 with seed 42) printed values that don't match what either the old or new code actually produces (e.g. 7.516076 vs the real 6.901163 for the first row) - confirmed by running the old (pre-migration) code directly, so this predates the migration. Fixed the docstring numbers to the verified real output. The docs/user_guide/transformation/ArcSinhTransformer.rst walkthrough's printed tables were re-run and already matched exactly, so those were left as-is; added a verified "With polars" section to both the docstring and the user guide. Rewrote test_arcsinh.py to parametrize every behavior over pandas and polars input (previously pandas-only). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate BaseImputer to narwhals, add polars support Shared base for the imputation module: _transform() (fit-state checks + column reorder) and transform() (fillna via imputer_dict_) are now dataframe-agnostic, with _get_feature_names_in() reading columns through narwhals on non-pandas input. Benchmarked the fillna step (select + fill from a per-column value dict) at 10k/100k/1M rows x 1/2/10 columns: pandas-native fillna runs ~1.3-1.6x faster than the narwhals-generic fill_null equivalent at the 10k-100k row sizes imputers are normally used at (the gap narrows to ~1.0x only past ~1M rows) - a real, not minimal, loss, so pandas keeps its own fast path (is_pandas = nwd.is_pandas_dataframe(X); if is_pandas is True: ... else narwhals fill_null per column). Also benchmarked a numpy rewrite (to_numpy + np.where per column, mirroring RelativeFeatures) but it did not beat pandas-native and was consistently slower than narwhals fill_null on polars, so it wasn't adopted here - unlike RelativeFeatures' arithmetic, a plain value fill is already close to a no-op for both pandas and narwhals/polars, leaving no room for a numpy win. The pandas<3 fillna-downcasting workaround (option_context + infer_objects) is preserved on the pandas branch but no longer imports pandas at module level - the module is fetched via nw.from_native(X).__native_namespace__() only once X is already confirmed to be a pandas dataframe, so no import is attempted on a polars-only install. Verified: tests/test_imputation full suite unchanged (95 passed, 7 pre-existing failures in test_check_estimator_imputers.py - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract, predates this change). flake8 and mypy clean on the file. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * tidy code * restore infer object * remove reordering of the df * Adapt BaseImputer to narwhals-returning check_X Since #1019, check_X returns a narwhals DataFrame instead of the native frame. BaseImputer._transform rebinds `X = check_X(X)` and returns it, so transform() then sees a narwhals frame: nwd.is_pandas_dataframe(X) is always False (and emits a UserWarning), skipping the pandas-native fillna fast path. check_X is pure validation, so drop the rebinding and keep returning the native X. transform()'s pandas / narwhals split then works as before, with no warning. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* address review feedback on dataframe_checks.py Follow-up to FBruzzesi's review on PR #965: - Clarify docstrings for check_X, check_y, check_X_y in terms of which dataframe libraries are safe to pass in (pandas, polars, PyArrow, modin, cuDF), instead of narwhals-specific "eager" terminology. - Use narwhals' IntoDataFrameT instead of IntoDataFrame for check_X and check_X_y, since both return the same concrete dataframe type they receive. - Fix a null/NaN detection bug: in polars, is_null() does not catch an explicit float("nan") value (only None counts as null), so check_y and _check_contains_na could silently miss NaNs in polars data. Now also check is_nan() for numeric columns/series, keeping numpy for the finite/inf checks since it benchmarks as fast or faster there. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * inline null/nan checks to match FBruzzesi's suggested one-liner Collapses the has_na/has_null/has_nan accumulator variables into a single short-circuiting if-condition, as suggested in review. This also avoids an unnecessary is_nan() call when is_null() already found a null value. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * speed up numeric column detection in _check_contains_na The schema-based list comprehension rebuilt narwhals' full column schema on every single-column access, making it scale roughly quadratically with column count on pandas (benchmarked up to ~500x slower than necessary at 200 columns). Switch to the pandas fast-path / narwhals-selector pattern already used in variable_handling (find_numerical_variables, check_numerical_variables) for the same "which of these columns are numeric" problem. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate MeanImputer/MeanMedianImputer to narwhals, add polars support Fit's mean()/median() computation is split by backend and, on the pandas branch, additionally rewritten to use NumPy directly. Benchmarked (10k-100k rows x 1-10 cols): narwhals-on-pandas vs pandas-native .mean()/.median() showed the same real, not minimal, loss (1.0-3.0x) already documented for BaseImputer's fillna and CategoricalImputer's mode(), so pandas keeps its own fast path. Going further, benchmarked a bulk NumPy nanmean/nanmedian pass (to_numpy() + axis=0 reduction, mirroring MathFeatures' reducer pattern) against pandas-native .mean()/.median() and found NumPy consistently as fast or faster (ratios 0.5-1.05x) - a real win, so the pandas branch now uses NumPy instead of pandas' own methods. For polars, the equivalent NumPy round-trip was benchmarked too and lost to narwhals' native per-column mean()/median() expressions (1.8-3.5x slower for mean; mixed but trending slower for median at scale), so the polars/narwhals branch computes stats with a single narwhals select() of one expression per variable instead - benchmarked against a per-column loop and against select()+to_native().to_dicts() and found select()+rows(named=True) is equal-or-faster and backend-agnostic (no reliance on a polars-only to_dicts() method). All-NaN/all-null columns produce matching values on both backends (verified directly): NumPy's nanmean/nanmedian warn on all-NaN slices where pandas' methods don't, so those warnings are suppressed the same way MathFeatures does. Nullable extension dtypes that would produce object arrays fall back to pandas' native .mean()/.median(), same guard as MathFeatures' dtype.kind check. Found and fixed a real crash: narwhals' select() with zero expressions collapses row count to 0 too, so stats.rows(named=True)[0] would IndexError when return_empty=True yields no numerical variables on polars input. Added an explicit empty-variables guard that skips the backend branch entirely instead of relying on backend-specific zero-column behaviour. Rewrote tests as one parametrized test per behaviour over pd.DataFrame/pl.DataFrame (a self-contained DATA dict replacing the pandas-only df_na fixture, matching the CategoricalImputer migration's pattern), keeping the MeanImputer/MeanMedianImputer deprecation-warning parametrization on top. Verified: tests/test_imputation full suite - 99 passed (up from 95 pre-migration, same tests plus new polars parametrizations), same 7 pre-existing failures in test_check_estimator_imputers.py (sklearn's check_estimator feeds raw numpy arrays, rejected by check_X's dataframe-only contract from the base migration - confirmed identical root cause against the pre-migration baseline via git stash). flake8 and mypy clean. mean_median.py's actual import chain (base_imputer, dataframe_checks, variable_handling) verified pandas-free with pandas blocked, using direct module loading to bypass the sibling not-yet-migrated imputers in imputation/__init__.py. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Every doc example (docstring pandas/polars examples and the new "With polars" section in MeanImputer.rst) re-run against live output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt MeanImputer.fit to narwhals-returning check_X check_X now returns a narwhals DataFrame; fit() passed it to find/check_numerical_variables, the is_pandas mean()/median() fast path and _get_feature_names_in, which then took their non-pandas path (spurious is_pandas_dataframe warning, hard failure on integer column names). check_X is pure validation, so stop rebinding X and keep working with the native input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * unify pandas/polar branches * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate EndTailImputer to narwhals, add polars support fit() now computes the Gaussian/IQR/max end-of-distribution values via a single narwhals aggregation (nw_X.select(...) of per-variable mean/std/ quantile/max expressions), instead of pandas-only .mean()/.std()/.quantile(). transform() already worked cross-backend via the already-migrated BaseImputer. Merge vs split: benchmarked pandas-native vs narwhals-generic (on both pandas and polars) at 10k/50k/100k rows x 1/2/10 columns, with NaNs present (this is an imputer, so skip-NaN semantics matter - mean/std/quantile must skip missing values like pandas' default skipna=True). Results: - gaussian: narwhals-on-pandas is 0.93-1.5x pandas-native's time (parity to a mild loss, narrowing towards 1.0x as rows scale up), and 3-10x *faster* than pandas-native when run on polars. - iqr: narwhals-on-pandas is consistently *faster* than pandas-native (~1.3-2x), on both backends. Nowhere near the "real loss" (1.7x+) split threshold, so one code path (no is_pandas branching) serves both backends - unlike BaseImputer's fillna, which stayed split because it *was* consistently 1.3-1.6x slower via narwhals on pandas. Also benchmarked a numpy rewrite (nanmean/nanstd/nanpercentile per column, mirroring RelativeFeatures' numpy-acceleration pattern) and rejected it: numpy's nan-aware reductions are slow (isnan-mask overhead), and at 10 columns narwhals-on-polars beat numpy-on-polars by ~10x (0.65ms vs 7.2ms at 100k rows x 10 cols) since polars aggregates columns natively/in parallel instead of looping in Python. RelativeFeatures' numpy win doesn't transfer here because that transformer's arithmetic has no NaN-skipping requirement, so plain (non-nan-aware) numpy ops sufficed there. Tests rewritten to one parametrized test per behavior over `@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame])`, replacing the pandas-only test_end_tail_imputer.py. Test data uses `None` for missing values instead of `np.nan`: polars treats a literal np.nan as a real float (not a null), so it would NOT be skipped by mean/std/quantile the way pandas skips NaN by default - `None` becomes a null on both backends and is skipped consistently. Docs: verified the existing house_prices example still runs and produces matching output; added a "With polars" section to both the class docstring and docs/user_guide/imputation/EndTailImputer.rst. No bugs found in the pre-migration code. The 7 pre-existing test_check_estimator_from_sklearn failures in this test module (numpy array input now rejected by check_X, e.g. for MeanImputer) predate this change and are unrelated to EndTailImputer. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt EndTailImputer.fit to narwhals-returning check_X check_X now returns a narwhals DataFrame; fit() passed it to find/check_numerical_variables and _get_feature_names_in, which then took their non-pandas path (spurious is_pandas_dataframe warning, hard failure on integer column names). check_X is pure validation, so stop rebinding X and keep working with the native input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply suggestion from @solegalli * Apply suggestion from @solegalli --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate ArbitraryImputer to narwhals, add polars support fit() never touches dataframe values - it only calls the already- narwhals-migrated check_X/check_numerical_variables/find_numerical_variables and builds imputer_dict_ via a plain dict comprehension over column names - so the only change needed was dropping the module-level `import pandas as pd` and swapping the X/y type hints for narwhals' IntoDataFrame/IntoSeries. transform() is fully inherited from the already-migrated BaseImputer. Benchmarked fit()+transform() (via fit_transform) at 10k/50k/100k rows x 1/2/10 cols, pandas vs polars, and old code vs migrated code on pandas input: fit() takes ~0.06-0.13ms regardless of row count, column count, or backend, both before and after the edit (within noise of each other) - confirming fit() truly does no per-row work. No backend split was needed or added; a single narwhals-agnostic path was kept (it already was one). Numpy: not applicable - fit() has no numeric computation over data at all, only dict/list building over variable names, so there is nothing for numpy to accelerate. While touching fit(), changed `if self.imputer_dict:` to `if self.imputer_dict is not None:` per AGENTS.md's ban on truthy container checks; this also fixes a latent edge case where imputer_dict={} was silently treated as "not provided" and fell through to the variables/arbitrary_number branch. Confirmed pre-existing on origin/narwhals-imputation-base (unrelated to this migration, no test previously covered it). Rewrote tests/test_imputation/test_arbitrary_imputer.py to the cross-backend parametrized style (@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame])) in place, replacing the pandas-only df_na fixture and pd.testing.assert_frame_equal/.isnull() assertions with a plain DATA dict and narwhals-based null/value assertions. The deprecation-warning test for ArbitraryNumberImputer and the arbitrary_number-type-validation test stayed single-backend since they never touch a dataframe. Added a "With polars" section to both the class docstring and docs/user_guide/imputation/ArbitraryImputer.rst, output verified by actually running the transformer. No staleness found in the existing rst (it builds its example from fetch_openml, no literal printed dataframe values to go stale). Verified: tests/test_imputation full suite 98 passed / 7 pre-existing unrelated failures in test_check_estimator_imputers.py (same 7 as on origin/narwhals-imputation-base's baseline of 95 passed - the 3 extra passes here are the new cross-backend parametrization, no regressions). flake8 and mypy clean. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt ArbitraryImputer.fit to narwhals-returning check_X check_X now returns a narwhals DataFrame; fit() passed it to check_numerical_variables / find_numerical_variables / _get_feature_names_in, which then took their non-pandas path (spurious is_pandas_dataframe warning, hard failure on integer column names). check_X is pure validation, so stop rebinding X and keep working with the native input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate MissingIndicator/AddMissingIndicator to narwhals, add polars support Removed the module-level `import pandas as pd`; X/y type hints now use narwhals' IntoDataFrame/IntoSeries. This file overrides transform() rather than extending BaseImputer's, so both the fit() null-count filter and the transform() indicator-column step needed their own narwhals path. Benchmarked both operations at 10k/50k/100k rows x 1/2/10 columns (varying how many columns need indicators), plus a mixed string+numeric-dtype dataset matching MissingIndicator's real "all variable types" usage: - fit()'s `[var for var in variables_ if X[var].isnull().sum() > 0]` loop is ~2-5x faster on pandas than a narwhals-generic `null_count()` call (e.g. 100k rows x 10 cols: 0.41ms loop vs 0.78ms narwhals-on-pandas). A vectorized `X[variables_].isnull().sum()` alternative didn't beat the loop either. narwhals-on-polars was consistently fastest of all (its own native path), so the split is pandas-loop vs narwhals-generic (used for polars/other backends), matching BaseImputer's is_pandas branch pattern. - transform()'s `X[vars].isna().astype("int8").add_suffix("_na")` + `pd.concat` is ~2-5x faster on pandas than narwhals' with_columns equivalent (100k rows x 10 cols: 0.28ms concat vs 1.27ms narwhals-on- pandas), and also beats `assign()`-per-column (0.91ms) and `join()` (0.44ms) alternatives - concat already batches all new columns in one op. So transform() keeps the same pandas fast path, split from a narwhals with_columns path for other backends. Both losses are >1.7x, past the "keep pandas fast path" threshold, so merging into one narwhals-generic path (as BaseImputer's docstring discusses for its own fillna step) was not justified here either. Numpy: converting columns via `.to_numpy()` + `pd.isna()` (the only numpy op that works across MissingIndicator's mixed string/numeric columns, since np.isnan raises on object arrays) was consistently ~1.7-2x slower than pandas-native isnull()/isna() for both fit and transform on mixed dtypes - the extra .to_numpy() copy plus pd.isna() dispatch outweighs any gain, same conclusion as BaseImputer's fillna numpy experiment. Tests: converted tests/test_imputation/test_missing_indicator.py from the pandas-only `df_na` fixture to a plain DATA dict parametrized over `make_df` in [pd.DataFrame, pl.DataFrame], asserting identical variables_ selection and identical `<var>_na` column values on both backends for the same input (one cross-backend PerformanceWarning regression test stays pandas-only, since it targets the pandas fast path specifically). Docs: docs/user_guide/imputation/MissingIndicator.rst has no inline printed output to go stale (it references a screenshot image instead of doctest-style text) - verified its house_prices code example's logic against the migrated transformer with a synthetic stand-in dataset (no network access in this environment) and it behaves identically. Added a verified "With polars" example to the class docstring. Verified: tests/test_imputation/test_missing_indicator.py 29 passed. tests/test_imputation full suite: 107 passed / 7 pre-existing failures in test_check_estimator_imputers.py (confirmed identical failures against a baseline run of origin/narwhals-imputation-base: 95 passed / same 7 failures - sklearn's check_estimator feeds raw numpy arrays, which check_X() has always rejected per the narwhals migration's dataframe-only contract; predates this change). flake8 and mypy clean. Module imports with pandas import blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt MissingIndicator.fit to narwhals-returning check_X check_X now returns a narwhals DataFrame; fit() passed it to find/check_all_variables, the is_pandas null-count fast path and _get_feature_names_in, which then took their non-pandas path (spurious is_pandas_dataframe warning, hard failure on integer column names). check_X is pure validation, so stop rebinding X and keep working with the native input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update missing_indicator.py --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate RandomSampleImputer to narwhals, add polars support fit()/transform() now accept pandas, polars, or any narwhals-supported dataframe. Split (not merged) into a pandas branch and a narwhals branch, mirroring BaseImputer's pattern, because this transformer stores a copy of the training data and draws random values from it - a correctness concern, not just a performance one. RNG/reproducibility decision: pandas' .sample() and polars'/narwhals' .sample() are backed by different random number generators, so they never draw the same values for the same seed even on identical data - this was already true within pure pandas usage across pandas versions in some cases, but is guaranteed different across backends. The contract adopted and documented (class docstring + new "With polars" user guide section) is "same seed, same backend -> same result", not cross-backend value parity. The pandas branch is the pre-migration code verbatim (still X.loc/.sample(random_state=...)/index reassignment, called directly on the pandas object already in hand - no pandas import needed per AGENTS.md), so existing pandas users see bit-identical sampled values after upgrading, seed-for-seed. The narwhals branch is a positional reimplementation for polars and other backends: null positions come from Series.is_null().arg_true(), replacement values come from Series.sample(n, with_replacement=True, seed=...) drawn from the stored training-data pool, and values are written back with Series.scatter() (mirrors the exact usage in narwhals' own scatter() docstring example). For seed="observation", pandas' per-row .loc-based seed lookup (_define_seed, kept pandas-only and untouched) is replaced for the narwhals branch by a single vectorized numpy pass over the seed columns (X.select(seed_vars).to_numpy() + sum/prod per row), since narwhals dataframes have no row-label-based access to loop against. Benchmarked fit()+transform() at 10k/50k/100k rows x 1/2/10 cols: the narwhals-generic (scatter-based) implementation running on pandas input is actually close to or faster than the pandas-native .loc-based implementation at most sizes (0.7-1.3x), so throughput alone would have allowed merging into one code path. The split is driven entirely by the backward-compatibility requirement above (existing users' random_state values must keep drawing the exact same pandas samples they did before this migration) rather than by a performance loss. Rewrote tests/test_imputation/test_random_sample_imputer.py: behavioral tests (general seed, per-observation seed with add/multiply/single variable, categorical dtype preservation, the input-validation error paths that touch a dataframe) are now single tests parametrized over pd.DataFrame/pl.DataFrame, asserting the backend-agnostic invariants that actually hold for this transformer (no nulls remain, every filled value came from the training pool, same seed + same backend reproduces the same result) rather than literal values, since literal sampled values are inherently backend-specific here. _define_seed's own test stays pandas-only (it exercises .loc label access directly, which has no narwhals equivalent). Added one dedicated pandas-only regression test asserting the exact historic literal values are unchanged post-migration, protecting the backward-compatibility guarantee above. Verified: full tests/test_imputation suite goes from 95 passed/7 pre-existing failures (baseline, via git stash) to 102 passed/same 7 pre-existing failures (MeanImputer et al. failing because sklearn's check_estimator feeds raw numpy arrays, which check_X() has rejected since the narwhals migration began - confirmed unrelated to this file). flake8 and mypy clean. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). random_sample.py itself contains no `import pandas` and loads standalone with pandas blocked; the feature_engine.imputation package as a whole still fails to import with pandas blocked, but only because arbitrary_imputer.py (untouched by this change, pre-existing on narwhals-imputation-base) still has a module-level `import pandas as pd` - out of scope here, flagged separately. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt RandomSampleImputer to narwhals-returning check_X - fit(): stop rebinding X = check_X(X); check_X is pure validation and the variable_handling / is_pandas copy paths detect the backend themselves, so keep passing them the native input (avoids the spurious is_pandas_dataframe warning and the integer-column-name failure in the narwhals select path). - _transform_pandas(): copy X before the in-place .loc NaN fills. BaseImputer._transform no longer returns a reordered copy (#1002), so the assignments were mutating the caller's dataframe (and self.X_), which broke the seed-reproducibility tests after rebase. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update RandomSampleImputer.rst * Update random_sample.py * Update random_sample.py * Update random_sample.py * Apply suggestion from @solegalli * Update random_sample.py * Apply suggestion from @solegalli --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate DropMissingData to narwhals, add polars support Split transform()/return_na_data() by backend: benchmarked (10k-100k rows x 1-10 cols) a numpy-backed pandas mask (X[vars].notna().to_numpy().sum (axis=1) / .isnull().to_numpy().any(axis=1)) against both pandas' own axis=1 isnull()/notna().sum() and a narwhals-generic any_horizontal/ sum_horizontal path on pandas input. The numpy mask won consistently - e.g. the threshold check at 100k rows x 10 cols: 1.04ms numpy vs 4.56ms pandas-native vs 2.64ms narwhals-on-pandas (up to ~9x over the naive narwhals path, since pandas' axis=1 reductions are a known-slow case) - so pandas keeps this dedicated fast path; polars/other backends use narwhals' any_horizontal/sum_horizontal, which is fastest of all on native polars input. fit()'s missing_only variable-detection loop keeps the same pandas-loop/narwhals-null_count() split already established by MissingIndicator's migration. Found and fixed a real, pre-existing complementary-logic bug in return_na_data(): its threshold branch computed `isnull_frac >= threshold` as "dropped", when the true complement of transform()'s dropna (kept if non-null count >= n_vars*threshold) is `non_null_count < n_vars*threshold`. These aren't algebraic complements except by coincidence at threshold=0.5, and even there the boundary row was double- counted: kept by transform() AND returned by return_na_data(). Verified against the old code (predates this migration, present on origin/main): with threshold=0.5, transform() kept row 2 (2/4 non-null, meets the threshold) while return_na_data() also returned it; at threshold=1 the bug was worse - return_na_data() silently dropped 2 of 3 truly-missing rows from its output entirely. Fixed by deriving transform() and return_na_data() from one "keep" mask/expression, negated for the drop side (_select_rows(X, keep)), so the two outputs are an exact partition by construction - added test_transform_and_return_na_data_partition_input to verify this explicitly across every threshold value, plus corrected test_return_na_data_method's threshold=0.5 expectation, which had baked the bug's wrong output into the assertion. Also fixed find_all_variables(X, self.return_empty) - a positional-arg bug (return_empty was landing in the exclude_datetime slot) present on origin/main; the same bug pattern is repeated in random_sample.py, categorical.py and missing_indicator.py but those are out of scope here. Guarded the narwhals row-filter path against variables_ == [] (a real case: missing_only=True on a clean training set finds nothing to check) since narwhals' any_horizontal/sum_horizontal raise on an empty expression list, unlike pandas' dropna(subset=[]) which silently keeps every row - added a test for it. Fixed a latent bug in TransformXyMixin.transform_x_y's narwhals branch: it injects a temporary row-index column before calling self.transform(), but BaseImputer._transform() validates X's column count/names against feature_names_in_/n_features_in_ first and rejected the extra column - this combination (TransformXyMixin + a strict-validating transform()) was never exercised before since no prior narwhals migration combined both on a row-dropping transformer. Fixed by widening feature_names_in_/ n_features_in_ just for that call and restoring them after. Rewrote tests as one parametrized test per behavior over pd.DataFrame/pl.DataFrame with a shared DATA dict, replacing pandas .index-based assertions (meaningless for polars) with value-based checks via a backend-agnostic _cols() helper. Verified: tests/test_imputation full suite unchanged except for the new cases (106 passed, same 7 pre-existing test_check_estimator_imputers.py failures that predate this change). flake8 clean; mypy clean on this file, and introduces zero new errors in mixins.py (8 pre-existing attr-defined errors, inherent to the mixin pattern, unchanged). Module's own import chain verified pandas-free with pandas blocked, run successfully against polars input. Every doc example in DropMissingData.rst re-verified against actual output; added a "With polars" section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix: TransformXyMixin.transform_x_y crashes when feature_names_in_ isn't set widened self.feature_names_in_/n_features_in_ unconditionally to smuggle a row-index marker column through transform()'s column-count validation. DropMissingData's own tests exercise transform_x_y() before fit() has run in some paths, where feature_names_in_ doesn't exist yet, raising AttributeError. Guard with hasattr() so the widening only happens when there's something to widen - identical behavior for every caller that already had feature_names_in_ set (OutlierTrimmer, forecasting base), verified via the existing mixin/imputation test suites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt DropMissingData.fit to narwhals-returning check_X check_X now returns a narwhals DataFrame; fit() passed it to find/check_all_variables, the is_pandas null-count fast path and _get_feature_names_in, which then took their non-pandas path (spurious is_pandas_dataframe warning, hard failure on integer column names). check_X is pure validation, so stop rebinding X and keep working with the native input. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Update drop_missing_data.py * Update mixins.py --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate CategoricalImputer to narwhals, add polars support Fit's mode() computation is split by backend: benchmarked (10k-100k rows x 1-10 cols) narwhals-on-pandas against pandas-native mode() and found a real, not minimal, 1.4-1.7x loss, consistent with BaseImputer's earlier split decision for fillna - so pandas keeps calling its own .mode(). Also benchmarked pandas' per-column mode() loop against its original batch X[variables_].mode() call and found no advantage to the batch form (ratios 0.77-0.97x), so both backends now share one per-variable loop structure, just with a different mode() call inside - simpler than the original single-var/multi-var split without losing performance. Found and fixed a real mode-tie bug: polars' native mode() does not drop nulls first (pandas' does, by default), so a column whose nulls outnumber any single category would make null "the mode" on polars instead of raising the multi-mode ValueError pandas raises. Fixed by calling drop_nulls() before mode(keep="all") on the narwhals branch; verified both backends now raise on the same tied columns and agree on the same single mode when there's no tie. Investigated pandas' category dtype vs polars' Categorical/Enum, since they aren't equivalent APIs. polars' Categorical auto-widens on fill_null (no add_categories-equivalent step needed, unlike pandas' category dtype which still needs the existing add_categories call or it raises TypeError). polars' Enum has a genuinely fixed category set: filling it with a value outside that set silently writes null instead of erroring - confirmed this is real, not hypothetical, so added an explicit check that raises a clear ValueError instead of corrupting data silently. Also confirmed polars never silently upcasts a string-typed column back to numeric the way pandas' fillna+ infer_objects does, so return_object is a documented no-op there. Rewrote tests as one parametrized test per behavior over pd.DataFrame/pl.DataFrame, using a shared DATA dict instead of the pandas-only df_na fixture. Kept pandas' object-dtype-for-numeric-vars tests and the category-dtype tests single-backend (genuinely pandas-specific dtype quirks with no polars equivalent), and added new single-backend polars tests for Categorical widening and the Enum fixed-category error path. Verified: tests/test_imputation full suite unchanged except for the new cases (105 passed, same 7 pre-existing failures in test_check_estimator_imputers.py that predate this change, per BaseImputer's migration). flake8 and mypy clean. Module's own import chain (dataframe_checks, variable_handling, base_imputer) verified pandas-free with pandas blocked - the whole feature_engine.imputation package still imports pandas only because sibling imputers are not yet migrated. Every doc example re-run against the live house_prices dataset and a pandas dtype-name string fixed to match pandas 3's actual output; added a "With polars" section with the Enum caveat. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt CategoricalImputer to narwhals-returning check_X - fit(): stop rebinding X = check_X(X); check_X is pure validation and the variable_handling / mode() paths detect the backend themselves, so keep passing them the native input (avoids the spurious is_pandas_dataframe warning and the integer-column-name failure). - transform(): copy X before widening pandas category columns in place. BaseImputer._transform no longer returns a reordered copy (#1002), so the in-place cat.add_categories() reassignment was mutating the caller's dataframe (broke test_variables_cast_as_category_missing after rebase). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * CategoricalImputer: impute with first mode instead of erroring on multi-mode variables CategoricalImputer(imputation_method="frequent") raised a ValueError at fit() whenever a variable had more than one mode, forcing the user to break ties themselves. It now resolves the tie automatically: it sorts the modes and imputes with the smallest one, deterministically and identically for pandas and polars. - fit(): the "frequent" branch is now one unified narwhals loop (no is_pandas split); it sorts drop_nulls().mode(keep="all") and takes [0]. multi_mode_vars, the len(mode_vals) > 1 checks and the raise are gone. Single-mode behaviour is unchanged. - tests: replace test_error_when_variable_contains_multiple_modes with test_uses_smallest_mode_when_variable_has_multiple_modes (both backends). CategoricalImputer has no post-variable-selection fit failure anymore, so drop its branch in test_raises_non_fitted_error_when_error_during_fit. - docs: rewrite the "Categorical features with 2 modes" user-guide section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Apply suggestion from @solegalli --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…1024) RandomSampleImputer._transform_narwhals had two bugs on the polars/narwhals path: - it wrote each variable's imputation into a fresh copy of the input (`nw_X = X.with_columns(...)`), so only the last imputed variable survived and every earlier variable kept its nulls; it also returned a narwhals frame instead of a native one. - the "observation" seed branch read `nw_X` before it was ever assigned, raising UnboundLocalError. Both branches now accumulate into `X` and the method returns `X.to_native()`, matching the pandas branch. TransformXyMixin.transform_x_y still assumed check_X_y returned a native dataframe. Since check_X_y now returns a narwhals frame, `is_pandas_dataframe` was always False, so pandas input took the positional-backend path and the `__feature_engine_row_index__` tag column made transform() fail the column-count check. The mixin now branches on `implementation.is_pandas()`, and `_check_X_matches_training_df` ignores the reserved tag column (its name is now a shared constant in dataframe_checks). Fixes the polars cases of test_random_sample_imputer.py and both cases of test_drop_missing_data.py::test_transform_x_y. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate DatetimeOrdinal to narwhals+numpy, add polars support Replaces the pandas-only row-by-row implementation (pd.to_datetime + .apply(lambda x: x.toordinal())) with a vectorized one: string/categorical variables are parsed to a real Date/Datetime dtype via narwhals' str.to_datetime() (shared across backends), then the ordinal itself is computed as (days-since-epoch + epoch_ordinal), verified to match datetime.date.toordinal() exactly, including pre-epoch and year-1 dates. Benchmarked the ordinal math at 10k/50k/100k rows x 1/2/10 columns: - old apply()-based pandas path vs a narwhals-generic dt.timestamp() path: 27x-234x faster, growing with row count (the old code was O(rows) in Python, this is fully vectorized). - narwhals dt.timestamp() vs a numpy datetime64[D] fast path on pandas: numpy wins by 3.4x-12x (bigger at low row counts, where per-call narwhals/polars-engine overhead dominates). This is a real, not minimal, gain, so pandas gets its own numpy branch (_transform_pandas: to_numpy().astype("datetime64[D]").astype("int64")), while polars stays on the narwhals dt.timestamp() path (_transform_narwhals), which was already fast enough (0.09-1.3ms) that a numpy round-trip through Arrow wouldn't pay for itself. start_date parsing in __init__ no longer imports pandas (pd.to_datetime -> dateutil.parser.parse, already a core dependency and already used elsewhere in feature_engine/variable_handling); datetime.date/datetime objects use their own .toordinal() directly, both stdlib. Missing-value representation is now backend-native instead of forcing object-dtype + pd.NA: NaN/float64 for pandas, null/Int64 for polars - tests and docs normalize/document this instead of asserting one fixed dtype. Bug found (pre-existing, not from this migration - verified against narwhals-migration base with git stash): the two "days from start_date" numbers in docs/user_guide/datetime/DatetimeOrdinal.rst were stale (-4343 and 3956 vs the actual -4342 and 3957); fixed against verified output. Also documents a real narwhals/polars limitation found while writing the polars doc example: polars' str.to_datetime() (unlike pandas' dateutil-backed pd.to_datetime) can't guess ambiguous or loosely-formatted date strings ("May-1989", "06/21/2012") without an explicit format - the polars example uses ISO-8601 strings instead, with a note explaining the difference. Also found and fixed a latent bug this migration's own cross-backend tests exposed in the *already-migrated* shared `_check_contains_na` (feature_engine/dataframe_checks.py): nw.col([]) raises on the polars backend, which crashed fit() for return_empty=True + missing_values= "raise" + polars input (no variables found). Worked around locally by skipping the na-check when variables_ is empty (nothing to check anyway); flagged the shared function itself for a proper fix since other transformers hitting the same combination will have the same problem (spawned as a separate follow-up task). Tests rewritten as one cross-backend parametrized test per behavior (`@pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame])`), 32 passed. Full tests/test_datetime suite: 152 passed, 2 pre-existing failures in test_datetime_features.py (DatetimeFeatures, unmigrated, unrelated file) confirmed present on narwhals-migration base too. flake8 and mypy clean. Module verified to import and run end-to-end on polars with pandas import blocked. sphinx -W build has the same single pre-existing linkcode_resolve warning as the unmigrated base, nothing new. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Address review: init params, drop reorder, fewer narwhals round-trips - __init__ stores raw self.start_date (user param) instead of deriving self.start_date_ at construction; start_date is now parsed into self.start_date_ordinal_ in fit(). Restores get_params()/clone(). - Inline nwd.is_pandas_dataframe(X) in the if statements. - Remove the "reorder variables to match train set" step in transform(); columns are selected by name, so it wasn't needed. - transform() now converts to narwhals once and back to native once in the per-backend helper, with no round-trips in between. - Tests updated: invalid start_date now raises from fit(); stale known-bug comment in test_return_empty corrected. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update datetime_ordinal.py * Sync docstrings with fit()-time start_date parsing - start_date param: document that datetime.date is also accepted. - fit() docstring: note it parses start_date and can raise ValueError (the raise moved here from __init__). - Doctests: `_ = dtf.fit(X)` since repr(dtf) now works and would otherwise echo in the >>> fit(X) line. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update datetime_ordinal.py * Update datetime_ordinal.py * Apply suggestion from @solegalli --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

@solegalli

* Migrate DatetimeFeatures to narwhals, add polars support DatetimeFeatures does heavy .dt-accessor work, and narwhals' dt namespace is missing 10 of the 20 supported features outright (quarter, week, month_start/end, quarter_start/end, year_start/end, leap_year, days_in_month - no isocalendar(), is_month_start, days_in_month, etc.). All 20 are reproducible from narwhals primitives (month()/day()/weekday()/ offset_by()/truncate()/to_string("%V")) and verified byte-for-byte against pandas' native FEATURES_FUNCTIONS across 8000 random dates x both backends, including nulls, leap days, and year/quarter/month boundaries. Benchmarked per-feature at 100k rows: running the new narwhals formulas through narwhals-on-*pandas* is fine for month/year/day/hour/minute/second/ day_of_year/day_of_week/quarter/semester/weekend/month_start (~1.0-1.3x, minimal loss) but a real loss for week (53x - to_string() round-trips through string parsing), and month_end/quarter_start/quarter_end/ year_start/year_end/leap_year/days_in_month (2.0x-3.3x - multi-condition boolean chains and offset_by/truncate are slow on the narwhals-pandas backend). Rather than split per-feature, the transformer splits per backend at the top of fit()/transform() (matching BaseImputer/ DecisionTreeFeatures): the pandas branch is the original, untested-for- regression pandas-native code, unchanged; the new FEATURES_FUNCTIONS_NARWHALS dict in _datetime_constants.py only runs for non-pandas input, where it's strictly faster than the pandas path ever was. `variables="index"` is pandas-only (narwhals dataframes have no index concept) and now raises a clear TypeError on other backends instead of silently doing the wrong thing. String-to-datetime parsing keeps `pandas.to_datetime` (dayfirst/yearfirst/utc/mixed-format) on the pandas branch via the native-namespace trick (no static pandas import); the narwhals branch uses `Series.str.to_datetime(format=...)`, which has no day/year-first heuristic, so ambiguous non-ISO strings need an explicit `format` there (documented in the docstring, .rst, and a dedicated test). Found and fixed a pre-existing bug on narwhals-migration: the variables="index" branch called `_is_categorical_and_is_datetime()` with a raw pandas Index, but that helper's signature was already changed (by the variable_handling narwhals refactor) to expect a narwhals Series, breaking NaN-in-index detection for 2 tests. Confirmed pre-existing via `git stash` against this same branch tip before starting this migration. Rewrote the cross-backend-relevant tests in test_datetime_features.py to single parametrized tests over pd.DataFrame/pl.DataFrame (ISO-8601 dates, portable across backends); left the pandas-only dateutil-format-inference, timezone, categorical-dtype, and "index" tests as pandas-only, since that behavior is genuinely pandas-specific. Added tests for the new variables="index" TypeError on non-pandas input and the ambiguous-format ComputeError on non-pandas string parsing. Verified: tests/test_datetime full suite 155 passed (up from 140 on the pre-migration baseline, which had 2 pre-existing failures from the bug above - both now fixed). flake8 and mypy clean. Module imports and a full polars fit/transform succeed with pandas import blocked at the interpreter level. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning; had to use `.. code:: text` instead of `.. code:: python` for the polars table output in the new "With polars" doc section, since Pygments' python lexer chokes on the box-drawing characters - matching the existing convention in MathFeatures.rst etc). All existing pandas doc examples in DatetimeFeatures.rst spot-checked against actual current output before and after - byte-identical, since the pandas code path is untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Apply suggestion from @solegalli * Apply suggestion from @solegalli * Update datetime.py * Update datetime.py * Update datetime.py * Fix DatetimeFeatures for narwhals-returning check_X Rebased onto narwhals-migration, where check_X returns a narwhals frame and no longer copies its input. Adapt DatetimeFeatures accordingly: - fit(): drop the leftover `is_pandas` references (NameError); take feature_names_in_ / n_features_in_ from the narwhals frame check_X built. - fit(): the variables="index" guard was inverted - it rejected pandas input instead of non-pandas. Flip it. - transform(): reuse check_X's frame for __native_namespace__ instead of re-wrapping; drop the redundant from_native in the non-pandas branch. - transform(): the pandas and index paths mutated the caller's dataframe in place (fine when check_X copied, not any more). Build the new columns and concat them into a fresh frame; drop_original no longer uses inplace. Docs: describe pandas/polars support without naming the internal dataframe library. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

) * Migrate DatetimeSubtraction to narwhals+numpy, add polars support Ports DatetimeSubtraction (extends the already-migrated BaseCreation) to narwhals, adding native polars support and removing the pandas-only computation path, following the RelativeFeatures precedent. Benchmarked pandas-native vs narwhals+numpy on pandas vs narwhals+numpy on polars at 10k/50k/100k rows x 1/2/10 datetime-pair combinations. Extracting each unique variable to a numpy datetime64 array once, then subtracting and dividing with plain numpy ops, is a clear MERGE win - no is_pandas branch needed for the arithmetic itself: rows=100000 pairs=10 | pandas_native=5.934ms | narwhals+numpy(pandas)= 3.011ms (0.51x) | narwhals+numpy(polars)=1.282ms (0.22x) End-to-end (including datetime parsing), the new pandas path is also consistently faster than the old pandas-only implementation (0.55x-0.96x across the grid), and polars is 4-20x faster than pandas at scale once parsing cost is amortized over more rows. "Y"/"M" output units are non-linear numpy timedelta units, so both the diff and the unit divisor are cast to timedelta64[ns] before dividing (numpy can't otherwise find a common divisor) - this mirrors what pandas does internally for Timedelta / Timedelta and was verified against all 14 supported output_unit values. Datetime parsing (dayfirst/yearfirst/utc/format) is inherently backend-specific, so it keeps a real is_pandas branch: the pandas path calls pandas.to_datetime via nw.get_native_namespace() (no "import pandas") to preserve exact prior behaviour; the non-pandas path uses narwhals' str.to_datetime first, then falls back to per-value dateutil parsing (honouring dayfirst/yearfirst/utc) for ambiguous formats narwhals can't infer - the same flexible, cross-backend date guessing check_datetime_variables/find_datetime_variables already promise, so a column that passes fit() can always be parsed in transform() on any backend. No bugs found in DatetimeSubtraction itself. Two pre-existing failures in test_datetime_features.py (DatetimeFeatures index/NaN handling) and 68 repo-wide pre-existing failures elsewhere are unchanged before/after this change (confirmed via git stash) and belong to other, not-yet-migrated modules. Rewrote tests/test_datetime/test_datetime_subtraction.py to parametrize every dataframe-dependent test over pandas and polars via make_df (122 tests, up from 83), and added a "With polars" section to DatetimeSubtraction.rst, verifying every doc example (old and new) against actual output. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update datetime_subtraction.py * Finish removing the is_pandas indicator from DatetimeSubtraction The previous commit half-removed it, leaving fit()/transform() broken: - fit() had a bare `nw_X.columns` expression that never assigned self.feature_names_in_. - transform() and _to_datetime() still referenced an undefined `is_pandas`. fit() now assigns self.feature_names_in_ = nw_X.columns; transform() calls _to_datetime(nw_X) with no flag; _to_datetime() derives the backend locally with nw_X.implementation.is_pandas() (the idiom used in dataframe_checks and the base mixin), keeping the pandas to_datetime fast path. Dropped the now unused narwhals.dependencies import. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Type _to_datetime / _sub dict keys as Union[str, int] Column names in feature-engine can be ints (find_datetime_variables / check_datetime_variables return List[Union[str, int]]), so the datetime array dict is keyed by str | int, not str. Fixes 3 mypy errors on `mypy feature_engine`. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

#999) * Migrate CategoricalMethodsMixin (encoding base) to narwhals, add polars support Shared base for all 8 encoders. _get_feature_names_in() and _check_transform_input_and_state() follow the same is_pandas-gated column-reorder pattern as BaseImputer/DecisionTreeFeatures. _check_or_select_variables() needed no change: the variable_handling helpers it calls are already fully narwhals-generic. The hot path is _encode()/inverse_transform(), a per-column dict-based map applied on every transform() call across every encoder. Benchmarked pandas-native .map(dict) vs narwhals Series.replace_strict(dict, default=...) at 10k/50k/100k rows x 1/2/10 columns x 5/50 categories (warmed up first to remove first-call JIT/import overhead): narwhals-on-pandas lands at ~1.06x-1.2x of pandas-native at realistic sizes (50k-100k rows), i.e. minimal loss - merged into a single narwhals path per the established decision rule, no pandas fast-path split. narwhals-on- polars is consistently ~4-5x faster than pandas-native at 100k rows. replace_strict() also *simplifies* the old logic: pandas' plain .map() leaves category-dtype columns as category dtype after mapping, which the old code corrected with a manual "cast to int if all-int else float" step. Verified narwhals' replace_strict resolves straight to a plain numeric dtype on both a pandas category column and a polars Categorical column, so that dtype fixup is dead code once replace_strict replaces .map() - dropped it entirely rather than porting it. Used Series.get_column().replace_strict() (not nw.col(), which only accepts string names) throughout, same as DecisionTreeFeatures' precedent for pandas integer column names - nw.col(feature) blew up on int-named columns (caught by the existing test_column_names_are_numbers test, which polars can't cover since it has no integer-column-name concept). _check_nan_values_after_transformation() rewritten off pandas' .isnull().sum().sum()/.columns[...] chain onto per-column Series.null_count(), for the same int-column-name reason. Verified: tests/test_encoding full suite unchanged (17 pre-existing failures - numpy-array-input rejection per the narwhals check_X() contract, plus 3 MeanEncoder inverse_transform failures caused by a pre-existing bug in mean_encoding.py's still-unmigrated fit() passing a numpy y into y.groupby(); reproduced identically against the unmodified base_encoder.py to confirm neither predates nor is introduced by this change - 326 passed both before and after, same failing test IDs). flake8 and mypy clean on the file. Module imports with pandas blocked (loaded standalone, since sibling encoder files in this package are not yet migrated and still import pandas at their own module level). sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Manually verified CountEncoder end-to-end on polars input (fit still pandas-only until its own migration, transform/inverse_transform now backend-agnostic via this mixin) produces identical values to the pandas path, including a pre-existing quirk where count-encoding inverse_transform is ambiguous for categories that share a count (confirmed identical, not a regression, on the old code too). _helper_functions.py checked: pure-python parameter validation, no dataframe interaction, no pandas import - left untouched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update base_encoder.py * Fix CategoricalMethodsMixin for narwhals-returning check_X After the rebase onto narwhals-migration, check_X / check_X_y return a narwhals frame. The previous "Update base_encoder.py" left the method bodies referencing a local nw_X that no longer exists. - _encode / _check_nan_values_after_transformation: use the narwhals frame that is actually passed in (was NameError on nw_X). - _check_nan_values_after_transformation now assumes a narwhals frame (its only caller, _encode, hands it one); no nw.from_native round-trip. - _get_feature_names_in: single branch-free `list(X.columns)` (normalises a narwhals column list and a pandas Index alike). - _check_transform_input_and_state keeps the native X for the column-count check and returns the narwhals frame. - Drop now-unused narwhals imports; refresh docstrings. - test_categorical_method_mixin: pass a narwhals frame to the two direct _check_nan_values_after_transformation calls. The encoder subclasses still run pandas-only fit()/transform() code and are adapted in their own migration PRs. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update base_encoder.py * test(encoding): assert error/warning text via pytest.raises/warns match= Replace the `as record: ... assert str(record.value) == msg` / `record[0].message.args[0] == msg` pattern in the CategoricalMethodsMixin tests with `match=re.escape(msg)` on pytest.raises / pytest.warns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…pport (#1025) * Migrate CountEncoder/CountFrequencyEncoder to narwhals, add polars support transform()/inverse_transform() came pre-migrated via base_encoder.py's CategoricalMethodsMixin (already narwhals-generic and benchmarked). The remaining work was fit(), which builds encoder_dict_ from pandas' .value_counts().to_dict() per variable. Replaced with narwhals Series.drop_nulls().value_counts(sort=True, normalize=...), converted to a dict via to_list() on both columns. Two behavioral gaps found and closed against the old pandas code: - narwhals' value_counts() has no dropna param and counts NaN as a category by default, unlike pandas' value_counts(dropna=True) default. Without drop_nulls() first, a NaN category picked up a real count instead of staying an "unseen" category under missing_values= "ignore" - would have been a silent behavior change. Verified against the old code (pandas value_counts() drops NaN by default) that this wasn't already the case. - sort=True (matching pandas' own value_counts() default, descending by count) rather than narwhals' own default of sort=False, so encoder_dict_ keeps the same category order as before - verified via the class docstring's doctest and the user guide's printed encoder_dict_ output, both unchanged byte-for-byte. Benchmarked pandas-native vs narwhals-on-pandas vs narwhals-on-polars at 10k-1M rows x 1/2/10 columns x 5/50 categories, warmed up first. Also compared value_counts() against group_by().agg(nw.len()) as an alternative - value_counts() was consistently faster (up to ~2x), so kept the simpler API. Decision: merge into one narwhals path, no pandas/polars branch. The numbers are noisier than the base's transform() benchmark: at 50k-100k rows (the "realistic size" range used for that decision) narwhals-on-pandas ran 1.3x-2.0x of pandas-native, higher than the 1.06x-1.2x band that justified merging the encode() hot path. But the ratio is dominated by fixed per-call overhead, not genuine scaling cost - it converges to 1.06x-1.2x by 200k-1M rows, and the absolute cost stays trivial throughout (under 2ms extra at 50k rows, under 6ms extra at 1M rows x 10 columns). Unlike encode(), fit() runs once per model lifecycle, not once per transform() call, so that one-time cost doesn't compound. narwhals-on- polars was consistently at or faster than pandas-native (0.8x-1.1x). Given AGENTS.md's stated priority (readability first, add a fast path only when a slow default isn't free), a pandas/polars split wasn't justified here. Rewrote test_count_frequency_encoder.py to one parametrized test per behavior over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]), replacing the module-level pandas-only fixtures (df_enc, df_enc_rare, df_enc_na, df_vartypes) with local dict constants both backends can build from, per the ArcsinTransformer/ PowerTransformer precedent. Kept test_column_names_are_numbers and test_variables_cast_as_category pandas-only, since integer column names and pandas category dtype are backend-specific per AGENTS.md. Switched exact-message pytest.raises(match=...) checks to match= re.escape(msg): one of the existing error strings contains literal parentheses ("feature(s)"), which pytest.raises interprets as a regex capture group and silently fails to match without escaping - caught this while converting the tests, not a pre-existing bug in the old code (the old tests used exact string equality, which doesn't have this problem). Verified: tests/test_encoding full suite - 350 passed, 17 pre-existing failures, identical failing test IDs to the pre-migration baseline (10 in test_check_estimator_encoders.py's numpy-array-input rejection checks, 3 MeanEncoder inverse_transform failures from mean_encoding.py's still-unmigrated fit()); confirmed by running the same suite against the unmodified code via git stash. flake8 and mypy clean. Module imports with pandas blocked. sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning, confirmed identical against the unmodified code too). Added a verified "With polars" section to both the class docstring and the CountEncoder.rst user guide page. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt CountEncoder to narwhals-returning check_X check_X now returns a narwhals frame, so bind it to nw_X and keep the original native X for _check_or_select_variables / _check_na / _get_feature_names_in (the variable_handling and _check_contains_na helpers still branch on nwd.is_pandas_dataframe and expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the now-redundant nw.from_native(X) round-trip and its unused `import narwhals as nw`; the fit() loop reuses nw_X from check_X. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Assert CountEncoder test results natively, check output backend The tests converted every result to pandas via _to_pandas() and compared with assert_frame_equal(check_dtype=False). That hid the output backend (a polars input returning pandas would pass) and required pyarrow, which is not a feature_engine dependency, so all polars cases failed in CI. Follow the imputation tests instead: read results back with to_dict(as_series=False) through a _cols() helper (NaN normalised to None), count nulls per column with narwhals, and assert the output is an instance of the input dataframe type. The pandas-only tests (integer column names, category dtype) keep assert_frame_equal. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Shorten value_counts comment in CountEncoder.fit() Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Rename CountEncoder test helper _cols to _to_dict The helper returns the whole dataframe as a {column: values} dict, not a selection of columns, so _to_dict describes it better. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Simplify unseen-category warning assertion in CountEncoder tests Use pytest.warns(match=...) like the other tests in the file instead of inspecting the recorded warnings by hand. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…sts (#1045) * Add shared backend test fixtures and helpers, use them in CountEncoder tests Tests for narwhals-migrated transformers each defined their own way of building pandas/polars inputs and reading results back (_to_backend, _assert_values, _cols, _to_dict, _to_pandas, ...), which makes the test suite hard to maintain. Standardise on one structure: - tests/conftest.py: `make_df` fixture parametrized over pd.DataFrame and pl.DataFrame (ids "pandas"/"polars"). Tests that request it run once per backend; pandas-only tests simply don't request it. - tests/backend_helpers.py: `to_dict` (contents as {column: values}, NaN normalised to None), `null_count`, and `make_series` (target on the same backend as X). - tests/test_encoding/conftest.py: data shared by the encoder tests, as fixtures returning plain dicts (missing values written as None) that tests build with make_df(data). CountEncoder tests are migrated to this structure: they check the output is of the input backend with isinstance(X, make_df) and compare contents with to_dict(). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Add data_enc_big_na and data_enc_top encoder test fixtures Shared by the OneHotEncoder, RareLabelEncoder and StringSimilarityEncoder tests, which each defined their own copy of this data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * rename functions * rename function * update tests --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

…gData output type (#1046) * Use shared backend test fixtures and helpers in imputation tests Move the data duplicated across the imputer test files into tests/test_imputation/conftest.py (data_na, and data_na_dob for the two transformers that need a never-null datetime column), and replace the file-local helpers (_cols, _null_count, _values, _same_values, assert_df_equal, _missing_count, _to_list, _make_series) with the shared ones: make_df fixture, to_dict, null_count and make_series. Every transform output is now also checked to be of the input backend with isinstance(X, make_df). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix DropMissingData returning a narwhals frame and skipping its pandas path _select_rows() receives the narwhals frame returned by check_X, but still dispatched with nwd.is_pandas_dataframe(X), which is never True for a narwhals frame (narwhals warns about it). As a result: - when no variable was selected (e.g. missing_only=True on a clean training set), transform() returned the narwhals frame itself instead of a pandas/polars dataframe; - the benchmarked pandas fast path never ran, so pandas input silently went through the slower narwhals expression path. Branch on X.implementation.is_pandas() instead, and always return the native frame. Caught by the new isinstance(X, make_df) output checks. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict in imputation tests The shared helper was renamed from to_dict to frame_to_dict in #1045. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * update conftest * update arbitrary imputer tests * update categorical imputer * reorder tests in drop missing data * refactor end tail tests * refactor mean median imputer tests * refactor random sampler and improve seeding procedure --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate MeanEncoder.fit() to narwhals, add polars support fit() computes, per variable, the mean of y per category (and, with smoothing="auto", the target variance per category), blended with the overall target mean via a weight that increases with category count. transform() and inverse_transform() already came dataframe-agnostic for free from CategoricalMethodsMixin (base_encoder.py, merged separately). Benchmarked a pure-narwhals fit() (group_by/agg for count+mean(+var)) against pandas-native (value_counts + groupby) at 10k-100k rows x 1-10 cols x 5-50 categories: narwhals-on-pandas ran ~1.5x-2.9x slower, worst at the most common shape (1-2 columns, 50k-100k rows), crossing the ~1.7x real-loss threshold; narwhals-on-polars was competitive to faster than pandas-native throughout. Per the benchmark-driven merge-vs-split rule, and matching what the OrdinalEncoder sibling migration found for the same y-groupby-by-category shape of fit(), this splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps a close variant of its original value_counts/groupby code, while polars (and other narwhals backends) goes through group_by()/agg(). Bug fixed (pre-existing, confirmed against the unmodified file): the old fit() always called `y.groupby(X[var])`, which raises AttributeError whenever y is a numpy array rather than a Series - e.g. list/array-like y input, which sklearn's check_X_y machinery converts to numpy. This is the exact same bug the OrdinalEncoder sibling found and fixed in its own fit(). Confirmed failing against the unmodified file (tests/test_encoding/test_mean_encoder.py:: test_inverse_transform_when_no_unseen, ::test_inverse_transform_when_ ignore_unseen, ::test_inverse_transform_when_encode_unseen, plus test_check_estimator_encoders.py::test_encoders_when_x_pandas_y_numpy [encoder1] for MeanEncoder) and now passing. Fixed on the pandas branch by pairing X[var] with y via `.assign()` when y isn't a Series (aligns a numpy y positionally, matching how `y.groupby(X[var])` aligned a Series y by index), and on the narwhals branch via `nw.new_series` for a numpy y. Unlike OrdinalEncoder, no cross-backend tie-break fix was needed: MeanEncoder's encoder_dict_ is a category-to-target-mean mapping (a dict), not a rank-ordered list, so backend-dependent group order doesn't affect the result - verified pandas and polars produce identical dicts across smoothing=0.0/100/ "auto" and all three `unseen` settings. Rewrote every test in test_mean_encoder.py as one @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case per behavior (41 tests, up from 20), using a narwhals-based, NaN-aware comparison helper; y is passed as a plain list in most tests, which also exercises the numpy-y bug fix on every parametrized case. test_variables_cast_as_category stays pandas-only - it exercises pandas Categorical dtype, which polars has no direct equivalent for. Verified: tests/test_encoding/test_mean_encoder.py 41 passed (was 20, 3 failing). tests/test_encoding full suite: 345 passed, 13 failed - same failing test IDs as the unmodified base minus the 4 MeanEncoder- specific ones fixed here (unmodified base: 17 failed/326 passed); remaining 13 are pre-existing and unrelated (numpy-X rejection per the narwhals check_X() contract, affecting every encoder; OrdinalEncoder's and WoEEncoder's own unmigrated fit() bugs on other in-progress branches). flake8 and mypy clean. Module imports with pandas blocked (verified in isolation from unmigrated sibling modules in the encoding package, which still import pandas on this per-file migration branch). sphinx -W build clean (only the pre-existing linkcode_resolve warning). Verified the class docstring example and every code example in docs/user_guide/encoding/MeanEncoder.rst that doesn't require the Titanic dataset against real output, and added a "With polars" section verified the same way; the Titanic-dataset examples could not be re-run in this sandbox (no network access to openml.org) but are untouched by this change. Downstream consumers (feature_engine/_prediction/base_predictor.py and target_mean_selection.py, which construct MeanEncoder internally) verified via their test suites: 66 passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt MeanEncoder to narwhals-returning check_X check_X_y now returns a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_na, _get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). The pandas value_counts/groupby fast path is unchanged - X stays native so no rehydration is needed. The narwhals branch reuses nw_X from check_X_y instead of nw.from_native(X). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in MeanEncoder tests Replace the file-local _to_backend/_assert_values helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=...). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict in MeanEncoder tests The shared helper was renamed from to_dict to frame_to_dict in #1045. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor mean encoder * Simplify MeanEncoder narwhals fit and drop init asserts from fit tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor mean enc tests --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

…#1047) * Add add_target_to_X helper, check init param types in encoders Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix get_feature_names_out error message, check missing_values type Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate OrdinalEncoder.fit() to narwhals, add polars support fit() has two paths: "arbitrary" (X[var].unique()) and "ordered" (target mean per category, via y.groupby(X[var])). transform() and inverse_transform() already came dataframe-agnostic for free from CategoricalMethodsMixin (base_encoder.py, merged separately). Benchmarked a pure-narwhals fit() (group_by/agg/sort for "ordered", unique() for "arbitrary") at 10k-100k rows x 1-10 cols x 5-50 categories: it ran 5x-18x slower than pandas-native fit() at every size tested - a large, consistent loss, unlike the ~1.1x seen for the encode/transform hot path in base_encoder.py. Per the benchmark-driven merge-vs-split rule, this is a real loss, so fit() splits on `is_pandas = nwd.is_pandas_dataframe(X)`: pandas keeps a close variant of its original groupby/unique code (confirmed via a like-for-like full-class benchmark to run within noise of the old code, ~1.0x), while polars (and any other narwhals backend) goes through group_by()/agg()/sort()/unique(). New pandas branch differs from the old code only in how "ordered" pairs y with X[var] (see bug below) - "arbitrary" is untouched. Two real issues found, confirmed against the unmodified pre-migration file (both predate this migration): 1. Bug (fixed): the old "ordered" fit() always called `y.groupby(X[var])`, which raises AttributeError whenever y is a numpy array rather than a Series - e.g. list/array-like y input, which sklearn's check_X_y machinery converts to numpy. This is exactly the scenario tests/test_encoding/test_check_estimator_encoders.py ::test_encoders_when_x_pandas_y_numpy exercises for OrdinalEncoder (encoder2, added in 2022 for issue #376) - it failed against the unmodified file and now passes. Fixed on both the pandas branch (pair X[var] with y via `.assign()`, which aligns a numpy y positionally and a Series y by index, instead of `y.groupby(X[var])`) and the narwhals branch (`nw.new_series` for a numpy y). 2. Cross-backend ordering hazard (avoided, not a regression since old code was pandas-only): grouping by category then sorting by target mean does not, by itself, guarantee the same tie-break order on ties across backends - verified polars reversed two tied categories relative to pandas without it. Old pandas code effectively tie-broke on the category itself (pandas groupby sorts keys ascending by default, and sort_values() is stable). Reproduced that explicitly with a compound sort `.sort([target_name, var])` in the narwhals branch; verified pandas and polars now produce the same dict for a deliberately tied-mean fixture, matching the old code's order exactly. Rewrote every test in test_ordinal_encoder.py as one @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]) case per behavior (43 tests, up from 26), using a narwhals-based, NaN-aware comparison helper. test_variables_cast_as_category stays pandas-only - it exercises pandas Categorical dtype, which polars has no direct equivalent for. Verified: tests/test_encoding/test_ordinal_encoder.py 43 passed. tests/test_encoding full suite: 344 passed, 16 failed - identical failing test IDs to the unmodified base (17 failures, one of which is the bug fixed above), all pre-existing and unrelated to OrdinalEncoder (numpy-X rejection per the narwhals check_X() contract, and MeanEncoder's own unmigrated fit() bug). flake8 and mypy clean. Module imports with pandas blocked. sphinx -W build clean (only the pre-existing linkcode_resolve warning, confirmed identical on the unmodified base). Verified every code example in docs/user_guide/encoding/OrdinalEncoder.rst against real output (California Housing dataset) and added a "With polars" section, verified the same way; the Titanic-dataset examples in that file could not be re-run in this sandbox (no network access to openml.org) but are untouched by this change and were not touched. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt OrdinalEncoder to narwhals-returning check_X check_X / check_X_y now return a narwhals frame, so bind that to nw_X and keep the original native X for _check_or_select_variables, _check_na, _get_feature_names_in and the nwd.is_pandas_dataframe(X) fast-path check (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). The pandas groupby/unique fast path is unchanged - X stays native so no rehydration is needed. The narwhals branch reuses nw_X from check_X / check_X_y instead of nw.from_native(X). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in OrdinalEncoder tests Replace the file-local _to_backend/_assert_values helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor code * Check encoding_method type, simplify OrdinalEncoder fit, group init tests Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use add_target_to_X in OrdinalEncoder Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate WoEEncoder to narwhals, add polars support fit() splits by backend: pandas keeps _calculate_woe()'s existing two-groupby implementation unchanged (it's directly unit-tested for that exact pandas-Series-with-category-index contract); polars/other narwhals backends use one group_by() instead of two, deriving the negative-class count as the complement of the positive-class count per category - benchmarked competitive with, and often faster than, pandas-native at 50k-100k rows. Zero-count-per-class fill_value handling preserved exactly. Bug fix: _check_fit_input() previously assumed y was always a pandas Series (y.nunique()/y.min()/y.max()), breaking on a numpy y (e.g. a plain list/array-like target, which sklearn's check_X_y machinery converts via column_or_1d). Wrapped numpy y into a narwhals Series aligned to X's backend; for pandas specifically, also had to line the wrapped Series up with X's actual index, since _calculate_woe()'s y.groupby(X[var]) aligns by index and a mismatched default RangeIndex silently drops every row instead of raising, leaving encoder_dict_ empty. Fixes test_encoders_when_x_pandas_y_numpy's WoEEncoder case (was failing on the unmigrated file, confirmed pre-existing). Verified: 44/44 own tests, full encoding suite 342 passed/16 failed (was 17 pre-existing on the narwhals-encoding-base baseline - one less here since this branch's own numpy-y bug is now fixed, rest confirmed unrelated), flake8 and mypy clean, sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt WoEEncoder to narwhals-returning check_X check_X_y now returns a narwhals frame. In _check_fit_input, bind that to nw_X and keep the original native X: the nwd.is_pandas_dataframe(X) check, the native_y.index = X.index alignment and the returned X all need native input, and fit()'s pandas _calculate_woe fast path and nwd checks are then unchanged (X stays native so no rehydration is needed). Take the y-series backend from nw_X.implementation instead of re-wrapping X. In transform(), bind _check_transform_input_and_state to nw_X, keep native X for _check_contains_na, and pass nw_X to _encode (which now expects narwhals). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in WoEEncoder tests Replace the file-local data dicts and assert_df_equal/_none_to_nan helpers with the shared test structure: make_df and data_enc* fixtures, y built with make_series on the backend under test, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Add a test passing the target as a list and as a numpy array, which take a different code path than a Series. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use add_target_to_X in WoEEncoder, group init tests, match errors Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Compute WoE with narwhals in _calculate_woe, shared by WoEEncoder and SelectByInformationValue Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Replace zero counts by 0.5 in WoE, remove fill_value, add variables_with_zero_counts_ Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Add CLAUDE.md with conventions from the narwhals migration Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Merge CLAUDE.md into AGENTS.md and drop migration-specific notes Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

* Migrate OneHotEncoder to narwhals, add polars support Uses narwhals' to_dummies() for the actual expansion rather than a manual numpy/dict loop, since it's a real vectorized one-hot op on both backends. Handles two edge cases to_dummies() doesn't cover directly: a fixed-length prefix placeholder ("__ohe_tmp__") swapped back out by slicing rather than by using the real column name, since to_dummies() only prefixes with the Series name when it's truthy - a falsy real name (e.g. an int column literally named 0) would otherwise silently drop the prefix; and learned categories absent from (or present-but-unlearned in) a given transform batch, filled with an explicit all-0 column so unseen categories are encoded as 0 across the board, matching the pre-narwhals behavior exactly. fit()'s value_counts()/unique() calls and transform()'s reassembly are a single unified narwhals path - no pandas/polars split needed, verified directly on both backends (identical dummy columns/values for identical input). Rewrote tests/test_encoding/test_onehot_encoder.py to the single cross-backend-parametrized-test convention: local dict fixtures (dropping the pandas-only global df_enc_big/df_enc_numeric/df_enc_binary fixtures) parametrized over make_df in [pd.DataFrame, pl.DataFrame], with narwhals- based column/sum assertions replacing pd.testing.assert_frame_equal. test_variables_cast_as_category stays pandas-only (pandas category dtype has no polars equivalent under test there). Verified: 43/43 own tests, full encoding suite 340 passed/17 pre-existing failures (matches the narwhals-encoding-base baseline exactly), flake8 and mypy clean, sphinx -W build clean (only the pre-existing unrelated linkcode_resolve warning), no pandas import in this file itself. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt OneHotEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_contains_na and _get_feature_names_in (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the now-redundant nw.from_native(X) round-trips in fit() and transform(); they reuse the narwhals frame returned by check_X / _check_transform_input_and_state. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in OneHotEncoder tests Replace the file-local data dicts and _columns/_colsum helpers with the shared test structure: make_df and data_enc* fixtures, isinstance(X, make_df) plus to_dict() checks (keeping the column-order assertions, which are part of this encoder's output contract), and pytest.raises(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group OneHotEncoder init tests, match errors, shorten comments Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Match the fixed get_feature_names_out error message Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor ohe * fix code style --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate RareLabelEncoder to narwhals, add polars support fit() replaces pandas .unique()/.value_counts(normalize=True) with narwhals Series.n_unique() (for the cardinality check - matches pandas' plain unique() length, which counts a null as its own category, unlike pandas' nunique() which drops it) and drop_nulls().value_counts(sort=True, normalize=True) (same drop_nulls()/sort=True reasoning as CountEncoder.fit(): narwhals' value_counts() has no dropna param and narwhals' own value_counts default is unsorted). transform() doesn't reuse CategoricalMethodsMixin._encode() (that's a dict-based numeric remap; this encoder keeps frequent categories as-is and only replaces the rest), so it's rewritten from pandas' .loc[~isin(...), feature] = replace_with onto nw.when(<Series>).then(<Series>).otherwise(nw.lit(replace_with)).alias( feature). Passing Series from get_column() (not nw.col()) into when/then/otherwise keeps this working for pandas integer column names, same as base_encoder.py's precedent. A pandas Categorical column still needs its own add_categories(replace_with) step before assignment - kept as a small is_pandas-gated block (structural, like base_encoder.py's existing reorder branches), since narwhals has no cross-backend equivalent and polars has no matching restriction. Unlike the old pandas-only code, no manual object-dtype fixup is needed before assignment for the ignore_format + numeric-variable + string replace_with case: narwhals resolves the common dtype itself (object in pandas, cast-to-string in polars). Benchmarked pandas-native vs narwhals-on-pandas vs narwhals-on-polars at 10k/50k/100k rows x 1/2/10 columns x 5/50 categories, warmed up. First pass (zip_with(col, new_series_filled_with_replace_with)) averaged 2.41x pandas-native at 50k-100k rows - most of that cost was constructing a full same-length replacement Series every transform() call (~2.5ms of a ~4.8ms transform at 100k rows, confirmed by isolating just the Series construction). Switched to nw.when(keep).then(col) .otherwise(nw.lit(replace_with)), which lets the backend broadcast the scalar instead of materialising a parallel array: dropped the average to 1.60x, converging to 1.12x-1.54x at 100k rows/10 columns, the "realistic size" range. narwhals-on-polars is faster than pandas-native throughout (0.7x-1.5x, mostly <1x at 50k+ rows). Merged into a single narwhals path per the established decision rule - no pandas/polars performance split - the remaining overhead is fixed per-call cost, not scaling cost, and stays under a few ms in absolute terms even at the largest sizes tested. Rewrote test_rare_label_encoder.py to one parametrized test per behaviour over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]), replacing the shared pandas-only module-level fixtures (df_enc_big, df_enc_big_na, df_enc_numeric, from tests/conftest.py, still used by other encoder test files) with local dict constants both backends can build from, per the CountEncoder precedent. Kept test_when_varnames_are_numbers and the three category-dtype tests pandas-only (integer column names and pandas Categorical dtype are backend-specific per AGENTS.md). Split test_max_n_categories_with_numeric_var into a pandas-only version (the existing str()-workaround test, unchanged) plus a new polars-only version documenting the real, expected behavioural difference: polars can't hold mixed int/str values in one column the way pandas' object dtype does, so a numeric variable with a string replace_with casts the whole column to string instead of leaving frequent numeric categories as numbers. Verified: tests/test_encoding/test_rare_label_encoder.py - 39 passed (up from 29, from parametrizing over both backends); full tests/test_encoding suite - 336 passed, 17 pre-existing failures with identical test IDs confirmed against the unmodified base_encoder.py baseline (numpy-array-input rejection checks plus 3 MeanEncoder inverse_transform failures from mean_encoding.py's still-unmigrated fit() - predate this change, reproduced identically on the unmodified rare_label.py too). flake8 clean on feature_engine and tests. mypy clean. Module imports with pandas blocked (loaded standalone, same technique as the base_encoder.py migration, since sibling encoder files in this package still import pandas at module level). sphinx -W build clean (only the pre-existing linkcode_resolve warning, confirmed identical against the unmodified baseline). Verified every doc example in RareLabelEncoder.rst against actual output; fixed a pre-existing, unrelated value_counts() Series-name drift ("Name: var_A" -> "Name: count", a pandas version difference, not caused by this migration) while touching that page, and added a verified "With polars" section to both the class docstring and the user guide (the polars value_counts() example needed an explicit .sort() - unlike pandas, its groupby-based value_counts() order isn't stable run to run). The Titanic-dataset section of the user guide could not be re-verified against live output in this sandboxed environment (SSL cert verification blocks urllib by default here, though curl succeeds) and was left untouched; a workaround (unverified SSL context) showed matching encoder_dict_/transform output, with only an unrelated .unique() repr-formatting difference from a newer pandas version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt RareLabelEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables, _check_na and _check_contains_na (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) round-trips in fit() and transform(). In transform(), detect the pandas Categorical fix-up path via nw_X.implementation .is_pandas() and run it on a copy so the user's dataframe is not mutated. Drop the now-unused narwhals.dependencies import. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in RareLabelEncoder tests Replace the file-local data dicts and the _to_pandas helper - whose polars to_pandas() call needs pyarrow, which is not a dependency, so the polars cases failed - with the shared test structure: make_df and data_enc_big* / data_enc_numeric fixtures, isinstance(X, make_df) plus to_dict() checks, and pytest.raises/warns(match=re.escape(msg)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Group RareLabelEncoder init tests, match errors, shorten comments Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix typo in RareLabelEncoder replace_with error Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * minor refactor to tests --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Migrate StringSimilarityEncoder to narwhals, add polars support fit() rebuilds encoder_dict_ with narwhals cast(nw.String)/value_counts, matching the CountEncoder/RareLabelEncoder convention. cast() preserves nulls as null on both pandas and polars (verified empirically), unlike pandas' own astype(str) which stringifies NaN to "nan" - this lets "impute" mode fill_null("") directly and "ignore" mode drop_nulls() before casting, replacing the old "nan"/"<NA>" text-sentinel workaround with a real null check (col.is_null()) that can't collide with a genuine category literally named "nan" or "<NA>" (both edge cases stay covered by test_string_dtype_with_literal_nan_strings). transform()'s per-row difflib.SequenceMatcher similarity has no vectorised narwhals equivalent, so it's computed once per unique value via numpy broadcasting (np.unique's inverse index fans the small per-unique-value matrix back out to all rows) and reassembled with nw.new_series()/with_columns(), same pattern DecisionTreeFeatures uses for externally-computed new columns. Benchmarked a pandas-specific fast path (X.join(dict-of-columns), as DecisionTreeFeatures uses) against the unified narwhals with_columns() here across 10k-100k rows x 1-10 columns x 5-50 categories: assembly overhead ranges 0.9x-6.25x depending on shape, but the difflib computation itself dominates wall time by 1-3 orders of magnitude in every realistic scenario (e.g. 30ms difflib vs <1ms assembly overhead at 100k rows/20 categories) - even the worst synthetic case (500 output columns) only costs ~10ms extra out of an already tens-of-ms-to-seconds transform. Went with the unified/merged implementation: no is_pandas split, one code path for both backends. Rewrote tests as single parametrized cases over @pytest.mark.parametrize("make_df", [pd.DataFrame, pl.DataFrame]), keeping only the pandas-NA-sentinel tests (np.nan/pd.NA/None, StringDtype) pandas-only since polars has no equivalent multi-sentinel behavior to exercise. All doc examples (including the Titanic worked example) re-verified against actual output; added a "With polars" section. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Adapt StringSimilarityEncoder to narwhals-returning check_X Bind check_X / _check_transform_input_and_state results to nw_X and keep the original native X for _check_or_select_variables and _check_contains_na (those helpers still expect native input, matching the CategoricalImputer migration on narwhals-migration). Drop the redundant nw.from_native(X) round-trips in fit() and transform(). The empty-variables short-circuit in transform() now returns nw_X.to_native() so callers still get a native frame. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use shared backend test fixtures and helpers in StringSimilarityEncoder tests Replace the file-local data dicts and _to_pandas/_columns helpers with the shared test structure: make_df and data_enc* fixtures, isinstance(X, make_df) plus to_dict() checks, and pytest.raises(match=re.escape(msg)). Tests of pandas-specific NA sentinels and the nullable string dtype stay pandas-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Use frame_to_dict after the shared helper rename in #1045 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Check missing_values type, group StringSimilarityEncoder init tests, match errors Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Match the fixed get_feature_names_out error message Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * refactor enc dict at the end --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

This was referenced Jul 26, 2026

Request to use polars dataframe package directly #809

Open

add missing_only=True to all imputers to use in combination with variables=None #974

Closed

ojassharma7 mentioned this pull request Jul 29, 2026

[MNT] migrate scaling module to narwhals #979

Closed

2 tasks

solegalli force-pushed the narwhals-migration branch 3 times, most recently from 8fe8359 to ea95750 Compare July 31, 2026 12:30

FBruzzesi reviewed Aug 23, 2026

View reviewed changes

solegalli mentioned this pull request Aug 24, 2026

address review feedback on dataframe_checks.py #989

Merged

solegalli and others added 6 commits August 24, 2026 12:12

start narwhals migration

fbbf160

fix doc test

7f33552

solegalli force-pushed the narwhals-migration branch from 0ef7acb to ff5d212 Compare August 24, 2026 10:14

solegalli and others added 6 commits August 24, 2026 17:00

FBruzzesi mentioned this pull request Aug 24, 2026

perf: Speedup pandas-like is_finite, sqrt, kurtosis and skew narwhals-dev/narwhals#3874

Merged

7 tasks

solegalli and others added 3 commits August 25, 2026 20:20

solegalli and others added 2 commits August 30, 2026 16:55

Adapt numerical base transformers to narwhals-returning check_X (#1022)

0fca9c6

solegalli force-pushed the narwhals-migration branch from 04c88dc to e9f7d69 Compare August 30, 2026 15:40

solegalli force-pushed the narwhals-migration branch from e9f7d69 to 04c88dc Compare August 30, 2026 15:49

solegalli and others added 25 commits August 30, 2026 18:55

update base imputer again (#1023)

211d79e

Shorten n_jobs docstring in DecisionTreeFeatures (#1049)

e2e3ff3

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

	if not np.isfinite(nw_y.to_numpy()).all():
	if not nw_y.is_finite().all():

	if nw_y.select(nw.all().is_null().any()).to_numpy().any():
	if nw_y.select(nw.any_horizontal(nw.all().is_null().any())).item():

Conversation

solegalli commented Jul 21, 2026

Uh oh!

ojassharma7 commented Jul 28, 2026

Uh oh!

solegalli commented Jul 29, 2026

Uh oh!

ojassharma7 commented Jul 29, 2026

Uh oh!

ojassharma7 commented Jul 29, 2026

Uh oh!

FBruzzesi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants