Repository navigation
All algorithms should be usable with generic containers #622
Description
Activity
Some thoughts: one conclusion from recent
columnarstuff is that it's really the container and its thoughts about reference types that drive a lot of the code. Additionally, the container is the thing that defines the layer, rather than the owned input type, and it seems its opinions go further than the type: also e.g. the order and layout, for effective merging/intersection/etc.I think point being, imo it may be fine to only be some amount of generic: if
half_joinneeds to take a container as input, it seems reasonable for it to say "it needs to be my container" rather than making it generic over other containers as well. Transforming the container is a thing we can do, but we can probably avoid needing to teach each operator how to do this, and instead could have a bump on the wire operator for moments where it turns out we want to fit together two unrelated containers.
Differential offers many algorithms that absorb input data in some form. Most (all?) are compatible with vector-based data, and some support generic containers, or specific instances of containers other than vectors. We'd like to support generic containers or make it possible to easily integrate new specific containers for all parts of Differential. Here's an overview where we stand:
Input, but be generic over some variant ofPush.VecD/T/R, but not very usable for other containers thanVec.Vecmap_in_placeflat_mapfilterexplodejoin_functionenter/leaveenter_atdelayinspectinspect_batchassert_empty: needsconsolidateconsolidate/consolidate_named: Work-around to build arrangement.consolidate_stream: UsesConsolidatingContainerBuildernegateis aware of containers (traitoperators::Negate), but generic parameters are in wrong order. See Introduce EnterTime, fix Negate #587ConsolidatingContainerBuilderTimelyStackThere are a bunch of types that we don't use a lot, so the list is likely incomplete.