Pipeline representation and relationship betwen KFP/Vertex AI Pipelines and TFX #7872

If the feature is related to a specific library below, please raise an issue in
the respective repo directly:

TensorFlow Data Validation Repo

TensorFlow Model Analysis Repo

TensorFlow Transform Repo

TensorFlow Serving Repo

System information

• TFX Version (you are using):
• Environment in which you plan to use the feature (e.g., Local
  (Linux/MacOS/Windows), Interactive Notebook, Google Cloud, etc..):
• Are you willing to contribute it (Yes/No):

Describe the feature and the current behavior/state.
Today, TFX pipelines have to be compiled to kubeflow pipelines to run on vertex. We should also support the other direction, taking kubeflow pipelines (e.g. pipeline spec/yaml representation) and converting them to a representation runnable by the tfx runner.

Once we have this working, we should enable vertex ai pipelines to be a runner for TFX pipelines natively. The reason for this is it is quite complex to build these pipelines without kubernetes knowhow. If we can build that bridge, and support jax, we can make it a lot easier to build large scale ML pipelines using re-usable components in a way that doesn't require knowledge/expertise of the underlying infrastructure.

Will this change the current API? How?

TFX would get a transpiler for kubeflow pipelines that use TFX components.
For the second step (which cannot be implemented as part of this project, but part of vertex ai pipelines), vertex ai pipelines would become a runner for TFX pipelines directly without having to go through the KFP route.

Who will benefit with this feature?

ML practitioners who do now want to deal with the infrastructure complexity. It would be a similar model to dataflow, write one pipeline, run it anywhere. Implementing this would improve the bridge between TFX and KFP, which today is quite brittle.

Do you have a workaround or are completely blocked by this? :

I have kubernetes infrastructure and knowhow, so I'm not blocked by this.

Name of your Organization (Optional)

Intuitive.ai

Any Other info.