✨ MOVE update

.gitignore

@@ -1,45 +1,52 @@
+# OS and editor files
+.DS_Store
+.vscode/
+.idea/
+
 # Jupyter NB checkpoints
 .ipynb_checkpoints
 
 # Python compiled bytecode
 __pycache__/
 *.py[cod]
 
-# NumPy binary files
-data*/*.npy
+# NumPy/PyTorch binary files
+*.npy
+*.pt
 
 # Distribution and packaging files
 build/
-develop-eggs/
 dist/
-downloads/
-eggs/
-.eggs/
-lib/
-lib64/
-parts/
-sdist/
-var/
-wheels/
-*.egg
 *.egg-info/
-.installed.cfg
-MANIFEST
-
-# Logs
-outputs/
-*.log
 
-# Tutorial files
-**/interim_data/
-**/processed_data/
-**/results/
-tutorial/maize/data
+# Testing
+.pytest_cache/
 
 # Virtual environment
 venv/
-virtualvenv/
+.venv/
+
+# Logs and outputs of MOVE runs
+outputs/
+logs/
+*.log
+/interim_data/
+/results/
 
 # docs files
 docs/build/
-docs/source/_templates/
+docs/source/_templates/
+
+# Tutorial files: keep configs, input data, notebooks and README; ignore the
+# rest (e.g., interim data and results generated when running the tutorial)
+tutorial/*
+!tutorial/config/
+!tutorial/data/
+!tutorial/maize/
+tutorial/maize/*
+!tutorial/maize/maize_dataset.py
+!tutorial/notebooks/
+tutorial/notebooks/*
+!tutorial/notebooks/*.ipynb
+!tutorial/notebooks/README.md
+!tutorial/README.md

README.md

@@ -58,6 +58,10 @@ you do not have powerful GPUs available, it is possible to run using only CPUs.
 For instance, the tutorial data set consisting of simulated drug, metabolomics
 and proteomics data for 500 individuals runs fine on a standard macbook.
 
+> Note: The pip installation of `move-dl` does not setup your local GPU automatically.
+> To use GPU acceleration, install a CUDA-enabled PyTorch build following the
+> instructions at [pytorch.org](https://pytorch.org/get-started/locally/).
+
 # The MOVE pipeline
 
 MOVE has five-six steps:
@@ -68,8 +72,7 @@ MOVE has five-six steps:
 03. Finding the right architecture of the network focusing on stability of the model
 04. Use model, determined from steps 02-03, to create and analyze the latent space
 05. Identify associations between a categorical and continuous datasets
-05a. Using an ensemble of VAEs with the t-test approach
-05b. Using an ensemble of VAEs with the Bayesian decision theory approach
+05. Using an ensemble of VAEs with the Bayesian decision theory approach
 06. If both 5a and 5b were run select the overlap between them
 ```
 

docs/source/index.rst

@@ -9,6 +9,7 @@ Welcome to MOVE's documentation!
    install
    method
    tutorial/index
+   pages/api/API
 
 MOVE (**m**\ ulti-\ **o**\ mics **v**\ ariational auto\ **e**\ ncoder) is a
 framework for integration of omics and other data modalities (including both

docs/source/method.rst

@@ -59,6 +59,25 @@ compare the change between the reconstruction generated from the original data
 and the perturbed data. To achieve this, we proposed two approaches: using
 *t*\ -test and Bayes factors. Both are described below:
 
+Continuous features can be perturbed too, via ``perturbation_type`` on
+:class:`~move.conf.tasks.PerturbationConfig` (used by
+:class:`~move.tasks.associations.Associations`). Unlike a categorical feature,
+where "change from category A to B" is unambiguous, a single hardcoded target
+value rarely means the same thing across differently-scaled continuous
+features, so five perturbation types are available:
+
+- ``value`` (the default, and the only type applicable to categorical
+  features) replaces every sample's value with a fixed ``target_value``.
+- ``minimum``/``maximum`` replace every sample's value with the feature's own
+  dataset-wide minimum/maximum.
+- ``plus_std``/``minus_std`` add or subtract one standard deviation **to each
+  sample's own existing value**, rather than replacing it. The only types
+  that perturb relative to each sample individually instead of moving every
+  sample to the same point.
+
+Whichever type is used, the resulting perturbed dataset is passed through the
+same procedure described below as the categorical case).
+
 MOVE *t*\ -test
 ^^^^^^^^^^^^^^^
 

docs/source/pages/api/API.rst

@@ -5,5 +5,8 @@ API
     :maxdepth: 2
 
     configuration_schemas.rst
-    functions.rst
+    data.rst
+    tasks.rst
     models.rst
+    training.rst
+    functions.rst

docs/source/pages/api/configuration_schemas.rst

@@ -1,2 +1,202 @@
 Configuration schemas
-=====================
+=====================
+
+MOVE is driven by `Hydra <https://hydra.cc/>`_ configuration files, whose
+structure is validated against the dataclasses documented on this page.
+
+Main configuration
+-------------------
+
+.. autoclass:: move.conf.schema.MOVEConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.DataConfig
+    :members:
+    :undoc-members:
+
+Task configurations
+--------------------
+
+.. autoclass:: move.conf.schema.TaskConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.EncodeDataConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.MoveTaskConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.LatentSpaceAnalysisConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.AssociationsConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.TuningConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.schema.StabilityTuningConfig
+    :members:
+    :undoc-members:
+
+Input configurations
+----------------------
+
+Used in ``categorical_inputs``/``continuous_inputs`` (``data`` config) to
+describe each dataset that should be pre-processed by the ``EncodeData`` task.
+
+.. autoclass:: move.conf.tasks.InputConfig
+    :members:
+
+.. autoclass:: move.conf.tasks.DiscreteInputConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.tasks.ContinuousInputConfig
+    :members:
+    :undoc-members:
+
+Dimensionality-reduction configurations
+-----------------------------------------
+
+Used in ``reducer_config`` (latent space analysis task) to further reduce the
+latent space to a plottable number of dimensions.
+
+.. autoclass:: move.conf.tasks.ReducerConfig
+    :members:
+
+.. autoclass:: move.conf.tasks.PcaConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.tasks.TsneConfig
+    :members:
+
+.. note::
+    If the optional ``umap-learn`` dependency is installed, a
+    ``move.conf.tasks.UmapConfig`` (wrapping ``umap.UMAP``, with an
+    ``n_neighbors`` attribute) is also registered under this group.
+
+Perturbation configuration
+----------------------------
+
+Used in ``perturbation_config`` (associations task) to describe the *in
+silico* perturbation experiment to run.
+
+.. autoclass:: move.conf.tasks.PerturbationConfig
+    :members:
+
+Model configurations
+----------------------
+
+.. autoclass:: move.conf.models.ModelConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.models.VaeConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.models.VaeNormalConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.models.VaeTConfig
+    :members:
+    :undoc-members:
+
+Training loop and dataloader configurations
+----------------------------------------------
+
+.. autoclass:: move.conf.training.DataLoaderConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.training.TrainingDataLoaderConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.training.TestDataLoaderConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.training.TrainingLoopConfig
+    :members:
+    :undoc-members:
+
+Optimizer configurations
+---------------------------
+
+.. autoclass:: move.conf.optim.OptimizerConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.AdamConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.AdamWConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.ProdigyConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.SgdConfig
+    :members:
+    :undoc-members:
+
+Learning-rate scheduler configurations
+------------------------------------------
+
+.. autoclass:: move.conf.optim.LrSchedulerConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.ExponentialLrConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.CosineAnnealingLrConfig
+    :members:
+    :undoc-members:
+
+.. autoclass:: move.conf.optim.ReduceLrOnPlateauConfig
+    :members:
+    :undoc-members:
+
+Resolvers
+-----------
+
+Custom `OmegaConf resolvers <https://omegaconf.readthedocs.io/en/2.3_branch/custom_resolvers.html>`_
+used to derive values (e.g., dataset names/weights) from the ``data`` config's
+``categorical_inputs``/``continuous_inputs`` lists.
+
+.. autofunction:: move.conf.resolvers.register_resolvers
+
+.. autofunction:: move.conf.resolvers.extract_names
+
+.. autofunction:: move.conf.resolvers.extract_weights
+
+Legacy configurations
+------------------------
+
+.. warning::
+    The dataclasses in ``move.conf.legacy`` configure the pre-refactor
+    function-based tasks (``move.tasks.tune_model``,
+    ``move.tasks.identify_associations``, ``move.tasks.analyze_latent``) and
+    the legacy ``move.models.vae_legacy.VAE`` model. New projects should use
+    the configurations documented above instead.
+
+.. automodule:: move.conf.legacy
+    :members:
+    :undoc-members:
+    :show-inheritance: