Performance & Helm Chart Improvements

.travis.yml

@@ -10,7 +10,7 @@ before_script:
   - python tr_sys/manage.py migrate
   - docker run -d -p 5672:5672 rabbitmq
   - sleep 10 # give some time for server to come up
-  - cd tr_sys; celery -A tr_sys worker -l info &
+  - cd tr_sys; celery -A tr_sys worker -l info -Q celery,heavy &
   - sleep 10 # give some time for server to come up
   - cd ..; python tr_sys/manage.py runserver --noreload &
   - sleep 10 # give some time for server to come up

config/start_celery.sh

@@ -3,7 +3,10 @@
 cd /ars/tr_sys
 # Start beat with log file
 celery -A tr_sys beat -l info -f /var/log/celerybeat.log &
-# Start worker in foreground
-
-celery -A tr_sys worker -l info --concurrency=10 --prefetch-multiplier=1
+# Start worker in foreground. Single-worker deployments (docker-compose, the
+# legacy deploy/ chart) must consume BOTH queues: merge-and-post-process and
+# ingest-ara-response are routed to `heavy` (tr_sys/celery.py task_routes) and
+# would otherwise sit in RabbitMQ with no consumer. helm/ars runs a separate
+# worker pool per queue instead.
+celery -A tr_sys worker -l info -Q celery,heavy --concurrency=10 --prefetch-multiplier=1
 

config/tr_sys_settings.py

@@ -42,7 +42,6 @@
     'django.contrib.messages',
     'django.contrib.staticfiles',
     'corsheaders',
-    #'channels',
     'tr_ars.apps.ARSConfig',
     #'tr_ars.default_ars_app.ars_app.AppConfig',
     'tr_ara_aragorn.aragorn_app.AppConfig',
@@ -100,17 +99,6 @@
 ]
 
 WSGI_APPLICATION = 'tr_sys.wsgi.application'
-# Channels
-ASGI_APPLICATION = 'tr_sys.routing.application'
-
-CHANNEL_LAYERS = {
-    'default': {
-        'BACKEND': 'channels_redis.core.RedisChannelLayer',
-        'CONFIG': {
-            "hosts": [('redis', 6379)],
-        },
-    },
-}
 
 # Database
 # https://docs.djangoproject.com/en/1.11/ref/settings/#databases

docker-compose.yml

@@ -34,12 +34,6 @@ services:
       - ./config/tr_sys_settings.py:/ars/tr_sys/tr_sys/settings.py
     depends_on:
       - arsdb
-      - redis
-  redis:
-    image: redis:6.2
-    container_name: ars_redis
-    expose:
-      - 6379
   celery_worker:
     command: sh -c "wait-for rabbitmq:5672 -t 30 && wait-for arsserver:8000 -t 60 -- /bin/bash /ars/config/start_celery.sh"
     container_name: celery_worker

helm/ars/Chart.yaml

@@ -0,0 +1,11 @@
+apiVersion: v2
+name: ars
+description: >-
+  Helm chart for the ARS (Autonomous Relay System) and its backing services.
+  Unlike deploy/ (the Jenkins-driven production chart), this chart is
+  self-contained: it deploys MySQL and RabbitMQ alongside the app so a
+  dev instance can be installed on any cluster with no externally supplied
+  config files.
+type: application
+version: 0.2.0
+appVersion: "1.42.1"

helm/ars/README.md

@@ -0,0 +1,72 @@
+# ars (self-contained dev chart)
+
+A Helm chart that deploys the ARS **and all of its backing services** — MySQL
+and RabbitMQ — so a dev instance comes up on any Kubernetes cluster with
+no externally supplied files. It is the outcome of `deploy/HELM_DEPENDENCIES.md`;
+the Jenkins-driven production chart in `deploy/` is untouched.
+
+Differences from `deploy/`:
+
+| | `deploy/` | `helm/ars/` |
+|---|---|---|
+| Topology | one pod, 4 containers over `localhost` | one pod per component (server / heavy worker / light worker / beat / mysql / rabbitmq) |
+| MySQL | external, invisible to the chart | in-cluster StatefulSet + PVC by default, or `mysql.enabled=false` + `mysql.external.*` |
+| RabbitMQ | sidecar, no volume | own pod with a PVC (celery is configured for persistent delivery) |
+| Redis | sidecar (token gate + unused Channels layer) | gone; nothing in the app uses it anymore |
+| `settings.py` | injected by Jenkins, `sed`-substituted | in the chart (`files/settings.py`), reads env vars; credentials via a Secret |
+| Migrations | `manage.py migrate` in the server container command | post-install/pre-upgrade hook Job (keep migrations backward compatible) |
+| Celery beat | started inside the worker script | own single-replica Deployment |
+| Scaling | impossible (replicating the pod duplicates the broker) | `arsserver.replicas` / `celeryworkers.<pool>.replicas` |
+
+## Quick start (kind / minikube / any dev cluster)
+
+```sh
+# 1. Build the app image and make it available to the cluster
+docker build -t relay_ars:latest .
+kind load docker-image relay_ars:latest        # or: minikube image load relay_ars:latest
+
+# 2. Install
+helm install ars ./helm/ars -n ars --create-namespace
+
+# 3. Talk to it
+kubectl -n ars port-forward svc/ars 8000:80
+curl http://localhost:8000/ars/api/health
+```
+
+On a cluster with ECR access, skip the build and use the CI image:
+
+```sh
+helm install ars ./helm/ars -n ars --create-namespace \
+  --set image.repository=853771734544.dkr.ecr.us-east-1.amazonaws.com/translator-ars \
+  --set image.tag=<version> --set image.pullPolicy=Always
+```
+
+## Using external backing services
+
+```yaml
+mysql:
+  enabled: false
+  external:
+    host: my-rds.cluster-xyz.us-east-1.rds.amazonaws.com
+    port: 3306
+    database: arsdb
+    username: ars
+```
+
+Passwords always come from the release Secret — set `secrets.mysqlPassword`
+etc., or point `secrets.existingSecret` at a Secret you manage (keys:
+`django-secret-key`, `aes-master-key`, `mysql-root-password`, `mysql-password`,
+`rabbitmq-password`).
+
+## Notes and limits
+
+- **Dev credentials ship in `values.yaml`.** Override every `secrets.*` value
+  (or use `secrets.existingSecret`) for any instance others can reach.
+- The in-cluster MySQL is a single node with no backups — fine for dev, not for
+  anything you'd miss.
+- Queries fan out to the live ARAs/KPs selected by `env.TR_ENV` (SmartAPI
+  maturity level).
+- OTEL trace export points at `otel.jaegerHost`; if no collector exists at that
+  address the exporter logs connection errors and the app carries on.
+- `affinity`/`tolerations` are empty by default so pods schedule anywhere; copy
+  the node pinning from `deploy/values.yaml` if an environment needs it.

helm/ars/files/settings.py

@@ -0,0 +1,201 @@
+"""
+Django settings for tr_sys, as deployed by the helm/ars chart.
+
+This file is mounted over tr_sys/tr_sys/settings.py (same mechanism as the
+production deploy/ chart) but takes every connection parameter and credential
+from environment variables, which the chart supplies from values.yaml and its
+Secret. It should track tr_sys/tr_sys/settings.py apart from those env reads.
+"""
+
+import os
+from .otel_config import configure_opentelemetry
+
+configure_opentelemetry()
+
+# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
+BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+SECRET_KEY = os.environ.get('DJANGO_SECRET_KEY', 'ars-dev-insecure-secret-key')
+
+DEBUG = os.environ.get('DJANGO_DEBUG', 'true').lower() in ('1', 'true', 'yes')
+
+ALLOWED_HOSTS = [h.strip() for h in os.environ.get('DJANGO_ALLOWED_HOSTS', '*').split(',') if h.strip()]
+
+SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')
+
+# Application definition
+
+INSTALLED_APPS = [
+    'django.contrib.admin',
+    'django.contrib.auth',
+    'django.contrib.contenttypes',
+    'django.contrib.sessions',
+    'django.contrib.messages',
+    'django.contrib.staticfiles',
+    'corsheaders',
+    'tr_ars.apps.ARSConfig',
+    'tr_ara_aragorn.aragorn_app.AppConfig',
+    'tr_ara_arax.arax_app.AppConfig',
+    'tr_ara_bte.bte_app.AppConfig',
+    'tr_ara_improving.improving_app.AppConfig',
+    'tr_ara_unsecret.unsecret_app.AppConfig',
+    'tr_ara_wfr.wfr_app.AppConfig',
+    'tr_ara_cqs.cqs_app.AppConfig',
+    'tr_ara_shepherd_aragorn.shepherd_aragorn_app.AppConfig',
+    'tr_ara_shepherd_arax.shepherd_arax_app.AppConfig',
+    'tr_ara_shepherd_bte.shepherd_bte_app.AppConfig',
+    'tr_kp_genetics.genetics_app.AppConfig',
+    'tr_kp_clinical.clinical_app.AppConfig',
+    'tr_kp_drug.drug_app.AppConfig',
+    'tr_kp_molecular.molecular_app.AppConfig',
+    'tr_kp_cam.cam_app.AppConfig',
+    'tr_kp_textmining.textmining_app.AppConfig',
+    'tr_kp_openpredict.openpredict_app.AppConfig',
+    'tr_kp_cohd.cohd_app.AppConfig',
+    'tr_kp_chp.chp_app.AppConfig',
+    'django_celery_results',
+    'markdownify',
+    'django_celery_beat',
+]
+
+MIDDLEWARE = [
+    'django.middleware.security.SecurityMiddleware',
+    'django.contrib.sessions.middleware.SessionMiddleware',
+    'corsheaders.middleware.CorsMiddleware',
+    'django.middleware.common.CommonMiddleware',
+    'django.middleware.csrf.CsrfViewMiddleware',
+    'django.contrib.auth.middleware.AuthenticationMiddleware',
+    'django.contrib.messages.middleware.MessageMiddleware',
+    'django.middleware.clickjacking.XFrameOptionsMiddleware',
+]
+
+ROOT_URLCONF = 'tr_sys.urls'
+
+CORS_ORIGIN_ALLOW_ALL = True # for further customization see https://pypi.org/project/django-cors-headers/
+
+TEMPLATES = [
+    {
+        'BACKEND': 'django.template.backends.django.DjangoTemplates',
+        'DIRS': [os.path.join(BASE_DIR, 'tr_ars','templates')],
+        'APP_DIRS': True,
+        'OPTIONS': {
+            'context_processors': [
+                'django.template.context_processors.debug',
+                'django.template.context_processors.request',
+                'django.contrib.auth.context_processors.auth',
+                'django.contrib.messages.context_processors.messages',
+            ],
+        },
+    },
+]
+
+WSGI_APPLICATION = 'tr_sys.wsgi.application'
+
+# Database
+# https://docs.djangoproject.com/en/1.11/ref/settings/#databases
+
+DATABASES = {
+    'default': {
+        'ENGINE': 'django.db.backends.mysql',
+        'NAME': os.environ.get('ARS_DB_NAME', 'arsdb'),
+        'USER': os.environ.get('ARS_DB_USER', 'ars'),
+        'PASSWORD': os.environ.get('ARS_DB_PASSWORD', ''),
+        'HOST': os.environ.get('ARS_DB_HOST', 'localhost'),
+        'PORT': os.environ.get('ARS_DB_PORT', '3306'),
+    }
+}
+
+DJANGO_LOG_LEVEL = os.environ.get('DJANGO_LOG_LEVEL', 'DEBUG')
+LOGGING = {
+    'formatters': {
+        'simple': {
+            'format': '{levelname} {asctime} {module} {process:d} {thread:d} {message}',
+            'style': '{',
+        }
+    },
+    'version': 1,
+    'disable_existing_loggers': False,
+    'handlers': {
+        'console': {
+            'class': 'logging.StreamHandler',
+            'formatter': 'simple'
+        }
+    },
+    'root': {
+        'handlers': ['console'],
+        'level': DJANGO_LOG_LEVEL,
+    },
+    'loggers': {
+        'tr_ars.tasks': {
+            'level': DJANGO_LOG_LEVEL,
+            'handlers': ['console'],
+        },
+        'tr_ars.default_ars_app.api': {
+            'level': DJANGO_LOG_LEVEL,
+            'handlers': ['console']
+        }
+    }
+}
+
+# Password validation
+# https://docs.djangoproject.com/en/1.11/ref/settings/#auth-password-validators
+
+AUTH_PASSWORD_VALIDATORS = [
+    {
+        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
+    },
+    {
+        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
+    },
+    {
+        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
+    },
+    {
+        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
+    },
+]
+
+
+# Internationalization
+# https://docs.djangoproject.com/en/1.11/topics/i18n/
+
+LANGUAGE_CODE = 'en-us'
+
+TIME_ZONE = 'UTC'
+
+USE_I18N = True
+
+USE_L10N = True
+
+USE_TZ = True
+
+# markdowninfy
+
+MARKDOWNIFY_STRIP = False
+MARKDOWNIFY_WHITELIST_TAGS = {
+    'a', 'p',
+    'h1', 'h2', 'h3','h4', 'h5', 'h6', 'h7',
+    'ul', 'li', 'span',
+}
+
+# Static files (CSS, JavaScript, Images)
+# https://docs.djangoproject.com/en/1.11/howto/static-files/
+
+STATIC_URL = '/static/'
+STATIC_ROOT = os.path.join(BASE_DIR, "static")
+
+# Celery settings
+
+CELERY_RESULT_BACKEND = 'django-db'
+CELERY_CACHE_BACKEND = 'django-cache'
+CELERY_BROKER_URL = os.environ.get('CELERY_BROKER_URL', 'amqp://localhost')
+CELERY_IMPORTS = [
+    'tr_ars.tasks',
+]
+# Other important shared settings
+DATA_UPLOAD_MAX_MEMORY_SIZE=1073741824
+CELERY_TASK_ALWAYS_EAGER=False
+CELERY_TASK_ACKS_LATE = True
+
+USE_CELERY = True
+DEFAULT_HOST = os.environ.get('ARS_DEFAULT_HOST', 'http://localhost:8000')