Skip to content

feat: add CompetencyFramework bulk term CSV workflow (validate/commit/download) - #1335

Open
aimansharief wants to merge 57 commits into
v1.1.2from
feat/competency-framework
Open

aimansharief wants to merge 57 commits into
v1.1.2from
feat/competency-framework

Conversation

@aimansharief

@aimansharief aimansharief commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Bulk term workflow (CSV): bulk/validate → bulk/commit diffs an uploaded CSV against the current graph state and writes it, attempting every create row regardless of earlier failures (a row that already succeeded stays persisted even if the overall commit is reported failed) — new rows land in Live status, updates/retires apply immediately. bulk/download (GET) rebuilds the current term set as CSV on demand, with category-name placeholder rows when a framework has zero terms yet.
  • Framework review cycle: adds the review/reject actions and their FrameworkManager support, so a framework can move Draft/Live → Review → Draft (reject) or Live (publish).
  • API cleanup: removes duplicate single-row endpoints under /competency-framework/v3/... (term create/update/retire, category CRUD, bulk/create) now that the bulk CSV flow covers them — no functionality lost, no other route affected.
  • CSV robustness: hardens the reader against real Excel/Google Sheets export divergence — UTF-8 BOM stripping, wrong-delimiter detection, non-UTF-8 encoding rejection, and date-shaped code mangling detection.

No new external dependencies — CSV parsing reuses commons-csv, already transitively resolved via actor-core → platform-common (same path collection-csv-actors uses); cloud storage reuses the existing StorageService.

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • Unit tests (ScalaTest + ScalaMock) for the taxonomy-api actors/managers: mvn test -pl taxonomy-api

Test Configuration:

  • Software versions: Java 11, Scala 2.13, Play 3.0.5
  • Hardware versions: 2 CPU / 4GB RAM

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@coderabbitai

coderabbitai Bot commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ea9c98ba-502c-42a1-91e8-ce15ddc78a33

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

aimansharief and others added 6 commits November 4, 2026 12:06
…k schema

The type field was copied verbatim from the Framework schema, carrying
Framework's K-12/TPD enum and a "K-12" default. Those values describe
curriculum taxonomies and mean nothing for a competency framework.

Nothing reads a framework's type: no business logic in taxonomy-api,
search-api or knowledge-platform-jobs branches on it, and NodeUtil's
relation projection keys on an exact "framework" objectType match that
CompetencyFramework never satisfied. CompetencyFramework is already a
distinct object type, so objectType does the discriminating that
K-12/TPD did for Framework.

Because BaseSchemaValidator fills defaults before validating, a create
that omitted type was silently stored as "K-12" -- including the create
example in the API reference. Removing it from both properties and
required avoids persisting a meaningless value.

Framework's own schema is untouched.
19 tests were failing on this branch. 15 of them were test-harness
defects, not product bugs; the other 4 need a running Redis.

FrameworkActorTest (14): update/review/reject resolve the request-body
channel to a Channel node before doing anything else. In production that
value arrives from the x-channel-id header via commonHeaders(), but
these tests build a bare Request and bypass the controller, so channel
was "" -- yielding either ERR_INVALID_CHANNEL_ID or an unmatched mock
call. Supply channel and stub the Channel node, as the already-passing
tests in this file do.

The .img test had a second, distinct cause: VersioningNode.getEditableNode
deliberately re-reads ".img" after addNode creates it, so a stub that
always reports not-found breaks that re-read. Make it state-aware.

TermBulkManagerTest (1): a successful commit surfaces only rows carrying
warnings, and the sheet's cm3 create row has no associations and nothing
referencing it, so it gets the documented WARN_ORPHAN_TERM -- behaviour a
sibling test already asserts. Expect that row instead of none.

taxonomy-actors: 189 -> 204 passing; the 4 remaining failures are
CategoryCacheTest/FrameworkCacheTest, which require Redis.
Identifiers are uniquely indexed in JanusGraph, so /framework/v3 and
/competency-framework/v3 resolve the same vertex for a given id. The
controllers differ only in the objectType/schemaName/bodyKey they put in
the request context, and that context drives validation and
serialization -- never the lookup. Neither endpoint checked what it got
back, so each served and mutated the other's objects.

Reads are gated at the response exit, after the cache, the hierarchy and
the graph converge on one map, so no store can answer without being
type-checked. Writes are gated on the node they load instead: by the
time there is a response the node is already mutated, which makes a
response-level check useless for them. That covers update, retire,
review, reject and publish.

A mismatch returns 404 rather than 400 -- for that endpoint the id
genuinely isn't there. A record carrying no objectType is also rejected:
every writer stamps it, so its absence means the record cannot be
type-verified.

Term APIs are deliberately left alone. They are type-agnostic by design
so competency frameworks can hold terms, and FrameworkTermController
puts objectType=Term in context. Gating the shared loaders
(getLiveEditNode / assertFrameworkEditable) would have compared a
framework node against "Term" and broken every term operation, so the
check lives in the framework-specific operations and the loaders are
untouched. assertFrameworkEditable now returns the node it already
loaded so those callers can assert on it; term callers ignore it.

No database data or structure is changed.

taxonomy-actors: 216 tests, 212 passing; the 4 failures are the
pre-existing CategoryCacheTest/FrameworkCacheTest ones that need Redis.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants