Skip to content

[FEATURE] Prometheus: add Exemplars support - #799

Open
julianocosta89 wants to merge 11 commits into
perses:mainfrom
julianocosta89:exemplars/3445-datasource
Open

julianocosta89 wants to merge 11 commits into
perses:mainfrom
julianocosta89:exemplars/3445-datasource

Conversation

@julianocosta89

@julianocosta89 julianocosta89 commented Sep 7, 2026

Copy link
Copy Markdown

Description

Part of #3445 (perses/perses#3445). Depends on the exemplars contract in perses/spec ().

Summary

Adds exemplars support to the Prometheus datasource and PrometheusTimeSeriesQuery — the data path. Rendering in TimeSeriesChart follows in a follow-up PR stacked on this one.

PrometheusDatasource

  • New optional exemplars: { enable: boolean } section in the datasource spec — off by default, so existing datasources and dashboards are unaffected:
    • prometheus/schemas/datasource/prometheus.cue — CUE schema for the new section
    • prometheus/src/plugins/types.tsPrometheusDatasourceSpec.exemplars?
    • PrometheusDatasourceEditor.tsx — new "Exemplars" section with an "Enable exemplars" Switch (respects readonly mode; the section is removed from the spec when toggled off)
  • New queryExemplars() client method hitting GET /api/v1/query_exemplars?query=...&start=...&end=..., with API types (QueryExemplarsRequestParameters, ExemplarSeries,
    ExemplarData, QueryExemplarsResponse) modeled on the Prometheus docs.
  • The endpoint is allowed through the proxy (GET /api/v1/query_exemplars) and exposed on PrometheusClient via the usual wrapClientMethod.

PrometheusTimeSeriesQuery

  • When the selected datasource has exemplars enabled and the query is in range mode (not instant), a query_exemplars request with the same PromQL + time range is fired in
    parallel
    with the range query.
  • Unit conversion at the datasource boundary: exemplar timestamps (Prometheus seconds) → UnixTimeMs (× 1000), values (Prometheus strings) → numbers.
  • Results attached to TimeSeriesData.exemplars per the spec contract.
  • A failing exemplar request never breaks the panel: it's logged (console.warn) and the series data is returned without exemplars.

Testing

  • Prometheus plugin: 27 tests pass, including mocked query_exemplars client call, exemplar enabled/disabled behavior, exemplar-request failure fallback, and unit conversion.
  • Repo-wide: turbo run type-check / test (30 packages), lint (31), oxfmt clean, make test-schemas-plugins (24 CUE schema tests), make lint-plugins pass.

Follow-up

Stacked on this branch: feat(timeserieschart) PR rendering exemplars as clickable diamond markers (#800).

Screenshots

edit_datasource Screenshot 2026-09-09 at 13 30 08

There are more screenshots available at: #800

Checklist

  • Pull request has a descriptive title and context useful to a reviewer.
  • Pull request title follows the [<catalog_entry>] <commit message> naming convention using one of the
    following catalog_entry values: FEATURE, ENHANCEMENT, BUGFIX, BREAKINGCHANGE, DOC,IGNORE.
  • All commits have DCO signoffs.

UI Changes

  • Changes that impact the UI include screenshots and/or screencasts of the relevant changes.
  • Code follows the UI guidelines.

…t method

Add an optional `exemplars: { enable: boolean }` section to the
PrometheusDatasource spec, exposed in the datasource editor as a
toggle and validated by the CUE schema.

Add a `queryExemplars` method to the Prometheus client hitting
GET /api/v1/query_exemplars with the same query parameters and
authentication handling as the other API calls. The endpoint is also
added to the default proxy allowedEndpoints list.

Related to perses/perses#3445

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
When the selected datasource has exemplars enabled in its spec and the
query runs in range mode, fire a query_exemplars request alongside the
range query (same PromQL and time range, in parallel) and attach the
converted exemplar data to the returned TimeSeriesData.

Exemplar timestamps are converted from seconds to ms and values are
parsed as numbers. A failing exemplar request never breaks the panel:
the error is logged and the query result is returned without exemplars.

Related to perses/perses#3445

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
@julianocosta89
julianocosta89 force-pushed the exemplars/3445-datasource branch from fc0b4a9 to 543b09a Compare September 8, 2026 12:07
@AntoineThebaud

Copy link
Copy Markdown
Contributor

Please add a screenshot to your PR desc 🙏

@julianocosta89

Copy link
Copy Markdown
Author

@AntoineThebaud screenshots added here and in perses/shared#278

@julianocosta89

Copy link
Copy Markdown
Author

This depends on perses/shared#278

Comment thread prometheus/src/plugins/PrometheusDatasourceEditor.tsx
julianocosta89 and others added 2 commits September 10, 2026 11:46
…ditor

Address maintainer feedback from PR perses#799: the Exemplars heading and
toggle now render before HTTPSettingsEditor as part of the General
Settings section.

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
@julianocosta89

Copy link
Copy Markdown
Author

@AntoineThebaud moved up 🙏🏽

image

@AntoineThebaud AntoineThebaud changed the title Exemplars/3445 datasource [FEATURE] Prometheus: add Exemplars support Sep 10, 2026
@AntoineThebaud

Copy link
Copy Markdown
Contributor

I think this new feature is worth mentionning in the functionnal documentation of the plugin, could you please add something there? https://github.com/perses/plugins/blob/main/docs%2Fprometheus%2FREADME.md

Resolve conflict in get-time-series-data.ts: keep upstream's earlier
'isInstant' declaration (moved before the time-range alignment) and drop
the duplicate introduced by the merge; keep the exemplars fetching logic.

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
Add a functional documentation section about the new exemplars support:
opt-in at the datasource level, fetched in parallel of range queries via
/api/v1/query_exemplars and rendered as pin-able diamond markers on the
Time Series Chart.

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
@julianocosta89

Copy link
Copy Markdown
Author

@AntoineThebaud, doc added ✅

@AntoineThebaud

Copy link
Copy Markdown
Contributor

@julianocosta89 the shared package has been released & it includes the exemplar metadata dialog. Actually #817 is upgrading the dep for all plugins so once this one is merged you'll just have to update your branch with the last state of main.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved moderate issues remain in error handling and Go SDK parity, along with a documentation nit.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds opt-in Prometheus exemplar support for datasource configuration and time-series data fetching, with rendering deferred to a follow-up.

Changes:

  • Adds exemplar schema, editor toggle, proxy access, and client API support.
  • Fetches and converts exemplars alongside range queries.
  • Adds tests and documentation.
File summaries
File Summary Final review note
prometheus/src/plugins/types.ts Adds exemplar datasource types. Moderate (2 votes): mirror the field and builder option in the Go SDK.
prometheus/src/plugins/PrometheusDatasourceEditor.tsx Adds exemplar configuration UI.
prometheus/src/plugins/prometheus-time-series-query/plugin.test.ts Tests exemplar behavior and conversion.
prometheus/src/plugins/prometheus-time-series-query/get-time-series-data.ts Fetches and transforms exemplars. Moderate (3 votes): handle rejection immediately. Moderate (1 vote): handle resolved API error responses.
prometheus/src/plugins/prometheus-datasource.tsx Exposes the exemplar client method.
prometheus/src/plugins/prometheus-datasource.test.ts Tests exemplar endpoint requests.
prometheus/src/model/prometheus-client.ts Adds the exemplar API client method.
prometheus/src/model/api-types.ts Defines exemplar API types.
prometheus/schemas/datasource/prometheus.cue Adds exemplar datasource schema. Moderate (1 vote): add the corresponding Go SDK field, option, documentation, and tests.
docs/prometheus/README.md Documents exemplar configuration and behavior. Nit (3 votes): correct the documented marker interaction on lines 90 and 100.
Review details

Suppressed comments (3)

docs/prometheus/README.md:102

  • The README links docs/prometheus/model.md as the technical model documentation, but that datasource example still omits the new exemplars field. Add the field there as well so users consulting the canonical model page can discover this configuration.
  exemplars:
    enable: true
**prometheus/schemas/datasource/prometheus.cue:28**
* This adds `exemplars` to the CUE/TypeScript datasource contract, but the public Go SDK still has no serialized `Exemplars` field in `prometheus/sdk/go/datasource/datasource.go:29-34` and no builder option in `options.go`. Dashboard-as-Code users therefore cannot enable this feature, leaving the datasource representations inconsistent; add the SDK field/option and corresponding docs/tests before publishing the schema change.
exemplars?: {
	enable: bool
}
**prometheus/src/plugins/prometheus-time-series-query/get-time-series-data.ts:155**
* Prometheus can return a query failure as a resolved `status: 'error'` response, since `QueryExemplarsResponse` is an `ApiResponse` union. This code only handles rejected promises, so that failure is silently converted to missing exemplars and no warning is logged, contrary to the documented fallback; check the response status and warn/ignore error responses before attaching the data.
    exemplarResponse = await exemplarPromise;
  } catch (err) {
    console.warn('Failed to fetch exemplars', err);
  }

- **Files reviewed:** 10/10 changed files
- **Comments generated:** 3
- **Review effort level:** Lite
</details>

---

💡 <a href="/perses/plugins/new/main?filename=.github/skills/code-review/SKILL.md" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Add a `code-review` agent skill</a> or configure MCP servers for context-aware, tailored reviews. <a href="https://docs.github.com/copilot/how-tos/use-copilot-agents/request-a-code-review/use-code-review?tool=webui#mcp-servers-and-agent-skills" class="Link--inTextBlock" target="_blank" rel="noopener noreferrer">Learn more in the docs.</a>

Comment thread prometheus/src/plugins/types.ts
Comment thread docs/prometheus/README.md Outdated
…exemplars in the Go SDK

- Attach the rejection handler when the exemplar promise is created so a
  fast-failing /api/v1/query_exemplars request cannot surface as an
  unhandled rejection while the range query is still running
- Ignore exemplar responses that resolve with an error status
- Add the exemplars field to the PrometheusDatasource Go SDK PluginSpec
  along with the EnableExemplars() builder option, test and docs
- Document exemplars in docs/prometheus/model.md and the Go SDK doc

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
Clicking an exemplar marker opens the ExemplarMetadataDialog with the
series labels, exemplar labels, value and timestamp; it does not pin
the chart tooltip. Align the docs with the actual TimeSeriesChart
behavior.

Signed-off-by: Juliano Costa <juliano.costa@datadoghq.com>
@julianocosta89
julianocosta89 force-pushed the exemplars/3445-datasource branch from 4f25939 to 71131a0 Compare September 14, 2026 20:00
@julianocosta89

Copy link
Copy Markdown
Author

Addressed comments from Copilot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants