Skip to content

fix(common): restore the overflow check in Timestamp::getEpochNanoSeconds - #1584

Merged
adsharma merged 10 commits into
LadybugDB:mainfrom
pratik-saptarshi:fix/timestamp-ns-overflow
Nov 14, 2026
Merged

adsharma merged 10 commits into
LadybugDB:mainfrom
pratik-saptarshi:fix/timestamp-ns-overflow

Conversation

@pratik-saptarshi

@pratik-saptarshi pratik-saptarshi commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1583.

Problem

Casting a timestamp outside the nanosecond-representable range to TIMESTAMP_NS silently wrapped instead of failing:

RETURN CAST(CAST('9999-01-01 00:00:00' AS TIMESTAMP_NS) AS STRING);
-- 1815-03-31 05:56:08.066278

The string, timestamp, and date cast paths to TIMESTAMP_NS route through Timestamp::getEpochNanoSeconds, which multiplies the microsecond value by 1000 via function::Multiply::operation. The overflow-checked int64_t specialization of Multiply took its operands as non-const lvalue references (int64_t&), while the primary template performed unchecked multiplication. When operands were const-qualified or otherwise not bindable as non-const lvalues (e.g. timestamp.value via const timestamp_t&, or constexpr scale constants such as Interval::NANOS_PER_MICRO), overload/template selection fell back to the unchecked primary template, allowing out-of-range products to wrap silently. The same trap was live in Timestamp::fromEpochMilliSeconds / fromEpochSeconds and in Date::getEpochNanoSeconds.

Fix

  • Multiply's checked integral operations now take their input operands by value (with a const-friendly generic template for non-integral cases), so const/constexpr/lvalue/rvalue operands select the checked implementation — defined on top of TryMultiply in multiply.cpp — without relying on caller-side local copies. This is the mechanism that covers every epoch helper above, including Date::getEpochNanoSeconds, which is kept (it is LBUG_API-exported public API) and now routes both of its multiplications through the checked overload instead of a raw wrapping multiply.
  • Strict string casts and try-cast are split. Plain CAST(string AS TIMESTAMP_NS/MS/SEC) uses dedicated TryCastStringToTimestamp::cast<> specializations that bypass tryCast, so an OverflowException from the checked multiply propagates with its specific message instead of collapsing into a generic parse error. The internal try-cast path (tryCast<>) returns false on overflow — catching OverflowException specifically — so try-cast consumers see failure (NULL / unselected field) rather than an exception, matching the engine's try-cast contract.
  • The strict-cast specializations duplicate the parse+convert sequence for the three scaled timestamp types rather than parameterizing tryCast; the ms/sec conversions divide and cannot overflow, so their specializations are behaviourally neutral symmetry with the ns one.
  • One consequence of the split outside the cast expressions themselves: the same strict cast<> specializations serve COPY/CSV string loading, so loading an out-of-range string into a TIMESTAMP_NS column now raises OverflowException instead of the parse ConversionException. Both errors abort the load; the new one names the actual problem.

Tests

  • test/test_files/cast/cast_bounds.test: the timestamp_ns case pins the strict side — overflow on the string, timestamp, and date cast paths to TIMESTAMP_NS in the positive direction and on the timestamp path in the negative direction, with in-range controls. A new timestamp_ns_trycast case pins the try-cast side: this engine has no TRY_CAST SQL syntax, so the pin goes through string→UNION casts, whose field selection consumes tryCast — an overflowing value falls through to a STRING union field instead of raising, an in-range value selects the TIMESTAMP_NS field, and a single-field union reports the conversion failure rather than the overflow error.
  • test/test_files/function/timestamp.test: EpochMsOverflow cases pin epoch_ms at both int64 extremes.
  • test/common/timestamp_test.cpp: a new TimestampTests.FromEpochSecondsOverflow unit case pins Timestamp::fromEpochSeconds at both int64 extremes with the exact checked-multiply message (Value … * 1000000 is not within INT64 range.). It is a unit case because no SQL surface can construct a TIMESTAMP_SEC value large enough to overflow that multiply: every SQL construction divides an int64 microsecond timestamp by MICROS_PER_SEC first, which bounds the seconds value below the overflow threshold.
  • Executed locally on the final branch tree (a68595c2…; the authorship rewrite described below is tree-identical, so these results are for the tree being pushed), incremental release build: cast~cast_bounds.* 9/9, TimestampTests.* 5/5 (including the new FromEpochSecondsOverflow), function~timestamp.FunctionTimestamp PASS, and the cast/timestamp sweep (cast~*, exceptions~types~timestamp.*, copy~copy_timestamp.*, reader~parquet~timestamp.*) 46/46. The full test suite was not run locally. Earlier, at the pre-remediation head d6027ca9 (a different, earlier tree), the same sweep passed 45/45 locally and the minimal test (linux) CI job succeeded; that CI run's overall conclusion was cancelled because its extension-test job was cancelled when the branch head moved, and no CI has run on the final tree yet — it will run on this push.

Review notes

  • Authorship: all commits on this branch have been reset to the submitting account (pratik-saptarshi), as requested in review.
  • The mechanism description above corrects the original PR text: the fix works through checked by-value Multiply overloads, not through callers passing by-value arguments to the old reference-based specialization.

— Musebot-Astro

Comment thread test/test_files/cast/cast_bounds.test Resolved
Comment thread src/common/types/timestamp_t.cpp Resolved
Comment thread src/common/types/timestamp_t.cpp Resolved
pratik-saptarshi pushed a commit to pratik-saptarshi/ladybug that referenced this pull request Nov 13, 2026
…h helpers

Address review feedback on PR LadybugDB#1584:
- Make Multiply checked integral operations accept inputs by value/const so
  constexpr/const operands (e.g. Interval scale constants) select the checked
  implementation instead of the unchecked primary template.
- Simplify Timestamp::getEpochNanoSeconds to rely on the safe Multiply API.
- Route Date::getEpochNanoSeconds through checked Multiply operations to avoid
  raw wrapping multiplication.
- Make string-to-timestamp tryCast paths return false on overflow instead of
  propagating exceptions, preserving try-cast failure semantics.

Note: build/tests not executed in this environment (cmake/ninja unavailable).
adsharma pushed a commit to pratik-saptarshi/ladybug that referenced this pull request Nov 13, 2026
…h helpers

Address review feedback on PR LadybugDB#1584:
- Make Multiply checked integral operations accept inputs by value/const so
  constexpr/const operands (e.g. Interval scale constants) select the checked
  implementation instead of the unchecked primary template.
- Simplify Timestamp::getEpochNanoSeconds to rely on the safe Multiply API.
- Route Date::getEpochNanoSeconds through checked Multiply operations to avoid
  raw wrapping multiplication.
- Make string-to-timestamp tryCast paths return false on overflow instead of
  propagating exceptions, preserving try-cast failure semantics.

Note: build/tests not executed in this environment (cmake/ninja unavailable).
@adsharma
adsharma force-pushed the fix/timestamp-ns-overflow branch from c3a1f1c to a38e3f6 Compare November 13, 2026 18:38
@adsharma

Copy link
Copy Markdown
Contributor

@pratik-saptarshi Could you look into the test failure? Also, you probably want to --reset-author on the commits to yourself instead of the bot so attribution is clear.

@pratik-saptarshi

Copy link
Copy Markdown
Contributor Author

@pratik-saptarshi Could you look into the test failure? Also, you probably want to --reset-author on the commits to yourself instead of the bot so attribution is clear.

I am training an expert system that is adhering to my rules. I have 9 bots, with self-learning loops.

…onds

Casting an out-of-range timestamp to TIMESTAMP_NS silently wrapped:
CAST('9999-01-01 00:00:00' AS TIMESTAMP_NS) produced an instant in 1815
instead of failing. getEpochNanoSeconds multiplies the microsecond value
by 1000 through function::Multiply::operation, intending its
overflow-checked int64_t specialization (which throws OverflowException),
but the parameter is const-qualified, so template argument deduction
selected the primary unchecked template and the product wrapped.

Copy the operands to non-const locals so the checked specialization
applies, matching how fromEpochMilliSeconds/fromEpochSeconds already call
it. All three conversion paths route through this helper (string,
timestamp, and date casts to TIMESTAMP_NS), so all three now throw
'Overflow exception: Value ... * 1000 is not within INT64 range.' for
instants outside the nanosecond-representable range (roughly
1677-09-21 to 2262-04-11); in-range casts are unchanged.

Adds timestamp_ns boundary cases to cast_bounds.test covering the three
cast paths in both directions plus in-range controls.
…h helpers

Address review feedback on PR LadybugDB#1584:
- Make Multiply checked integral operations accept inputs by value/const so
  constexpr/const operands (e.g. Interval scale constants) select the checked
  implementation instead of the unchecked primary template.
- Simplify Timestamp::getEpochNanoSeconds to rely on the safe Multiply API.
- Route Date::getEpochNanoSeconds through checked Multiply operations to avoid
  raw wrapping multiplication.
- Make string-to-timestamp tryCast paths return false on overflow instead of
  propagating exceptions, preserving try-cast failure semantics.

Note: build/tests not executed in this environment (cmake/ninja unavailable).
Keep tryCast returning false on overflow for try-cast paths, but let strict
CAST propagate overflow exceptions instead of converting them to parse errors.
pratik-saptarshi added a commit to pratik-saptarshi/ladybug that referenced this pull request Nov 14, 2026
…h helpers

Address review feedback on PR LadybugDB#1584:
- Make Multiply checked integral operations accept inputs by value/const so
  constexpr/const operands (e.g. Interval scale constants) select the checked
  implementation instead of the unchecked primary template.
- Simplify Timestamp::getEpochNanoSeconds to rely on the safe Multiply API.
- Route Date::getEpochNanoSeconds through checked Multiply operations to avoid
  raw wrapping multiplication.
- Make string-to-timestamp tryCast paths return false on overflow instead of
  propagating exceptions, preserving try-cast failure semantics.

Note: build/tests not executed in this environment (cmake/ninja unavailable).
@pratik-saptarshi
pratik-saptarshi force-pushed the fix/timestamp-ns-overflow branch from d6027ca to 27dcb1c Compare November 14, 2026 04:33
@pratik-saptarshi
pratik-saptarshi force-pushed the fix/timestamp-ns-overflow branch from 27dcb1c to 4149bdc Compare November 14, 2026 06:00
@pratik-saptarshi

Copy link
Copy Markdown
Contributor Author

Authorship: all commits on this branch have been reset to the submitting account (pratik-saptarshi), as requested by the maintainer in review.

What changed in response to the review:

  • The minimal test (linux) failure: the failing case was this branch's own cast_bounds timestamp_ns case. The try-cast hardening had collapsed strict CAST's overflow error into a generic parse error, because strict string casts shared tryCast. Strict string casts to TIMESTAMP_NS/TIMESTAMP_MS/TIMESTAMP_SEC now have dedicated specializations that bypass tryCast and let the OverflowException propagate; the case passes again, and the CI root cause is fixed at the source rather than in the test.
  • Thread 1 (try-cast behaviour): the split is confirmed: strict CAST throws the specific OverflowException, internal try-cast returns false on overflow, and is now pinned on both sides. The try-cast pin goes through string→UNION casts (this engine has no TRY_CAST SQL syntax); see the timestamp_ns_trycast case in test/test_files/cast/cast_bounds.test. The try-cast catches were narrowed to OverflowException specifically.
  • Thread 2 (fromEpochMilliSeconds/fromEpochSeconds): both are covered by the checked by-value Multiply overloads, verified at object-code level. EpochMsOverflow e2e cases pin the millisecond path; a new FromEpochSecondsOverflow unit case pins the seconds path.
  • Thread 3 (Date::getEpochNanoSeconds): both multiplications route through the checked overload. The function is kept, not deleted, because it is LBUG_API-exported public API.

Verification, run locally on the final tree (a68595c2…) with an incremental release build: cast~cast_bounds.* 9/9, TimestampTests.* 5/5, and the cast/timestamp sweep (cast~*, exceptions~types~timestamp.*, copy~copy_timestamp.*, reader~parquet~timestamp.*) 46/46. The full test suite was not run locally.

— Musebot-Astro

}

template<>
void TryCastStringToTimestamp::cast<timestamp_ns_t>(const char* input, uint64_t len,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With IGNORE_ERRORS=true, an out-of-range row now aborts the whole COPY, and the error gives no file or line:

// ts.csv: id,t / 1,2020-01-01 00:00:00 / 2,9999-01-01 00:00:00
COPY T FROM 'ts.csv' (HEADER=true, IGNORE_ERRORS=true);
-- Error: Overflow exception: Value 253370764800000000 * 1000 is not within INT64 range.

The CSV reader only skips rows on ConversionException (csv/driver.cpp:47). That is how 300 in an INT8 column gets skipped with a warning. Could cast<timestamp_ns_t> catch the OverflowException, as tryCast already does, and rethrow a ConversionException naming the value and TIMESTAMP_NS? That matches CAST('300' AS INT8), which raises a conversion error for string input. The string case in cast_bounds would change to that error, and a COPY test with IGNORE_ERRORS=true would pin it. One more thing: the PR body says COPY used to raise a ConversionException. On main the row loaded with the wrapped value.

}

template<>
void TryCastStringToTimestamp::cast<timestamp_ms_t>(const char* input, uint64_t len,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getEpochMilliSeconds and getEpochSeconds divide, so they can't throw. The try/catch in tryCast<timestamp_ms_t>/tryCast<timestamp_sec_t> never fires, and the new cast<timestamp_ms_t>/cast<timestamp_sec_t> specializations behave exactly like the generic cast. The PR body calls them symmetry, but could they go anyway, leaving only the timestamp_ns_t specializations?

@adsharma

Copy link
Copy Markdown
Contributor

Looks good. I'll address the two @zahariash comments in another commit in the PR and merge.

@adsharma
adsharma force-pushed the fix/timestamp-ns-overflow branch from 98e072c to e9c3317 Compare November 14, 2026 21:47
…; drop dead ms/sec specializations

Strict string casts to TIMESTAMP_NS wrapped an out-of-range value in an
OverflowException, which aborts COPY ... (IGNORE_ERRORS=true) for the whole
file because the CSV reader only skips rows on ConversionException. Catch
the overflow from getEpochNanoSeconds and rethrow a ConversionException
naming the value and TIMESTAMP_NS, matching e.g. CAST('300' AS INT8).
tryCast<timestamp_ns_t> already returns failure there.

Drop the dead code flagged in review: getEpochMilliSeconds/getEpochSeconds
only divide and cannot throw, so remove the try/catch from the ms/sec
tryCast specializations and delete the ms/sec strict-cast specializations
(the generic cast<T> behaves identically).

Tests: cast_bounds pins the new Conversion error for the string path
(TIMESTAMP/DATE-intermediate paths still raise OverflowException);
copy_to_csv pins that COPY with IGNORE_ERRORS skips the overflowing row
with a warning instead of aborting.
@adsharma
adsharma force-pushed the fix/timestamp-ns-overflow branch from e9c3317 to 7017f9b Compare November 14, 2026 22:06
@adsharma
adsharma merged commit 43f4af6 into LadybugDB:main Nov 14, 2026
4 checks passed
adsharma pushed a commit that referenced this pull request Nov 14, 2026
…h helpers

Address review feedback on PR #1584:
- Make Multiply checked integral operations accept inputs by value/const so
  constexpr/const operands (e.g. Interval scale constants) select the checked
  implementation instead of the unchecked primary template.
- Simplify Timestamp::getEpochNanoSeconds to rely on the safe Multiply API.
- Route Date::getEpochNanoSeconds through checked Multiply operations to avoid
  raw wrapping multiplication.
- Make string-to-timestamp tryCast paths return false on overflow instead of
  propagating exceptions, preserving try-cast failure semantics.

Note: build/tests not executed in this environment (cmake/ninja unavailable).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TIMESTAMP_NS casts produce wrong instants at large magnitudes (year 9999 renders as 1815)

3 participants