Skip to content

std.base64 fails on unicode strings with codepoints > 255 (emoji, CJK, etc.) #887

Open

Description

@He-Pin

Bug Description

std.base64() fails when the input string contains characters with codepoints greater than 255, such as emoji, CJK characters, and other non-Latin scripts.

Reproduction

std.base64("🎉")

Expected Behavior

"8J+OiQ=="

(This is the correct base64 encoding of the UTF-8 bytes f0 9f 8e 89 for 🎉)

Actual Behavior

RUNTIME ERROR: base64 encountered invalid codepoint value in the array (must be 0 <= X <= 255), got 127881

More Examples

std.base64("你好")    // fails - Chinese characters
std.base64("مرحبا")   // fails - Arabic characters
std.base64("héllo")   // works - all codepoints <= 255
std.base64("hello")   // works - ASCII

Version

$ jsonnet --version
Jsonnet commandline interpreter (Go implementation) v0.22.0

Analysis

The implementation appears to convert the string to an array of Unicode codepoints and then try to treat each codepoint as a byte. This fails for any character with a codepoint > 255.

The correct behavior should be:

  1. Encode the string as UTF-8 bytes
  2. Base64-encode the resulting byte array

sjsonnet handles this correctly:

std.base64("🎉")  // returns "8J+OiQ==" (correct)
std.base64("你好") // returns "5L2g5aW9" (correct)

Verification that the expected output is correct:

echo -n "🎉" | base64
# Output: 8J+OiQ==

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions