Triage and debug go.dev/issue/81021.
#81021 (Unicode NFC Hangul)
/home/claude/golang_go_81021/go/src · Tue 25 Aug 2026 08:34:16 · 2 messages499f4eb1-6fd9-4190-be20-22ac8f0f6e51
You Tue 25 Aug 2026 08:08:16
Assistant Tue 25 Aug 2026 08:08:18
I'll start by looking at the issue and the working directory.
Ran tools (49 Bash, 1 ToolSearch, 3 WebFetch, 1 SendUserFile)
Bash ls /home/claude/golang_go_81021 && echo "---" && git log --oneline -3 2>&1 | head
ls /home/claude/golang_go_81021 && echo "---" && git log --oneline -3 2>&1 | head
List working directory contents
env.sh go --- fatal: not a git repository (or any of the parent directories): .git
ToolSearch select:WebFetch
select:WebFetch
3
Bash cat /home/claude/golang_go_81021/env.sh; echo "---"; cd /home/claude/golang_go_81021/go && git log --oneline -3
cat /home/claude/golang_go_81021/env.sh; echo "---"; cd /home/claude/golang_go_81021/go && git log --oneline -3
Inspect env.sh and go repo
export WORK=/home/claude/golang_go_81021 export GOROOT=$WORK/go export GOPATH=$WORK export GOBIN=$WORK/bin export PATH=$GOBIN:/home/claude/.local/bin:/home/claude/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin --- 13b477edc7 runtime: fix flaky refill detection in TestScanAllocIssue77573 72755de33c os, internal/poll: use splice for copies involving pipes 3e6ad2cd0a cmd/compile: always do rewriteCondSelectIntoMath for powers of two
WebFetch https://go.dev/issue/81021
https://go.dev/issue/81021
Reproduce the full issue text: title, body, all comments, labels, status, and any code snippets.
REDIRECT DETECTED: The URL redirects to a location that was not fetched automatically. Original URL: https://go.dev/issue/81021 Redirect URL (from the server's Location header — server-supplied, not verified): https://github.com/golang/go/issues/81021 Status: 302 Found To complete your request, I need to fetch content from the redirected URL. Please use WebFetch again with these parameters: - url: "https://github.com/golang/go/issues/81021" - prompt: "Reproduce the full issue text: title, body, all comments, labels, status, and any code snippets."
Bash gh issue view 81021 --repo golang/go --comments 2>&1 | head -200
gh issue view 81021 --repo golang/go --comments 2>&1 | head -200
Fetch issue via gh
/bin/bash: line 1: gh: command not found
WebFetch https://github.com/golang/go/issues/81021
https://github.com/golang/go/issues/81021
Reproduce the full issue text verbatim: title, body, all comments, labels, status, and any code snippets.
# Issue Summary: NFC Unicode Normalization Bug with Hangul Syllables **Title:** "x/text/unicode/norm: NFC drops a canonical composition when a Hangul syllable precedes it (U+113C5, Unicode 17 tables) #81021" **Status:** Open **Reporter:** tannevaled (opened Aug 22, 2026) ## Problem Description The NFC normalization function in `golang.org/x/text` incorrectly handles the Unicode character U+113C5 (TULU-TIGALARI VOWEL SIGN OO) when preceded by a Hangul syllable. **Expected behavior:** The character should recompose from its canonical decomposition (U+113C2 U+113C2) in all contexts. **Actual behavior:** The composition fails when a Hangul syllable precedes it. For example: - `NFC(U+AC00 U+113C5)` produces `U+AC00 U+113C2 U+113C2` (incorrect) - Instead of producing `U+AC00 U+113C5` (correct) ## Root Cause The issue stems from the `reorderBuffer.compose()` function's Hangul handling path. When `isJamoVT` detects any Jamo character, it routes the entire segment to `combineHangul()`, which processes characters without attempting non-Hangul compositions. This prevents the recomposition of U+113C2 + U+113C2 → U+113C5. ## Context - Affects Go 1.27.0+ (when Unicode 17 tables are selected) - Not caught by existing tests in the normalization test suite - Independent of issue #81001
Bash ls src/vendor/golang.org/x/text/unicode/norm/ 2>/dev/null; echo "---"; find / -path /proc -prune -o -name "composition.g...
ls src/vendor/golang.org/x/text/unicode/norm/ 2>/dev/null; echo "---"; find / -path /proc -prune -o -name "composition.go" -path "*norm*" -print 2>/dev/null
Locate norm package
composition.go forminfo.go input.go iter.go normalize.go readwriter.go tables15.0.0.go tables17.0.0.go transform.go trie.go --- /home/claude/golang_go_80855/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/sdk/go1.26.0/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80855/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.27rc3.linux-amd64/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/sonnet/gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.linux-amd64/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_79754/pkg/mod/golang.org/x/text@v0.40.0/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.27rc3.linux-amd64/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/sonnet/gopath/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.linux-amd64/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_79754/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.7.linux-amd64/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/x/text@v0.40.0/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition.go /home/claude/golang_go_79754/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.7.linux-amd64/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.13.linux-amd64/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.6.linux-amd64/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.7.linux-amd64/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.13.linux-amd64/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.7.linux-amd64/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/sdk/go1.26.0/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80746/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/pkg/mod/golang.org/toolchain@v0.0.1-go1.26.6.linux-amd64/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80746/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_68111/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_68111/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_78576/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_78576/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/go1/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/issue9/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/issue9/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80654/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_78877/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80883/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_78438/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80654/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_79754/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_80883/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_79754/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_81021/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_78438/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_78877/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_33564/go/src/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/golang_go_33564/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /home/claude/go1/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go /usr/share/go-1.23/src/vendor/golang.org/x/text/unicode/norm/composition.go /usr/share/go-1.22/src/vendor/golang.org/x/text/unicode/norm/composition.go /usr/share/go-1.23/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go
Bash cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && ls -la && head -20 tables17.0.0.go 2>/de...
cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && ls -la && head -20 tables17.0.0.go 2>/dev/null; echo "=== cmd/vendor ==="; ls /home/claude/golang_go_81021/go/src/cmd/vendor/golang.org/x/text/unicode/norm/
Inspect vendored norm package
total 864
drwxrwxr-x 2 claude claude 4096 Aug 25 08:05 .
drwxrwxr-x 4 claude claude 4096 Aug 25 08:05 ..
-rw-rw-r-- 1 claude claude 14452 Aug 25 08:05 composition.go
-rw-rw-r-- 1 claude claude 9422 Aug 25 08:05 forminfo.go
-rw-rw-r-- 1 claude claude 2070 Aug 25 08:05 input.go
-rw-rw-r-- 1 claude claude 11219 Aug 25 08:05 iter.go
-rw-rw-r-- 1 claude claude 15293 Aug 25 08:05 normalize.go
-rw-rw-r-- 1 claude claude 2957 Aug 25 08:05 readwriter.go
-rw-rw-r-- 1 claude claude 395028 Aug 25 08:05 tables15.0.0.go
-rw-rw-r-- 1 claude claude 402801 Aug 25 08:05 tables17.0.0.go
-rw-rw-r-- 1 claude claude 2422 Aug 25 08:05 transform.go
-rw-rw-r-- 1 claude claude 1233 Aug 25 08:05 trie.go
// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
//go:build go1.27
package norm
import "sync"
const (
// Version is the Unicode edition from which the tables are derived.
Version = "17.0.0"
// MaxTransformChunkSize indicates the maximum number of bytes that Transform
// may need to write atomically for any Form. Making a destination buffer at
// least this size ensures that Transform can always make progress and that
// the user does not need to grow the buffer on an ErrShortDst.
MaxTransformChunkSize = 35 + maxNonStarters*4
)
var ccc = [56]uint8{
=== cmd/vendor ===
composition.go
forminfo.go
input.go
iter.go
normalize.go
readwriter.go
tables15.0.0.go
tables17.0.0.go
transform.go
trie.goBash cd /home/claude/golang_go_81021 && source env.sh && which go; go version; ls bin 2>/dev/null; ls go/bin 2>/dev/null
cd /home/claude/golang_go_81021 && source env.sh && which go; go version; ls bin 2>/dev/null; ls go/bin 2>/dev/null
Check go toolchain
/usr/bin/go go version go1.22.2 linux/amd64 go gofmt
Bash ./go/bin/go version
./go/bin/go version
Check built GOROOT go version
go version go1.28-devel_13b477edc7 Thu Aug 20 11:16:25 2026 -0400 linux/amd64
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
mkdir -p $S/normtest/norm
cp /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm/*.go $S/normtest/norm/
cd $S/normtest
cat > go.mod <<'EOF'
module normtest
go 1.27
EOF
cat > main.go <<'EOF'
package main
import (
"fmt"
"normtest/norm"
)
func dump(name string, r []rune) {
s := string(r)
out := []rune(norm.NFC.String(s))
fmt.Printf("%s: in=%U -> NFC=%U\n", name, r, out)
}
func main() {
fmt.Println("Version:", norm.Version)
dump("bare", []rune{0x113C2, 0x113C2})
dump("after A", []rune{'A', 0x113C2, 0x113C2})
dump("after hangul syl", []rune{0xAC00, 0x113C2, 0x113C2})
dump("after hangul LV jamo", []rune{0x1100, 0x1161, 0x113C2, 0x113C2})
dump("precomposed after hangul", []rune{0xAC00, 0x113C5})
dump("precomposed bare", []rune{0x113C5})
// NFD then NFC roundtrip
for _, s := range []string{"가\U000113C5", "\U000113C5"} {
d := norm.NFD.String(s)
c := norm.NFC.String(d)
fmt.Printf("roundtrip %U: NFD=%U NFC=%U ok=%v\n", []rune(s), []rune(d), []rune(c), c == s)
}
}
EOF
sed -i 's|^//go:build go1.27$|//go:build go1.27|' norm/tables17.0.0.go
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run . 2>&1 | head -40Reproduce the NFC bug in scratch module
norm/normalize.go:15:2: no required module provides package golang.org/x/text/transform; to add it: go get golang.org/x/text/transform Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest && cp -r /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/transform ./transform && sed -i 's|"golang.org/x/text/transform"|"normtest/transform"|' norm/*.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run . 2>&1 | head -40
Add transform dep and run
Version: 17.0.0 bare: in=[U+113C2 U+113C2] -> NFC=[U+113C5] after A: in=[U+0041 U+113C2 U+113C2] -> NFC=[U+0041 U+113C5] after hangul syl: in=[U+AC00 U+113C2 U+113C2] -> NFC=[U+AC00 U+113C2 U+113C2] after hangul LV jamo: in=[U+1100 U+1161 U+113C2 U+113C2] -> NFC=[U+AC00 U+113C2 U+113C2] precomposed after hangul: in=[U+AC00 U+113C5] -> NFC=[U+AC00 U+113C2 U+113C2] precomposed bare: in=[U+113C5] -> NFC=[U+113C5] roundtrip [U+AC00 U+113C5]: NFD=[U+1100 U+1161 U+113C2 U+113C2] NFC=[U+AC00 U+113C2 U+113C2] ok=false roundtrip [U+113C5]: NFD=[U+113C2 U+113C2] NFC=[U+113C5] ok=true Shell cwd was reset to /home/claude/golang_go_81021
Bash cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && grep -n "isJamoVT\|combineHangul\|func (...
cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && grep -n "isJamoVT\|combineHangul\|func (rb \*reorderBuffer) compose\|Hangul\|hangul" composition.go forminfo.go normalize.go | head -60
Find Hangul handling
forminfo.go:156: // TODO: create the decomposition for Hangul?
composition.go:219: if rune := src.hangul(i); rune != 0 {
composition.go:220: rb.decomposeHangul(rune)
composition.go:235: if rune := src.hangul(i); rune != 0 {
composition.go:236: rb.decomposeHangul(rune)
composition.go:277:// appendRune inserts a rune at the end of the buffer. It is used for Hangul.
composition.go:286:// assignRune sets a rune at position pos. It is used for Hangul and recomposition.
composition.go:293:// runeAt returns the rune at position n. It is used for Hangul and recomposition.
composition.go:301:// It is used for Hangul and recomposition.
composition.go:307:// For Hangul we combine algorithmically, instead of using tables.
composition.go:309: hangulBase = 0xAC00 // UTF-8(hangulBase) -> EA B0 80
composition.go:310: hangulBase0 = 0xEA
composition.go:311: hangulBase1 = 0xB0
composition.go:312: hangulBase2 = 0x80
composition.go:314: hangulEnd = hangulBase + jamoLVTCount // UTF-8(0xD7A4) -> ED 9E A4
composition.go:315: hangulEnd0 = 0xED
composition.go:316: hangulEnd1 = 0x9E
composition.go:317: hangulEnd2 = 0xA4
composition.go:334:const hangulUTF8Size = 3
composition.go:336:func isHangul(b []byte) bool {
composition.go:337: if len(b) < hangulUTF8Size {
composition.go:341: if b0 < hangulBase0 {
composition.go:346: case b0 == hangulBase0:
composition.go:347: return b1 >= hangulBase1
composition.go:348: case b0 < hangulEnd0:
composition.go:350: case b0 > hangulEnd0:
composition.go:352: case b1 < hangulEnd1:
composition.go:355: return b1 == hangulEnd1 && b[2] < hangulEnd2
composition.go:358:func isHangulString(b string) bool {
composition.go:359: if len(b) < hangulUTF8Size {
composition.go:363: if b0 < hangulBase0 {
composition.go:368: case b0 == hangulBase0:
composition.go:369: return b1 >= hangulBase1
composition.go:370: case b0 < hangulEnd0:
composition.go:372: case b0 > hangulEnd0:
composition.go:374: case b1 < hangulEnd1:
composition.go:377: return b1 == hangulEnd1 && b[2] < hangulEnd2
composition.go:381:func isJamoVT(b []byte) bool {
composition.go:386:func isHangulWithoutJamoT(b []byte) bool {
composition.go:388: c -= hangulBase
composition.go:392:// decomposeHangul writes the decomposed Hangul to buf and returns the number
composition.go:394:func decomposeHangul(buf []byte, r rune) int {
composition.go:396: r -= hangulBase
composition.go:408:// decomposeHangul algorithmically decomposes a Hangul rune into
composition.go:410:// See https://unicode.org/reports/tr15/#Hangul for details on decomposing Hangul.
composition.go:411:func (rb *reorderBuffer) decomposeHangul(r rune) {
composition.go:412: r -= hangulBase
composition.go:422:// combineHangul algorithmically combines Jamo character components into Hangul.
composition.go:423:// See https://unicode.org/reports/tr15/#Hangul for details on combining Hangul.
composition.go:424:func (rb *reorderBuffer) combineHangul(s, i, k int) {
composition.go:438: l := rb.runeAt(s) // also used to compare to hangulBase
composition.go:444: rb.assignRune(s, hangulBase+
composition.go:446: case hangulBase <= l && l < hangulEnd &&
composition.go:448: ((l-hangulBase)%jamoTCount) == 0:
composition.go:462:// handle alternations between Hangul and non-Hangul characters correctly.
composition.go:463:func (rb *reorderBuffer) compose() {
composition.go:480: if isJamoVT(rb.bytesAt(i)) {
composition.go:481: // Redo from start in Hangul mode. Necessary to support
composition.go:483: rb.combineHangul(s, i, k)Bash sed -n '300,520p' composition.go
sed -n '300,520p' composition.go
Read compose and Hangul code
// bytesAt returns the UTF-8 encoding of the rune at position n.
// It is used for Hangul and recomposition.
func (rb *reorderBuffer) bytesAt(n int) []byte {
inf := rb.rune[n]
return rb.byte[inf.pos : int(inf.pos)+int(inf.size)]
}
// For Hangul we combine algorithmically, instead of using tables.
const (
hangulBase = 0xAC00 // UTF-8(hangulBase) -> EA B0 80
hangulBase0 = 0xEA
hangulBase1 = 0xB0
hangulBase2 = 0x80
hangulEnd = hangulBase + jamoLVTCount // UTF-8(0xD7A4) -> ED 9E A4
hangulEnd0 = 0xED
hangulEnd1 = 0x9E
hangulEnd2 = 0xA4
jamoLBase = 0x1100 // UTF-8(jamoLBase) -> E1 84 00
jamoLBase0 = 0xE1
jamoLBase1 = 0x84
jamoLEnd = 0x1113
jamoVBase = 0x1161
jamoVEnd = 0x1176
jamoTBase = 0x11A7
jamoTEnd = 0x11C3
jamoTCount = 28
jamoVCount = 21
jamoVTCount = 21 * 28
jamoLVTCount = 19 * 21 * 28
)
const hangulUTF8Size = 3
func isHangul(b []byte) bool {
if len(b) < hangulUTF8Size {
return false
}
b0 := b[0]
if b0 < hangulBase0 {
return false
}
b1 := b[1]
switch {
case b0 == hangulBase0:
return b1 >= hangulBase1
case b0 < hangulEnd0:
return true
case b0 > hangulEnd0:
return false
case b1 < hangulEnd1:
return true
}
return b1 == hangulEnd1 && b[2] < hangulEnd2
}
func isHangulString(b string) bool {
if len(b) < hangulUTF8Size {
return false
}
b0 := b[0]
if b0 < hangulBase0 {
return false
}
b1 := b[1]
switch {
case b0 == hangulBase0:
return b1 >= hangulBase1
case b0 < hangulEnd0:
return true
case b0 > hangulEnd0:
return false
case b1 < hangulEnd1:
return true
}
return b1 == hangulEnd1 && b[2] < hangulEnd2
}
// Caller must ensure len(b) >= 2.
func isJamoVT(b []byte) bool {
// True if (rune & 0xff00) == jamoLBase
return b[0] == jamoLBase0 && (b[1]&0xFC) == jamoLBase1
}
func isHangulWithoutJamoT(b []byte) bool {
c, _ := utf8.DecodeRune(b)
c -= hangulBase
return c < jamoLVTCount && c%jamoTCount == 0
}
// decomposeHangul writes the decomposed Hangul to buf and returns the number
// of bytes written. len(buf) should be at least 9.
func decomposeHangul(buf []byte, r rune) int {
const JamoUTF8Len = 3
r -= hangulBase
x := r % jamoTCount
r /= jamoTCount
utf8.EncodeRune(buf, jamoLBase+r/jamoVCount)
utf8.EncodeRune(buf[JamoUTF8Len:], jamoVBase+r%jamoVCount)
if x != 0 {
utf8.EncodeRune(buf[2*JamoUTF8Len:], jamoTBase+x)
return 3 * JamoUTF8Len
}
return 2 * JamoUTF8Len
}
// decomposeHangul algorithmically decomposes a Hangul rune into
// its Jamo components.
// See https://unicode.org/reports/tr15/#Hangul for details on decomposing Hangul.
func (rb *reorderBuffer) decomposeHangul(r rune) {
r -= hangulBase
x := r % jamoTCount
r /= jamoTCount
rb.appendRune(jamoLBase + r/jamoVCount)
rb.appendRune(jamoVBase + r%jamoVCount)
if x != 0 {
rb.appendRune(jamoTBase + x)
}
}
// combineHangul algorithmically combines Jamo character components into Hangul.
// See https://unicode.org/reports/tr15/#Hangul for details on combining Hangul.
func (rb *reorderBuffer) combineHangul(s, i, k int) {
b := rb.rune[:]
bn := rb.nrune
for ; i < bn; i++ {
cccB := b[k-1].ccc
cccC := b[i].ccc
if cccB == 0 {
s = k - 1
}
if s != k-1 && cccB >= cccC {
// b[i] is blocked by greater-equal cccX below it
b[k] = b[i]
k++
} else {
l := rb.runeAt(s) // also used to compare to hangulBase
v := rb.runeAt(i) // also used to compare to jamoT
switch {
case jamoLBase <= l && l < jamoLEnd &&
jamoVBase <= v && v < jamoVEnd:
// 11xx plus 116x to LV
rb.assignRune(s, hangulBase+
(l-jamoLBase)*jamoVTCount+(v-jamoVBase)*jamoTCount)
case hangulBase <= l && l < hangulEnd &&
jamoTBase < v && v < jamoTEnd &&
((l-hangulBase)%jamoTCount) == 0:
// ACxx plus 11Ax to LVT
rb.assignRune(s, l+v-jamoTBase)
default:
b[k] = b[i]
k++
}
}
}
rb.nrune = k
}
// compose recombines the runes in the buffer.
// It should only be used to recompose a single segment, as it will not
// handle alternations between Hangul and non-Hangul characters correctly.
func (rb *reorderBuffer) compose() {
// Lazily load the map used by the combine func below, but do
// it outside of the loop.
recompMapOnce.Do(buildRecompMap)
// UAX #15, section X5 , including Corrigendum #5
// "In any character sequence beginning with starter S, a character C is
// blocked from S if and only if there is some character B between S
// and C, and either B is a starter or it has the same or higher
// combining class as C."
bn := rb.nrune
if bn == 0 {
return
}
k := 1
b := rb.rune[:]
for s, i := 0, 1; i < bn; i++ {
if isJamoVT(rb.bytesAt(i)) {
// Redo from start in Hangul mode. Necessary to support
// U+320E..U+321E in NFKC mode.
rb.combineHangul(s, i, k)
return
}
ii := b[i]
// We can only use combineForward as a filter if we later
// get the info for the combined character. This is more
// expensive than using the filter. Using combinesBackward()
// is safe.
if ii.combinesBackward() {
cccB := b[k-1].ccc
cccC := ii.ccc
blocked := false // b[i] blocked by starter or greater or equal CCC?
if cccB == 0 {
s = k - 1
} else {
blocked = s != k-1 && cccB >= cccC
}
if !blocked {
combined := combine(rb.runeAt(s), rb.runeAt(i))
if combined != 0 {
rb.assignRune(s, combined)
continue
}
}
}
b[k] = b[i]
k++
}
rb.nrune = k
}Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/dbg_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
)
func TestDbg(t *testing.T) {
recompMapOnce.Do(buildRecompMap)
for _, r := range []rune{0x113C2, 0x113C5, 0x1161, 0xAC00} {
p := NFC.properties([]byte(string(r)))
fmt.Printf("%U ccc=%d combinesBackward=%v combinesForward=%v isInert=%v\n", r, p.ccc, p.combinesBackward(), p.combinesForward(), p.isInert())
}
fmt.Println("recompMap size:", len(recompMap))
// count how many entries whose decomposition breaks after a Hangul syllable
bad := 0
var examples []rune
for key, val := range recompMap {
a := rune(key >> 16)
b := rune(key & 0xffff)
_ = a
_ = b
src := "가" + NFD.String(string(val))
want := "가" + string(val)
if got := NFC.String(src); got != want {
bad++
if len(examples) < 20 {
examples = append(examples, val)
}
}
}
fmt.Printf("Version %s: %d/%d recomposition targets fail after Hangul syllable\n", Version, bad, len(recompMap))
for _, e := range examples {
fmt.Printf(" %U (NFD=%U)\n", e, []rune(NFD.String(string(e))))
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestDbg -v 2>&1 | head -40Instrument rune properties and scan recompMap
# normtest/norm [normtest/norm.test] norm/dbg_test.go:11:12: NFC.properties undefined (type Form has no field or method properties, but does have method Properties) FAIL normtest/norm [build failed] FAIL Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest && sed -i 's/NFC.properties(/NFC.Properties(/' norm/dbg_test.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestDbg -v 2>&1 | head -40
Rerun debug test
=== RUN TestDbg U+113C2 ccc=0 combinesBackward=true combinesForward=true isInert=false U+113C5 ccc=0 combinesBackward=true combinesForward=false isInert=false U+1161 ccc=0 combinesBackward=true combinesForward=true isInert=false U+AC00 ccc=0 combinesBackward=false combinesForward=true isInert=false recompMap size: 961 Version 17.0.0: 12/961 recomposition targets fail after Hangul syllable U+16126 (NFD=[U+1611E U+1611E U+1611F]) U+16121 (NFD=[U+1611E U+1611E]) U+16127 (NFD=[U+1611E U+16129 U+1611F]) U+113C5 (NFD=[U+113C2 U+113C2]) U+16125 (NFD=[U+1611E U+16120]) U+16128 (NFD=[U+1611E U+1611E U+16120]) U+113C8 (NFD=[U+113C2 U+113C9]) U+16123 (NFD=[U+1611E U+1611F]) U+113C7 (NFD=[U+113C2 U+113B8]) U+16122 (NFD=[U+1611E U+16129]) U+16124 (NFD=[U+16129 U+1611F]) U+16D68 (NFD=[U+16D67 U+16D67]) --- PASS: TestDbg (0.00s) PASS ok normtest/norm 0.004s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest && sed -i 's/^go 1.27$/go 1.21/' go.mod && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestDbg -v 2>&1 | head -30; sed -i 's/^go 1.21$/go 1.27/' go.mod
Test with Unicode 15 tables
=== RUN TestDbg U+113C2 ccc=0 combinesBackward=true combinesForward=true isInert=false U+113C5 ccc=0 combinesBackward=true combinesForward=false isInert=false U+1161 ccc=0 combinesBackward=true combinesForward=true isInert=false U+AC00 ccc=0 combinesBackward=false combinesForward=true isInert=false recompMap size: 961 Version 17.0.0: 12/961 recomposition targets fail after Hangul syllable U+16D68 (NFD=[U+16D67 U+16D67]) U+16123 (NFD=[U+1611E U+1611F]) U+16127 (NFD=[U+1611E U+16129 U+1611F]) U+16124 (NFD=[U+16129 U+1611F]) U+113C5 (NFD=[U+113C2 U+113C2]) U+16128 (NFD=[U+1611E U+1611E U+16120]) U+16125 (NFD=[U+1611E U+16120]) U+113C7 (NFD=[U+113C2 U+113B8]) U+113C8 (NFD=[U+113C2 U+113C9]) U+16121 (NFD=[U+1611E U+1611E]) U+16126 (NFD=[U+1611E U+1611E U+1611F]) U+16122 (NFD=[U+1611E U+16129]) --- PASS: TestDbg (0.00s) PASS ok normtest/norm 0.004s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest && cp -r norm norm15 && rm norm15/tables17.0.0.go && sed -i 's|^//go:build !go1.27$|//go:build ignoreme_never|;s|^//go:build !go1\.27|//go:build |' norm15/tables15.0.0.go && head -5 norm15/tables15.0.0.go && sed -i 's/^package norm$/package norm15/' norm15/*.go && sed -i 's|normtest/norm"|normtest/norm15"|' norm15/*.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm15/ -run TestDbg -v 2>&1 | tail -25
Build a Unicode 15 variant and test
// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT. //go:build ignoreme_never package norm # normtest/norm15 found packages norm15 (composition.go) and norm (normalize.go) in /tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/normtest/norm15 FAIL normtest/norm15 [setup failed] FAIL Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest/norm15 && sed -i '1,10s|^//go:build ignoreme_never$||' tables15.0.0.go && sed -i 's/^package norm$/package norm15/' *.go && head -4 tables15.0.0.go && cd .. && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm15/ -run TestDbg -v 2>&1 | tail -25
Fix build tag and rerun U15 test
// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT. # normtest/norm15 found packages norm15 (composition.go) and norm (normalize.go) in /tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/normtest/norm15 FAIL normtest/norm15 [setup failed] FAIL Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest/norm15 && grep -n "^package" *.go
Check package clauses
dbg_test.go:1:package norm15 forminfo.go:5:package norm15 input.go:5:package norm15 transform.go:5:package norm15 composition.go:5:package norm15 readwriter.go:5:package norm15 iter.go:5:package norm15 trie.go:5:package norm15 normalize.go:10:package norm // import "golang.org/x/text/unicode/norm" tables15.0.0.go:5:package norm15 Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest && sed -i 's|^package norm // import.*|package norm15|' norm15/normalize.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm15/ -run TestDbg -v 2>&1 | tail -25
Run U15 variant test
=== RUN TestDbg U+113C2 ccc=0 combinesBackward=false combinesForward=false isInert=true U+113C5 ccc=0 combinesBackward=false combinesForward=false isInert=true U+1161 ccc=0 combinesBackward=true combinesForward=true isInert=false U+AC00 ccc=0 combinesBackward=false combinesForward=true isInert=false recompMap size: 941 Version 15.0.0: 0/941 recomposition targets fail after Hangul syllable --- PASS: TestDbg (0.00s) PASS ok normtest/norm15 0.002s Shell cwd was reset to /home/claude/golang_go_81021
Bash cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && sed -n '277,300p' composition.go
cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && sed -n '277,300p' composition.go
Read assignRune/runeAt
// appendRune inserts a rune at the end of the buffer. It is used for Hangul.
func (rb *reorderBuffer) appendRune(r rune) {
bn := rb.nbyte
sz := utf8.EncodeRune(rb.byte[bn:], rune(r))
rb.nbyte += utf8.UTFMax
rb.rune[rb.nrune] = Properties{pos: bn, size: uint8(sz)}
rb.nrune++
}
// assignRune sets a rune at position pos. It is used for Hangul and recomposition.
func (rb *reorderBuffer) assignRune(pos int, r rune) {
bn := rb.rune[pos].pos
sz := utf8.EncodeRune(rb.byte[bn:], rune(r))
rb.rune[pos] = Properties{pos: bn, size: uint8(sz)}
}
// runeAt returns the rune at position n. It is used for Hangul and recomposition.
func (rb *reorderBuffer) runeAt(n int) rune {
inf := rb.rune[n]
r, _ := utf8.DecodeRune(rb.byte[inf.pos : inf.pos+inf.size])
return r
}
// bytesAt returns the UTF-8 encoding of the rune at position n.Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest/norm && python3 - <<'PY'
import re
p='composition.go'
s=open(p).read()
old=""" default:
b[k] = b[i]
k++
}"""
new=""" default:
// Not a Hangul composition. The segment may still
// contain regular canonical compositions, such as
// combining marks following a Hangul syllable, so
// fall back to the composition table.
if b[i].combinesBackward() {
if combined := combine(l, v); combined != 0 {
rb.assignRune(s, combined)
continue
}
}
b[k] = b[i]
k++
}"""
assert s.count(old)==1
open(p,'w').write(s.replace(old,new))
PY
cd $S/normtest && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestDbg -v 2>&1 | tail -20 && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run . 2>&1Apply candidate fix and test
=== RUN TestDbg U+113C2 ccc=0 combinesBackward=true combinesForward=true isInert=false U+113C5 ccc=0 combinesBackward=true combinesForward=false isInert=false U+1161 ccc=0 combinesBackward=true combinesForward=true isInert=false U+AC00 ccc=0 combinesBackward=false combinesForward=true isInert=false recompMap size: 961 Version 17.0.0: 0/961 recomposition targets fail after Hangul syllable --- PASS: TestDbg (0.00s) PASS ok normtest/norm 0.002s Version: 17.0.0 bare: in=[U+113C2 U+113C2] -> NFC=[U+113C5] after A: in=[U+0041 U+113C2 U+113C2] -> NFC=[U+0041 U+113C5] after hangul syl: in=[U+AC00 U+113C2 U+113C2] -> NFC=[U+AC00 U+113C5] after hangul LV jamo: in=[U+1100 U+1161 U+113C2 U+113C2] -> NFC=[U+AC00 U+113C5] precomposed after hangul: in=[U+AC00 U+113C5] -> NFC=[U+AC00 U+113C5] precomposed bare: in=[U+113C5] -> NFC=[U+113C5] roundtrip [U+AC00 U+113C5]: NFD=[U+1100 U+1161 U+113C2 U+113C2] NFC=[U+AC00 U+113C5] ok=true roundtrip [U+113C5]: NFD=[U+113C2 U+113C2] NFC=[U+113C5] ok=true Shell cwd was reset to /home/claude/golang_go_81021
Bash ls /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/ && grep -c "combinesBackward" /home/claude/pkg/mod/golan...
ls /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/ && grep -c "combinesBackward" /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition.go; diff /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition.go /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm/composition.go && echo IDENTICAL
Check upstream x/text norm package
composition.go composition_test.go data15.0.0_test.go data17.0.0_test.go example_iter_test.go example_test.go forminfo.go forminfo_test.go input.go iter.go iter_test.go maketables.go normalize.go normalize_test.go readwriter.go readwriter_test.go tables15.0.0.go tables17.0.0.go transform.go transform_test.go triegen.go trie.go ucd_test.go 2 IDENTICAL
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad rm -rf $S/xtext && cp -r /home/claude/pkg/mod/golang.org/x/text@v0.41.0 $S/xtext && chmod -R u+w $S/xtext && cd $S/xtext && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ 2>&1 | tail -20
Run upstream x/text norm tests baseline
ok golang.org/x/text/unicode/norm 2.918s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/xtext && cp $S/normtest/norm/composition.go unicode/norm/composition.go && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ 2>&1 | tail -20 && echo "=== verbose list ===" && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ -v 2>&1 | grep -E "^(---|===|ok|FAIL|PASS|\s+---)" | grep -v "^=== RUN\|^=== PAUSE\|^=== CONT" | head -60
Apply fix to x/text copy and run full norm tests
ok golang.org/x/text/unicode/norm 1.534s
=== verbose list ===
--- PASS: TestFlush (0.00s)
--- PASS: TestInsert (0.00s)
--- PASS: TestDecomposition (0.00s)
--- PASS: TestComposition (0.00s)
--- PASS: TestIterNext (0.03s)
--- PASS: TestIterNext/NFC/0 (0.00s)
--- PASS: TestIterNext/NFC/0/fn (0.00s)
--- PASS: TestIterNext/NFC/0/NFD (0.00s)
--- PASS: TestIterNext/NFC/0/NFKC (0.00s)
--- PASS: TestIterNext/NFC/0/NFKD (0.00s)
--- PASS: TestIterNext/NFC/1 (0.00s)
--- PASS: TestIterNext/NFC/1/fn (0.00s)
--- PASS: TestIterNext/NFC/1/NFD (0.00s)
--- PASS: TestIterNext/NFC/1/NFKC (0.00s)
--- PASS: TestIterNext/NFC/1/NFKD (0.00s)
--- PASS: TestIterNext/NFC/2 (0.00s)
--- PASS: TestIterNext/NFC/2/fn (0.00s)
--- PASS: TestIterNext/NFC/2/NFD (0.00s)
--- PASS: TestIterNext/NFC/2/NFKC (0.00s)
--- PASS: TestIterNext/NFC/2/NFKD (0.00s)
--- PASS: TestIterNext/NFC/3 (0.00s)
--- PASS: TestIterNext/NFC/3/fn (0.00s)
--- PASS: TestIterNext/NFC/3/NFD (0.00s)
--- PASS: TestIterNext/NFC/3/NFKC (0.00s)
--- PASS: TestIterNext/NFC/3/NFKD (0.00s)
--- PASS: TestIterNext/NFC/4 (0.00s)
--- PASS: TestIterNext/NFC/4/fn (0.00s)
--- PASS: TestIterNext/NFC/4/NFD (0.00s)
--- PASS: TestIterNext/NFC/4/NFKC (0.00s)
--- PASS: TestIterNext/NFC/4/NFKD (0.00s)
--- PASS: TestIterNext/NFC/5 (0.00s)
--- PASS: TestIterNext/NFC/5/fn (0.00s)
--- PASS: TestIterNext/NFC/5/NFD (0.00s)
--- PASS: TestIterNext/NFC/5/NFKC (0.00s)
--- PASS: TestIterNext/NFC/5/NFKD (0.00s)
--- PASS: TestIterNext/NFC/6 (0.00s)
--- PASS: TestIterNext/NFC/6/fn (0.00s)
--- PASS: TestIterNext/NFC/6/NFD (0.00s)
--- PASS: TestIterNext/NFC/6/NFKC (0.00s)
--- PASS: TestIterNext/NFC/6/NFKD (0.00s)
--- PASS: TestIterNext/NFC/7 (0.00s)
--- PASS: TestIterNext/NFC/7/fn (0.00s)
--- PASS: TestIterNext/NFC/7/NFD (0.00s)
--- PASS: TestIterNext/NFC/7/NFKC (0.00s)
--- PASS: TestIterNext/NFC/7/NFKD (0.00s)
--- PASS: TestIterNext/NFC/8 (0.00s)
--- PASS: TestIterNext/NFC/8/fn (0.00s)
--- PASS: TestIterNext/NFC/8/NFD (0.00s)
--- PASS: TestIterNext/NFC/8/NFKC (0.00s)
--- PASS: TestIterNext/NFC/8/NFKD (0.00s)
--- PASS: TestIterNext/NFC/9 (0.00s)
--- PASS: TestIterNext/NFC/9/fn (0.00s)
--- PASS: TestIterNext/NFC/9/NFD (0.00s)
--- PASS: TestIterNext/NFC/9/NFKC (0.00s)
--- PASS: TestIterNext/NFC/9/NFKD (0.00s)
--- PASS: TestIterNext/NFC/10 (0.00s)
--- PASS: TestIterNext/NFC/10/fn (0.00s)
--- PASS: TestIterNext/NFC/10/NFD (0.00s)
--- PASS: TestIterNext/NFC/10/NFKC (0.00s)
--- PASS: TestIterNext/NFC/10/NFKD (0.00s)
Shell cwd was reset to /home/claude/golang_go_81021Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/xtext/unicode/norm && grep -n "func Test" *_test.go | head -40; echo "=== ucd ==="; sed -n '1,40p' ucd_test.go; echo "=== conformance ==="; grep -rn "NormalizationTest\|Conformance\|conformance" *_test.go ../../internal/testtext/*.go 2>/dev/null | head -20
Survey norm tests and conformance harness
composition_test.go:46:func TestFlush(t *testing.T) {
composition_test.go:92:func TestInsert(t *testing.T) {
composition_test.go:111:func TestDecomposition(t *testing.T) {
composition_test.go:128:func TestComposition(t *testing.T) {
forminfo_test.go:11:func TestProperties(t *testing.T) {
iter_test.go:32:func TestIterNext(t *testing.T) {
iter_test.go:83:func TestIterSegmentation(t *testing.T) {
readwriter_test.go:29:func TestReader(t *testing.T) {
readwriter_test.go:51:func TestWriter(t *testing.T) {
ucd_test.go:220:func TestCharacterByCharacter(t *testing.T) {
ucd_test.go:246:func TestStandardTests(t *testing.T) {
ucd_test.go:258:func TestPerformance(t *testing.T) {
transform_test.go:14:func TestTransform(t *testing.T) {
transform_test.go:103:func TestTransformNorm(t *testing.T) {
normalize_test.go:155:func TestDecomposeSegment(t *testing.T) {
normalize_test.go:190:func TestFirstBoundary(t *testing.T) {
normalize_test.go:195:func TestNextBoundary(t *testing.T) {
normalize_test.go:312:func TestDecomposeToLastBoundary(t *testing.T) {
normalize_test.go:373:func TestLastBoundary(t *testing.T) {
normalize_test.go:488:func TestSpan(t *testing.T) {
normalize_test.go:566:func TestIsNormal(t *testing.T) {
normalize_test.go:579:func TestIsNormalString(t *testing.T) {
normalize_test.go:903:func TestAppend(t *testing.T) {
normalize_test.go:909:func TestAppendString(t *testing.T) {
normalize_test.go:915:func TestBytes(t *testing.T) {
normalize_test.go:924:func TestString(t *testing.T) {
normalize_test.go:956:func TestLinking(t *testing.T) {
=== ucd ===
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package norm
import (
"bufio"
"bytes"
"fmt"
"regexp"
"runtime"
"strconv"
"strings"
"sync"
"testing"
"time"
"unicode/utf8"
"golang.org/x/text/internal/gen"
"golang.org/x/text/internal/testtext"
)
var once sync.Once
func skipShort(t *testing.T) {
testtext.SkipIfNotLong(t)
once.Do(func() { loadTestData(t) })
}
// This regression test runs the test set in NormalizationTest.txt
// (taken from https://www.unicode.org/Public/<unicode.Version>/ucd/).
//
// NormalizationTest.txt has form:
// @Part0 # Specific cases
// #
// 1E0A;1E0A;0044 0307;1E0A;0044 0307; # (Ḋ; Ḋ; D◌̇; Ḋ; D◌̇; ) LATIN CAPITAL LETTER D WITH DOT ABOVE
// 1E0C;1E0C;0044 0323;1E0C;0044 0323; # (Ḍ; Ḍ; D◌̣; Ḍ; D◌̣; ) LATIN CAPITAL LETTER D WITH DOT BELOW
//
=== conformance ===
ucd_test.go:32:// This regression test runs the test set in NormalizationTest.txt
ucd_test.go:35:// NormalizationTest.txt has form:
ucd_test.go:78:// Holds data from NormalizationTest.txt
ucd_test.go:107:// Load the data form NormalizationTest.txt
ucd_test.go:109: f := gen.OpenUCDFile("NormalizationTest.txt")
ucd_test.go:205:func doConformanceTests(t *testing.T, tc *Test, partn int) {
ucd_test.go:241: doConformanceTests(t, &tests[i], 1)
ucd_test.go:250: doConformanceTests(t, &test, j)
normalize_test.go:1224:// https://unicode.org/Public/UNIDATA/NormalizationTest.txt
Shell cwd was reset to /home/claude/golang_go_81021Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && rm -rf normold && mkdir normold && cp /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm/*.go normold/ && sed -i 's|"golang.org/x/text/transform"|"normtest/transform"|' normold/*.go && sed -i 's|^package norm // import.*|package normold|;s/^package norm$/package normold/' normold/*.go && cat > cmp.go <<'EOF'
//go:build ignore
EOF
rm cmp.go
mkdir -p cmp && cat > cmp/main.go <<'EOF'
package main
import (
"fmt"
"normtest/norm"
"normtest/normold"
)
var prefixes = []string{
"", "A", "Á",
"가", // LV syllable
"각", // LVT syllable
"가", // L V
"각", // L V T
"ᄀ", "ᅡ", "ᆨ",
"㈎", // parenthesized hangul (NFKC)
"힣",
}
type form struct {
name string
n norm.Form
o normold.Form
}
func main() {
forms := []form{
{"NFC", norm.NFC, normold.NFC},
{"NFKC", norm.NFKC, normold.NFKC},
{"NFD", norm.NFD, normold.NFD},
{"NFKD", norm.NFKD, normold.NFKD},
}
diffs, badNew, badOld := 0, 0, 0
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
c := string(r)
for _, p := range prefixes {
for _, f := range forms {
// input in decomposed form
in := p + norm.NFD.String(c)
gn, go_ := f.n.String(in), f.o.String(in)
if gn != go_ {
diffs++
if diffs <= 40 {
fmt.Printf("DIFF %s prefix=%U rune=%U: old=%U new=%U\n", f.name, []rune(p), r, []rune(go_), []rune(gn))
}
}
// invariant: normalizing the decomposed form must equal
// normalizing the original form
ref := p + c
if f.n.String(in) != f.n.String(ref) {
badNew++
if badNew <= 20 {
fmt.Printf("NEW-INCONSISTENT %s prefix=%U rune=%U: fromNFD=%U fromRaw=%U\n", f.name, []rune(p), r, []rune(f.n.String(in)), []rune(f.n.String(ref)))
}
}
if f.o.String(in) != f.o.String(ref) {
badOld++
}
// idempotence
if x := f.n.String(f.n.String(in)); x != gn {
fmt.Printf("NOT-IDEMPOTENT %s prefix=%U rune=%U\n", f.name, []rune(p), r)
}
}
}
}
fmt.Printf("diffs=%d newInconsistent=%d oldInconsistent=%d\n", diffs, badNew, badOld)
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp 2>&1 | tail -60Exhaustive old-vs-new differential test
DIFF NFC prefix=[U+AC00] rune=U+113C5: old=[U+AC00 U+113C2 U+113C2] new=[U+AC00 U+113C5] DIFF NFKC prefix=[U+AC00] rune=U+113C5: old=[U+AC00 U+113C2 U+113C2] new=[U+AC00 U+113C5] DIFF NFC prefix=[U+AC01] rune=U+113C5: old=[U+AC01 U+113C2 U+113C2] new=[U+AC01 U+113C5] DIFF NFKC prefix=[U+AC01] rune=U+113C5: old=[U+AC01 U+113C2 U+113C2] new=[U+AC01 U+113C5] DIFF NFC prefix=[U+1100 U+1161] rune=U+113C5: old=[U+AC00 U+113C2 U+113C2] new=[U+AC00 U+113C5] DIFF NFKC prefix=[U+1100 U+1161] rune=U+113C5: old=[U+AC00 U+113C2 U+113C2] new=[U+AC00 U+113C5] DIFF NFC prefix=[U+1100 U+1161 U+11A8] rune=U+113C5: old=[U+AC01 U+113C2 U+113C2] new=[U+AC01 U+113C5] DIFF NFKC prefix=[U+1100 U+1161 U+11A8] rune=U+113C5: old=[U+AC01 U+113C2 U+113C2] new=[U+AC01 U+113C5] DIFF NFC prefix=[U+D7A3] rune=U+113C5: old=[U+D7A3 U+113C2 U+113C2] new=[U+D7A3 U+113C5] DIFF NFKC prefix=[U+D7A3] rune=U+113C5: old=[U+D7A3 U+113C2 U+113C2] new=[U+D7A3 U+113C5] DIFF NFC prefix=[U+AC00] rune=U+113C7: old=[U+AC00 U+113C2 U+113B8] new=[U+AC00 U+113C7] DIFF NFKC prefix=[U+AC00] rune=U+113C7: old=[U+AC00 U+113C2 U+113B8] new=[U+AC00 U+113C7] DIFF NFC prefix=[U+AC01] rune=U+113C7: old=[U+AC01 U+113C2 U+113B8] new=[U+AC01 U+113C7] DIFF NFKC prefix=[U+AC01] rune=U+113C7: old=[U+AC01 U+113C2 U+113B8] new=[U+AC01 U+113C7] DIFF NFC prefix=[U+1100 U+1161] rune=U+113C7: old=[U+AC00 U+113C2 U+113B8] new=[U+AC00 U+113C7] DIFF NFKC prefix=[U+1100 U+1161] rune=U+113C7: old=[U+AC00 U+113C2 U+113B8] new=[U+AC00 U+113C7] DIFF NFC prefix=[U+1100 U+1161 U+11A8] rune=U+113C7: old=[U+AC01 U+113C2 U+113B8] new=[U+AC01 U+113C7] DIFF NFKC prefix=[U+1100 U+1161 U+11A8] rune=U+113C7: old=[U+AC01 U+113C2 U+113B8] new=[U+AC01 U+113C7] DIFF NFC prefix=[U+D7A3] rune=U+113C7: old=[U+D7A3 U+113C2 U+113B8] new=[U+D7A3 U+113C7] DIFF NFKC prefix=[U+D7A3] rune=U+113C7: old=[U+D7A3 U+113C2 U+113B8] new=[U+D7A3 U+113C7] DIFF NFC prefix=[U+AC00] rune=U+113C8: old=[U+AC00 U+113C2 U+113C9] new=[U+AC00 U+113C8] DIFF NFKC prefix=[U+AC00] rune=U+113C8: old=[U+AC00 U+113C2 U+113C9] new=[U+AC00 U+113C8] DIFF NFC prefix=[U+AC01] rune=U+113C8: old=[U+AC01 U+113C2 U+113C9] new=[U+AC01 U+113C8] DIFF NFKC prefix=[U+AC01] rune=U+113C8: old=[U+AC01 U+113C2 U+113C9] new=[U+AC01 U+113C8] DIFF NFC prefix=[U+1100 U+1161] rune=U+113C8: old=[U+AC00 U+113C2 U+113C9] new=[U+AC00 U+113C8] DIFF NFKC prefix=[U+1100 U+1161] rune=U+113C8: old=[U+AC00 U+113C2 U+113C9] new=[U+AC00 U+113C8] DIFF NFC prefix=[U+1100 U+1161 U+11A8] rune=U+113C8: old=[U+AC01 U+113C2 U+113C9] new=[U+AC01 U+113C8] DIFF NFKC prefix=[U+1100 U+1161 U+11A8] rune=U+113C8: old=[U+AC01 U+113C2 U+113C9] new=[U+AC01 U+113C8] DIFF NFC prefix=[U+D7A3] rune=U+113C8: old=[U+D7A3 U+113C2 U+113C9] new=[U+D7A3 U+113C8] DIFF NFKC prefix=[U+D7A3] rune=U+113C8: old=[U+D7A3 U+113C2 U+113C9] new=[U+D7A3 U+113C8] DIFF NFC prefix=[U+AC00] rune=U+16121: old=[U+AC00 U+1611E U+1611E] new=[U+AC00 U+16121] DIFF NFKC prefix=[U+AC00] rune=U+16121: old=[U+AC00 U+1611E U+1611E] new=[U+AC00 U+16121] DIFF NFC prefix=[U+AC01] rune=U+16121: old=[U+AC01 U+1611E U+1611E] new=[U+AC01 U+16121] DIFF NFKC prefix=[U+AC01] rune=U+16121: old=[U+AC01 U+1611E U+1611E] new=[U+AC01 U+16121] DIFF NFC prefix=[U+1100 U+1161] rune=U+16121: old=[U+AC00 U+1611E U+1611E] new=[U+AC00 U+16121] DIFF NFKC prefix=[U+1100 U+1161] rune=U+16121: old=[U+AC00 U+1611E U+1611E] new=[U+AC00 U+16121] DIFF NFC prefix=[U+1100 U+1161 U+11A8] rune=U+16121: old=[U+AC01 U+1611E U+1611E] new=[U+AC01 U+16121] DIFF NFKC prefix=[U+1100 U+1161 U+11A8] rune=U+16121: old=[U+AC01 U+1611E U+1611E] new=[U+AC01 U+16121] DIFF NFC prefix=[U+D7A3] rune=U+16121: old=[U+D7A3 U+1611E U+1611E] new=[U+D7A3 U+16121] DIFF NFKC prefix=[U+D7A3] rune=U+16121: old=[U+D7A3 U+1611E U+1611E] new=[U+D7A3 U+16121] diffs=120 newInconsistent=0 oldInconsistent=0 Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && mkdir -p cmp2 && cat > cmp2/main.go <<'EOF'
package main
import (
"fmt"
"normtest/norm"
"normtest/normold"
)
func main() {
var set []rune
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
p := norm.NFC.Properties([]byte(string(r)))
if p.CCC() != 0 || p.String() != "" || !p.BoundaryBefore() {
set = append(set, r)
}
}
fmt.Println("interesting runes:", len(set))
prefixes := []string{"", "가", "각", "가", "A"}
diffs := 0
seen := map[string]bool{}
for _, a := range set {
for _, b := range set {
for _, p := range prefixes {
in := p + string(a) + string(b)
for _, f := range []struct {
n norm.Form
o normold.Form
s string
}{{norm.NFC, normold.NFC, "NFC"}, {norm.NFKC, normold.NFKC, "NFKC"}} {
gn, go_ := f.n.String(in), f.o.String(in)
if gn != go_ {
diffs++
key := fmt.Sprintf("%s %U %U -> %U vs %U", f.s, a, b, []rune(go_), []rune(gn))
if !seen[key] && len(seen) < 30 {
seen[key] = true
fmt.Printf("DIFF %s prefix=%U in=%U old=%U new=%U\n", f.s, []rune(p), []rune(in), []rune(go_), []rune(gn))
}
}
// NFC output must itself be stable and canonically minimal:
if x := f.n.String(gn); x != gn {
fmt.Printf("NOT-IDEMPOTENT %s in=%U\n", f.s, []rune(in))
}
}
}
}
}
fmt.Println("total diffs:", diffs)
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp2 2>&1 | tail -45Pairwise differential test over combining/composing runes
# normtest/cmp2 cmp2/main.go:16:24: p.String undefined (type norm.Properties has no field or method String) Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/normtest && sed -i 's|p.CCC() != 0 \|\| p.String() != "" \|\| !p.BoundaryBefore()|p.CCC() != 0 \|\| len(p.Decomposition()) != 0 \|\| !p.BoundaryBefore()|' cmp2/main.go && grep -n "CCC()" cmp2/main.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp2 2>&1 | tail -45
Fix predicate and run pairwise differential test
16: if p.CCC() != 0 || len(p.Decomposition()) != 0 || !p.BoundaryBefore() {
interesting runes: 3126
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+113C5] old=[U+AC00 U+0300 U+113C2 U+113C2] new=[U+AC00 U+0300 U+113C5]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+113C5] old=[U+AC00 U+0300 U+113C2 U+113C2] new=[U+AC00 U+0300 U+113C5]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+113C5] old=[U+AC01 U+0300 U+113C2 U+113C2] new=[U+AC01 U+0300 U+113C5]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+113C5] old=[U+AC01 U+0300 U+113C2 U+113C2] new=[U+AC01 U+0300 U+113C5]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+113C7] old=[U+AC00 U+0300 U+113C2 U+113B8] new=[U+AC00 U+0300 U+113C7]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+113C7] old=[U+AC00 U+0300 U+113C2 U+113B8] new=[U+AC00 U+0300 U+113C7]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+113C7] old=[U+AC01 U+0300 U+113C2 U+113B8] new=[U+AC01 U+0300 U+113C7]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+113C7] old=[U+AC01 U+0300 U+113C2 U+113B8] new=[U+AC01 U+0300 U+113C7]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+113C8] old=[U+AC00 U+0300 U+113C2 U+113C9] new=[U+AC00 U+0300 U+113C8]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+113C8] old=[U+AC00 U+0300 U+113C2 U+113C9] new=[U+AC00 U+0300 U+113C8]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+113C8] old=[U+AC01 U+0300 U+113C2 U+113C9] new=[U+AC01 U+0300 U+113C8]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+113C8] old=[U+AC01 U+0300 U+113C2 U+113C9] new=[U+AC01 U+0300 U+113C8]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+16121] old=[U+AC00 U+0300 U+1611E U+1611E] new=[U+AC00 U+0300 U+16121]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+16121] old=[U+AC00 U+0300 U+1611E U+1611E] new=[U+AC00 U+0300 U+16121]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+16121] old=[U+AC01 U+0300 U+1611E U+1611E] new=[U+AC01 U+0300 U+16121]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+16121] old=[U+AC01 U+0300 U+1611E U+1611E] new=[U+AC01 U+0300 U+16121]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+16122] old=[U+AC00 U+0300 U+1611E U+16129] new=[U+AC00 U+0300 U+16122]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+16122] old=[U+AC00 U+0300 U+1611E U+16129] new=[U+AC00 U+0300 U+16122]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+16122] old=[U+AC01 U+0300 U+1611E U+16129] new=[U+AC01 U+0300 U+16122]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+16122] old=[U+AC01 U+0300 U+1611E U+16129] new=[U+AC01 U+0300 U+16122]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+16123] old=[U+AC00 U+0300 U+1611E U+1611F] new=[U+AC00 U+0300 U+16123]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+16123] old=[U+AC00 U+0300 U+1611E U+1611F] new=[U+AC00 U+0300 U+16123]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+16123] old=[U+AC01 U+0300 U+1611E U+1611F] new=[U+AC01 U+0300 U+16123]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+16123] old=[U+AC01 U+0300 U+1611E U+1611F] new=[U+AC01 U+0300 U+16123]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+16124] old=[U+AC00 U+0300 U+16129 U+1611F] new=[U+AC00 U+0300 U+16124]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+16124] old=[U+AC00 U+0300 U+16129 U+1611F] new=[U+AC00 U+0300 U+16124]
DIFF NFC prefix=[U+AC01] in=[U+AC01 U+0300 U+16124] old=[U+AC01 U+0300 U+16129 U+1611F] new=[U+AC01 U+0300 U+16124]
DIFF NFKC prefix=[U+AC01] in=[U+AC01 U+0300 U+16124] old=[U+AC01 U+0300 U+16129 U+1611F] new=[U+AC01 U+0300 U+16124]
DIFF NFC prefix=[U+AC00] in=[U+AC00 U+0300 U+16125] old=[U+AC00 U+0300 U+1611E U+16120] new=[U+AC00 U+0300 U+16125]
DIFF NFKC prefix=[U+AC00] in=[U+AC00 U+0300 U+16125] old=[U+AC00 U+0300 U+1611E U+16120] new=[U+AC00 U+0300 U+16125]
total diffs: 302022
Shell cwd was reset to /home/claude/golang_go_81021Bash cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && grep -n "recompMap\|func combine" formin...
cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && grep -n "recompMap\|func combine" forminfo.go composition.go normalize.go | head -20; echo ---; grep -n "func combine" -A 12 forminfo.go
Look at combine and recompMap
composition.go:466: recompMapOnce.Do(buildRecompMap)
forminfo.go:195: recompMap = make(map[uint32]rune, len(recompMapPacked)/8)
forminfo.go:197: for i := 0; i < len(recompMapPacked); i += 8 {
forminfo.go:198: copy(buf[:], recompMapPacked[i:i+8])
forminfo.go:201: recompMap[key] = rune(val)
forminfo.go:215:// recompMapOnce.Do(buildRecompMap) sometime before this is called.
forminfo.go:216:func combine(a, b rune) rune {
forminfo.go:218: if recompMap == nil {
forminfo.go:221: return recompMap[key]
---
216:func combine(a, b rune) rune {
217- key := uint32(uint16(a))<<16 + uint32(uint16(b))
218- if recompMap == nil {
219- panic("caller error") // see func comment
220- }
221- return recompMap[key]
222-}
223-
224-func lookupInfoNFC(b input, i int) Properties {
225- v, sz := b.charinfoNFC(i)
226- return compInfo(v, sz)
227-}
228-Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/dbg2_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
)
func TestRecompCollisions(t *testing.T) {
recompMapOnce.Do(buildRecompMap)
fmt.Printf("packed entries=%d map entries=%d\n", len(recompMapPacked)/8, len(recompMap))
// Also: does any composition target exceed BMP in either operand?
supp := 0
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
d := NFD.String(string(r))
if d == string(r) {
continue
}
rs := []rune(d)
if len(rs) >= 2 && (rs[0] > 0xFFFF || rs[1] > 0xFFFF) {
supp++
}
}
fmt.Println("decompositions with supplementary operands:", supp)
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestRecompCollisions -v 2>&1 | headCheck recompMap key collisions
=== RUN TestRecompCollisions packed entries=961 map entries=961 decompositions with supplementary operands: 46 --- PASS: TestRecompCollisions (0.04s) PASS ok normtest/norm 0.038s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/ref_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
"unicode/utf8"
)
// refNFC is a straightforward, table-driven implementation of the
// canonical composition algorithm from UAX #15 (D117), used as an
// oracle for the optimized implementation in this package.
func refNFC(s string) string {
recompMapOnce.Do(buildRecompMap)
return refCompose(NFD.String(s))
}
func refNFKC(s string) string {
recompMapOnce.Do(buildRecompMap)
return refCompose(NFKD.String(s))
}
func refCCC(r rune) uint8 {
return NFC.Properties([]byte(string(r))).CCC()
}
func refHangulCompose(l, c rune) rune {
const (
sbase = 0xAC00
lbase = 0x1100
vbase = 0x1161
tbase = 0x11A7
lcnt = 19
vcnt = 21
tcnt = 28
ncnt = vcnt * tcnt
scnt = lcnt * ncnt
)
if lbase <= l && l < lbase+lcnt && vbase <= c && c < vbase+vcnt {
return sbase + ((l-lbase)*vcnt+(c-vbase))*tcnt
}
if sbase <= l && l < sbase+scnt && (l-sbase)%tcnt == 0 && tbase < c && c < tbase+tcnt {
return l + (c - tbase)
}
return 0
}
func refCompose(d string) string {
r := []rune(d)
if len(r) == 0 {
return d
}
out := make([]rune, 0, len(r))
// index in out of the last starter, or -1
last := -1
for i := 0; i < len(r); i++ {
c := r[i]
if last >= 0 {
// blocked if there is a char between the starter and c
// that is a starter or has ccc >= ccc(c)
blocked := false
cccC := refCCC(c)
// chars between out[last] and c are out[last+1:]
if len(out) > last+1 {
prev := refCCC(out[len(out)-1])
if prev == 0 || prev >= cccC {
blocked = true
}
} else if cccC == 0 {
// adjacent starter: allowed only if the pair composes
}
if !blocked {
l := out[last]
var comp rune
if h := refHangulCompose(l, c); h != 0 {
comp = h
} else {
comp = combine(l, c)
}
if comp != 0 {
out[last] = comp
continue
}
}
}
out = append(out, c)
if refCCC(c) == 0 {
last = len(out) - 1
}
}
return string(out)
}
func TestAgainstReference(t *testing.T) {
var interesting []rune
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
p := NFC.Properties([]byte(string(r)))
if p.CCC() != 0 || len(p.Decomposition()) != 0 || !p.BoundaryBefore() {
interesting = append(interesting, r)
}
}
t.Logf("interesting runes: %d", len(interesting))
prefixes := []string{"", "A", "가", "각", "가", "각", "㈀", "㈎"}
bad := 0
buf := make([]byte, 0, 64)
check := func(in string) {
if got, want := NFC.String(in), refNFC(in); got != want {
bad++
if bad <= 20 {
t.Errorf("NFC(%U) = %U, reference %U", []rune(in), []rune(got), []rune(want))
}
}
if got, want := NFKC.String(in), refNFKC(in); got != want {
bad++
if bad <= 20 {
t.Errorf("NFKC(%U) = %U, reference %U", []rune(in), []rune(got), []rune(want))
}
}
}
for _, a := range interesting {
for _, b := range interesting {
for _, p := range prefixes {
buf = buf[:0]
buf = append(buf, p...)
buf = utf8.AppendRune(buf, a)
buf = utf8.AppendRune(buf, b)
check(string(buf))
}
}
}
fmt.Println("mismatches:", bad)
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestAgainstReference -v 2>&1 | tail -30Validate fixed impl against reference NFC/NFKC oracle
Exit code 143 Command timed out after 2m 0s
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && rm norm/ref_test.go && cat > cmp2/main.go <<'EOF'
package main
import (
"fmt"
"normtest/norm"
"normtest/normold"
)
func main() {
var set []rune
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
p := norm.NFC.Properties([]byte(string(r)))
if p.CCC() != 0 || len(p.Decomposition()) != 0 || !p.BoundaryBefore() {
set = append(set, r)
}
}
fmt.Println("interesting runes:", len(set))
prefixes := []string{"", "가", "각", "가", "각", "A", "㈎"}
diffs, notEquiv, notShorter := 0, 0, 0
changed := map[rune]bool{}
for _, a := range set {
for _, b := range set {
for _, p := range prefixes {
in := p + string(a) + string(b)
for _, f := range []struct {
n norm.Form
o normold.Form
s string
}{{norm.NFC, normold.NFC, "NFC"}, {norm.NFKC, normold.NFKC, "NFKC"}} {
gn, go_ := f.n.String(in), f.o.String(in)
if gn == go_ {
continue
}
diffs++
// (a) canonical equivalence preserved
if norm.NFD.String(gn) != norm.NFD.String(go_) {
notEquiv++
if notEquiv <= 10 {
fmt.Printf("NOT-EQUIVALENT %s in=%U old=%U new=%U\n", f.s, []rune(in), []rune(go_), []rune(gn))
}
}
// (b) new output is strictly more composed
if len([]rune(gn)) >= len([]rune(go_)) {
notShorter++
if notShorter <= 10 {
fmt.Printf("NOT-MORE-COMPOSED %s in=%U old=%U new=%U\n", f.s, []rune(in), []rune(go_), []rune(gn))
}
}
for _, r := range gn {
if r != a && r != b {
changed[r] = true
}
}
}
}
}
}
fmt.Printf("diffs=%d notEquiv=%d notShorter=%d\n", diffs, notEquiv, notShorter)
fmt.Print("newly produced runes: ")
for r := range changed {
fmt.Printf("%U ", r)
}
fmt.Println()
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp2 2>&1 | tail -20Classify all behavioral diffs
interesting runes: 3126 diffs=403464 notEquiv=0 notShorter=0 newly produced runes: U+5ED2 U+23D1E U+7DC7 U+6D77 U+03CC U+9F8D U+98EF U+05D2 U+4FBB U+6094 U+34B9 U+5FF5 U+60D8 U+83DC U+8AFE U+03B9 U+2054B U+7FFA U+881F U+9DFA U+6599 U+6DF9 U+7A40 U+2300A U+8D1B U+7409 U+66C6 U+753B U+7D5B U+5922 U+6081 U+681F U+8353 U+518D U+5D6E U+250F3 U+8CAB U+911B U+91B4 U+51F5 U+5D7C U+63E4 U+72C0 U+704A U+4995 U+AC09 U+52B3 U+8AA0 U+0020 U+52F5 U+9756 U+25C80 U+980B U+633D U+AC1B U+038F U+84EE U+6BAE U+8336 U+7949 U+6469 U+264DA U+985E U+7948 U+5145 U+05D4 U+59EC U+70AD U+4AB2 U+516D U+7BC0 U+1D1BA U+6B21 U+0313 U+09A2 U+6FFE U+9234 U+7AEE U+732A U+5716 U+5C22 U+9723 U+9304 U+88E1 U+8667 U+278AE U+AC1A U+0B3C U+7DA0 U+9CFD U+5375 U+8606 U+68A8 U+63C4 U+05DB U+3D96 U+88D7 U+559D U+6148 U+03CE U+7669 U+26D6B U+20804 U+67F3 U+24FB8 U+771F U+83CA U+8AAA U+7469 U+9686 U+5349 U+5962 U+40E3 U+5951 U+52D2 U+0300 U+7E41 U+113C9 U+4FBF U+795D U+5A66 U+5BFF U+25133 U+8AF8 U+2002 U+916A U+9675 U+50DA U+5140 U+7956 U+05D3 U+5246 U+7F85 U+964B U+9DB4 U+8941 U+6700 U+6ADB U+23A8D U+3D33 U+AC19 U+02B9 U+0389 U+8CC2 U+54F6 U+2A20E U+7565 U+722B U+76F4 U+6368 U+26247 U+27CA8 U+8DBC U+9094 U+0301 U+5317 U+6DCB U+67FA U+6A13 U+962E U+53F1 U+7485 U+46C7 U+9E1E U+6E2F U+6200 U+7406 U+7235 U+8323 U+0916 U+091C U+7740 U+440B U+8363 U+8860 U+16125 U+0F90 U+6B79 U+61A4 U+8060 U+113B8 U+0B22 U+523B U+49E6 U+614C U+62D4 U+3FFC U+0388 U+99F1 U+792A U+5C3F U+502B U+7A81 U+8DF0 U+4A76 U+2003 U+73DE U+6E9C U+28D77 U+7210 U+578B U+3AE4 U+7D00 U+273CA U+0917 U+8449 U+649A U+6717 U+9E9F U+7CD6 U+6C4E U+80B2 U+34BB U+113C8 U+9199 U+5002 U+23AFA U+25F86 U+6AD3 U+585A U+77A7 U+22B0C U+05B7 U+5154 U+7145 U+79EB U+79CA U+5B28 U+6756 U+8CC8 U+3B19 U+2921A U+AC02 U+786B U+541D U+96A3 U+96B7 U+7039 U+AC00 U+88C2 U+669C U+0F40 U+70D9 U+8201 U+25AA7 U+8986 U+771E U+5EEC U+674E U+233D5 U+6D34 U+64C4 U+5717 U+6477 U+2569A U+76CA U+05E0 U+83AD U+2A600 U+16121 U+52FA U+5651 U+23F8E U+8650 U+29145 U+81E8 U+4FAE U+6168 U+20A2C U+7F95 U+4EE4 U+84FC U+7891 U+55AB U+5832 U+2597C U+9D67 U+5587 U+5E74 U+7422 U+5197 U+828B U+0F80 U+58F7 U+5195 U+96C3 U+9F16 U+53E5 U+6578 U+85FA U+5272 U+86A9 U+45D7 U+0F5B U+967C U+38C7 U+3F1B U+285ED U+AC11 U+9E97 U+56F9 U+4E26 U+2219F U+92D7 U+91D1 U+91CC U+7CE8 U+8103 U+58D8 U+934A U+9C57 U+43D9 U+745C U+753E U+24FA1 U+5584 U+6753 U+8B01 U+AC08 U+797F U+807E U+8F26 U+79AE U+83BD U+27F2F U+6A02 U+9732 U+78FB U+1D170 U+0915 U+721B U+7C60 U+6674 U+05B4 U+1D172 U+5BE7 U+5674 U+226D4 U+647E U+2A105 U+8AD2 U+4F8B U+16124 U+4ECC U+86E2 U+03AC U+05B8 U+05EA U+82B3 U+8669 U+05DC U+05E7 U+5B08 U+6721 U+23CBC U+908F U+6012 U+9928 U+5099 U+58F2 U+0029 U+9F9C U+62CF U+8D08 U+16127 U+038E U+5EC9 U+7D10 U+8AED U+7CE3 U+AC0F U+85CD U+5207 U+5C64 U+617A U+675E U+7280 U+3EAC U+7489 U+9748 U+4EC0 U+6649 U+83E7 U+0F74 U+6C88 U+65E3 U+719C U+24C36 U+4227 U+5D19 U+3B49 U+6D78 U+62C9 U+57CE U+964D U+52E4 U+57F4 U+6E6E U+2633E U+0F4C U+51AC U+580D U+4096 U+05E9 U+9678 U+5168 U+614E U+219C8 U+8FB0 U+5305 U+55B3 U+43D5 U+540F U+214E4 U+5E28 U+927C U+16128 U+637B U+96F6 U+96E2 U+8B8A U+382F U+6350 U+98FC U+05D9 U+5406 U+4561 U+88FA U+78CA U+798D U+59D8 U+AC15 U+3009 U+58AC U+7524 U+7E02 U+51CC U+5973 U+7C92 U+5EB6 U+621B U+80AD U+865C U+71CE U+9038 U+6383 U+412F U+95B7 U+0A1C U+0FB5 U+05D8 U+54A2 U+24263 U+05C2 U+2063A U+99A7 U+7149 U+5563 U+7A4F U+7559 U+5BF3 U+AC01 U+8ABF U+51A4 U+591A U+96F7 U+8204 U+456B U+788C U+58DF U+8005 U+05E1 U+73A5 U+9EBB U+55E2 U+05D1 U+7A1C U+63C5 U+5C6E U+7F72 U+6EDB U+5D43 U+5FCD U+8779 U+00B4 U+81ED U+5553 U+55C2 U+0FB2 U+00C5 U+8279 U+AC05 U+AC16 U+5373 U+7BC6 U+4301 U+87E1 U+2940A U+7570 U+4EAE U+671B U+9F8E U+5BD8 U+9415 U+0338 U+6F0F U+7DF4 U+05E4 U+216EA U+238A7 U+863F U+62FE U+7B20 U+5EFE U+28BFA U+6DEA U+8352 U+4018 U+5192 U+4202 U+4D56 U+00B7 U+7506 U+6EBA U+761F U+5EB0 U+3EB8 U+7262 U+8612 U+831D U+1D165 U+0A32 U+9B12 U+4E3D U+3781 U+5C60 U+69EA U+250F2 U+5DB2 U+0F42 U+66B4 U+3862 U+6D1B U+5169 U+5599 U+291DF U+81D8 U+24814 U+6DDA U+5FA9 U+6613 U+05C1 U+7D63 U+8AD6 U+83F1 U+677B U+1D157 U+688E U+6914 U+6D6A U+7DBE U+3515 U+793E U+AC04 U+0385 U+8728 U+9238 U+98E2 U+320E U+58B3 U+24C92 U+25044 U+29B30 U+1E61 U+6852 U+6F6E U+8CC1 U+5ED3 U+798E U+0F56 U+0FAB U+0060 U+6ED1 U+7387 U+95AD U+7F79 U+774A U+73B2 U+641C U+5831 U+6B9F U+26523 U+4BCE U+25CD0 U+349E U+393A U+8001 U+654F U+5ED9 U+5D6B U+9C40 U+03AE U+622E U+05DA U+82B1 U+585E U+6691 U+0A3C U+716E U+7C7B U+5BC3 U+649D U+9F3B U+03CD U+6FEB U+5C62 U+96E3 U+5203 U+535A U+61B2 U+3C4E U+24608 U+8786 U+51DE U+537D U+25BAB U+8863 U+8C55 U+9829 U+9929 U+113C5 U+8F62 U+55C0 U+5840 U+72AF U+38E3 U+5D50 U+52A3 U+5167 U+3008 U+843D U+6D16 U+97E0 U+AC14 U+3B9D U+62FC U+7950 U+0F72 U+6CCC U+5954 U+7AB1 U+737A U+16122 U+6CE5 U+793C U+1D16F U+0342 U+90CE U+2335F U+AC0D U+71D0 U+5906 U+216A8 U+3C18 U+63A0 U+798F U+53EB U+AC13 U+8C48 U+5BEE U+6108 U+267B5 U+4F80 U+452B U+9F05 U+806F U+5E69 U+03AF U+5DBA U+AC12 U+5510 U+4008 U+5606 U+9F43 U+34DF U+22183 U+707D U+42A0 U+29496 U+AC0B U+6BBA U+6D1E U+4039 U+7D2F U+05BC U+2541D U+0F51 U+5442 U+6F23 U+507A U+6B72 U+7077 U+61F6 U+22844 U+701B U+876B U+6F22 U+5F6B U+2872E U+092F U+0314 U+88F8 U+7F9A U+5277 U+2A392 U+256C5 U+233C3 U+0308 U+66F8 U+26B3C U+AC0A U+7CBE U+05B9 U+21D0B U+8964 U+808B U+65E2 U+4CED U+0FA1 U+7ACB U+05E8 U+3A2E U+6B54 U+6210 U+7CE7 U+5668 U+5944 U+5F62 U+26228 U+5AB5 U+2339C U+76E7 U+9E7F U+8FB6 U+76DB U+243AB U+45F9 U+09BC U+05D0 U+5F22 U+6D69 U+25626 U+829D U+2A0CE U+AC10 U+AC18 U+5448 U+36EE U+60C7 U+61DE U+8564 U+26F2C U+8F2A U+75E2 U+6885 U+22331 U+8801 U+4635 U+5217 U+8B58 U+50E7 U+6C67 U+27667 U+5B85 U+232B8 U+9F0F U+66F4 U+6D41 U+514D U+8B80 U+05E6 U+5CC0 U+236A3 U+24735 U+16126 U+0A17 U+88CF U+05D6 U+7701 U+9072 U+1D158 U+5F53 U+5E3D U+656C U+7E45 U+958B U+537F U+5FAD U+78CC U+22BF1 U+0F9C U+90DE U+6075 U+5180 U+69A3 U+6D3E U+8F38 U+6CCD U+813E U+52C9 U+03A9 U+97FF U+455D U+270D2 U+618E U+0A16 U+90FD U+05E3 U+7099 U+7965 U+9EFE U+AC06 U+898B U+5B3E U+93F9 U+0386 U+53C3 U+88DE U+16D68 U+09AF U+7070 U+53CA U+7250 U+6817 U+038C U+5ECA U+AC07 U+20B63 U+72FC U+5A62 U+05BF U+6422 U+907C U+242EE U+6234 U+50CF U+826F U+6144 U+5F69 U+1D1B9 U+53DF U+549E U+7CD2 U+8DCB U+092B U+58EE U+03B0 U+862D U+5A1B U+3B08 U+20525 U+58A8 U+5555 U+387C U+5F9A U+8070 U+0921 U+0922 U+7C3E U+5C65 U+7498 U+8F3B U+1D16E U+5468 U+AC17 U+6B04 U+62D3 U+8457 U+05D5 U+5164 U+391C U+4CF8 U+8910 U+6556 U+5A27 U+46BE U+6B77 U+7642 U+7FBD U+6785 U+2A291 U+09A1 U+701E U+83EF U+9276 U+267A7 U+99FE U+03AD U+4046 U+61F2 U+4E41 U+62B1 U+6CBF U+87BA U+4F86 U+6452 U+7471 U+784E U+7A4A U+8ECA U+4E0D U+6881 U+91CF U+601C U+884C U+25249 U+285D2 U+9B6F U+6E1A U+97DB U+8516 U+208DE U+90F1 U+0390 U+7295 U+262D9 U+870E U+AC03 U+0FB7 U+65C5 U+21DE6 U+5351 U+3A6C U+27966 U+16129 U+51B7 U+82E5 U+4E86 U+761D U+27ED3 U+2051C U+74CA U+99C2 U+AC0E U+7E37 U+5438 U+21B18 U+83CC U+0041 U+113C7 U+61AF U+4E82 U+6EC7 U+738B U+8ED4 U+4A6E U+004B U+7D22 U+5F8B U+093C U+6190 U+5229 U+5DE2 U+23F5E U+0028 U+4E39 U+7375 U+7E09 U+4334 U+23393 U+4CCE U+0F92 U+60E1 U+0A38 U+8B39 U+4F60 U+5177 U+7610 U+26CD5 U+4B33 U+AC0C U+9ECE U+4E38 U+26FB1 U+51C9 U+5C06 U+6FC6 U+446B U+52DE U+5F04 U+82E6 U+6160 U+36FC U+261DA U+625D U+25119 U+8688 U+295B6 U+0FB3 U+5948 U+9818 U+5EA6 U+8996 U+05DE U+84F1 U+05F2 U+20122 U+70C8 U+8CD3 U+8ACB U+5DFD U+2346D U+265A8 U+26C36 U+03AB U+2ADD U+54BD U+533F U+52C7 U+1D171 U+21DE4 U+7228 U+9023 U+983B U+7447 U+0A2B U+0B21 U+8046 U+6688 U+523A U+8F9E U+92D8 U+51B5 U+2284A U+5DE1 U+5EB3 U+8291 U+23ED1 U+84F3 U+16123 U+6797 U+8218 U+6ECB U+7F7A U+9EF9 U+9A6A U+96B8 U+5306 U+0F71 U+63A9 U+6AA8 U+6D85 U+8DEF U+51DC U+038A U+5C8D U+5FF9 U+6942 U+8D77 U+529B U+778B U+9111 U+0FA6 U+795E U+7F3E U+5FD7 U+7BC9 U+82BD U+003B U+4E32 U+5289 U+4359 U+6BBB U+83C9 Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/ref_test.go <<'EOF'
package norm
import (
"strings"
"testing"
)
// refNFC/refNFKC are straightforward implementations of the canonical
// composition algorithm of UAX #15 (D117), used as an oracle for the
// optimized implementation in this package.
var cccCache = map[rune]uint8{}
func refCCC(r rune) uint8 {
if c, ok := cccCache[r]; ok {
return c
}
c := NFC.Properties([]byte(string(r))).CCC()
cccCache[r] = c
return c
}
func refHangulCompose(l, c rune) rune {
const (
sbase, lbase, vbase, tbase = 0xAC00, 0x1100, 0x1161, 0x11A7
lcnt, vcnt, tcnt = 19, 21, 28
ncnt = vcnt * tcnt
scnt = lcnt * ncnt
)
if lbase <= l && l < lbase+lcnt && vbase <= c && c < vbase+vcnt {
return sbase + ((l-lbase)*vcnt+(c-vbase))*tcnt
}
if sbase <= l && l < sbase+scnt && (l-sbase)%tcnt == 0 && tbase < c && c < tbase+tcnt {
return l + (c - tbase)
}
return 0
}
func refCompose(d string) string {
recompMapOnce.Do(buildRecompMap)
r := []rune(d)
out := make([]rune, 0, len(r))
last := -1 // index in out of the last starter
for _, c := range r {
if last >= 0 {
cccC := refCCC(c)
blocked := false
if len(out) > last+1 {
prev := refCCC(out[len(out)-1])
blocked = prev == 0 || prev >= cccC
}
if !blocked {
comp := refHangulCompose(out[last], c)
if comp == 0 {
comp = combine(out[last], c)
}
if comp != 0 {
out[last] = comp
continue
}
}
}
out = append(out, c)
if refCCC(c) == 0 {
last = len(out) - 1
}
}
return string(out)
}
func refNFC(s string) string { return refCompose(NFD.String(s)) }
func refNFKC(s string) string { return refCompose(NFKD.String(s)) }
func TestAgainstReference(t *testing.T) {
var all []rune
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
p := NFC.Properties([]byte(string(r)))
if p.CCC() != 0 || len(p.Decomposition()) != 0 || !p.BoundaryBefore() {
all = append(all, r)
}
}
t.Logf("interesting runes: %d", len(all))
// Runes that exercise the Hangul path plus the Unicode 17 starter
// compositions and a few ordinary combining marks.
lead := []rune{
0x1100, 0x1101, 0x1112, 0x1113, // jamo L (and one past the end)
0x1161, 0x1162, 0x1175, 0x1176, // jamo V (and one past the end)
0x11A7, 0x11A8, 0x11C2, 0x11C3, // jamo T (and the boundaries)
0xAC00, 0xAC01, 0xD7A3, 0xD7A4, // Hangul syllables and boundary
0x320E, 0x321E, 0x3200, 0x3260, // circled/parenthesized Hangul (NFKC)
0x0041, 0x00C0, 0x0300, 0x0301, 0x0316, 0x05B0,
0x113C2, 0x113B8, 0x113C5, 0x113C9, 0x1611E, 0x1611F,
0x16120, 0x16121, 0x16129, 0x16D67, 0x16D68,
0x0344, 0x0F73, 0x1E14, 0x212B,
}
prefixes := []string{"", "A", "가", "각", "ᄀ", "가", "̀"}
fail := 0
check := func(in string) {
if got, want := NFC.String(in), refNFC(in); got != want {
fail++
if fail <= 20 {
t.Errorf("NFC(%U) = %U, want %U", []rune(in), []rune(got), []rune(want))
}
}
if got, want := NFKC.String(in), refNFKC(in); got != want {
fail++
if fail <= 20 {
t.Errorf("NFKC(%U) = %U, want %U", []rune(in), []rune(got), []rune(want))
}
}
}
// lead x all interesting runes, under several prefixes
var sb strings.Builder
for _, a := range lead {
for _, b := range all {
for _, p := range prefixes {
sb.Reset()
sb.WriteString(p)
sb.WriteRune(a)
sb.WriteRune(b)
check(sb.String())
}
}
}
// exhaustive triples over the curated set
for _, a := range lead {
for _, b := range lead {
for _, c := range lead {
for _, p := range prefixes {
sb.Reset()
sb.WriteString(p)
sb.WriteRune(a)
sb.WriteRune(b)
sb.WriteRune(c)
check(sb.String())
}
}
}
}
t.Logf("mismatches: %d", fail)
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestAgainstReference -v -timeout 20m 2>&1 | tail -30Run reference-oracle conformance test on fixed impl
=== RUN TestAgainstReference
ref_test.go:85: interesting runes: 3126
ref_test.go:107: NFC([U+0300 U+11F41]) = [U+1F43], want [U+11F41 U+0300]
ref_test.go:113: NFKC([U+0300 U+11F41]) = [U+1F43], want [U+11F41 U+0300]
ref_test.go:107: NFC([U+0300 U+0300 U+11F41]) = [U+1F43 U+0300], want [U+11F41 U+0300 U+0300]
ref_test.go:113: NFKC([U+0300 U+0300 U+11F41]) = [U+1F43 U+0300], want [U+11F41 U+0300 U+0300]
ref_test.go:107: NFC([U+0301 U+11F41]) = [U+1F45], want [U+11F41 U+0301]
ref_test.go:113: NFKC([U+0301 U+11F41]) = [U+1F45], want [U+11F41 U+0301]
ref_test.go:107: NFC([U+0300 U+0301 U+11F41]) = [U+1F43 U+0301], want [U+11F41 U+0300 U+0301]
ref_test.go:113: NFKC([U+0300 U+0301 U+11F41]) = [U+1F43 U+0301], want [U+11F41 U+0300 U+0301]
ref_test.go:107: NFC([U+0300 U+0316 U+11F41]) = [U+1F43 U+0316], want [U+11F41 U+0316 U+0300]
ref_test.go:113: NFKC([U+0300 U+0316 U+11F41]) = [U+1F43 U+0316], want [U+11F41 U+0316 U+0300]
ref_test.go:107: NFC([U+0300 U+05B0 U+11F41]) = [U+1F43 U+05B0], want [U+11F41 U+05B0 U+0300]
ref_test.go:113: NFKC([U+0300 U+05B0 U+11F41]) = [U+1F43 U+05B0], want [U+11F41 U+05B0 U+0300]
ref_test.go:107: NFC([U+0300 U+0344 U+11F41]) = [U+1F43 U+0308 U+0301], want [U+11F41 U+0300 U+0308 U+0301]
ref_test.go:113: NFKC([U+0300 U+0344 U+11F41]) = [U+1F43 U+0308 U+0301], want [U+11F41 U+0300 U+0308 U+0301]
ref_test.go:107: NFC([U+0300 U+0F73 U+11F41]) = [U+1F43 U+0F71 U+0F72], want [U+11F41 U+0F71 U+0F72 U+0300]
ref_test.go:113: NFKC([U+0300 U+0F73 U+11F41]) = [U+1F43 U+0F71 U+0F72], want [U+11F41 U+0F71 U+0F72 U+0300]
ref_test.go:107: NFC([U+0041 U+113C2 U+0300 U+0316]) = [U+00C0 U+113C2 U+0316], want [U+0041 U+113C2 U+0316 U+0300]
ref_test.go:113: NFKC([U+0041 U+113C2 U+0300 U+0316]) = [U+00C0 U+113C2 U+0316], want [U+0041 U+113C2 U+0316 U+0300]
ref_test.go:107: NFC([U+0041 U+113C2 U+0300 U+05B0]) = [U+00C0 U+113C2 U+05B0], want [U+0041 U+113C2 U+05B0 U+0300]
ref_test.go:113: NFKC([U+0041 U+113C2 U+0300 U+05B0]) = [U+00C0 U+113C2 U+05B0], want [U+0041 U+113C2 U+05B0 U+0300]
ref_test.go:145: mismatches: 304
--- FAIL: TestAgainstReference (3.23s)
FAIL
FAIL normtest/norm 3.238s
FAIL
Shell cwd was reset to /home/claude/golang_go_81021Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && mkdir -p cmp3 && cat > cmp3/main.go <<'EOF'
package main
import (
"fmt"
"normtest/norm"
"normtest/normold"
)
func main() {
cases := []string{
"̀\U00011F41",
"́\U00011F41",
"A\U000113C2̖̀",
"A\U000113C2̀",
"À\U000113C2",
}
for _, in := range cases {
fmt.Printf("in=%U\n NFD =%U\n new =%U\n old =%U\n", []rune(in),
[]rune(norm.NFD.String(in)), []rune(norm.NFC.String(in)), []rune(normold.NFC.String(in)))
}
for _, r := range []rune{0x11F41, 0x0316, 0x0300, 0x113C2} {
p := norm.NFC.Properties([]byte(string(r)))
fmt.Printf("%U ccc=%d cb=%v boundaryBefore=%v\n", r, p.CCC(), p.String() == "", p.BoundaryBefore())
}
}
EOF
sed -i 's/p.String() == ""/true/' cmp3/main.go
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp3Compare specific mismatch cases old vs new
in=[U+0300 U+11F41] NFD =[U+11F41 U+0300] new =[U+1F43] old =[U+1F43] in=[U+0301 U+11F41] NFD =[U+11F41 U+0301] new =[U+1F45] old =[U+1F45] in=[U+0041 U+113C2 U+0300 U+0316] NFD =[U+0041 U+113C2 U+0316 U+0300] new =[U+00C0 U+113C2 U+0316] old =[U+00C0 U+113C2 U+0316] in=[U+0041 U+113C2 U+0300] NFD =[U+0041 U+113C2 U+0300] new =[U+0041 U+113C2 U+0300] old =[U+0041 U+113C2 U+0300] in=[U+0041 U+0300 U+113C2] NFD =[U+0041 U+0300 U+113C2] new =[U+00C0 U+113C2] old =[U+00C0 U+113C2] U+11F41 ccc=9 cb=true boundaryBefore=false U+0316 ccc=220 cb=true boundaryBefore=false U+0300 ccc=230 cb=true boundaryBefore=false U+113C2 ccc=0 cb=true boundaryBefore=false Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/dbg3_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
)
func TestTraceCompose(t *testing.T) {
recompMapOnce.Do(buildRecompMap)
fmt.Printf("combine(0x41,0x113C2)=%U\n", combine(0x41, 0x113C2))
fmt.Printf("combine(0x113C2,0x316)=%U\n", combine(0x113C2, 0x316))
fmt.Printf("combine(0x113C2,0x300)=%U\n", combine(0x113C2, 0x300))
fmt.Printf("combine(0x41,0x300)=%U\n", combine(0x41, 0x300))
fmt.Printf("combine(0x11F41,0x300)=%U\n", combine(0x11F41, 0x300))
for _, r := range []rune{0x41, 0x113C2, 0x316, 0x300} {
p := lookupInfoNFC(inputString(string(r)), 0)
fmt.Printf("%U info ccc=%d cb=%v cf=%v\n", r, p.ccc, p.combinesBackward(), p.combinesForward())
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestTraceCompose -v 2>&1 | head -20Trace combine lookups
=== RUN TestTraceCompose combine(0x41,0x113C2)=U+0000 combine(0x113C2,0x316)=U+0000 combine(0x113C2,0x300)=U+0000 combine(0x41,0x300)=U+00C0 combine(0x11F41,0x300)=U+1F43 U+0041 info ccc=0 cb=false cf=true U+113C2 info ccc=0 cb=true cf=true U+0316 info ccc=46 cb=false cf=false U+0300 info ccc=51 cb=true cf=false --- PASS: TestTraceCompose (0.00s) PASS ok normtest/norm 0.005s Shell cwd was reset to /home/claude/golang_go_81021
WebFetch https://github.com/golang/go/issues/81001
https://github.com/golang/go/issues/81001
Reproduce the full issue text verbatim: title, body, all comments, labels, status, and any code snippets.
# Issue Summary: Unicode Normalization Bug in Go 1.27 **Title:** x/text/unicode/norm: NFC/NFKC composes across a ccc=0 character that combines backward (24-33 of 35 affected, not go1.27-specific) #81001 **Status:** Open **Reporter:** tannevaled (opened August 21, 2026) ## Problem Description The `golang.org/x/text` package (v0.41.0) exhibits incorrect Unicode normalization behavior when built with Go 1.27.0, but works correctly with Go 1.26.4. The issue stems from a conditional build that selects Unicode 17.0.0 tables only for Go 1.27+. ## Technical Details When normalizing the input sequence `U+16D68 U+FB01 U+113C2 U+0300 U+0316` using NFKC: **Expected output:** "f096b5a86669f0918f82cc96cc80" (U+16D68 f i U+113C2 U+0316 U+0300) **Actual output (Go 1.27):** "f096b5a866c3acf0918f82cc96" - a codepoint is lost The composition incorrectly occurs across `U+113C2` (which has combining class 0), violating UAX #15 rule D115 that prevents composition when a ccc-0 character intervenes. ## Key Observations - The `PropertiesString` function correctly reports "ccc(U+113C2): 0" - Yet the composition step violates the rule that should apply based on this value - Ruby 4.0.6 (Unicode 17.0.0) produces the expected output - The trigger appears narrow and reproducible ## Workarounds Evaluated (Insufficient) Composing from NFKD or segmenting at ccc-0 boundaries creates other regressions, particularly with Hangul syllable composition.
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/dbg4_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
)
// scan for combine() key collisions: pairs (a,b) for which combine returns a
// rune whose canonical decomposition is not [a,b].
func TestCombineCollisions(t *testing.T) {
recompMapOnce.Do(buildRecompMap)
// collect all second operands appearing in real decompositions
second := map[rune]bool{}
first := map[rune]bool{}
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
d := []rune(NFD.String(string(r)))
if len(d) >= 2 {
first[d[0]] = true
second[d[1]] = true
}
}
fmt.Printf("distinct first operands=%d second operands=%d\n", len(first), len(second))
bogus := 0
var ex []string
for a := rune(0); a <= 0x10FFFF; a++ {
if a >= 0xD800 && a <= 0xDFFF {
continue
}
for b := range second {
c := combine(a, b)
if c == 0 {
continue
}
d := []rune(NFD.String(string(c)))
if !(len(d) == 2 && d[0] == a && d[1] == b) {
bogus++
if len(ex) < 25 {
ex = append(ex, fmt.Sprintf("combine(%U,%U)=%U but NFD(%U)=%U", a, b, c, c, d))
}
}
}
}
fmt.Printf("bogus combine results: %d\n", bogus)
for _, e := range ex {
fmt.Println(" " + e)
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestCombineCollisions -v -timeout 20m 2>&1 | head -40Quantify combine() key collisions
=== RUN TestCombineCollisions distinct first operands=336 second operands=110 bogus combine results: 15632 combine(U+00C2,U+0300)=U+1EA6 but NFD(U+1EA6)=[U+0041 U+0302 U+0300] combine(U+00C2,U+0301)=U+1EA4 but NFD(U+1EA4)=[U+0041 U+0302 U+0301] combine(U+00C2,U+0309)=U+1EA8 but NFD(U+1EA8)=[U+0041 U+0302 U+0309] combine(U+00C2,U+0303)=U+1EAA but NFD(U+1EAA)=[U+0041 U+0302 U+0303] combine(U+00C4,U+0304)=U+01DE but NFD(U+01DE)=[U+0041 U+0308 U+0304] combine(U+00C5,U+0301)=U+01FA but NFD(U+01FA)=[U+0041 U+030A U+0301] combine(U+00C7,U+0301)=U+1E08 but NFD(U+1E08)=[U+0043 U+0327 U+0301] combine(U+00CA,U+0301)=U+1EBE but NFD(U+1EBE)=[U+0045 U+0302 U+0301] combine(U+00CA,U+0309)=U+1EC2 but NFD(U+1EC2)=[U+0045 U+0302 U+0309] combine(U+00CA,U+0303)=U+1EC4 but NFD(U+1EC4)=[U+0045 U+0302 U+0303] combine(U+00CA,U+0300)=U+1EC0 but NFD(U+1EC0)=[U+0045 U+0302 U+0300] combine(U+00CF,U+0301)=U+1E2E but NFD(U+1E2E)=[U+0049 U+0308 U+0301] combine(U+00D4,U+0301)=U+1ED0 but NFD(U+1ED0)=[U+004F U+0302 U+0301] combine(U+00D4,U+0309)=U+1ED4 but NFD(U+1ED4)=[U+004F U+0302 U+0309] combine(U+00D4,U+0303)=U+1ED6 but NFD(U+1ED6)=[U+004F U+0302 U+0303] combine(U+00D4,U+0300)=U+1ED2 but NFD(U+1ED2)=[U+004F U+0302 U+0300] combine(U+00D5,U+0308)=U+1E4E but NFD(U+1E4E)=[U+004F U+0303 U+0308] combine(U+00D5,U+0304)=U+022C but NFD(U+022C)=[U+004F U+0303 U+0304] combine(U+00D5,U+0301)=U+1E4C but NFD(U+1E4C)=[U+004F U+0303 U+0301] combine(U+00D6,U+0304)=U+022A but NFD(U+022A)=[U+004F U+0308 U+0304] combine(U+00DC,U+0300)=U+01DB but NFD(U+01DB)=[U+0055 U+0308 U+0300] combine(U+00DC,U+0304)=U+01D5 but NFD(U+01D5)=[U+0055 U+0308 U+0304] combine(U+00DC,U+0301)=U+01D7 but NFD(U+01D7)=[U+0055 U+0308 U+0301] combine(U+00DC,U+030C)=U+01D9 but NFD(U+01D9)=[U+0055 U+0308 U+030C] combine(U+00E2,U+0303)=U+1EAB but NFD(U+1EAB)=[U+0061 U+0302 U+0303] --- PASS: TestCombineCollisions (1.68s) PASS ok normtest/norm 1.682s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/dbg4_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
)
// oneStepDecomp returns the canonical (single-step) decomposition of r.
func oneStepDecomp(r rune) []rune {
return []rune(string(NFD.Properties([]byte(string(r))).Decomposition()))
}
func TestCombineCollisions(t *testing.T) {
recompMapOnce.Do(buildRecompMap)
second := map[rune]bool{}
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
if d := oneStepDecomp(r); len(d) == 2 {
second[d[1]] = true
}
}
fmt.Printf("distinct second operands: %d\n", len(second))
bogus, bogusSupp := 0, 0
var ex []string
for a := rune(0); a <= 0x10FFFF; a++ {
if a >= 0xD800 && a <= 0xDFFF {
continue
}
for b := range second {
c := combine(a, b)
if c == 0 {
continue
}
d := oneStepDecomp(c)
if len(d) == 2 && d[0] == a && d[1] == b {
continue
}
bogus++
if a > 0xFFFF || b > 0xFFFF {
bogusSupp++
if len(ex) < 30 {
ex = append(ex, fmt.Sprintf("combine(%U,%U)=%U (real decomp of %U is %U)", a, b, c, c, d))
}
}
}
}
fmt.Printf("bogus=%d, of which involving supplementary operands=%d\n", bogus, bogusSupp)
for _, e := range ex {
fmt.Println(" " + e)
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestCombineCollisions -v -timeout 20m 2>&1 | head -45Refine collision scan using single-step decomposition
=== RUN TestCombineCollisions distinct second operands: 89 bogus=15632, of which involving supplementary operands=15378 combine(U+1099,U+110BA)=U+1109A (real decomp of U+1109A is [U+11099 U+110BA]) combine(U+109B,U+110BA)=U+1109C (real decomp of U+1109C is [U+1109B U+110BA]) combine(U+10A5,U+110BA)=U+110AB (real decomp of U+110AB is [U+110A5 U+110BA]) combine(U+1131,U+11127)=U+1112E (real decomp of U+1112E is [U+11131 U+11127]) combine(U+1132,U+11127)=U+1112F (real decomp of U+1112F is [U+11132 U+11127]) combine(U+1347,U+11357)=U+1134C (real decomp of U+1134C is [U+11347 U+11357]) combine(U+1347,U+1133E)=U+1134B (real decomp of U+1134B is [U+11347 U+1133E]) combine(U+1382,U+113C9)=U+11383 (real decomp of U+11383 is [U+11382 U+113C9]) combine(U+1384,U+113BB)=U+11385 (real decomp of U+11385 is [U+11384 U+113BB]) combine(U+138B,U+113C2)=U+1138E (real decomp of U+1138E is [U+1138B U+113C2]) combine(U+1390,U+113C9)=U+11391 (real decomp of U+11391 is [U+11390 U+113C9]) combine(U+13C2,U+113C2)=U+113C5 (real decomp of U+113C5 is [U+113C2 U+113C2]) combine(U+13C2,U+113B8)=U+113C7 (real decomp of U+113C7 is [U+113C2 U+113B8]) combine(U+13C2,U+113C9)=U+113C8 (real decomp of U+113C8 is [U+113C2 U+113C9]) combine(U+14B9,U+114B0)=U+114BC (real decomp of U+114BC is [U+114B9 U+114B0]) combine(U+14B9,U+114BA)=U+114BB (real decomp of U+114BB is [U+114B9 U+114BA]) combine(U+14B9,U+114BD)=U+114BE (real decomp of U+114BE is [U+114B9 U+114BD]) combine(U+15B8,U+115AF)=U+115BA (real decomp of U+115BA is [U+115B8 U+115AF]) combine(U+15B9,U+115AF)=U+115BB (real decomp of U+115BB is [U+115B9 U+115AF]) combine(U+1935,U+11930)=U+11938 (real decomp of U+11938 is [U+11935 U+11930]) combine(U+611E,U+1611E)=U+16121 (real decomp of U+16121 is [U+1611E U+1611E]) combine(U+611E,U+1611F)=U+16123 (real decomp of U+16123 is [U+1611E U+1611F]) combine(U+611E,U+16120)=U+16125 (real decomp of U+16125 is [U+1611E U+16120]) combine(U+611E,U+16129)=U+16122 (real decomp of U+16122 is [U+1611E U+16129]) combine(U+6121,U+1611F)=U+16126 (real decomp of U+16126 is [U+1611E U+1611E U+1611F]) combine(U+6121,U+16120)=U+16128 (real decomp of U+16128 is [U+1611E U+1611E U+16120]) combine(U+6122,U+1611F)=U+16127 (real decomp of U+16127 is [U+1611E U+16129 U+1611F]) combine(U+6129,U+1611F)=U+16124 (real decomp of U+16124 is [U+16129 U+1611F]) combine(U+6D63,U+16D67)=U+16D69 (real decomp of U+16D69 is [U+16D63 U+16D67]) combine(U+6D67,U+16D67)=U+16D68 (real decomp of U+16D68 is [U+16D67 U+16D67]) --- PASS: TestCombineCollisions (1.45s) PASS ok normtest/norm 1.450s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > norm/dbg5_test.go <<'EOF'
package norm
import (
"fmt"
"testing"
"unicode/utf8"
)
// NFC must preserve canonical equivalence: NFD(NFC(x)) == NFD(x).
func TestCanonicalEquivalencePairs(t *testing.T) {
second := map[rune]bool{}
for r := rune(0); r <= 0x10FFFF; r++ {
if r >= 0xD800 && r <= 0xDFFF {
continue
}
if d := []rune(string(NFD.Properties([]byte(string(r))).Decomposition())); len(d) == 2 {
second[d[1]] = true
}
}
var secs []rune
for r := range second {
secs = append(secs, r)
}
bad := 0
var ex []string
buf := make([]byte, 0, 16)
for a := rune(0); a <= 0x10FFFF; a++ {
if a >= 0xD800 && a <= 0xDFFF {
continue
}
for _, b := range secs {
buf = buf[:0]
buf = utf8.AppendRune(buf, a)
buf = utf8.AppendRune(buf, b)
in := string(buf)
got := NFC.String(in)
if NFD.String(got) != NFD.String(in) {
bad++
if len(ex) < 25 {
ex = append(ex, fmt.Sprintf("NFC(%U) = %U — not canonically equivalent", []rune(in), []rune(got)))
}
}
}
}
fmt.Printf("non-equivalent NFC results over 2-rune inputs: %d\n", bad)
for _, e := range ex {
fmt.Println(" " + e)
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm/ -run TestCanonicalEquivalencePairs -v -timeout 20m 2>&1 | head -35End-to-end canonical-equivalence scan over rune pairs
=== RUN TestCanonicalEquivalencePairs non-equivalent NFC results over 2-rune inputs: 15379 NFC([U+05D2 U+0307]) = [U+105C9] — not canonically equivalent NFC([U+05DA U+0307]) = [U+105E4] — not canonically equivalent NFC([U+1099 U+110BA]) = [U+1109A] — not canonically equivalent NFC([U+109B U+110BA]) = [U+1109C] — not canonically equivalent NFC([U+10A5 U+110BA]) = [U+110AB] — not canonically equivalent NFC([U+1131 U+11127]) = [U+1112E] — not canonically equivalent NFC([U+1132 U+11127]) = [U+1112F] — not canonically equivalent NFC([U+1347 U+11357]) = [U+1134C] — not canonically equivalent NFC([U+1347 U+1133E]) = [U+1134B] — not canonically equivalent NFC([U+1382 U+113C9]) = [U+11383] — not canonically equivalent NFC([U+1384 U+113BB]) = [U+11385] — not canonically equivalent NFC([U+138B U+113C2]) = [U+1138E] — not canonically equivalent NFC([U+1390 U+113C9]) = [U+11391] — not canonically equivalent NFC([U+13C2 U+113C9]) = [U+113C8] — not canonically equivalent NFC([U+13C2 U+113C2]) = [U+113C5] — not canonically equivalent NFC([U+13C2 U+113B8]) = [U+113C7] — not canonically equivalent NFC([U+14B9 U+114BD]) = [U+114BE] — not canonically equivalent NFC([U+14B9 U+114BA]) = [U+114BB] — not canonically equivalent NFC([U+14B9 U+114B0]) = [U+114BC] — not canonically equivalent NFC([U+15B8 U+115AF]) = [U+115BA] — not canonically equivalent NFC([U+15B9 U+115AF]) = [U+115BB] — not canonically equivalent NFC([U+1935 U+11930]) = [U+11938] — not canonically equivalent NFC([U+611E U+16129]) = [U+16122] — not canonically equivalent NFC([U+611E U+16120]) = [U+16125] — not canonically equivalent NFC([U+611E U+1611E]) = [U+16121] — not canonically equivalent --- PASS: TestCanonicalEquivalencePairs (43.98s) PASS ok normtest/norm 43.981s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cp norm/dbg5_test.go norm15/ && sed -i 's/^package norm$/package norm15/' norm15/dbg5_test.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./norm15/ -run TestCanonicalEquivalencePairs -v -timeout 20m 2>&1 | head -12
mkdir -p cmp4 && cat > cmp4/main.go <<'EOF'
package main
import (
"fmt"
"normtest/normold"
)
func main() {
fmt.Println("unmodified vendored x/text (Unicode", normold.Version+"):")
for _, in := range []string{"ג̇", "ך̇", "Ꮒ\U000113C2", "̀\U00011F41"} {
fmt.Printf(" NFC(%U) = %U\n", []rune(in), []rune(normold.NFC.String(in)))
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp4Check U15 tables and unmodified code for the aliasing bug
=== RUN TestCanonicalEquivalencePairs non-equivalent NFC results over 2-rune inputs: 15056 NFC([U+1099 U+110BA]) = [U+1109A] — not canonically equivalent NFC([U+109B U+110BA]) = [U+1109C] — not canonically equivalent NFC([U+10A5 U+110BA]) = [U+110AB] — not canonically equivalent NFC([U+1131 U+11127]) = [U+1112E] — not canonically equivalent NFC([U+1132 U+11127]) = [U+1112F] — not canonically equivalent NFC([U+1347 U+11357]) = [U+1134C] — not canonically equivalent NFC([U+1347 U+1133E]) = [U+1134B] — not canonically equivalent NFC([U+14B9 U+114B0]) = [U+114BC] — not canonically equivalent NFC([U+14B9 U+114BD]) = [U+114BE] — not canonically equivalent NFC([U+14B9 U+114BA]) = [U+114BB] — not canonically equivalent unmodified vendored x/text (Unicode 17.0.0): NFC([U+05D2 U+0307]) = [U+105C9] NFC([U+05DA U+0307]) = [U+105E4] NFC([U+13C2 U+113C2]) = [U+113C5] NFC([U+0300 U+11F41]) = [U+1F43] Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cd $S/normtest && cat > cmp4/main.go <<'EOF'
package main
import (
"fmt"
"normtest/norm"
"normtest/normold"
)
func main() {
cases := []string{
"\U00016D68fi\U000113C2̖̀", // issue #81001 repro
"가\U000113C2̖̀", // Hangul analogue
"가\U000113C5", // issue #81021 repro
"가\U000113C2\U000113C2",
}
for _, in := range cases {
fmt.Printf("in =%U\n NFKC old=%U\n NFKC new=%U\n NFC old=%U\n NFC new=%U\n",
[]rune(in),
[]rune(normold.NFKC.String(in)), []rune(norm.NFKC.String(in)),
[]rune(normold.NFC.String(in)), []rune(norm.NFC.String(in)))
}
}
EOF
GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go run ./cmp4Check 81001 repro is unaffected by the 81021 fix
in =[U+16D68 U+FB01 U+113C2 U+0300 U+0316] NFKC old=[U+16D68 U+0066 U+00EC U+113C2 U+0316] NFKC new=[U+16D68 U+0066 U+00EC U+113C2 U+0316] NFC old=[U+16D68 U+FB01 U+113C2 U+0316 U+0300] NFC new=[U+16D68 U+FB01 U+113C2 U+0316 U+0300] in =[U+AC00 U+113C2 U+0300 U+0316] NFKC old=[U+AC00 U+113C2 U+0316 U+0300] NFKC new=[U+AC00 U+113C2 U+0316 U+0300] NFC old=[U+AC00 U+113C2 U+0316 U+0300] NFC new=[U+AC00 U+113C2 U+0316 U+0300] in =[U+AC00 U+113C5] NFKC old=[U+AC00 U+113C2 U+113C2] NFKC new=[U+AC00 U+113C5] NFC old=[U+AC00 U+113C2 U+113C2] NFC new=[U+AC00 U+113C5] in =[U+AC00 U+113C2 U+113C2] NFKC old=[U+AC00 U+113C2 U+113C2] NFKC new=[U+AC00 U+113C5] NFC old=[U+AC00 U+113C2 U+113C2] NFC new=[U+AC00 U+113C5] Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
for d in src/vendor src/cmd/vendor; do
f=/home/claude/golang_go_81021/go/$d/golang.org/x/text/unicode/norm/composition.go
python3 - "$f" <<'PY'
import sys
p=sys.argv[1]
s=open(p).read()
old=""" default:
b[k] = b[i]
k++
}"""
new=""" default:
// Not a Hangul composition, but the segment may
// still hold a regular canonical composition, as
// the characters following a Hangul syllable are
// not necessarily Hangul themselves. Fall back to
// the composition table, as compose does.
if b[i].combinesBackward() {
if combined := combine(l, v); combined != 0 {
rb.assignRune(s, combined)
continue
}
}
b[k] = b[i]
k++
}"""
assert s.count(old)==1, p
open(p,'w').write(s.replace(old,new))
print("patched", p)
PY
done
cd /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm && sed -n '424,470p' composition.goApply fix to both vendored copies in the Go tree
patched /home/claude/golang_go_81021/go/src/vendor/golang.org/x/text/unicode/norm/composition.go
patched /home/claude/golang_go_81021/go/src/cmd/vendor/golang.org/x/text/unicode/norm/composition.go
func (rb *reorderBuffer) combineHangul(s, i, k int) {
b := rb.rune[:]
bn := rb.nrune
for ; i < bn; i++ {
cccB := b[k-1].ccc
cccC := b[i].ccc
if cccB == 0 {
s = k - 1
}
if s != k-1 && cccB >= cccC {
// b[i] is blocked by greater-equal cccX below it
b[k] = b[i]
k++
} else {
l := rb.runeAt(s) // also used to compare to hangulBase
v := rb.runeAt(i) // also used to compare to jamoT
switch {
case jamoLBase <= l && l < jamoLEnd &&
jamoVBase <= v && v < jamoVEnd:
// 11xx plus 116x to LV
rb.assignRune(s, hangulBase+
(l-jamoLBase)*jamoVTCount+(v-jamoVBase)*jamoTCount)
case hangulBase <= l && l < hangulEnd &&
jamoTBase < v && v < jamoTEnd &&
((l-hangulBase)%jamoTCount) == 0:
// ACxx plus 11Ax to LVT
rb.assignRune(s, l+v-jamoTBase)
default:
// Not a Hangul composition, but the segment may
// still hold a regular canonical composition, as
// the characters following a Hangul syllable are
// not necessarily Hangul themselves. Fall back to
// the composition table, as compose does.
if b[i].combinesBackward() {
if combined := combine(l, v); combined != 0 {
rb.assignRune(s, combined)
continue
}
}
b[k] = b[i]
k++
}
}
}
rb.nrune = k
}Bash cd /home/claude/golang_go_81021/go/src && source ../../env.sh && GOTOOLCHAIN=local ../bin/go build std cmd 2>&1 | head -...
cd /home/claude/golang_go_81021/go/src && source ../../env.sh && GOTOOLCHAIN=local ../bin/go build std cmd 2>&1 | head -20 && echo BUILD_OK && GOTOOLCHAIN=local ../bin/go test -count=1 net net/http/httpguts vendor/golang.org/x/net/idna golang.org/x/text/unicode/norm 2>&1 | tail -20
Build std/cmd and run dependent tests
BUILD_OK # net/http/httpguts package net/http/httpguts is not in std (/home/claude/golang_go_81021/go/src/net/http/httpguts) FAIL net/http/httpguts [setup failed] ok net 29.318s ? vendor/golang.org/x/net/idna [no test files] ? golang.org/x/text/unicode/norm [no test files] FAIL
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad sed -n '110,160p' $S/xtext/unicode/norm/composition_test.go; echo "=== normalize_test excerpt ==="; sed -n '1215,1260p' $S/xtext/unicode/norm/normalize_test.go
Look at existing test style
func TestDecomposition(t *testing.T) {
runTests(t, "TestDecompositionNFD", NFD, decompositionNFDTest)
runTests(t, "TestDecompositionNFKD", NFKD, decompositionNFKDTest)
}
var compositionTest = []TestCase{
{[]rune{0x41, 0x300}, []rune{0xC0}},
{[]rune{0x41, 0x316}, []rune{0x41, 0x316}},
{[]rune{0x41, 0x300, 0x35D}, []rune{0xC0, 0x35D}},
{[]rune{0x41, 0x316, 0x300}, []rune{0xC0, 0x316}},
// blocking starter
{[]rune{0x41, 0x316, 0x40, 0x300}, []rune{0x41, 0x316, 0x40, 0x300}},
{[]rune{0x1100, 0x1161}, []rune{0xAC00}},
// parenthesized Hangul, alternate between ASCII and Hangul.
{[]rune{0x28, 0x1100, 0x1161, 0x29}, []rune{0x28, 0xAC00, 0x29}},
}
func TestComposition(t *testing.T) {
runTests(t, "TestComposition", NFC, compositionTest)
}
=== normalize_test excerpt ===
doTextBenchmark(b, txt_cn)
}
func BenchmarkOverflow(b *testing.B) {
doTextBenchmark(b, overflow)
}
var overflow = string(bytes.Repeat([]byte("\u035D"), 4096)) + "\u035B"
// Tests sampled from the Canonical ordering tests (Part 2) of
// https://unicode.org/Public/UNIDATA/NormalizationTest.txt
const txt_canon = `\u0061\u0315\u0300\u05AE\u0300\u0062 \u0061\u0300\u0315\u0300\u05AE\u0062
\u0061\u0302\u0315\u0300\u05AE\u0062 \u0061\u0307\u0315\u0300\u05AE\u0062
\u0061\u0315\u0300\u05AE\u030A\u0062 \u0061\u059A\u0316\u302A\u031C\u0062
\u0061\u032E\u059A\u0316\u302A\u0062 \u0061\u0338\u093C\u0334\u0062
\u0061\u059A\u0316\u302A\u0339 \u0061\u0341\u0315\u0300\u05AE\u0062
\u0061\u0348\u059A\u0316\u302A\u0062 \u0061\u0361\u0345\u035D\u035C\u0062
\u0061\u0366\u0315\u0300\u05AE\u0062 \u0061\u0315\u0300\u05AE\u0486\u0062
\u0061\u05A4\u059A\u0316\u302A\u0062 \u0061\u0315\u0300\u05AE\u0613\u0062
\u0061\u0315\u0300\u05AE\u0615\u0062 \u0061\u0617\u0315\u0300\u05AE\u0062
\u0061\u0619\u0618\u064D\u064E\u0062 \u0061\u0315\u0300\u05AE\u0654\u0062
\u0061\u0315\u0300\u05AE\u06DC\u0062 \u0061\u0733\u0315\u0300\u05AE\u0062
\u0061\u0744\u059A\u0316\u302A\u0062 \u0061\u0315\u0300\u05AE\u0745\u0062
\u0061\u09CD\u05B0\u094D\u3099\u0062 \u0061\u0E38\u0E48\u0E38\u0C56\u0062
\u0061\u0EB8\u0E48\u0E38\u0E49\u0062 \u0061\u0F72\u0F71\u0EC8\u0F71\u0062
\u0061\u1039\u05B0\u094D\u3099\u0062 \u0061\u05B0\u094D\u3099\u1A60\u0062
\u0061\u3099\u093C\u0334\u1BE6\u0062 \u0061\u3099\u093C\u0334\u1C37\u0062
\u0061\u1CD9\u059A\u0316\u302A\u0062 \u0061\u2DED\u0315\u0300\u05AE\u0062
\u0061\u2DEF\u0315\u0300\u05AE\u0062 \u0061\u302D\u302E\u059A\u0316\u0062`
// Taken from http://creativecommons.org/licenses/by-sa/3.0/vn/
const txt_vn = `Với các điều kiện sau: Ghi nhận công của tác giả.
Nếu bạn sử dụng, chuyển đổi, hoặc xây dựng dự án từ
nội dung được chia sẻ này, bạn phải áp dụng giấy phép này hoặc
một giấy phép khác có các điều khoản tương tự như giấy phép này
cho dự án của bạn. Hiểu rằng: Miễn — Bất kỳ các điều kiện nào
trên đây cũng có thể được miễn bỏ nếu bạn được sự cho phép của
người sở hữu bản quyền. Phạm vi công chúng — Khi tác phẩm hoặc
bất kỳ chương nào của tác phẩm đã trong vùng dành cho công
chúng theo quy định của pháp luật thì tình trạng của nó không
bị ảnh hưởng bởi giấy phép trong bất kỳ trường hợp nào.`
// Taken from http://creativecommons.org/licenses/by-sa/1.0/deed.ru
const txt_ru = `При обязательном соблюдении следующих условий:
Attribution — Вы должны атрибутировать произведение (указывать
автора и источник) в порядке, предусмотренном автором или
лицензиаром (но только так, чтобы никоим образом не подразумевалось,Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad head -20 $S/xtext/unicode/norm/data17.0.0_test.go; echo ...; grep -n "^var \|^func \|^)" $S/xtext/unicode/norm/data17.0.0_test.go | head -20; echo "=== runTests ==="; sed -n '1,45p' $S/xtext/unicode/norm/composition_test.go
Inspect version-specific test data files
// Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT.
//go:build go1.27
package norm
const (
Yes = iota
No
Maybe
)
type formData struct {
qc uint8
combinesForward bool
decomposition string
}
type runeData struct {
r rune
...
11:)
27:func f(qc uint8, cf bool, dec string) [2]formData {
31:func g(qc, qck uint8, cf, cfk bool, d, dk string) [2]formData {
35:var testData = []runeData{
=== runTests ===
// Copyright 2011 The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
package norm
import "testing"
// TestCase is used for most tests.
type TestCase struct {
in []rune
out []rune
}
func runTests(t *testing.T, name string, fm Form, tests []TestCase) {
rb := reorderBuffer{}
rb.init(fm, nil)
for i, test := range tests {
rb.setFlusher(nil, appendFlush)
for j, rune := range test.in {
b := []byte(string(rune))
src := inputBytes(b)
info := rb.f.info(src, 0)
if j == 0 {
rb.ss.first(info)
} else {
rb.ss.next(info)
}
if rb.insertFlush(src, 0, info) < 0 {
t.Errorf("%s:%d: insert failed for rune %d", name, i, j)
}
}
rb.doFlush()
was := string(rb.out)
want := string(test.out)
if len(was) != len(want) {
t.Errorf("%s:%d: length = %d; want %d", name, i, len(was), len(want))
}
if was != want {
k, pfx := pidx(was, want)
t.Errorf("%s:%d: \nwas %s%+q; \nwant %s%+q", name, i, pfx, was[k:], pfx, want[k:])
}
}
}Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cat > $S/hangul_test_snippet.go <<'EOF'
// Issue #81021: a Hangul syllable at the start of a segment must not
// disable regular canonical composition for the rest of the segment.
// The reorderBuffer switches to Hangul mode as soon as it sees a Jamo,
// so any composition further along in the same segment used to be lost.
func TestCompositionAfterHangul(t *testing.T) {
prefixes := []string{
"가", // Jamo L V
"각", // Jamo L V T
"가", // Hangul syllable LV
"각", // Hangul syllable LVT
"힣", // last Hangul syllable
}
for _, p := range prefixes {
for r := rune(0); r <= unicode.MaxRune; r++ {
c := string(r)
if !utf8.ValidString(c) || NFC.String(c) != c {
continue // not a composed character
}
d := NFD.String(c)
if d == c {
continue // nothing to recompose
}
// Skip characters that may legitimately merge with the
// Hangul prefix rather than recompose among themselves.
if f, _ := utf8.DecodeRuneInString(d); f >= 0x1100 && f <= 0x11ff ||
f >= hangulBase && f < hangulEnd {
continue
}
in, want := p+d, NFC.String(p)+c
if got := NFC.String(in); got != want {
t.Errorf("NFC(%+q) = %+q; want %+q", in, got, want)
}
if got := NFKC.String(in); got != want {
t.Errorf("NFKC(%+q) = %+q; want %+q", in, got, want)
}
}
}
}
EOF
cd $S/xtext/unicode/norm && python3 - <<'PY'
p='composition_test.go'
s=open(p).read()
s=s.replace('import "testing"','import (\n\t"testing"\n\t"unicode"\n\t"unicode/utf8"\n)')
s=s.rstrip()+"\n\n"+open('/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/hangul_test_snippet.go').read()
open(p,'w').write(s)
PY
cd $S/xtext && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ -run 'TestCompositionAfterHangul|TestComposition|TestDecomposition' -v 2>&1 | tail -15Add regression test to x/text and run against fixed code
composition_test.go:169: NFKC("\ud7a3\u03d2\u0301") = "\ud7a3\u038e"; want "\ud7a3\u03d3"
composition_test.go:169: NFKC("\ud7a3\u03d2\u0308") = "\ud7a3\u03ab"; want "\ud7a3\u03d4"
composition_test.go:169: NFKC("\ud7a3\u017f\u0307") = "\ud7a3\u1e61"; want "\ud7a3\u1e9b"
composition_test.go:169: NFKC("\ud7a3\u00a8\u0342") = "\ud7a3 \u0308\u0342"; want "\ud7a3\u1fc1"
composition_test.go:169: NFKC("\ud7a3\u1fbf\u0300") = "\ud7a3 \u0313\u0300"; want "\ud7a3\u1fcd"
composition_test.go:169: NFKC("\ud7a3\u1fbf\u0301") = "\ud7a3 \u0313\u0301"; want "\ud7a3\u1fce"
composition_test.go:169: NFKC("\ud7a3\u1fbf\u0342") = "\ud7a3 \u0313\u0342"; want "\ud7a3\u1fcf"
composition_test.go:169: NFKC("\ud7a3\u1ffe\u0300") = "\ud7a3 \u0314\u0300"; want "\ud7a3\u1fdd"
composition_test.go:169: NFKC("\ud7a3\u1ffe\u0301") = "\ud7a3 \u0314\u0301"; want "\ud7a3\u1fde"
composition_test.go:169: NFKC("\ud7a3\u1ffe\u0342") = "\ud7a3 \u0314\u0342"; want "\ud7a3\u1fdf"
composition_test.go:169: NFKC("\ud7a3\u00a8\u0300") = "\ud7a3 \u0308\u0300"; want "\ud7a3\u1fed"
--- FAIL: TestCompositionAfterHangul (0.35s)
FAIL
FAIL golang.org/x/text/unicode/norm 0.355s
FAIL
Shell cwd was reset to /home/claude/golang_go_81021Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad
cat > $S/hangul_test_snippet.go <<'EOF'
// Issue #81021: a Hangul syllable at the start of a segment must not
// disable regular canonical composition for the rest of the segment.
// The reorderBuffer switches to Hangul mode as soon as it sees a Jamo,
// and used to drop every non-Hangul composition after it.
func TestCompositionAfterHangul(t *testing.T) {
prefixes := []string{
"가", // Jamo L V
"각", // Jamo L V T
"가", // Hangul syllable LV
"각", // Hangul syllable LVT
"힣", // last Hangul syllable
}
forms := []struct {
name string
c, d Form
}{
{"NFC", NFC, NFD},
{"NFKC", NFKC, NFKD},
}
for _, p := range prefixes {
for _, f := range forms {
for r := rune(0); r <= unicode.MaxRune; r++ {
s := string(r)
if !utf8.ValidString(s) {
continue
}
want := f.c.String(s)
in := f.d.String(s)
if in == want {
continue // nothing to recompose
}
// Skip characters that may legitimately merge with the
// Hangul prefix rather than recompose among themselves.
if c, _ := utf8.DecodeRuneInString(in); c >= jamoLBase && c <= 0x11ff ||
c >= hangulBase && c < hangulEnd {
continue
}
in, want = p+in, f.c.String(p)+want
if got := f.c.String(in); got != want {
t.Errorf("%s(%+q) = %+q; want %+q", f.name, in, got, want)
}
}
}
}
}
EOF
cd $S/xtext/unicode/norm && python3 - <<'PY'
p='composition_test.go'
s=open(p).read()
i=s.index('// Issue #81021')
s=s[:i]+open('/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/hangul_test_snippet.go').read()
open(p,'w').write(s)
PY
cd $S/xtext && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ -run 'TestCompositionAfterHangul' -v 2>&1 | tail -15Rework and rerun regression test
=== RUN TestCompositionAfterHangul --- PASS: TestCompositionAfterHangul (0.69s) PASS ok golang.org/x/text/unicode/norm 0.692s Shell cwd was reset to /home/claude/golang_go_81021
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/xtext && cp unicode/norm/composition.go /tmp/fixed.go && cp /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition.go unicode/norm/composition.go && echo "--- WITHOUT FIX ---" && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ -run 'TestCompositionAfterHangul' 2>&1 | head -12 && cp /tmp/fixed.go unicode/norm/composition.go && echo "--- WITH FIX (full suite) ---" && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ 2>&1 | tail -5
Confirm test fails without fix, passes with
--- WITHOUT FIX ---
--- FAIL: TestCompositionAfterHangul (0.73s)
composition_test.go:175: NFC("\uac00\U000113c2\U000113c2") = "\uac00\U000113c2\U000113c2"; want "\uac00\U000113c5"
composition_test.go:175: NFC("\uac00\U000113c2\U000113b8") = "\uac00\U000113c2\U000113b8"; want "\uac00\U000113c7"
composition_test.go:175: NFC("\uac00\U000113c2\U000113c9") = "\uac00\U000113c2\U000113c9"; want "\uac00\U000113c8"
composition_test.go:175: NFC("\uac00\U0001611e\U0001611e") = "\uac00\U0001611e\U0001611e"; want "\uac00\U00016121"
composition_test.go:175: NFC("\uac00\U0001611e\U00016129") = "\uac00\U0001611e\U00016129"; want "\uac00\U00016122"
composition_test.go:175: NFC("\uac00\U0001611e\U0001611f") = "\uac00\U0001611e\U0001611f"; want "\uac00\U00016123"
composition_test.go:175: NFC("\uac00\U00016129\U0001611f") = "\uac00\U00016129\U0001611f"; want "\uac00\U00016124"
composition_test.go:175: NFC("\uac00\U0001611e\U00016120") = "\uac00\U0001611e\U00016120"; want "\uac00\U00016125"
composition_test.go:175: NFC("\uac00\U0001611e\U0001611e\U0001611f") = "\uac00\U0001611e\U0001611e\U0001611f"; want "\uac00\U00016126"
composition_test.go:175: NFC("\uac00\U0001611e\U00016129\U0001611f") = "\uac00\U0001611e\U00016129\U0001611f"; want "\uac00\U00016127"
composition_test.go:175: NFC("\uac00\U0001611e\U0001611e\U00016120") = "\uac00\U0001611e\U0001611e\U00016120"; want "\uac00\U00016128"
--- WITH FIX (full suite) ---
ok golang.org/x/text/unicode/norm 2.054s
Shell cwd was reset to /home/claude/golang_go_81021Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/xtext/unicode/norm && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/gofmt -l . ; U=/home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm diff -u $U/composition.go composition.go > $S/81021-xtext.patch; diff -u $U/composition_test.go composition_test.go >> $S/81021-xtext.patch; cat $S/81021-xtext.patch
Generate the x/text patch
composition_test.go
--- /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition.go 2026-08-19 10:51:41.929559908 -0400
+++ composition.go 2026-08-25 08:29:40.310769401 -0400
@@ -449,6 +449,16 @@
// ACxx plus 11Ax to LVT
rb.assignRune(s, l+v-jamoTBase)
default:
+ // Not a Hangul composition. The segment may still
+ // contain regular canonical compositions, such as
+ // combining marks following a Hangul syllable, so
+ // fall back to the composition table.
+ if b[i].combinesBackward() {
+ if combined := combine(l, v); combined != 0 {
+ rb.assignRune(s, combined)
+ continue
+ }
+ }
b[k] = b[i]
k++
}
--- /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition_test.go 2026-08-19 10:51:41.929650830 -0400
+++ composition_test.go 2026-08-25 08:29:28.958448288 -0400
@@ -4,7 +4,11 @@
package norm
-import "testing"
+import (
+ "testing"
+ "unicode"
+ "unicode/utf8"
+)
// TestCase is used for most tests.
type TestCase struct {
@@ -128,3 +132,49 @@
func TestComposition(t *testing.T) {
runTests(t, "TestComposition", NFC, compositionTest)
}
+
+// Issue #81021: a Hangul syllable at the start of a segment must not
+// disable regular canonical composition for the rest of the segment.
+// The reorderBuffer switches to Hangul mode as soon as it sees a Jamo,
+// and used to drop every non-Hangul composition after it.
+func TestCompositionAfterHangul(t *testing.T) {
+ prefixes := []string{
+ "가", // Jamo L V
+ "각", // Jamo L V T
+ "가", // Hangul syllable LV
+ "각", // Hangul syllable LVT
+ "힣", // last Hangul syllable
+ }
+ forms := []struct {
+ name string
+ c, d Form
+ }{
+ {"NFC", NFC, NFD},
+ {"NFKC", NFKC, NFKD},
+ }
+ for _, p := range prefixes {
+ for _, f := range forms {
+ for r := rune(0); r <= unicode.MaxRune; r++ {
+ s := string(r)
+ if !utf8.ValidString(s) {
+ continue
+ }
+ want := f.c.String(s)
+ in := f.d.String(s)
+ if in == want {
+ continue // nothing to recompose
+ }
+ // Skip characters that may legitimately merge with the
+ // Hangul prefix rather than recompose among themselves.
+ if c, _ := utf8.DecodeRuneInString(in); c >= jamoLBase && c <= 0x11ff ||
+ c >= hangulBase && c < hangulEnd {
+ continue
+ }
+ in, want = p+in, f.c.String(p)+want
+ if got := f.c.String(in); got != want {
+ t.Errorf("%s(%+q) = %+q; want %+q", f.name, in, got, want)
+ }
+ }
+ }
+ }
+}
Shell cwd was reset to /home/claude/golang_go_81021Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cd $S/xtext/unicode/norm && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/gofmt -w composition_test.go && GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/gofmt -l . # sync the exact comment wording into the Go tree copies for d in src/vendor src/cmd/vendor; do cp $S/xtext/unicode/norm/composition.go /home/claude/golang_go_81021/go/$d/golang.org/x/text/unicode/norm/composition.go done U=/home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm diff -u $U/composition.go composition.go > $S/81021-xtext.patch diff -u $U/composition_test.go composition_test.go >> $S/81021-xtext.patch cd $S/xtext && GOFLAGS=-mod=mod GOTOOLCHAIN=local /home/claude/golang_go_81021/go/bin/go test ./unicode/norm/ 2>&1 | tail -3 cd /home/claude/golang_go_81021/go/src && GOTOOLCHAIN=local ../bin/go build std cmd && echo GO_TREE_BUILD_OK && GOTOOLCHAIN=local ../bin/go test -count=1 net 2>&1 | tail -3
Format, sync into Go tree, rebuild and retest
ok golang.org/x/text/unicode/norm 2.075s GO_TREE_BUILD_OK ok net 28.976s
Bash S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad ...
S=/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad cp $S/81021-xtext.patch $S/../ 2>/dev/null; ls -la $S/81021-xtext.patch; sed -n '1,20p' $S/81021-xtext.patch
Confirm patch file
-rw-rw-r-- 1 claude claude 2480 Aug 25 08:30 /tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/81021-xtext.patch
--- /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition.go 2026-08-19 10:51:41.929559908 -0400
+++ composition.go 2026-08-25 08:29:40.310769401 -0400
@@ -449,6 +449,16 @@
// ACxx plus 11Ax to LVT
rb.assignRune(s, l+v-jamoTBase)
default:
+ // Not a Hangul composition. The segment may still
+ // contain regular canonical compositions, such as
+ // combining marks following a Hangul syllable, so
+ // fall back to the composition table.
+ if b[i].combinesBackward() {
+ if combined := combine(l, v); combined != 0 {
+ rb.assignRune(s, combined)
+ continue
+ }
+ }
b[k] = b[i]
k++
}
--- /home/claude/pkg/mod/golang.org/x/text@v0.41.0/unicode/norm/composition_test.go 2026-08-19 10:51:41.929650830 -0400SendUserFile
Fix + regression test for #81021, against golang.org/x/text@v0.41.0 (also applied to both vendored copies in the local GOROOT).
[ "/tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/81021-xtext.patch" ]
normal
1 file delivered to user. /tmp/claude-1003/-home-claude-golang-go-81021/499f4eb1-6fd9-4190-be20-22ac8f0f6e51/scratchpad/81021-xtext.patch → file_uuid: 583f2e0e-178e-4424-b792-5a0893d329e7
go.dev/issue/81021 — triage
Confirmed, valid, and the reporter's root-cause analysis is correct. Reproduced against the tree at 13b477edc7 with the Unicode 17 tables:
NFC(U+AC00 U+113C5) = U+AC00 U+113C2 U+113C2 (want U+AC00 U+113C5)
NFC(U+113C5) = U+113C5 (correct)
Mechanism
reorderBuffer.compose (composition.go:480) bails to combineHangul the moment it sees a Jamo:
if isJamoVT(rb.bytesAt(i)) {
rb.combineHangul(s, i, k)
return
}
combineHangul only knows the two algorithmic Hangul rules; its default: arm copies the rune through without ever consulting combine(). So once a segment starts with a Hangul syllable (which NFD-expands to L V [T], hitting isJamoVT at index 1), every table-driven composition later in that segment is silently dropped.
Why it surfaced only now
U+113C2 has ccc=0 and combinesBackward=true, so it has no boundary before it and gets pulled into the preceding Hangul syllable's segment. Unicode 17 is the first version to add canonical compositions of starter + starter of this shape. Scanning the recomposition map: 12 of 961 targets are affected under Unicode 17, 0 of 941 under Unicode 15 — Tulu-Tigalari (U+113C5/C7/C8), Gurung Khema (U+16121–U+16128), Kirat Rai (U+16D68). So it's a latent bug in combineHangul, newly reachable, and Go 1.27+ only (matching the //go:build go1.27 table selection).
Fix
Fall back to the composition table in combineHangul's default: arm, mirroring what compose does. Patch attached (also applied to src/vendor and src/cmd/vendor in the local tree).
Verification:
- All 12 affected targets now compose correctly after every Hangul prefix form (LV, LVT,
L V,L V T). - Differential old-vs-new over 3,126 "interesting" runes × pairs × 7 prefixes × {NFC, NFKC}: 403,464 outputs change, 0 break canonical equivalence (
NFD(new) == NFD(old)) and 0 are less composed. Nothing changes outside the intended cases. - Full upstream
x/text/unicode/normsuite passes;go build std cmdandgo test netpass. - The new test fails on unpatched v0.41.0 and passes with the fix.
The real fix belongs upstream in golang.org/x/text and then gets vendored into the Go tree.
Two adjacent bugs found while debugging (not fixed here)
-
#81001 is a distinct bug and this fix does not address it.
composeonly refreshess(last-starter index) inside theii.combinesBackward()branch, so a non-combining mark after accc=0backward-combining starter leavessstale and composition happens across the intervening starter.combineHangulupdatessunconditionally each iteration, which is why the Hangul analogue is already correct — and why this fix inherits no #81001 behavior. -
combine()truncates both operands to 16 bits (forminfo.go:216:uint32(uint16(a))<<16 + uint32(uint16(b))), so supplementary-plane runes alias BMP table entries. This is not a Unicode 17 regression — it's long-standing and reachable from plain BMP text:NFC(U+05D2 U+0307) = U+105C9 // Hebrew gimel + dot above → a Garay letter NFC(U+0300 U+11F41) = U+1F43 // Kawi char destroyed, Greek letter inventedExhaustive two-rune scan: 15,379 inputs under Unicode 17 (15,056 under Unicode 15) where NFC output is not canonically equivalent to its input. Fixing it needs a wider key in the generated
recompMapPacked, i.e. amaketables.gochange. This looks unreported and is arguably more severe than #81021 — worth filing separately if you'd like.