EC
bufio.ErrTooLong
GoERRORNotableIO

bufio.Scanner: token too long

Quick Answer

Call scanner.Buffer(buf, maxSize) before scanning to increase the token size limit.

What this means

Returned by bufio.Scanner when a token exceeds the scanner maximum buffer size. Default is 64 KiB; use Scanner.Buffer to increase it.

Why it happens
  1. 1Input line longer than bufio.MaxScanTokenSize (65536 bytes)
  2. 2Custom split function producing tokens larger than the scanner buffer

Fix

Set larger buffer

Set larger buffer
const max = 1024 * 1024
buf := make([]byte, max)
scanner.Buffer(buf, max)

Why this works

Scanner.Buffer replaces both the initial buffer and the hard max token size.

Code examples
Detectgo
s := bufio.NewScanner(r)
for s.Scan() {}
if errors.Is(s.Err(), bufio.ErrTooLong) {
    fmt.Println("token too long")
}
Increase limitgo
s := bufio.NewScanner(r)
s.Buffer(make([]byte, 512*1024), 512*1024)
Custom splitgo
s.Split(bufio.ScanLines)
for s.Scan() {
    fmt.Println(s.Text())
}
Sources
Official documentation ↗

Go standard library

Content generated with AI assistance and reviewed for accuracy. Found an error? hello@errcodes.dev

← All Go errors