0359647a82
Fedora 35 support was dropped, so we can update to a newer Go. Stable RHEL 8 and 9 and Fedora 36 ships Go 1.18, so let's switch to it. "//go:build" directives are now apparently enforced by go fmt, so that's why there were added. Also, all the github actions were adjusted to use Go 1.18. Signed-off-by: Ondřej Budai <ondrej@budai.cz> |
||
|---|---|---|
| .. | ||
| .gitignore | ||
| .travis.yml | ||
| LICENSE | ||
| README.md | ||
| utfbom.go | ||
utfbom 
The package utfbom implements the detection of the BOM (Unicode Byte Order Mark) and removing as necessary. It can also return the encoding detected by the BOM.
Installation
go get -u github.com/dimchansky/utfbom
Example
package main
import (
"bytes"
"fmt"
"io/ioutil"
"github.com/dimchansky/utfbom"
)
func main() {
trySkip([]byte("\xEF\xBB\xBFhello"))
trySkip([]byte("hello"))
}
func trySkip(byteData []byte) {
fmt.Println("Input:", byteData)
// just skip BOM
output, err := ioutil.ReadAll(utfbom.SkipOnly(bytes.NewReader(byteData)))
if err != nil {
fmt.Println(err)
return
}
fmt.Println("ReadAll with BOM skipping", output)
// skip BOM and detect encoding
sr, enc := utfbom.Skip(bytes.NewReader(byteData))
fmt.Printf("Detected encoding: %s\n", enc)
output, err = ioutil.ReadAll(sr)
if err != nil {
fmt.Println(err)
return
}
fmt.Println("ReadAll with BOM detection and skipping", output)
fmt.Println()
}
Output:
$ go run main.go
Input: [239 187 191 104 101 108 108 111]
ReadAll with BOM skipping [104 101 108 108 111]
Detected encoding: UTF8
ReadAll with BOM detection and skipping [104 101 108 108 111]
Input: [104 101 108 108 111]
ReadAll with BOM skipping [104 101 108 108 111]
Detected encoding: Unknown
ReadAll with BOM detection and skipping [104 101 108 108 111]