Report for github.com/nlpodyssey/gotokenizers
A+ Excellent! Found 20 issues across 59 files
gofmt100%
Gofmt formats Go programs. We run gofmt -s on your code, where -s is for the "simplify" command
No problems detected. Good job!
go_vet100%
go vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string.
No problems detected. Good job!
gocyclo98%
Gocyclo calculates cyclomatic complexities of functions in Go source code. The cyclomatic complexity of a function is calculated according to the following rules: 1 is the base complexity of a function +1 for each 'if', 'for', 'case', '&&' or '||' Go Report Card warns on functions with cyclomatic complexity > 15.
-
-
gotokenizers/normalizedstring/normalizedstring.go
- Line 498: warning: cyclomatic complexity 25 of function (*NormalizedString).Split() is high (> 15) (gocyclo)
- Line 343: warning: cyclomatic complexity 18 of function (*NormalizedString).CoerceRangeToNormalized() is high (> 15) (gocyclo)
golint66%
Golint is a linter for Go source code.
-
-
gotokenizers/models/models.go
- Line 16: warning: exported type Token should have comment or be unexported (golint)
-
-
gotokenizers/pretokenizedstring/pretokenizedstring.go
- Line 27: warning: exported function FromString should have comment or be unexported (golint)
- Line 31: warning: exported function FromNormalizedString should have comment or be unexported (golint)
- Line 158: warning: exported method PreTokenizedString.Splits should have comment or be unexported (golint)
-
-
gotokenizers/splitpattern/string.go
- Line 12: warning: exported type StringSplitPattern should have comment or be unexported (golint)
- Line 19: warning: exported function FromString should have comment or be unexported (golint)
- Line 27: warning: exported method StringSplitPattern.FindMatches should have comment or be unexported (golint)
-
-
gotokenizers/models/bpemodel/word.go
- Line 42: warning: exported method WordSymbol.HasPrev should have comment or be unexported (golint)
- Line 46: warning: exported method WordSymbol.HasNext should have comment or be unexported (golint)
- Line 65: warning: exported method Word.Len should have comment or be unexported (golint)
- Line 89: warning: exported method Word.MergeAll should have comment or be unexported (golint)
-
-
gotokenizers/pretokenizedstring/split.go
- Line 28: warning: exported type OriginalByteSplit should have comment or be unexported (golint)
- Line 37: warning: exported type NormalizedByteSplit should have comment or be unexported (golint)
-
-
gotokenizers/models/wordpiecemodel/wordpiecemodel.go
- Line 14: warning: exported var ErrUnknownTokenOutOfVocabulary should have comment or be unexported (golint)
- Line 32: warning: exported function New should have comment or be unexported (golint)
- Line 46: warning: exported function NewDefault should have comment or be unexported (golint)
- Line 55: warning: exported method WordPieceModel.Tokenize should have comment or be unexported (golint)
-
-
gotokenizers/pretokenizers/whitespacepretokenizer/whitespacepretokenizer.go
- Line 23: warning: comment on exported var DefaultWordRegexp should be of the form "DefaultWordRegexp ..." (golint)
- Line 33: warning: exported function NewDefault should have comment or be unexported (golint)
-
-
gotokenizers/strutils/strutils.go
- Line 49: warning: exported type BytesToRuneOffsetConverter should have comment or be unexported (golint)
- Line 53: warning: exported function NewBytesToRuneOffsetConverter should have comment or be unexported (golint)
- Line 63: warning: exported method BytesToRuneOffsetConverter.Convert should have comment or be unexported (golint)
-
-
gotokenizers/splitpattern/regexp.go
- Line 12: warning: exported type RegexpSplitPattern should have comment or be unexported (golint)
- Line 18: warning: exported function FromRegexp should have comment or be unexported (golint)
- Line 22: warning: exported method RegexpSplitPattern.FindMatches should have comment or be unexported (golint)
-
-
gotokenizers/splitpattern/rune.go
- Line 7: warning: exported type RuneSplitPattern should have comment or be unexported (golint)
- Line 14: warning: exported function FromRune should have comment or be unexported (golint)
- Line 23: warning: exported method RuneSplitPattern.FindMatches should have comment or be unexported (golint)
-
-
gotokenizers/models/bpemodel/bpemodel.go
- Line 14: warning: exported var ErrUnknownTokenOutOfVocabulary should have comment or be unexported (golint)
- Line 72: warning: exported function NewDefault should have comment or be unexported (golint)
- Line 85: warning: exported method BPEModel.Tokenize should have comment or be unexported (golint)
-
-
gotokenizers/splitpattern/invertedpattern.go
- Line 7: warning: exported type InvertedPattern should have comment or be unexported (golint)
- Line 13: warning: exported function Invert should have comment or be unexported (golint)
- Line 17: warning: exported method InvertedPattern.FindMatches should have comment or be unexported (golint)
-
-
gotokenizers/models/bpemodel/mergemap.go
- Line 98: warning: exported method MergeMap.Set should have comment or be unexported (golint)
-
-
gotokenizers/models/bpemodel/wordmerge.go
- Line 9: warning: exported type WordMerge should have comment or be unexported (golint)
- Line 14: warning: exported type WordMergeHeap should have comment or be unexported (golint)
- Line 35: warning: exported method WordMergeHeap.Push should have comment or be unexported (golint)
- Line 39: warning: exported method WordMergeHeap.Pop should have comment or be unexported (golint)
-
-
gotokenizers/normalizedstring/range.go
- Line 30: warning: exported function NewOriginalRange should have comment or be unexported (golint)
- Line 34: warning: exported method OriginalRange.Start should have comment or be unexported (golint)
- Line 38: warning: exported method OriginalRange.End should have comment or be unexported (golint)
- Line 42: warning: exported method OriginalRange.Len should have comment or be unexported (golint)
- Line 55: warning: exported function NewNormalizedRange should have comment or be unexported (golint)
- Line 59: warning: exported method NormalizedRange.Start should have comment or be unexported (golint)
- Line 63: warning: exported method NormalizedRange.End should have comment or be unexported (golint)
- Line 67: warning: exported method NormalizedRange.Len should have comment or be unexported (golint)
-
-
gotokenizers/encodings/encodings.go
- Line 83: warning: exported function EncodingFromEncodableTokens should have comment or be unexported (golint)
-
-
gotokenizers/splitpattern/func.go
- Line 9: warning: exported type FuncSplitPattern should have comment or be unexported (golint)
- Line 15: warning: exported function FromFunc should have comment or be unexported (golint)
- Line 19: warning: exported method FuncSplitPattern.FindMatches should have comment or be unexported (golint)
-
-
gotokenizers/splitpattern/regexp2.go
- Line 12: warning: exported type Regexp2SplitPattern should have comment or be unexported (golint)
- Line 18: warning: exported function FromRegexp2 should have comment or be unexported (golint)
- Line 22: warning: exported method Regexp2SplitPattern.FindMatches should have comment or be unexported (golint)
-
-
gotokenizers/models/bpemodel/wordcache.go
- Line 10: warning: exported type WordCache should have comment or be unexported (golint)
- Line 28: warning: exported method WordCache.SetValues should have comment or be unexported (golint)
- Line 32: warning: exported method WordCache.Set should have comment or be unexported (golint)
- Line 36: warning: exported method WordCache.Get should have comment or be unexported (golint)
- Line 41: warning: exported method WordCache.GetValues should have comment or be unexported (golint)
-
-
gotokenizers/normalizedstring/normalizedstring.go
- Line 490: warning: exported const SplitDelimiterRemoved should have comment (or a comment on this block) or be unexported (golint)
ineffassign100%
IneffAssign detects ineffectual assignments in Go code.
No problems detected. Good job!