gofmt98%

Gofmt formats Go programs. We run gofmt -s on your code, where -s is for the "simplify" command

• tokenizer/model/bpe/trainer.go
  • Line 1: warning: file is not gofmted with -s (gofmt)

go_vet100%

go vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string.

No problems detected. Good job!

gocyclo92%

Gocyclo calculates cyclomatic complexities of functions in Go source code. The cyclomatic complexity of a function is calculated according to the following rules: 1 is the base complexity of a function +1 for each 'if', 'for', 'case', '&&' or '||' Go Report Card warns on functions with cyclomatic complexity > 15.

• tokenizer/pretokenizer.go
  • Line 143: warning: cyclomatic complexity 21 of function (*PreTokenizedString).IntoEncoding() is high (> 15) (gocyclo)

• tokenizer/encoding_test.go
  • Line 108: warning: cyclomatic complexity 19 of function TestTokenizer_Mapping() is high (> 15) (gocyclo)

• tokenizer/model/bpe/trainer.go
  • Line 598: warning: cyclomatic complexity 31 of function (*BpeTrainer).train() is high (> 15) (gocyclo)

• tokenizer/model/bpe/word.go
  • Line 257: warning: cyclomatic complexity 22 of function (*Word).MergeAll() is high (> 15) (gocyclo)

• tokenizer/normalizer/normalized.go
  • Line 455: warning: cyclomatic complexity 40 of function (*NormalizedString).TransformRange() is high (> 15) (gocyclo)
  • Line 1194: warning: cyclomatic complexity 18 of function (*NormalizedString).Split() is high (> 15) (gocyclo)

golint47%

Golint is a linter for Go source code.

• tokenizer/processor/roberta.go
  • Line 27: warning: exported function NewRobertaProcessing should have comment or be unexported (golint)
  • Line 48: warning: exported method RobertaProcessing.AddedTokens should have comment or be unexported (golint)
  • Line 51: warning: if block ends with a return statement, so drop this else and outdent its block (golint)

• tokenizer/decoder/bpe.go
  • Line 7: warning: comment on exported type BpeDecoder should be of the form "BpeDecoder ..." (with optional leading article) (golint)
  • Line 23: warning: exported method BpeDecoder.Decode should have comment or be unexported (golint)

• tokenizer/normalizer/pattern.go
  • Line 19: warning: comment on exported type OffsetsMatch should be of the form "OffsetsMatch ..." (with optional leading article) (golint)
  • Line 32: warning: exported function NewRunePattern should have comment or be unexported (golint)
  • Line 36: warning: comment on exported method RunePattern.FindMatches should be of the form "FindMatches ..." (golint)
  • Line 87: warning: comment on exported type StringPattern should be of the form "StringPattern ..." (with optional leading article) (golint)
  • Line 93: warning: exported function NewStringPattern should have comment or be unexported (golint)
  • Line 97: warning: exported method StringPattern.FindMatches should have comment or be unexported (golint)
  • Line 184: warning: exported type RegexpPattern should have comment or be unexported (golint)
  • Line 188: warning: exported function NewRegexpPattern should have comment or be unexported (golint)
  • Line 212: warning: exported type FnPattern should have comment or be unexported (golint)
  • Line 216: warning: exported function NewFnPattern should have comment or be unexported (golint)

• tokenizer/tokenizer.go
  • Line 26: warning: exported type Token should have comment or be unexported (golint)
  • Line 99: warning: comment on exported function NewToken should be of the form "NewToken ..." (golint)
  • Line 112: warning: exported type InputType should have comment or be unexported (golint)
  • Line 115: warning: exported const RawInput should have comment (or a comment on this block) or be unexported (golint)
  • Line 121: warning: exported type InputSequence should have comment or be unexported (golint)
  • Line 151: warning: exported type Single should have comment or be unexported (golint)
  • Line 154: warning: exported type Dual should have comment or be unexported (golint)
  • Line 159: warning: exported type EncodeInput should have comment or be unexported (golint)
  • Line 161: warning: exported function NewSingleEncodeInput should have comment or be unexported (golint)
  • Line 165: warning: exported function NewDualEncodeInput should have comment or be unexported (golint)
  • Line 187: warning: comment on exported function NewTokenizer should be of the form "NewTokenizer ..." (golint)
  • Line 201: warning: exported method Tokenizer.WithNormalizer should have comment or be unexported (golint)
  • Line 205: warning: exported method Tokenizer.GetNormalizer should have comment or be unexported (golint)
  • Line 209: warning: exported method Tokenizer.WithPreTokenizer should have comment or be unexported (golint)
  • Line 213: warning: exported method Tokenizer.GetPreTokenizer should have comment or be unexported (golint)
  • Line 217: warning: exported method Tokenizer.WithPostProcessor should have comment or be unexported (golint)
  • Line 221: warning: exported method Tokenizer.GetPostProcessor should have comment or be unexported (golint)
  • Line 225: warning: exported method Tokenizer.WithDecoder should have comment or be unexported (golint)
  • Line 229: warning: exported method Tokenizer.GetDecoder should have comment or be unexported (golint)
  • Line 233: warning: exported method Tokenizer.WithModel should have comment or be unexported (golint)
  • Line 237: warning: exported method Tokenizer.GetModel should have comment or be unexported (golint)
  • Line 241: warning: exported method Tokenizer.WithTruncation should have comment or be unexported (golint)
  • Line 245: warning: exported method Tokenizer.GetTruncation should have comment or be unexported (golint)
  • Line 249: warning: exported method Tokenizer.WithPadding should have comment or be unexported (golint)
  • Line 253: warning: exported method Tokenizer.GetPadding should have comment or be unexported (golint)
  • Line 537: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 613: warning: comment on exported method Tokenizer.TrainAndReplace should be of the form "TrainAndReplace ..." (golint)

• tokenizer/pretokenizer/bert.go
  • Line 68: warning: exported type BertPreTokenizer should have comment or be unexported (golint)
  • Line 70: warning: exported function NewBertPreTokenizer should have comment or be unexported (golint)

• tokenizer/pretrained/bert.go
  • Line 16: warning: comment on exported function BertBaseUncased should be of the form "BertBaseUncased ..." (golint)

• tokenizer/util/slice/int.go
  • Line 1: warning: package comment should be of the form "Package util ..." (golint)
  • Line 139: warning: comment on exported function FilterIntNoAllocate should be of the form "FilterIntNoAllocate ..." (golint)

• tokenizer/util/slice/string.go
  • Line 1: warning: package comment should be of the form "Package util ..." (golint)

• tokenizer/normalizer/strip.go
  • Line 3: warning: exported type Strip should have comment or be unexported (golint)
  • Line 8: warning: exported function NewStrip should have comment or be unexported (golint)
  • Line 18: warning: exported method Strip.Normalize should have comment or be unexported (golint)
  • Line 22: warning: if block ends with a return statement, so drop this else and outdent its block (golint)

• tokenizer/encoding.go
  • Line 9: warning: exported type PaddingDirection should have comment or be unexported (golint)
  • Line 12: warning: exported const Left should have comment (or a comment on this block) or be unexported (golint)
  • Line 48: warning: exported function NewEncodingWithCapacity should have comment or be unexported (golint)
  • Line 61: warning: comment on exported function DefaultEncoding should be of the form "DefaultEncoding ..." (golint)
  • Line 113: warning: comment on exported method Encoding.GetTokens should be of the form "GetTokens ..." (golint)
  • Line 191: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 202: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 213: warning: if block ends with a return statement, so drop this else and outdent its block (golint)

• tokenizer/model/common.go
  • Line 3: warning: exported type Vocab should have comment or be unexported (golint)
  • Line 4: warning: exported type VocabR should have comment or be unexported (golint)

• tokenizer/decoder/wordpiece.go
  • Line 17: warning: comment on exported function NewWordPieceDecoder should be of the form "NewWordPieceDecoder ..." (golint)
  • Line 25: warning: comment on exported function DefaultWordpieceDecoder should be of the form "DefaultWordpieceDecoder ..." (golint)
  • Line 33: warning: exported method WordPieceDecoder.Decode should have comment or be unexported (golint)

• tokenizer/normalizer/normalizer.go
  • Line 7: warning: exported type Normalizer should have comment or be unexported (golint)
  • Line 30: warning: exported type Option should have comment or be unexported (golint)
  • Line 47: warning: exported function NewNormalizer should have comment or be unexported (golint)

• tokenizer/processor/bytelevel.go
  • Line 8: warning: exported type ByteLevelProcessing should have comment or be unexported (golint)
  • Line 12: warning: exported function NewByteLevelProcessing should have comment or be unexported (golint)
  • Line 21: warning: exported method ByteLevelProcessing.AddedTokens should have comment or be unexported (golint)
  • Line 25: warning: exported method ByteLevelProcessing.Process should have comment or be unexported (golint)

• tokenizer/model/wordpiece/trainer.go
  • Line 101: warning: exported method WordPieceTrainer.Train should have comment or be unexported (golint)
  • Line 108: warning: exported method WordPieceTrainer.ProcessTokens should have comment or be unexported (golint)
  • Line 112: warning: exported method WordPieceTrainer.WithProgressBar should have comment or be unexported (golint)

• tokenizer/util/iterator.go
  • Line 52: warning: if block ends with a return statement, so drop this else and outdent its block (golint)

• tokenizer/util/rune-reader.go
  • Line 31: warning: exported type RuneReader should have comment or be unexported (golint)

• tokenizer/normalizer/default.go
  • Line 1: warning: package comment should be of the form "Package normalizer ..." (golint)
  • Line 21: warning: exported type DefaultNormalizer should have comment or be unexported (golint)
  • Line 28: warning: exported type DefaultOption should have comment or be unexported (golint)
  • Line 30: warning: exported function WithLowercase should have comment or be unexported (golint)
  • Line 36: warning: exported function WithStrip should have comment or be unexported (golint)
  • Line 50: warning: exported method DefaultNormalizer.Normalize should have comment or be unexported (golint)
  • Line 65: warning: exported function NewDefaultNormalizer should have comment or be unexported (golint)

• tokenizer/normalizer/unicode.go
  • Line 42: warning: exported type UnicodeNormalizer should have comment or be unexported (golint)
  • Line 46: warning: exported function NewUnicodeNormalizer should have comment or be unexported (golint)
  • Line 52: warning: exported method UnicodeNormalizer.Normalize should have comment or be unexported (golint)

• tokenizer/model/bpe/cache.go
  • Line 18: warning: exported type CacheItem should have comment or be unexported (golint)
  • Line 62: warning: exported method Cache.SetValues should have comment or be unexported (golint)

• tokenizer/model/wordlevel/wordlevel.go
  • Line 73: warning: comment on exported function NewWorldLevelFromFile should be of the form "NewWorldLevelFromFile ..." (golint)

• tokenizer/normalizer/bert.go
  • Line 7: warning: exported type BertNormalizer should have comment or be unexported (golint)
  • Line 14: warning: exported function NewBertNormalizer should have comment or be unexported (golint)
  • Line 102: warning: comment on exported function IsChinese should be of the form "IsChinese ..." (golint)

• tokenizer/model/bpe/bpe.go
  • Line 22: warning: exported type Merges should have comment or be unexported (golint)
  • Line 29: warning: exported type Config should have comment or be unexported (golint)
  • Line 46: warning: exported function NewBpeBuilder should have comment or be unexported (golint)
  • Line 92: warning: comment on exported method BpeBuilder.ContinuingSubwordPrefix should be of the form "ContinuingSubwordPrefix ..." (golint)
  • Line 222: warning: exported function DefaultBPE should have comment or be unexported (golint)
  • Line 252: warning: comment on exported method BPE.FromFiles should be of the form "FromFiles ..." (golint)
  • Line 369: warning: comment on exported method BPE.GetContinuingSubwordPrfix should be of the form "GetContinuingSubwordPrfix ..." (golint)
  • Line 495: warning: exported method BPE.TokenizeWithCache should have comment or be unexported (golint)
  • Line 499: warning: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
  • Line 511: warning: exported method BPE.TokenToId should have comment or be unexported (golint)
  • Line 516: warning: exported method BPE.IdToToken should have comment or be unexported (golint)
  • Line 521: warning: exported method BPE.GetVocabSize should have comment or be unexported (golint)
  • Line 525: warning: exported method BPE.Save should have comment or be unexported (golint)

• tokenizer/pretokenizer/bytelevel.go
  • Line 23: warning: exported var BytesChar should have comment or be unexported (golint)
  • Line 25: warning: exported var CharBytes should have comment or be unexported (golint)
  • Line 74: warning: comment on exported function GenerateBytesChar should be of the form "GenerateBytesChar ..." (golint)
  • Line 181: warning: comment on exported method ByteLevel.PreTokenize should be of the form "PreTokenize ..." (golint)
  • Line 248: warning: exported method ByteLevel.AddedToken should have comment or be unexported (golint)
  • Line 252: warning: exported method ByteLevel.Process should have comment or be unexported (golint)
  • Line 360: warning: exported function ProcessOffsets should have comment or be unexported (golint)

• tokenizer/model/wordpiece/wordpiece.go
  • Line 16: warning: error var MissingUnkToken should have name of the form ErrFoo (golint)
  • Line 16: warning: exported var MissingUnkToken should have comment or be unexported (golint)
  • Line 32: warning: exported function NewWordPieceBuilder should have comment or be unexported (golint)
  • Line 66: warning: comment on exported method WordPieceBuilder.ContinuingSubwordPrefix should be of the form "ContinuingSubwordPrefix ..." (golint)
  • Line 73: warning: comment on exported method WordPieceBuilder.MaxInputCharsPerWord should be of the form "MaxInputCharsPerWord ..." (golint)
  • Line 218: warning: comment on exported function NewWordPieceFromBPE should be of the form "NewWordPieceFromBPE ..." (golint)
  • Line 241: warning: exported method WordPiece.GetVocab should have comment or be unexported (golint)
  • Line 245: warning: exported method WordPiece.GetVocabSize should have comment or be unexported (golint)
  • Line 249: warning: exported method WordPiece.Tokenize should have comment or be unexported (golint)
  • Line 326: warning: exported method WordPiece.TokenToId should have comment or be unexported (golint)
  • Line 331: warning: exported method WordPiece.IdToToken should have comment or be unexported (golint)
  • Line 336: warning: exported method WordPiece.Save should have comment or be unexported (golint)

• tokenizer/normalizer/normalized.go
  • Line 28: warning: exported const RemovedBehavior should have comment (or a comment on this block) or be unexported (golint)
  • Line 34: warning: exported type OffsetsRemove should have comment or be unexported (golint)
  • Line 39: warning: comment on exported type IndexOn should be of the form "IndexOn ..." (with optional leading article) (golint)
  • Line 45: warning: exported const OriginalTarget should have comment (or a comment on this block) or be unexported (golint)
  • Line 57: warning: exported function NewRange should have comment or be unexported (golint)
  • Line 65: warning: exported method Range.Start should have comment or be unexported (golint)
  • Line 69: warning: exported method Range.End should have comment or be unexported (golint)
  • Line 79: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 93: warning: comment on exported method Range.On should be of the form "On ..." (golint)
  • Line 129: warning: comment on exported type NormalizedString should be of the form "NormalizedString ..." (with optional leading article) (golint)
  • Line 195: warning: exported function NewNormalizedString should have comment or be unexported (golint)
  • Line 235: warning: comment on exported method NormalizedString.ConvertOffset should be of the form "ConvertOffset ..." (golint)
  • Line 442: warning: exported type ChangeMap should have comment or be unexported (golint)
  • Line 447: warning: comment on exported method NormalizedString.TransformRange should be of the form "TransformRange ..." (golint)
  • Line 870: warning: exported method NormalizedString.NFD should have comment or be unexported (golint)
  • Line 918: warning: exported method NormalizedString.NFC should have comment or be unexported (golint)
  • Line 956: warning: exported method NormalizedString.NFKD should have comment or be unexported (golint)
  • Line 992: warning: exported method NormalizedString.NFKC should have comment or be unexported (golint)
  • Line 1105: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 1411: warning: if block ends with a return statement, so drop this else and outdent its block (move short variable declaration to its own line if necessary) (golint)
  • Line 1416: warning: exported method NormalizedString.Replace should have comment or be unexported (golint)

• tokenizer/pretokenizer.go
  • Line 13: warning: exported type PreToken should have comment or be unexported (golint)
  • Line 23: warning: exported const Byte should have comment (or a comment on this block) or be unexported (golint)
  • Line 47: warning: comment on exported type PreTokenizedString should be of the form "PreTokenizedString ..." (with optional leading article) (golint)
  • Line 281: warning: comment on exported function NewPreTokenizedStringFromNS should be of the form "NewPreTokenizedStringFromNS ..." (golint)

• tokenizer/util.go
  • Line 8: warning: exported type TruncationParams should have comment or be unexported (golint)
  • Line 14: warning: exported type PaddingParams should have comment or be unexported (golint)
  • Line 36: warning: exported type PaddingStrategyOption should have comment or be unexported (golint)
  • Line 38: warning: exported function WithBatchLongest should have comment or be unexported (golint)
  • Line 45: warning: exported function WithFixed should have comment or be unexported (golint)
  • Line 52: warning: exported function NewPaddingStrategy should have comment or be unexported (golint)
  • Line 72: warning: exported const LongestFirst should have comment (or a comment on this block) or be unexported (golint)
  • Line 78: warning: exported const SecondSequenceNotProvided should have comment (or a comment on this block) or be unexported (golint)
  • Line 82: warning: exported function TruncateEncodings should have comment or be unexported (golint)
  • Line 127: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 150: warning: exported function PadEncodings should have comment or be unexported (golint)

• tokenizer/processor/bert.go
  • Line 7: warning: exported type PostToken should have comment or be unexported (golint)
  • Line 12: warning: exported type BertProcessing should have comment or be unexported (golint)
  • Line 17: warning: exported function NewBertProcessing should have comment or be unexported (golint)
  • Line 27: warning: exported method BertProcessing.AddedTokens should have comment or be unexported (golint)
  • Line 30: warning: if block ends with a return statement, so drop this else and outdent its block (golint)

• tokenizer/added-vocabulary.go
  • Line 45: warning: exported type ATOption should have comment or be unexported (golint)
  • Line 94: warning: comment on exported method AddedToken.SetSingleWord should be of the form "SetSingleWord ..." (golint)
  • Line 101: warning: comment on exported method AddedToken.SetLStrip should be of the form "SetLStrip ..." (golint)
  • Line 108: warning: comment on exported method AddedToken.SetRStrip should be of the form "SetRStrip ..." (golint)
  • Line 115: warning: comment on exported method AddedToken.SetNormalized should be of the form "SetNormalized ..." (golint)
  • Line 220: warning: exported function NewAddedVocabulary should have comment or be unexported (golint)
  • Line 242: warning: comment on exported method AddedVocabulary.TokenToId should be of the form "TokenToId ..." (golint)
  • Line 253: warning: comment on exported method AddedVocabulary.IdToToken should be of the form "IdToToken ..." (golint)
  • Line 263: warning: comment on exported method AddedVocabulary.IsSpecialToken should be of the form "IsSpecialToken ..." (golint)
  • Line 270: warning: comment on exported method AddedVocabulary.AddSpecialTokens should be of the form "AddSpecialTokens ..." (golint)
  • Line 286: warning: comment on exported method AddedVocabulary.AddTokens should be of the form "AddTokens ..." (golint)
  • Line 467: warning: exported type SplitIdx should have comment or be unexported (golint)
  • Line 528: warning: exported type AddedTokenWithId should have comment or be unexported (golint)

• tokenizer/util/error.go
  • Line 32: warning: comment on exported function TraceError should be of the form "TraceError ..." (golint)

• tokenizer/model/bpe/trainer.go
  • Line 21: warning: comment on exported type UintSet should be of the form "UintSet ..." (with optional leading article) (golint)
  • Line 25: warning: exported type CharSet should have comment or be unexported (golint)
  • Line 27: warning: exported type TMerge should have comment or be unexported (golint)
  • Line 34: warning: comment on exported type TConfig should be of the form "TConfig ..." (with optional leading article) (golint)
  • Line 52: warning: exported function NewBPETrainerBuilder should have comment or be unexported (golint)
  • Line 68: warning: comment on exported method BpeTrainerBuilder.MinFrequency should be of the form "MinFrequency ..." (golint)
  • Line 83: warning: comment on exported method BpeTrainerBuilder.SpecialTokens should be of the form "SpecialTokens ..." (golint)
  • Line 153: warning: exported function NewBpeTrainer should have comment or be unexported (golint)
  • Line 221: warning: should omit 2nd value from range; this loop is equivalent to `for initChar := range ...` (golint)
  • Line 233: warning: should omit 2nd value from range; this loop is equivalent to `for k := range ...` (golint)
  • Line 570: warning: exported method BpeTrainer.WithProgressBar should have comment or be unexported (golint)
  • Line 587: warning: comment on exported method BpeTrainer.ProcessTokens should be of the form "ProcessTokens ..." (golint)
  • Line 743: warning: should omit 2nd value from range; this loop is equivalent to `for p := range ...` (golint)
  • Line 787: warning: should omit 2nd value from range; this loop is equivalent to `for i := range ...` (golint)

• tokenizer/model/bpe/word.go
  • Line 12: warning: exported const DefaultCacheCapacity should have comment or be unexported (golint)
  • Line 14: warning: exported type Merge should have comment or be unexported (golint)
  • Line 25: warning: exported const Less should have comment (or a comment on this block) or be unexported (golint)
  • Line 30: warning: comment on exported method Merge.Eq should be of the form "Eq ..." (golint)
  • Line 38: warning: exported method Merge.PartialCmp should have comment or be unexported (golint)
  • Line 57: warning: exported method Merge.Cmp should have comment or be unexported (golint)
  • Line 62: warning: exported type Symbol should have comment or be unexported (golint)
  • Line 69: warning: comment on exported type Symbols should be of the form "Symbols ..." (with optional leading article) (golint)
  • Line 94: warning: exported method Symbol.MergeWith should have comment or be unexported (golint)
  • Line 100: warning: exported type Word should have comment or be unexported (golint)
  • Line 104: warning: exported function NewWord should have comment or be unexported (golint)
  • Line 111: warning: exported method Word.Add should have comment or be unexported (golint)
  • Line 155: warning: exported type Pair should have comment or be unexported (golint)
  • Line 166: warning: exported type WChange should have comment or be unexported (golint)
  • Line 257: warning: exported method Word.MergeAll should have comment or be unexported (golint)
  • Line 413: warning: exported method Word.GetChars should have comment or be unexported (golint)
  • Line 421: warning: exported method Word.GetOffsets should have comment or be unexported (golint)

• tokenizer/util/util.go
  • Line 15: warning: comment on exported function MakeRange should be of the form "MakeRange ..." (golint)
  • Line 34: warning: if block ends with a return statement, so drop this else and outdent its block (golint)
  • Line 82: warning: comment on exported type RuneGen should be of the form "RuneGen ..." (with optional leading article) (golint)
  • Line 220: warning: comment on exported function MinMaxFloat64 should be of the form "MinMaxFloat64 ..." (golint)

license100%

Checks whether your project has a LICENSE file.

No problems detected. Good job!

ineffassign92%

IneffAssign detects ineffectual assignments in Go code.

• tokenizer/pretokenizer/bert.go
  • Line 62: warning: ineffectual assignment to word (ineffassign)

• tokenizer/encoding.go
  • Line 173: warning: ineffectual assignment to end (ineffassign)

• tokenizer/model/bpe/bpe.go
  • Line 202: warning: ineffectual assignment to b (ineffassign)
  • Line 213: warning: ineffectual assignment to b (ineffassign)

• tokenizer/model/bpe/cache.go
  • Line 36: warning: ineffectual assignment to c (ineffassign)

• tokenizer/model/bpe/trainer.go
  • Line 643: warning: ineffectual assignment to pairCounts (ineffassign)
  • Line 644: warning: ineffectual assignment to whereToUpdate (ineffassign)

misspell86%

Misspell Finds commonly misspelled English words

• tokenizer/util/slice/int.go
  • Line 1: warning: "manupulation" is a misspelling of "manipulation" (misspell)

• tokenizer/util/slice/string.go
  • Line 1: warning: "manupulation" is a misspelling of "manipulation" (misspell)

• tokenizer/normalizer/normalized.go
  • Line 473: warning: "occured" is a misspelling of "occurred" (misspell)
  • Line 576: warning: "begining" is a misspelling of "beginning" (misspell)
  • Line 1076: warning: "begining" is a misspelling of "beginning" (misspell)
  • Line 1406: warning: "substracts" is a misspelling of "subtracts" (misspell)
  • Line 1407: warning: "substraction" is a misspelling of "subtraction" (misspell)

• tokenizer/normalizer/pattern.go
  • Line 271: warning: "usefull" is a misspelling of "useful" (misspell)

• tokenizer/model/bpe/word.go
  • Line 355: warning: "asign" is a misspelling of "assign" (misspell)

• tokenizer/example/truncation/main.go
  • Line 55: warning: "lenght" is a misspelling of "length" (misspell)

• tokenizer/tokenizer.go
  • Line 765: warning: "proccessed" is a misspelling of "processed" (misspell)

• tokenizer/model/bpe/trainer.go
  • Line 219: warning: "intial" is a misspelling of "initial" (misspell)
  • Line 222: warning: "asign" is a misspelling of "assign" (misspell)
  • Line 341: warning: "neccessary" is a misspelling of "necessary" (misspell)
  • Line 546: warning: "exisitng" is a misspelling of "existing" (misspell)
  • Line 639: warning: "occurence" is a misspelling of "occurrence" (misspell)

• tokenizer/pretokenizer/bytelevel.go
  • Line 135: warning: "neccessary" is a misspelling of "necessary" (misspell)