String::Tokenizer String::Tokenizer is a simple string tokenizer.
Download

String::Tokenizer Ranking & Summary

String::Tokenizer Tags

Perl module string Perl extension Tokenizer string tokenizer

String::Tokenizer Description

String::Tokenizer is a simple string tokenizer. String::Tokenizer is a simple string tokenizer.SYNOPSIS use String::Tokenizer; # create the tokenizer and tokenize input my $tokenizer = String::Tokenizer->new("((5+5) * 10)", '+*()'); # create tokenizer my $tokenizer = String::Tokenizer->new(); # ... then tokenize the string $tokenizer->tokenize("((5 + 5) - 10)", '()'); # will print '(, (, 5, +, 5, ), -, 10, )' print join ", " => $tokenizer->getTokens(); # create tokenizer which retains whitespace my $st = String::Tokenizer->new( 'this is a test with, (signifigant) whitespace', ',()', String::Tokenizer->RETAIN_WHITESPACE ); # this will print: # 'this', ' ', 'is', ' ', 'a', ' ', 'test', ' ', 'with', ' ', '(', 'signifigant', ')', ' ', 'whitespace' print "'" . (join "', '" => $tokenizer->getTokens()) . "'"; # get a token iterator my $i = $tokenizer->iterator(); while ($i->hasNextToken()) { my $next = $i->nextToken(); # peek ahead at the next token my $look_ahead = $i->lookAheadToken(); # ... # skip the next 2 tokens $i->skipTokens(2); # ... # then backtrack 1 token my $previous = $i->prevToken(); # ... # get the current token my $current = $i->currentToken(); # ... }A simple string tokenizer which takes a string and splits it on whitespace. It also optionally takes a string of characters to use as delimiters, and returns them with the token set as well. This allows for splitting the string in many different ways.This is a very basic tokenizer, so more complex needs should be either addressed with a custom written tokenizer or post-processing of the output generated by this module. Basically, this will not fill everyones needs, but it spans a gap between simple split / /, $string and the other options that involve much larger and complex modules.Also note that this is not a lexical analyser. Many people confuse tokenization with lexical analysis. A tokenizer mearly splits its input into specific chunks, a lexical analyzer classifies those chunks. Sometimes these two steps are combined, but not here.Requirements:· Perl Requirements: · Perl

String::Tokenizer Related Software