Package antlr
Class PythonCodeGenerator
- java.lang.Object
-
- antlr.CodeGenerator
-
- antlr.PythonCodeGenerator
-
public class PythonCodeGenerator extends CodeGenerator
Generate MyParser.java, MyLexer.java and MyParserTokenTypes.java
-
-
Field Summary
Fields Modifier and Type Field Description (package private) int
astVarNumber
static int
caseSizeThreshold
(package private) java.lang.String
commonExtraArgs
(package private) java.lang.String
commonExtraParams
(package private) java.lang.String
commonLocalVars
(package private) java.lang.String
currentASTResult
Tracks the rule or labeled subrule being generated.(package private) RuleBlock
currentRule
Tracks the rule being generated.(package private) java.util.Hashtable
declaredASTVariables
Used to keep track of which AST variables have been defined in a rule (except for the #rule_name and #rule_name_in var's(package private) java.lang.String
exceptionThrown
protected boolean
genAST
static java.lang.String
initHeaderAction
(package private) java.lang.String
labeledElementASTType
(package private) java.lang.String
labeledElementInit
(package private) java.lang.String
labeledElementType
(package private) java.lang.String
lexerClassName
(package private) java.lang.String
lt1Value
static java.lang.String
mainHeaderAction
protected static java.lang.String
NONUNIQUE
Special value used to mark duplicate in treeVariableMap(package private) java.lang.String
parserClassName
protected boolean
saveText
protected int
syntacticPredLevel
(package private) java.lang.String
throwNoViable
(package private) java.util.Hashtable
treeVariableMap
Mapping between the ids used in the current alt, and the names of variables used to represent their AST values.(package private) java.lang.String
treeWalkerClassName
-
Fields inherited from class antlr.CodeGenerator
analyzer, antlrTool, behavior, BITSET_OPTIMIZE_INIT_THRESHOLD, bitsetsUsed, bitsetTestThreshold, charFormatter, currentOutput, DEBUG_CODE_GENERATOR, DEFAULT_BITSET_TEST_THRESHOLD, DEFAULT_MAKE_SWITCH_THRESHOLD, grammar, makeSwitchThreshold, tabs, TokenTypesFileExt, TokenTypesFileSuffix
-
-
Constructor Summary
Constructors Constructor Description PythonCodeGenerator()
-
Method Summary
All Methods Static Methods Instance Methods Concrete Methods Modifier and Type Method Description protected void
_printAction(java.lang.String s)
Print an action without leading tabs, attempting to preserve the current indentation level for multi-line actions Ignored if string is null.protected void
_printJavadoc(java.lang.String s)
protected int
addSemPred(java.lang.String predicate)
Adds a semantic predicate string to the sem pred vector These strings will be used to build an array of sem pred names when building a debugging parser.protected void
checkCurrentOutputStream()
void
exitIfError()
protected java.lang.String
extractIdOfAction(java.lang.String s, int line, int column)
Get the identifier portion of an argument-action.protected java.lang.String
extractTypeOfAction(java.lang.String s, int line, int column)
Get the type portion of an argument-action.protected void
flushTokens()
void
gen()
Generate the parser, lexer, treeparser, and token types in Javavoid
gen(ActionElement action)
Generate code for the given grammar element.void
gen(AlternativeBlock blk)
Generate code for the given grammar element.void
gen(BlockEndElement end)
Generate code for the given grammar element.void
gen(CharLiteralElement atom)
Generate code for the given grammar element.void
gen(CharRangeElement r)
Generate code for the given grammar element.void
gen(LexerGrammar g)
Generate the lexer Java filevoid
gen(OneOrMoreBlock blk)
Generate code for the given grammar element.void
gen(ParserGrammar g)
Generate the parser Java filevoid
gen(RuleRefElement rr)
Generate code for the given grammar element.void
gen(StringLiteralElement atom)
Generate code for the given grammar element.void
gen(TokenRangeElement r)
Generate code for the given grammar element.void
gen(TokenRefElement atom)
Generate code for the given grammar element.void
gen(TreeElement t)
Generate code for the given grammar element.void
gen(TreeWalkerGrammar g)
Generate the tree-parser Java filevoid
gen(WildcardElement wc)
Generate code for the given grammar element.void
gen(ZeroOrMoreBlock blk)
Generate code for the given grammar element.protected void
genAlt(Alternative alt, AlternativeBlock blk)
Generate an alternative.protected void
genASTDeclaration(AlternativeElement el)
protected void
genASTDeclaration(AlternativeElement el, java.lang.String node_type)
protected void
genASTDeclaration(AlternativeElement el, java.lang.String var_name, java.lang.String node_type)
protected void
genBitsets(Vector bitsetList, int maxVocabulary)
Generate all the bitsets to be used in the parser or lexer Generate the raw bitset data like "long _tokenSet1_data[] = {...}" and the BitSet object declarations like "BitSet _tokenSet1 = new BitSet(_tokenSet1_data)" Note that most languages do not support object initialization inside a class definition, so other code-generators may have to separate the bitset declarations from the initializations (e.g., put the initializations in the generated constructor instead).protected void
genBlockInitAction(AlternativeBlock blk)
Generate the init action for a block, which may be a RuleBlock or a plain AlternativeBLock.protected void
genBlockPreamble(AlternativeBlock blk)
Generate the header for a block, which may be a RuleBlock or a plain AlternativeBLock.protected void
genCases(BitSet p)
Generate a series of case statements that implement a BitSet test.PythonBlockFinishingInfo
genCommonBlock(AlternativeBlock blk, boolean noTestForSingle)
Generate common code for a block of alternatives; return a postscript that needs to be generated at the end of the block.protected void
genHeader()
Generate a header that is common to all Python filesprotected void
genHeaderInit(Grammar grammar)
protected void
genHeaderMain(Grammar grammar)
protected void
genJavadocComment(Grammar g)
protected void
genJavadocComment(RuleSymbol g)
protected void
genLexerTest()
Generate an automated test for Python CharScanner (sub)classes.protected void
genMatch(BitSet b)
protected void
genMatch(GrammarAtom atom)
protected void
genMatchUsingAtomText(GrammarAtom atom)
protected void
genMatchUsingAtomTokenType(GrammarAtom atom)
void
genNextToken()
Generate the nextToken() rule.void
genRule(RuleSymbol s, boolean startSymbol, int ruleNum)
Gen a named rule block.protected void
genSemPred(java.lang.String pred, int line)
protected void
genSemPredMap()
Write an array of Strings which are the semantic predicate expressions.protected void
genSynPred(SynPredBlock blk, java.lang.String lookaheadExpr)
protected void
genTokenASTNodeMap()
Create and set Integer token type objects that map to Java Class objects (which AST node to create).void
genTokenStrings()
Generate a static array containing the names of the tokens, indexed by the token type values.protected void
genTokenTypes(TokenManager tm)
Generate the token types Java filejava.lang.String
getASTCreateString(Vector v)
Get a string for an expression to generate creation of an AST subtree.java.lang.String
getASTCreateString(GrammarAtom atom, java.lang.String astCtorArgs)
Get a string for an expression to generate creating of an AST nodejava.lang.String
getASTCreateString(java.lang.String astCtorArgs)
Get a string for an expression to generate creating of an AST node.protected java.lang.String
getLookaheadTestExpression(Alternative alt, int maxDepth)
Generate a lookahead test expression for an alternate.protected java.lang.String
getLookaheadTestExpression(Lookahead[] look, int k)
protected java.lang.String
getLookaheadTestTerm(int k, BitSet p)
Generate a depth==1 lookahead test expression given the BitSet.java.lang.String
getRangeExpression(int k, int[] elems)
Return an expression for testing a contiguous renage of elements(package private) static boolean
isEmpty(java.lang.String s)
protected boolean
isspace(char c)
protected boolean
lookaheadIsEmpty(Alternative alt, int maxDepth)
Is the lookahead for this alt empty?java.lang.String
mapTreeId(java.lang.String idParam, ActionTransInfo transInfo)
Map an identifier to it's corresponding tree-node variable.protected void
od(java.lang.String s, int i, int end, java.lang.String msg)
protected void
printAction(java.lang.String s)
Print an action with leading tabs, attempting to preserve the current indentation level for multi-line actions Ignored if string is null.protected void
printActionCode(java.lang.String actionStr, int line)
protected void
printGrammarAction(Grammar grammar)
protected void
printMainFunc(java.lang.String s)
protected void
printTabs()
Create a Java code-generator using the given Grammar.protected java.lang.String
processActionCode(java.lang.String actionStr, int line)
protected java.lang.String
processActionForSpecialSymbols(java.lang.String actionStr, int line, RuleBlock currentRule, ActionTransInfo tInfo)
Lexically process $var and tree-specifiers in the action.void
setupOutput(java.lang.String className)
This method exists so a subclass, namely VAJCodeGenerator, can open the file in its own evil way.(package private) java.lang.String
toString(boolean v)
-
Methods inherited from class antlr.CodeGenerator
_print, _println, decodeLexerRuleName, elementsAreRange, encodeLexerRuleName, extractIdOfAction, extractTypeOfAction, genTokenInterchange, getBitsetName, getFIRSTBitSet, getFOLLOWBitSet, markBitsetForGen, print, println, processStringForASTConstructor, removeAssignmentFromDeclaration, reverseLexerRuleName, setAnalyzer, setBehavior, setGrammar, setTool
-
-
-
-
Field Detail
-
syntacticPredLevel
protected int syntacticPredLevel
-
genAST
protected boolean genAST
-
saveText
protected boolean saveText
-
labeledElementType
java.lang.String labeledElementType
-
labeledElementASTType
java.lang.String labeledElementASTType
-
labeledElementInit
java.lang.String labeledElementInit
-
commonExtraArgs
java.lang.String commonExtraArgs
-
commonExtraParams
java.lang.String commonExtraParams
-
commonLocalVars
java.lang.String commonLocalVars
-
lt1Value
java.lang.String lt1Value
-
exceptionThrown
java.lang.String exceptionThrown
-
throwNoViable
java.lang.String throwNoViable
-
initHeaderAction
public static final java.lang.String initHeaderAction
- See Also:
- Constant Field Values
-
mainHeaderAction
public static final java.lang.String mainHeaderAction
- See Also:
- Constant Field Values
-
lexerClassName
java.lang.String lexerClassName
-
parserClassName
java.lang.String parserClassName
-
treeWalkerClassName
java.lang.String treeWalkerClassName
-
currentRule
RuleBlock currentRule
Tracks the rule being generated. Used for mapTreeId
-
currentASTResult
java.lang.String currentASTResult
Tracks the rule or labeled subrule being generated. Used for AST generation.
-
treeVariableMap
java.util.Hashtable treeVariableMap
Mapping between the ids used in the current alt, and the names of variables used to represent their AST values.
-
declaredASTVariables
java.util.Hashtable declaredASTVariables
Used to keep track of which AST variables have been defined in a rule (except for the #rule_name and #rule_name_in var's
-
astVarNumber
int astVarNumber
-
NONUNIQUE
protected static final java.lang.String NONUNIQUE
Special value used to mark duplicate in treeVariableMap
-
caseSizeThreshold
public static final int caseSizeThreshold
- See Also:
- Constant Field Values
-
-
Method Detail
-
printTabs
protected void printTabs()
Create a Java code-generator using the given Grammar. The caller must still call setTool, setBehavior, and setAnalyzer before generating code.- Overrides:
printTabs
in classCodeGenerator
-
addSemPred
protected int addSemPred(java.lang.String predicate)
Adds a semantic predicate string to the sem pred vector These strings will be used to build an array of sem pred names when building a debugging parser. This method should only be called when the debug option is specified
-
exitIfError
public void exitIfError()
-
checkCurrentOutputStream
protected void checkCurrentOutputStream()
-
extractIdOfAction
protected java.lang.String extractIdOfAction(java.lang.String s, int line, int column)
Get the identifier portion of an argument-action. For Python the ID of an action is assumed to be everything before the assignment, as Python does not support a type.- Overrides:
extractIdOfAction
in classCodeGenerator
- Parameters:
s
- The action textline
- Line used for error reporting.column
- Line used for error reporting.- Returns:
- A string containing the text of the identifier
-
extractTypeOfAction
protected java.lang.String extractTypeOfAction(java.lang.String s, int line, int column)
Get the type portion of an argument-action. Python does not have a type declaration before an identifier, so we just return the empty string.- Overrides:
extractTypeOfAction
in classCodeGenerator
- Parameters:
s
- The action textline
- Line used for error reporting.- Returns:
- A string containing the text of the type
-
flushTokens
protected void flushTokens()
-
gen
public void gen()
Generate the parser, lexer, treeparser, and token types in Java- Specified by:
gen
in classCodeGenerator
-
gen
public void gen(ActionElement action)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The {...} action to generate
-
gen
public void gen(AlternativeBlock blk)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The "x|y|z|..." block to generate
-
gen
public void gen(BlockEndElement end)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The block-end element to generate. Block-end elements are synthesized by the grammar parser to represent the end of a block.
-
gen
public void gen(CharLiteralElement atom)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The character literal reference to generate
-
toString
java.lang.String toString(boolean v)
-
gen
public void gen(CharRangeElement r)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The character-range reference to generate
-
gen
public void gen(LexerGrammar g) throws java.io.IOException
Generate the lexer Java file- Specified by:
gen
in classCodeGenerator
- Throws:
java.io.IOException
-
genHeaderMain
protected void genHeaderMain(Grammar grammar)
-
genHeaderInit
protected void genHeaderInit(Grammar grammar)
-
printMainFunc
protected void printMainFunc(java.lang.String s)
-
gen
public void gen(OneOrMoreBlock blk)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The (...)+ block to generate
-
gen
public void gen(ParserGrammar g) throws java.io.IOException
Generate the parser Java file- Specified by:
gen
in classCodeGenerator
- Throws:
java.io.IOException
-
gen
public void gen(RuleRefElement rr)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The rule-reference to generate
-
gen
public void gen(StringLiteralElement atom)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The string-literal reference to generate
-
gen
public void gen(TokenRangeElement r)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The token-range reference to generate
-
gen
public void gen(TokenRefElement atom)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The token-reference to generate
-
gen
public void gen(TreeElement t)
Description copied from class:CodeGenerator
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
-
gen
public void gen(TreeWalkerGrammar g) throws java.io.IOException
Generate the tree-parser Java file- Specified by:
gen
in classCodeGenerator
- Throws:
java.io.IOException
-
gen
public void gen(WildcardElement wc)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
wc
- The wildcard element to generate
-
gen
public void gen(ZeroOrMoreBlock blk)
Generate code for the given grammar element.- Specified by:
gen
in classCodeGenerator
- Parameters:
blk
- The (...)* block to generate
-
genAlt
protected void genAlt(Alternative alt, AlternativeBlock blk)
Generate an alternative.- Parameters:
alt
- The alternative to generateblk
- The block to which the alternative belongs
-
genBitsets
protected void genBitsets(Vector bitsetList, int maxVocabulary)
Generate all the bitsets to be used in the parser or lexer Generate the raw bitset data like "long _tokenSet1_data[] = {...}" and the BitSet object declarations like "BitSet _tokenSet1 = new BitSet(_tokenSet1_data)" Note that most languages do not support object initialization inside a class definition, so other code-generators may have to separate the bitset declarations from the initializations (e.g., put the initializations in the generated constructor instead).- Parameters:
bitsetList
- The list of bitsets to generate.maxVocabulary
- Ensure that each generated bitset can contain at least this value.
-
genBlockInitAction
protected void genBlockInitAction(AlternativeBlock blk)
Generate the init action for a block, which may be a RuleBlock or a plain AlternativeBLock.
-
genBlockPreamble
protected void genBlockPreamble(AlternativeBlock blk)
Generate the header for a block, which may be a RuleBlock or a plain AlternativeBLock. This generates any variable declarations and syntactic-predicate-testing variables.
-
genCases
protected void genCases(BitSet p)
Generate a series of case statements that implement a BitSet test.- Parameters:
p
- The Bitset for which cases are to be generated
-
genCommonBlock
public PythonBlockFinishingInfo genCommonBlock(AlternativeBlock blk, boolean noTestForSingle)
Generate common code for a block of alternatives; return a postscript that needs to be generated at the end of the block. Other routines may append else-clauses and such for error checking before the postfix is generated. If the grammar is a lexer, then generate alternatives in an order where alternatives requiring deeper lookahead are generated first, and EOF in the lookahead set reduces the depth of the lookahead. @param blk The block to generate @param noTestForSingle If true, then it does not generate a test for a single alternative.
-
genASTDeclaration
protected void genASTDeclaration(AlternativeElement el)
-
genASTDeclaration
protected void genASTDeclaration(AlternativeElement el, java.lang.String node_type)
-
genASTDeclaration
protected void genASTDeclaration(AlternativeElement el, java.lang.String var_name, java.lang.String node_type)
-
genHeader
protected void genHeader()
Generate a header that is common to all Python files
-
genLexerTest
protected void genLexerTest()
Generate an automated test for Python CharScanner (sub)classes.
-
genMatch
protected void genMatch(BitSet b)
-
genMatch
protected void genMatch(GrammarAtom atom)
-
genMatchUsingAtomText
protected void genMatchUsingAtomText(GrammarAtom atom)
-
genMatchUsingAtomTokenType
protected void genMatchUsingAtomTokenType(GrammarAtom atom)
-
genNextToken
public void genNextToken()
Generate the nextToken() rule. nextToken() is a synthetic lexer rule that is the implicit OR of all user-defined lexer rules.
-
genRule
public void genRule(RuleSymbol s, boolean startSymbol, int ruleNum)
Gen a named rule block. ASTs are generated for each element of an alternative unless the rule or the alternative have a '!' modifier. If an alternative defeats the default tree construction, it must set_AST to the root of the returned AST. Each alternative that does automatic tree construction, builds up root and child list pointers in an ASTPair structure. A rule finishes by setting the returnAST variable from the ASTPair. - Parameters:
rule
- The name of the rule to generatestartSymbol
- true if the rule is a start symbol (i.e., not referenced elsewhere)
-
genSemPred
protected void genSemPred(java.lang.String pred, int line)
-
genSemPredMap
protected void genSemPredMap()
Write an array of Strings which are the semantic predicate expressions. The debugger will reference them by number only
-
genSynPred
protected void genSynPred(SynPredBlock blk, java.lang.String lookaheadExpr)
-
genTokenStrings
public void genTokenStrings()
Generate a static array containing the names of the tokens, indexed by the token type values. This static array is used to format error messages so that the token identifers or literal strings are displayed instead of the token numbers. If a lexical rule has a paraphrase, use it rather than the token label.
-
genTokenASTNodeMap
protected void genTokenASTNodeMap()
Create and set Integer token type objects that map to Java Class objects (which AST node to create).
-
genTokenTypes
protected void genTokenTypes(TokenManager tm) throws java.io.IOException
Generate the token types Java file- Throws:
java.io.IOException
-
getASTCreateString
public java.lang.String getASTCreateString(Vector v)
Get a string for an expression to generate creation of an AST subtree.- Specified by:
getASTCreateString
in classCodeGenerator
- Parameters:
v
- A Vector of String, where each element is an expression in the target language yielding an AST node.
-
getASTCreateString
public java.lang.String getASTCreateString(GrammarAtom atom, java.lang.String astCtorArgs)
Get a string for an expression to generate creating of an AST node- Specified by:
getASTCreateString
in classCodeGenerator
- Parameters:
atom
- The grammar node for which you are creating the nodestr
- The arguments to the AST constructor
-
getASTCreateString
public java.lang.String getASTCreateString(java.lang.String astCtorArgs)
Get a string for an expression to generate creating of an AST node. Parse the first (possibly only) argument looking for the token type. If the token type is a valid token symbol, ask for it's AST node type and add to the end if only 2 arguments. The forms are #[T], #[T,"t"], and as of 2.7.2 #[T,"t",ASTclassname].- Parameters:
str
- The arguments to the AST constructor
-
getLookaheadTestExpression
protected java.lang.String getLookaheadTestExpression(Lookahead[] look, int k)
-
getLookaheadTestExpression
protected java.lang.String getLookaheadTestExpression(Alternative alt, int maxDepth)
Generate a lookahead test expression for an alternate. This will be a series of tests joined by '&&' and enclosed by '()', the number of such tests being determined by the depth of the lookahead.
-
getLookaheadTestTerm
protected java.lang.String getLookaheadTestTerm(int k, BitSet p)
Generate a depth==1 lookahead test expression given the BitSet. This may be one of: 1) a series of 'x==X||' tests 2) a range test using >= && <= where possible, 3) a bitset membership test for complex comparisons- Parameters:
k
- The lookahead levelp
- The lookahead set for level k
-
getRangeExpression
public java.lang.String getRangeExpression(int k, int[] elems)
Return an expression for testing a contiguous renage of elements- Parameters:
k
- The lookahead levelelems
- The elements representing the set, usually from BitSet.toArray().- Returns:
- String containing test expression.
-
lookaheadIsEmpty
protected boolean lookaheadIsEmpty(Alternative alt, int maxDepth)
Is the lookahead for this alt empty?
-
mapTreeId
public java.lang.String mapTreeId(java.lang.String idParam, ActionTransInfo transInfo)
Map an identifier to it's corresponding tree-node variable. This is context-sensitive, depending on the rule and alternative being generated- Specified by:
mapTreeId
in classCodeGenerator
- Parameters:
idParam
- The identifier name to map- Returns:
- The mapped id (which may be the same as the input), or null if the mapping is invalid due to duplicates
-
processActionForSpecialSymbols
protected java.lang.String processActionForSpecialSymbols(java.lang.String actionStr, int line, RuleBlock currentRule, ActionTransInfo tInfo)
Lexically process $var and tree-specifiers in the action. This will replace #id and #(...) with the appropriate function calls and/or variables etc...- Specified by:
processActionForSpecialSymbols
in classCodeGenerator
-
isEmpty
static boolean isEmpty(java.lang.String s)
-
processActionCode
protected java.lang.String processActionCode(java.lang.String actionStr, int line)
-
printActionCode
protected void printActionCode(java.lang.String actionStr, int line)
-
setupOutput
public void setupOutput(java.lang.String className) throws java.io.IOException
This method exists so a subclass, namely VAJCodeGenerator, can open the file in its own evil way. JavaCodeGenerator simply opens a text file...- Throws:
java.io.IOException
-
isspace
protected boolean isspace(char c)
-
_printAction
protected void _printAction(java.lang.String s)
Description copied from class:CodeGenerator
Print an action without leading tabs, attempting to preserve the current indentation level for multi-line actions Ignored if string is null.- Overrides:
_printAction
in classCodeGenerator
- Parameters:
s
- The action string to output
-
od
protected void od(java.lang.String s, int i, int end, java.lang.String msg)
-
printAction
protected void printAction(java.lang.String s)
Description copied from class:CodeGenerator
Print an action with leading tabs, attempting to preserve the current indentation level for multi-line actions Ignored if string is null.- Overrides:
printAction
in classCodeGenerator
- Parameters:
s
- The action string to output
-
printGrammarAction
protected void printGrammarAction(Grammar grammar)
-
_printJavadoc
protected void _printJavadoc(java.lang.String s)
-
genJavadocComment
protected void genJavadocComment(Grammar g)
-
genJavadocComment
protected void genJavadocComment(RuleSymbol g)
-
-