Mass Programming Language (4) Lexer and Parser

Previous Post
Next Post

In the Mass programming language implementation, I have an enumeration and a class:

A token represents a word of code to be processed. Lexer is in charge of separate the code in words/tokens. And Parse takes that token stream and returns expressions and commands:

Lexer constructor receives a string:

public Lexer(string text)
    this.text = text;

This string is processed to be separated into tokens. Notice the lexer distinguish between operators (like +) and separators (like parenthesis). It take into account the end of line as a token, too (in other programming language, like C, the end of line is simply a blank space). The main method of Lexer is NextToken that returns the next token in code text. In some situations, it is needed to save a consumend token, so there are methods like PushToken and its variants.

Internally, Parser manages a Lexer. You can get the next command calling ParseCommand method, and the next expression using ParseExpression. When the text is exhausted, those methods return null.

I should modify Lexer to consume a text stream, instead of a string, so it could process an input, like console.

I should think about unifying commands and expressions, a la Ruby where all is an expression.

Keep tuned!

Angel “Java” Lopez

4 thoughts on “Mass Programming Language (4) Lexer and Parser

  1. Pingback: Mass Programming Language (3) Commands | Angel "Java" Lopez on Blog

  2. Pingback: Mass Programming Language (4) Lexer And Parser | MVPs de LATAM

  3. Pingback: New Month’s Resolutions: June 2013 | Angel "Java" Lopez on Blog

  4. Pingback: Mass Programming Language (5) Variables Scope | Angel "Java" Lopez on Blog

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s