Angel \”Java\” Lopez on Blog

May 25, 2013

Mass Programming Language (4) Lexer and Parser

Filed under: .NET, C Sharp, Mass, Open Source Projects, Programming Languages — ajlopez @ 10:38 am

Previous Post
Next Post

In the Mass programming language implementation, I have an enumeration and a class:

A token represents a word of code to be processed. Lexer is in charge of separate the code in words/tokens. And Parse takes that token stream and returns expressions and commands:

Lexer constructor receives a string:

public Lexer(string text)
{
    this.text = text;
}

This string is processed to be separated into tokens. Notice the lexer distinguish between operators (like +) and separators (like parenthesis). It take into account the end of line as a token, too (in other programming language, like C, the end of line is simply a blank space). The main method of Lexer is NextToken that returns the next token in code text. In some situations, it is needed to save a consumend token, so there are methods like PushToken and its variants.

Internally, Parser manages a Lexer. You can get the next command calling ParseCommand method, and the next expression using ParseExpression. When the text is exhausted, those methods return null.

I should modify Lexer to consume a text stream, instead of a string, so it could process an input, like console.

I should think about unifying commands and expressions, a la Ruby where all is an expression.

Keep tuned!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

4 Comments »

  1. […] Previous Post Next Post […]

    Pingback by Mass Programming Language (3) Commands | Angel "Java" Lopez on Blog — May 25, 2013 @ 10:39 am

  2. […] Mass Programming Language (4) Lexer And Parser […]

    Pingback by Mass Programming Language (4) Lexer And Parser | MVPs de LATAM — May 28, 2013 @ 8:18 pm

  3. […] version repo – Update ClojSharp [complete] repo – New posts about Mass [complete] post, post, post – New posts about Distributed Applications and Node.js [complete] […]

    Pingback by New Month’s Resolutions: June 2013 | Angel "Java" Lopez on Blog — June 2, 2013 @ 10:56 am

  4. […] Previos Post […]

    Pingback by Mass Programming Language (5) Variables Scope | Angel "Java" Lopez on Blog — July 12, 2014 @ 6:18 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Theme: Shocking Blue Green. Get a free blog at WordPress.com

Follow

Get every new post delivered to your Inbox.

Join 67 other followers

%d bloggers like this: