TDD and Code Kata: Writing a Lexer for AjRuby

Usually, I write programs as code katas, to train me in some software development topics. During last months, I wrote many mini interpreters, using TDD-like approach, code coverage, and C#. Past week, I was working in the kickoff of yet another one, AjRuby. The starting point: the lexer.

A lexer is a program that analizes a source code text, and returns the tokens, the "words" that compose the program. In the case of AjRuby, I wrote a lexer that follows the Ruby language grammar. It’s not complete yet, but it’s working.

You can see my progress at my code katas Google code project:

http://code.google.com/p/ajcodekatas/source/browse#svn/trunk/AjRuby

The solution has a library project, and a test project:

It has a Token class, that represents each token:

    public class Token
    {
        public string Value { get; set; }

        public TokenType TokenType { get; set; }
    }

The TokenType is an enum:

    public enum TokenType
    {
        Name,
        Integer,
        Real,
        Boolean,
        String,
        Operator,
        Separator
    }

There is a LexerTest.cs class, containing tests like:

        [TestMethod]
        public void ShouldProcessName()
        {
            Lexer lexer = new Lexer("name");

            Token token;

            token = lexer.NextToken();

            Assert.IsNotNull(token);
            Assert.AreEqual(TokenType.Name, token.TokenType);
            Assert.AreEqual("name", token.Value);

            token = lexer.NextToken();

            Assert.IsNull(token);
        }

There is a test for every kind of token to be processed. Other example:

        [TestMethod]
        public void ShouldProcessNameWithInitialUnderscore()
        {
            Lexer lexer = new Lexer("_name");

            Token token;

            token = lexer.NextToken();

            Assert.IsNotNull(token);
            Assert.AreEqual(TokenType.Name, token.TokenType);
            Assert.AreEqual("_name", token.Value);

            token = lexer.NextToken();

            Assert.IsNull(token);
        }

I wrote the tests using TDD ideas: write the test, compile with error, fix the compiler issues, run the tests in red, fix them, get green status. I skipped most of the refactor steps. I guess the tested code could be improved, but by now, I’m trying to get a working lexer. The tests are in green state, good!

My goal is to have > 80% of code coverage, at any time during the project development:

It’s good to write something only for fun and learning. For me, it’s part of the continuous improvement every software developer should pursuit.

Angel "Java" Lopez


http://www.ajlopez.com/en

http://twitter.com/ajlopez