Angel \”Java\” Lopez on Blog

June 8, 2016

New Month’s Resolutions: June 2016

Filed under: Blockchain, C Sharp, JavaScript, NodeJs, Open Source Projects — ajlopez @ 9:01 am

Let’s review my previous new month’s resolutions:

– Improve WangTiles [pending]
– Improve WangTilesJS [complete] see repo
– Improve CrysSharp [complete] see repo
– Improve BlockchainSharp [complete] see repo
– Improve SimpleBlockchain [complete] see repo
– Prepare and give a Bitcoin related talk [complete]

I also worked on:

– Improve SimpleForth [complete] see repo

My new month’s resolutions:

– Improve WangTiles
– Improve WangTilesJS
– Improve CrysSharp
– Improve GoSharp 
– Improve SimpleForth
– Improve BlockchainSharp
– Improve SimpleBlockchain

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

May 30, 2016

Building A Blockchain (11)

Previous Post
Next Post

I refactored the blockchain implementation in my personal JavaScript/NodeJS project:

https://github.com/ajlopez/SimpleBlockchain

In the previous post I showed some tests I wrote using TDD (Test-Driven Development) workflow. Now, I want to show the current implementation, that is an evolution from the past week one.

The implementation is at:

https://github.com/ajlopez/SimpleBlockchain/blob/master/lib/blockchains.js

There is a JavaScript “class”, implemented as a function:

function Blockchain(block) {
    var self = this;
    var blocks = [block];
    var blockstore = stores.blockstore();
    blockstore.save(block);
    
    this.bestBlock = function () { return blocks[blocks.length - 1]; }

In my new implementation, I’m using a block store. This object was able to save a block, and retrieve a block by hash, or retrieve the blocks with same number. But now, I added the retrieve by parent hash: that is, given a block hash, blockstore can retrieve all its know children. Given such functionality, the blockchain implementation evolved to use that feature. In this way, I don’t need to have a tree of blocks: given a block, I can retrieve all the tree of its descendants, from the blockstore.

Notice that the state of the blockchain has a naive implementation: only a JavaScript array, where the index coincides with the block number, starting with genesis block having 0 as number. My plan is to refactor this implementation, to support thousands or millions of blocks, using a block store based on disk. But now, using this naive implementation, I could explore the behavior of the node application.

The exposed method to add a block is:

this.add = function (block) {
    if (blockstore.hasBlockHash(block.hash))
        return;
        
    blockstore.save(block);
    
    if (getUnknownAncestor(block) != null)
        return;
    
    tryAdd(block);
}

If the block is know, then it implies it was already processed, then return.

If not, the block is saved in the in-memory store, and the first unknown ancestor hash is calculated. Maybe, the store has not ALL the ancestor chain up to genesis block. In this case, I cannot process the block.

If the block has a chain of ancestor that connects it to the genesis block, then I try to add the block to the blockchain:

function tryAdd(block) {
    if (isBestBlockChild(block))
        blocks.push(block);
    else if (isBetterBestBlock(block))
        tryFork(block);
        
    tryChildren(block);
}

It it is a direct child of the best block, it is added to the block (remember, a naive implementation, a simple array). If not, but it is a better block (it has a higher number), a fork to the block is performed. In any case, the children of the block are processed: maybe, the new block is the missing link to new candidate chains based on previously known blocks that were disconnected from genesis until the arrival of the new block.

These are the predicates I wrote:

function isBestBlockChild(block) {
    var bblock = self.bestBlock();
    
    return bblock.hash === block.parentHash && bblock.number === block.number - 1;
}

function isBetterBestBlock(block) {
    var bblock = self.bestBlock();
    
    return bblock.number < block.number;
}

This is the process of the children:

function tryChildren(block) {
    var children = blockstore.getChildren(block.hash);

    for (var n in children)
        tryAdd(children[n]);
}

Note: again, it is a naive implementation, that implies a recursion using tryAdd, that calls tryChildren. I could refactor to avoid recursion, a pending task.

Given a block, this function calculates the upward chain to its first ancestor included in the current blockchain, and then change the blockchain to have that chain:

function tryFork(block) {
    var newbranch = [block];
    var parentHash = block.parentHash;
    
    while (parentHash && blockstore.hasBlockHash(parentHash)) {
        var parent = blockstore.getByHash(parentHash);
        
        if (parent.hash === blocks[parent.number].hash)
            return switchToFork(newbranch);
            
        newbranch.push(parent);
        
        parentHash = parent.parentHash;
    }
}

The find of the first unknow ancestor of a block:

function getUnknownAncestor(block) {
    var parentHash = block.parentHash;

    while (parentHash && blockstore.hasBlockHash(parentHash)) {
        var parent = blockstore.getByHash(parentHash);
        
        if (parent.hash === blocks[parent.number].hash)
            return null;
        
        parentHash = parent.parentHash;
    }
    
    return parentHash;
}

The change of the blockchain to a new fork:

function switchToFork(newbranch) {
    for (var n = newbranch.length; n-- > 0😉 {
        var block = newbranch[n];
        blocks[block.number] = block;
    }
}

All the test passed. As I mentioned, my plan is to improve the implementation. But, as usual in my TDD workflow, I prefer to give baby steps, make it works, and only then, make it right and make it fast. I had good experiences using this way of writing code, obtaining simple and solid implementations, having all the test to help me to change any internal implementation, without pain.

Encouraged by this result, I also started to refactor my blockchain implementation in my personal C# project, too:

https://github.com/ajlopez/BlockchainSharp

I started to use a block store with GetByParentHash into my BlockChain code:

https://github.com/ajlopez/BlockchainSharp/blob/master/Src/BlockchainSharp/Core/BlockChain.cs

I wrote about my previous implementation in this post.

Stay tuned!

Angel “Java” Lopez

@ajlopez

May 9, 2016

Building A Blockchain (8)

Previous Post
Next Post

In previous post, I wrote about the virtual machine I’m implementing in my personal blockchain project, in C#, using TDD (Test-Driven Development) workflow:

https://github.com/ajlopez/SimpleBlockchain

I added a simple bytecode compiler, in order to simplify some tests. The class is not needed for production code, but it is included in the core project:

public class BytecodeCompiler
{
    private IList<byte> bytes = new List<byte>();

    public void Stop()
    {
        this.Compile(Bytecodes.Stop);
    }

    public void Add()
    {
        this.Compile(Bytecodes.Add);
    }

    public void Subtract()
    {
        this.Compile(Bytecodes.Subtract);
    }
    
    // more methods
    
    public byte[] ToBytes()
    {
        return this.bytes.ToArray();
    }
}

It collects a series of bytecodes, with optional arguments. There is a method to return a byte array with the compiled program. A sample test:

[TestMethod]
public void LessThan()
{
    BytecodeCompiler compiler = new BytecodeCompiler();

    compiler.Push(2);
    compiler.Push(2);
    compiler.LessThan();
    compiler.Push(0);
    compiler.Push(1);
    compiler.LessThan();
    compiler.Push(1);
    compiler.Push(0);
    compiler.LessThan();

    Machine machine = new Machine();

    machine.Execute(compiler.ToBytes());

    var stack = machine.Stack;

    Assert.IsNotNull(stack);
    Assert.AreEqual(3, stack.Size);
    Assert.AreEqual(DataWord.Zero, stack.ElementAt(2));
    Assert.AreEqual(DataWord.Zero, stack.ElementAt(1));
    Assert.AreEqual(DataWord.One, stack.ElementAt(0));
}

I also wrote a simple line compiler, named SimpleCompiler. Now, I can use it in my machine tests:

[TestMethod]
public void IsZeroUsingSimpleCompiler()
{
    string program = "push 2\n" +
        "iszero\n" +
        "push 0\n" +
        "iszero";

    SimpleCompiler compiler = new SimpleCompiler(program);

    Machine machine = new Machine();

    machine.Execute(compiler.Compile());

    var stack = machine.Stack;

    Assert.IsNotNull(stack);
    Assert.AreEqual(2, stack.Size);
    Assert.AreEqual(DataWord.Zero, stack.ElementAt(1));
    Assert.AreEqual(DataWord.One, stack.ElementAt(0));
}

These are auxiliary classes, but they simplify the writing of new tests. It is part of my simplification strategy: after some initial and long tests, try to simplify the writing of the newer ones.

In the next post, a surprise: a new blockchain project, but using NodeJS, JavaScript.

Stay tuned!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

May 8, 2016

New Month’s Resolutions: May 2016

Filed under: C Sharp, JavaScript, NodeJs, Open Source Projects, Wang Tiles — ajlopez @ 5:27 pm

Time to write my new month’s resolutions, and review the previous month’s ones:

– Improve WangTiles [complete] see repo
– Improve BlockchainSharp [complete] see repo
– Start Blockchain in JavaScript [complete] see repo
– Work on EthSharp [pending]
– Improve SimpleGA [pending]
– Improve AjGenesisNode-Express [pending]
– Work on CrysJS [pending]
– Work on CrysSharp [pending]

Additionally, I worked on:

– Improve SimpleForth [complete] see repo
– Start WangTilesJS [complete] see repo

My new month’s resolutions are:

– Improve WangTiles
– Improve WangTilesJS
– Improve CrysSharp
– Improve BlockchainSharp
– Improve SimpleBlockchain
– Prepare and give a Bitcoin related talk

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

May 2, 2016

Building A Blockchain (7)

Previous Post
Next Post

In my personal blockchain project:

https://github.com/ajlopez/BlockchainSharp

I want to execute simple code chain, using a virtual machine. The programs are usually called smart contracts. I adopted the Ethereum virtual machine design (read Ethereum Yellow Paper). Some classes:

A DataWord represents a number using 32 bytes. I implemented simple arithmetic operations using System.Numerics.BigInteger internally. I can create DataWord from integers or from byte arrays.

The Stack is an stack of DataWords, manipulated by the Machine. There is an enumeration, named Bytecodes, with the values of byte codes to be executed in the machine implementation. A program consists of a series of bytecodes, contained in a byte array. The Machine.Execute method is the place where the bytecodes are executed, manipulating the stack. Excerpt:

public void Execute(byte[] bytecodes)
{
    int pc = 0;
    int pl = bytecodes.Length;

    while (pc < pl)
    {
        byte bytecode = bytecodes[pc++];

        switch (bytecode)
        {
            case (byte)Bytecodes.Stop:
                return;
            case (byte)Bytecodes.Add:
                this.stack.Push(this.stack.Pop().Add(this.stack.Pop()));
                break;
            case (byte)Bytecodes.Multiply:
                this.stack.Push(this.stack.Pop().Multiply(this.stack.Pop()));
                break;
            case (byte)Bytecodes.Subtract:
                this.stack.Push(this.stack.Pop().Subtract(this.stack.Pop()));
                break;
                
            // more operations
        }
    }
}

I will add Storage and Memory to the execution of a program. The Storage will be associated and persisted with the contract account. Each contract is an account, with address, balance, but with code and storage state. The Memory will be created and used during the execution of the program, but it won’t be persisted: it is a temporary memory, used only at execution time.

I have a simple bytecode compiler, and a line compiler, to facilitate the creation of programs.

Next posts: description of compilers, block execution with transactions, storage and memory, persistent stores.

Keep tuned!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

April 25, 2016

Building A Blockchain (6)

Previous Post
Next Post

These days, I added transaction processing to my personal project:

https://github.com/ajlopez/BlockchainSharp

In the previous post, I described the immutable Trie structure. Now, I have an AccountState that is saved by hash in a such Trie:

public class AccountState
{
    private BigInteger balance;

    public AccountState(BigInteger balance)
    {
        if (BigInteger.Compare(BigInteger.Zero, balance) > 0)
            throw new InvalidOperationException("Invalid balance");

        this.balance = balance;
    }

    public BigInteger Balance { get { return this.balance; } }

    public AccountState AddToBalance(BigInteger amount)
    {
        return new AccountState(BigInteger.Add(this.balance, amount));
    }

    public AccountState SubtractFromBalance(BigInteger amount)
    {
        return new AccountState(BigInteger.Subtract(this.balance, amount));
    }
}

I decided to implement the Ethereum way: having an account state, instead of inputs and outputs. The only property is Balance, but I will add more data. Notice that negative balances are rejeceted. Then, I added a TransactionProcessor:

public class TransactionProcessor
{
    private Trie<AccountState> states;

    public TransactionProcessor(Trie<AccountState> states)
    {
        this.states = states;
    }

    public Trie<AccountState> States { get { return this.states; } }

    public bool ExecuteTransaction(Transaction transaction)
    {
        var states = this.states;

        try
        {
            foreach (var av in transaction.Inputs)
            {
                var addr = av.Address.ToString();
                var state = states.Get(addr);
                var newstate = state.SubtractFromBalance(av.Value);
                states = states.Put(addr, newstate);
            }

            foreach (var av in transaction.Outputs)
            {
                var addr = av.Address.ToString();
                var state = states.Get(addr);
                var newstate = state.AddToBalance(av.Value);
                states = states.Put(addr, newstate);
            }

            this.states = states;

            return true;
        }
        catch (Exception ex)
        {
            return false;
        }
    }
}

If the transaction is processed, a new account state trie is generated, and ExecuteTransaction returns true. If the transaction is rejected, the initial accout state trie still has the original values. A typical test:

[TestMethod]
public void ExecuteTransaction()
{
    var transaction = CreateTransaction(100);

    var addr1 = transaction.Inputs.First().Address;
    var addr2 = transaction.Outputs.First().Address;

    var states = new Trie<AccountState>(new AccountState(BigInteger.Zero));

    states = states.Put(addr1.ToString(), new AccountState(new BigInteger(200)));

    var processor = new TransactionProcessor(states);

    Assert.IsTrue(processor.ExecuteTransaction(transaction));

    var newstates = processor.States;

    Assert.IsNotNull(newstates);
    Assert.AreNotSame(states, newstates);

    Assert.AreEqual(new BigInteger(200), states.Get(addr1.ToString()).Balance);
    Assert.AreEqual(BigInteger.Zero, states.Get(addr2.ToString()).Balance);

    Assert.AreEqual(new BigInteger(100), newstates.Get(addr1.ToString()).Balance);
    Assert.AreEqual(new BigInteger(100), newstates.Get(addr2.ToString()).Balance);
}

The auxiliary CreateTransaction method creates a transaction with an amount, and two random addresses.

I’m thinking to have only one sender account and receiving account per transaction, as in Ethereum. The change is easy, I have all the TDD tests to help me to do redesigns without a lot of work.

Next topics: executing blocks with transactions, saving state in persistent store, virtual machine and its bytecodes, a simple compiler, etc…

Stay tuned!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

April 17, 2016

Building A Blockchain (5)

Filed under: Bitcoin, Blockchain, C Sharp, Ethereum, Test-Driven Development — ajlopez @ 10:02 am

Previous Post
Next Post

I was working a lot on my personal project:

https://github.com/ajlopez/BlockchainSharp

implementing a blockchain in C#, using TDD (Test-Driven Development) workflow. One element I need to implement, is the store of account states (initially their balances). The balance of an account should be retrieved by account id (a hash). But in many use cases, I need to know the balances of accounts at a given time. It is not enough to have the LATESTS balances. So, I implemented a data structure, a trie, but with immutable properties.

A trie is a tree where the non-leaf nodes stores part of the key:

In the above example, the value V1 is associated with key AAA, and the value V5 is associated with key ACA. When I change the value associated with key ABC from V4 to V7, part of a new trie is created, so the original one persists without modification:

I can access the original key/values using the “old root”, and then, I can switch to use the “new root”, at any moment. If any node/root is not reachable from a variable, garbage collector will release the memory associated with that node.

I wrote a typed trie implementation, using TDD. An example test:

[TestMethod]
public void PutAndGetKeyValue()
{
    Trie<string> trie = new Trie<string>();

    var trie2 = trie.Put("012", "foo");

    Assert.IsNotNull(trie2);
    Assert.AreNotSame(trie2, trie);
    Assert.IsNull(trie.Get("012"));
    Assert.AreEqual("foo", trie2.Get("012"));
}

An example that shows the persistent of tries after updates:

[TestMethod]
public void ReplaceValue()
{
    Trie<string> trie = new Trie<string>();

    var trie2 = trie.Put("012", "foo");
    var trie3 = trie2.Put("012", "bar");

    Assert.IsNotNull(trie2);
    Assert.AreNotSame(trie2, trie);
    Assert.IsNull(trie.Get("012"));
    Assert.AreEqual("foo", trie2.Get("012"));

    Assert.IsNotNull(trie3);
    Assert.AreNotSame(trie3, trie2);
    Assert.AreEqual("bar", trie3.Get("012"));
}

My idea is to use Trie<AccountState> as a store for account balances. At the end of a block (with transactions) processing, there is an account state trie. Ant the end of the next block processing, another trie will be generated. At any moment, I could retrieve the account balances at time “block 1”, and at time “block 2”, using the generated tries.

Stay tuned!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

April 11, 2016

Building A Blockchain (4)

Filed under: Bitcoin, Blockchain, C Sharp, Ethereum, Open Source Projects — ajlopez @ 10:20 am

Previous Post
Next Post

In this post, I want to describe the key parts of a blockchain example I’m writing at:

https://github.com/ajlopez/BlockchainSharp

using C# and TDD (Test-Driven Development). The essential things to be implemented are:

– Blocks
– Transactions

the concepts of:

– Account
– Account State
– Smart Contracts (executed in a virtual machine)

and utilities for:

– Encode/Decode entities to be transmited/received via the network of nodes
– Hash calculations
– Immutable tries
– Sign transactions
– Verify transaction sign
– Verify block integrity
– Verify transactions are valid
– Proof of Work calculation (as usual in Bitcoin and Ethereum implementations)
– Store some entities: complete blocks, partial blocks (block headers in Ethereum), account state, … in general, tries with storage

Some days ago, I started to write a Transaction class, having pairs of accounts and values. I adopted Ethereum approach: instead of having inputs/outputs as in Bitcoin, I have accounts with balances/states. Each account is identified by an address (actually a random hexadecimal string):

Each transaction can have one or more inputs and outputs. Each input/output has an account address and a value. The values are instances of BigInteger, so I can manage large integer numbers. It’s the way the criptocurrencies implementations allow the use of value fractions: the unity is a power of 10, and the amounts are expressed in integer quantity of minimal fractions.

Next steps: add transactions to blocks, apply transaction to account states, validate that input values are available, store states in an immutable trie,…

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

April 9, 2016

New Month’s Resolutions: Aprill 2016

Filed under: Blockchain, C Sharp, Ethereum, JavaScript, NodeJs, Open Source Projects, SimpleForth — ajlopez @ 4:34 pm

A new month started, and it’s time to write the new resolutions. And to review the past month ones:

– Improve AjGenesisNode-Express [pending]
– Work on CrysJS [complete] see repo
– Work on CrysSharp [pending]
– Improve my SimpleGA samples [pending
– Work on SharpGo [pending]
– Work on EthSharp [complete] see repo

Also I worked on:

– Improve SimpleForth in JavaScript [complete] see repo
– Start WangTiles in C# [complete] see repo
– Publish new version of SimpleArgs [complete] see repo
– Start BlockchainSharp in C# [complete] see repo
– Improve SimpleDT samples [complete] see repo
– Move AjGo to GitHub [complete] see repo
– Improve RuScript [complete] see repo
– Add Code Coverate to ethereumjs rlp [complete] see repo
– New Machine Learning in JavaScript presentation [complete] see repo see talk

My new resolutions:

– Improve WangTiles
– Improve BlockchainSharp
– Start Blockchain in JavaScript
– Work on EthSharp
– Improve SimpleGA
– Improve AjGenesisNode-Express
– Work on CrysJS
– Work on CrysSharp

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

April 4, 2016

Building A Blockchain (3)

Previous Post
Next Post

I added a lot of code to my simple blockchain implementation written in C#:

https://github.com/ajlopez/BlockchainSharp

As usual, I followed TDD (Test-Driven Development) workflow, pursuing simplicity, doing baby steps. In the last post I mentioned the use of a DSL (Domain Specific Language) to specify some block processing tests that have complex setup.

Initially, I wrote tests like:

[TestMethod]
public void ProcessTwoBlocksAndTwoUncles()
{
    Block genesis = new Block(0, null);
    Block block = new Block(1, genesis.Hash);
    Block uncle1 = new Block(1, genesis.Hash);
    Block uncle2 = new Block(2, uncle1.Hash);

    BlockProcessor processor = new BlockProcessor();

    processor.Process(genesis);
    processor.Process(block);
    processor.Process(uncle1);
    processor.Process(uncle2);

    Assert.IsNotNull(processor.BlockChain);
    Assert.AreEqual(2, processor.BlockChain.BestBlockNumber);
    Assert.AreEqual(genesis, processor.BlockChain.GetBlock(0));
    Assert.AreEqual(uncle1, processor.BlockChain.GetBlock(1));
    Assert.AreEqual(uncle2, processor.BlockChain.GetBlock(2));
}

The idea is:

– Create some blocks

– Send the blocks to the block processor

– Check the blocks in the block chain

The created blocks are related by parent relationships. Sometimes, a competing block is created, so the block processor must deal with block branches.

But the setup code could be long and complex. So, I wrote a DSL, and now I have tests like:

[TestMethod]
public void SendTwoBlocksAndTwoUncles()
{
    var processor = new BlockProcessor();
    var dsl = new BlockProcessorDsl(processor);

    dsl.Run(new string[] 
    {
        "chain g0 b1 b2",
        "chain b1 c2 c3",
        "send b1 b2",
        "send c2 c3",
        "top c3"
    });
}

Each command is an string, with verb and arguments. The block g0 is the genesis block. The verb “chain” enumerates a list of blocks to be created, each block is child of its previous block. The verb “send” sends the created blocks to the block processor. The verb “top” checks if the specified block is the top block in the current blockchain.

The result should not depend on order of arriving blocks, ie:

[TestMethod]
public void SendTwoBlocksInReversedOrder()
{
    var processor = new BlockProcessor();
    var dsl = new BlockProcessorDsl(processor);

    dsl.Run(new string[] 
    {
        "chain g0 b1 b2",
        "send b2 b1",
        "top b2"
    });
}

I could add text files, each one specifying command lines to be executed as sentences of the DSL.

In the next posts: core implementations needed by a blockchain, byte serialization, and implementing immutable tries for storing states.

Stay tuned!

Angel “Java” Lopez

http://www.ajlopez.com

http://twitter.com/ajlopez

« Newer PostsOlder Posts »

Blog at WordPress.com.