Category Archives: Scalability

Blockchain: Links And Resources (176)

Previous Post
Next Post

Ethereum’s First Production Scaling Project Is Here – But With a Catch
https://www.coindesk.com/ethereums-first-production-scaling-project-catch/

Vitalik Reveals New Idea for Plasma Scaling On Ethereum
https://www.coindesk.com/vitalik-reveals-new-idea-plasma-scaling-ethereum-event/

Ethereum’s Raiden Network Has New Scaling Competiton
https://www.coindesk.com/raiden-isnt-ethereums-hope-off-chain-payments-anymore/

Blockchain Engineering
https://ercim-news.ercim.eu/en110

Can ETH Keep Up vs. NEO and EOS in 2018? Scaling Ethereum | Casper Raiden Sharding Plasma
https://www.youtube.com/watch?v=FB42OWN3Z0w

State Tree Pruning
https://blog.ethereum.org/2015/06/26/state-tree-pruning/

The Ethereum-blockchain size will not exceed 1TB anytime soon
https://dev.to/5chdn/the-ethereum-blockchain-size-will-not-exceed-1tb-anytime-soon-58a

Blockchain Bibliography
https://p2preview.github.io/#/bibliography

Stay tuned!

Angel “Java” Lopez
https://github.com/ajlopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Scaling Ethereum/RSK: Multi Blockchains (2)

Previous Post

In the previous post I described a primary blockchain (based on Ethereum RSK) that could connect with a secondary level blockchains, using special transactions. In this post, I describe the mechanism to make it happens. Usually, an Ethereum RSK has transactions and a parent block:

The new trick is to have SOME blocks with TWO parents, connecting TWO blockchains (the primary one and a secondary blockchain), and special transactions that affects the TWO blockchains:

So, we could have two blockchains with normal blocks, with some special blocks that allows the execution of interchain transactions:

The primary blockchain could have special blocks with MORE THAN ONE secondary blockchain. But the secondary blockchains only allow the special blocks that connect with the primary one:

The normal blocks are mined by miners/producers in each blockchain. But the new special blocks ARE MINED by miners that keep BOTH BLOCKCHAINS states of word. There are machines that verifies the primary blockchain, other machines that verifies a secondary blockchain, and so on. But SOME machines verifies TWO blockchains.

Suppose we have the blue blockchain (the primary blockchain) and the green blockchain (a secondary one). Then, there is an special account (the bridge blue-green account) to be used in the special transactions. That bridge account in the blue blockchain has initial 0 balance. The same bridge account in the green blockchain has a total balance of the total amount managed in the blue (primary) blockchain (in RSK, the total amount is 21M of ethers, smart bitcoins).

A user signs a transaction in the blue blockchain, from his/her account TO bridge account. If a transaction has the blue-green bridge account as the target account, then IT IS AN SPECIAL transaction. For blue machines, that transaction transfer the specified value FROM user blue account, to blue bridge account. For green machines, the same transaction transfer the specified value from green bridge account to user green account.

Then, the user could use his/her green balance in green transactions. But at any moment, he/she could transfer from green user account to green bridge account. That is another special transaction: for green machines, it represents a transfer from user green account to green bridge account; for blue machines, it is a transfer from blue bridge account to blue user account.

Both green and blue user account, have the same address, public and private keys. It is up to the user decision to send a transaction to the green blockchain or to the blue blockchain.

In each secondary blockchain, there is only one bridge account. In the primary blockchain, there is a bridge account FOR EACH secondary blockchain.

A miner that keeps blue and green world state could build the special block. But other nodes could validate only the blue part or the green part. In this way, a secondary blockchain could add scalability to the primary one, and it could have its own fees and process time. A new secondary blockchain could be added when an special operation in primary blockchain creates a new bridge account. Nobody controls these accounts: there are target accounts in all special transactions. Only the sign of the source user account is needed.

In this way, the inter-blockchain transactions are build over the same blocks, you don’t need convoluted atomic swap mechanism, the same blocks execute the TWO transfers in one special transaction.

More about this mechanism, and miner incentive for bi-blockchain blocks, in next posts.

Stay tuned!

Angel “Java” Lopez
https://github.com/ajlopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Scaling Ethereum/RSK: Multi Blockchains (1)

Next Post

I already wrote some proposals in my other post series:

Connecting Blockchains
Multi-Blockchains in Ethereum/RSK
Offchain Transactions in Ethereum/RSK

commenting on using two or more blockchains (homegeneous or heterogeneous), and offchain transactions. They are alternative ways to explore solutions to scalability, with a fresh start and pursuing the simplest path in each idea.

After reviewing the multi-blockchain, now I think a have a simpler path. First, this proposal is about having many HOMOGENEOUS blockchains, Ethereum/RSK-like ones. One is the mainchain, and we could have n additional blockchains:

The main blockchain has the initial ether amount of ether to use. For sake of clarity, let’s start with a main blockchain, the blockchain zero, has an initial stock of 100M ethers (in case of RSK network, it could be the 21M smart bitcoins backed by the 21M bitcoins of the BTC mainnet),

All accounts are the same in all networks: same private, public keys, addresses. So, if you have control of a control in blockchain zero, you can use the same account IN OTHER blockchain.

But the secondary blockchains starts with 0 balance in each account.

The second idea is to have inter-chain transactions, to transfer from one account in source blockchain, to the SAME account in target blockchain:;

An special bridge account is used. There is a bridge account in mainchain FOR EACH secondary chain. So, if you transfer value in blockchain 0, to bridge account corresponding to secondary blockchain 3, THEN a similar transaction will appear in this secondary blockchain, transferring the same amout from the dedicated BRIDGE account. To allow such transfer, each bridge account in SECONDARY blockchain starts with 100M of initial balance,

If there are 10 secondary blockchain, then there are 10 BRIDGE ACCOUNTS in the main blockchain (with initial 0 balance), and ONE BRIDGE ACCOUNT in each secondary blockchain (with 100M inital balance),

You also can send a transaction to a secondary blockchain, transferring value back to the mainchain:

The motivation is to have scalabilty: if the mainchain is overloaded, we can use a secondary blockchain. Also, the different blockchain could have different gas price, and maybe other difference in contract execution cost.

We could use a free graph of blockchains. But limiting the proposal to a hierarchical schema with only one level simplifies the discussion. The transactions between secondary blockchains are not allowed: only main to secondary one are supported. And no generation of new Ether is allowed in the secondary blockchains. These restrictions exist to have a better control of the total amout of value.

I could imagine a secondary blockchain by world region, country, even by vertical market. Or competing blockchains, in gas price and contract execution cost. The client software (that is, a dapp, a mobile application) could simplify the user experience, doing some of these transfer inter-blockchain in automatic.

But, how to reflect one transaction to bridge in one blockchain, to other transaction in the target blockchain? My next post will descibe a simple solution, to be discussed and reviewed. But I think it is a good starting point, to avoid the complications of some atomic swap and alike implementions.

Stay tuned!

Stay tuned!

Angel “Java” Lopez
https://github.com/ajlopez
http://twitter.com/ajlopez
http://www.ajlopez.com

 

 

 

Scaling Ethereum/RSK (1)

I’m a member of development team at @RSKSmart. We are working implementing a blockchain based on Ethereum (Java version) that is connected with Bitcoin using a two way peg.

There are many ideas (not published yet) to improve scalability in Ethereum/RSK. In this new post series, I want to share a personal idea to achieve scalability: to run transactions in parallel, taking advantage of current processors.

Ethereum/RSK, as other blockchains, have blocks with transactions. The key difference with Bitcoin is that they support the execution of smart contracts. So, each transaction could be a simple value transference, or it could be a call to a contract method. Each created contract has an address, and an associated storage. The transactions in a block are executed in order, to obtain a deterministic result (state of world), at the end of its execution, in any node of the network. The whole block has an associated state of world, result of execute all its transactions in order, starting from previous block state of world.

But in many situations, two consecutive transactions are independent. The state of world at the end of their execution, is the same, without regarding the order of execution. Even calling the same contract, two transactions, in some cases, could be executed in parallel. Ie, a contract manages an asset/tokens by user, and transaction T1 alters the asset value associated to user U1, and transaction T2 alters the asset value associated to user U2. Both transactions could be interchangeable in order. And they could be executed in parallel. In this way, a contract is not the bottleneck of execution. The transaction semantic determines if the transactions are independent or not.

I was experimenting with lightweight implementations of tries, the data structure used to keep storage and calculate hash roots of state of worlds (see Building a blockchain (5)). My idea to parallelize the execution of transactions is to use a trie that support multithreading execution, detecting any conflict of read/write storage btw transactions. If N transactions can be executed without conflicts (ie, a conflict could be two transactions writing the same storage cell), then they could be executed in parallel. The base idea: use ideas from software transactional memory implementations (see Memory Transactions In AjSharp Using References).

Stay tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Scalability: Links, News And Resources (7)

Previous Post

What Does Your Webserver Do When a User Hits Refresh? — Ecommerce Blog by Shopify
http://www.shopify.com/technology/7535298-what-does-your-webserver-do-when-a-user-hits-refresh#axzz2O1D5ZhXG

VMware vFabric GemFire: High Performance Data Management for Cloud-Based Applications
http://www.vmware.com/products/application-platform/vfabric-gemfire

Indeed Engineering Blog » Blog Archive » From 1 to 1 Billion: Evolution of a Document Serving System
http://engineering.indeed.com/blog/2013/03/from-1-to-1-billion-part-1/

Scaling Node.js Applications | Colin J. Ihrig’s Blog
http://cjihrig.com/blog/scaling-node-js-applications/

Facebook kisses DRAM goodbye, builds memcached for flash — Tech News and Analysis
http://gigaom.com/2013/03/05/facebook-kisses-dram-goodbye-builds-memcached-for-flash/

Splout SQL
http://sploutsql.com/
MapReduce, using Pangool, from Spain

Pomelo home
http://pomelo.netease.com/

NetEase/pomelo · GitHub
https://github.com/NetEase/pomelo
Game development with Node.js

Scaling Facebook Engineering
http://www.infoq.com/presentations/Scaling-Facebook-Engineering

(10) Needle in a haystack: efficient storage of billions of photos
https://www.facebook.com/note.php?note_id=76191543919

Fully Loaded Node – A Node.JS Holiday Season, part 2 ✩ Mozilla Hacks – the Web developer blog
https://hacks.mozilla.org/2012/11/fully-loaded-node-a-node-js-holiday-season-part-2/

Structure:Data | GigaOM Events
http://event.gigaom.com/structuredata/

My Links
http://delicious.com/ajlopez/scalability

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Scalability: Links, News And Resources (6)

Previous Post
Next Post

What Does Your Webserver Do When a User Hits Refresh? — Ecommerce Blog by Shopify
http://www.shopify.com/technology/7535298-what-does-your-webserver-do-when-a-user-hits-refresh#axzz2O1D5ZhXG

VMware vFabric GemFire: High Performance Data Management for Cloud-Based Applications
http://www.vmware.com/products/application-platform/vfabric-gemfire

Indeed Engineering Blog » Blog Archive » From 1 to 1 Billion: Evolution of a Document Serving System
http://engineering.indeed.com/blog/2013/03/from-1-to-1-billion-part-1/

Scaling Node.js Applications | Colin J. Ihrig’s Blog
http://cjihrig.com/blog/scaling-node-js-applications/

Facebook kisses DRAM goodbye, builds memcached for flash — Tech News and Analysis
http://gigaom.com/2013/03/05/facebook-kisses-dram-goodbye-builds-memcached-for-flash/

Splout SQL
http://sploutsql.com/
MapReduce, using Pangool, from Spain

Pomelo home
http://pomelo.netease.com/

NetEase/pomelo · GitHub
https://github.com/NetEase/pomelo
Game development with Node.js

Scaling Facebook Engineering
http://www.infoq.com/presentations/Scaling-Facebook-Engineering

(10) Needle in a haystack: efficient storage of billions of photos
https://www.facebook.com/note.php?note_id=76191543919

Fully Loaded Node – A Node.JS Holiday Season, part 2 ✩ Mozilla Hacks – the Web developer blog
https://hacks.mozilla.org/2012/11/fully-loaded-node-a-node-js-holiday-season-part-2/

Structure:Data | GigaOM Events
http://event.gigaom.com/structuredata/

My Links
http://delicious.com/ajlopez/scalability

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez

Scalability: Links, News And Resources (5)

Previous Post
Next Post

Twitter’s programmers speed Hadoop development | Big Data – InfoWorld
http://www.infoworld.com/d/big-data/twitters-programmers-speed-hadoop-development-211931

Twitter’s programmers speed Hadoop development | Big Data – InfoWorld
http://www.infoworld.com/d/big-data/twitters-programmers-speed-hadoop-development-211931?source=IFWNLE_nlt_stradev_2013-02-05

Cross-Post: Windows Azure SQL Database and SQL Server — Performance and Scalability Compared and Contrasted – Windows Azure – Site Home – MSDN Blogs
http://blogs.msdn.com/b/windowsazure/archive/2013/02/01/cross-post-windows-azure-sql-database-and-sql-server-performance-and-scalability-compared-and-contrasted.aspx

Building A Node.JS Server That Won’t Melt – A Node.JS Holiday Season, part 5 ✩ Mozilla Hacks – the Web developer blog
https://hacks.mozilla.org/2013/01/building-a-node-js-server-that-wont-melt-a-node-js-holiday-season-part-5/

Disk-Locality in Datacenter Computing Considered Irrelevant
http://www.cs.berkeley.edu/~ganesha/talks/disk-irrelevant.pdf

bitly/nsq
https://github.com/bitly/nsq
realtime distributed message processing at scale

High Scalability – High Scalability – Switch your databases to Flash storage. Now. Or you’re doing it wrong.
http://highscalability.com/blog/2012/12/10/switch-your-databases-to-flash-storage-now-or-youre-doing-it.html

Gangnam Ons S4 Recording on 2012-11-07 1410-Vimeo1 +6db on Vimeo
http://vimeo.com/53261709

4store – Scalable RDF storage
http://4store.org/

Amazon Redshift
http://aws.amazon.com/redshift/
petabyte-scale data warehouse service in the cloud

High Scalability – High Scalability – Gone Fishin’: Tumblr Architecture – 15 Billion Page Views A Month And Harder To Scale Than Twitter
http://highscalability.com/blog/2012/11/19/gone-fishin-tumblr-architecture-15-billion-page-views-a-mont.html?87125f76=t

Expanding the Cloud – Announcing Amazon Redshift, a Petabyte-scale Data Warehouse Service – All Things Distributed
http://www.allthingsdistributed.com/2012/11/amazon-redshift.html

High Scalability – High Scalability – BigData using Erlang, C and Lisp to Fight the Tsunami of Mobile Data
http://highscalability.com/blog/2012/11/26/bigdata-using-erlang-c-and-lisp-to-fight-the-tsunami-of-mobi.html

Facebook News Feed: Social Data at Scale
http://www.infoq.com/presentations/Facebook-News-Feed

SQLFire: Scalable SQL instead of NoSQL
http://www.infoq.com/presentations/SQLFire-Scalable-SQL-instead-of-NoSQL

How to Scale Your Start-up | Inc. 5000
http://www.inc.com/karl-and-bill/how-to-scale-your-start-up.html

When the Nerds Go Marching In – Alexis C. Madrigal – The Atlantic
http://www.theatlantic.com/technology/archive/2012/11/when-the-nerds-go-marching-in/265325/
How a dream team of engineers from Facebook, Twitter, and Google built the software that drove Barack Obama’s reelection

Scaling Software with Akka
http://www.infoq.com/presentations/Scalability-Akka

Erlang Scales … Do You?
http://www.infoq.com/presentations/Erlang-Scalability

RethinkDB: An open-source distributed database built with love over three years | Hacker News
http://news.ycombinator.com/item?id=4763879

Twitter survives election after Ruby-to-Java move • The Register
http://www.theregister.co.uk/2012/11/08/twitter_epic_traffic_saved_by_java/

Cases – Two Screen – Angry Bytes
http://two-screen.tv/cases/

Windows Azure’s Flat Network Storage and 2012 Scalability Targets – Windows Azure – Site Home – MSDN Blogs
http://blogs.msdn.com/b/windowsazure/archive/2012/11/02/windows-azure-s-flat-network-storage-and-2012-scalability-targets.aspx

The Startup Hangover: Supporting 15M Users
http://www.infoq.com/presentations/Scallability-SoundCloud

My Links
http://delicious.com/ajlopez/scalability

Keep tuned!

Angel “Java” Lopez
http://www.ajlopez.com
http://twitter.com/ajlopez