For the past 3 months I’ve been working on designing an algorithm to compress random data. Read on to learn about the extreme difficulty in solving this problem and why my work eventually lead me to building a small super computer to assist in finding a solution.

Why compress random data

Existing compression algorithms are based on finding simple patterns in data or reducing the quality of a message. They cannot handle random data. What this means in practice is in order to squeeze more content down the tubes (the Internet is a series of tubes) companies like YouTube and Netflix reduce the quality of their content to compensate.

You can only use this approach so many times before the content starts to look too shitty for your customers to enjoy. The obvious downside is this approach involves loss. You’re not enjoying how the message originally looked. It’s been warped to fit your...

Algorithms like merkle trees and bloom filters allow you to take a group of data items and construct a unique fingerprint of the set. The fingerprint can then be used as a way to prove that some or all members in the group are part of that “fingerprint.”

These constructs are useful and you seem them used a lot in blockchains. The issue for me with these data structures is really the size of the meta data relative to the set size: Bloom filters suck because they only compress the original data like 30%; Merkle trees suck because you still need to retain a shit-load of meta-data to be able to construct proofs.

I think in some situations you might already have a large list of candidates and you want to compactly see if a candidate is in a set. I’ve been thinking about this problem today and I actually think that hashcash might be the solution here. The idea would be to generate an IV that...

Reworking the algorithm. Most likely there is no good algorithm

If one were to control a “funded” private key and agree to run a program that controlled the release of that private key, there would be nothing to stop the owner from failing to run that program. Some people aren’t honest and would seek to disrupt the process. And yet: if you were to repeat this same experiment at scale some people would still stay honest. Why is that? Well, not everyone is unethical (like the people in Japan who routinely return wallets with their full sums intact!) Then there are thieves who are only thieves on occasion. That is to say – if someone found a wallet full of money on an isolated street they would take it.

Opportunistic attackers aren’t the kind of people who would rob a bank or trouble themselves to steal money. In our experiment, you could defeat these attackers with a trusted execution environment or TEE. But what happens if we stop our security there...

If you’ve ever used a SIM card before then, you already know you need one to place calls. But what you might not know is a SIM card is actually a fully functional computer, complete with its own RAM, “disk space”, and operating system [uuic-hw-spec][uuic-sw-spec].

The latest SIMs are called “USIMs” and they are universal integrated circuit cards (UICC) programmed to store subscriber details. A UICC has about 16 KB of RAM– that means with four of these you have as much memory as an original Commodore 64 and it would all fit in your wallet [u/sim-mem]!

If you find that crazy wait until you learn that these cards can be powered through the air using magnetic fields. Bank cards operate using the same chips – they use UICC too – which means you can turn an average SIM into a swipe card if you wanted [nfc]. But I won’t go into that today. I want to talk more about eSIM cards, they’re even...

A game of war for decentralized threat detection:

• One side plays the attacker
• The other side plays defence
• The field is a virtual machine
• The ombudsman is software that monitors the VMs health
• The ombudsman is concerned with what has happened to the VM
• Are key system files damaged?
• Have files been locked / crypted by malware?
• Are certain network resources hijacked?
• Attackers submit vectors to disrupt the VM
• Defenders submit vectors to protect the VM
• False positive and negative problem: design the ombudsman to accept random nonce values that impact the tests, i.e. measure performance at date stamp X rather than at a fixed date each time.
• Statistically validity becomes more apparent over time.
• Reward function is yet unknown: but it could be a dividend system based on stopping or evading vectors – I find the potential here fascinating
• I should note because the ombudsman can observe the...

After reading many papers on blockchain scalability over the years, I am starting to see the painful truth that none of these systems can be scaled. For years I thought that some genius researcher would find a way to do it, and we would get blockchains with on-chain Visa scalability with the same security properties as Bitcoin… but that just hasn’t happened.

Here’s why that is currently impossible and maybe a path forwards…

The consensus basket

If you make a list of all the properties that make up a good consensus system you might end up with a basket with the following things inside it:

• Decentralization
• Censorship resistance
• Byzantine fault tolerance
• Open, equal participation
• Privacy / pseudo-anonymity

Unfortunately, you cannot have everything and keep scalablity.

If you want to have more scalable p2p networks just reduce the number of nodes, and lose decentralization. If you...

The problem with resource coins

Resource coins like Storj and Filecoin are systems that let anyone contribute their spare computational resources in exchange for payment. By using Storj, a person can lease out their extra hard drive space to other people and receive Storjcoins in return (a kind of special “utility token” in quotes…) The idea is very cool, but functionally it has problems.

The most significant problem, in my opinion, is the way in which these systems are designed to let anyone sell resources on an open market. At first impression, this might sound like an excellent property to have for a decentralized network, but in this particular context, it means that anyone can sell resources even if there is little demand for it.

In a cryptoeconomic system, this property is highly undesirable as an unregulated supply of storage space will cause wild fluctuations in the price of...

Survivability of an organism refers to its ability to survive in a worst case scenario. Any animal that has been caged or raised in captivity has had its survivability reduced. A caged animal cannot survive in the wild as well as a non-caged animal because its survival instincts have been altered (and often even its ability to reproduce has been adversely affected.)

An ICO-token is like a caged animal, removed from any cryptoeconomic context. In the cryptoeconomic wilderness, these tokens would be considered mistakes of nature, with no means to survive by themselves. They survive only in the context of an economic zoo, which has been artificially engineered for their sustained existence. If the company providing this zoo ceases to exist, then the shitcoin organism will not survive.

Centralization in the design of shitcoins reduces their innate survivability. We must not allow people to...