Introducing jr: A Peer-to-Peer Social Network

Intro

A friend of mine, John R., has been extolling the virtues of Clojure for years now. At one point, I bought Clojure for the Brave and True, read it, and implemented a few toy programs. It’s an excellent book. Recently I found myself looking for a language to implement a project to measure analyze a Secure Scuttlebutt style social network. While trying to decide between Guile, Chicken Scheme, and Racket I remembered John’s advice and instead went with Clojure. This project is named for John R.: jr (pronounced junior). Consequently, this project is also a tiny simulation of the ideas used in a real, featureful, established social network: Secure Scuttlebutt. This project is very much ssb’s junior.

You can find the source code for jr here.

Data Stored on the jr Network

The jr network consists of nodes which pass messages back and forth. Each node has an ed25519 keypair. A node’s public key is its identity on the network and most data sets just use the public key. Node’s use their private key to sign all of their messages. All messages include a timestamp.

Nodes maintain three main sets of data:

Following Network

A set of nodes that the node would like to follow. When a user decides to follow another node they announce it publicly by creating a following message in their message set.

Extended Network

A set of nodes being followed as well as the nodes their following network nodes are following. This is the friend-of-a-friend concept. You store messages not just for your friends, but also for the friends of your friends.

Messages

Each node maintains a list of messages that they have discovered or created. Each message is guaranteed to have the following mappings:

It should be noted that the signature is an ed25519 signature of the EDN representation of a message map with the :public and :signature mappings removed.

Currently the only supported message is a following message. A following message is a public announcement that one node is following another. It is used to build the extended network. A following messages adds the following mapping:

Syncing

In this type of network, nodes connect to other nodes opportunistically. There is no central, coordinated exchange. When two nodes sync each node gets all the message from the peer node that are authored by any nodes in its extended network. These messages are then added to the nodes set of messages. You can think of it like talking to someone getting information about only the people you know or may encounter later (friends-of-friends).

Extended Network Sync Simulation

Now that we understand how the jr network works, lets run a simulation to demonstrate how the extended network grows and to showcase some of the simulation features currently built into jr. Specifically we are going to be testing in-memory node syncing.

core/extended-sync-test creates three nodes and gives them the aliases “Alice”, “Bob”, and “Carol” using sim/create-aliases. Aliases are used by sim/pprint to print friendly strings instead of byte arrays for keys. Alice then follows Bob and Bob follows Carol. When Alice syncs with Bob, she should end up with Carol in her extended network (as Carol is a friend of a friend). core/extended-sync-test prints out the state of the network at each phase.

This is our initial network state. You can see what data a node stores (public, private, messages, following, and extended):

({:public "Alice",
  :private nil,
  :messages #{},
  :following #{},
  :extended #{}}

 {:public "Bob",
  :private nil,
  :messages #{},
  :following #{},
  :extended #{}}

 {:public "Carol",
  :private nil,
  :messages #{},
  :following #{},
  :extended #{}})

As you can see everyone has an empty message set, is not following anyone, and does not have anyone in their extended network. They do have public and private keys though. The private key is not printed out by sim/pprint as sim/create-aliases does not map private keys or signatures to a name.

In our second network state, Alice decides to follow Bob and Bob decides to follow Carol. This gives us the following state:

({:public "Alice",
  :private nil,
  :messages
  #{{:signature nil,
     :public "Alice",
     :following "Bob",
     :timestamp 1576107325835}},
  :following #{"Bob"},
  :extended #{"Bob"}}

 {:public "Bob",
  :private nil,
  :messages
  #{{:signature nil,
     :public "Bob",
     :following "Carol",
     :timestamp 1576107325846}},
  :following #{"Carol"},
  :extended #{"Carol"}}

 {:public "Carol",
  :private nil,
  :messages #{},
  :following #{},
  :extended #{}})

As you can see Bob is now in Alice’s following network and Alice’s extended network. Alice has also created a message saying she is following Bob so that when nodes sync with her they can update their own extended network.

Carol is now in Bob’s following network and extended network. Bob has also created a message saying he is following Carol.

Now to test the sync functionality grow our extended network, lets have Alice sync with Bob.

({:public "Alice",
  :private nil,
  :messages
  #{{:signature nil,
     :public "Bob",
     :following "Carol",
     :timestamp 1576107325846}
    {:signature nil,
     :public "Alice",
     :following "Bob",
     :timestamp 1576107325835}},
  :following #{"Bob"},
  :extended #{"Carol" "Bob"}}

 {:public "Bob",
  :private nil,
  :messages
  #{{:signature nil,
     :public "Bob",
     :following "Carol",
     :timestamp 1576107325846}},
  :following #{"Carol"},
  :extended #{"Carol"}}

 {:public "Carol",
  :private nil,
  :messages #{},
  :following #{},
  :extended #{}})

Alice now has two messages: a following message Alice created and Bob’s message saying Bob is following Carol. By going through Alice’s messages, Alice was able to update her extended network to include Carol.

While Alice may not be interested in what Carol has to say, her friend Bob is. As such, Alice will carry messages authored by Carol.

Summary

This is really just the tip of the iceberg when it comes to analysis of this network. I expect to expand on the code base quite a bit.

As simple as it is, it has provided few insights into the Clojure mindset. I frequently found myself designing functions that I thought were single-use but actually turned out to be applicable in the general sense. It’s just as easy to turn all the byte arrays in any combination of maps / lists into strings as it is to write a specific function to do that for the maps that nodes use. It’s easy to just pass a map and work with only the mappings you need, ignoring the others thereby inadvertently making an extensible message format.

I ended up spending a lot of time in the REPL poking, prodding, and trying before I bothered to put anything into a file. I also ended up spending many commutes just thinking about problems and then only a few minutes coding the solution. From a more practical lens Leiningen made it easy to get started and utilize dependencies. The Java interop made it possible to use Bouncy Castle as my crypto API.

So far, I’ve had a very positive Clojure experience. Thanks John!