Introduction

Link to GitHub

Developing P2P systems is hard, and blockchains are no different. A developer can't just focus on the functionality of a node, but needs to consider the dynamics of the network, the interaction of nodes and emergent system properties. A lot of testing can't be addressed by unit testing, but needs a special kind of integration testing, where the code runs in interaction with other nodes, attempting to simulate real network scenarios.

In the context of Avalanche, Subnets are a special focus which requires new tooling and support for playing, working and testing with this unique feature of the Avalanche ecosystem.

The ANR aims at being a tool for developers and system integrators alike, offering functionality to run networks of AvalancheGo nodes with support for custom node, Subnet and network configurations, allowing to locally test code before deploying to Mainnet or even public testnets like fuji.

You can also use the Avalanche Network Runner Postman collection.

Note that this tool is not for running production nodes, and that because it is being heavily developed right now, documentation might differ slightly from the actual code.

Installation

curl -sSfL https://raw.githubusercontent.com/ava-labs/avalanche-network-runner/main/scripts/install.sh | sh

The script installs the binary inside the ~/bin directory. If the directory doesn't exist, it will be created.

Please make sure that ~/bin is in your $PATH:

To add it to your path permanently, add an export command to your shell initialization script. If you run bash, use .bashrc. If you run zsh, use .zshrc.

Furthermore, AVALANCHEGO_EXEC_PATH should be set properly in all shells you run commands related to Avalanche Network Runner. We strongly recommend that you put the following in to your shell's configuration file.

# replace execPath with the path to AvalancheGo on your machine
# e.g., ${HOME}/go/src/github.com/ava-labs/avalanchego/build/avalanchego
AVALANCHEGO_EXEC_PATH="${HOME}/go/src/github.com/ava-labs/avalanchego/build/avalanchego"

Unless otherwise specified, file paths given below are relative to the root of this repository.

Usage

There are two main ways to use the network-runner:

• Run ANR as a binary

  This is the recommended approach for most use cases. Doesn't require Golang installation and provides a RPC server with an HTTP API and a client library for easy interaction.

• Import this repository into your go program

  This allows for custom network scenarios and high flexibility, but requires more code to be written.

Running the binary, the user can send requests to the RPC server in order to start a network, create Subnets, add nodes to the network, remove nodes from the network, restart nodes, etc. You can make requests through the avalanche-network-runner command or by making API calls. Requests are "translated" into gRPC and sent to the server.

Each node can then also be reached via API endpoints which each node exposes.

Examples

When running with the binary, ANR runs a server process as an RPC server which then waits for API calls and handles them. Therefore we run one shell with the RPC server, and another one for issuing calls.

Start the Server

avalanche-network-runner server \
--log-level debug \
--port=":8080" \
--grpc-gateway-port=":8081"

Note that the above command will run until you stop it with CTRL + C. Further commands will have to be run in a separate terminal.

The RPC server listens to two ports:

• port: the main gRPC port (see gRPC).
• grpc-gateway-port: the gRPC gateway port (see gRPC-gateway), which allows for HTTP requests.

When using the binary to issue calls, the main port will be hit. In this mode, the binary executes compiled code to issue calls. Alternatively, plain HTTP can be used to issue calls, without the need to use the binary. In this mode, the grpc-gateway-port should be queried.

Each of the examples below will show both modes, clarifying its usage.

Run Queries

Ping the Server

curl -X POST -k http://localhost:8081/v1/ping -d ''

or

avalanche-network-runner ping \
--log-level debug \
--endpoint="0.0.0.0:8080"

Start a New Avalanche Network with Five Nodes

curl -X POST -k http://localhost:8081/v1/control/start -d '{"execPath":"'${AVALANCHEGO_EXEC_PATH}'","numNodes":5}'

or

avalanche-network-runner control start \
--log-level debug \
--endpoint="0.0.0.0:8080" \
--number-of-nodes=5 \
--avalanchego-path ${AVALANCHEGO_EXEC_PATH}

Additional optional parameters which can be passed to the start command:

  --plugin-dir ${AVALANCHEGO_PLUGIN_PATH} \
  --blockchain-specs '[{"vm_name":"subnetevm","genesis":"/tmp/subnet-evm.genesis.json"}]' \
	--global-node-config '{"index-enabled":false, "api-admin-enabled":true,"network-peer-list-gossip-frequency":"300ms"}' \
	--custom-node-configs" '{"node1":{"log-level":"debug","api-admin-enabled":false},"node2":{...},...}'

--plugin-dir and --blockchain-specs are parameters relevant to Subnet operation.

--plugin-dir can be used to indicate to ANR where it will find plugin binaries for your own VMs. It is optional. If not set, ANR will assume a default location which is relative to the avalanchego-path given.

--blockchain-specs specifies details about how to create your own blockchains. It takes a JSON array for each blockchain, with the following possible fields:

   "vm_name": human readable name for the VM
   "genesis": path to a file containing the genesis for your blockchain (must be a valid path)

See the Avalanche-CLI documentation for details about how to create and run Subnets with our Avalanche-CLI tool.

The network-runner supports AvalancheGo node configuration at different levels.

1. If neither --global-node-config nor --custom-node-configs is supplied, all nodes get a standard set of config options. Currently this set contains:

{
 "network-peer-list-gossip-frequency": "250ms",
 "network-max-reconnect-delay": "1s",
 "public-ip": "127.0.0.1",
 "health-check-frequency": "2s",
 "api-admin-enabled": true,
 "index-enabled": true
}

2. --global-node-config is a JSON string representing a single AvalancheGo config, which will be applied to all nodes. This makes it easy to define common properties to all nodes. Whatever is set here will be combined with the standard set above.

3. --custom-node-configs is a map of JSON strings representing the complete network with individual configs. This allows to configure each node independently. If set, --number-of-nodes will be ignored to avoid conflicts.

4. The configs can be combined and will be merged, that is one could set global --global-node-config entries applied to each node, and also set --custom-node-configs for additional entries.

5. Common --custom-node-configs entries override --global-node-config entries which override the standard set.

6. The following entries will be ignored in all cases because the network-runner needs to set them internally to function properly:

  --log-dir
  --db-dir
  --http-port
  --staking-port
  --public-ipc

Wait for All the Nodes in the Cluster to Become Healthy

curl -X POST -k http://localhost:8081/v1/control/health -d ''

or

avalanche-network-runner control health \
--log-level debug \
--endpoint="0.0.0.0:8080"

The response to this call is actually pretty large, as it contains the state of the whole cluster. At the very end of it there should be a text saying healthy:true (it would say false if it wasn't healthy).

Get API Endpoints of All Nodes in the Cluster

curl -X POST -k http://localhost:8081/v1/control/uris -d ''

or

avalanche-network-runner control uris \
--log-level debug \
--endpoint="0.0.0.0:8080"

Query Cluster Status from the Server

curl -X POST -k http://localhost:8081/v1/control/status -d ''

or

avalanche-network-runner control status \
--log-level debug \
--endpoint="0.0.0.0:8080"

Stream Cluster Status

avalanche-network-runner control \
--request-timeout=3m \
stream-status \
--push-interval=5s \
--log-level debug \
--endpoint="0.0.0.0:8080"

Remove (Stop) a Node

curl -X POST -k http://localhost:8081/v1/control/removenode -d '{"name":"node5"}'

or

avalanche-network-runner control remove-node node5 \
--request-timeout=3m \
--log-level debug \
--endpoint="0.0.0.0:8080" \

Restart a Node

In this example we are stopping the node named node1.

Note: By convention all node names start with node and a number. We suggest to stick to this convention to avoid issues.

# e.g., ${HOME}/go/src/github.com/ava-labs/avalanchego/build/avalanchego
AVALANCHEGO_EXEC_PATH="avalanchego"

Note that you can restart the node with a different binary by providing

curl -X POST -k http://localhost:8081/v1/control/restartnode -d '{"name":"node1","execPath":"'${AVALANCHEGO_EXEC_PATH}'"}'

or

avalanche-network-runner control restart-node node1 \
--request-timeout=3m \
--log-level debug \
--endpoint="0.0.0.0:8080" \
--avalanchego-path ${AVALANCHEGO_EXEC_PATH}

Add a Node

In this example we are adding a node named node99.

# e.g., ${HOME}/go/src/github.com/ava-labs/avalanchego/build/avalanchego
AVALANCHEGO_EXEC_PATH="avalanchego"

Note that you can add the new node with a different binary by providing

curl -X POST -k http://localhost:8081/v1/control/addnode -d '{"name":"node99","execPath":"'${AVALANCHEGO_EXEC_PATH}'"}'

or

avalanche-network-runner control add-node node99 \
--request-timeout=3m \
--endpoint="0.0.0.0:8080" \
--avalanchego-path ${AVALANCHEGO_EXEC_PATH}

It's also possible to provide individual node config parameters:

	--node-config '{"index-enabled":false, "api-admin-enabled":true,"network-peer-list-gossip-frequency":"300ms"}'

--node-config allows to specify specific AvalancheGo config parameters to the new node. See here for the reference of supported flags.

Note: The following parameters will be ignored if set in --node-config, because the network runner needs to set its own in order to function properly: --log-dir --db-dir

Note: The following Subnet parameters will be set from the global network configuration to this node: --track-subnets --plugin-dir

Terminate the Cluster

Note that this will still require to stop your RPC server process with Ctrl-C to free the shell.

curl -X POST -k http://localhost:8081/v1/control/stop -d ''

or

avalanche-network-runner control stop \
--log-level debug \
--endpoint="0.0.0.0:8080"

Subnets

For general Subnet documentation, please refer to Subnets. ANR can be a great helper working with Subnets, and can be used to develop and test new Subnets before deploying them in public networks. However, for a smooth and guided experience, we recommend using Avalanche-CLI. These examples expect a basic understanding of what Subnets are and their usage.

RPC Server Subnet-EVM Example

The Subnet-EVM is a simplified version of Coreth VM (C-Chain). This chain implements the Ethereum Virtual Machine and supports Solidity smart-contracts as well as most other Ethereum client functionality. It can be used to create your own fully Ethereum-compatible Subnet running on Avalanche. This means you can run your Ethereum-compatible dApps in custom Subnets, defining your own gas limits and fees, and deploying solidity smart-contracts while taking advantage of Avalanche's validator network, fast finality, consensus mechanism and other features. Essentially, think of it as your own Ethereum where you can concentrate on your business case rather than the infrastructure. See Subnet-EVM for further information.

Using Avalanche Network as a Library

The Avalanche Network Runner can also be imported as a library into your programs so that you can use it to programmatically start, interact with and stop Avalanche networks. For an example of using the Network Runner in a program, see an example.

Creating a network is as simple as:

network, err := local.NewDefaultNetwork(log, binaryPath)

where log is a logger of type logging.Logger and binaryPath is the path of the AvalancheGo binary that each node that exists on network startup will run.

For example, the below snippet creates a new network using default configurations, and each node in the network runs the binaries at /home/user/go/src/github.com/ava-labs/avalanchego/build:

network, err := local.NewDefaultNetwork(log,"/home/user/go/src/github.com/ava-labs/avalanchego/build")

Once you create a network, you must eventually call Stop() on it to make sure all of the nodes in the network stop. Calling this method kills all of the Avalanche nodes in the network. You probably want to call this method in a defer statement to make sure it runs.

To wait until the network is ready to use, use the network's Healthy method. It returns a channel which will be notified when all nodes are healthy.

Each node has a unique name. Use the network's GetNodeNames() method to get the names of all nodes.

Use the network's method GetNode(string) to get a node by its name. For example:

names, _ := network.GetNodeNames()
node, _ := network.GetNode(names[0])

Then you can make API calls to the node:

id, _ := node.GetAPIClient().InfoAPI().GetNodeID() // Gets the node's node ID
balance, _ := node.GetAPIClient().XChainAPI().GetBalance(address,assetID,false) // Pretend these arguments are defined

After a network has been created and is healthy, you can add or remove nodes to/from the network:

newNode, _ := network.AddNode(nodeConfig)
err := network.RemoveNode(names[0])

Where nodeConfig is a struct which contains information about the new node to be created. For a local node, the most important elements are its name, its binary path and its identity, given by a TLS key/cert.

You can create a network where nodes are running different binaries -- just provide different binary paths to each:

  stakingCert, stakingKey, err := staking.NewCertAndKeyBytes()
  if err != nil {
   return err
  }
  nodeConfig := node.Config{
    Name: "New Node",
    ImplSpecificConfig: local.NodeConfig{
      BinaryPath: "/tmp/my-avalanchego/build",
    },
    StakingKey:  stakingKey,
    StakingCert: stakingCert,
  }

After adding a node, you may want to call the network's Healthy method again and wait until the new node is healthy before making API calls to it.

Creating Custom Networks

To create custom networks, pass a custom config (the second parameter) to the local.NewNetwork(logging.Logger, network.Config) function. The config defines the number of nodes when the network starts, the genesis state of the network, and the configs for each node.

Please refer to NetworkConfig for more details.