Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Moving Validators

The lighthouse validator-manager move command uses the VC HTTP API to move validators from one VC (the "src" VC) to another VC (the "dest" VC). The move operation is comprehensive; it will:

  • Disable the validators on the src VC.
  • Remove the validator keystores from the src VC file system.
  • Export the slashing database records for the appropriate validators from the src VC to the dest VC.
  • Enable the validators on the dest VC.
  • Generally result in very little or no validator downtime.

It is capable of moving all validators on the src VC, a count of validators or a list of pubkeys.

The move command is only guaranteed to work between two Lighthouse VCs (i.e., there is no guarantee that the commands will work between Lighthouse and Teku, for instance).

The move command only supports moving validators using a keystore on the local file system, it does not support Web3Signer validators.

Although all efforts are taken to avoid it, it's possible for the move command to fail in a way that removes the validator from the src VC without adding it to the dest VC. Therefore, it is recommended to never use the move command without having a backup of all validator keystores (e.g. the mnemonic).

Simple Example

The following command will move all validators from the VC running at http://localhost:6062 to the VC running at http://localhost:5062.

lighthouse \
	validator-manager \
	move \
	--src-vc-url http://localhost:6062 \
	--src-vc-token ~/src-token.txt \
	--dest-vc-url http://localhost:5062 \
	--dest-vc-token ~/.lighthouse/mainnet/validators/api-token.txt \
	--validators all

Detailed Guide

This guide describes the steps to move validators between two validator clients (VCs) which are able to SSH between each other. This guide assumes experience with the Linux command line and SSH connections.

There will be two VCs in this example:

  • The source VC which contains the validators/keystores to be moved.
  • The destination VC which is to take the validators/keystores from the source.

There will be two hosts in this example:

  • Host 1 ("source host"): Is running the src-vc.
  • Host 2 ("destination host"): Is running the dest-vc.

The example assumes that Host 1 is able to SSH to Host 2.

In reality, many host configurations are possible. For example:

  • Both VCs on the same host.
  • Both VCs on different hosts and the validator-manager being used on a third host.

1. Configure the Source VC

The source VC needs to have the following flags at a minimum:

  • --http
  • --http-allow-keystore-export

Therefore, the source VC command might look like:

lighthouse \
    vc \
    --http \
    --http-allow-keystore-export

2. Configure the Destination VC

The destination VC needs to have the following flags at a minimum:

  • --http
  • --enable-doppelganger-protection

Therefore, the destination VC command might look like:

lighthouse \
    vc \
    --http \
    --enable-doppelganger-protection

The --enable-doppelganger-protection flag is not strictly required, however it is recommended for an additional layer of safety. It will result in 2-3 epochs of downtime for the validator after it is moved, which is generally an inconsequential cost in lost rewards or penalties.

Optionally, users can add the --http-store-passwords-in-secrets-dir flag if they'd like to have the import validator keystore passwords stored in separate files rather than in the validator-definitions.yml file. If you don't know what this means, you can safely omit the flag.

3. Obtain the Source API Token

The VC API is protected by an API token. This is stored in a file on each of the hosts. Since we'll be running our command on the destination host, it will need to have the API token for the source host on its file-system.

On the source host, find the location of the api-token.txt file and copy the contents. The location of the file varies, but it is located in the "validator directory" of your data directory, alongside validator keystores. For example: ~/.lighthouse/mainnet/validators/api-token.txt. If you are unsure of the api-token.txt path, you can run curl http://localhost:5062/lighthouse/auth which will show the path.

Copy the contents of that file into a new file on the destination host at ~/src-token.txt. The API token is a random string, e.g., hGut6B8uEujufDXSmZsT0thnxvdvKFBvh.

4. Create an SSH Tunnel

In the source host, open a terminal window, SSH to the destination host and establish a reverse-SSH connection between the destination host and the source host.

ssh dest-host
ssh -L 6062:localhost:5062 src-host

It's important that you leave this session open throughout the rest of this tutorial. If you close this terminal window then the connection between the destination and source host will be lost.

5. Move

With the SSH tunnel established between the dest-host and src-host, from the destination host run the command to move the validators:

lighthouse \
	validator-manager \
	move \
	--src-vc-url http://localhost:6062 \
	--src-vc-token ~/src-token.txt \
	--dest-vc-url http://localhost:5062 \
	--dest-vc-token ~/.lighthouse/mainnet/validators/api-token.txt \
	--validators all

The command will provide information about the progress of the operation and emit Done. when the operation has completed successfully. For example:

Running validator manager for mainnet network
Validator client is reachable at http://localhost:5062/ and reports 2 validators
Validator client is reachable at http://localhost:6062/ and reports 0 validators
Moved keystore 1 of 2
Moved keystore 2 of 2
Done.

At the same time, lighthouse vc will log:

INFO Importing keystores via standard HTTP API, count: 1
INFO Enabled validator                       voting_pubkey: 0xab6e29f1b98fedfca878edce2b471f1b5ee58ee4c3bd216201f98254ef6f6eac40a53d74c8b7da54f51d3e85cacae92f, signing_method: local_keystore
INFO Modified key_cache saved successfully

Once the operation completes successfully, there is nothing else to be done. The validators have been removed from the src-host and enabled at the dest-host. If the --enable-doppelganger-protection flag was used it may take 2-3 epochs for the validators to start attesting and producing blocks on the dest-host. If you would only like to move some validators, you can replace the flag --validators all with one or more validator public keys. For example:

lighthouse \
	validator-manager \
	move \
	--src-vc-url http://localhost:6062 \
	--src-vc-token ~/src-token.txt \
	--dest-vc-url http://localhost:5062 \
	--dest-vc-token ~/.lighthouse/mainnet/validators/api-token.txt \
	--validators 0x9096aab771e44da149bd7c9926d6f7bb96ef465c0eeb4918be5178cd23a1deb4aec232c61d85ff329b54ed4a3bdfff3a,0x90fc4f72d898a8f01ab71242e36f4545aaf87e3887be81632bb8ba4b2ae8fb70753a62f866344d7905e9a07f5a9cdda1

Note: If you have the validator-monitor-auto turned on, the source beacon node may still be reporting the attestation status of the validators that have been moved:

INFO Previous epoch attestation(s) success   validators: ["validator_index"], epoch: 100000, service: val_mon, service: beacon

This is fine as the validator monitor does not know that the validators have been moved (it does not mean that the validators have attested twice for the same slot). A restart of the beacon node will resolve this.

Any errors encountered during the operation should include information on how to proceed. Assistance is also available on our Discord.