How to set up a high-availability sequencer

IntroductionCopy Anchor Link

The sequencer is a critical component of your Arbitrum chain and queues transactions submitted to the network. It serves as the transaction-ordering engine, accepting transactions forwarded from full nodes, queuing them, and returning feed messages to the nodes. It subsequently sends queued transactions to batch posters for data posting.

If your sequencer goes offline, the network cannot process new transactions arriving at the child chain's RPC nodes, impacting the user experience. This guide provides detailed instructions for setting up a high-availability (HA) sequencer architecture to minimize downtime and ensure your Arbitrum chain remains operational even if individual components fail.

PrerequisitesCopy Anchor Link

Before you begin, ensure you have:

• Experience with Kubernetes and container orchestration
• Access to a Kubernetes cluster with multiple availability zones
• Understanding of Redis and cloud infrastructure
• A properly configured parent chain node or RPC node endpoint
• Sequencer keys and permissions
• Sufficient storage and compute resources

High-availability sequencer architectureCopy Anchor Link

A high-availability sequencer deployment consists of seven key components:

1. CDN/Load Balancer: Manages traffic routing and bot detection
2. Nitro full nodes: Process read requests and forward write transactions
3. External Relays: Handle public feed traffic
4. Sequencer Relays: Combine feeds from all sequencers
5. Sequencers: Multiple redundant transaction queuing machines
6. Redis: Coordinates active sequencer selection and sequencer-related information sharing
7. Batch Poster: Posts transaction batches to the parent chain

Architecture diagramsCopy Anchor Link

The architecture varies slightly depending on whether your full nodes use Redis to identify the active sequencer or forward transactions to a predefined endpoint.

Send to active enabledCopy Anchor Link

In this configuration, full nodes query Redis to determine the active sequencer and forward transactions directly to it:

Send to active disabledCopy Anchor Link

In this configuration, full nodes forward transactions to a predefined endpoint without checking which sequencer is active:

Using Helm charts for deploymentCopy Anchor Link

We recommend using the Offchain Labs community Helm charts to deploy your high-availability sequencer setup. These charts provide pre-configured templates for all the necessary components and make it easier to maintain your deployment. The examples below provide base configuration values. However, adjust them to fit your needs, and use values files for production deployments.

Detailed component setupCopy Anchor Link

1. Load balancing (CDN)Copy Anchor Link

We strongly recommend using a CDN for managing traffic and security concerns:

• Recommendation: Use Cloudflare or a similar CDN service
• Configuration:
  • Direct RPC traffic to the Nitro full node fleet
  • Direct feed traffic to public-facing relays
  • Implement rate limiting and bot detection as needed
• Benefits: Distributes load, improves security, and enhances availability

2. Relays setupCopy Anchor Link

The architecture requires two types of relays:

Sequencer relaysCopy Anchor Link

• Deployment should have multiple replicas
• Configure to listen to feed outputs from all sequencers
• All other components connect to these relays instead of directly to the sequencers
• Minimize direct load on sequencer nodes

Deploy sequencer relays using the relay Helm chart:

helm install sequencer-relay offchainlabs/relay \
  --set replicaCount=2 \
  --set configmap.data.chain.id=<your-chain-id> \
  --set configmap.data.node.feed.input.url=ws://sequencer-nitro-0.sequencer-nitro:9642,ws://sequencer-nitro-1.sequencer-nitro:9642,ws://sequencer-nitro-2.sequencer-nitro:9642

Key configuration parameters:

• replicaCount: Number of relay replicas to deploy (recommend at least two for high availability)
• configmap.data.chain.id: Your chain ID
• configmap.data.node.feed.input.url: Comma-separated list of WebSocket URLs for all sequencer feed outputs. This relies on the perReplicaHeadlessService.enabled=true parameter in the sequencer deployment to create individual services for each sequencer replica.

External relaysCopy Anchor Link

• Connect to Sequencer Relays (not directly to sequencers)
• Handle all public feed requests
• Provide an additional layer of isolation for production sequencers
• Since these are public-facing, ensure they scale appropriately based on your traffic needs:

helm install external-relay offchainlabs/relay \
  --set replicaCount=2 \
  --set configmap.data.chain.id=<your-chain-id> \
  --set configmap.data.node.feed.input.url=ws://sequencer-relay:9642

Refer to the Run a feed relay page to learn how to set up a relay node.

3. Nitro full node setupCopy Anchor Link

helm install fullnode offchainlabs/nitro \
  --set replicaCount=2 \
  --set configmap.data.parent-chain.id=<parent-chain-id> \
  --set configmap.data.parent-chain.connection.url=<parent-node-url> \
  --set configmap.data.chain.id=<child-chain-id> \
  --set configmap.data.execution.forwarding-target=http://sequencer-nitro:8547

Send to active configuration (optional)Copy Anchor Link

To enable Redis-based active sequencer discovery:

• Monitor Redis to identify the active sequencer
• Enable with: -execution.forwarder.redis-url=redis://<redis-url>:6379
• Ensure connectivity to individual sequencer services and Redis
• Test failover scenarios before production deployment

helm install fullnode offchainlabs/nitro \
  --set replicaCount=2 \
  --set configmap.data.parent-chain.id=<parent-chain-id> \
  --set configmap.data.parent-chain.connection.url=<parent-node-url> \
  --set configmap.data.chain.id=<child-chain-id> \
  --set configmap.data.execution.forwarder.redis-url=redis://<redis-url>:6379

Health check routing (optional)Copy Anchor Link

If your full nodes forward to a single predefined endpoint, let your load balancer find the active sequencer. Each sequencer can serve a health check that reports whether it holds the lockout:

• Set node.seq-coordinator.chosen-healthcheck-addr on every sequencer. Nitro starts an HTTP service on that address. The value is a listen address in the form host:port or :port, for example :8080. There is no default port.
• The service returns 200 when it chooses a sequencer and 503 otherwise.
• Point your load balancer's health check at this service. The load balancer then sends traffic only to the chosen sequencer.

The health check switches to 503 as soon as the holder stops sequencing, which is lockout-spare before its lockout expires in Redis. See What lockout-spare protects against. This makes it a more accurate signal than inferring the active sequencer from logs or metrics.

Mutating-only endpoint (optional)Copy Anchor Link

For high availability, we recommend routing mutating transactions to a fleet of precheckers, which forward them to the sequencer. To do this, set up a separate endpoint for mutating transactions and allow only calls such as eth_sendRawTransaction to route to the precheckers, which then route to the sequencer. This insulates the sequencer from unnecessary load and enables it to focus on transaction ordering. However, this configuration requires custom load-balancing logic and is out of scope for this guide.

4. Redis setupCopy Anchor Link

Set up a highly available Redis cluster for sequencer coordination:

Deployment optionsCopy Anchor Link

• Use a managed service like AWS ElastiCache (recommended)
• Deploy within Kubernetes using a StatefulSet with PersistentVolumeClaims

RequirementsCopy Anchor Link

• Minimum of three replicas across different availability zones (recommended)

• Secured access (only accessible within the Kubernetes cluster)

• Backups enabled

• Configuration:

  • Use a Redis cluster or Redis Sentinel for high availability
  • Secure the endpoint with proper network policies
  • Monitor Redis health as part of your overall monitoring strategy

5. Sequencer setupCopy Anchor Link

Deploy multiple sequencer replicas with availability zone spread using the Nitro Helm chart (availability zone spread is not demonstrated in the example below):

helm install sequencer offchainlabs/nitro \
  --set replicaCount=3 \
  --set configmap.data.parent-chain.id=<parent-chain-id> \
  --set configmap.data.parent-chain.connection.url=<parent-node-url> \
  --set configmap.data.chain.id=<child-chain-id> \
  --set configmap.data.node.sequencer=true \
  --set configmap.data.node.delayed-sequencer.enable=true \
  --set configmap.data.node.seq-coordinator.enable=true \
  --set configmap.data.node.seq-coordinator.redis-url=<redis-url> \
  --set configmap.data.node.feed.output.enable=true \
  --set configmap.data.node.feed.output.port=9642 \
  --set configmap.data.execution.sequencer.enable=true \
  --set perReplicaHeadlessService.enabled=true

Critical sequencer coordinator parametersCopy Anchor Link

The sequencer coordinator is the key to high availability. These parameters are essential:

For the full set of coordinator flags and their defaults, refer to the Sequencer configuration reference page.

Redis priority registrationCopy Anchor Link

Configuring redis-url and my-url is not enough to make a sequencer eligible. Each sequencer must also appear in the Redis priority list, stored under the key coordinator.priorities as a comma-separated list of URLs in priority order.

Nitro treats this key as read-only. No sequencer can write itself into it. The Sequencer Coordination Manager (SQM) populates it; it's the only component that writes the key.

A sequencer that is not in the list starts cleanly, syncs, serves RPC, and broadcasts to the feed—but never becomes the active sequencer. At startup, it logs Sequencer did not find its URL in the priorities list in redis.

How the coordinator picks the active sequencer:

1. It reads coordinator.priorities from Redis.
2. It goes through the list in order and takes the first sequencer that has published a liveliness key at coordinator.liveliness.<my-url>. Each sequencer writes its own key when it syncs and is ready.
3. That sequencer acquires the lockout, recorded at coordinator.chosen, and activates. Everything else forwards to it.

Two consequences follow from step 2:

• Order is priority—the first healthy, synced entry wins. Put your preferred sequencer first.
• my-url must match its list entry exactly. The comparison is a string match. A trailing slash, a different port format, or an IP address instead of a hostname all cause a silent mismatch.

Verifying registrationCopy Anchor Link

Read the key directly:

redis-cli -u <redis-url> GET coordinator.priorities

Compare each URL against the node.seq-coordinator.my-url of the sequencer it should match. Then check which sequencer currently holds the lockout, and which ones report themselves ready:

redis-cli -u <redis-url> GET coordinator.chosen
redis-cli -u <redis-url> --scan --pattern 'coordinator.liveliness.*'

A sequencer that appears in coordinator.priorities but has no matching coordinator.liveliness.<my-url> key is not eligible. It is either unsynced or its my-url does not match its list entry.

Two log messages tell you the priority list is the problem:

• sequencer priorities unset — the key does not exist. This is expected on a new Redis instance and means no sequencer can ever be chosen until you populate it.
• no sequencer appears to want the lockout on redis — the key exists, but nothing on it is eligible. The log includes the priorities value, so compare it against your running sequencers. This starts at DEBUG, escalates to WARN after 10 seconds and ERROR after 20.

For a full diagnostic path, see Sequencer troubleshooting.

Setting the sequencer's self URLCopy Anchor Link

A critical configuration for the sequencer coordinator is setting a unique URL for each sequencer instance. This configuration can be adjusted using Kubernetes environment variables:

extraEnv:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
  - name: NITRO_NODE_SEQ__COORDINATOR_MY__URL
    value: 'http://$(POD_NAME).<NAMESPACE>.svc.cluster.local:8547/rpc'

This configuration:

1. Gets the pod name from Kubernetes metadata
2. Uses it to create a unique URL for each sequencer instance
3. Sets this URL as the node.seq-coordinator.my-url parameter

Adjust the domain name (<NAMESPACE>.svc.cluster.local) to match your Kubernetes cluster's DNS configuration. This configuration requires Nitro to be configured to read environment variables beginning with NITRO_.

Individual sequencer servicesCopy Anchor Link

You can enable the automatic creation of headless services for each sequencer replica by setting the perReplicaHeadlessService.enabled=true parameter in the Helm chart (as shown in the installation command above). This configuration creates individual services named <release-name>-nitro-<index> that allow direct access to each sequencer replica.

These individual services are critical for a proper high-availability setup because they allow components like sequencer relays to connect directly to specific sequencer instances. This direct connection is essential for:

• Proper failover: When the active sequencer changes, other components can address the new active sequencer directly
• Feed aggregation: Sequencer relays need to collect feeds from all sequencer instances to ensure no messages get lost during transitions

6. Sequencer Coordination ManagerCopy Anchor Link

To manage active sequencer selection, use the built-in sequencer coordinator UI:

• Follow detailed instructions at: How to run a Sequencer Coordination Manager (SQM)
• Use this interface to manually switch between sequencer replicas when needed
• Configure permissions and access controls appropriately

7. Batch poster setupCopy Anchor Link

Deploy the batch poster using the Nitro Helm chart:

helm install batchposter offchainlabs/nitro \
  --set configmap.data.parent-chain.id=<parent-chain-id> \
  --set configmap.data.parent-chain.connection.url=<parent-node-url> \
  --set configmap.data.chain.id=<child-chain-id> \
  --set configmap.data.execution.forwarding-target=null \
  --set configmap.data.node.seq-coordinator.enable=true \
  --set configmap.data.node.seq-coordinator.redis-url=<redis-url> \
  --set configmap.data.node.batch-poster.enable=true \
  --set "configmap.data.node.batch-poster.parent-chain-wallet.private-key=<your-private-key>"

How failover worksCopy Anchor Link

The active sequencer keeps its role by renewing a time-limited lockout in Redis. If it stops renewing, the lockout expires, and the next eligible sequencer takes over. You don’t need to act for this to happen.

Lockout renewalCopy Anchor Link

• A sequencer that acquires the lockout holds it for lockout-duration (default 1m). This is the expiry Nitro sets on the coordinator.chosen key.
• The holder renews the lockout every time it sequences a message. On a busy chain, the lockout is almost always fresh.
• When the chain is idle, the holder renews the lockout shortly before its local deadline. With the defaults, it renews about every 18 seconds.
• Each sequencer runs a coordinator update every update-interval (default 250ms). After a Redis error, it waits retry-interval (default 50ms) times the number of consecutive errors, up to update-interval.

What lockout-spare protects againstCopy Anchor Link

The holder stops treating itself as chosen lockout-spare (default 30s) before its lockout expires in Redis. At that moment, it stops sequencing and returns a 503 health check. This margin keeps two sequencers from being active at once: the old holder stops before another sequencer can acquire the lockout.

Keep lockout-spare well below lockout-duration. A smaller margin shortens the gap between the old holder stopping and the lockout expiring.

Unplanned failoverCopy Anchor Link

If the active sequencer crashes or loses its network, it stops renewing the lockout. Its liveliness key expires at the same moment as the lockout. The other sequencers then pick a new one the same way they picked the first:

1. Each sequencer walks coordinator.priorities in order.
2. It skips any sequencer that has no liveliness key. A sequencer publishes this key only while it is synced.
3. The first sequencer with a liveliness key tries to acquire the lockout. It succeeds only if it holds every message stored in Redis and has processed all of them.

Block production can stop for up to lockout-duration after the holder's last renewal, plus one update-interval and the time the new sequencer needs to start. With the defaults, plan for a gap of up to about one minute.

Two failures stop the chain instead of failing over:

• Redis is unreachable. The holder can’t renew, so it stops sequencing at its local deadline. With the defaults, that is at most 30 seconds after its last renewal. No other sequencer can acquire the lockout until Redis returns. Run Redis in a high-availability setup, as described in Redis setup.
• No listed sequencer is synced. No sequencer publishes a liveliness key, so none acquires the lockout.

During the switch, full nodes can briefly return sequencer temporarily not available. This is normal during a lockout handoff. If it lasts and every node returns it, no sequencer is chosen. See Sequencer troubleshooting.

The sequencer relays listen to every sequencer's feeds. Feed subscribers keep receiving messages from whichever sequencer is active.

Automatic failbackCopy Anchor Link

A sequencer that ranks higher than the current holder takes the role back as soon as it is eligible again. The holder sees that a higher-ranked sequencer wants the lockout, forwards transactions to it, and releases the lockout. This happens in two cases:

• A crashed higher-ranked sequencer recovers and syncs.
• You move a sequencer above the current holder in the SQM.

Planned handoffCopy Anchor Link

A sequencer that receives SIGINT or SIGTERM hands off the lockout before it stops. Nitro also hands off the lockout before a scheduled maintenance run. The handoff runs in four steps:

1. The sequencer deletes its own liveliness key, so it is no longer eligible.
2. The next eligible sequencer in coordinator.priorities becomes the recommended one. The holder forwards transactions to it and releases the lockout.
3. The holder waits up to handoff-timeout for coordinator.chosen to name another sequencer.
4. On shutdown, the sequencer then releases its liveliness key and the lockout, retrying each release on failure.

If no other sequencer is eligible, the holder keeps the lockout. The handoff times out, and Nitro logs timed out waiting for another sequencer to become chosen. On shutdown, the sequencer then releases the lockout anyway, and the chain stops until another sequencer becomes eligible. Before you stop the active sequencer, confirm that at least one standby has a liveliness key.

If you run on Kubernetes, set the pod's terminationGracePeriodSeconds (default: 30 seconds) higher than handoff-timeout. Nitro stops its other components after the handoff finishes. If Kubernetes kills the pod before the handoff ends, the lockout expires on its own, as in an unplanned failover.

Planned maintenanceCopy Anchor Link

Use a planned handoff to upgrade or restart sequencers without stopping the chain. Upgrade the standby sequencers first, then move the active role, then upgrade the last one.

Plan the order around automatic failback. A sequencer ranked above the current holder takes the active role as soon as it syncs.

1. Read the current state:

   Copy Text

   redis-cli -u <redis-url> GET coordinator.chosen
   redis-cli -u <redis-url> GET coordinator.priorities

2. Upgrade one standby sequencer that ranks below the chosen sequencer. Wait until it publishes its liveliness key again:

   Copy Text

   redis-cli -u <redis-url> --scan --pattern 'coordinator.liveliness.*'

3. Repeat step 2 for each remaining standby sequencer.

4. In the Sequencer Coordination Manager (SQM), move an upgraded sequencer to the top of the priority list. Press s to save the change to Redis. The current holder hands off the lockout right away.

5. Confirm that coordinator.chosen now names the upgraded sequencer and that block height keeps rising.

6. Upgrade the previously active sequencer. It now ranks lower, so it stays on standby when it returns. Wait for its liveliness key.

7. Optional: restore your preferred priority order in the SQM. This triggers another handoff.

Test failover before productionCopy Anchor Link

Test both failover paths on a testnet deployment before you go live:

• Planned handoff: Follow the maintenance steps above, or stop the chosen sequencer normally. A normal pod deletion sends SIGTERM, so it tests the planned path. Confirm that coordinator.chosen changes within a few seconds and block production continues.
• Unplanned failover: Stop the chosen sequencer without a graceful shutdown. For example, kill its process with SIGKILL or block its network access to Redis. Confirm that another sequencer acquires the lockout within about lockout-duration and block height keeps rising.

For both tests, watch for sequencer temporarily not available on your full nodes and note how long it lasts.

Monitoring and maintenanceCopy Anchor Link

Health checksCopy Anchor Link

Implement comprehensive health checks for all components:

• Sequencer health: Monitor the sequencer's logs and metrics
• Redis connectivity: Ensure all components can access Redis
• Feed availability: Verify feed connectivity between components
• Transaction processing: Monitor end-to-end transaction flow

Failover monitoringCopy Anchor Link

Alert on these signals so you know when a failover happens or when none can:

For the full list of sequencer metrics, see What to monitor, by component.

TroubleshootingCopy Anchor Link

If you run into any issues, visit the node-running troubleshooting guide.

ReferencesCopy Anchor Link

• How to run a full node
• How to run a Sequencer Coordination Manager (SQM)
• Kubernetes Documentation
• Offchain Labs Community Helm Charts