---
title: "Split-brain Issue on vRealize Automation Cluster Nodes"
canonical: "https://www.virtcloudrocks.com/space/vblog01/blog/721144/Split-brain%20Issue%20on%20vRealize%20Automation%20Cluster%20Nodes"
format: markdown
---
In this post I am going to describe a recent experience of mine... Little History Our Cloud Management (CM) production environment is based on a distributed deployment of VMware vRealize Automation (vRA) appliances, load-balanced through an external F5 LB. For storage redundancy purposes each one of the redundant components of the vRA deployment resides in its own datastore type - we have two types of datastores NFS and iSCSI served from different storage arrays. Few days ago we faced a major outage with our core back-end storage, and as you can understand our ESXis lost connection with the datastores, leading to all the VMs that were served from those datastores to be disconnected. When storage failure was fixed and connectivity with the datastores was regained, I tried to bring all the vRA components to full operational status, and I noticed that both of the vRA appliances appeared to act as Master nodes, (see below), thus leading the Cloud Management Portal not to be accessible. When I was powering off either of the nodes, and I had only one vRA node operational, all services were coming with no issues, and I could access the Cloud Management portal and all the internal services. I continued working for a two-three days having only one of the vRA nodes operational, since there were some urgent tasks to be done; in my case, vRA node 1 was powered off and vRA node 2 was powered on. Issue Resolution / Next steps When I found a free time slot, I raised a support request to VMware GSS, since I couldn't figure out what was going on, and I needed to revert back full redundancy of the cluster -  my mistake that I hadn't raised the request since Day 1 of the issue . Following some logs analysis and extended troubleshooting with VMware GSS, the problem was identified and support told me that we had to re-join one of the nodes to cluster. I explained to him that I was working on vRA node 2 and I had data that I didn't want to loose, and he told me that I should make vRA node 2 master, in order for my data not get lost. Below I describe all the tasks that we did, 1 - Shutdown both nodes 2 - Restart the vRA node 2, wait for all services to come in Registered status 3 - Restart the vRA node 1, wait for all services to come in Registered status 4 - After all the services come up on both vRA appliances (except from vco, which will be registered to one of vRA nodes), took snapshots of both the appliances 5 - Go to the VAMI of vRA node 1 →   Cluster  tab, provide the FQDN of second node (vRA node 2) and enter the root password 6 - Press  Initiate the Join Cluster  operation Wait until the process to finish - this is a lengthy process so go grab a coffee  When the process finishes successfully I verified the master / replica status of the two nodes and the Cloud Management portal was accessible with no issues. Thanks VMware GSS for the great support and help. Tasos