Resolving Veeam Plug-in Authentication & Configuration Issues for RMAN Backups on Exadata Cloud@Customer

When managing Oracle RMAN backups on Exadata Cloud@Customer (ExaCC) using the Veeam Plug-in for Oracle RMAN, authentication failures, expired credentials, or missing repository settings can cause backup jobs to fail silently or throw connection errors.

In this guide, I will walk through how to diagnose, re-authenticate, and verify the configuration of VeeamPluginforOracleRMAN using command-line administration.

1. Problem Overview

During routine operations or after plugin upgrades (such as updating to VeeamPluginforOracleRMAN-12.3.x), RMAN backup jobs may fail to communicate with the Veeam Backup & Replication repository. Common root causes include:

• Service account password updates on the Veeam server.
• Stale authentication tokens or configuration drift after plugin updates.
• Incomplete setup wizard configuration.

Step 1: Navigate to the Plugin Directory

Log in as root (or a privileged user) on your ExaCC database node VM:

$ sudo su - root 

# cd /opt/veeam/VeeamPluginforOracleRMAN

 

Step 2: Inspect Current Configuration & Status

$ ./OracleRMANConfigTool --show-config

$ cd /home/oracle/local/obackup/DBNAME 

$ grep -n -C 2 -iE "ORA-19511|RMAN-03009|failed" .

 The RMAN log file had ORA-19511 & RMAN-03009 errors, and the backups were not running. 

Step 3: Update Veeam Plugin Credentials 

If the error stems from credential mismatch or password updates, reset the stored credentials using OracleRMANConfigTool. Run the following command, passing your Veeam username and updated password

$./OracleRMANConfigTool --set-credentials "veeoracle" "YourNewPassword"

Step 4: Run the Interactive Configuration Wizard (If needed)

If updating credentials alone does not resolve the issue, or if the repository association is broken, re-run the interactive wizard to re-register the plugin with your Veeam Backup Server.

Step 5: Verify the Configuration

Once credentials and settings have been reapplied, verify that the configuration tool reflects the active and authenticated status. Run the RMAN backup again and check the current RMAN logs. 

$ ./OracleRMANConfigTool --show-config

$ cd /home/oracle/local/obackup/DBNAME 

$ grep -n -C 2 -iE "ORA-19511|RMAN-03009|failed" .

Troubleshooting OEM & Agent “No Route to Host” Errors on OCI

 

Resolving Oracle Enterprise Manager (OEM) on Oracle Cloud Infrastructure (OCI) can sometimes feel like peeling back layers of an onion. You’ve verified your VCN security lists, confirmed your ingress rules are wide open, and even checked that firewalld is inactive. Yet, the OEM console persists with a frustrating “No route to hosts” error on port 3872. If you’ve hit this wall, the culprit isn’t your cloud network – it’s hiding right underneath the OS hood. Oracle Linux images in OCI frequently carry pre-configured, low-level kernel packet filters or iptables rule chains that silently drop traffic on non-standard ports by default, completely bypassing higher-lelve firewall managers.

The below steps shows how to resolve “No Rout to Host” error on the OEM console.

 

1.       Check the ingress rule on the VCN to confirm that the OEM is allowed access to the VM on port 3872.

 

 

2.       Check Low-Level iptables Rules on the target VM

Run this command as root to see if a hidden drop or reject rule is intercepting port 3872 traffic:

sudo iptables -L -n -v | grep 3872

If this returns nothing or shows packets being dropped/rejected, iptables is blocking the connection.

3.       Test telnet from the OMS to the target VM on port 3872

Test telnet access from the OMS server to the DR server on port 3872.

The above output shows that the OEM is not able to reach the target VM on port 3872.

 

4.       Explicitly Force Open Port 3872 via iptables

To instantly force the local Linux firewall kernel to accept connections on port 3872, run:

  sudo iptables -I INPUT -p tcp --dport 3872 -j ACCEPT

5.       Test telnet access again from the OMS server to the DR server on port 3872.

The output shows that the OEM sever is now able to reach the target on port 3872.

 

6.       Add the internal targets and execute the upload agent command.

7.       The “No Route to Host” error is resolved and the target is now up.

The metric evaluation error is resolved, and the target is up on the OEM.

Fixing OCI Base DB GI Precheck Error DCS-10045 Caused by Metastore Duplication after OS Upgrade from Linux 7 to 8.

If you are upgrading an Oracle Cloud Infrastructure (OCI) Base Database Service operating system from Oracle Linux 7 to Oracle Linux 8, you might encounter a roadblock during Grid Infrastructure (GI) prechecks: 'DCS-10045: Validation error encountered: userNames are not unique' (or related DCS agent failures).

While standard DCS troubleshooting guides suggest restarting system services (‘systemctl restart initdcsagent’), this error often persists. In this blog, I’ll dive into why this happens after an OS upgrade and how to resolve it by cleaning up duplicate rows in the local MySQL metastore.

The below error is seen when the GI precheck process is executed.

                  

 Step by Step Procedure to resolve the DCS-10045 error

1. Connect to the MySQL instance on the DBCS

Switch to the root user and connect to local MySQL on port 3306 and switch to SQL mode.

2.        2. Identify Duplicate Entries

Run a query to check for duplicated user records across SysInstance_id, userId, userName, and userRole.

If this returns rows with cnt > 1, duplicate records are causing the conflict.

 3.       Deduplicate using a Temporary Table

To remove duplicates safely within a single transaction:

 4.       Verify that the user has been created

5.      Run the GI patch precheck again from the OCI console.

The precheck completes without any DCS-10045 error.

                

Resolving OCI Database Management "UNKNOWN" Status and "Monitoring Stopped" Alerts

Introduction

If you are managing Oracle databases on Oracle Cloud Infrastructure (OCI), few sights are as jarring as seeing a critical "Monitoring Stopped - Metric collection issue" alarm fire out of nowhere. If you navigate to the OCI Console only to find your Managed Database status marked as UNKNOWN, you are likely dealing with a communication breakdown.

In this blog, I’ll walk through how a standard database password expiration policy can completely blindside your OCI cloud monitoring—and how to quickly fix it by resetting the DBSNMP user and updating the OCI vault. 

The issue typically begins with a critical automated alert from OCI Monitoring:

Alarm: OK_TO_FIRING : CRITICAL : Monitoring Stopped - Metric collection issue

When you log into the OCI Console to investigate, the database page shows:

• Status: UNKNOWN

• Metrics: All charts (CPU, Memory, Session limits) flatlined or showing empty gaps.

• 
  
  
  
  

Because the status is UNKNOWN (rather than DOWN), it indicates that while the infrastructure may be fine, the OCI Management Agent cannot pull telemetry from the database.

How to Resolve It (Step-by-Step)

1. First, you need to log into your database host via SSH and run SQL*Plus as SYSDBA to check the account's actual status and reset the password.

Tip: If you want to prevent this from happening again to your service accounts, consider assigning DBSNMP to a custom DB profile with a longer (or unlimited) password lifetime policy.

2. Since the DBSNMP user password has EXPIRED, I decided to extend the password by reusing the encrypted hash value. You can get the hash value using the below command. 

     

With the method of extending the DBSNMP password, there is no need to update the OCI vault secret because it is the same password. 

3. After extending the DBSNMP password, check the status of the database monitoring The status should change from UNKNOWN to UP.

Fixing False-Positive OCI IPsec Alarms in Active-Passive Topologies

 

When deploying an Oracle Cloud Infrastructure (OCI) Site-to-Site VPN in an active-passive (standby) configuration, Tunnel 2 is often intentionally left down on the customer-premises equipment (CPE). However, a default OCI Monitoring alarm will treat this intentional downtime as a critical failure, flooding your inbox with repeat alerts.

This happens because an un-scoped Monitoring Query Language (MQL) expression evaluates all metric streams across the entire compartment, and tunnel 1 which is active remains in FIRING state. The screenshot below shows that Tunnel2 is in the resourceName of the notification sent from the monitoring instead of Tunnel1. This blog provides a guided approach to avoid false alarm for the active tunnel.

The Problem: Blanket MQL Queries causing active Tunnel to remain in FIRING state

A standard alarm query often looks like this:

TunnelState[5m].mean() == 0

Because no specific dimensions are defined inside the curly braces {}, OCI checks the TunnelState of both Tunnel 1 and Tunnel 2. Even if Tunnel 1 is perfectly healthy (returning a value of 1), Tunnel 2 being down (returning 0) satisfies the condition, triggering a false CRITICAL state and tunnel 1 remains in FIRING state.

 

Step-by-Step Resolution Procedure

Step 1: Identify the Affected Alarm

1. Log into the OCI Console.
2. Open the navigation menu, go to Observability & Management, and select Alarm Definitions under the Monitoring section.
3. Locate and click on the alarm that is generating the false alerts 
4. 
   
   

4. Step 2: Edit the Alarm Query

1. On the alarm's detail page, click the Edit button at the top.
2. Scroll down to the Creation Mode section and choose advance mode to enable you edit the  MQL expression.
3. Look at the Query text box. Notice that it lacks specific resource dimensions.

 

Step 3: Scope the Query to the Active Tunnel

To force the monitoring engine to ignore the standby tunnel, modify the query to explicitly filter by the resourceName dimension of your active tunnel.

Update the expression to the following:

Step 4: Verify and Save

1. Click the Edit query or Stream graph preview button to ensure the metric stream updates correctly and shows only the data for Tunnel 1.
2. Scroll to the bottom of the page and click Save changes.

The Result

Once saved, OCI's Monitoring service drops the Tunnel 2 metric stream from its evaluation loop. Within one evaluation cycle 5 minutes based on my window, the alarm state will transition from FIRING back to OK, permanently stopping the repetitive alert emails while keeping your primary production path fully monitored.

 

 

Steps to Deploy a 4-Node MySQL 8.4 InnoDB Cluster

 

   

Introduction

MySQL InnoDB Cluster provides native, highly available, and fault-tolerant database services. It packages three components—MySQL Shell (for configuration), MySQL Router (for connection handling), and the Group Replication plugin—into one cohesive solution. This guide outlines the setup on four Linux nodes.

Prerequisites

Before starting, ensure all four Linux nodes (node1 through node4) meet these requirements:

• MySQL Server: MySQL 8.4 installed on the first three nodes (Refer to my blog on how to install MySQL 8.4.0 https://edson-help.blogspot.com/2025/08/step-by-step-guide-to-setup-mysql-84-on.html)
• Network: All nodes can communicate with each other on the necessary ports (3306, 33060, and Group Replication ports 33061-33062).
• Enable ports 6446 and 6447 on the database nodes for read/write and read only connections.
• Initial Setup: Each MySQL instance must be initialized, running, and accessible via the MySQL Shell.

• Download the required binaries for MySQL Shell & MySQL Router

Steps For to Install MySQL Shell & InnoDB Cluster Setup

 1.     Install MySQL Shell on all VMs (Repeat for all four nodes)

              unzip MySQLShell.zip

              yum install mysql-shell-commercial-8.4.7-1.1.el8.x86_64.rpm

 2.     Add the hostname of each VM on each server for internal DNS resolution.

 3.      Test connection to the MySQL Databases using mysql shell  (mysqlsh)

        mysqlsh root@mysql-db-02:3307

        mysqlsh root@mysql-db-02:3307

        mysqlsh root@mysql-db-03:3307

4. Check the instance configuration 

  dba.checkInstanceConfiguration('admin@mysql-db-01:3307')

 

5. Use MySQL Shell to fix issues (confirm required changes)

The MySQL instance will restart during this process.

 dba.configureInstance('admin@mysql2:3307')

 
6. Execute the check the instance configuration again after fixing the issues. Repeat for all three database nodes.

7. Create the InnoDB Cluster from the primary node

\connect admin@mysql-db-01:3307

var cluster = dba.createCluster('cdlcluster')

8. Add the remaining two nodes to the newly created cluster.

cluster.addInstance('admin@mysql-db-02:3307')

cluster.addInstance('admin@mysql-db-03:3307')

9. Check the status of the InnoDB Cluster

 

MySQL Router Deployment

1.     Enable port 6446 and 6447 for connections from the router to the InnoDB cluster.

Execute the below commands for all three database nodes.

2. Unzip the MySQLRouter binary file

3. Install MySQL Router via rpm package

4. Grant access on the databases to allow connections from the Router.

Repeat these commands on all three database nodes.        

5. Configure the MySQL Router

sudo mysqlrouter --bootstrap admin@mysql-db-01:3307 --user=mysqlrouter

6.   Start the MySQL Router

 sudo systemctl start mysqlrouter

7. Test the connection with a mysql client connect to 6446 port (read/write). The connection takes you to the primary node mysql-db-01

 

mysqlsh admin@127.0.0.1:6446

 

8. Test the connection with a client connect to 6447 port (read only mode). The connection takes you to one of the secondary node mysql-db-02/03

mysqlsh admin@127.0.0.1:6447