When deploying an Oracle Cloud
Infrastructure (OCI) Site-to-Site VPN in an active-passive (standby)
configuration, Tunnel 2 is often intentionally left down on the
customer-premises equipment (CPE). However, a default OCI Monitoring alarm will
treat this intentional downtime as a critical failure, flooding your inbox with
repeat alerts.
This happens because an un-scoped
Monitoring Query Language (MQL) expression evaluates all metric streams across
the entire compartment, and tunnel 1 which is active remains in FIRING state. The
screenshot below shows that Tunnel2 is in the resourceName of the notification sent
from the monitoring instead of Tunnel1. This blog provides a guided approach to
avoid false alarm for the active tunnel.
The Problem: Blanket MQL
Queries causing active Tunnel to remain in FIRING state
A standard alarm query often
looks like this:
TunnelState[5m].mean() == 0
Because no specific dimensions
are defined inside the curly braces {}, OCI checks the TunnelState of both
Tunnel 1 and Tunnel 2. Even if Tunnel 1 is perfectly healthy (returning a value
of 1), Tunnel 2 being down (returning 0) satisfies the condition, triggering a
false CRITICAL state and tunnel 1 remains in FIRING state.
Step-by-Step Resolution
Procedure
Step 1: Identify the Affected
Alarm
- Log into the OCI Console.
- Open the navigation menu, go to Observability
& Management, and select Alarm Definitions under the Monitoring
section.
- Locate and click on the alarm that is generating the
false alerts
- Step 2: Edit the Alarm Query
- On the alarm's detail page, click the Edit
button at the top.
- Scroll down to the Creation Mode section and
choose advance mode to enable you edit the MQL expression.
- Look at the Query text box. Notice that it
lacks specific resource dimensions.
Step 3: Scope the Query to the
Active Tunnel
To force the monitoring engine to
ignore the standby tunnel, modify the query to explicitly filter by the
resourceName dimension of your active tunnel.
Update the expression to the
following:
Step 4: Verify and Save
- Click the Edit query or Stream graph
preview button to ensure the metric stream updates correctly and shows
only the data for Tunnel 1.
- Scroll to the bottom of the page and click Save
changes.
The Result
Once saved, OCI's Monitoring
service drops the Tunnel 2 metric stream from its evaluation loop. Within one
evaluation cycle 5 minutes based on my window, the alarm state will transition
from FIRING back to OK, permanently stopping the repetitive alert emails while
keeping your primary production path fully monitored.
No comments:
Post a Comment