01Executive summary

Monitoring is only useful if the alert that wakes someone up is real and the alert that matters is not buried. At an MSP, the same symptom (a site goes dark) can be an ISP blip, a firewall in trouble, or a power cut, and every one of them generates its own pile of alerts.

This case study covers the alert engineering and automation work around that: correlating events, refining WAN and VPN alerting, managing a FortiGate fleet, scripting RMM components, and a featured automation design for recovering FortiGates that enter memory conserve mode.

02Disciplines & technologies

03The problem

When a site’s internet drops, monitoring sees a VPN tunnel down, devices offline, a WAN link failed and perhaps a firewall unreachable. Four alerts, one cause, and a technician who has to work that out at speed. Meanwhile real problems, like a firewall slowly running out of memory, can look quiet until they fail.

FortiGates that run short on memory enter conserve mode, where they start dropping new sessions or skipping inspection. Catching that early across a fleet, and recovering safely, is a good automation candidate, and also a risky one.

04Environment & complexity

05My contributions

• VPN outage monitoring and event filtering so one cause produces one actionable alert
• WAN alert refinement and false-positive reduction
• Remote maintenance and firmware work across the FortiGate fleet
• RMM scripting: components and scripts for repeatable remediation and deployment
• Design of automated remediation with guard rails
• FortiManager automation planning for memory conserve mode across 43 FortiGates
• Scheduled documentation sync from the documentation platform into a version-controlled repository, with credentials excluded

06Featured: conserve-mode automation design

Evaluation checklist before anything runs automatically

1. Prove the triggerConfirm the event fires reliably when conserve mode is entered, and does not fire on normal memory spikes.
2. Lab and pilotTest on a lab or low-impact device, then a small pilot group, before the fleet.
3. Guard railsRequire the condition to persist, cap the number of automated reboots per device per window, and never act during business-critical hours without approval.
4. Escalate, do not loopIf recovery fails or the cap is hit, stop and raise it to a human with context.
5. Root cause stays manualA reboot treats the symptom. Repeat offenders get investigated: firmware, features, sizing.

# FortiOS: memory and conserve-mode state
› get system performance status
› diagnose hardware sysinfo conserve
# top memory consumers
› diagnose sys top-mem 10

Alert correlation

07Challenges & decisions

08Outcomes

• Ongoing

  Ongoing VPN, WAN and fleet monitoring and refinement.

• Design / evaluation

  FortiManager conserve-mode automation for 43 FortiGates: designed and under evaluation, not deployed.

• Ongoing

  Fleet firmware: staged rollout validated and in progress by model.

• Verified

  Documentation-to-repository sync workflow in production, excluding credentials.

• Not documented

  No false-positive reduction percentages or uptime figures are claimed.

09Lessons & capabilities

• An automation that can reboot a firewall needs more testing than the firewall itself
• Guard rails and escalation paths are the design, not extras
• Alert quality matters more than alert quantity
• Fix the cause; automate the recovery only while you do

Capabilities demonstrated

• Alert engineering
• FortiManager
• Automation design
• RMM scripting
• Risk analysis
• Fleet operations

10Related work