維持Web Server在資安服務掃描狀態下效能穩定策略—以Trend Micro Deep Security 為例 @ 儒道哲學的浪漫人生

　　最近因為部門的Web application都已經搬到AWS上以EC2 Instance VM來運轉，所以每一台VM都被配置公司的Deep Security Agent Service作為標準背景資訊防護軟體來為所有Web Server進行資安監控，也因此發生過一個問題是Deep Security在掃描系統時因為掃到一個檔案量龐大的資料夾，所以整個系統CPU跟Memory就這樣被hand在那邊，把資源吃光，Web Latency因此變得很高，甚至可能Response Time Out。

　　針對這個問題，由於長官問我有沒有解決辦法，雖然一開始我也只是說只能在部署機器時與之後記得要觀察是不是有因為Deep Security掃描造成的系統異常，但終究這不是一個Solution，只是一個勉勉強強的Workround。

　　所以我想了一下，也許應該從QA或DevOps的角度來思考如何從基本面改善這種防毒軟體引起的效能不穩定問題，畢竟Web Server跟我們個人電腦不一樣，不是說都因為裝有防毒軟體，所以不穩定只要重開機或關閉監控就可以，Web Server是24*7都在運作的，而且是在網路上工作，任何資安防護與停維運計畫都是不能被忽略的，不能跟PC一樣說關就關。

　　終於我想到一個策略就是：我們應該針對資安服務的行為模式去思考如何透過預先的測試與優化設定來防止這樣的問題，而且部門所有機器都應該統一這一套針對資安服務的作業程序，做為Operation的Security Policy for Web Server。

　　所以我從兩方面去思考如何實施這樣的策略。先從測試開始，我找了不少網路資料，但都沒有什麼針對防毒軟體對Web Server影響的測試相關討論文章，大抵來說，應該還沒有什麼人特別為了自己的Web AP去撰寫任何搭配資安軟體配置下的測試，畢竟這應該算系統層的問題，而不是APP本身該去思考的事。

　　在一陣子摸不著頭緒之後，我轉而思考能不能直接先從了解Deep Security行為開始，然後去挖掘優化監控掃描設定以使系統任何情況下都可以穩定運作的可能性。我終於發現到一些有高度價值的資訊，也許從這些設定Deep Security運作方式的configuration參數中，可以找到讓系統與資安服務完美共存的無暇地帶。

　　首先，Trend Micro Deep Security是一項針對VM平台提供的雲端式資安服務軟體，基本上就是在AWS這一類Cloud Platform上創建一個Deep Security Manager服務，然後在這個Deep Security Manger統一設定所有將來納入他底下管理的所有VM(在這邊即是指AWS EC2 Instance)要套用的Deep Security Policy，可以針對不同VM作客製化的Policy，也可以將VM群組化再套用群組用的Policy，但所有這一切Deep Security Policy設定只能在Deep Security Manager上編輯¹，如此當底下任何VM機器生成時，Deep Security Manger便可以直接套用設定到VM上，不必再額外為新VM作任何設定，VM上唯一要做的就是讓Deep Security VM Agent Service Enable( Get DS Agent Installer script from DSM and then run install script on VM)，其他都交給 Deep Security Manager來管理。

[1] For most anti-malware settings, you can either configure them for each individual computer or in a policy that applies to multiple computers.

關於AWS 上DSA的快速安裝可以參考：　Deploying DSA for Linux on Deep Security

　　在了解Deep Security 的管理架構之後，接下來就可以看看有哪些設定是Deep Security提供且對我們管理安全監控即時掃描的system resource usage是有幫助的，在查閱Deep Security一些線上文件後，基本上不外乎以下三種設定最重要：

Real-Time Scanning Schedule
Inclusion and Exclusion Policy
CPU and Memory and Disk Usage Limitation

之所以以上三點重要是因為當我們知道這些設定的方式之後，便可以用他們來幫助我們穩定Web Server在當有資安即時掃描發生時的效能，當我們充份了解這幾項設定之後，基本上便可以依此制定我們部門Window Web Server and Linux Web Server 專用的Security Policy，並且要求在所有VM生成時都主動套用這些Policy。同時這個執行策略是通用的，也就是不只侷限於Deep Security上²，他應該也要同時適用於其他資產服務產品。

[2] The following section provides recommendations and information about some of the key settings and options for running anti-malware scans. For detailed information about all available settings, see Anti-malware settings and Malware Scan Configurations.

Real-Time Scanning Schedule

　　根據”Protect servers from malware in four steps”文件，當DS運作Anti-Malware即時掃描時，我們可以在Policy上設定三種掃描方式：

Real Time Sacn : 即時監控防護掃描。
Manually Scan : 即時手動掃描。
Scheduled Scan : 設定固定時間排程掃描。

在以上幾種掃描類型中，我們應該可以套用Schedule Scan做固定時間掃描，便於掌握系統效能可能因為DS Scan造成不穩定的時間範圍。

Inclusion and Exclusion Policy

　　這是目前我們部門唯一比較知道的設定項目，應該要在Policy中明確排除掉可能放有大量Log檔案的資料夾以及我們信任的資料夾或檔案(如：Deploy Build Folder)。除此之外，我們也可以by檔案容量大小或數量以及一些特定檔案屬性來決定是否排除這項檔案掃描，例如當一個放備份圖檔的壓縮檔內含5層資料夾10000個檔案時，我們可以排除掉。而以上這些設定該怎麼統一處理則仍需要全Team各系統的Owner幫忙界定。

CPU and Memory and Disk Usage Limitation

　　最後這一項是我最感興趣也覺得最重要的一項，畢竟前兩項其實現在來看已經是已知且在運作中的policy，只是應該還有些許優化設定的空間，然此項是我覺得可以完全避免系統資源被DS Scanning吃光的設定關鍵，最主要是我們可以限制當DS Scanning佔用超過X% CPU或Y% RAM或Z% Disk usage時應暫停掃描以維持系統穩定，只要知道這些設定的範圍，便可以防止系統在掃描時被DS吃掉大部份資源而不能正常response使用者request。

　　以下內容擷取自” Performance tips for anti-malware” ，是我們可以針對DS Scanning使用的系統資源所作的限制：

System Resource	Settings That Impact Performance
CPU usage	Exclude files from real-time scans if they are normally safe but have high I/O, such as databases, Microsoft Exchange quarantines, and network shares (on Windows, you can use procmon to find files with high I/O). To specify exclusions, go to Policies > Other > Malware Scan Configurations, double-click the malware scan configuration you want to change, go toExclusions and then select an exclusion list. If you need to create an exclusion list, select New from the list. Deselect Enable Network Directory Scan Deselect Smart Scan if the computer doesn't have reliable network connectivity to the Trend MicroSmart Protection Network or your Smart Protection server Reduce the CPU impact of malware scans by setting CPU Usage to Medium (Recommended; pauses when overall CPU usage exceeds 50%,) or Low (Pauses when overall CPU usage exceeds 20%). To change CPU Usage, go to Policies > Other > Malware Scan Configurations, double-click the malware scan configuration you want to change, then go to Advanced > CPU Usage. You can also create a scheduled task to run scans at a time when CPU resources are more readily available. In VM Scan Cache, select a Real-Time Scan Cache Configuration; if scans are not frequent, increase the Expiry Time (avoid repeated scans) Use agentless deployments (CPU usage is in one centralized virtual appliance, not every computer) Reduce or keep small default values for Maximum file size to scan, Maximum levels of compression from which to extract files, Maximum size of individual extracted files,Maximum number of files to extract, and OLE Layers to scan Most malware is small, and nested compression indicates malware. But if you don't scan large files, there is a small risk that anti-malware won't detect some malware. You can mitigate this risk with other features such as integrity monitoring. Select Use multithreaded processing for Malware scans (if available)
RAM usage	Reduce or keep small default values for Maximum file size to scan, Maximum levels of compression from which to extract files, Maximum size of individual extracted files, andOLE Layers to scan Most malware is small, and nested compression indicates malware. But if you don't scan large files, there is a small risk that anti-malware won't detect some malware. You can mitigate this risk with other features such as integrity monitoring. Use agentless deployments (RAM usage is in one centralized virtual appliance, not every computer) Reduce Maximum file size to scan
Disk usage	Reduce Maximum disk space used to store identified files

System Resource

Settings That Impact Performance

CPU usage

Exclude files from real-time scans if they are normally safe but have high I/O, such as databases, Microsoft Exchange quarantines, and network shares (on Windows, you can use procmon to find files with high I/O). To specify exclusions, go to Policies > Other > Malware Scan Configurations, double-click the malware scan configuration you want to change, go toExclusions and then select an exclusion list. If you need to create an exclusion list, select New from the list.
Deselect Enable Network Directory Scan
Deselect Smart Scan if the computer doesn't have reliable network connectivity to the Trend MicroSmart Protection Network or your Smart Protection server
Reduce the CPU impact of malware scans by setting CPU Usage to Medium (Recommended; pauses when overall CPU usage exceeds 50%,) or Low (Pauses when overall CPU usage exceeds 20%). To change CPU Usage, go to Policies > Other > Malware Scan Configurations, double-click the malware scan configuration you want to change, then go to Advanced > CPU Usage.

You can also create a scheduled task to run scans at a time when CPU resources are more readily available.

In VM Scan Cache, select a Real-Time Scan Cache Configuration; if scans are not frequent, increase the Expiry Time (avoid repeated scans)
Use agentless deployments (CPU usage is in one centralized virtual appliance, not every computer)
Reduce or keep small default values for Maximum file size to scan, Maximum levels of compression from which to extract files, Maximum size of individual extracted files,Maximum number of files to extract, and OLE Layers to scan

Most malware is small, and nested compression indicates malware. But if you don't scan large files, there is a small risk that anti-malware won't detect some malware. You can mitigate this risk with other features such as integrity monitoring.

Select Use multithreaded processing for Malware scans (if available)

RAM usage

Reduce or keep small default values for Maximum file size to scan, Maximum levels of compression from which to extract files, Maximum size of individual extracted files, andOLE Layers to scan
Most malware is small, and nested compression indicates malware. But if you don't scan large files, there is a small risk that anti-malware won't detect some malware. You can mitigate this risk with other features such as integrity monitoring.
Use agentless deployments (RAM usage is in one centralized virtual appliance, not every computer)
Reduce Maximum file size to scan

Disk usage

Reduce Maximum disk space used to store identified files

　　除了以上三項外，Deep Security本身還提供獨有的”Virtual Appliance Scan Caching”項目，可以進一步優化DS Scanning效能，至於其內容原理，大家就自己進去”Virtual Appliance Scan Caching”文件參考囉，但這也可以當成是我們設定優化的選項之一。

　　綜合以上大概就是我想到的利用對資安服務產品本身行為與設定的了解來執行Web Application系統效能穩定最佳化的策略，雖然是以公司的Deep Security為例，但相信如果這策略成功，也一定適合直接套用到其他資安產品上。當根據這些設定制定一套統一的Security Policy後，接下來就是把他交給有權限設定policy的人處理啦，以Deep Security來說當然就是指DSM Administrator囉，相信這方法應該多少對我們的系統維運可以起到不錯的功效。

2017年6月7日星期三　6:35 AM

jackterrylau

儒道哲學的浪漫人生

jackterrylau 發表在痞客邦留言(0) 人氣()

E-mail轉寄

儒道哲學的浪漫人生

願生活像一本書，把每一頁都寫得豐富精彩; 願生命像顆太陽，把每一天都注入滿滿活力；願感情像把吉他，把每一聲都彈得浪浪漫漫；願心靈像花鮮豔，把每一寸都灌概儒道哲學。

屬於金牛座小子的草原