使用Graylog和Opensearch進行日誌管理和備份

近期專案需求，需要使用到Log工具，在效能、快速搭建、費用的考量下，決定使用Gray & Opensearch

創建 Graylog 和 Opensearch

1. 使用官方提供的Docker yaml檔就能快速建立，不過因為有備份需求，opensearch要增加快照設定

version: '3'
services:
  opensearch:
    image: "opensearchproject/opensearch:2.15.0"
    environment:
      - "OPENSEARCH_JAVA_OPTS=-Xms1g -Xmx1g"
      - "bootstrap.memory_lock=true"
      - "discovery.type=single-node"
      - "path.repo=/home/mmsd/graylog" # 配置快照路徑
      - "plugins.security.disabled=true"
    ulimits:
      memlock:
        hard: -1
        soft: -1
    ports:
      - "9203:9200"
      - "9303:9300"
    volumes:
      - "opensearch:/usr/share/opensearch/data"  # 數據卷
      - "/path/on/host/graylog:/home/mmsd/graylog"  # 快照存儲路徑
volumes:
  opensearch:

2. 啟動Docker container

docker compose up -d

手動備份Log

1. Docker container啟動後，就建立快照存儲庫

curl -X PUT "http://localhost:9203/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/home/mmsd/graylog",
    "compress": true
  }
}'

• 如果出現{"error":{"root_cause":[{"type":"exception","reason":"failed to create blob container"}],"type":"repository_verification_exception","reason":"[my_backup] path is not accessible on cluster-manager node","caused_by":{"type":"exception","reason":"failed to create blob container","caused_by":{"type":"access_denied_exception","reason":"/usr/share/opensearch/snapshot/tests-G5Qo2ODjTz6wqjlTiKsRDw"}}},"status":500}的錯誤，通常是因為容器對該路徑的權限不足，請參考權限不足解決方案

2.創建快照

curl -X PUT "http://localhost:9203/_snapshot/my_backup/snapshot_1"

3.確保快照已正確配置

curl -X GET "http://localhost:9203/_snapshot/my_backup?pretty"

4.確保 /path/on/host/graylog 目錄中能看到生成的快照文件

ls /path/on/host/graylog

5.備份索引

curl -X PUT "http://localhost:9203/_snapshot/my_backup/graylog_backup_$(date +%Y%m%d%H%M%S)?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "indices": "graylog_0",
  "ignore_unavailable": true,
  "include_global_state": false
}'

6.刪除已備份的索引

curl -X DELETE "http://localhost:9203/graylog_0?pretty"

Recover Log data

1.關閉索引

curl -X POST "http://localhost:9203/graylog_0/_close?pretty"

2.Recover Log data

curl -X POST "http://localhost:9203/_snapshot/my_backup/graylog_backup/_restore" -H 'Content-Type: application/json' -d'
{
  "indices": "graylog_0"
}'

3.重新開啟索引

curl -X POST "http://localhost:9203/graylog_0/_open?pretty"

自動備份

1.建立shell script

#!/bin/bash

# 備份索引
curl -X PUT "http://localhost:9203/_snapshot/my_backup/backup_$(date +%Y%m%d)?wait_for_completion=true" -H 'Content-Type: application/json' -d'
{
  "indices": "graylog_*",
  "ignore_unavailable": true,
  "include_global_state": false
}'

# 刪除索引
curl -X DELETE "http://localhost:9203/graylog_0?pretty"

# 日誌
echo "$(date): Backup and deletion completed." >> /var/log/opensearch_backup.log

2.設定Cron Job

crontab -e

0 2 * * * /path/to/backup_script.sh

權限不足解決方案

排查與解決方式

1. 檢查宿主機目錄的權限

由於你使用的是 bind 挂载 /path/on/host/graylog:/home/mmsd/graylog，宿主機的目錄權限可能限制了 OpenSearch 的操作。

執行以下命令，檢查該目錄的所有者與權限：

ls -ld /path/on/host/graylog

假設輸出類似以下內容：

drwxr-xr-x 2 root root 4096 Nov 21 14:00 /path/on/host/graylog

可以看到，該目錄歸屬於 root，而 Docker 容器內的 OpenSearch 默認運行用戶不是 root，因此無法寫入。

2. 修改宿主機目錄的權限

給該目錄賦予容器的用戶寫入權限：

• 改變目錄的擁有者 找到容器內的 OpenSearch 用戶 UID 和 GID，然後修改宿主機路徑的擁有者：

docker exec -it <opensearch_container_id> id

輸出類似：

uid=1000 gid=1000(opensearch)

在宿主機執行：

sudo chown -R 1000:1000 /path/on/host/graylog

3. 重啟 OpenSearch 容器

權限修改後，重新啟動容器：

docker compose restart opensearch

4. 重新創建快照存儲庫

再次執行 API 請求來創建快照存儲庫：

curl -X PUT "http://localhost:9203/_snapshot/my_backup" -H 'Content-Type: application/json' -d'
{
  "type": "fs",
  "settings": {
    "location": "/home/mmsd/graylog",
    "compress": true
  }
}'