Очистка Elasticsearch: настройка ILM и retention для data streams

Ниже — рабочая настройка автоматического удаления старых логов в Elasticsearch. Сначала определяем, используются ли обычные индексы или data streams, затем выбираем один lifecycle-механизм и проверяем его на тестовом потоке.

1. Проверяем состояние кластера

export ES='https://es.example.com:9200'
export API_KEY='CHANGE_ME'
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_cluster/health?pretty"
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_cat/indices?v&s=store.size:desc"
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_data_stream?pretty"

2. Проверяем текущий lifecycle

curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_ilm/status?pretty"
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_ilm/policy?pretty"
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/logs-*/_ilm/explain?pretty"

3. Создаём ILM policy

curl -sS -X PUT -H "Authorization: ApiKey $API_KEY" -H 'Content-Type: application/json' \
  "$ES/_ilm/policy/logs-30d" -d '{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {"max_primary_shard_size": "30gb", "max_age": "1d"}
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {"delete": {}}
      }
    }
  }
}'

4. Создаём index template для data stream

curl -sS -X PUT -H "Authorization: ApiKey $API_KEY" -H 'Content-Type: application/json' \
  "$ES/_index_template/logs-app-template" -d '{
  "index_patterns": ["logs-app-*"],
  "data_stream": {},
  "template": {
    "settings": {"index.lifecycle.name": "logs-30d"}
  },
  "priority": 500
}'

5. Создаём тестовый data stream и документ

curl -sS -X POST -H "Authorization: ApiKey $API_KEY" -H 'Content-Type: application/json' \
  "$ES/logs-app-default/_doc" -d '{"@timestamp":"2026-08-02T06:00:00Z","message":"ILM test"}'

curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_data_stream/logs-app-default?pretty"

6. Проверяем привязку ILM

curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/logs-app-default/_ilm/explain?pretty"
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_cat/indices/.ds-logs-app-default-*?v"

7. Проверяем rollover вручную

curl -sS -X POST -H "Authorization: ApiKey $API_KEY" \
  "$ES/logs-app-default/_rollover?dry_run=true&pretty"

curl -sS -X POST -H "Authorization: ApiKey $API_KEY" \
  "$ES/logs-app-default/_rollover?pretty"

8. Вариант Data Stream Lifecycle

Для потока, которым не управляет ILM, можно задать простой retention:

curl -sS -X PUT -H "Authorization: ApiKey $API_KEY" -H 'Content-Type: application/json' \
  "$ES/_data_stream/logs-app-default/_lifecycle" -d '{"data_retention":"30d"}'

curl -sS -H "Authorization: ApiKey $API_KEY" \
  "$ES/_data_stream/logs-app-default/_lifecycle?pretty"

Не назначайте ILM и Data Stream Lifecycle одному потоку одновременно без понимания приоритета и версии Elasticsearch.

9. Проверяем snapshots перед удалением старых данных

curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_snapshot/_all?pretty"
curl -sS -H "Authorization: ApiKey $API_KEY" "$ES/_cat/snapshots/backup-repo?v"

10. Rollback тестовой настройки

curl -sS -X DELETE -H "Authorization: ApiKey $API_KEY" "$ES/_data_stream/logs-app-default"
curl -sS -X DELETE -H "Authorization: ApiKey $API_KEY" "$ES/_index_template/logs-app-template"
curl -sS -X DELETE -H "Authorization: ApiKey $API_KEY" "$ES/_ilm/policy/logs-30d"

Критерий успеха: data stream создан, backing index связан с нужной policy, explain API показывает управляемую фазу, dry-run rollover возвращает ожидаемые условия, а retention виден через lifecycle API.