[Loki] #6.Docker기반 Loki 설치 및 설정 가이드
Grafana Promtail Loki Architecture
- 아래 그림에서 Loki 에 해당하는 부분에 해당함
- Prometheus Loki는 Grafana에서 제공하는 오픈소스 기반의 로그 집계 시스템
- Loki의 주요특징: 수평확장, 고가용성지원, Multi-tenancy
Loki 인덱스 구성
- 메타정보만 인덱스하도록 강제되어 있어, 전체를 설정을 통해 구성하는 elasticsearch에 비해 빠르고 간결하며 적은 저장소를 필요로 하는 구조
테스트 수행한 환경
- 인터넷가능한 환경, Rocky Linux 8.6, Docker가 구성된 환경
1. Loki docker-compose 구동을 위한 스크립트 준비
전체파일 구성
tree
├── create.sh
├── delete.sh
├── docker-compose.yml
├── log.sh
├── start.sh
└── stop.sh
1) Loki의 docker-compose.yml 샘플
주요설정 설명
- -config.file: loki에서 사용하는 config파일 위치 정의. 아래와 같이 설정시 loki 컨테이너 이미지에서 제공하는 기본설정값 그대로 사용
vi docker-compose.yml
version: '3.7'
# monitor-net이름으로 이미 구성한 경우에는 networks부분은 주석처리
networks:
monitor-net:
driver: bridge
services:
loki:
# image: grafana/loki:2.7.4
image: grafana/loki:latest
container_name: loki
ports:
- 3100:3100
command: -config.file=/etc/loki/local-config.yaml
networks:
- monitor-net
labels:
logging: "promtail"
logging_jobname: "containerlogs"
참고: /etc/loki/local-config.yaml 설정 내용 조회방법
docker exec -it loki cat /etc/loki/local-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2020-10-24
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
ruler:
alertmanager_url: http://localhost:9093
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# If you would like to disable reporting, uncomment the following lines:
#analytics:
# reporting_enabled: false
2) docker 컨테이너 생성,삭제,구동,중지,로그조회 등 스크립트 작성
cat > create.sh <<EOF
#!/usr/bin/bash
docker-compose -f docker-compose.yml up -d
docker-compose -f docker-compose.yml logs -f
EOF
cat > delete.sh <<EOF
#!/usr/bin/bash
docker-compose -f docker-compose.yml down -v
EOF
cat > start.sh <<EOF
#!/usr/bin/bash
docker-compose -f docker-compose.yml start
EOF
cat > stop.sh <<EOF
#!/usr/bin/bash
docker-compose -f docker-compose.yml stop
EOF
cat > log.sh <<EOF
#!/usr/bin/bash
docker-compose -f docker-compose.yml logs -f
EOF
chmod +x *.sh
2. Loki 정상구동여부 확인
### 구동하기
./create.sh
### 도커컨테이너 정상구동여부 확인
docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c16cdc7ba488 grafana/loki:latest "/usr/bin/loki -conf…" 6 days ago Up 35 minutes 0.0.0.0:3100->3100/tcp, :::3100->3100/tcp loki
docker logs loki
[root@rocky8 promtail]# docker logs loki
level=warn ts=2023-04-12T00:20:46.856282084Z caller=loki.go:286 msg="per-tenant timeout not configured, using default engine timeout (\"5m0s\"). This behavior will change in the next major to always use the default per-tenant timeout (\"5m\")."
level=info ts=2023-04-12T00:20:46.870600208Z caller=main.go:108 msg="Starting Loki" version="(version=2.8.0, branch=HEAD, revision=90888a0cc)"
level=info ts=2023-04-12T00:20:46.871668305Z caller=server.go:323 http=[::]:3100 grpc=[::]:9095 msg="server listening on addresses"
level=warn ts=2023-04-12T00:20:46.895185176Z caller=cache.go:114 msg="fifocache config is deprecated. use embedded-cache instead"
level=warn ts=2023-04-12T00:20:46.895219711Z caller=experimental.go:20 msg="experimental feature in use" feature="In-memory (FIFO) cache - chunksembedded-cache"
level=info ts=2023-04-12T00:20:46.895570888Z caller=table_manager.go:262 msg="query readiness setup completed" duration=1.322µs distinct_users_len=0
level=info ts=2023-04-12T00:20:46.895605572Z caller=shipper.go:131 msg="starting index shipper in RW mode"
level=info ts=2023-04-12T00:20:46.895770641Z caller=shipper_index_client.go:78 msg="starting boltdb shipper in RW mode"
level=info ts=2023-04-12T00:20:46.89613884Z caller=worker.go:112 msg="Starting querier worker using query-scheduler and scheduler ring for addresses"
level=info ts=2023-04-12T00:20:46.8972083Z caller=mapper.go:47 msg="cleaning up mapped rules directory" path=/loki/rules-temp
이후 로그 생략..
3. Loki 메트릭 데이터 조회
curl http://아이피:3100/metrics
# HELP cortex_consul_request_duration_seconds Time spent on consul requests.
# TYPE cortex_consul_request_duration_seconds histogram
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.005"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.01"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.025"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.05"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.1"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.25"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="0.5"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="1"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="2.5"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="5"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="10"} 725
cortex_consul_request_duration_seconds_bucket{kv_name="ingester-ring",operation="CAS loop",status_code="200",le="+Inf"} 725
cortex_consul_request_duration_seconds_sum{kv_name="ingester-ring",operation="CAS loop",status_code="200"} 0.052866492999999945
cortex_consul_request_duration_seconds_count{kv_name="ingester-ring",operation="CAS loop",status_code="200"} 725
# HELP cortex_distributor_ingester_clients The current number of ingester clients.
# TYPE cortex_distributor_ingester_clients gauge
cortex_distributor_ingester_clients 2
# HELP cortex_dns_failures_total The number of DNS lookup failures
# TYPE cortex_dns_failures_total counter
cortex_dns_failures_total{name="memberlist"} 0
# HELP cortex_dns_lookups_total The number of DNS lookups resolutions attempts
# TYPE cortex_dns_lookups_total counter
cortex_dns_lookups_total{name="memberlist"} 0
# HELP cortex_frontend_query_range_duration_seconds Total time spent in seconds doing query range requests.
# TYPE cortex_frontend_query_range_duration_seconds histogram
4. 구동화면 예시
- container 로그파일에 대해 promtail로 수집하는 부분 추가 후 Grafana 대시보드에 추가하고 조회하는 방법 설명예정
댓글남기기