ELK分布式日志收集

2023-10-07 约 230 字预计阅读 1 分钟

ELK 架构

flowchart LR
    subgraph server-1
        log-file-1 --> filebeat-1
    end
    subgraph server-2
        log-file-2 --> filebeat-2
    end
    filebeat-1 -- push log --> kafka
    filebeat-2 -- push log --> kafka
    kafka -- sub log --> logstash-filter
    logstash-filter -- pub log --> kafka
    kafka -- sub log --> logstash
    logstash -- push --> es
    kibana -- query --> es

配置ES

修改文件config/elasticsearch.yml

path.data
path.logs
network.host
http.port

生成相应账户密码

./bin/elasticsearch-reset-password -u elastic
./bin/elasticsearch-reset-password -u kibana-system

配置Kibana

server.port
server.host
elasticsearch.username
elasticsearch.password
i18n.locale

配置Kafka

新建 config/kafka_server_jass.conf

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26


KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password"
    user_tom="tom-password";
};
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password"
    user_tom="tom-password";
};
Client {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password";
};
Server {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password";
};

修改 bin/kafka-server-start.sh, 最后一行前添加 export KAFKA_OPTS="-Djava.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf"

新建 config/tom-client.conf

1
2
3
4


sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
    required username="tom" password="tom-password";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

测试执行命令 ./bin/kafka-topics.sh --bootstrap-server ip:port --command-config config/tom-client.conf --list

目录

ELK分布式日志收集

ELK 架构

配置ES

配置Kibana

配置Kafka