ELK分布式日志收集

ELK 架构

文档

  flowchart LR
    subgraph server-1
        log-file-1 --> filebeat-1
    end
    subgraph server-2
        log-file-2 --> filebeat-2
    end
    filebeat-1 -- push log --> kafka
    filebeat-2 -- push log --> kafka
    kafka -- sub log --> logstash-filter
    logstash-filter -- pub log --> kafka
    kafka -- sub log --> logstash
    logstash -- push --> es
    kibana -- query --> es

配置ES

修改文件config/elasticsearch.yml

• path.data
• path.logs
• network.host
• http.port

生成相应账户密码

• ./bin/elasticsearch-reset-password -u elastic
• ./bin/elasticsearch-reset-password -u kibana-system

配置Kibana

• server.port
• server.host
• elasticsearch.username
• elasticsearch.password
• i18n.locale

配置Kafka

新建 config/kafka_server_jass.conf

KafkaServer {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password"
    user_tom="tom-password";
};
KafkaClient {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password"
    user_tom="tom-password";
};
Client {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password";
};
Server {
  org.apache.kafka.common.security.plain.PlainLoginModule required
    username="admin"
    password="admin-password"
    user_admin="admin-password";
};

修改 bin/kafka-server-start.sh, 最后一行前添加 export KAFKA_OPTS="-Djava.security.auth.login.config=$base_dir/../config/kafka_server_jaas.conf"

新建 config/tom-client.conf

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule \
    required username="tom" password="tom-password";
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN

测试执行命令 ./bin/kafka-topics.sh --bootstrap-server ip:port --command-config config/tom-client.conf --list