大数据学习笔记第2课(续) 通过filebeat收集nginx访问日志到kafka集群

1、说明

本文是接着我的上一篇博文《大数据学习笔记第2课 Zookeeper & Kafka集群搭建》往下写的,请先阅读上一篇文章,不然会感到比较突兀。

2、环境说明

本次搭建的环境是在华为云上
1、服务器:三台华为云ECS(都是4核8G)

计算机名内网IP公网IP别名说明
ecs-ae8a-0001192.168.0.177x.x.x.xhadoop01zookeeper/broker01
ecs-ae8a-0002192.168.0.56x.x.x.xhadoop02broker02
ecs-ae8a-0003192.168.0.59x.x.x.xhadoop03broker03

2、系统版本:CentOS Linux release 7.5.1804 (Core)
3、zookeeper与kafka服务都已正常启动,且已创建了1个topic,值为qingdao

3、Nginx安装与配置

1、在CentOS上安装Nginx非常简单,我们再hadoop01服务器上执行以下命令:

yum install nginx

安装日志如下:

[root@ecs-ae8a-0001 ~]# yum install nginx
Loaded plugins: fastestmirror
Determining fastest mirrors
epel/x86_64/metalink                                                                                                                                                                                                                                   | 8.4 kB  00:00:00     
 * base: mirrors.huaweicloud.com
 * epel: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.huaweicloud.com
base                                                                                                                                                                                                                                                   | 3.6 kB  00:00:00     
epel                                                                                                                                                                                                                                                   | 4.7 kB  00:00:00     
extras                                                                                                                                                                                                                                                 | 2.9 kB  00:00:00     
updates                                                                                                                                                                                                                                                | 2.9 kB  00:00:00     
(1/7): epel/x86_64/group_gz                                                                                                                                                                                                                            |  95 kB  00:00:00     
(2/7): extras/7/x86_64/primary_db                                                                                                                                                                                                                      | 222 kB  00:00:00     
(3/7): base/7/x86_64/primary_db                                                                                                                                                                                                                        | 6.1 MB  00:00:00     
(4/7): epel/x86_64/updateinfo                                                                                                                                                                                                                          | 1.0 MB  00:00:00     
(5/7): epel/x86_64/primary_db                                                                                                                                                                                                                          | 6.9 MB  00:00:00     
(6/7): base/7/x86_64/group_gz                                                                                                                                                                                                                          | 153 kB  00:00:07     
(7/7): updates/7/x86_64/primary_db                                                                                                                                                                                                                     | 3.7 MB  00:00:08     
Resolving Dependencies
--> Running transaction check
---> Package nginx.x86_64 1:1.16.1-3.el7 will be installed
--> Processing Dependency: nginx-all-modules = 1:1.16.1-3.el7 for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: nginx-filesystem = 1:1.16.1-3.el7 for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libcrypto.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_0)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libssl.so.1.1(OPENSSL_1_1_1)(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: nginx-filesystem for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: redhat-indexhtml for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libcrypto.so.1.1()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libprofiler.so.0()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Processing Dependency: libssl.so.1.1()(64bit) for package: 1:nginx-1.16.1-3.el7.x86_64
--> Running transaction check
---> Package centos-indexhtml.noarch 0:7-9.el7.centos will be installed
---> Package gperftools-libs.x86_64 0:2.6.1-1.el7 will be installed
---> Package nginx-all-modules.noarch 1:1.16.1-3.el7 will be installed
--> Processing Dependency: nginx-mod-http-image-filter = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-http-perl = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-http-xslt-filter = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-mail = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
--> Processing Dependency: nginx-mod-stream = 1:1.16.1-3.el7 for package: 1:nginx-all-modules-1.16.1-3.el7.noarch
---> Package nginx-filesystem.noarch 1:1.16.1-3.el7 will be installed
---> Package openssl11-libs.x86_64 1:1.1.1g-1.el7 will be installed
--> Running transaction check
---> Package nginx-mod-http-image-filter.x86_64 1:1.16.1-3.el7 will be installed
--> Processing Dependency: gd for package: 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libgd.so.2()(64bit) for package: 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64
---> Package nginx-mod-http-perl.x86_64 1:1.16.1-3.el7 will be installed
---> Package nginx-mod-http-xslt-filter.x86_64 1:1.16.1-3.el7 will be installed
--> Processing Dependency: libxslt.so.1(LIBXML2_1.0.11)(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libxslt.so.1(LIBXML2_1.0.18)(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libexslt.so.0()(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
--> Processing Dependency: libxslt.so.1()(64bit) for package: 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64
---> Package nginx-mod-mail.x86_64 1:1.16.1-3.el7 will be installed
---> Package nginx-mod-stream.x86_64 1:1.16.1-3.el7 will be installed
--> Running transaction check
---> Package gd.x86_64 0:2.0.35-26.el7 will be installed
--> Processing Dependency: libjpeg.so.62(LIBJPEG_6.2)(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libjpeg.so.62()(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libfontconfig.so.1()(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libXpm.so.4()(64bit) for package: gd-2.0.35-26.el7.x86_64
--> Processing Dependency: libX11.so.6()(64bit) for package: gd-2.0.35-26.el7.x86_64
---> Package libxslt.x86_64 0:1.1.28-6.el7 will be installed
--> Running transaction check
---> Package fontconfig.x86_64 0:2.13.0-4.3.el7 will be installed
--> Processing Dependency: fontpackages-filesystem for package: fontconfig-2.13.0-4.3.el7.x86_64
--> Processing Dependency: dejavu-sans-fonts for package: fontconfig-2.13.0-4.3.el7.x86_64
---> Package libX11.x86_64 0:1.6.7-3.el7_9 will be installed
--> Processing Dependency: libX11-common >= 1.6.7-3.el7_9 for package: libX11-1.6.7-3.el7_9.x86_64
--> Processing Dependency: libxcb.so.1()(64bit) for package: libX11-1.6.7-3.el7_9.x86_64
---> Package libXpm.x86_64 0:3.5.12-1.el7 will be installed
---> Package libjpeg-turbo.x86_64 0:1.2.90-8.el7 will be installed
--> Running transaction check
---> Package dejavu-sans-fonts.noarch 0:2.33-6.el7 will be installed
--> Processing Dependency: dejavu-fonts-common = 2.33-6.el7 for package: dejavu-sans-fonts-2.33-6.el7.noarch
---> Package fontpackages-filesystem.noarch 0:1.44-8.el7 will be installed
---> Package libX11-common.noarch 0:1.6.7-3.el7_9 will be installed
---> Package libxcb.x86_64 0:1.13-1.el7 will be installed
--> Processing Dependency: libXau.so.6()(64bit) for package: libxcb-1.13-1.el7.x86_64
--> Running transaction check
---> Package dejavu-fonts-common.noarch 0:2.33-6.el7 will be installed
---> Package libXau.x86_64 0:1.0.8-2.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================================
 Package                                                                        Arch                                                      Version                                                            Repository                                                  Size
==============================================================================================================================================================================================================================================================================
Installing:
 nginx                                                                          x86_64                                                    1:1.16.1-3.el7                                                     epel                                                       563 k
Installing for dependencies:
 centos-indexhtml                                                               noarch                                                    7-9.el7.centos                                                     base                                                        92 k
 dejavu-fonts-common                                                            noarch                                                    2.33-6.el7                                                         base                                                        64 k
 dejavu-sans-fonts                                                              noarch                                                    2.33-6.el7                                                         base                                                       1.4 M
 fontconfig                                                                     x86_64                                                    2.13.0-4.3.el7                                                     base                                                       254 k
 fontpackages-filesystem                                                        noarch                                                    1.44-8.el7                                                         base                                                       9.9 k
 gd                                                                             x86_64                                                    2.0.35-26.el7                                                      base                                                       146 k
 gperftools-libs                                                                x86_64                                                    2.6.1-1.el7                                                        base                                                       272 k
 libX11                                                                         x86_64                                                    1.6.7-3.el7_9                                                      updates                                                    607 k
 libX11-common                                                                  noarch                                                    1.6.7-3.el7_9                                                      updates                                                    164 k
 libXau                                                                         x86_64                                                    1.0.8-2.1.el7                                                      base                                                        29 k
 libXpm                                                                         x86_64                                                    3.5.12-1.el7                                                       base                                                        55 k
 libjpeg-turbo                                                                  x86_64                                                    1.2.90-8.el7                                                       base                                                       135 k
 libxcb                                                                         x86_64                                                    1.13-1.el7                                                         base                                                       214 k
 libxslt                                                                        x86_64                                                    1.1.28-6.el7                                                       base                                                       242 k
 nginx-all-modules                                                              noarch                                                    1:1.16.1-3.el7                                                     epel                                                        20 k
 nginx-filesystem                                                               noarch                                                    1:1.16.1-3.el7                                                     epel                                                        21 k
 nginx-mod-http-image-filter                                                    x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        30 k
 nginx-mod-http-perl                                                            x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        39 k
 nginx-mod-http-xslt-filter                                                     x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        29 k
 nginx-mod-mail                                                                 x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        57 k
 nginx-mod-stream                                                               x86_64                                                    1:1.16.1-3.el7                                                     epel                                                        85 k
 openssl11-libs                                                                 x86_64                                                    1:1.1.1g-1.el7                                                     epel                                                       1.4 M

Transaction Summary
==============================================================================================================================================================================================================================================================================
Install  1 Package (+22 Dependent packages)

Total download size: 5.9 M
Installed size: 18 M
Is this ok [y/d/N]: y
Downloading packages:
warning: /var/cache/yum/x86_64/7/base/packages/centos-indexhtml-7-9.el7.centos.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Public key for centos-indexhtml-7-9.el7.centos.noarch.rpm is not installed
(1/23): centos-indexhtml-7-9.el7.centos.noarch.rpm                                                                                                                                                                                                     |  92 kB  00:00:00     
(2/23): dejavu-sans-fonts-2.33-6.el7.noarch.rpm                                                                                                                                                                                                        | 1.4 MB  00:00:00     
(3/23): fontconfig-2.13.0-4.3.el7.x86_64.rpm                                                                                                                                                                                                           | 254 kB  00:00:00     
(4/23): fontpackages-filesystem-1.44-8.el7.noarch.rpm                                                                                                                                                                                                  | 9.9 kB  00:00:00     
(5/23): gd-2.0.35-26.el7.x86_64.rpm                                                                                                                                                                                                                    | 146 kB  00:00:00     
(6/23): gperftools-libs-2.6.1-1.el7.x86_64.rpm                                                                                                                                                                                                         | 272 kB  00:00:00     
(7/23): libXau-1.0.8-2.1.el7.x86_64.rpm                                                                                                                                                                                                                |  29 kB  00:00:00     
(8/23): libjpeg-turbo-1.2.90-8.el7.x86_64.rpm                                                                                                                                                                                                          | 135 kB  00:00:00     
(9/23): dejavu-fonts-common-2.33-6.el7.noarch.rpm                                                                                                                                                                                                      |  64 kB  00:00:00     
(10/23): libxcb-1.13-1.el7.x86_64.rpm                                                                                                                                                                                                                  | 214 kB  00:00:00     
warning: /var/cache/yum/x86_64/7/epel/packages/nginx-1.16.1-3.el7.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID 352c64e5: NOKEY
Public key for nginx-1.16.1-3.el7.x86_64.rpm is not installed
(11/23): nginx-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                                 | 563 kB  00:00:00     
(12/23): nginx-all-modules-1.16.1-3.el7.noarch.rpm                                                                                                                                                                                                     |  20 kB  00:00:00     
(13/23): nginx-filesystem-1.16.1-3.el7.noarch.rpm                                                                                                                                                                                                      |  21 kB  00:00:00     
(14/23): nginx-mod-http-image-filter-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                           |  30 kB  00:00:00     
(15/23): nginx-mod-http-perl-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                   |  39 kB  00:00:00     
(16/23): nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                            |  29 kB  00:00:00     
(17/23): nginx-mod-mail-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                        |  57 kB  00:00:00     
(18/23): nginx-mod-stream-1.16.1-3.el7.x86_64.rpm                                                                                                                                                                                                      |  85 kB  00:00:00     
(19/23): libxslt-1.1.28-6.el7.x86_64.rpm                                                                                                                                                                                                               | 242 kB  00:00:00     
(20/23): openssl11-libs-1.1.1g-1.el7.x86_64.rpm                                                                                                                                                                                                        | 1.4 MB  00:00:00     
(21/23): libXpm-3.5.12-1.el7.x86_64.rpm                                                                                                                                                                                                                |  55 kB  00:00:00     
Public key for libX11-common-1.6.7-3.el7_9.noarch.rpm is not installed
(22/23): libX11-common-1.6.7-3.el7_9.noarch.rpm                                                                                                                                                                                                        | 164 kB  00:00:00     
(23/23): libX11-1.6.7-3.el7_9.x86_64.rpm                                                                                                                                                                                                               | 607 kB  00:00:00     
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                                          14 MB/s | 5.9 MB  00:00:00     
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
 Userid     : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
 Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
 Package    : centos-release-7-5.1804.el7.centos.x86_64 (@anaconda)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Is this ok [y/N]: y
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Importing GPG key 0x352C64E5:
 Userid     : "Fedora EPEL (7) <epel@fedoraproject.org>"
 Fingerprint: 91e9 7d7c 4a5e 96f1 7f3e 888f 6a2f aea2 352c 64e5
 Package    : epel-release-7-12.noarch (@epel)
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Warning: RPMDB altered outside of yum.
  Installing : fontpackages-filesystem-1.44-8.el7.noarch                                                                                                                                                                                                                 1/23 
  Installing : dejavu-fonts-common-2.33-6.el7.noarch                                                                                                                                                                                                                     2/23 
  Installing : dejavu-sans-fonts-2.33-6.el7.noarch                                                                                                                                                                                                                       3/23 
  Installing : fontconfig-2.13.0-4.3.el7.x86_64                                                                                                                                                                                                                          4/23 
  Installing : gperftools-libs-2.6.1-1.el7.x86_64                                                                                                                                                                                                                        5/23 
  Installing : 1:nginx-filesystem-1.16.1-3.el7.noarch                                                                                                                                                                                                                    6/23 
  Installing : libxslt-1.1.28-6.el7.x86_64                                                                                                                                                                                                                               7/23 
  Installing : libXau-1.0.8-2.1.el7.x86_64                                                                                                                                                                                                                               8/23 
  Installing : libxcb-1.13-1.el7.x86_64                                                                                                                                                                                                                                  9/23 
  Installing : 1:openssl11-libs-1.1.1g-1.el7.x86_64                                                                                                                                                                                                                     10/23 
  Installing : centos-indexhtml-7-9.el7.centos.noarch                                                                                                                                                                                                                   11/23 
  Installing : libX11-common-1.6.7-3.el7_9.noarch                                                                                                                                                                                                                       12/23 
  Installing : libX11-1.6.7-3.el7_9.x86_64                                                                                                                                                                                                                              13/23 
  Installing : libXpm-3.5.12-1.el7.x86_64                                                                                                                                                                                                                               14/23 
  Installing : libjpeg-turbo-1.2.90-8.el7.x86_64                                                                                                                                                                                                                        15/23 
  Installing : gd-2.0.35-26.el7.x86_64                                                                                                                                                                                                                                  16/23 
  Installing : 1:nginx-mod-mail-1.16.1-3.el7.x86_64                                                                                                                                                                                                                     17/23 
  Installing : 1:nginx-mod-http-perl-1.16.1-3.el7.x86_64                                                                                                                                                                                                                18/23 
  Installing : 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                         19/23 
  Installing : 1:nginx-mod-stream-1.16.1-3.el7.x86_64                                                                                                                                                                                                                   20/23 
  Installing : 1:nginx-1.16.1-3.el7.x86_64                                                                                                                                                                                                                              21/23 
  Installing : 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                        22/23 
  Installing : 1:nginx-all-modules-1.16.1-3.el7.noarch                                                                                                                                                                                                                  23/23 
  Verifying  : fontconfig-2.13.0-4.3.el7.x86_64                                                                                                                                                                                                                          1/23 
  Verifying  : 1:nginx-mod-mail-1.16.1-3.el7.x86_64                                                                                                                                                                                                                      2/23 
  Verifying  : 1:nginx-1.16.1-3.el7.x86_64                                                                                                                                                                                                                               3/23 
  Verifying  : fontpackages-filesystem-1.44-8.el7.noarch                                                                                                                                                                                                                 4/23 
  Verifying  : dejavu-fonts-common-2.33-6.el7.noarch                                                                                                                                                                                                                     5/23 
  Verifying  : 1:nginx-mod-http-perl-1.16.1-3.el7.x86_64                                                                                                                                                                                                                 6/23 
  Verifying  : libxcb-1.13-1.el7.x86_64                                                                                                                                                                                                                                  7/23 
  Verifying  : libXpm-3.5.12-1.el7.x86_64                                                                                                                                                                                                                                8/23 
  Verifying  : libjpeg-turbo-1.2.90-8.el7.x86_64                                                                                                                                                                                                                         9/23 
  Verifying  : libX11-common-1.6.7-3.el7_9.noarch                                                                                                                                                                                                                       10/23 
  Verifying  : libX11-1.6.7-3.el7_9.x86_64                                                                                                                                                                                                                              11/23 
  Verifying  : 1:nginx-mod-http-xslt-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                         12/23 
  Verifying  : dejavu-sans-fonts-2.33-6.el7.noarch                                                                                                                                                                                                                      13/23 
  Verifying  : gd-2.0.35-26.el7.x86_64                                                                                                                                                                                                                                  14/23 
  Verifying  : 1:nginx-mod-stream-1.16.1-3.el7.x86_64                                                                                                                                                                                                                   15/23 
  Verifying  : centos-indexhtml-7-9.el7.centos.noarch                                                                                                                                                                                                                   16/23 
  Verifying  : 1:openssl11-libs-1.1.1g-1.el7.x86_64                                                                                                                                                                                                                     17/23 
  Verifying  : libXau-1.0.8-2.1.el7.x86_64                                                                                                                                                                                                                              18/23 
  Verifying  : libxslt-1.1.28-6.el7.x86_64                                                                                                                                                                                                                              19/23 
  Verifying  : 1:nginx-filesystem-1.16.1-3.el7.noarch                                                                                                                                                                                                                   20/23 
  Verifying  : gperftools-libs-2.6.1-1.el7.x86_64                                                                                                                                                                                                                       21/23 
  Verifying  : 1:nginx-all-modules-1.16.1-3.el7.noarch                                                                                                                                                                                                                  22/23 
  Verifying  : 1:nginx-mod-http-image-filter-1.16.1-3.el7.x86_64                                                                                                                                                                                                        23/23 

Installed:
  nginx.x86_64 1:1.16.1-3.el7                                                                                                                                                                                                                                                 

Dependency Installed:
  centos-indexhtml.noarch 0:7-9.el7.centos          dejavu-fonts-common.noarch 0:2.33-6.el7  dejavu-sans-fonts.noarch 0:2.33-6.el7    fontconfig.x86_64 0:2.13.0-4.3.el7      fontpackages-filesystem.noarch 0:1.44-8.el7        gd.x86_64 0:2.0.35-26.el7                 
  gperftools-libs.x86_64 0:2.6.1-1.el7              libX11.x86_64 0:1.6.7-3.el7_9            libX11-common.noarch 0:1.6.7-3.el7_9     libXau.x86_64 0:1.0.8-2.1.el7           libXpm.x86_64 0:3.5.12-1.el7                       libjpeg-turbo.x86_64 0:1.2.90-8.el7       
  libxcb.x86_64 0:1.13-1.el7                        libxslt.x86_64 0:1.1.28-6.el7            nginx-all-modules.noarch 1:1.16.1-3.el7  nginx-filesystem.noarch 1:1.16.1-3.el7  nginx-mod-http-image-filter.x86_64 1:1.16.1-3.el7  nginx-mod-http-perl.x86_64 1:1.16.1-3.el7 
  nginx-mod-http-xslt-filter.x86_64 1:1.16.1-3.el7  nginx-mod-mail.x86_64 1:1.16.1-3.el7     nginx-mod-stream.x86_64 1:1.16.1-3.el7   openssl11-libs.x86_64 1:1.1.1g-1.el7   

Complete!
[root@ecs-ae8a-0001 ~]# which nginx
/usr/sbin/nginx

中间有几步Is this OK?的提问的地方输入y就行。
2、安装完成后
nginx日志目录为:/var/log/nginx
nginx配置文件目录为:/etc/nginx
3、修改/etc/nginx/nginx.conf,内容如下:

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  escape=none '$http_x_forwarded_for -[$time_local] "$request" '
			'$status $body_bytes_sent '
			'"#http_user_agent"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;
    
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
		proxy_pass http://127.0.0.1:5000;
        }

        error_page 404 /404.html;
        location = /404.html {
        }

        error_page 500 502 503 504 /50x.html;
        location = /50x.html {
        }
    }
}

上面的配置主要修改了2个内容

  • 把80端口接收到的请求转发到5000端口下处理。
  • 对访问日志的格式进行格式化处理。

4、启动nginx服务,在终端执行以下命令:

nginx

5、如果中间手动删除了/var/log/nginx目录下的日志文件,则需要让Nginx重新加载一下,否则不会自动生成新的日志文件,如下:

killall -s USR1 nginx

4、启动一个Mini Http Server处理5000端口的请求

1、为了能够处理Nginx转发至5000端口的请求,我们可以通过python开启一个httpserver,如下命令:

nohup phython -m SimpleHTTPServer 5000 &

5、filebeat下载、配置与安装

1、切换当前目录为/opt/soft

cd /opt/soft

2、执行以下命令下载filebeat,如下:

wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz

3、解压filebeat

tar -xzvf filebeat-6.2.4-linux-x86_64.tar.gz

4、复制filebeat至/program/目录下:

cp -r filebeat-6.2.4-linux-x86_64 /program/

5、切换当前目录至/program/filebeat-6.2.4-linux-x86_64/

cd /program/filebeat-6.2.4-linux-x86_64

6、创建一个空文件kafka.yml

touch kafka.yml

7、通过vim kafka.yml修改kafka.yml文件内容,内容如下:

filebeat:
  prospectors:
    -
      paths:
        - /var/log/nginx/access.log
      input_type: log
output:
  kafka:
    hosts: ["hadoop01:9092","hadoop02:9092","hadoop03:9092"]
    topic: qingdao

6、启动filebeat开启日志收集

执行以下命令

nohup ./filebeat -e -c kafka.yml &

7、通过java程序发送http请求至nginx,使nginx产生访问日志。

1、运行test01的模拟java程序。
2、在浏览器地址栏输入访问地址,访问格式如下:

http:// 启 动 服 务 的 i p 及 端 口 / t e s t / m o c k d a t a / s t a r t ? u r l = h t t p : / / {启动服务的ip及端口}/test/mockdata/start?url=http:// ip/test/mockdata/start?url=http://{nginx的ip}&thread=1

在这里插入图片描述
在这里插入图片描述

8、通过java程序模拟kafka消息消费

1、运行kafka消费者程序(完整代码),执行效果如下:

在这里插入图片描述

9、结尾

至此使用filebeat收集日志数据至kafka集群的环境就搭建完成了,小伙伴们如果觉得有用就点个赞吧~

已标记关键词 清除标记
©️2020 CSDN 皮肤主题: 编程工作室 设计师:CSDN官方博客 返回首页