B: Installing the Cloud Logging agent on a single VM  |  Google Cloud

code:fluentd

# Master configuration file for google-fluentd

# Include any configuration files in the config.d directory.

#

# An example "catch-all" configuration can be found at

@include config.d/*.conf

# Prometheus monitoring.

<source>

@type prometheus

port 24231

</source>

<source>

@type prometheus_monitor

</source>

# Do not collect fluentd's own logs to avoid infinite loops.

<match fluent.**>

@type null

</match>

# Add a unique insertId to each log entry that doesn't already have it.

# This helps guarantee the order and prevent log duplication.

<filter **>

@type add_insert_ids

</filter>

# Configure all sources to output to Google Cloud Logging

<match **>

@type google_cloud

buffer_type file

buffer_path /var/log/google-fluentd/buffers

# Set the chunk limit conservatively to avoid exceeding the recommended

# chunk size of 5MB per write request.

buffer_chunk_limit 512KB

# Flush logs every 5 seconds, even if the buffer is not full.

flush_interval 5s

# Enforce some limit on the number of retries.

disable_retry_limit false

# After 3 retries, a given chunk will be discarded.

retry_limit 3

# Wait 10 seconds before the first retry. The wait interval will be doubled on

# each following retry (20s, 40s...) until it hits the retry limit.

retry_wait 10

# Never wait longer than 5 minutes between retries. If the wait interval

# reaches this limit, the exponentiation stops.

# Given the default config, this limit should never be reached, but if

# retry_limit and retry_wait are customized, this limit might take effect.

max_retry_wait 300

# Use multiple threads for processing.

num_threads 8

# Use the gRPC transport.

use_grpc true

# If a request is a mix of valid log entries and invalid ones, ingest the

# valid ones and drop the invalid ones instead of dropping everything.

partial_success true

# Enable monitoring via Prometheus integration.

enable_monitoring true

monitoring_type opencensus

</match>

code:bash

$ ls -lFa /etc/google-fluentd/config.d

total 120

drwxr-xr-x 2 root root 4096 Jan 9 05:02 ./

drwxr-xr-x 4 google-fluentd google-fluentd 4096 Jan 9 05:02 ../

-rw-r--r-- 1 root root 526 Mar 21 2018 apache.conf

-rw-r--r-- 1 root root 624 Mar 21 2018 cassandra.conf

-rw-r--r-- 1 root root 2471 Mar 21 2018 chef.conf

-rw-r--r-- 1 root root 162 Mar 21 2018 forward.conf

-rw-r--r-- 1 root root 1614 Mar 21 2018 gitlab.conf

-rw-r--r-- 1 root root 238 Mar 21 2018 jenkins.conf

-rw-r--r-- 1 root root 537 Mar 21 2018 jetty.conf

-rw-r--r-- 1 root root 322 Mar 21 2018 joomla.conf

-rw-r--r-- 1 root root 747 Mar 21 2018 magento.conf

-rw-r--r-- 1 root root 368 Mar 21 2018 mediawiki.conf

-rw-r--r-- 1 root root 167 Mar 21 2018 memcached.conf

-rw-r--r-- 1 root root 163 Mar 21 2018 mongodb.conf

-rw-r--r-- 1 root root 936 Mar 21 2018 mysql.conf

-rw-r--r-- 1 root root 351 Mar 21 2018 nginx.conf

-rw-r--r-- 1 root root 192 Mar 21 2018 postgresql.conf

-rw-r--r-- 1 root root 5197 Mar 21 2018 puppet-enterprise.conf

-rw-r--r-- 1 root root 485 Mar 21 2018 puppet.conf

-rw-r--r-- 1 root root 1096 Mar 21 2018 rabbitmq.conf

-rw-r--r-- 1 root root 177 Mar 21 2018 redis.conf

-rw-r--r-- 1 root root 163 Mar 21 2018 redmine.conf

-rw-r--r-- 1 root root 674 Mar 21 2018 salt.conf

-rw-r--r-- 1 root root 154 Mar 21 2018 solr.conf

-rw-r--r-- 1 root root 166 Mar 21 2018 sugarcrm.conf

-rw-r--r-- 1 root root 230 Mar 21 2018 syslog.conf

-rw-r--r-- 1 root root 271 Mar 21 2018 syslog_endpoint.conf

-rw-r--r-- 1 root root 639 Mar 21 2018 tomcat.conf

-rw-r--r-- 1 root root 373 Mar 21 2018 zookeeper.conf

code:/etc/google-fluentd/config.d/nginx.conf

<source>

@type tail

format nginx

path /var/log/nginx/access.log

pos_file /var/lib/google-fluentd/pos/nginx-access.pos

read_from_head true

tag nginx-access

</source>

<source>

@type tail

format none

path /var/log/nginx/error.log

pos_file /var/lib/google-fluentd/pos/nginx-error.pos

read_from_head true

tag nginx-error

</source>

code:etc/google-fluentd/config.d/forward.conf

<source>

@type forward

# default port

port 24224

# only accept connections from localhost - to open this up, change to 0.0.0.0

bind 127.0.0.1

</source>

root の conf は変わってなさそうだった。内容かぶってんのかなと思ったけど若干違うか。

デフォルトのやつは prometheus ね、ふんふん、聞いたことはある。 soundcloud で使われてるのか。モニタリングとかアラート。ということは vs としては AWS CloudWatch や GCP monitoring とか alert ということになるのかな。 GCP が内部で使ってるってこと？わからん。

うん、ちょっと無理やりエラー起こしてみないとわからんやつ

code:bash

nmap localhost -p 24224

Nmap scan report for localhost (127.0.0.1)

Host is up (0.000070s latency).

PORT STATE SERVICE

24224/tcp open unknown

ポートはおきてらっしゃる

https://gyazo.com/016c9f2aedf6616172fee06d08c6873d

成功！やったぜ。はじめての fluentd ~

やっぱエラー見にくいし積極的に prod でも sourcemap すべきだな。仕組み詳しく知らないからなんとも言えないけどエラー起きたあとに南下するだけだろうしゼロコストじゃないかな…？（適当）

というかソースマップに対する観点として、正直バックエンドはよくて、フロントエンド、というか配信されるものはくっつけたら minify の意味ないので、いらない & つけない、かな。

（作業終わったらデバッグ用とかのログは切ったほうが良さそう、クソコストかかりそう。）

https://gyazo.com/8009e310dfb28d6c8ce252b673c7655c

必要ないルール消すとかの工夫はありそうだけど。