Kinesis

Amazon Kinesis Data Streams ノート。

Intro

Kinesis Data Streams は、数十万(すうじゅうまん) のソースから毎秒(まいびょう) ギガバイト単位(たんい) のデータを継続的(けいぞくてき) にキャプチャできる、スケーラブルで耐久性(たいきゅうせい) のあるリアルタイムデータストリーミングサービス。

Setup Kinesis Agent On EC2

Create IAM Role

EC2 に付与(ふよ) :

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "kinesis:PutRecords"
            ],
            "Resource": "*"
        }
    ]
}

Install Agent

sudo yum update

sudo yum install –y aws-kinesis-agent

Create a folder for streaming data

cd /opt/
sudo mkdir app-stream-1
cd app-stream-1

Config

sudo cat /etc/aws-kinesis/agent.json

デフォルト設定(せってい) :

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "https://kinesis.ap-southeast-1.amazonaws.com",
  "firehose.endpoint": "",

  "flows": [
    {
      "filePattern": "/opt/app-stream-1/logfile.log",
      "kinesisStream": "kds-test",
      "partitionKeyOption": "RANDOM"
    },
    {
      "dataProcessingOptions": {
        "optionName": "CSVTOJSON",
        "customFieldNames": [ "field1", "field2" ],
        "delimiter": "yourdelimiter"
      },
      "filePattern": "/tmp/app.log*",
      "kinesisStream": "stream name",
      "initialPosition": "END_OF_FILE",
      "maxBufferAgeMillis": "60000",
      "maxBufferSizeBytes": "4194304",
      "maxBufferSizeRecords": "500",
      "minTimeBetweenFilePollsMillis": "100",
      "multiLineStartPattern": "",
      "partitionKeyOption": "RANDOM",
      "skipHeaderLines": "0",
      "truncatedRecordTerminator": "\n"
    }
  ]
}

Set permissions

sudo chown aws-kinesis-agent-user:aws-kinesis-agent-user -R /opt/stream-1

Kinesis Agent Services

sudo service aws-kinesis-agent status

sudo service aws-kinesis-agent start
sudo service aws-kinesis-agent restart
sudo service aws-kinesis-agent stop

Tail log files

アプリケーションログファイルを監視(かんし) :

sudo tail -f /opt/kds/file.log

Kinesis Agent ログファイルを監視(かんし) :

sudo tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

Glue Redshift