摘要:本篇教程探讨了大数据技术之flume抽取mysql数据到kafka,希望阅读本篇文章以后大家有所收获,帮助大家对相关内容的理解更加深入。
本篇教程探讨了大数据技术之flume抽取mysql数据到kafka,希望阅读本篇文章以后大家有所收获,帮助大家对相关内容的理解更加深入。
flume安装:
1、下载地址
2、安装-下图
新建数据库和表
3、配置新增conf/mysql-flume.conf
[root@node191 apache-flume-1.8.0-bin]# cat conf/mysql-flume.conf a1.channels = ch-1a1.sources = src-1a1.sinks = k1###########sql source################## For each one of the sources, the type is defineda1.sources.src-1.type = org.keedio.flume.source.SQLSourcea1.sources.src-1.hibernate.connection.url = jdbc:mysql://192.168.3.191:3306/chenhuachao# Hibernate Database connection propertiesa1.sources.src-1.hibernate.connection.user = roota1.sources.src-1.hibernate.connection.password = abcd@1234a1.sources.src-1.hibernate.connection.autocommit = truea1.sources.src-1.hibernate.dialect = org.hibernate.dialect.MySQL5Dialecta1.sources.src-1.hibernate.connection.driver_class = com.mysql.jdbc.Drivera1.sources.src-1.run.query.delay=5000a1.sources.src-1.status.file.path = /opt/apache-flume-1.8.0-bina1.sources.src-1.status.file.name = sqlSource.status# Custom querya1.sources.src-1.start.from = 0a1.sources.src-1.custom.query = select `id`, `name` from testa1.sources.src-1.batch.size = 1000a1.sources.src-1.max.rows = 1000a1.sources.src-1.hibernate.connection.provider_class = org.hibernate.connection.C3P0ConnectionProvidera1.sources.src-1.hibernate.c3p0.min_size=1a1.sources.src-1.hibernate.c3p0.max_size=10################################################################a1.channels.ch-1.type = memorya1.channels.ch-1.capacity = 10000a1.channels.ch-1.transactionCapacity = 10000a1.channels.ch-1.byteCapacityBufferPercentage = 20a1.channels.ch-1.byteCapacity = 800000################################################################a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSinka1.sinks.k1.topic = TestTopica1.sinks.k1.brokerList = 192.168.3.191:9092,192.168.3.193:9092,192.168.3.194:9092a1.sinks.k1.requiredAcks = 1a1.sinks.k1.batchSize = 20a1.sinks.k1.channel = c1a1.sinks.k1.channel = ch-1a1.sources.src-1.channels=ch-1
4、添加mysql驱动到flume的lib目录下
$ wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.35.tar.gz$ tar xzf mysql-connector-java-5.1.35.tar.gz$ cp mysql-connector-java-5.1.35-bin.jar lib/
5、添加kafka的topic
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic TestTopic
6、启动flume agent
./bin/flume-ng agent -n a1 -c conf -f conf/mysql-flume.conf -Dflume.root.logger=INFO,console
7、操作数据库,新增表数据,查看topic数据
本文由职坐标整理发布,学习更多的相关知识,请关注职坐标IT知识库!
您输入的评论内容中包含违禁敏感词
我知道了
请输入正确的手机号码
请输入正确的验证码
您今天的短信下发次数太多了,明天再试试吧!
我们会在第一时间安排职业规划师联系您!
您也可以联系我们的职业规划师咨询:
版权所有 职坐标-一站式IT培训就业服务领导者 沪ICP备13042190号-4
上海海同信息科技有限公司 Copyright ©2015 www.zhizuobiao.com,All Rights Reserved.
沪公网安备 31011502005948号