文章目录

• 1、基础环境
• 2、开发环境
• • 2.1、pom.xml
  • 2.2、log4j.properties
  • 2.3、测试用的代码
  • • 2.3.1、Kafka工具
    • 2.3.3、Flink执行环境工具
    • 2.3.3、测试Flink读写Kafka
    • 2.3.4、测试FlinkSQL读写Kafka
  • 2.4、打包后上传到服务器
• 3、生产环境
• • 3.1、Flink安装
  • 3.2、Flink on YARN下3种模式
  • • 3.2.1、Session-Cluster
    • 3.2.2、Per-Job Cluster Mode
    • 3.2.3、Application Mode
  • 3.3、关闭Flink

1、基础环境

• 开发环境：Windows
  WIN10+JDK1.8+IDEA2022.3+Maven3.8.1
• 生产环境：Linux
  CentOS7.5+JDK1.8+Hadoop3.1.3+Kafka3.0.0
  Flink集群生产环境常用部署模式为 YARN模式

2、开发环境

创建Maven项目

2.1、pom.xml

本依赖支持：Flink、FlinkSQL、Flink读写Kafka、JSON解析

88UTF-81.13.62.123.1.32.0.52.19.01.2.83


org.apache.flinkflink-java${flink.version}org.apache.flinkflink-streaming-java_${scala.version}${flink.version}org.apache.flinkflink-streaming-scala_${scala.version}${flink.version}org.apache.flinkflink-clients_${scala.version}${flink.version}org.apache.flinkflink-runtime-web_${scala.version}${flink.version}org.apache.flinkflink-connector-kafka_${scala.version}${flink.version}org.apache.flinkflink-table-planner-blink_${scala.version}${flink.version}org.apache.flinkflink-json${flink.version}org.apache.hadoophadoop-client${hadoop.version}com.alibabafastjson${fastjson.version}org.slf4jslf4j-api${slf4j.version}org.slf4jslf4j-log4j12${slf4j.version}org.apache.logging.log4jlog4j-to-slf4j${log4j.version}


org.apache.maven.pluginsmaven-shade-plugin3.1.1packageshadecom.google.code.findbugs:jsr305org.slf4j:*log4j:**:*META-INF/*.SFMETA-INF/*.DSAMETA-INF/*.RSA

2.2、log4j.properties

log4j.rootLogger=error, stdout
log4j.appender.stdout=org.apache.log4j.ConsoleAppender
log4j.appender.stdout.layout=org.apache.log4j.PatternLayout
log4j.appender.stdout.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

2.3、测试用的代码

代码架构

2.3.1、Kafka工具

package org.example;import org.apache.flink.api.common.serialization.SimpleStringSchema;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer;import java.util.Properties;import org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer;public class KafkaTool {private static final String BOOTSTRAP_SERVER = "hadoop105:9092,hadoop106:9092,hadoop107:9092";private static final String CONSUMER_GROUP_ID = "Flink01";public static FlinkKafkaProducer getFlinkKafkaProducer(String topic) {return new FlinkKafkaProducer<>(BOOTSTRAP_SERVER, topic, new SimpleStringSchema());}public static FlinkKafkaConsumer getFlinkKafkaConsumer(String topic) {Properties properties = new Properties();properties.setProperty("bootstrap.servers", BOOTSTRAP_SERVER);properties.setProperty("group.id", CONSUMER_GROUP_ID);properties.setProperty("auto.offset.reset", "latest");return new FlinkKafkaConsumer<>(topic, new SimpleStringSchema(), properties);}public static DataStreamSource getKafkaSource(StreamExecutionEnvironment env, String topic) {return env.addSource(getFlinkKafkaConsumer(topic));}public static String getInputTable(String topic) {return " WITH ("+ "'connector'='kafka',"+ "'topic'='" + topic + "',"+ "'properties.bootstrap.servers'='" + BOOTSTRAP_SERVER + "',"+ "'properties.group.id'='" + CONSUMER_GROUP_ID + "',"+ "'scan.startup.mode'='latest-offset',"+ "'format'='json'"+ ")";}public static String getOutputTable(String topic) {return " WITH ("+ "'connector'='kafka',"+ "'topic'='" + topic + "',"+ "'properties.bootstrap.servers'='" + BOOTSTRAP_SERVER + "',"+ "'format'='json'"+ ")";}
}

2.3.3、Flink执行环境工具

package org.example;import org.apache.flink.api.common.restartstrategy.RestartStrategies;
import org.apache.flink.runtime.state.hashmap.HashMapStateBackend;
import org.apache.flink.streaming.api.CheckpointingMode;
import org.apache.flink.streaming.api.environment.CheckpointConfig;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;public class FlinkEnv {private final static String CHECKPOINT_DIRECTORY = "hdfs://hadoop105:8020/Flink/Checkpoint";private final static String HADOOP_USER_NAME = "hjw";public static StreamExecutionEnvironment getEnv() {//创建流执行环境StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();//设置并行度（=Kafka分区数）env.setParallelism(3);//获取checkpoint撇嘴CheckpointConfig checkpointConfig = env.getCheckpointConfig();//开启CheckPoint：每隔5分钟1次，精准一次模式env.enableCheckpointing(300 * 1000L, CheckpointingMode.EXACTLY_ONCE);//设置CheckPoint超时：10分钟checkpointConfig.setCheckpointTimeout(600 * 1000L);//设置Checkpoint最大数量（10/5=2）checkpointConfig.setMaxConcurrentCheckpoints(2);//设置重启策略：重启3次，执行延时5秒env.setRestartStrategy(RestartStrategies.fixedDelayRestart(3, 5000L));//设置状态后端env.setStateBackend(new HashMapStateBackend());checkpointConfig.setCheckpointStorage(CHECKPOINT_DIRECTORY);System.setProperty("HADOOP_USER_NAME", HADOOP_USER_NAME);//返回return env;}
}

2.3.3、测试Flink读写Kafka

package org.example.test;import com.alibaba.fastjson.JSONObject;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.example.FlinkEnv;
import org.example.KafkaTool;public class FlinkTest {public static void main(String[] args) throws Exception {StreamExecutionEnvironment env = FlinkEnv.getEnv();DataStreamSource kafkaSource = KafkaTool.getKafkaSource(env, "topic01");SingleOutputStreamOperator s = kafkaSource.map(JSONObject::parseObject).map(Object::toString);s.addSink(KafkaTool.getFlinkKafkaProducer("topic02"));env.execute();}
}

2.3.4、测试FlinkSQL读写Kafka

package org.example.test;import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
import org.example.FlinkEnv;
import org.example.KafkaTool;public class FlinkSqlTest {public static void main(String[] args) {//TODO 1 创建执行环境StreamExecutionEnvironment env = FlinkEnv.getEnv();StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);//TODO 2 数据来源tableEnv.executeSql("CREATE TABLE tb1 (database STRING, ts BIGINT, data STRING, type STRING)"+ KafkaTool.getInputTable("topic02"));//TODO 3 数据终点tableEnv.executeSql("CREATE TABLE tb2 (database STRING, ts BIGINT, data STRING, type STRING)"+ KafkaTool.getOutputTable("topic03"));//TODO 4 插入数据tableEnv.executeSql("INSERT INTO tb2 SELECT * FROM tb1 WHERE type IN ('insert','update')");}
}

2.4、打包后上传到服务器

打包后

把带original的jar上传到服务器（文件更小，上传更快）

3、生产环境

Flink集群YARN模式

1. 把Flink应用提交给YARN的ResourceManager
2. 从NodeManager上申请容器
3. 在容器内创建JobManager和TaskManager
4. 根据运行在JobManger上的Job的需要的插槽数量，动态地给TaskManager分配资源

3.1、Flink安装

1、解压

wget -b https://archive.apache.org/dist/flink/flink-1.13.6/flink-1.13.6-bin-scala_2.12.tgz
tar -zxf flink-1.13.6-bin-scala_2.12.tgz
mv flink-1.13.6 /opt/module/flink

2、环境变量

vim /etc/profile.d/my_env.sh

export HADOOP_CLASSPATH=`hadoop classpath`

3、分发环境变量

source ~/bin/source.sh

4、Per-Job-Cluster时报错：Exception in thread “Thread-5” java.lang.IllegalStateException:
Trying to access closed classloader.
Please check if you store classloaders directly or indirectly in static fields.
If the stacktrace suggests that the leak occurs in a third party library and cannot be fixed immediately,
you can disable this check with the configuration ‘classloader.check-leaked-classloader’.
对此，编辑配置文件

vim /opt/module/flink/conf/flink-conf.yaml

在配置文件添加下面这行，可解决上面报错

classloader.check-leaked-classloader: false

5、下载Flink-Kafka和fastjson的jar（去Maven官网找链接）

cd /opt/module/flink/lib
wget https://repo1.maven.org/maven2/org/apache/flink/flink-sql-connector-kafka_2.12/1.13.6/flink-sql-connector-kafka_2.12-1.13.6.jar
wget https://repo1.maven.org/maven2/com/alibaba/fastjson/1.2.83/fastjson-1.2.83.jar

3.2、Flink on YARN下3种模式

Session-Cluster

在YARN中初始化一个Flink集群，开辟一定的资源
这个Flink集群会常驻在YARN集群中，可持续接收Job
若资源用完，则新的Job不能正常提交

Per-Job-Cluster、Application Mode

每次提交都会创建一个新的Flink集群，任务之间互相独立
任务执行完成之后，Flink集群也会消失

3.2.1、Session-Cluster

1、开启会话

/opt/module/flink/bin/yarn-session.sh \
-s 1 \
-jm 1024 \
-tm 1024 \
-nm a1 \
-d

2、在会话上运行jar

/opt/module/flink/bin/flink run -c org.example.test.FlinkTest FlinkDW-1.0-SNAPSHOT.jar

/opt/module/flink/bin/flink run -c org.example.test.FlinkSqlTest FlinkDW-1.0-SNAPSHOT.jar

3.2.2、Per-Job Cluster Mode

/opt/module/flink/bin/flink run \
-t yarn-per-job \
-nm a2 \
-ys 1 \
-yjm 1024 \
-ytm 1024 \
-c org.example.test.FlinkSqlTest \
original-FlinkDW-1.0-SNAPSHOT.jar

3.2.3、Application Mode

/opt/module/flink/bin/flink run-application \
-t yarn-application \
-nm a3 \
-ys 1 \
-yjm 1024 \
-ytm 1024 \
-c org.example.test.FlinkTest \
original-FlinkDW-1.0-SNAPSHOT.jar

3.3、关闭Flink

yarn top可动态查看应用状况：应用名、应用所属用户、应用开始时间、容器数量、内存和CPU消耗…

yarn top

使用yarn application --list查看应用ID，使用yarn application --kill关闭会话