Jar作业相关问题
更新时间: 2024-12-04 15:35:13
问题描述
用户Flink程序的依赖包与平台的内置依赖包冲突,导致提交失败。
解决方案
需要排除 flink 基础包、table、log4j 以及hadoop 相关的包。连接器相关的包需要打进jar包中
flink1.12引擎,集群目前已有的包,参考下图:
查看是否已包含Flink的包,如果存在,则需要排除集群相关的依赖,重新打包。
排包的pom.xml 案例
以下案例是除provide 之外的排除方式,只做参考,如果直接使用,可能会排除掉一些程序中用到的包,需要根据程序使用情况去排除
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.2</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<finalName>flink-demo</finalName>
<transformers>
<!-- append default configs -->
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
<artifactSet>
<excludes>
<exclude>org.apache.flink:flink-parquet</exclude>
<exclude>org.apache.flink:flink-s*</exclude>
<exclude>org.apache.flink:flink-t*</exclude>
<exclude>org.apache.flink:flink-c*</exclude>
<exclude>org.apache.flink:flink-f*</exclude>
<exclude>org.apache.flink:flink-o*</exclude>
<exclude>org.apache.flink:flink-a*</exclude>
<exclude>org.apache.flink:flink-m*</exclude>
<exclude>org.apache.flink:flink-r*</exclude>
<exclude>org.apache.flink:flink-q*</exclude>
<exclude>org.apache.flink:flink-java</exclude>
<exclude>org.apache.hadoop:*</exclude>
<exclude>log4j:log4j</exclude>
<exclude>org.slf4j:*</exclude>
<exclude>org.scala-lang:*</exclude>
<exclude>org.scala-lang.modules:*</exclude>
<exclude>com.typesafe.akka:*</exclude>
</excludes>
</artifactSet>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>