[Bug] [seatunnal-server] task status is running,but pipelineStatus is FAILING

[13813586515]

Search before asking

• I had searched in the issues and found no similar issues.

What happened

任务运行后，通过seatunnel.sh -l 发现task是runing状态，但是通过seatunnel.sh -j jobid 发现pipelineStatus 是FAILING

SeaTunnel Version

2.3.7

SeaTunnel Config

env{
   job.name="future_mkt01_同步商品销售日结表"
  parallelism = 10
  job.mode = "BATCH"
}

source {
  Jdbc {
    url = "jdbc:oracle:thin:@192.168.98.38:1521:mkt01"
    driver = "oracle.jdbc.OracleDriver"
    connection_check_timeout_sec = 100
    user = "123"
    password = "123"
    split.size = 10000
    table_list=[
      {
        table_path = "DBUSRMKT.SALECOSTDAY"
        query = "select *  from DBUSRMKT.SALECOSTDAY where scddate>=TO_DATE('${this_day}', 'YYYY/MM/DD')"
      },
      {
        table_path = "DBUSRMKT.JXCGOODSLIST"
        query = "select *  from DBUSRMKT.JXCGOODSLIST where JGLFSDATE>=TO_DATE('${this_day}', 'YYYY/MM/DD')"
      }
    ]
  }
}



sink {
  Doris {
    fenodes = "192.168.0.168:8030,192.168.0.169:8030,192.168.0.170:8030"
    username = 123
    password = "123"
    database = "ods_future_mkt01"
    table = "ods_${table_name}"
    # 保证每个文件唯一，拷贝之后文件需要修改此处
    sink.label-prefix = "future-mkt01_${table_name}-jdbc"
    sink.enable-2pc = "true"
    sink.enable-delete = "true"
    save_mode_create_template="CREATE TABLE IF NOT EXISTS `${database}`.`${table_name}` (${rowtype_primary_key},${rowtype_fields},dt datetime(0) default current_timestamp(0) on update current_timestamp(0)) ENGINE=OLAP UNIQUE KEY (${rowtype_primary_key}) DISTRIBUTED BY HASH (${rowtype_primary_key}) BUCKETS 3 PROPERTIES ( \"replication_allocation\" = \"tag.location.default: 3\",\"in_memory\" = \"false\",\"storage_format\" = \"V2\", \"disable_auto_compaction\" = \"false\")"
    doris.config {
      format = "json"
      read_json_by_line = "true"
    }
  }
}

Running Command

${SEATUNNEL_HOME}/bin/seatunnel.sh --config /tmp/dolphinscheduler/exec/process/default/118604607510400/118604881362816_14/154516/198346/seatunnel_154516_198346.conf --deploy-mode cluster

Error Exception

任务无法结束，然后一直读取的数据是0

Zeta or Flink or Spark Version

zeta

Java or Scala Version

No response

Screenshots

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[corgy-w]

Your third picture and it’s wrong. Can provide more comprehensive info, ss zeta server run log and stack log

[13813586515]

Your third picture and it’s wrong. Can provide more comprehensive info, ss zeta server run log and stack log

2024-09-28 23:08:41,076 INFO [o.a.s.e.s.CoordinatorService ] [hz.main.generic-operation.thread-4] - [hadoop01]:55801 [seatunnel] [5.1] Received task end from execution TaskGroupLocation{jobId=892425438989647878, pipelineId=1, taskGroupId=30007}, state FAILED
2024-09-28 23:08:41,104 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.generic-operation.thread-4] - Job future_hdqtls_供应商结算单 (892425438989647878), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc]-SourceTask (8/10)] turned from state RUNNING to FAILED.
2024-09-28 23:08:41,104 INFO [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.generic-operation.thread-4] - Job future_hdqtls_供应商结算单 (892425438989647878), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc]-SourceTask (8/10)] state process is stopped
2024-09-28 23:08:41,104 ERROR [o.a.s.e.s.d.p.PhysicalVertex ] [hz.main.generic-operation.thread-4] - Job future_hdqtls_供应商结算单 (892425438989647878), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc]-SourceTask (8/10)] end with state FAILED and Exception: com.hazelcast.core.HazelcastException: java.lang.OutOfMemoryError: Java heap space
at com.hazelcast.internal.util.ExceptionUtil.lambda$static$0(ExceptionUtil.java:47)
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:125)
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:115)
at com.hazelcast.internal.util.ExceptionUtil.peel(ExceptionUtil.java:68)
at com.hazelcast.internal.util.ExceptionUtil.rethrow(ExceptionUtil.java:130)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:490)
at com.hazelcast.map.impl.proxy.MapProxySupport.getInternal(MapProxySupport.java:371)
at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:123)
at org.apache.seatunnel.engine.server.execution.TaskExecutionContext.getOrCreateMetricsContext(TaskExecutionContext.java:64)
at org.apache.seatunnel.engine.server.task.SeaTunnelTask.init(SeaTunnelTask.java:128)
at org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask.init(SourceSeaTunnelTask.java:72)
at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:715)
at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:1039)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
at ------ submitted from ------.()
at com.hazelcast.internal.util.ExceptionUtil.cloneExceptionWithFixedAsyncStackTrace(ExceptionUtil.java:336)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.returnOrThrowWithGetConventions(InvocationFuture.java:112)
at com.hazelcast.spi.impl.operationservice.impl.InvocationFuture.resolveAndThrowIfException(InvocationFuture.java:100)
at com.hazelcast.spi.impl.AbstractInvocationFuture.get(AbstractInvocationFuture.java:617)
at com.hazelcast.map.impl.proxy.MapProxySupport.invokeOperation(MapProxySupport.java:479)
... 12 more
在master日志中发现了这么一段，后来观察发现，只要运行一段时间，imap配置在hdfs，就会出现内存溢出的情况，导致任务失败，我现在配置的历史任务存储时间是8个小时，理论上不会有内存溢出，是否是因为现在配置的imap在hdfs但是还是会全部加载到内存中？

[corgy-w]

@13813586515 有dump的数据吗

[13813586515]

@13813586515 有dump的数据吗

群内大佬提醒，是否是impa挂载hdfs出现的内存泄露问题导致的

[13813586515]

@13813586515 有dump的数据吗

还发现一个问题，无论是挂载imap在hdfs还是本地节点，运行过程中的数据会不断地被加载到内存当中，不会被释放（即使设置了历史任务最大时间依然不会被释放），将seatunnel-server shut down 内存会被释放，但是如果再次启动又会加载imap中的数据到内存，导致重启时内存又一次溢出

[corgy-w]

dump file

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[shenzhy5]

I met the similar problem, the data has been extracted, but the job state is always running

[shenzhy5]

任务无法结束，然后一直读取的数据是0