MapReduce作业比较大的 接入RSS 出现AppMaster内存不够用

[yanfangwang]

set mapred.max.split.size=1073741824;
set yarn.app.mapreduce.am.resource.mb=8192;
set yarn.app.mapreduce.am.command-opts=-Djava.net.preferIPv4Stack=true -Xmx12288m;
set mapreduce.map.memory.mb=8192;
set mapreduce.map.java.opts=-Xmx6144m;
set mapreduce.reduce.memory.mb=8192;
set mapreduce.reduce.java.opts=-Xmx6144m;
add jar hdfs:///tmp/celeborn-client-mr-shaded_2.12-0.4.0-incubating.jar;
set mapreduce.job.reduce.slowstart.completedmaps=1;
set mapreduce.celeborn.master.endpoints=yj-rsscelebornmaster-01:9097,yj-rsscelebornmaster-02:9097,yj-rsscelebornmaster-03:9097;
set mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.CelebornMapOutputCollector;
set mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.CelebornShuffleConsumer;
set yarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn;
set yarn.app.mapreduce.am.job.recovery.enable=false;
set celeborn.rpc.lookupTimeout=6000s

相关的参数

Application application_1713940740527_488674 failed 4 times (global limit =6; local limit is =4) due to AM Container for appattempt_1713940740527_488674_000004 exited with exitCode: -104
Failing this attempt.Diagnostics: [2024-05-09 09:57:30.819]Container [pid=223169,containerID=container_e38_1713940740527_488674_04_000001] is running 7455248384B beyond the 'PHYSICAL' memory limit. Current usage: 14.9 GB of 8 GB physical memory used; 34.4 GB of 16.8 GB virtual memory used. Killing container.

[FMX]

Can you get a memory dump?

[yanfangwang]

Yeah, yeah, it's a little big. No dump. I'll try to dump one

[yanfangwang]

目前用了14G，dump出来了5G左右

org.apache.celeborn.common.rpc.netty.Inbox这个占了2个多G

[FMX]

好的，感谢配合。我这边查一下。

[FMX]

这个图片还没有上传完成你就点击关闭编辑框了。

[yanfangwang]

我后边直接把压缩包上传了，你打开之后网页打开index.html就可以

[yanfangwang]

Uploading heap_Top_Components.zip…

[FMX]

压缩包也没有上传完。

[FMX]

@yanfangwang [CELEBORN-1314] add capacity-bounded inbox for rpc endpoint
看看这个 PR，我觉得和你的问题有关联。

[yanfangwang]

我合入之后还是没有解决 ,拿了一个标准sql q11.sql测试的，AM内存仍然用的很高 ,但是dump出来的内存只有几百M
add jar hdfs:///tmp/wyf/celeborn-client-mr-shaded_2.12-0.4.0-incubating.jar;
set mapreduce.job.reduce.slowstart.completedmaps=1;
set mapreduce.celeborn.master.endpoints=yj-rsscelebornmaster-01:9097,yj-rsscelebornmaster-02:9097,yj-rsscelebornmaster-03:9097;
set mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.CelebornMapOutputCollector;
set mapreduce.job.reduce.shuffle.consumer.plugin.class=org.apache.hadoop.mapreduce.task.reduce.CelebornShuffleConsumer;
set yarn.app.mapreduce.am.command-opts=org.apache.celeborn.mapreduce.v2.app.MRAppMasterWithCeleborn;
set yarn.app.mapreduce.am.job.recovery.enable=false;

use dwd_insys_test;

WITH year_total AS (
SELECT
c_customer_id customer_id,
c_first_name customer_first_name,
c_last_name customer_last_name,
c_preferred_cust_flag customer_preferred_cust_flag,
c_birth_country customer_birth_country,
c_login customer_login,
c_email_address customer_email_address,
d_year dyear,
sum(ss_ext_list_price - ss_ext_discount_amt) year_total,
's' sale_type
FROM customer, store_sales, date_dim
WHERE c_customer_sk = ss_customer_sk
AND ss_sold_date_sk = d_date_sk
GROUP BY c_customer_id
, c_first_name
, c_last_name
, d_year
, c_preferred_cust_flag
, c_birth_country
, c_login
, c_email_address
, d_year
UNION ALL
SELECT
c_customer_id customer_id,
c_first_name customer_first_name,
c_last_name customer_last_name,
c_preferred_cust_flag customer_preferred_cust_flag,
c_birth_country customer_birth_country,
c_login customer_login,
c_email_address customer_email_address,
d_year dyear,
sum(ws_ext_list_price - ws_ext_discount_amt) year_total,
'w' sale_type
FROM customer, web_sales, date_dim
WHERE c_customer_sk = ws_bill_customer_sk
AND ws_sold_date_sk = d_date_sk
GROUP BY
c_customer_id, c_first_name, c_last_name, c_preferred_cust_flag, c_birth_country,
c_login, c_email_address, d_year)
SELECT t_s_secyear.customer_preferred_cust_flag
FROM year_total t_s_firstyear
, year_total t_s_secyear
, year_total t_w_firstyear
, year_total t_w_secyear
WHERE t_s_secyear.customer_id = t_s_firstyear.customer_id
AND t_s_firstyear.customer_id = t_w_secyear.customer_id
AND t_s_firstyear.customer_id = t_w_firstyear.customer_id
AND t_s_firstyear.sale_type = 's'
AND t_w_firstyear.sale_type = 'w'
AND t_s_secyear.sale_type = 's'
AND t_w_secyear.sale_type = 'w'
AND t_s_firstyear.dyear = 2001
AND t_s_secyear.dyear = 2001 + 1
AND t_w_firstyear.dyear = 2001
AND t_w_secyear.dyear = 2001 + 1
AND t_s_firstyear.year_total > 0
AND t_w_firstyear.year_total > 0
AND CASE WHEN t_w_firstyear.year_total > 0
THEN t_w_secyear.year_total / t_w_firstyear.year_total
ELSE NULL END

CASE WHEN t_s_firstyear.year_total > 0
THEN t_s_secyear.year_total / t_s_firstyear.year_total
ELSE NULL END
ORDER BY t_s_secyear.customer_preferred_cust_flag
LIMIT 100

[yanfangwang]

[yanfangwang]

同样的我还跑了一个1万个map的hive作业，AM消耗的资源可以达到20G左右还没有跑完

同样 dump出来也只有4G左右，大部分感觉像是堆外用的，下图是在18G的时候dump出来的内存使用的情况

[FMX]

问题收到，我找环境重现一下问题。感谢反馈。

[FMX]

It's necessary to set the correct -Xmx for the application master.