merge upstream changes before adding support for torchbench#8

Merged

tianyu-l merged 55 commits intomainfrom benchmark

Aug 13, 2024

+3,378-45,945

Commits on Jun 26, 2024

by default disable heavy memory profiling
tianyu-l
committed

Commits on Jun 27, 2024

Add the option to turn on async-TP
yifuwang
committed

Commits on Jul 1, 2024

Modifying memory estimation options and minor changes
sanketpurandare
committed

Commits on Jul 8, 2024

add comment pointing to Sequence Parallel optimization example
tianyu-l
committed
switch float8 logic from Float8DynamicLinear to Float8Linear (pytorch#436 )
vkuzo
authored

Commits on Jul 11, 2024

Commits on Jul 16, 2024

enable FSDP2 + fp8 all-gather and fix TP fp8 all-gather (pytorch#413 )
weifengpy
authored

Commits on Jul 18, 2024

Add support of DDP and experimental CompiledAutograd
fegin
committed

Commits on Jul 19, 2024

Commits on Jul 20, 2024

remove CI for FSDP2 + fp8 all-gather (pytorch#470 )
weifengpy
authored

Commits on Jul 21, 2024

dynamically update torch.compile cache config to ensure async tp support, enhance async tp UX (pytorch#471 )
lessw2020
authored

Commits on Jul 26, 2024

Fix 8gpu PP failure due to 2D DCP disablement
wconstab
committed
update float8 integration after UX changes (pytorch#484 )
vkuzo
authored
Re-enable FSDP2 Mem Tracker integration tests
Sanket Jayant Purandare
committed

Commits on Jul 29, 2024

Used partial instead of global vars for LR scheduling
awgu
committed

Commits on Aug 2, 2024

[TP] Infer local n_heads instead of ad-hoc model changes
kwen2501
committed

Commits on Aug 3, 2024

some compile-related updates
tianyu-l
committed

Commits on Aug 5, 2024

Commits on Aug 6, 2024

Add warning to compile rmsnorm (pytorch#505 )
wanchaol
authored

Commits on Aug 7, 2024

Commits on Aug 8, 2024

Commits on Aug 9, 2024

Commits on Aug 13, 2024

merge upstream changes
tianyu-l
committed