unexpected keyword argument 'block_list'

[yuhuanyuan]

python3 needle_in_haystack_with_mask.py
TypeError: Qwen2ForCausalLM.forward() got an unexpected keyword argument 'block_list'

[zhouliang-yu]

@yuhuanyuan
I met exactly same issue when following the default setting by using from source.modeling_qwen2 import Qwen2ForCausalModel, and got the same error
TypeError: Qwen2ForCausalLM.forward() got an unexpected keyword argument 'block_list'

If you look at the code at faiss_attn/source/modeling_qwen2.py, you would notice that the class Qwen2ForCausalModel did not implement a mask operation indicator which is the block_list to indict the index of blocks to be masked.
However, faiss_attn/source/modeling_llama.py do have the block list in its LlamaForCausalLM, so I guess maybe the author somehow forget to add this feature in Qwen2ForCausalModel.

To solve the problem

simply use Qwen2Model by from source.modeling_qwen2 import Qwen2Model can make it work for

Retrieval_Head/needle_in_haystack_with_mask.py

Line 342 in 2d9fb9d

output, retrieval_score = self.decode(q_outputs, input_ids[:,-1], 50, block_list=block_list)

However, I dont know is it a workaround? or we can use Qwen2Model instead of Qwen2ForCausalLM for reimplement the result. Pls correct me if I was wrong. @nightdessert

[nightdessert]

@yuhuanyuan I met exactly same issue when following the default setting by using from source.modeling_qwen2 import Qwen2ForCausalModel, and got the same error TypeError: Qwen2ForCausalLM.forward() got an unexpected keyword argument 'block_list'

If you look at the code at faiss_attn/source/modeling_qwen2.py, you would notice that the class Qwen2ForCausalModel did not implement a mask operation indicator which is the block_list to indict the index of blocks to be masked. However, faiss_attn/source/modeling_llama.py do have the block list in its LlamaForCausalLM, so I guess maybe the author somehow forget to add this feature in Qwen2ForCausalModel.

To solve the problem

simply use Qwen2Model by from source.modeling_qwen2 import Qwen2Model can make it work for

Retrieval_Head/needle_in_haystack_with_mask.py

Line 342 in 2d9fb9d

output, retrieval_score = self.decode(q_outputs, input_ids[:,-1], 50, block_list=block_list)

However, I dont know is it a workaround? or we can use Qwen2Model instead of Qwen2ForCausalLM for reimplement the result. Pls correct me if I was wrong. @nightdessert

Hey guys, sorry for the waiting. I have updated the file 'source/modeling_qwen2.py'. Hope that works well.