Refactor inf2 streamer handler

[lxning]

Description

Please read our CONTRIBUTING.md prior to creating your first pull request.

Please include a summary of the feature or issue being fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes #(issue)

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Feature/Issue validation/testing

Please describe the Unit or Integration tests that you ran to verify your changes and relevant result summary. Provide instructions so it can be reproduced.
Please also list any relevant details for your test configuration.

• Test on notebook: examples/large_models/inferentia2/llama2/streamer/inf2-llama-2-micro-batching.ipynb

python examples/large_models/utils/test_llm_streaming_response.py -m llama-2-70b --demo-streaming --prompt-text "Today the weather is really nice and I am planning on "
payload={'prompt': 'Today the weather is really nice and I am planning on ', 'max_new_tokens': 64}
, output=
Today the weather is really nice and I am planning on 200km ride. I am going to ride to the north of the city and then to the west. I am going to ride to the city of Krasnogorsk.
I am going to ride to the city of KrasTasks are completed

• Test on notebook: examples/large_models/inferentia2/llama2/continuous_batching/inf2-llama-2-continuous-batching.ipynb

python examples/large_models/utils/test_llm_streaming_response.py -m llama-2-70b -o 50 -t 2 -n 4 --prompt-text "Today the weather is really nice and I am planning on " --prompt-randomize
Tasks are completed
payload={'prompt': 'n j Today the weather is really nice and I am planning on ', 'max_new_tokens': 52}
, output=n j Today the weather is really nice and I am planning on 100% of the time.
I'm not sure if I'm going to be able to make it to the meetup. I'm going to be in the area, but I'm not sure if I'll be able

payload={'prompt': 't d e s k h e h y o z m a f e f x Today the weather is really nice and I am planning on ', 'max_new_tokens': 67}
, output=t d e s k h e h y o z m a f e f x Today the weather is really nice and I am planning on 100% going to the beach. I am going to go to the beach with my friends and we are going to have a lot of fun. I am going to go to the beach with my friends and we are going to have a lot of fun. I am going to go to the beach with my friends and we are

payload={'prompt': 'Today the weather is really nice and I am planning on ', 'max_new_tokens': 50}
, output=Today the weather is really nice and I am planning on 200km ride. I am going to ride to the north of the city and then to the west. I am going to ride to the city of Krasnogorsk.
I am going to ride to the city of Kras

payload={'prompt': 'c r y g g g v Today the weather is really nice and I am planning on ', 'max_new_tokens': 57}
, output=c r y g g g v Today the weather is really nice and I am planning on 100% going to the beach. I am going to take my camera and take some pictures of the beach and the ocean. I am going to take some pictures of the sunset and the sunrise. I am going to take some pictures of the waves and the sand

payload={'prompt': 'z m w n r Today the weather is really nice and I am planning on ', 'max_new_tokens': 55}
, output=z m w n r Today the weather is really nice and I am planning on 100% going to the beach. I am going to go to the beach with my friends and we are going to have a lot of fun. I am going to go to the beach with my friends and we are going to have a lot of fun. I am

payload={'prompt': 's e d t p d w Today the weather is really nice and I am planning on ', 'max_new_tokens': 57}
, output=s e d t p d w Today the weather is really nice and I am planning on 100% going to the beach. I am going to go to the beach with my friends and we are going to have a lot of fun. I am going to go to the beach with my friends and we are going to have a lot of fun. I am going to

payload={'prompt': 'u x t o s o t Today the weather is really nice and I am planning on ', 'max_new_tokens': 57}
, output=u x t o s o t Today the weather is really nice and I am planning on 100% enjoying it. I am going to go to the beach and then to the pool. I am going to have a great time with my friends and I am going to enjoy the sun. I am going to have a great time and I am going to enjoy the

payload={'prompt': 'q p f z v z t u o e f d g Today the weather is really nice and I am planning on ', 'max_new_tokens': 63}
, output=q p f z v z t u o e f d g Today the weather is really nice and I am planning on 1. (go) to the park with my friends. We are going to2. (play) football and then we are going to3. (have) a picnic. I am going to4. (bring) some sandwiches and some fruit. I am also going to

Checklist:

• Did you have fun?
• Have you added tests that prove your fix is effective or that this feature works?
• Has code been commented, particularly in hard-to-understand areas?
• Have you made corresponding changes to the documentation?

[mreso]

LGTM, but lets move the neuronx handlers into the example folder, see comments

[mreso]

LGTM