Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

wip add new proposed message structure #1904

Merged
merged 16 commits into from
Oct 13, 2024
Merged

wip add new proposed message structure #1904

merged 16 commits into from
Oct 13, 2024

Commits on Oct 13, 2024

  1. wip add new proposed message structure

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    ae835b5 View commit details
    Browse the repository at this point in the history

  2. tokenization

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    e7c7598 View commit details
    Browse the repository at this point in the history

  3. wip

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    c7e7a01 View commit details
    Browse the repository at this point in the history

  4. wip transform builder

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    8925b71 View commit details
    Browse the repository at this point in the history

  5. wip make the chat dataset loadable

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    4c607c1 View commit details
    Browse the repository at this point in the history

  6. wip chatml + llama 3 new chat objects

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    bff63c6 View commit details
    Browse the repository at this point in the history

  7. chore: lint

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    9ba5150 View commit details
    Browse the repository at this point in the history

  8. chore: lint

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    9deced2 View commit details
    Browse the repository at this point in the history

  9. fix tokenization

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    7c66bcc View commit details
    Browse the repository at this point in the history

  10. remove dacite dependency since we're using pydantic now

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    4716b9b View commit details
    Browse the repository at this point in the history

  11. fix handling when already correctly split in messages

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    413ce81 View commit details
    Browse the repository at this point in the history

  12. make sure to remove chat features from tokenized ds

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    71020d3 View commit details
    Browse the repository at this point in the history

  13. move chat to be a input transform for messages

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    f393bfc View commit details
    Browse the repository at this point in the history

  14. make sure llama3 has the bos token

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    9709e86 View commit details
    Browse the repository at this point in the history

  15. remove non-working special token code

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    0fcd6d3 View commit details
    Browse the repository at this point in the history

  16. fix messages strat loader

    @winglian
    winglian committed Oct 13, 2024
    Configuration menu
    Copy the full SHA
    0b717fb View commit details
    Browse the repository at this point in the history