Skip to content

Emojis are broken #3

Closed
Closed
Emojis are broken#3

Description

@tiguchi
tiguchi
opened on Sep 11, 2023

Problem

When using the LlamaModel.generate() method, the resulting tokens are corrupted in case the model generates emojis as response. Instead of getting the correct emojis, I get question mark blocks instead.

It seems emoji characters are split into multiple tokens, and buffering might be necessary?

Similar Issue

Here's a related issue with a comment from a user who figured out a fix for the C# binding library

ggml-org/llama.cpp#2231 (comment)

Activity

  1. kherud commented on Sep 11, 2023

    @kherud
    kherud
    on Sep 11, 2023
    Owner

    Thanks for the issue, the related link is really helpful! I'll look into it.

  2. kherud
    added a commit that references this issue on Sep 11, 2023

    Add SliceableByteBuffer, fix utf-8 codepoint de-tokenization #3

    55face6

  3. kherud commented on Sep 11, 2023

    @kherud
    kherud
    on Sep 11, 2023
    Owner

    I implemented the fix from the related issue. When using the outputs of LlamaModel.generate() unicode code points should now be buffered until the end of the emoji.

    It's still not perfect, though, I think:

    • Consecutive emojis will all be buffered and then be output at once. I think this requires something more sophisticated than simply comparing the most significant bit with 0x80.
    • Each individual token is still output to be able to get its probability. Only the final token has a String representation, though. Each preceding one has an empty String.

    I'll have a closer look at it when I have more time.

  4. kherud
    added a commit that references this issue on Sep 20, 2023

    Merge branch 'jni' (#3 #4)

    01253b5

  5. kherud commented on Sep 20, 2023

    @kherud
    kherud
    on Sep 20, 2023
    Owner

    Version 2.0 just released and solves the above mentioned issues. Emojis are now output individually.

  6. kherud
    closed this as completedon Sep 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions