Skip to content

#1 Updating to latest repo version with LLM Monitoring metrics#1

Merged
juanroesel merged 16 commits into
mainfrom
llm-monitoring
May 9, 2024
Merged

#1 Updating to latest repo version with LLM Monitoring metrics#1
juanroesel merged 16 commits into
mainfrom
llm-monitoring

Conversation

@juanroesel

@juanroesel juanroesel commented May 7, 2024 •
edited
Loading

Copy link
Copy Markdown

Closes ZenHubHQ/devops#2205

It also adds includes the following:

  • A new metric kv_cache_usage_ratio, which measures how much KV cache is being used.
  • Synced commits with the parent repo (not relevant for the PR review).
  • A Llama 3 8B model baked into the image.

New image us.gcr.io/zenhub-ops/llama_cpp_python-llama3_8b_f16:v0.3.1 was successfully deployed into staging.

abetlen and others added 15 commits May 2, 2024 11:32
@abetlen
feat: Add llama-3-vision-alpha chat format
31b1d95
@abetlen
feat: Update llama.cpp
d75dea1
@abetlen
chore: Bump version
2117122
@dthuerck
fix(server): Propagate flash_attn to model load. (abetlen#1424)
2138561
@abetlen
feat(server): Remove temperature bounds checks for server. Closes abe…
0a454be 
…tlen#1384
@abetlen
fix: Use memmove to copy str_value kv_override. Closes abetlen#1417
9f7a855
@abetlen
Merge branch 'main' of github.com:abetlen/llama_cpp_python into main
f9b7221
@jeffrey-fong @abetlen
feat: Implement streaming for Functionary v2 + Bug fixes (abetlen#1419)
1f56c64 
* set up streaming for v2

* assert v2 streaming, fix tool_call vs function_call

* fix streaming with tool_choice/function_call

* make functions return 1 function call only when 'auto'

* fix

---------

Co-authored-by: Andrei <abetlen@gmail.com>
@noamgat @abetlen
fix: detokenization case where first token does not start with a lead…
e0d7674 
…ing space (abetlen#1375)

* Fix tokenization edge case where llama output does not start with a space

See this notebook:
https://colab.research.google.com/drive/1Ooz11nFPk19zyJdMDx42CeesU8aWZMdI#scrollTo=oKpHw5PZ30uC

* Update _internals.py

Fixing to compare to b' ' instead of (str)' '

---------

Co-authored-by: Andrei <abetlen@gmail.com>
@abetlen
feat: Update llama.cpp
3e2597e
@Smartappli
feat(ci): Add docker checks and check deps more frequently (abetlen#1426
3666833 
)

* Update dependabot.yml

Add github-actions update

* Update dependabot.yml

* Update dependabot.yml
@abetlen
feat(server): Add support for setting root_path. Closes abetlen#1420
0318702
Ported over prometheus implementation from previous repo
1972445
Added kn_cache_usage_ratio metric
edd0ec6
@juanroesel
Merge branch 'abetlen:main' into llm-monitoring
e42c1d6
@juanroesel juanroesel requested review from cwarje and m62534 May 7, 2024 02:32
@juanroesel

juanroesel commented May 7, 2024

Copy link
Copy Markdown
Author

NOTE: GH Actions need to be updated in this repo. I will create a ticket for this soon.

@juanroesel juanroesel requested a review from blacklander May 7, 2024 17:17
cwarje
cwarje approved these changes May 7, 2024
View reviewed changes

@cwarje cwarje left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@juanroesel

juanroesel commented May 9, 2024 •
edited
Loading

Copy link
Copy Markdown
Author

@m62534 @cwarje Just FYI, given today's events with Llama3, I built a new image us.gcr.io/zenhub-ops/llama_cpp_python_zh-mistral7b_f16:v0.2.1 containing these code changes plus the Mistral model and redeployed it in staging.

@juanroesel juanroesel merged commit 8cd638c into main May 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@m62534 m62534 m62534 approved these changes

@blacklander blacklander Awaiting requested review from blacklander

+1 more reviewer

@cwarje cwarje cwarje approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@juanroesel @m62534 @cwarje @abetlen @dthuerck @jeffrey-fong @noamgat @Smartappli