jealk/TTC-L2V-supervised-2

TTC-L2V-2 (Danish, Swedish and Norwegian)

Model Description

Supervised model for sentence embeddings.

• Developed by: Jesper Alkestrup, The Tech Collective
• Model type: Embedding model
• Language(s) (NLP): Danish, Swedish and Norwegian
• Finetuned from model : AI-Sweden-Models/Llama-3-8B-instruct
• Finetuning procedure: LLM2Vec

Trained by using the approach outlined in the paper LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders.

MNTP Finetuning on: https://huggingface.co/datasets/jealk/scandi-wiki-combined Initial Supervised SimCSE on:: https://huggingface.co/datasets/jealk/supervised-da Followed by Supervised SimCSE w. in hard-negatives and instructions on: https://huggingface.co/datasets/DDSC/nordic-embedding-training-data

Requires the llm2vec package to encode sentences. Credits to https://huggingface.co/McGill-NLP/LLM2Vec-Meta-Llama-3-8B-Instruct-mntp-supervised for the below instructions:

Installation

pip install llm2vec

Usage

from llm2vec import LLM2Vec

import torch
from transformers import AutoTokenizer, AutoModel, AutoConfig
from peft import PeftModel

# Loading base Llama model, along with custom code that enables bidirectional connections in decoder-only LLMs. MNTP LoRA weights are merged into the base model.
tokenizer = AutoTokenizer.from_pretrained(
    "jealk/llm2vec-scandi-mntp-v2"
)
config = AutoConfig.from_pretrained(
    "jealk/llm2vec-scandi-mntp-v2", trust_remote_code=True
)
model = AutoModel.from_pretrained(
    "jealk/llm2vec-scandi-mntp-v2",
    trust_remote_code=True,
    config=config,
    torch_dtype=torch.bfloat16,
    device_map="cuda" if torch.cuda.is_available() else "cpu",
)
model = PeftModel.from_pretrained(
    model,
    "jealk/llm2vec-scandi-mntp-v2",
)
model = model.merge_and_unload()  # This can take several minutes on cpu

# Loading supervised model. This loads the trained LoRA weights on top of MNTP model. Hence the final weights are -- Base model + MNTP (LoRA) + supervised (LoRA).
model = PeftModel.from_pretrained(
    model, "jealk/TTC-L2V-supervised-2"
)

# Wrapper for encoding and pooling operations
l2v = LLM2Vec(model, tokenizer, pooling_mode="mean", max_length=8124)

# Encoding queries using instructions
instruction = (
    "Givet et spørgsmål, find relevante tekstudsnit, der besvarer det:"
)
queries = [
    [instruction, "Hvordan påvirker søvn vores koncentrationsevnet"],
    [instruction, "Hvad skal man være opmærksom på, når man køber brugt cykel"],
]
q_reps = l2v.encode(queries)

# Encoding documents. Instruction are not required for documents
documents = [
    "Forskning viser, at for lidt søvn kan nedsætte både koncentration og hukommelse. Allerede efter én nat med dårlig søvn kan man opleve problemer med at fokusere og træffe beslutninger. En god nattesøvn er derfor vigtig for både mental og fysisk sundhed.",
    "Når du køber en brugt cykel, bør du tjekke, om stellet har skader eller rust, og om gear og bremser fungerer korrekt. Det er også en god idé at sikre sig, at cyklen ikke er stjålet – det kan du gøre ved at slå stelnummeret op i politiets cykelregister.",
]
d_reps = l2v.encode(documents)

# Compute cosine similarity
q_reps_norm = torch.nn.functional.normalize(q_reps, p=2, dim=1)
d_reps_norm = torch.nn.functional.normalize(d_reps, p=2, dim=1)
cos_sim = torch.mm(q_reps_norm, d_reps_norm.transpose(0, 1))

print(cos_sim)
"""
tensor([[0.7161, 0.4021],
        [0.4525, 0.7370]])
"""

Note: When running the above a warning will appear indicating that lm_head.weight was not used. This layer is only for token prediction and is not required for embedding extraction, so the warning safely be ignored.