Image-to-Text
Transformers
Safetensors
Japanese
llava-jp
text-generation
vision
image-captioning
VQA
toshi456 commited on
Commit
996c477
·
verified ·
1 Parent(s): 5d355b8

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +146 -3
README.md CHANGED
@@ -1,3 +1,146 @@
1
- ---
2
- license: cc-by-nc-4.0
3
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: cc-by-nc-4.0
3
+ datasets:
4
+ - turing-motors/LLaVA-Pretrain-JA
5
+ - turing-motors/LLaVA-v1.5-Instruct-620K-JA
6
+ language:
7
+ - ja
8
+ pipeline_tag: image-to-text
9
+ tags:
10
+ - vision
11
+ - image-captioning
12
+ - VQA
13
+ ---
14
+
15
+ # ConvLLaVA-JP Model Card
16
+
17
+ ## Model detail
18
+
19
+ **Model type:**
20
+
21
+ ConvLLaVA-JP is a vision-language model that can converse about input images.<br>
22
+ This model is an LVLM model trained using [laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft](https://huggingface.co/laion/CLIP-convnext_large_d_320.laion2B-s29B-b131K-ft) as the image encoder and [llm-jp/llm-jp-1.3b-v1.0](https://huggingface.co/llm-jp/llm-jp-1.3b-v1.0) as the text decoder. Input of 768 x 768 high resolution.
23
+
24
+ **Training:**
25
+
26
+ This model was initially trained with Vision Projector and Stage 5 using LLaVA-Pretrain-JA.<br>
27
+ In the second phase, it was trained Image Encoder, Vision Projector, Stage 5 and LLM using LLaVA-Pretrain-JA.<br>
28
+ In the third phase, it was fine-tuned with Vision Projector and LLM using LLaVA-v1.5-Instruct-620K-JA.
29
+
30
+ resources for more information: https://github.com/tosiyuki/LLaVA-JP/tree/main
31
+
32
+ **Comparing VLMs**
33
+ |Model|JA-VG-VQA-500<br>(ROUGE-L)|JA-VLM-Bench-In-the-Wild<br>(ROUGE-L)|Heron-Bench(Detail)|Heron-Bench(Conv)|Heron-Bench(Complex)|Heron-Bench(Average)
34
+ |-|-|-|-|-|-|-|
35
+ |[Japanese Stable VLM](https://huggingface.co/stabilityai/japanese-stable-vlm)|-|40.50|25.15|51.23|37.84|38.07|
36
+ |[EvoVLM-JP-v1-7B](https://huggingface.co/SakanaAI/EvoVLM-JP-v1-7B)|**19.70**|**51.25**|50.31|44.42|40.47|45.07|
37
+ |[Heron BLIP Japanese StableLM Base 7B llava-620k](https://huggingface.co/turing-motors/heron-chat-blip-ja-stablelm-base-7b-v1-llava-620k)|14.51|33.26|49.09|41.51|45.72|45.44|
38
+ |[Heron GIT Japanese StableLM Base 7B](https://huggingface.co/turing-motors/heron-chat-git-ja-stablelm-base-7b-v1)|15.18|37.82|42.77|**54.20**|43.53|46.83|
39
+ |[llava-jp-1.3b-v1.0-620k](https://huggingface.co/toshi456/llava-jp-1.3b-v1.0-620k)|12.69|44.58|**51.21**|41.05|45.95|44.84|
40
+ |[llava-jp-1.3b-v1.1](https://huggingface.co/toshi456/llava-jp-1.3b-v1.1)|13.33|44.40|50.00|51.83|**48.98**|**50.39**|
41
+ |[ConvLLaVA-JP-1.3b-768](https://huggingface.co/toshi456/ConvLLaVA-JP-1.3b-768)|12.05|42.80|44.24|40.00|48.16|44.96|
42
+
43
+ ## How to use the model
44
+ **1. Download dependencies**
45
+ ```
46
+ git clone https://github.com/tosiyuki/LLaVA-JP.git
47
+ ```
48
+
49
+ **2. Inference**
50
+ ```python
51
+ import requests
52
+ import torch
53
+ import transformers
54
+ from PIL import Image
55
+
56
+ from transformers.generation.streamers import TextStreamer
57
+ from llava.constants import DEFAULT_IMAGE_TOKEN, IMAGE_TOKEN_INDEX
58
+ from llava.conversation import conv_templates, SeparatorStyle
59
+ from llava.model.llava_gpt2 import LlavaGpt2ForCausalLM
60
+ from llava.train.dataset import tokenizer_image_token
61
+
62
+
63
+ if __name__ == "__main__":
64
+ model_path = 'toshi456/ConvLLaVA-JP-1.3b-768'
65
+ device = "cuda" if torch.cuda.is_available() else "cpu"
66
+ torch_dtype = torch.bfloat16 if device=="cuda" else torch.float32
67
+
68
+ model = LlavaGpt2ForCausalLM.from_pretrained(
69
+ model_path,
70
+ low_cpu_mem_usage=True,
71
+ use_safetensors=True,
72
+ torch_dtype=torch_dtype,
73
+ device_map=device,
74
+ )
75
+ tokenizer = transformers.AutoTokenizer.from_pretrained(
76
+ model_path,
77
+ model_max_length=1532,
78
+ padding_side="right",
79
+ use_fast=False,
80
+ )
81
+ model.eval()
82
+
83
+ conv_mode = "v1"
84
+ conv = conv_templates[conv_mode].copy()
85
+
86
+ # image pre-process
87
+ image_url = "https://huggingface.co/rinna/bilingual-gpt-neox-4b-minigpt4/resolve/main/sample.jpg"
88
+ image = Image.open(requests.get(image_url, stream=True).raw).convert('RGB')
89
+
90
+ if device == "cuda":
91
+ image_tensor = model.get_model().vision_tower.image_processor(image).unsqueeze(0).half().cuda().to(torch_dtype)
92
+ else:
93
+ image_tensor = model.get_model().vision_tower.image_processor(image).unsqueeze(0).to(torch_dtype)
94
+
95
+ # create prompt
96
+ # ユーザー: <image>\n{prompt}
97
+ prompt = "猫の隣には何がありますか?"
98
+ inp = DEFAULT_IMAGE_TOKEN + '\n' + prompt
99
+ conv.append_message(conv.roles[0], inp)
100
+ conv.append_message(conv.roles[1], None)
101
+ prompt = conv.get_prompt()
102
+
103
+ input_ids = tokenizer_image_token(
104
+ prompt,
105
+ tokenizer,
106
+ IMAGE_TOKEN_INDEX,
107
+ return_tensors='pt'
108
+ ).unsqueeze(0)
109
+ if device == "cuda":
110
+ input_ids = input_ids.to(device)
111
+
112
+ input_ids = input_ids[:, :-1] # </sep>がinputの最後に入るので削除する
113
+ stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
114
+ keywords = [stop_str]
115
+ streamer = TextStreamer(tokenizer, skip_prompt=True, timeout=20.0)
116
+
117
+ # predict
118
+ with torch.inference_mode():
119
+ output_id = model.generate(
120
+ inputs=input_ids,
121
+ images=image_tensor,
122
+ do_sample=False,
123
+ temperature=1.0,
124
+ top_p=1.0,
125
+ max_new_tokens=256,
126
+ streamer=streamer,
127
+ use_cache=True,
128
+ )
129
+ """猫の隣にはノートパソコンがあります。"""
130
+
131
+ ```
132
+
133
+ ## Training dataset
134
+ **Stage1 and Stage2 Pretrain**
135
+ - [LLaVA-Pretrain-JA](https://huggingface.co/datasets/turing-motors/LLaVA-Pretrain-JA)
136
+
137
+ **Stage3 Fine-tuning**
138
+ - [LLaVA-v1.5-Instruct-620K-JA](https://huggingface.co/datasets/turing-motors/LLaVA-v1.5-Instruct-620K-JA)
139
+
140
+ ## Acknowledgement
141
+ - [ConvLLaVA](https://arxiv.org/abs/2405.15738)
142
+ - [LLM-jp](https://llm-jp.nii.ac.jp/)
143
+ - [Open CLIP](https://github.com/mlfoundations/open_clip)
144
+
145
+ ## License
146
+ cc-by-nc-4.0