--- language: - en license: apache-2.0 tags: - sentence-transformers - cross-encoder - generated_from_trainer - dataset_size:578402 - loss:BinaryCrossEntropyLoss base_model: answerdotai/ModernBERT-base pipeline_tag: text-ranking library_name: sentence-transformers metrics: - map - mrr@10 - ndcg@10 model-index: - name: ModernBERT-base trained on GooAQ results: - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: gooaq dev type: gooaq-dev metrics: - type: map value: 0.7323 name: Map - type: mrr@10 value: 0.7309 name: Mrr@10 - type: ndcg@10 value: 0.7731 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoMSMARCO R100 type: NanoMSMARCO_R100 metrics: - type: map value: 0.4464 name: Map - type: mrr@10 value: 0.4352 name: Mrr@10 - type: ndcg@10 value: 0.525 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNFCorpus R100 type: NanoNFCorpus_R100 metrics: - type: map value: 0.3794 name: Map - type: mrr@10 value: 0.5704 name: Mrr@10 - type: ndcg@10 value: 0.4269 name: Ndcg@10 - task: type: cross-encoder-reranking name: Cross Encoder Reranking dataset: name: NanoNQ R100 type: NanoNQ_R100 metrics: - type: map value: 0.5135 name: Map - type: mrr@10 value: 0.518 name: Mrr@10 - type: ndcg@10 value: 0.5685 name: Ndcg@10 - task: type: cross-encoder-nano-beir name: Cross Encoder Nano BEIR dataset: name: NanoBEIR R100 mean type: NanoBEIR_R100_mean metrics: - type: map value: 0.4464 name: Map - type: mrr@10 value: 0.5079 name: Mrr@10 - type: ndcg@10 value: 0.5068 name: Ndcg@10 --- # ModernBERT-base trained on GooAQ This is a [Cross Encoder](https://www.sbert.net/docs/cross_encoder/usage/usage.html) model finetuned from [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) using the [sentence-transformers](https://www.SBERT.net) library. It computes scores for pairs of texts, which can be used for text reranking and semantic search. ## Model Details ### Model Description - **Model Type:** Cross Encoder - **Base model:** [answerdotai/ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base) - **Maximum Sequence Length:** 8192 tokens - **Number of Output Labels:** 1 label - **Language:** en - **License:** apache-2.0 ### Model Sources - **Documentation:** [Sentence Transformers Documentation](https://sbert.net) - **Documentation:** [Cross Encoder Documentation](https://www.sbert.net/docs/cross_encoder/usage/usage.html) - **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) - **Hugging Face:** [Cross Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=cross-encoder) ## Usage ### Direct Usage (Sentence Transformers) First install the Sentence Transformers library: ```bash pip install -U sentence-transformers ``` Then you can load this model and run inference. ```python from sentence_transformers import CrossEncoder # Download from the 🤗 Hub model = CrossEncoder("tomaarsen/reranker-ModernBERT-base-gooaq-bce-no-pos-weight") # Get scores for pairs of texts pairs = [ ['what is a default final judgment?', 'Default judgment is a binding judgment in favor of either party based on some failure to take action by the other party. Most often, it is a judgment in favor of a plaintiff when the defendant has not responded to a summons or has failed to appear before a court of law. The failure to take action is the default.'], ['what is a default final judgment?', "A default judgment is a judgment issued against a party that doesn't bother to defend itself at all. ... A summary judgment is a judgment issued against a party that doesn't have any evidence to support its claims. Summary judgment means: “You can't prove it; therefore you lose.”"], ['what is a default final judgment?', 'This judgment is seen as being mentioned in Hebrews 9:27, which states that "it is appointed unto men once to die, but after this the judgment".'], ['what is a default final judgment?', "If you don't file an answer or go to court, your landlord can ask the judge to find you in default. Then the judge may let your landlord show there is reason for you to be evicted. If the landlord does that, the judge can enter a default judgment against you."], ['what is a default final judgment?', 'What can High Court Enforcement Officers do to enforce judgment? HCEOs can take control of goods or possessions to the value of the unpaid judgment, and may also attempt to take goods to cover the costs of enforcement, court costs, and interest on the debt.'], ] scores = model.predict(pairs) print(scores.shape) # (5,) # Or rank different texts based on similarity to a single text ranks = model.rank( 'what is a default final judgment?', [ 'Default judgment is a binding judgment in favor of either party based on some failure to take action by the other party. Most often, it is a judgment in favor of a plaintiff when the defendant has not responded to a summons or has failed to appear before a court of law. The failure to take action is the default.', "A default judgment is a judgment issued against a party that doesn't bother to defend itself at all. ... A summary judgment is a judgment issued against a party that doesn't have any evidence to support its claims. Summary judgment means: “You can't prove it; therefore you lose.”", 'This judgment is seen as being mentioned in Hebrews 9:27, which states that "it is appointed unto men once to die, but after this the judgment".', "If you don't file an answer or go to court, your landlord can ask the judge to find you in default. Then the judge may let your landlord show there is reason for you to be evicted. If the landlord does that, the judge can enter a default judgment against you.", 'What can High Court Enforcement Officers do to enforce judgment? HCEOs can take control of goods or possessions to the value of the unpaid judgment, and may also attempt to take goods to cover the costs of enforcement, court costs, and interest on the debt.', ] ) # [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...] ``` ## Evaluation ### Metrics #### Cross Encoder Reranking * Dataset: `gooaq-dev` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": false } ``` | Metric | Value | |:------------|:---------------------| | map | 0.7323 (+0.2012) | | mrr@10 | 0.7309 (+0.2069) | | **ndcg@10** | **0.7731 (+0.1818)** | #### Cross Encoder Reranking * Datasets: `NanoMSMARCO_R100`, `NanoNFCorpus_R100` and `NanoNQ_R100` * Evaluated with [CrossEncoderRerankingEvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderRerankingEvaluator) with these parameters: ```json { "at_k": 10, "always_rerank_positives": true } ``` | Metric | NanoMSMARCO_R100 | NanoNFCorpus_R100 | NanoNQ_R100 | |:------------|:---------------------|:---------------------|:---------------------| | map | 0.4464 (-0.0431) | 0.3794 (+0.1184) | 0.5135 (+0.0939) | | mrr@10 | 0.4352 (-0.0423) | 0.5704 (+0.0706) | 0.5180 (+0.0913) | | **ndcg@10** | **0.5250 (-0.0154)** | **0.4269 (+0.1018)** | **0.5685 (+0.0679)** | #### Cross Encoder Nano BEIR * Dataset: `NanoBEIR_R100_mean` * Evaluated with [CrossEncoderNanoBEIREvaluator](https://sbert.net/docs/package_reference/cross_encoder/evaluation.html#sentence_transformers.cross_encoder.evaluation.CrossEncoderNanoBEIREvaluator) with these parameters: ```json { "dataset_names": [ "msmarco", "nfcorpus", "nq" ], "rerank_k": 100, "at_k": 10, "always_rerank_positives": true } ``` | Metric | Value | |:------------|:---------------------| | map | 0.4464 (+0.0564) | | mrr@10 | 0.5079 (+0.0399) | | **ndcg@10** | **0.5068 (+0.0514)** | ## Training Details ### Training Dataset #### Unnamed Dataset * Size: 578,402 training samples * Columns: question, answer, and label * Approximate statistics based on the first 1000 samples: | | question | answer | label | |:--------|:-----------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------|:------------------------------------------------| | type | string | string | int | | details | | | | * Samples: | question | answer | label | |:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:---------------| | what is a default final judgment? | Default judgment is a binding judgment in favor of either party based on some failure to take action by the other party. Most often, it is a judgment in favor of a plaintiff when the defendant has not responded to a summons or has failed to appear before a court of law. The failure to take action is the default. | 1 | | what is a default final judgment? | A default judgment is a judgment issued against a party that doesn't bother to defend itself at all. ... A summary judgment is a judgment issued against a party that doesn't have any evidence to support its claims. Summary judgment means: “You can't prove it; therefore you lose.” | 0 | | what is a default final judgment? | This judgment is seen as being mentioned in Hebrews 9:27, which states that "it is appointed unto men once to die, but after this the judgment". | 0 | * Loss: [BinaryCrossEntropyLoss](https://sbert.net/docs/package_reference/cross_encoder/losses.html#binarycrossentropyloss) with these parameters: ```json { "activation_fn": "torch.nn.modules.linear.Identity", "pos_weight": null } ``` ### Training Hyperparameters #### Non-Default Hyperparameters - `eval_strategy`: steps - `per_device_train_batch_size`: 64 - `per_device_eval_batch_size`: 64 - `learning_rate`: 2e-05 - `num_train_epochs`: 1 - `warmup_ratio`: 0.1 - `seed`: 12 - `bf16`: True - `dataloader_num_workers`: 4 - `load_best_model_at_end`: True #### All Hyperparameters
Click to expand - `overwrite_output_dir`: False - `do_predict`: False - `eval_strategy`: steps - `prediction_loss_only`: True - `per_device_train_batch_size`: 64 - `per_device_eval_batch_size`: 64 - `per_gpu_train_batch_size`: None - `per_gpu_eval_batch_size`: None - `gradient_accumulation_steps`: 1 - `eval_accumulation_steps`: None - `torch_empty_cache_steps`: None - `learning_rate`: 2e-05 - `weight_decay`: 0.0 - `adam_beta1`: 0.9 - `adam_beta2`: 0.999 - `adam_epsilon`: 1e-08 - `max_grad_norm`: 1.0 - `num_train_epochs`: 1 - `max_steps`: -1 - `lr_scheduler_type`: linear - `lr_scheduler_kwargs`: {} - `warmup_ratio`: 0.1 - `warmup_steps`: 0 - `log_level`: passive - `log_level_replica`: warning - `log_on_each_node`: True - `logging_nan_inf_filter`: True - `save_safetensors`: True - `save_on_each_node`: False - `save_only_model`: False - `restore_callback_states_from_checkpoint`: False - `no_cuda`: False - `use_cpu`: False - `use_mps_device`: False - `seed`: 12 - `data_seed`: None - `jit_mode_eval`: False - `use_ipex`: False - `bf16`: True - `fp16`: False - `fp16_opt_level`: O1 - `half_precision_backend`: auto - `bf16_full_eval`: False - `fp16_full_eval`: False - `tf32`: None - `local_rank`: 0 - `ddp_backend`: None - `tpu_num_cores`: None - `tpu_metrics_debug`: False - `debug`: [] - `dataloader_drop_last`: False - `dataloader_num_workers`: 4 - `dataloader_prefetch_factor`: None - `past_index`: -1 - `disable_tqdm`: False - `remove_unused_columns`: True - `label_names`: None - `load_best_model_at_end`: True - `ignore_data_skip`: False - `fsdp`: [] - `fsdp_min_num_params`: 0 - `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} - `fsdp_transformer_layer_cls_to_wrap`: None - `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} - `deepspeed`: None - `label_smoothing_factor`: 0.0 - `optim`: adamw_torch - `optim_args`: None - `adafactor`: False - `group_by_length`: False - `length_column_name`: length - `ddp_find_unused_parameters`: None - `ddp_bucket_cap_mb`: None - `ddp_broadcast_buffers`: False - `dataloader_pin_memory`: True - `dataloader_persistent_workers`: False - `skip_memory_metrics`: True - `use_legacy_prediction_loop`: False - `push_to_hub`: False - `resume_from_checkpoint`: None - `hub_model_id`: None - `hub_strategy`: every_save - `hub_private_repo`: None - `hub_always_push`: False - `gradient_checkpointing`: False - `gradient_checkpointing_kwargs`: None - `include_inputs_for_metrics`: False - `include_for_metrics`: [] - `eval_do_concat_batches`: True - `fp16_backend`: auto - `push_to_hub_model_id`: None - `push_to_hub_organization`: None - `mp_parameters`: - `auto_find_batch_size`: False - `full_determinism`: False - `torchdynamo`: None - `ray_scope`: last - `ddp_timeout`: 1800 - `torch_compile`: False - `torch_compile_backend`: None - `torch_compile_mode`: None - `dispatch_batches`: None - `split_batches`: None - `include_tokens_per_second`: False - `include_num_input_tokens_seen`: False - `neftune_noise_alpha`: None - `optim_target_modules`: None - `batch_eval_metrics`: False - `eval_on_start`: False - `use_liger_kernel`: False - `eval_use_gather_object`: False - `average_tokens_across_devices`: False - `prompts`: None - `batch_sampler`: batch_sampler - `multi_dataset_batch_sampler`: proportional
### Training Logs | Epoch | Step | Training Loss | gooaq-dev_ndcg@10 | NanoMSMARCO_R100_ndcg@10 | NanoNFCorpus_R100_ndcg@10 | NanoNQ_R100_ndcg@10 | NanoBEIR_R100_mean_ndcg@10 | |:----------:|:--------:|:-------------:|:--------------------:|:------------------------:|:-------------------------:|:--------------------:|:--------------------------:| | -1 | -1 | - | 0.1386 (-0.4527) | 0.0206 (-0.5198) | 0.2387 (-0.0863) | 0.0515 (-0.4491) | 0.1036 (-0.3517) | | 0.0001 | 1 | 1.0425 | - | - | - | - | - | | 0.0221 | 200 | 0.5627 | - | - | - | - | - | | 0.0443 | 400 | 0.4593 | - | - | - | - | - | | 0.0664 | 600 | 0.3714 | - | - | - | - | - | | 0.0885 | 800 | 0.2955 | - | - | - | - | - | | 0.1106 | 1000 | 0.2829 | 0.7083 (+0.1171) | 0.4992 (-0.0412) | 0.3110 (-0.0141) | 0.4795 (-0.0211) | 0.4299 (-0.0255) | | 0.1328 | 1200 | 0.2696 | - | - | - | - | - | | 0.1549 | 1400 | 0.2548 | - | - | - | - | - | | 0.1770 | 1600 | 0.2485 | - | - | - | - | - | | 0.1992 | 1800 | 0.2326 | - | - | - | - | - | | 0.2213 | 2000 | 0.241 | 0.7461 (+0.1549) | 0.5350 (-0.0054) | 0.3701 (+0.0451) | 0.5339 (+0.0332) | 0.4797 (+0.0243) | | 0.2434 | 2200 | 0.2373 | - | - | - | - | - | | 0.2655 | 2400 | 0.2356 | - | - | - | - | - | | 0.2877 | 2600 | 0.2207 | - | - | - | - | - | | 0.3098 | 2800 | 0.222 | - | - | - | - | - | | 0.3319 | 3000 | 0.2258 | 0.7443 (+0.1531) | 0.5554 (+0.0150) | 0.3921 (+0.0671) | 0.5368 (+0.0361) | 0.4948 (+0.0394) | | 0.3541 | 3200 | 0.2182 | - | - | - | - | - | | 0.3762 | 3400 | 0.215 | - | - | - | - | - | | 0.3983 | 3600 | 0.2161 | - | - | - | - | - | | 0.4204 | 3800 | 0.2202 | - | - | - | - | - | | 0.4426 | 4000 | 0.2147 | 0.7542 (+0.1629) | 0.5465 (+0.0061) | 0.4047 (+0.0797) | 0.5199 (+0.0193) | 0.4904 (+0.0350) | | 0.4647 | 4200 | 0.2177 | - | - | - | - | - | | 0.4868 | 4400 | 0.2129 | - | - | - | - | - | | 0.5090 | 4600 | 0.2099 | - | - | - | - | - | | 0.5311 | 4800 | 0.2105 | - | - | - | - | - | | 0.5532 | 5000 | 0.2101 | 0.7644 (+0.1731) | 0.5448 (+0.0044) | 0.4157 (+0.0907) | 0.5746 (+0.0739) | 0.5117 (+0.0563) | | 0.5753 | 5200 | 0.2034 | - | - | - | - | - | | 0.5975 | 5400 | 0.2047 | - | - | - | - | - | | 0.6196 | 5600 | 0.2043 | - | - | - | - | - | | 0.6417 | 5800 | 0.2029 | - | - | - | - | - | | 0.6639 | 6000 | 0.2021 | 0.7699 (+0.1786) | 0.5250 (-0.0154) | 0.4264 (+0.1013) | 0.5491 (+0.0484) | 0.5002 (+0.0448) | | 0.6860 | 6200 | 0.2048 | - | - | - | - | - | | 0.7081 | 6400 | 0.2033 | - | - | - | - | - | | 0.7303 | 6600 | 0.2017 | - | - | - | - | - | | 0.7524 | 6800 | 0.1976 | - | - | - | - | - | | 0.7745 | 7000 | 0.1989 | 0.7722 (+0.1810) | 0.5732 (+0.0328) | 0.4206 (+0.0956) | 0.6013 (+0.1007) | 0.5317 (+0.0763) | | 0.7966 | 7200 | 0.1925 | - | - | - | - | - | | 0.8188 | 7400 | 0.1917 | - | - | - | - | - | | 0.8409 | 7600 | 0.2002 | - | - | - | - | - | | 0.8630 | 7800 | 0.1913 | - | - | - | - | - | | 0.8852 | 8000 | 0.191 | 0.7707 (+0.1794) | 0.5412 (+0.0007) | 0.4332 (+0.1082) | 0.5508 (+0.0502) | 0.5084 (+0.0530) | | 0.9073 | 8200 | 0.1929 | - | - | - | - | - | | 0.9294 | 8400 | 0.1989 | - | - | - | - | - | | 0.9515 | 8600 | 0.1889 | - | - | - | - | - | | 0.9737 | 8800 | 0.1874 | - | - | - | - | - | | **0.9958** | **9000** | **0.1863** | **0.7731 (+0.1818)** | **0.5250 (-0.0154)** | **0.4269 (+0.1018)** | **0.5685 (+0.0679)** | **0.5068 (+0.0514)** | | -1 | -1 | - | 0.7731 (+0.1818) | 0.5250 (-0.0154) | 0.4269 (+0.1018) | 0.5685 (+0.0679) | 0.5068 (+0.0514) | * The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.11.10 - Sentence Transformers: 3.5.0.dev0 - Transformers: 4.49.0 - PyTorch: 2.5.1+cu124 - Accelerate: 1.5.2 - Datasets: 2.21.0 - Tokenizers: 0.21.0 ## Citation ### BibTeX #### Sentence Transformers ```bibtex @inproceedings{reimers-2019-sentence-bert, title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", author = "Reimers, Nils and Gurevych, Iryna", booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", month = "11", year = "2019", publisher = "Association for Computational Linguistics", url = "https://arxiv.org/abs/1908.10084", } ```