2nd model rf

.gitignore

@@ -6,7 +6,9 @@ __pycache__/
 .env
 .ipynb_checkpoints
 .vscode/
-
+notebooks
+Pipfile
+Pipfile.lock
 eval-queue/
 eval-results/
 eval-queue-bk/
@@ -14,4 +16,3 @@ eval-results-bk/
 logs/
 
 emissions.csv
-notebooks/test.ipynb

README.md

@@ -7,65 +7,40 @@ sdk: docker
 pinned: false
 ---
 
+# Classification Model for Climate Disinformation Classification
+
+## Global Informations
+
+The aim of this model is to detect illegal deforestation thanks to audio clips. Our objective is to make this AI system the most frugal possible.
+When you're new to AI for audio processing and you're looking for information on the Internet, the following methodology is often described: transform the audio signal into a spectrogram (2D image) and then have it analyzed by a CNN. It can be necessary when you're working on very precise task such as transcription, but it's too heavy for our task, which is simply to detect chainsaw noises. So, for our baseline, we used an mfcc transform to preprocess the data with, then we transformed the 2D output un a 1D by taking the mean for each feature and finally we applied a basic ml classification algorithm like Random Forest. 
+Then, we tried to optimize the 2 different stages: the data preprocessing (to make it simple, quick and to create the smallest possible preprocessed dataset) and the ml model itself. At this point, we noticed than the preprocessing of our data (with the mfcc transformation) was consuming 12 times more than our model training and inference so we mostly worked on optimizing the data preprocessing part. Here are our main ideas :
+1. The data preprocessing:
+   * Compare different methods of audio feature extraction 
+   * Decrease the size of the data to analyse (i.e. by decreasing the sampling rate, take only a small sample of the initial audio of 3s, since we don’t really need several seconds to identify a sound, remove unnecessary data (as with plots we can see that the characteristic sounds of chainsraw are observed between 200Hz & 1000Hz))
+   * Decrease the number of features extracted 
+ 
+2. The ml model: 
+   * Use the most lightweighted model : Avoid neural networks and compare basic ml classification algorithms for example knn is 3 times less energy consuming than the Random Forest with a non-significant loss of accuracy
+
+## Submitted models
+Model 1:
+* Data preprocessing: Resampling to 6000Hz and using librosa mfcc method to calculate 7 MFCC 
+  * Energy consumption for the processing of the training dataset : 0.005349 kWh
+* Model: KNN 
+  * Energy consumption for the training of the model : 0.000002 kWh
+
+Model 2:
+* Data preprocessing: Resampling to 6000Hz and using librosa mfcc method to calculate 10 MFCC  
+    * Energy consumption for the processing of the training dataset : 0.005648 kWh
+* Model: Random Forest
+    * Energy consumption for the training of the model : 0.000326 kWh
+
+The 2nd model has better accuracy but is less energy efficient.
+
+## Other avenues for optimization 
+
+We learned a lot about all the possible optimizations but unfortunately, we didn't keep all of them for the final submission, as they  led to an excessive loss of precision (~85% accuracy), they are described below :
+* Use a less complex method of audio feature extraction (example: spectral centroid is 5 times faster than mfcc)
+* Focus our analysis on the frequencies that really matter for chainsaws (from 0 to 20,000Hz)
+* Use less data (shortest audio), i.e. keep randomly only 0.2s of audio for each audio of the dataset before extracting features with mfcc and then also take only a small sample of each audio of the test dataset to predict the label
 
-# Random Baseline Model for Climate Disinformation Classification
-
-## Model Description
-
-This is a random baseline model for the Frugal AI Challenge 2024, specifically for the text classification task of identifying climate disinformation. The model serves as a performance floor, randomly assigning labels to text inputs without any learning.
-
-### Intended Use
-
-- **Primary intended uses**: Baseline comparison for climate disinformation classification models
-- **Primary intended users**: Researchers and developers participating in the Frugal AI Challenge
-- **Out-of-scope use cases**: Not intended for production use or real-world classification tasks
-
-## Training Data
-
-The model uses the QuotaClimat/frugalaichallenge-text-train dataset:
-- Size: ~6000 examples
-- Split: 80% train, 20% test
-- 8 categories of climate disinformation claims
-
-### Labels
-0. No relevant claim detected
-1. Global warming is not happening
-2. Not caused by humans
-3. Not bad or beneficial
-4. Solutions harmful/unnecessary
-5. Science is unreliable
-6. Proponents are biased
-7. Fossil fuels are needed
-
-## Performance
-
-### Metrics
-- **Accuracy**: ~12.5% (random chance with 8 classes)
-- **Environmental Impact**:
-  - Emissions tracked in gCO2eq
-  - Energy consumption tracked in Wh
-
-### Model Architecture
-The model implements a random choice between the 8 possible labels, serving as the simplest possible baseline.
-
-## Environmental Impact
-
-Environmental impact is tracked using CodeCarbon, measuring:
-- Carbon emissions during inference
-- Energy consumption during inference
-
-This tracking helps establish a baseline for the environmental impact of model deployment and inference.
-
-## Limitations
-- Makes completely random predictions
-- No learning or pattern recognition
-- No consideration of input text
-- Serves only as a baseline reference
-- Not suitable for any real-world applications
-
-## Ethical Considerations
-
-- Dataset contains sensitive topics related to climate disinformation
-- Model makes random predictions and should not be used for actual classification
-- Environmental impact is tracked to promote awareness of AI's carbon footprint
-```

models/audio_classification_rf.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d180471cec554f4b107a578e7badc3972e70a5702efb6258a0f4ba6172808a9b
+size 6603238

tasks/audio.py

@@ -2,8 +2,10 @@ from fastapi import APIRouter
 from datetime import datetime
 from datasets import load_dataset
 from sklearn.metrics import accuracy_score
-import random
 import os
+import joblib
+import librosa
+import numpy as np
 
 from .utils.evaluation import AudioEvaluationRequest
 from .utils.emissions import tracker, clean_emissions_data, get_space_info
@@ -13,17 +15,16 @@ load_dotenv()
 
 router = APIRouter()
 
-DESCRIPTION = "Random Baseline"
+DESCRIPTION = "Model 2 : Random Forest audio classification"
 ROUTE = "/audio"
 
 
-
 @router.post(ROUTE, tags=["Audio Task"],
              description=DESCRIPTION)
 async def evaluate_audio(request: AudioEvaluationRequest):
     """
     Evaluate audio classification for rainforest sound detection.
-    
+
     Current Model: Random Baseline
     - Makes random predictions from the label space (0-1)
     - Used as a baseline for comparison
@@ -38,35 +39,56 @@ async def evaluate_audio(request: AudioEvaluationRequest):
     }
     # Load and prepare the dataset
     # Because the dataset is gated, we need to use the HF_TOKEN environment variable to authenticate
-    dataset = load_dataset(request.dataset_name,token=os.getenv("HF_TOKEN"))
-    
+    dataset = load_dataset(request.dataset_name, token=os.getenv("HF_TOKEN"))
+
     # Split dataset
     train_test = dataset["train"]
     test_dataset = dataset["test"]
-    
+
     # Start tracking emissions
     tracker.start()
     tracker.start_task("inference")
-    
-    #--------------------------------------------------------------------------------------------
+
+    # --------------------------------------------------------------------------------------------
     # YOUR MODEL INFERENCE CODE HERE
     # Update the code below to replace the random baseline by your model inference within the inference pass where the energy consumption and emissions are tracked.
-    #--------------------------------------------------------------------------------------------   
-    
-    # Make random predictions (placeholder for actual model inference)
+    # --------------------------------------------------------------------------------------------
+    # data formatting
+
+    def preprocess(dataset):
+        features = []
+        for row in dataset:
+            # Load the audio file and resample it
+            target_sr = 6000
+            audio = row['audio']['array']
+            audio = librosa.resample(audio, orig_sr=12000, target_sr=target_sr)
+
+            # Extract MFCC features
+            mfccs = librosa.feature.mfcc(y=audio, sr=target_sr, n_mfcc=10)
+            mfccs_scaled = np.mean(mfccs.T, axis=0)
+
+            # Append features and labels
+            features.append(mfccs_scaled)
+
+        return np.array(features)
+
+    X_test = preprocess(test_dataset)
+
+    classification_model = joblib.load("./models/audio_classification_rf.pkl")
+
+    predictions = classification_model.predict(X_test)
     true_labels = test_dataset["label"]
-    predictions = [random.randint(0, 1) for _ in range(len(true_labels))]
-    
-    #--------------------------------------------------------------------------------------------
+
+    # --------------------------------------------------------------------------------------------
     # YOUR MODEL INFERENCE STOPS HERE
-    #--------------------------------------------------------------------------------------------   
-    
+    # --------------------------------------------------------------------------------------------
+
     # Stop tracking emissions
     emissions_data = tracker.stop_task()
-    
+
     # Calculate accuracy
     accuracy = accuracy_score(true_labels, predictions)
-    
+
     # Prepare results dictionary
     results = {
         "username": username,
@@ -84,5 +106,4 @@ async def evaluate_audio(request: AudioEvaluationRequest):
             "test_seed": request.test_seed
         }
     }
-    
-    return results 
+    return results