This kaggle is：
https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/overview
The Goal is：
to classify toxic comments
especially to recognize unintended bias towards identities
toxic comment is a comment that is rude, disrespectful or otherwise likely to make someone leave a discussion
challenge is：
some neutral comments regarding some identity like "gay" would be classified as toxic，eg："I am a gay woman" .
reason is：
identities associated with toxicity outnumbered neutral comments regarding the same identity
Dataset
dataset labeled with the associated identity
https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/data
These are subtypes of toxicity, do not need to predict:
severe_toxicity
obscene
threat
insult
identity_attack
sexual_explicit
These columns are corresponding to identity attributes:
representing the identities that are mentioned in the comment
male
female
transgender
other_gender
heterosexual
homosexual_gay_or_lesbian
bisexual
other_sexual_orientation
christian
jewish
muslim
hindu
buddhist
atheist
other_religion
black
white
asian
latino
other_race_or_ethnicity
physical_disability
intellectual_or_learning_disability
psychiatric_or_mental_illness
other_disability
Additional columns:
toxicity_annotator_count and identity_annotator_count, and metadata from Civil Comments: created_date, publication_id, parent_id, article_id, rating, funny, wow, sad, likes, disagree. Civil Comments' label rating is the civility rating Civil Comments users gave the comment.
Example:
Comment: Continue to stand strong LGBT community. Yes, indeed, you'll overcome and you have.
Toxicity Labels: All 0.0
Identity Mention Labels: homosexual_gay_or_lesbian: 0.8, bisexual: 0.6, transgender: 0.3 (all others 0.0)
1. Libs and Data:
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import os
print(os.listdir("../input"))
train_df = pd.read_csv('../input/train.csv')
test_df = pd.read_csv('../input/test.csv')
2. Shape of data
train_len, test_len = len(train_df.index), len(test_df.index)
print(f'train size: {train_len}, test size: {test_len}')
train size: 1804874, test size: 97320
train_df.head()
3. Count the amount of missing values
miss_val_train_df = train_df.isnull().sum(axis=0) / train_len
miss_val_train_df = miss_val_train_df[miss_val_train_df > 0] * 100
miss_val_train_df
- a large portion of the data doesn't have the identity tag
- but the numbers are same
4. Visualization
Q1: which identity appears the most in the dataset?
According to the data details, just care about the identities tagged in this dataset, and make a list of them:
identities = ['male','female','transgender','other_gender','heterosexual','homosexual_gay_or_lesbian',
'bisexual','other_sexual_orientation','christian','jewish','muslim','hindu','buddhist',
'atheist','other_religion','black','white','asian','latino','other_race_or_ethnicity',
'physical_disability','intellectual_or_learning_disability','psychiatric_or_mental_illness',
'other_disability']
From below diagram we can also see distributions of toxic and non-toxic in each identity:
# getting the dataframe with identities tagged
train_labeled_df = train_df.loc[:, ['target'] + identities ].dropna()
# lets define toxicity as a comment with a score being equal or .5
# in that case we divide it into two dataframe so we can count toxic vs non toxic comment per identity
toxic_df = train_labeled_df[train_labeled_df['target'] >= .5][identities]
non_toxic_df = train_labeled_df[train_labeled_df['target'] < .5][identities]
# at first, we just want to consider the identity tags in binary format. So if the tag is any value other than 0 we consider it as 1.
toxic_count = toxic_df.where(train_labeled_df == 0, other = 1).sum()
non_toxic_count = non_toxic_df.where(train_labeled_df == 0, other = 1).sum()
# now we can concat the two series together to get a toxic count vs non toxic count for each identity
toxic_vs_non_toxic = pd.concat([toxic_count, non_toxic_count], axis=1)
toxic_vs_non_toxic = toxic_vs_non_toxic.rename(index=str, columns={1: "non-toxic", 0: "toxic"})
# here we plot the stacked graph but we sort it by toxic comments to (perhaps) see something interesting
toxic_vs_non_toxic.sort_values(by='toxic').plot(kind='bar', stacked=True, figsize=(30,10), fontsize=20).legend(prop={'size': 20})
Q2: which identities are more frequently related to toxic comments?
- consider the score (target) of how toxic the comment is
- also count in the value of how each identity been targeted
# First we multiply each identity with the target
weighted_toxic = train_labeled_df.iloc[:, 1:].multiply(train_labeled_df.iloc[:, 0], axis="index").sum()
# changing the value of identity to 1 or 0 only and get comment count per identity group
identity_label_count = train_labeled_df[identities].where(train_labeled_df == 0, other = 1).sum()
# then we divide the target weighted value by the number of time each identity appears
weighted_toxic = weighted_toxic / identity_label_count
weighted_toxic = weighted_toxic.sort_values(ascending=False)
# plot the data using seaborn like before
plt.figure(figsize=(30,20))
sns.set(font_scale=3)
ax = sns.barplot(x = weighted_toxic.values , y = weighted_toxic.index, alpha=0.8)
plt.ylabel('Demographics')
plt.xlabel('Weighted Toxicity')
plt.title('Weighted Analysis of Most Frequent Identities')
plt.show()
Conclusion: race based identities (White and Black) and religion based identities (Muslim and Jews) are heavily associated with toxic comments.