Greetings everyone and happy new year!
Data Filtering
- replace zeros with nans (df.replace(0,np.nan,inplace=True)
- remove outliers (spikes) using 3 standard divation filter! (hint , use pandas mask!)
- fill nans with mean per column!
Building the training data
quite simple , some array minpulations to build the data set , change number of samples to add more data points
families = list(geochem_database.family.unique())
num_samples = 10000
r_0=[]
r_1=[]
r_2=[]
r_3=[]
r_4=[]
r_5=[]
r_6=[]
r_7=[]
r_8=[]
r_9=[]
r_10=[]
r_11=[]
r_12=[]
r_13=[]
r_14=[]
r_15=[]
r_16=[]
for f in families:
n_data = geochem_database[geochem_database['family']==f].reset_index(drop=True)
for i in range(len(n_data)):
mean=n_data['mean'][i]
stdev=n_data['stdev'][i]
globals()[f"r_{i}"].append(np.random.normal(mean, stdev, size =num_samples))
r_16.append([f]*num_samples)
train_data = pd.DataFrame()
train_data['19']=np.concatenate(r_0).ravel()
train_data['20']=np.concatenate(r_1).ravel()
train_data['21']=np.concatenate(r_2).ravel()
train_data['22']=np.concatenate(r_3).ravel()
train_data['23']=np.concatenate(r_4).ravel()
train_data['24']=np.concatenate(r_5).ravel()
train_data['25']=np.concatenate(r_6).ravel()
train_data['26']=np.concatenate(r_7).ravel()
train_data['27']=np.concatenate(r_8).ravel()
train_data['28']=np.concatenate(r_9).ravel()
train_data['29']=np.concatenate(r_10).ravel()
train_data['30']=np.concatenate(r_11).ravel()
train_data['31']=np.concatenate(r_12).ravel()
train_data['32']=np.concatenate(r_13).ravel()
train_data['OL']=np.concatenate(r_14).ravel()
train_data['GA']=np.concatenate(r_15).ravel()
train_data['family']=sum(r_16, [])
Data processing
simple, just use min-max scaler!
ML
using
VotingClassifier of LogisticRegression KNeighborsClassifier RidgeClassifier
final touches
to put the cherry on the icing
we filter weird classes out in this case we have two points with a lot of marine neighbors! we locally edit them with iloc and change them to marine!
enjoy!