# Some Graphical Comparison and Results (with fitted polynomials) of Naive Bayes, Logistic Regression and Linear Regression

This article provides results, graphical plots and analysis of Linear, Logistic Regression, Naive Bayes for three kinds of datasets, namely, (i) Linearly Seperable, (ii) Non-Linearly Seperable and (iii) Banana Data. This article plays a good role for new mentors, academicians and students to understand the process of experimentations and analysis, result presentation in a Machine Learning tasks. It also plots the classifier hyperplane along with the distinct class in this 2-class problem. The results are not benchmarks, they are elaborated for experimental setups in labs, and to be followed by mentors teaching Machine Learning Lab Work. For benchmark results look into peer-reviewed research papers.

# RESULTS & PLOTS

Here some results obtained after the dividing the data into 30-70 ratio of testing and training respectively. This was done for 5 folds. Finally, average was taken. This has been done for all thre classifiers who’s results are discussed below.

The following table shows results for linearly seperable data for the three classifiers (i) Linear Regression (ii) Logistic Regression and (iii) Naïve Bayes.  The results are averages over 5 folds and are comparable.

## Plots of original Data

Following are the plots of given data.

### Linearly Separable Data

Non Linearly Seperable Data plot

Banana Data

LS Data Results  (Note the data are averages of 5 fold data)

The following table shows results for non  linearly seperable data for the three classifiers (i) Linear Regression (ii) Logistic Regression and (iii) Naïve Bayes.  The best results are with Linear and logistic regression.

NLS Data Results (Note the data are averages of 5 fold data)

The following table shows results for banana data for the three classifiers (i) Linear Regression (ii) Logistic Regression and (iii) Naïve Bayes.  The results are averages over 5 folds and Naive Bayes gives the best results.

## The Surface and figures

The surfaces and figures of the discriminant are given below.

### The surface for Linearly Seperable Data using linear regression

Below are the two plots of linearly separable data surface generated by linear regression. The two plots have different axis coordinates.

### The surface for  Non Linearly Seperable data using linear regression

Below are the two plots of non linearly separable data surface generated by linear regression. The two plots have different axis coordinates.

### The surface for Banana Data using linear regression

Below are the two plots of banana  data surface generated by linear regression. The two plots have different axis coordinates.

### The Surface of LS Data uisng Logistic regression

Below are the two plots of linearly separable data surface generated by logistic regression. The two plots with different runs and different training and testing data.

The following are surface generated over two runs.

### The Surface of NLS Data with Logistic Regression

Below are the two plots of non linearly separable data surface generated by logistic regression.

### The Surface of BananaData with Logistic Regression

Below are the two plots of banana data surface generated by logistic regression. The two plots with different runs and different training and testing data.

(The outputs are over two runs)

## More Results: On Segmented Data

Now the samples of training sizes (500, 1000, 1500,….,4500) and {{5000}-{training-size}} as the testing size is being analyzed here. I iterate over with incrementing each time the partition(train-test ratio) percentage by  0.1% of 5000 to .2% of 5000 and so on to -9% of 5000, remaining to be for testing. This is how we will get the training size of (500, 1000, 1500, ……,4500)and testing ratio as 5000-{training-size}.

Average time over 5 such samples of each of  the given training size. Following are the experiment results of execution times. Of different algorithms and different data sets.

## RESULTS

### Logistic Regression

I have converted data to 1 and 0 class instead of 1 and -1 class before using logit. Following are the results of logistic regression of all the three data samples (i)Banana Data (ii) NLS Data and (iii) LS Data.

In the below table the training sizes are given and the corresponding testing sizes are 5000-{training size}

### Linear Regression

Following are the results of linear regression of all the three data samples (i)Banana Data (ii) NLS Data and (iii) LS Data

### Naïve Bayes

To use Naïve Bayes I have used discretizer present in weka before applying Naïve Bayes. Following are the results of Nayes Bayes of all the three data samples (i)Banana Data (ii) NLS Data and (iii) LS Data.

First Order Polynomial Fitting for NLS Data:-

I have used polyfit and polyval for this part

Each is linear in number of Samples . ie O(N).

1. Linear Regression for degree 1 polynomial fitting with NLSData

I have computer x = x1*x1+y1*y1

And used polyfit the parameter with 1 degree polyfit are:-

-0.0237x+ 1.4974. Graph Below

With x1 I am getting the parameter with 1 degree polyfit are:-

-0.2301 x+ 1.5416. Graph below

With x2 I am getting the parameter with 1 degree polyfit are:-

1. X+  2.0415  .
• Logistic Regression  with degree 1 polynomial fitting for NLSData

I have computer x = x1*x1+y1*y1

And used polyfit the parameter with 1 degree polyfit are:-

-0.0081x+ 1.2010

With x1 I am getting the parameter with 1 degree polyfit are:-

-0.1628x+ 1.4918

With x2 I am getting the parameter with 1 degree polyfit are:-

-0.1684x+ 1.4518

• Logistic Regression  with degree 2 polynomial fitting for NLSData

x = x1*x1+y1*y1

And used polyfit the parameter with 1 degree polyfit are:-

0+-0.0151 x+1.4139

This is almost linear

With x1 I am getting the parameter with 1 degree polyfit are:-

-0.0041x^2+-0.1254x + 1.4189

This is almost linear

With x2 I am getting the parameter with 1 degree polyfit are:-

0.0003x^2 -0.17702x+ 1.5425

This is also almost straight line

So polynomial of degree 1 fits in Logistic regression and similar results were coming for linear regression

# Newtons Logistic Regression

The following are results obtained using the logistic regression using Newton’s method  classifier. I have split the datasets into 70% training and 30% testing randomly in 5 folds. Below are the details of how we choose these 5 folds.

Here the first weight represent the bias

sum = sum + W_Old(j+1)*trainingSet(t,j);

P(t) = 1/(1+ exp(-1*sum));

Z= (X’) * (Y-P)’;

W = diag(P.* (1-P));

Hessian =  X’ * W * X;

etaMatrix(1:3) = eta;

W_New = W_Old + etaMatrix .* (Hessian \ Z)’  ;

Its derivation and complete form in two ways is given as:

Parameters used for computations:

Maximum Number of  numIteration =10000;

eta = 0.5 and 0.2;  // both tested

errorBound = 0.0001;

The following are the final weights for the three data:-

The following are the TP, TN , FP, FN, Precision, recall, accuracy  for the three data:

The Decision Boundaries of the three data sets are as follows:-

The following are the figues showing (i) the decision surface and (ii) decision surface and dat both. Note different colored are used to show different classes.

1. Linearly Seperable  Data(Green is with + sign the positive and blue with –ve)

This following figure is for the plane of the Newtons method for linearly seperable data. Red dots shows the predicted line projected in 2D

This following  figure is for data for linearly separable data.

This  following figure is for the plane of the Newtons method for linearly seperable data plotted in 3D

1. NLS Data   (Green is with + sign the positive and blue with –ve)

This  following figure is for the data  for non linearly seperable data.

This  following figure is of the Newtons method for NLS seperable data. Red dots shows the predicted line projected in 2D.

This  figure is for the plane of the Newtons method for non linearly seperable data over two runs.

1. Banana Data

This following  figure is for the separating plane of the Newtons method for banana data.

This  following figure is for the plane of the Newtons method for banana data projected to 2D.

This  figure is for the plane of the Newtons method for banana data. Red dots shows the predicted line projected in 2D

The results are self explanatory with the help of image data and are not benchmarks, they are elaborated for experimental setups in labs, and to be followed by mentors teaching Machine Learning Lab Work. For benchmark results look into peer-reviewed research papers.