This article provides results, graphical plots and analysis of Linear, Logistic Regression, Naive Bayes for three kinds of datasets, namely, (i) Linearly Seperable, (ii) Non-Linearly Seperable and (iii) Banana Data. This article plays a good role for new mentors, academicians and students to understand the process of experimentations and analysis, result presentation in a Machine Learning tasks. It also plots the classifier hyperplane along with the distinct class in this 2-class problem. The results are not benchmarks, they are elaborated for experimental setups in labs, and to be followed by mentors teaching Machine Learning Lab Work. For benchmark results look into peer-reviewed research papers.
RESULTS & PLOTS
Here some results obtained after the dividing the data into 30-70 ratio of testing and training respectively. This was done for 5 folds. Finally, average was taken. This has been done for all thre classifiers who’s results are discussed below.
The following table shows results for linearly seperable data for the three classifiers (i) Linear Regression (ii) Logistic Regression and (iii) Naïve Bayes. The results are averages over 5 folds and are comparable.
Plots of original Data
Following are the plots of given data.
Linearly Separable Data

Non Linearly Seperable Data plot




LS Data Results (Note the data are averages of 5 fold data)
TP | TN | FP | FN | Precision | Recall | F-Value | Accuracy | |
Linear Regression | 750 | 750 | 0 | 0 | 1 | 1 | 1 | 1 |
Logistic regression | 749.6 | 750 | 0 | 0.4 | 1 | 0.9995 | 0.9997 | 0.9997 |
Naïve Bayes | 750 | 750 | 0 | 0 | 1 | 1 | 1 | 1 |
The following table shows results for non linearly seperable data for the three classifiers (i) Linear Regression (ii) Logistic Regression and (iii) Naïve Bayes. The best results are with Linear and logistic regression.
NLS Data Results (Note the data are averages of 5 fold data)
TP | TN | FP | FN | Precision | Recall | F-Value | Accuracy | |
Linear Regression | 735.5 | 734.8000 | 15.2000 | 14.6000 | 0.9797 | 0.9805 | 0.9801 | 0.9801 |
Logistic regression | 733.8 | 733.2000 | 16.8000 | 16.2000 | 0.9776 | 0.9784 | 0.9780 | 0.9780 |
Naïve Bayes | 721 | 726 | 29 | 24 | 0.965 | 0.965 | 0.888 | 0.96 |
The following table shows results for banana data for the three classifiers (i) Linear Regression (ii) Logistic Regression and (iii) Naïve Bayes. The results are averages over 5 folds and Naive Bayes gives the best results.
TP | TN | FP | FN | Precision | Recall | F-Value | Accuracy | |
Linear Regession | 626.4000 | 633.8000 | 122.2000 | 116.6000 | 0.8370 | 0.8431 | 0.8399 | 0.8407 |
Logistic Regression | 665 | 515 | 240 | 78 | 0.7425 | 0.8950 | 0.8094 | 0.7876 |
Naïve Bayes | 666 | 666 | 96 | 72 | 0.88 | 0.88 | 0.888 | 0.88 |
The Surface and figures
The surfaces and figures of the discriminant are given below.
The surface for Linearly Seperable Data using linear regression
Below are the two plots of linearly separable data surface generated by linear regression. The two plots have different axis coordinates.


The surface for Non Linearly Seperable data using linear regression
Below are the two plots of non linearly separable data surface generated by linear regression. The two plots have different axis coordinates.


The surface for Banana Data using linear regression
Below are the two plots of banana data surface generated by linear regression. The two plots have different axis coordinates.


The Surface of LS Data uisng Logistic regression
Below are the two plots of linearly separable data surface generated by logistic regression. The two plots with different runs and different training and testing data.
The following are surface generated over two runs.


The Surface of NLS Data with Logistic Regression
Below are the two plots of non linearly separable data surface generated by logistic regression.

The Surface of BananaData with Logistic Regression
Below are the two plots of banana data surface generated by logistic regression. The two plots with different runs and different training and testing data.
(The outputs are over two runs)


More Results: On Segmented Data
Now the samples of training sizes (500, 1000, 1500,….,4500) and {{5000}-{training-size}} as the testing size is being analyzed here. I iterate over with incrementing each time the partition(train-test ratio) percentage by 0.1% of 5000 to .2% of 5000 and so on to -9% of 5000, remaining to be for testing. This is how we will get the training size of (500, 1000, 1500, ……,4500)and testing ratio as 5000-{training-size}.
Average time over 5 such samples of each of the given training size. Following are the experiment results of execution times. Of different algorithms and different data sets.
RESULTS
Logistic Regression
I have converted data to 1 and 0 class instead of 1 and -1 class before using logit. Following are the results of logistic regression of all the three data samples (i)Banana Data (ii) NLS Data and (iii) LS Data.
In the below table the training sizes are given and the corresponding testing sizes are 5000-{training size}
Testing Sample | 4500 | 4000 | 3500 | 3000 | 2500 | 2000 | 1500 | 1000 | 500 |
Banana Data | 3.5992 | 2.7031 | 2.8467 | 2.6833 | 1.4648 | 1.1170 | 1.0087 | 0.6808 | 0.4955 |
NLS Data | 3.5631 | 3.3242 | 3.4129 | 3.1384 | 3.3511 | 1.5047 | 1.4308 | 0.8716 | 0.5782 |
LS Data | 0.4260 | 0.3063 | 0.1808 | 0.6101 | 0.3813 | 0.0842 | 0.2085 | 0.0642 | 0.0364 |
Linear Regression
Following are the results of linear regression of all the three data samples (i)Banana Data (ii) NLS Data and (iii) LS Data
Training Sample | 4500 | 4000 | 3500 | 3000 | 2500 | 2000 | 1500 | 1000 | 500 |
Banana Data | 0 | 0 | 0 | 0 | 0 | 0.0633 | 0 | 0 | 0 |
NLS Data | 0 | 0.0989 | 0 | 0 | 0.1900 | 0.0570 | 0 | 0 | 0 |
LS Data | 0 | 0 | 0 | 0 | 0 | 0.0566 | 0 | 0 | 0 |
Naïve Bayes
To use Naïve Bayes I have used discretizer present in weka before applying Naïve Bayes. Following are the results of Nayes Bayes of all the three data samples (i)Banana Data (ii) NLS Data and (iii) LS Data.
Training Sample | 4500 | 4000 | 3500 | 3000 | 2500 | 2000 | 1500 | 1000 | 500 |
Banana Data | 0 | 0 | 0.02 | 0.02 | 0.02 | 0 | 0 | 0.02 | 0 |
NLS Data | 0 | 0 | 0 | 0.02 | 0.02 | 0.02 | 0 | 0.02 | 0.02 |
LS Data | 0 | 0.02 | 0 | 0 | 0.02 | 0 | 0 | 0 | 0.02 |
First Order Polynomial Fitting for NLS Data:-
I have used polyfit and polyval for this part
Each is linear in number of Samples . ie O(N).
- Linear Regression for degree 1 polynomial fitting with NLSData
I have computer x = x1*x1+y1*y1
And used polyfit the parameter with 1 degree polyfit are:-
-0.0237x+ 1.4974. Graph Below

With x1 I am getting the parameter with 1 degree polyfit are:-
-0.2301 x+ 1.5416. Graph below

With x2 I am getting the parameter with 1 degree polyfit are:-
- X+ 2.0415 .
- Logistic Regression with degree 1 polynomial fitting for NLSData
I have computer x = x1*x1+y1*y1
And used polyfit the parameter with 1 degree polyfit are:-
-0.0081x+ 1.2010
With x1 I am getting the parameter with 1 degree polyfit are:-
-0.1628x+ 1.4918
With x2 I am getting the parameter with 1 degree polyfit are:-
-0.1684x+ 1.4518
- Logistic Regression with degree 2 polynomial fitting for NLSData
x = x1*x1+y1*y1
And used polyfit the parameter with 1 degree polyfit are:-
0+-0.0151 x+1.4139
This is almost linear
With x1 I am getting the parameter with 1 degree polyfit are:-
-0.0041x^2+-0.1254x + 1.4189
This is almost linear
With x2 I am getting the parameter with 1 degree polyfit are:-
0.0003x^2 -0.17702x+ 1.5425
This is also almost straight line
So polynomial of degree 1 fits in Logistic regression and similar results were coming for linear regression
Linear Regression | Parameters of polyfit for degree 1 with x = x1*x1+y1*y1 | Parameters of polyfit for degree 1 with x = x1 | Parameters of polyfit for degree 1 with x = x2 |
NLS Data | -0.0237x+ 1.4974 | -0.2301x+ 1.5416 | 1.709x+2.0415 |
LS Data | -0.0018x+ 1.2346 | -0.1477x+ 1.5013 | -0.1794x+1.9014 |
Banana Data | -0.0129x+ 2.2202 | -0.2264x+ -1.4032 | 0.9265x+ -1.6075 |
Logistic Regression | Parameters of polyfit for degree 1 with x = x1*x1+y1*y1 | Parameters of polyfit for degree 1 with x = x1 | Parameters of polyfit for degree 1 with x = x2 |
NLS Data | -0.0081x+ 1.2010 | -0.1628x+ 1.4918 | -0.1684x+1.4518 |
LS Data | -0.0020x+ 1.0548 | -0.0794x+ 1.2986 | -0.0785x+1.2847 |
Banana Data | -0.0021x+ 0.8505 | 0.0438x+ 0.5558 | 0.0191x+ 0.5530 |
Newtons Logistic Regression
The following are results obtained using the logistic regression using Newton’s method classifier. I have split the datasets into 70% training and 30% testing randomly in 5 folds. Below are the details of how we choose these 5 folds.
Here the first weight represent the bias
sum = sum + W_Old(j+1)*trainingSet(t,j);
P(t) = 1/(1+ exp(-1*sum));
Z= (X’) * (Y-P)’;
W = diag(P.* (1-P));
Hessian = X’ * W * X;
etaMatrix(1:3) = eta;
W_New = W_Old + etaMatrix .* (Hessian \ Z)’ ;
Its derivation and complete form in two ways is given as:

Parameters used for computations:
Maximum Number of numIteration =10000;
eta = 0.5 and 0.2; // both tested
errorBound = 0.0001;
The following are the final weights for the three data:-
W0 | W1 | W2 | |
LS Data | 8.4137 | -0.4298 | -0.4125 |
NLS Data | 26.2879 | -2.2458 | -2.1134 |
Banana Data | 0.6495 | 0.3533 | -0.1829 |
The following are the TP, TN , FP, FN, Precision, recall, accuracy for the three data:
TP | TN | FP | FN | Precision | Recall | F-Value | Accuracy | |
LS Data | 750 | 750 | 0 | 0 | 1 | 1 | 1 | 1 |
NLS Data | 737.6 | 732.8 | 17.2 | 12.4 | 0.9772 | 0.9835 | 0.9835 | 0.9803 |
Banana Data | 630.6 | 647.4 | 108.6 | 112.4 | 0.8531 | 0.8487 | 0.8509 | 0.8526 |
The Decision Boundaries of the three data sets are as follows:-
The following are the figues showing (i) the decision surface and (ii) decision surface and dat both. Note different colored are used to show different classes.
- Linearly Seperable Data(Green is with + sign the positive and blue with –ve)
This following figure is for the plane of the Newtons method for linearly seperable data. Red dots shows the predicted line projected in 2D

This following figure is for data for linearly separable data.

This following figure is for the plane of the Newtons method for linearly seperable data plotted in 3D

- NLS Data (Green is with + sign the positive and blue with –ve)
This following figure is for the data for non linearly seperable data.

This following figure is of the Newtons method for NLS seperable data. Red dots shows the predicted line projected in 2D.

This figure is for the plane of the Newtons method for non linearly seperable data over two runs.


- Banana Data
This following figure is for the separating plane of the Newtons method for banana data.

This following figure is for the plane of the Newtons method for banana data projected to 2D.

This figure is for the plane of the Newtons method for banana data. Red dots shows the predicted line projected in 2D

The results are self explanatory with the help of image data and are not benchmarks, they are elaborated for experimental setups in labs, and to be followed by mentors teaching Machine Learning Lab Work. For benchmark results look into peer-reviewed research papers.
You must log in to post a comment.