When Vector Space and Matrix Met

Vectors Spaces can be studied without Matrices and Matrices can be studied without Vector Space.

In reality when we expand the study of both, we will find that Matrices can take 3 vector spaces.

The 3 vector spaces are row space, column space and null space. These spaces also have their basis.

Vector Spaces of Matrices

If A is an m×n matrix

The subspace of R^n spanned by the row vectors of A is called the row space of A.

The subspace of R^m spanned by the column vectors is called the column space of A.

The solution space of the homogeneous system of equation Ax = 0, which is a subspace of R^n, is called the nullspace of A.

Basis of Matrices Vector spaces

If a matrix R is in row echelon form,

The row vectors with the leading 1’s (i.e., the nonzero row vectors) form a basis for the row space of R.

The column vectors with the leading 1’s of the row vectors form a basis for the column space of R.

I hope if there is a unique solution for matrix AX=B, then we this unique solution form the basis for null space in R. Still doubtful, but not required to be investigated further. [Found it true, Finding a basis of the null space of a matrix , was present in tool and it maps to solution space as basis]


Rank of a matrix in terms of Vector Spaces

If Row and column space have equal dimensions dim(RS(A)) = dim(CS(A)) then rank(A) = dim(RS(A)) = dim(CS(A))

rank(A^T) = dim(RS(A^T)) = dim(CS(A)) = rank(A) Therefore rank(A^T ) = rank(A)


Nullity of Matrix

If A is an m×n matrix of rank r, then the dimension of the solution space of Ax = 0 is n – r.

nullity(A) = dim(NS(A))

nullity(A) =n - rank(A)= n-r

n=rank(A)+nullity(A) --> Rank Nullity Theorem.


Rank for solution finding

Rank we can obtain from row reduction by getting rref.

If A is an m×n matrix and rank(A) = r, then

Fundamental Space => Dimension
RS(A)=CS(AT) => r
CS(A)=RS(A^T) => r
NS(A) => n-r
NS(AT) => m-r

We can only use guassian elimination method to find rank and the get the system of equation without 0 values and replace dependent rows with substituted values, we can find the nullity. Then we can check whether rank + nullity provides dimension n of matrix (or total no. of rows).

Normal Distribution, Skewness, Kurtosis and Box plots

Please try to understand Probability Distribution a little bit, especially the particular section of "Common probability distributions and their applications".

Understanding distribution of data during mining is very important as we require to find patterns.

Normal distribution is very much useful for error detection and outlier detection.

If we take Normal Curve or Bell shaped Curve or Gaussian Curve of Normal distribution. The center portion of the curve has the highest probability of an event happening. In term of data analysis, we can find around 60% falling under center part of curve.

For a symmetric curve, the mean, median and mode will be same and will fall in the center. For a skewed distribution, it will be different. We can plot an historgram chart


Normal distribution curve showing higher probability of events happening in the center of the curve.





Covering and removing errors from the distribution results in 1 sigma, 2 sigma, 3 sigma. People try to perform 6 sigma projects based on impurity reduction or error reduction in project.



The below image show what happens to mean, median and mode when normal curve is skewed.





Quantile Quantile Plot

A box-and-whisker plot or boxplot is a diagram based on the five-number summary (completely based on median, range calculation) of a data set. The skewness also can be identified with Quantile Quantile plots.

Positive skew indirectly means that some non-conformance occurrence of events helps to minimizing the entire distribution overall value / performance (despite mean remains same, median and mode shifted towards maximum)

Negative skew indirectly means that non-conformance occurrence of events helps to maximizing the entire distribution overall value / performance (despite mean remains same, median and mode shifted towards maximum) .




Below image show how Histogram is related to Normal Distribution Curve and Box Plot





Below image show how to identify the outliers from box plot. Even i above histogram there are dots representing outliers.

Kurtosis

Kurtosis is another measure of deviation from Normal Distribution just like skewness. While skewness tell us whether there is any +ve or -ve translation of the event population. Kurtosis actually tell us the way how dispersion takes place.

While Skewness holds the central axis of the distribution and and bring an external change by changing the internal axis. Kurtosis takes the external curve or external population distribution and rearranges them to smoother it or flatten it or sharpen it.

Skewness is an internal change in the behavior of population. Kurtosis is the external change in the behavior of population.

Skewness denotes a sidewards movement, while kurtosis denotes the decrease or increase of height of the bell curve with respect to population.

Spectral Radius, Power Series and Series in general

Spectral Radius and Spectrum of a matrix are formed by the Eigen vectors and values. I started reviewing this concept that I documented as a take away in my earlier post. While reviewing what is Spectral Radius and What is it significance, I came across a Quora Response, which gave me hint that it is related to Power Series.

So what is it (Spectrum & Spectral Radius) Significance?

I could see that any series could help us to approximation of values. And please note that there are 2 kinds of series,
1. Convergence Series - A Series which converge to a finite number (real or complex does not matter)
2. Divergent Series - A series which becomes infinite.

Spectral radius has an impact on Convergence of Matrix Power Sequence M^n and Matrix Series.

if the power series ∑j=0∞cjxj${\displaystyle \underset{}{\overset{}{=0}}}$converges for some radius of convergence |x|<r$|x|<\mathrm{r.}$


There was a related Question in Quora related to Spectral Decomposition. Yes Spectral Decomposition is a familiar term it also is related to Frequency Decomposition (As Spectrum is essentially a decomposition of Electromagnetic radiation.

Spectral Decomposition is related to Data Mining in term of Analysis of Cube or Multidimensional data as like solving rubik cube collecting all similar colors on particular faces.

While doing Matrix decomposition, we prefer to perform spectral decomposition, this is possible with eigen vectors and values.

What is a Series? What are the other series available and why spectum is compared to power series?

If we are familiar with Arithmetic and Geometric Progression, we definitely know what is a sequence. A summation of Sequence is a Series. Both sequence and series will converge or diverge.

There are many associated function to series like Riemann zeta function, gamma function, polygamma function, polylogarithm  and special functions and polynomials like Bernoulli polynomial and constants like Bernoulli number, Euler number, binomial coefficient etc., which can act like tools to provide Summation of series for application of the series in real life.

Greek mathematician Archimedes seems to have produced first known summation of an infinite series with a method that is still used in the area of calculus today. He used the method of exhaustion to calculate the area under the arc of a parabola with the summation of an infinite series, and gave a remarkably accurate approximation of π. There are also many approximation of π. Please find below picture, I love the way how approximation are done by converging the series to π value.

The geometric series is a simplified form of a larger set of series called the power series. A power series is any series of the following form:

Notice how the power series differs from the geometric series:
In a geometric series, every term has the same coefficient.
In a power series, the coefficients may be different — usually according to a rule that’s specified in the sigma notation.
One can think of a power series as a polynomial with an infinite number of terms. For this reason, many useful features of polynomials carry over to power series.
Matrices are actually polynomials of linear order, so i hope there is no surprise when power series comes into the Picture.

Little bit More about Series

The well know series based on the sequences are as below,

The Power Series is very interesting for Polynomial Analysis as it is related to much of the other interesting series and very much relevant to mining and data science especially in terms of dimensional reduction with spectral decomposition.

I created a Youtube playlist with few videos related to series - check this out 

Matrix Decomposition to reduce Dimensionality of Matrices in terms of Principle Axes

Blindly any system can be defined by a Matrix as any system of equation formed with inputs and empirical outputs.

Determination of solution to these matrices is the Grant Goal of Linear Algebra.

Encountering Matrices naively as "solving simultaneous linear equation in school" is not possible as the no. of dimensional will be very large. Then how do we solve these.

Solution Approach:

We encounter this problem by "Divide And Conquer" Rule.

First and Foremost, we should ensure the vectors (columns from system of equations) are linearly independent otherwise all the below stuff is nothing but a fluff.

We first like to find the Eigen Vectors and Eigen Values of given Non Singular Matrix (i.e., consistent has one or many solutions).

Eigen Vectors are vectors which are inline with Transformation forces (system of equations), they only scale by a factor of corresponding eigen values.

We find the Orthonormal form of Eigen Vectors and determine the Principle Axes. These Orthonormal form are used forming basis of the Vector space to represent entire Matrix.

The Principle Component of the Matrix is Identified based on the Eigen Values. We give large important to largest Eigen Value.

But these procedures are not enough as not always we get Square Matrix. hence we have various other tools in our Arsenal.

Gram Schimt Orthogonalization - This Removes one basis component from other one by one and ensures that the eigen vectors are orthogonal.

We have different Matrix Decomposition Methods.
1. LU (Lower Triangular, Upper Triangular)
Divides the problem into 2 parts, makes the matrix independent of B.
2. QR
Q - Quadratic form of Matrix (Formed based on Eigen vectors and Matrix to represent the Principle Component)
R - Upper Triangular Matrix

3. SVD - UWV^T
For find the W (Sigma Matrix) which represent the eigen vectors and Principle axis together here.

SVD is the Epitome as it applies for both Square and Non Square Matrix. Dimensional Reduction is easily possible with it.

The Question then is why do we require other tools in our Arsenal?? For which problems they are Useful?

Eigen values and vectors to Principle axes

As a part of BITS WILP MFDS Session 3 I learned about eigen values and vectors.

My Take away from the session are as follows.
1. Few Jargons
2. How to calculate Eigen values and Eigen vectors? - NAIVE METHOD
3. Applications of Eigen values and Eigen vectors.
4. Non intuitive but important theorems
5. How to search for Eigen Values in large matrices with GERSCHGORIN THEOREM
6. How to identify the defect or linear dependent?
7. How to identify a basis from Eigen vectors?
8. How to Diagonalize a matrix with similarity theorem and eigen vectors as basis?
9. How to get Power of matrix A with Diagonalized Matrix?
10. Orthogonal, Orthonormal and Normalization.
11. How to determine principle axes or canonical form, if A is a Symmetry and an take Quadratic form with eigen values as basis eigen vectors will be orthonormal (orthogonal and unit vector)?

Few Jargons

1. Eigen Values - A scaling factor
2. Eigen Vectors - A Vector which does not change its direction (or does not rotate) despite the shape of the matrix sees a transformation.

Eigen if has to be said in English It can be said as "its own". Eigen is a German word. 

By Eigen values, we mean its own value and by eigen vector we mean its own vector. 

Consider the below Normal faces and Eigen faces. Eigen faces are the primary characteristics of our faces which mostly does not change despite we twist and turn our face.

3. Spectrum & Spectral Radius

Spectrum - All Eigen values of Matrix.
Spectral Radius - Longest Eigen value in the Specturm (set of eigen values).

4. Characteristic matrix - (A-lambda.I)
5. Characteristic determinant - determinant of characteristic matrix
6. Characteristic equation - Characteristic determinant equated to 0.
7. Characteristic Polynomial - Outcome of reducing Characteristic equation

Characteristic matrix is Obtained by stating  

1. say Ax = lambda.x then lambda is scalar real or complex number which just scales x without changing the direction of any of its vectors. 

2. if so, we can also say (A-lambda.I)x = 0

8. Eigen space  - a space formed by eigen vectors.

9. algebraic multiplicity
10. geometric multiplicity

The order Mλ of an eigenvalue λ as a root of the characteristic polynomial is called the algebraic multiplicity of λ.

The number mλ of linearly independent eigenvectors corresponding to λ is called the geometric multiplicity of λ. hence, mλ can be also called as dimension of eigen space.

11. Similar Matrices - Transforming Matrices which does not shifts the angle of transforming objects.


How to get Eigen Values and Eigen Vectors?

Ax = lambda.x
or
(A- lambda.I)x =0  --> It is a homogeneous system of equation. Now we can formulate a polynomial involving lambda and x and get the values of lambda. This polynomial is called as "Characteristic Polynomial"

Determinant of (A- lambda.I) is called as Characteristic Determinant.
Characteristic determinant equated to 0 gives Characteristic equation.
Reducing Characteristic equation we get a Characteristic Polynomial whose solution is our lambda.

Applying any one of lambda to homogeneous system of equation and solving the homogeneous equation by Gaussian elimination or other means, we get its corresponding eigen vectors.


Where does these Eigen Vectors and Eigen values can be applied?
1. To solve Linear Differential Equation.
2. To describe natural frequency of vibration.
3. Distinguish states of Energy.

There were stated in class, but I feel they are important to understand the normal or not changing natural states of differential equation, natural frequencies and natural states of energy.

Non Intuitive but very Important theorems.
1. Eigen vectors cannot be trivial. That is they cannot be zero vector.
2. A Matrix is singular if and only if it has Zero Eigen Values
3. Sum of Eigen Values of the Matrix = Trace of Matrix (sum of diagonal elements of diagonal square matrix)
4. Product of Eigen Values of the Matrix = Determinant of Matrix (product of diagonal elements of diagonal square matrix)
5. The diagonal matrix of upper or lower triangular matrix are the Eigen Values.
6.  A has lambda as eigen value. A inverse has 1/lambda as eigen values.
7. A and transpose of A has same eigen values (of course transpose does not changes the diagonal elements)
8. Eigen values of Real symmetric matrix are real. Eigen values of orthogonal matrix will be either 1 or -1 or can be said as their modules will be 1.


Eigen Space
If w and x are eigenvectors of a matrix A corresponding to the same eigenvalue λ
Then w + x (provided x ≠ −w) and kx for any k ≠ 0 are also eigen vectors. All these eigen vectors form a space called Eigen Space.

That is Eigen values can be repeated but has as many independent eigen vectors any matrix A could have.

What is the impact of having more linearly independent and repeated eigen values and what is the impact of not having more linearly independent for repeated eigen values?


How to search for Eigen Values in large matrices with GERSCHGORIN THEOREM?

If we could draw circles or disc with radius = sum of element value  of row other than diagonal element and with center of disc as the diagonal element value

Then,
1. Range of Eigen values defined by union of all discs.
2. Every eigen value must lie within one the disc.
3. A can be without eigen value.

Based on inequality.
λ – {aii} ≤ Σ |{aij}|, i = 1,2,....n where j≠ i

How to identify "the defect" or linear dependent?

Every polynomial has root whose sum and products make the polynomial equation itself. hence we have to get the algebraic multiplicity and geometric multiplicity.

sum of algebraic multiplicity = degree of polynomial
Mλ - algebraic mulitiplicty
mλ - Geometric multiplicity

In general it can be mλ ≤ Mλ

Δλ = Mλ − mλ is called the defect.

If rank is equal to degree of polynomial, then we don't have defect, if rank is lesser, we will have defect.

If there is no defect mλ = Mλ.

λ comes along diagonal hence the degree is related to algebraic multiplicity.

Rank or no.of linearly independent row vectors of matrix determines the geometric multiplicity.


How to identify a basis from Eigen vectors?

Eigen vectors can act as basis if there are n eigen values for nxn matrix. As Eigen vectors corresponding to distinct eigen vectors are linearly independent and they make the largest span.

How to Diagonalize a matrix with similarity theorem and eigen vectors as basis? 

Similar Matrices with respect to Eigen values or vectors are the matrices, whose eigen values are same but eigen vectors are different

y = P−1x  (x, and y are vector, where P is nxn non singular square matrix)

 is similar Matrix of A if  = P−1AP


In order to get Diagonal Matrix, we have to use similarity theorem  of similar matrices.
D = X−1 A X

here X obtained from eigen vectors of A forming the basis. It is a non singular square matrix.

D is a diagonal matrix which is a similar matrix to A.


How to get Power of matrix A with Diagonalized Matrix?

D^m = X−1A^mX

so A^m = XD^mX−1


Orthogonal, Orthonormal and Normalization.

Two vectors are Orthogonal, if they are perpendicular to each other.

Normalization is done to find unit vector kind of scaled down version with respect to Magnitude i.e norm of vector.

Orthonormal - vectors which is orthogonal and normalized are called as Orthonormal.


How to determine principle axes if A is a Symmetry and an take Quadratic form with eigen values as basis eigen vectors will be orthonormal (orthogonal and unit vector)?

if A is a symmetric Matrix,

Q = x^TAx can be related to eigen vectors in quadratic form, here x is any vector.

We know D = X−1 A X, here X which is the matrix formed from basis with eigen vectors.

and  A = XAX−1

orthogonal means X−1 = X^T

A = XAX^T

so Q = x^T.X.D.X^Tx.

Quadratic form or Canonical form or Principle axis form helps us to determine the direction of principle axis. This is very important for Dimensional Reduction.

Monte Carlo Integration

Monte Carlo Integration

I felt it as a complex topic when I wanted to check out, what it is. But the below video made it damn simple.

It has three components as stated below,

1. Arbitary Shapes for Integration

2. Random Samples which falls within curve and outside curve are taken.

3. As the sample increases to a larger number, we get to the result of Integration.

https://www.youtube.com/watch?v=AyBNnkYrSWY

.

Evaluation Metrics of Models

My Take away

1. The metrics are provided for Regression and Classification and mostly for classification but not provided for Clustering.

2. RMSE (Root Mean Square Error) better suits for Regression and RLMSE (Root Logarithmic Mean Square Error) suits better for Classification and Variance was also used to compare. This is my prior understanding.

The below measures are for classification. The main concern they are addressing seems to be the gap in classification. Either the item should be classified as this or that based on the labels or attributes despite the fact that they are spread curvy linearly.

Static Measures - Measures does not changes over time, done for whole population.

3. Confusion Matrix - I got confused initially with it, but they are just the false positives and false negative measures. It is named as Sensitivity state the model is sensitive with real input and Specificity stating the model is not missing any part of classification or capturing wrong classification. It also capture accuracy, all correctly predicted outcomes.

4. F1 Score - Harmonic Mean of Sensitivity and Specificity. That is being stated as Harmonic mean of Recall and Precision and being confused. It actually ensures that the extreme values are diminished. FBeta has a parameter to tune either to Sensitivity or Specificity.


Dynamic Measures - Measures done over Changing time or population.

5. Gain vs Lift Chart - We could split our data into various sets and keep checking our outcome metrics. it perfectly identity the thresold post which the model inflects (pass/fail).

6. Kolomogorov Smirnov chart - similar to Gain vs Lift Chart. Provides a difference between +ve and -ve distribution

7. Area Under the ROC curve (AUC – ROC) - I love this term Receiver operating characteristic (ROC) , you can see that we are treating the model as thought it is an partly impaired Human Ear and measuring how good it is at listening. As said it is measuring the gap in the classification. Compared to above 2 measures, this measure does not change with respect to population much.

8. Log Loss - Takes model capability, not just the training data distribution probability as in prior models and figures out the loss. Lower the loss better the model.

9. Gini Coefficient - A derived measure of AUC-ROC

10. Concord and discord Ratio - again a listening related measure. This measure states something related to be in tune or not.

Again Linear Regression & Classification Measure

11. R-Squared/Adjusted R-Squared - Derived measure of RMSE.

Cross Validation nothing but a strategy for testing instead of testing in production, not sure how it becomes a measure, but also considered over here.

The Reference blog does not deals with Gradient Descent & Regularization Parameters and Derivative of Gradient descent to find optimal point post gradient descent to improve model. They are part of Training Model not evaluation.

Reference:
https://www.analyticsvidhya.com/blog/2019/08/11-important-model-evaluation-error-metrics/