Meditation of a computer programmer

I don't have exact link from where i took the 1st 2 pictures but I did picture the 3rd one I took it while going through meme about python. I went through all 112 meditation techniques stated in Vigyan Bhairava Tantra. I have also done few dynamic meditations of Osho. The third image demonstrates nothing less all the different techniques of meditation. Despite the fact the first 2 images intimidates us with different technical terms and stages in meditation. Once you have attained Liberation, we have to come back to normal life. Please check this link and this link. Like Ox we should have Objective/Object in life and we should dissolve our self erasing everything else and finally the Objective/Object also dissolves leading us to Vanity.

My Review on "A Beginner’s Guide to Data Engineering - a 3 parts blog"

The main reason for the review is to summarize my takeaways from the blog.

The writer of the blog is a data scientist working in Airbnb and has also worked in Twitter. The blog is a writing of his understanding about his Adjacent field Data Engineering. I read many links provided in his blogs including his blogs on Data Science where he documents his work experience and also read an article about "Mastering Adjacent Disciplines".
His Main Objective for the blog was to document his learning of the Adjacent Disciplines.

Let me first summarize the points to take away in "Mastering Adjacent Disciplines"

1. First figure out what are your Adjacent discipline. Like the blogger, I am an aspirant data scientist, wanted to understand data engineering as an adjacent discipline. let us understand Adjacent discipline through examples and figure out what is it.

Product engineer, adjacent disciplines might include user interface design, user research, server development, or automated testing. 

For Infrastructure engineer, they might include database internals, basic web development, or machine learning. 

User growth engineers could benefit by increasing their skills in data science, marketing, behavioral psychology, and writing. 

For technical or project leads, adjacent disciplines might include both product management and people management. 

And if you’re a product manager or designer, you might learn to code.

2.  Understand the benefits to expend efforts

You will become self-sufficient and effective in your day-to-day job. 

Gives you the flexibility to potentially tackle those areas on your own. 

In comparison to learning a completely unrelated but perhaps still valuable discipline, you’re almost guaranteed to use these new skills you acquire in your day-to-day work. 

It benefits you and your team by increasing empathy of your team mates with other teams.

Let me also finish of with my takeaways from blog of the writer on Data Science.

1. We are not expected to be Unicorns, Unicorns do exists. I like to become one Unicorn.

Data science is not Teenage sex. i definitely know this. But we can't help people speaking about it like Teenage sex. They are just Marketing, Sales motivators but over-emphasizes real Data Scientist role and the need of Data Science.  

All DS need not be unicorns with expertise from Math/Stat, CS/ML/Algorithms, to data. We don't have such demands in the industry but Unicorns do exists.

2.  There are 2 types of Data Scientist. My skills and opportunities are almost of that of Type B, I had to move towards Type A to add more meaning to the domain of operation.

Type A Data Scientist: The A is for Analysis. This type is primarily concerned with making sense of data or working with it in a fairly static way. The Type A Data Scientist is very similar to a statistician (and may be one) but knows all the practical details of working with data that aren’t taught in the statistics curriculum: data cleaning, methods for dealing with very large data sets, visualization, deep knowledge of a particular domain, writing well about data, and so on.

Type B Data Scientist: The B is for Building. Type B Data Scientists share some statistical background with Type A, but they are also very strong coders and may be trained software engineers. The Type B Data Scientist is mainly interested in using data “in production.” They build models which interact with users, often serving recommendations (products, people you may know, ads, movies, search results).

3. Where to land for a Job and what will be the nature of work? I have taken the startup as "investigation start" on data and not as a startup company. All I had dream seem to fall only in scaled companies. I am currently under some early start up stage :(

At early stage start-ups: the primary analytic focus is to implement logging, to build ETL processes, to model data and design schemas so data can be tracked and stored. The goal here is focused on building the analytics foundation rather than analysis itself. 

At mid-stage growing start-ups: Since the company is growing, the data is probably growing too. The data platform needs to adapt, but with the foundation laid out already, there will be a natural shift to insight generation. Unless the company leverages Data Science for its strategic differentiation to start with, many analytics work are around defining KPI, attributing growth, and finding the next opportunities to grow. 

Companies who achieved scale: When the company scales up, data also scales up. It needs to leverage data to create or maintain competitive edge. e.g. Search results need to be better, recommendations need to be more relevant, logistics or operations need to be more efficient — this is the time where specialist like ML engineers, Optimization experts, Experimentation designers can play a huge role in stepping up the game. 

4. Understand the Job Nature as a whole. I am Nowhere near here. It is well taken, it is completely a different world. I shall move from Nowhere to Now here.

Skill that are required - Programming, Analytical and Experimentation.
Understanding of Infrastructure & Data pipelines - the Product, Instrument, Experiment, A/B test and Deploy

Hope that convinced you to read my blog further despite you being a data scientist aspirant just like me. Let me also put forth my take away with respect to "Data Engineering" blog.

1. Monica Rogati’s call out

Think of Artificial Intelligence as the top of a pyramid of needs. Yes, self-actualization (AI) is great, but you first need food, water, and shelter (data literacy, collection, and infrastructure).

 

2. Better understanding of "Data Engineering" field

The data engineering field could be thought of as a superset of business intelligence and data warehousing that brings more elements from software engineering. This discipline also integrates specialization around the operation of so called “big data” distributed systems, along with concepts around the extended Hadoop ecosystem, stream processing, and in computation at scale. 

A link reference from the writer blog led to The Rise of the Data Engineer. Where I could make better sense of Data Engineering field.

1. The need for flexible ETL tools lead to developement of new ETL tools like Airflow, Oozie, Azkabhan or Luigi.

2. Old ETL tools which had drag and drop facilities like Informatica, IBM Datastage, Cognos, AbInitio or Microsoft SSIS have become obsolete. 

3. New ETL tools provides flexibility and abstractions to maintain experiments, schedule experiments, allow A/B testing. They are more Open Systems.

4. Data modeling has changed - Much denormalization possiblities, better blob support, dynamic creation of schemas, snapshoting and conformane of dimensions of schemas have become less imperative. 

5. Datawarehouse is the gravity around which data engineering still moves around. Yet Datawarehouse is also publicly shared with Data Scientist & Analyst. It has become to much Centric to the IT organization as a whole, rather than Data Engineer being its owner.

6. Heavy performance tuning & optimization are being achieved as more money is invested to pour in more data and experiment with same resources. 

7. Data Integration from SAAS based OLTP applications have become difficult. Non Standard and Changing API of OLTP systems are disrupting OLAP system.

3. ETL Paradigms: JVM based ETLs and SQL based ETLs are two track of choice.

4. Understanding of Job Nature of "Data Engineer"

Build Data Warehouses with ETLs and managing data pipelines (DAG - Directed Acyclic graphs). 

Data modeling (Data Normalization and Star Schema), Data Partitioning and back filling historical records. Fact and Dimension Tables.

5. Understanding the need of moving from pipelines to frameworks.

Standalone pipelines to Dynamic pipelines have become need of the hour. It is now possible by constructing DAG via simple configuration files such yaml and has to deploy well known patterns as frameworks.

Incremental Computation Framework 

To avoid full table scans for aggregation functions, this framework pre-calculating them daily, monthly, quarterly and avoids them when data scientist does such operations.

Back fill Framework 

Back filling of historical or update records is a tedious job. But it will have to take frequently, such jobs are run with this framework. 

Global Metrics Framework

De-normalization Machines to make Dimensional cut based metrics to build de-normalized schema automatically as required for both data scientist and market facing business people

Experimentation Reporting Framework

Every data company builds experimental models in a modular fashion which remains very lengthy than production models. These most complex ETL jobs have to executed and statistical calculation are captured per module instead of complete workflow to make decisions. 

The Data Mining I did to Understand Data Mining

Why you should read this blog? What will be your takeaway?

1. Better Understanding of "Data Mining"
2. Picture data mining perfectly among the misty jargon
3. Helps to understand a student's journey.


It was my first Data Mining class of M.Tech in Data science. I was not completely focused during the class hour as the class lecture bewildered me to think What is Data Mining?

Is the word "Mining" in "Data Mining" Misleading?

I started looking for the difference between data science and data mining. My Initial though was that "Data Mining" is nothing but data collection. I thought so because, when i went through "Statistical Mathematics", the collection and cleaning of data for performing some analysis itself was huge task. Consider the age where there was no social media and no internet. Statistics had its birth out of mathematics especially probability theory. Census was very much important for proper governance during that era and people have to visit each household, each and every village and cities to collect data.  Consider every revolution any country had seen. At all these revolution huge of amount of data was required and people did collect from every required corner of the globe to bring in revolutions. So was my assumption that "Data Mining" is collection of data.

Black Revolution	Related with Petroleum Production
Blue Revolution	Related with Fish Production
Brown Revolution	Related with Leather, Cocoa
Golden Fibre Revolution	Related with Jute Production
Golden Revolution	Related with Overall Horticulture, Honey, Fruit Production
Green Revolution	Related with Agriculture Production
Grey Revolution	Related with Fertilizers
Pink Revolution	Related with Onions, Prawn
Red Revolution	Related with Meat, Tomato Production
Evergreen Revolution	Intended for overall agriculture production growth
Round Revolution	Related with Potato Production
Silver Fibre Revolution	Related with Cotton Production
Silver Revolution	Related with Egg Production
White Revolution	Related with Dairy, Milk Production
Yellow Revolution	Related with Oil Seed Production
Round Revolution	Related with Potato

Hopes turned despair and I was confused

More through the lecture my hopes were turning into disparity as the lecture took a direction different from "Data Collection". The course of the lecture was not anywhere near how to collect data, where to find for sources. How to select the sources. I started reading about the difference between Data Mining and Data Science. Instead of clearing my doubts, it catalyzed my already burning confusion.

Here is the site, I checked out .

I came across other names of "Data mining"
1. Data Archaeology
2. Data Discovery
3. Information Harvesting
4. Knowledge Extraction

Got confused and started thinking more in terms of Archaeology. Selecting site for digging after a long and thorough analysis of histories (like keeladi site excavation), theories and speculating few finding based on other findings in hand. Looks I am right, data mining seems to be data collection, but collection of data from rare sites and collection of Golden Nuggets among the debris.

With the term Data Discovery, I can say, Discovery of natural phenomenon have never been straight forward, 99.999999%, human just stumbled upon them. While "Need is mother of all Inventions", discovery unlike it has a bizzare path. Discovery has everything to do with Nature. One has to look into to Nature to discover as one is just finding what is there all the time, while invention is just a process of putting things together as per the need. Only once in a while someone discovers something meaningful in the era in which he discovers.

While thinking about Information harvesting, when did we sow the seeds to grow data to harvest?. Yes, we do sow the seeds via all our OLTP systems. Consider every form we fill to give our personal details or fill some events as part of job or fill workflow input data to generate events. Forms are our fields, input data are our seeds. Data grows in velocity, variety and volume to produce Information and we harvest Information.

While thinking about Knowledge Extraction, Knowledge is nothing but Connecting dots of Information.







None of the other names of "Data Mining" brought clarity. Data Mining, Data Archaeology and Data Discovery point a direction towards searching dirty data pile, while Knowledge discovery and Information Harvesting point out the act of extracting the Golden Nuggets.


Classify the data to get information, connect information to get knowledge. Exercising the knowledge at the required situation or context in a globally acceptable way is wisdom. Wisdom creates an impact. All is well. Yet what is data Mining?






Better Picture of the "Extraction of Golden Nuggets" Appeared

Given the confusion, i started searching for images and more clarification linking data science and data mining.









With the above images, I hit the jackpot. Everything fell into the singularity. With data science we seem to analyze the past to predict the future by penetrating the data via analysis, then analytics - automating analysis a bit with logic(completely analytical math), then with data mining - proactively making sense with heuristics with causation and correlation of different dimensional data.

The above stated comparison seems sufficient. I think we should never compare the data mining & tasks (mostly i see classification, regression etc.,) with storage systems like data lakes and data warehouses, techniques & tools like statistics methods or BI tools or Machine learning and  Roles like Data Engineer, Data Scientists,  Data Analyst, despite they are required while performing Data mining. Confusing roles, storage, technique and methods seems to be the cause of ill. At least I was receiving some information so and so during the lecture with my antenna.

Given the relief I shared one more link which I read to differentiate "Data lakes" and "Data Warehouses" with my class students - here it is.

To conclude, Data Mining is a Technique, focused on Business Process to extract Patterns of Information with the purpose of finding trends previously not found. In order to perform data mining, one has to have the understanding of data whereabouts in order to navigate across and its statistical understanding to conduct Mining operation. It is a part of Data Science to conduct Data Mining on Structured data while dealing with both structured and unstructured data. AI is part of Data Mining. There are 4 perspectives for AI. Only one winner which is what is required for Data science and the Winner is "AI which acts rationally and achieves result in optimal expense of resources (time & memory) while applying heuristics over data mine field. Machine learning and deep learning or anyway a part of AI and so they become part of Data Mining.







P.S - The lecture also expounded some business areas where data mining is applied but i was not able to appreciate them without proper understanding or definition of the term "Data Mining" itself. Hope I missed it or lecture never had it.

Feynmann Diagrams

Below are the two videos related to Feynman's diagram.

Paul Dirac out of despair writes down a little paper on Quantum Mechanics stating how in many possibilities that some particle interactions could happen.

It is this little paper which invites Feynman's attention to make the problem intuitive with Feynman's diagram.

In the below videos, I could clearly understand the diagram in relativity or space time Continuum.

It bound to works taking Space and time into consideration and particle, particle interactions and their combinations. What is more interesting is that how probability plays a role in figuring out what is dominant strategy of interaction or occurring most of the time and what is not dominant or occurring very less no. of times. This usage marks the practical applications of Quantum Mechanics or QED - Quantum Electrodynamics.

The Strong and Weak Nuclear forces which makes the world of Quantum Mechanics, Dynamics or Computing, for me looks to be attractive or repulsive forces with very less or more flux as in magnetism. I am not sure why there are 4 forces put apart separated out - Gravity, Electomagnetic, Strong Nuclear and Weak Nuclear. Can't we explain them with Biology or Life forces of plants, animal or solar systems or universe?

Believe Alignment and Magnetism generates Gravity

Prof  Eric Laithwaite, an england professor and marvelous engineer show his incredible work in the below videos. His engineering tactics are really marvelous, what impressed me was that, he was totally practical without any much of mathematics and theory, he tries out all faulty things and discovers good solution meticulously keep in mind the economy of the solution if implemented in large scale for public.

I stumble up on this video when I was trying to re-imagine gravity with magnetism, which was a constant intuition, as Gravity is the only component that does not fits in "Theory of Everything" or "Unification of forces". I happend to read "Theory of Universal Magnetism" which was propounded by Vedanthiri Maharishi and admired that this thought had not been anything new and I do know that in Hindu Literature and Yogic science, there is lot of talk about Aura and the flux energy of a person.

The "Universal Magnetism", tries to explain both about microcosm and the macrocosm with magnetic forces i.e., via bio magnets. It seems to explains wonderfully about how magnetism can help explain the universe. I had one disagreement which i am not clear what was it about while writing this blog. I remember that it has to do something with 5 elements, earth, fire, water, sky(akash) and air and how the flux energy should reside outside body as AURA, but somehow it was stated in an opposite sense that AURA remains within body.

I came across Laithwaite videos, while searching to understand Earth & Sun magnetic poles and their impact on each other and why planets and starts revolves.

Below are take away i consider post watching Laithwaite videos,

1. Creativity with Engine wingdings and models.

    a. how to change circular motion to linear motion

    b. how to change concentrated circular coils help sustain levitation.

2. Shape of things and how to obtain inspiration with a Tea Pot.

3. How one can explain other laws of nature like Boyle's law, thermodynamics with just magnetism.

4. How to think about economy of a Solution.

5. Loved his conclusion that Big Motors are efficient that Small Motors and how magnetism plays its role in both microcosm and macrocosm.

Laithwaite videos helped me to understand nature in terms of merging duality into one and witness the duality.

Linear motion and Electrical Waves

Motors Small & Large

Magnetism for explaining Gas flow & laws , Liquid and Gun

Change Shape, engineering tactics to Levitation & Linear Motion

Siddhargal

What is the difference between Siddha and Ayurveda? What are its Origin and What they deal with?

Ayurveda is considered as sister science of YOGA.
It deals with 3 Dhosas / Humors
Vata = ether+air.
Pitta - Water+fire.
and
Kapha = Water + Earth.

Child is pre-dominantly Kapha. Adult is pre-dominantly Pitta and Oldage is pre-dominantly Vata.
Ayurveda is mentioned in Vedas and Upanishads. Deals with Herds. "Disease Preventive Science".


Siddha like YOGA seems to have been provided by LORD Shiva. Agastiya Muni / Kumba Muni who wrote the grammar for Tamil including 18 Siddhas (one who attained Astma Siddhi - Utimate form) are considered as father/mother of Siddha medicine. Even Patanjali who expounded traditional YOGA is also one of the Siddhas.

It take all elements ether, air, water, fire and earth into account for medicine. Considers 3 doshas for all age group. Deals with Herbs and minerals. "Curative Science"

What is Astma Siddhi? What the 8 parts of it?

Siddhi is nothing but GREAT PERFECTION. Perfection in gaining MASS, losing MASS,  gaining SPACE, losing SPACE, gaining TIME, losing TIME, gaining TRUTH/Nature, controlling TRUTH/Nature.

• Anima: reducing one's body to the size of an atom
• Mahima: expanding one's body to an infinitely larger size
• Garima: becoming infinitely heavy
• Laghima: becoming almost weightless
• Prapthi: ability to be anywhere at will
• Prakamyam: realizing whatever one desires
• Isatvam: supremacy over nature
• Vasatva: control of natural forces

Who are these 18 Sidhhars?

Below are the 18 Siddhars mentioned in Tamil. The Samadi of each siddhar is mentioned below in tamil, most of them fall in TAMILNADU. They have written may books and scripts all have been written in Tamil.

What were their life time? When did they lived? Who was whose guru? What is their lineage? do they have some relations with Vedic period and Vedas? Why the literatures of them have been propounded in tamil?

Come let see them one by one.....


Ramanathar - Algarmalai.

Arabian Siddhar, also known as YAKOBU. Born in Nagapattinam, in the nose of Tamilnadu. Has met Bogar. Has lived in Chaturgiri (சதுரகிரி) hills in Villupuram district (where almost all 18 siddhas lived together). Has his samadi in Alagarmalai.

Bogar - Palani

Chinese Siddhar, also known as BO YANG & Lao Tse. One of the famous one, next to Agastyar. He whose divine brought  9 poision (navabashanam) to make medicine for all kinds of people via Palani Shrine. He had been kind to help all kinds of people, he who made cat to tell vedas, he who made secret of shive paravathi yoga to everyone, not knowing how to help people despite looking for Seeranjeevi and Philosopher Stone (life rejunevator) made palani Shrine. He also seems to have propounded Taoism.

Today, we can visit bogar samadhi in palani, which is guarded by pulipani lineage.

His Guru: Kalanginathar
His Student: Pulipani

Korakkar  - Kokadu Malai (Pondicherry / Puducherry)

Student of Machendra Muni / Malayala King. Killed Menanathan, son of Machendra muni & Premala by washing him and brought him back to bring his guru to senses. Became part of drama artist to escape from Premala trap. Machendra Muni who gave a boon to needy brought Korakkar out of Ashes (Vibhuthi).

Korakkar, He who gave his eyes for his guru to gain a special tasty vada which his guru.

Ganzha (Korakkar Muigai) and Pugaiilai (Bramha Pattiram) are born out of Korakkar atma sakthi, along with Bhrama muni.

He seems to have been to Gujarat and the reason for Gorakpur. Nostradamus or Vaanga should be called as Korakkar as he like them has produced "Chandrarekha" predictions.

Samadhi at Palani, Perur or Kokadu - Pondicherry. Has lived in Chaturgiri (சதுரகிரி) hills in Villupuram district

Thirumoolar - Thiruvadudurai, Nagapatinam

kuduvittukoodu paibhavar. He took the form of cow herd "Moolan", then king Veerasena and Jambhukeswaran (a student yogi). he who locked veerasena body into "Yaani thumbi/hundi" maram and made every one to call arasa maram.

Samadi in Thillai.

Karuvurrar - Karuvur, Thanjavur

One who make Shrines of god. who killed bhramarakshasi in Thanjavur temple during incarnation. Known for karuvuthal. who went to jail for making nataraja via another chola king.

World, Function, Object, Method, Behavior, Structure

1. World - Considered as Self Sustained System where computation takes place.

The world is dual in nature as we describe light being a particle and wave dual. Programming (like a sequence of timed steps) is nothing but the choreographic capture & reflection of the real world, it can be described with Object and Function duality.

2. Function - mapping the relation from one domain to another domain. Ideally without any side effects like throwing an exception because of Input/output unreliability or time/space insufficiency. 

3. Object - an Organized Structure that embodies both properties and behaviors. 

4. Method - a function can be elevated to method when it is ubiquitously occurring.

5. Behavior - a function that binds to Object.

6. Structure - schemes for Organizing Objects, a data structure.