Revisiting R and R Studio: The Normal/ Gaussian Distribution.


Binomial vs Normal Distribution

The Normal/ Gaussian Distribution is just the Binomial distribution but Extended to very large N (Sample Size), logical probabilities, and it is continuous. That is a key point of Understanding: Binomial Distribution: Discrete Normal/ Gaussian Distribution: Continuous.

An application of the normal/ Gaussian distribution can be found in Noise modelling. Noise is so random, so unpredictable that as a whole we treat it as a distribution of probabilistic values that may appear, centered on the mean.

Sometimes its high, sometimes its low most of the times we don't really know. So we roll the dice.

On some level, this is a Normal Distribution that crackle and hiss through your microphone, emanating from the blue sky beyond your window.

The Normal/ Gaussian Distribution is modeled on a bell curve with 0 being on the left. 1 being on the right. It reads cumulatively from left to right. with the mean being the center. Plugging in different statistics we can shift the mean left or right.

Here are some sample Gaussian distribution bell curve:

Playing with the mean shifts the graph around, playing with the standard deviations makes it compressed or expanded

These are the commands used to generate these graphs, we have to plot first then we can play with points:

> x <- seq(from=-4,to=4,len=100)

> plot(x,dnorm(x),pch=16,ylim=c(0,0.6),type='l',main='some Gaussian distributions')

# Prepare some Gaussian distributions

> points(x, dnorm(x, mean = 1), col = 'red', type = 'l')

> points(x, dnorm(x, mean = 2), col = 'green', type = 'l')

> points(x, dnorm(x, sd = 2), col = 'blue', type = 'l')

> points(x, dnorm(x, mean = -1 , sd = 2), col = 'purple', type = 'l')

> points(x, dnorm(x, mean = -1 , sd = 2), col = 'orange', type = 'l')

> points(x, dnorm(x, mean = -1.5 , sd = 2), col = 'cyan', type = 'l')

> points(x, dnorm(x, mean = -2 , sd = 2), col = 'magenta', type = 'l')

# plot distributions of various shapes and sizes

We can use R to further Explore the Normal Distribution with some simple commands. Lets check them out.

...

we can use pnorm(X) function to key in an X value and find the cumulative area of the graph. The area returned is the cumulative probabilities from the left side of the graph until the keyed in X value.

Example:

pnorm(-1000) = 0 # no probability of happening? 

pnorm(-10) = 7.619853e-24 # fairly low probability of Happening

pnorm(-1.75) = 0.04005916 # low probability of happening

pnorm(0) = 0.5 # Half probability

pnorm(8) = 0.8413447 # fairly high probability of happening

pnorm(1000) = 1 very # High likely probability of happening


...

Normal/ Gaussian Distribution is a key concept in Hypothesis Testing where our objective isn't to test whether to accept a null hypothesis but to determine whether to reject it or fail to reject it.

This will involve something called a p-value...

10/3/2018 COMP 809/10 Data Mining and Machine Learning  Assignment


Ok I'm going to get a bit ranty to start. I made a move to AUT to do work with IRASR, become a Professional Astronomer, and perhaps one day get a chance to work with the SKA the biggest Radio Telescope Human Beings have ever made.

Inspired by movies like Contact and 2001: A Space Odyssey, this young man decided to offer himself to the alter of the Astrophysical Sciences

I'm not, here to read papers about Data Mining related to the banking sector, loan eligibility assessment, Telecom Trouble Ticketing, ML for Self Driving Cars.

Honestly I thought I had left planet Earth behind already.

Kind of feels like that.

But here I am, doing a Series of Assignments for COMP 809/10 Data Mining and Machine Learning which requires me to read a bunch of papers on DM and ML and extract crucial information about them and put them in a two page report per paper.

Here are the list of Papers:

1. Coussement, Kristof & Lessmann, Stefan & Verstraeten, Geert. (2016). A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry. Decision Support Systems. 95. 10.1016/j.dss.2016.11.007.
2. Gerritsen, R. (1999). Assessing loan risks: a data mining case study.
3. Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D. Jackel, Mathew Monfort, Urs Muller, Jiakai Zhang, Xin Zhang, Jake Zhao: “End to End Learning for Self-Driving Cars”, 2016;
4. M. R. Kraft, K. C. Desouza and I. Androwich, "Data mining in healthcare information systems: case study of a veterans' administration spinal cord injury population," 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the, Big Island, HI, USA, 2003,
5. Nuno Carneiro, Gonçalo Figueira, Miguel Costa, A data mining based system for credit-card fraud detection in e-tail, Decision Support Systems, Volume 95, 2017, Pages 91-101, ISSN 0167-9236,
6. Aslıhan Dursun, Meltem Caber, Using data mining techniques for profiling profitable hotel customers: An application of RFM analysis, Tourism Management Perspectives, Volume 18, 2016, Pages 153-160, ISSN 2211-9736,
7. Y. Temprado, F. J. Molinero, C. Garcia and J. Gomez, "Knowledge Discovery from Trouble Ticketing Reports in a Large Telecommunication Company," 2008 International Conference on Computational Intelligence for Modelling Control & Automation, Vienna, 2008, pp. 37-42.
   doi: 10.1109/CIMCA.2008.116
8. C. Leon, F. Biscarri, I. Monedero, J. I. Guerrero, J. Biscarri and R. Millan, "Variability and Trend-Based Generalized Rule Induction Model to NTL Detection in Power Companies," in IEEE Transactions on Power Systems, vol. 26, no. 4, pp. 1798-1807, Nov. 2011.
9. R. M. Gardner, J. Bieker and S. Elwell, "Solving tough semiconductor manufacturing problems using data mining," 2000 IEEE/SEMI Advanced Semiconductor Manufacturing Conference and Workshop. ASMC 2000 (Cat. No.00CH37072), Boston, MA, USA, 2000

I have to choose 3 from among the 9 listed papers. One of which will be article [3] on Self Driving cars and Extract the following information from them to put in a 2 page report.

Background Information: Who was it that was doing the data mining, which organization ie; Google Deep Mind, Nvidia Corporation, or Universite Catholique De Lille etc...

Target Application: What Problem they were trying to solve. Usually this problem comes in the form of situations that human beings face every day such as whether a bank is deciding whether to issue a loan or not, Non Technical Losses in power Generation faced by Power Companies, or Semiconductor Manufacturing in a factory, determining the churn rate for Telecommunications Operators.

These are all situations that are relevant to every day life where data mining and machine learning can make a difference if applied properly. (Well have to read the paper to find out)

Description of the Data Collected: What are the data features/ feature vectors that the papers looked at, The Columns of Data that they collected.

Identification of Algorithm: Decision Tree, k-Nearest Neighbour,  k Mean Clustering, CNN. This is where we describe the machine learning model used. (No details required)

Pre-processing Method: How the data was pre-processed before being fed to the model. Massaging/ cleaning and filling in the blanks. Data Munging is the word. How was the data Munged.

Potential/ Actual Outcome: What was the expected result and what was the actual result. What we expect to see in the model output.

Organizational Benefit: What positive outcome was gained from Data Mining. Here is where we have to go into the nitty gritty details. We have to talk about the accuracy of the results and Potential/ Actual Benefits gained in terms of the $ and Time Saving.

Some critical thinking is involved here because your own reflection on the level of success achieved must be included. Argue the case.  With your knowledge on Data Mining and Machine Learning how could it have been done differently. Would you do it the same or tweak the model in a Certain way to get better results.

Now all we have to do is await the rise of our robotic Overlords

And that's my Assignment. Its not worth much, and there's no hands on exercise in it unlike the subsequent part 2 and unlike my other class ENSE 807 Digital Signal Processing. But it is due in 3 weeks. I have about 2 weeks to pen most of it down.

I'll keep you updated if I find anything interesting.

Sincerely yours,
SofOfTerra92

6/3/2019 Finalized subject list... and the software packages needed to do them.

So I think I've finally got together the things that I will be up to during this 12 week semester:

STAT 805 Computational Mathematics and Statistics: R and RStudio to do math
COMP 809 Data Mining and Machine Learning: python/ Anaconda/ Tensorflow for ML model training
ENSE 807 Digital Signal Processing: Matlab & Simulink for Signals Impulse/ Step Analysis

ASTR 800 Advanced Topics in Astrophysics: Is a readings paper, I need to read about the Universe, alot. and write a rather lengthy paper about it by the end of the semester.

STAT 805 Lecture on 6/3/2019

STAT 805 class today. We moved to Chapter 2 (normal distribution/ Gaussian) from Chapter 1 (binomial distribution)

Binomial Distribution

As the name implies its a 'two number' distribution. Two numbers meaning that the result of the experiment that conduct is either one case or the other, success or fail, heads or tail ,dark or light, day or night, good or evil, Batman or Superman.

note: this can be extended to continuous values as well. Like age (age >= 13 || age < 13) or weight (weight > 67.5 Kg || weight <= 67.5 kg). Treating it as a critical value. Then we can conduct a 'binomial trial' on a sample of n people with these critical value characteristics.

In a binomial trial of n repeated bouts in a multiverse (where the outcome of 1 Universe does not affect the other) with p being the probability of Batman coming out victorious is a success (my own Bias).

• The Experiment consists of 'n' number of repeated Trials - They fight in 'n' number of Universes.

• Each trial can result in only one of two possible outcomes - Either batman wins or Superman wins.

• The probability of a success case is constant - In every Universe batman has the same probability of winning (p=0.5).

• The trials are independent of each other - A batman win in one Universe does not affect the batman win in another Universe. (and in none of the Universe are both their moms named Martha. I think they both lose in that one)

There's no way to ace Computational Mathematics without actually getting down and dirty and playing with some RStudio. So lets get down to it and start simulating some non-Interdimensional Hero Bouts shall we?

Lets take the function: rbinom(20,5,0.5)

which tells us, if we sample 20 Universes (vector quantile), each Universe they fight (3 times) and in each Universe Batman (p = success) has a 0.5 percent chance in winning. Yields us:

[1] 2 3 3 1 4 3 2 5 3 0 3 4 2 4 3 2 3 2 2 1

[1] 3 4 1 2 4 3 1 3 0 3 1 2 2 1 0 2 3 2 3 1

[1] 2 2 3 2 1 3 3 3 2 3 2 3 1 4 2 4 4 2 1 5

Each of these numbers represents the 'n' number of Batman wins in each Universe. If we switch to n trials = 1, we get something that looks like Binary: rbinom(20,1,0.5)

[1] 1 0 1 1 0 1 0 0 1 1 0 0 1 0 0 0 0 1 0 0

[2]  1 1 1 1 0 0 1 0 1 1 1 0 0 0 0 0 1 0 1 1

[3] 1 0 0 1 0 0 0 0 0 1 0 0 1 1 0 0 0 0 1 1


Think of vector quantiles as your samples. Number of samples. Number of Universes or Number of kids asked to toss a coin.

5/3/2019 Club Registration Week at UoA/ AUT

Astronomically Speaking, we're all from the same place.

From the 4th of March 2019 to the 7th of March 2019 is Club Registration Week. So I decided to show up to participate more in Student Body Experiences. Admittedly, I hadn't participated in a similar event to this for 3 years now in MMU. I was eager to see how they did it here in Auckland.

To add to the campus festivities, a soiree is being held at Albert Park with plenty of performances and games.

UoA creative writers club. They do short Stories, poems and prose. I thought joining would help in my storytelling ability.

TEDx UOA. The event is in September. Registering ahead of time to get prior notifications.

Science Students Association, I though to join to improve my Science Communicator Skills. They seem to be more of an avenue for social gathering than an actual Science Association. Registration was 5 Dollars

Physics Association. Also a social gathering type society but oriented to some technical skill learning. They will be organizing a python bootcamp.

Debate Society. Debating helps us get closer to the truth, help us become better people, and help us to keep and open mind. Registration was 10 Dollars

I've always wanted to play Airsoft. The regulations are too strict in Malaysia.

Everyone should do their part to help protect the environment of  planet Earth, The only home we've ever known.

Party in the Park with a performance by a steel drums band

3/3/2019 The Lives and Deaths of Pulsars

"Consider Again that Periodically Rotating Dot. That's a Pulsar. and its slowing down."

We know that Pulsars (Rotating Neutron Stars) are Rotating Periodically on their Rotational Axis while emitting an intense beam of Electromagnetic Radiation along its magnetic Axis. This beam actually Sweeps out a cone like shape in space and if planet Earth (or any sentient beings with decent radar technology) happen to be in the way of this beam, then the Pulsar will be detectable as a repeating signal.

This rapid rotation actually slows down over time.

Pulsar P - Pdot diagram

Lets understand this diagram

The diagram is plotted by horizontal Rotation Period (P) axis against vertical Period Derivative (dp/dt) axis

Period (P): time taken to spin once
Period derivative (Pdot): instantaneous rate of change of P, basically variability of much the Pulsar is slow down.

Pulsars exist on this diagram. Pulsars of Different Ages and type/ Signature populate the diagram and as they age and slow down they move from the top left (small P and high Pdot) into the lower right hand corner (large P, low Pdot) of the mapping.

Young Pulsars spin really fast taking less time to complete a revolution and have a high rate of change of that Period. Old Pulsars spin Slowly and take a long time to alter the total time to make a complete spin.

They're like old men who can't seem to switch it up on the dance floor and move like before. So after a while, they just stifle up, shut off and die. So the trick to staying young is always keep moving.

And there's a line there called a Death Line where the Pulsar is too slow (Large Period) to generate that signature Electromagnetic beam that makes it detectable from Earth and eventually they too will shut off, ending the Pulsars life as a Radio Source, resulting in one less lighthouse in the Cosmos.

The generation of the EM beam is also tied to the strength of magnetic field (B). Low B equals no Beam.

The Pulsar inside the Heart of the Crab Nebula of 1054 is pretty young.

The P -Pdot diagram is sort of like a HR Diagram but for the lives of Pulsars. Detailing their ages until their final fates

But there is a twist, one that involves a bit of Stellar Vampirism.

How Pulsars may be born again Shining and Fast (Really, really Fast)

(That was a Fury Road reference btw)

1/3/2019 Had my first Space class today. I guess that makes me an Astronomer in training.

ASTR 800 is a Readings paper course held at AUT for postgraduate students looking to do a Mathematical Science Postgraduate Degree related to Astronomy field.

The class is being taught by Dr. Willem van Straten of the IRASR.

Dr. Willem van Straten

We will have to do a lit review on various topics in Astronomy. Hopefully to get a guideline of what we will do in our research methodologies paper.

Some takeaways from today:

The Lives of Stars

"Whats the similarity between Ogres, Onions and Stars? They all have layers"

The Class Started with some basics of Stellar Evolution. The idea is that Stars end up as either 1 of 3 things (Stellar Core Remnants) depending on their mass. White Dwarves, Neutron Stars or Black Holes.

As Stars deplete their fuel they burn different elements until there is nothing left but the iron core.

Our Star the Sun is bound to expand into a Red Giant and then shed off its layers into a white Dwarf star which has the mass of the Sun but the volume of planet Earth. After of course it has swallowed up the Earth and the Inner Planets during its Red Giant phase. No one left to sing songs of the Earth by then.

Big Stars Live Fast and Die Hard with a bang as after they undergo core collapse and Explode in Supernova event they either turn into a Neutron Star which is a super dense, super Gravitationally Intense object or a Black Hole which is also Gravitationally Intense.

Pulsars and Pulsar Timing 

"Pulsars are like Lighthouses in the Universe"

We then talked about something that Dr. van Straten was involved with personally, that is the search for Pulsars. Pulsars are rapidly rotating Neutron Stars that have a strong magnetic field. An Electromagnetic Beam is emitted along the Magnetic Axis which not necessarily the same as its Rotational Axis

Pulsars are interesting things because they represent places where Nature may exist in its most Extreme. Their Magnetic Fields definitely represent a key area of study.

Fast Radio Burst

"We are Baryons"

ASTR 800 subject is called "Advanced Topics in Astronomy and Astrophysics" and the subject title is definitely not misleading.

It was the first class and already we were studying something completely new in the field of Astrophysics and something that Dr Willem was involved in which are Fast Radio Bursts.

Strange Signals from Outer Space

Fast Radio Bursts or 'Lorimer Bursts' as they were initially known, were discovered about 20 years ago. They very brief and intense flashes (Saturated the Receiver) of electromagnetic energy emanating from Extra-galactic Sources (Places outside the Milky Way).

Those sources could range from hypothetical Magnetars or copious other violent phenomena in the Universe, perhaps even extraterrestrial technology...

Why should we hunt for FRB signals? Because it may give us insight into the Interstellar Medium (ISM) and Intergalactic Medium (IGM).

If we can know more about the ISM and IGM then we can know more about the the density/ distribtion of stuff in the Universe. By doing survey survey of all the mass in the Universe, we can understand more about Baryonic Matter. (We are made of Baryons btw) and if we can know more about Baryonic matter then we could perhaps build a better model to fit Dark Matter and Dark Energy, and if we can build a suitable model to understand DM/DE we could perhaps unlock some new physical laws down the line.

27/2/2019 Crashed a an Undergraduate Astronomy Class

Crashed an ASTR 500 Introduction to Astronomy. On my path to become a radio Astronomer I am fretted by the fact that I dont have a formal Astronomy background. My understanding of Astronomy is limited to reading popular science books like A Brief History of the Universe by J.P. McEvoy, + Lucky Planet by David Waltham, TIME: New Frontiers of Space, Our Cosmic Origins by Armand Delsemme. Also shout out to Nikos Prantzos who wrote our Cosmic Future.

Armand Delsemme studied Comets which gives him insight to talk about how Life Arose on Planet Earth.

These Books, although fun and insightful count little in the way of a formal education in Astronomy.

My Background is in Electronics Engineering Majoring in Microwave Communications, and with 2 

years grappling with some data processing (python) and web technologies(NodeJS and Express Framework).

So, on my way down from WS Science building I saw a notice on the screen that lists all ongoing lectures. It showed ASTR 500 Introduction to Astronomy was just about to start.

I went in as the class was starting and placed myself in the back, hopefully out of notice of the professor.

The class was taught by Dr. Jordan Alexander of the IRASR Institute. We were asked to Introduce ourselves. Here's what I said:

"Hi, my name is Afiq and I'm an engineer. I read Carl Sagan's Cosmos once I thought that was cool..."

And then we were off on our journey...

Hand Gestures FTW

Kicking it with the youngins

We had an exercise where we shrank the radius of the Earth down to 1 meter. Then marked the position of some things:

• Deepest Ocean Trench
• Height of Sky Scraper
• Tallest Mountains
• Low Earth Orbit/ ISS Operating height.

Also, I learned how to find shit in the Universe: Right Ascension and Declination.

Many of my follow up blogs will probably involve some things I learn from ASTR 500