Sometimes when you are processing log files with the Spark, you need to have some data fields of the next (or previous) row in hand. For example, you have data file containing some credit ca… Read More

A Practical Guide on Modeling Customer Behavior Most people think deep neural networks1 or any other modern machine learning techniques are capable of doing almost anything at least in spe… Read More

The effect of time of observation in detecting anomalies or frauds I have been working on many anomaly and fraud detection systems for five years and what I have found in these years shows… Read More

Here is a white paper on time series anomaly detection. This white paper is about finding anomalies in time series, which we encounter in almost every system. I usually keep notes when I wor… Read More

One of my friends and I have built a service based on the recent posts of the blog on visualizing computer's port distribution and monitoring the usage. It is like the tools we already had h… Read More

It is alway important to remember why we do classification. We do it because we want to build a general model to support our problem, not to model the given training datasets only. Sometimes… Read More

I am working on a new project; it is about recognizing usage patterns in network traffic by having as minimum as possible data. Usually, you need to do DPI or look for signatures, ... T… Read More

We saw we could use Bayes Theorem and Machine Learning methods to catch changes in your computer's port usage. There is another complementary way we can use to find out any changes in port u… Read More

Screen snapshot of the toolIn recent 5 posts, we talked about how important is data in Machine Language algorithms and introduced a source of data every one of us who uses a c… Read More

Screen snapshot of the toolIn recent 5 posts, we talked about how important is data in Machine Language algorithms and introduced a source of data every one of us who uses a c… Read More

Screen snapshot of the toolIn recent 5 posts, we talked about how important is data in Machine Language algorithms and introduced a source of data every one of us who uses a c… Read More

The applicationI set up a web application which enables you to get the idea of how the applications you are working with on your computer, use the TCP connection ports. You can go to the fol… Read More

In the previous posts, we saw that our raw data from your computer comes to our classification program as bellow:443:47,5223:2,5228:1, ...Then we apply the following function to get the port… Read More

I used daily values of the following information from "Bank of Canada" and "Federal Reserve Bank of St. Louis." for ten years to build a model to predict the behavior of the Canadian dollar… Read More

Human eyes can process 10-15 images per second,it can handle higher rates too, but will lower precision.Consider you are sitting in a car, the car is moving on a road, and you are looking th… Read More

Random WalkConsider the following Bayesian Network or Markov Chain, which shows your walking direction. Every time you want to take a step; the network says you have the equal chance of walk… Read More

Visual perception is the power of processing information we typically get from our eyes. Why typically? Because even if we cover our eyes, we still have the ability to get visual perception… Read More

Data for Machine Learning (ML) algorithms is like fuel for a car, no fuel no movement. If you do not have access to enough data, it is better to think about using rule-based systems rather t… Read More

If you've read the posts about Bayesian Network, you are now ready to write your first pattern recognition system based on a Bayesian Network. Since I'm working with Java in these days, I pr… Read More

If you check the meaning of the word "sin" in a dictionary it is something like "transgression of the law of God", I am going to rephrase it as "violation of the society accepted law" to tal… Read More

Suppose you have a simple system which has only four different states, and your observation shows the system changes its state like the following Markov Chain:Sample Markov Chain to walk onW… Read More

I posted a simple time series pattern recognition ten days ago, and I didn't  expect readers may come, download and test it. I'm happy it happened and since some of them have asked to p… Read More

1- IntroductionOnce in a while, we decide to get used to a new habit or forget a one, like doing morning exercise, not drinking a sugary beverage, studying every night, learning to play or s… Read More

I always ask myself, do we do some complex Mathematics in our brain when we want to recognize an object or make a decision? Or do we solve tens of time-related multidimensional partial diffe… Read More

If you have ever tried to model a complex system with Bayesian Network or have an attempted to calculate the joint probability of many outcomes you know what I'm saying! Multiplication of ma… Read More

This post is about how you can have a scientific approach to thinking or analyzing different phenomena using Bayesian Network. The only tools I used to prepare information were OpenOffi… Read More

I showed in the last post (How to convert "Time Series" to "Bayesian Network") how we can convert a sample time series to a Bayesian Network as you may have seen it is easy, but there are so… Read More

Consider the following simple state transition network, it is just a simpler version of what we had in our previous post. In each state, we have only two options either staying in the state… Read More

It is not wrong if we say human vision is one the most advanced pattern recognition systems that exists. Just consider our vision system starts to work since we get born even before it! And… Read More

Suppose you need to train a machine with (or learn from) an online stream of data and the given data is not tagged, so you don't have any idea of if the recent data complies your classificat… Read More

We saw we can represent a neuron memory layer as a matrix like the following:[  [a11, a12, ... , a1n],  [a21, a22, ... , a2n],  .  .  .  [am1,&nb&hell…Read More

It turns out that human brain uses coarse coding to store information it gathers. In fact, this might be the easiest way to store this huge amount of information we gather as we explore the… Read More

dense vs coarse codefor 16 bits number in dense and equivalent 13 bits coarse code.OK, suppose you have a single variable that represents the current value of a sensor. A network usage… Read More

Although neurons can be considered having active state or not, but brain at its bare neocortex layer  doesn't understand binary information or processing binary data. I mean the neocort… Read More

The example in the previous post was about a very simple 2 states system, now let us consider a bit more complicated system which has 3 states. Things we are going to talk about or calculati… Read More

Although we can discuss this topic from mathematical point of view and specifically Markov process model but it is not that much difficult to talk about it in plain english too. What I… Read More

Our habits are strange, the more you do them the more it's difficult for you to get out of them. In a sense, it seems like some simple probability rules are controlling our habits, as you us… Read More

We want to see how the stuff we talked about before (in the following posts) can help us to learn patterns. The example is about learning the way we use words in simple sentences; the senten… Read More

I already talked about time series in "Time series as a point in space" post and also discussed a way we can find its anomalies in "A simple way of calculating anomaly" p… Read More

Have you ever thought about how a goalkeeper dives for the ball or a dog runs to catch something?! Mathematical modeling and solving some complex time related differential equations? No way!… Read More

There are some ways you can predict the future or next state of a system. In most of them, you build a statistical model from the information you have and the try to predict the next outcome… Read More

I modified previously provided tools to build a new one which gets both training and test datasets of a time series to see how much the given test dataset satisfies the learnt pattern (or si… Read More

If you ask me what are the main factors which don't let even carefully designed, developed and even tested software continue working in long run, I'd say "Memory & Disk". These two facto… Read More

Loading only one page when you run a single page application (SPA) is not the only benefit of this model for building an application. I think keeping the state of the application in the clie… Read More

If you search the web for big and famous software bugs or failures you'll find plenty of them like the followings:The famous Y2K problem in the year 2000 we experience.Similar to Y2K, we wil… Read More

If you have ever tried to use d3.js to visualize your data, you already have used some of the features of functional programming. I never forget the difficulty of the first time I tried… Read More

Java's Function and UnaryOperator provided in "java.util.function" package are two examples of functional interfaces and you can use them as a target for lambda expressions or method referen… Read More

The thing I'm going to talk about is just a sample study of why software applications get complex and buggy through the time. I'm trying to build a simple model for software application's bu… Read More

In the last post, we implicitly assumed that we have a fully connected mesh topology in our software modules while there is almost no application in which all modules have a connection to al… Read More

In two previous posts, we tried to see the effect of having a modular topology for bug generation in a development cycle. We saw that if we add some new lines to project and edit some previo… Read More

Look at the following numbers which show the number of lines in some famous software applications, I've got them from informationisbeautiful.net:Average iPhone app:      … Read More

I don't know why, but we have learned that we somehow always are in out of storage memory or disk space situation. We have learned we always have to compress data, for example, to show 256 s… Read More

Image from www.shutterstock.com Random access memory is simply an array of directly addressable elements like Bytes, Words, Longs, ... or even Strings and ... Each memory element is dir… Read More

Why do we need to work with sparse matrices or data representation? Look at the following video from "Max Planck institute of brain research". It shows the working neurons of a small pa… Read More

Before all else let me say I have no deep knowledge and experience in biology or neuroscience so most of the things I'm going to talk about is based on the Jeff Hawkins and some ot… Read More

I haven't written even a word in months, you start writing about stuff you like with love and excitement and you know you have to sacrifices something for it. At least, you have to get time… Read More