Sorting is a fundamental operation in the field of computer science, and because of this, there are various algorithms available to solve this problem. Each one is chosen based on factors such as the number of items to sort, the degree of order already present, the computer architecture where the algorithm will be executed, the type of storage device, among others. In this article, we will explore the insertion sort algorithm, understanding its nuances, strengths, and limitations.

What is insertion sort?

Insertion sort is a comparison-based algorithm that constructs its output one element at a time. It works similarly to the method we use to sort a deck of cards: we take one card at a time, compare it with the ones we already have in our hand, place the card in the correct position, and repeat this action until we finish our deck.

It is an adaptive algorithm, meaning it is efficient for small data sets, as well as other quadratic complexity algorithms ($O(n^2)$). It is simple to implement, requires a constant amount of memory, as changes in the list are made in the list itself (without the need to create a new list, which would double the use of memory), and is capable of sorting the list as it receives it.

How does insertion sort work?

Initialization: We assume that the first element of our list is already sorted. We proceed to the next element, consider it as our key, and insert it in the correct position in the sorted part of the list;

Iteration: For each item in the list (starting from the second element), we store the current item (key) and its position. Then we compare the key with the elements in the sorted part of the list (elements before the key);

Insertion: If the current element in the sorted part is greater than the key, we move that element one position up. This creates space for the new key to be inserted;

Repositioning the Key: We continue moving elements one position up until we find the correct position for the key. This position is found when we encounter an element that is less than or equal to the key or when we reach the beginning of the list;

Repeat: The process is repeated for all the elements in the list.

Implementation in JavaScript

To better understand the algorithm, let’s implement it in JavaScript:

/**
* Sorts an array of numbers using the insertion sort algorithm.
* 
* @param  {number[]}  numbers - The list of numbers to be sorted.
* @returns  {number[]} - The sorted list of numbers.
*/
function insertionSort(numbers) {
  for (let i = 1; i < numbers.length; i++) {
    const key = numbers[i]
    let j = i - 1
    
    while (j >= 0 && numbers[j] > key) {
      numbers[j  +  1] =  numbers[j]
      j--
    }

    numbers[j  +  1] = key
  }
}

Complexity Analysis

Time Complexity

Best Case (Array is already sorted): $O(n)$. This is because the inner loop (while) is not executed at all; Average Case and Worst Case (Array is sorted in reverse order): $O(n^2)$. In the worst case, each iteration will cause an element to be moved. This makes the algorithm inefficient for large data sets.

Space Complexity

Space Complexity: $O(1)$. Insertion sort is an in-place algorithm; it requires a constant amount of memory space.

What is an Algorithm?

An algorithm is a precise and unambiguous specification of a sequence of computational steps that can be mechanically performed^[1]. From this, we can think of a function that receives a value or a set of values as input and returns a value or a set of values as its output.

An algorithm can be correct or incorrect. It is correct when, given its input parameters, its output is correct and, therefore, solves the computational problem for which it was developed. An incorrect algorithm, on the other hand, may stop with an incorrect output or may not stop at all for some input instances. Still, some incorrect algorithms can still have useful applications.

There can be different algorithms that solve the same problem, some more efficient, that is, faster than others. But, not every problem has an efficient solution. Such problems are known as NP-complete.

NP-complete problems are very interesting: even though no efficient algorithm has been found for this class of problems, it has not been proven that it is not possible to find an efficient algorithm (from class P, which can be solved in polynomial time) for such a problem. Moreover, if there were an efficient algorithm to solve an NP-complete problem, it would mean that there is an efficient algorithm for all NP-complete problems.

Algorithmic Complexity

When we talk about algorithms, most of the time we are interested in the growth rate of time and space required to solve increasingly larger instances of certain problems. If we are interested in the time a particular algorithm takes to perform its function, we are interested in its time complexity. And the behavior of the time complexity limit of our algorithm in relation to the increase of the problem instances is called asymptotic time complexity. And it is this asymptotic complexity that determines the size of the problem that can be solved by algorithms^[2].

If an algorithm takes a time $cn^2$ for a constant $c$ to process an input of size $n$, we say that the complexity of the algorithm is of the order of $n^2$, or, in Bachmann–Landau notation (also called asymptotic notation and Big O notation), the algorithm has complexity $O(n^2)$.

To get a better idea of what this means in relation to the runtime of our algorithm, consider that one unit of time on the computer on which we run our algorithm is 1 millisecond. Now, we want to know what the maximum size of input that our algorithm can process within a certain time limit (one second, one hour, and one day). Note, in the table below, how much the complexity of the algorithm interferes with the maximum size of the input it can handle, given the time limit:

Time Complexity	1 second	1 minute	1 hour
$n$	1000	60000	360000
$n \log_2 n$	140	4895	204095
$n^2$	31	244	1897
$n^3$	10	39	153
$2^n$	9	15	21

Even though we can build faster computers, the increase in the execution speed of less efficient algorithms would not be so significant, so we should always seek the most efficient algorithm to address a given problem.

---

1. AHO, Alfred V.; ULLMAN, Jeffrey D. Foundations of Computer Science. Stanford, 1994. ↩︎

2. AHO, Alfred V.; HOPCROFT, John E.; ULLMAN, Jeffrey D. The Design and Analysis of Computer Algorithms. Addison-Wesley, 1974. ↩︎

Consider the following function that calculates the factorial of a number:

const factorial = (n) => {
  let result = 1;

  while (n > 1) {
    result *= n;
    n--;
  }

  return result;
};

The function above was implemented iteratively, that is, it uses a loop to calculate the factorial of a number. However, it is possible to implement the same function recursively (that is, a function that references itself):

const factorial = (n) => {
  if (n === 0) return 1;

  return n * factorial(n - 1);
};

The result of both functions is the same, however, the iterative function is much more efficient (in JavaScript) than the recursive function. In addition, if we try to calculate the factorial of a very large number, we encounter the error RangeError: Maximum call stack size exceeded. Let’s understand why this happens and how we can improve the recursive function.

Call Stack

A call stack is a data structure that stores information about a program’s functions. When a function is called, it is added to the execution stack, as well as all the functions it calls. When a function returns, it is removed from the execution stack. Each function added to the stack is called a stack frame.

In order to understand what is happening, let’s try to represent, graphically, how the calculation of the factorial of 6 is done with the iterative function:

Now, compare it with the substitution model for calculating the factorial of 6 using the recursive function:

Note that, in the iterative function, the arrow shape is linear and we can see the state of each variable at each step. In addition, at each iteration of our loop, a calculation is performed and the variables stored in memory are updated. In the recursive function, the arrow shape is exponential and we cannot see the state of all variables in the first half of the processing. In addition, each time the function is executed, more memory needs to be used to store the resulting values of each execution.

But what does this mean? In order for JavaScript to calculate the factorial of 6 using the iterative function, the while condition is added to the stack, where its calculation is performed, the result variable is updated, and then the executed code block of the while is removed from the stack. This is done until the while condition is false, that is, until the value of n is less than or equal to 1.

In the recursive function, each call to the factorial function is added to the stack as many times as necessary until the if condition is false, that is, until the value of n is less than or equal to 1. This means that, to calculate the factorial of 6, the factorial function is added to the stack 6 times before being executed. And that’s why, when we try to calculate the factorial of a large number (100,000, for example), we encounter the error RangeError: Maximum call stack size exceeded: there is not enough space in the stack to store all the calls to the factorial function.

Introducing Tail Call Optimization

As defined by Dr. Axel Rauschmayer:

[…] whenever the last thing a function does is call another function, then this last function does not need to return to its caller. As a consequence, no information needs to be stored on the call stack and the function call is more like a goto (a jump). This type of call is called a tail call; not increasing the stack is called tail call optimization (TCO).

Now, we have discovered that our factorial calculation function is not tail recursive. But how can we make it tail recursive? With the help of another function:

const factorial = (n) => {
  return factorialHelper(n, 1);
};

const factorialHelper = (x, accumulator) => {
  if (x <= 1) {
    return accumulator;
  }

  return factorialHelper(x - 1, x * accumulator);
};

Now, our function is tail recursive: the last thing it does is call a function (and not calculate an expression, as in the first implementation). Now, let’s see the substitution model for calculating the factorial of 6 with our new factorial function:

The performance is superior to our first implementation, although it still doesn’t beat the performance of the iterative function. However, we still encounter the error RangeError: Maximum call stack size exceeded. But why does this happen? Because, despite our function being tail recursive, current versions of Node.js and browsers (with the exception of Safari) do not implement Tail Call Optimization (despite its inclusion in the EcmaScript specification since 2015).

But how will we solve this problem? With the help of another function, of course! For that, we will rely on the Trampoline pattern:

const trampoline = (fn) => {
  while (typeof fn === "function") {
    fn = fn();
  }

  return result;
};

Our trampoline function consists of a loop that invokes a function that wraps another function (what we call a thunk) until there are no more functions to execute. Let’s see how the implementation of our factorial function would look like with the Trampoline pattern:

const trampoline = (fn) => {
  while (typeof fn === "function") {
    fn = fn();
  }

  return fn;
};

const factorialHelper = (x, accumulator) => {
  if (x <= 1) {
    return accumulator;
  }

  // Now, a function returns another function
  return () => factorialHelper(x - 1, x * accumulator);
};

const factorial = (n) => {
  return trampoline(factorialHelper(n, 1));
};

And now, we can call our factorial function with a large number, without encountering the error RangeError: Maximum call stack size exceeded. Of course, depending on the factorial we want to calculate, we will encounter an Infinity, as it is a very large number (a number greater than Number.MAX_SAFE_INTEGER: 253 - ¹). In this case, we can use BigInt:

const trampoline = (fn) => {
  while (typeof fn === "function") {
    fn = fn();
  }

  return fn;
};

const factorialHelper = (x, accumulator) => {
  if (x <= 1) {
    return accumulator;
  }

  return () => factorialHelper(x - 1n, x * accumulator);
};

const factorial = (n) => {
  // Converting values to BigInt
  //-------------------------------\/----------\/
  return trampoline(factorialHelper(BigInt(n), 1n));
};

Typing our function

And finally, let’s add the necessary types to our factorial function:

type Thunk = bigint | (() => Thunk);

const trampoline = (fn: Thunk) => {
  while (typeof fn === "function") {
    fn = fn();
  }

  return fn;
};

const factorialHelper = (x: bigint, accumulator: bigint): Thunk => {
  if (x <= 1) {
    return accumulator;
  }

  return () => factorialHelper(x - 1n, x * accumulator);
};

const factorial = (n: number) => {
  return trampoline(factorialHelper(BigInt(n), 1n));
};

References

• What happened to proper tail calls in JavaScript?
• Tail Call Optmization
• Limites da recursão em JavaScript, TCO e o pattern Trampoline
• What is an Activation object in JavaScript?
• Factorial
• Tail Recursion Explained - Computerphile
• Tail Call Optimization: The Musical!!