Lecture #15 Mon 18 Feb 2002 NEC 0. Linear Search (Continued . . .) 1. Introduction to Big-O 2. Binary Search 3. Insertion Sort READ CHAPTER 6!!!!! Excellent!!!! Copy of Algoritmics, the 2nd edition, is now on reserve! HOMEWORK DUE NEXT CLASS!!!! 0. Linear Search (continued) (Put up functions in projector -- DrScheme) Last time, we finished up by talking about an algorithm that would search through a list of numbers to find a particular number. We discussed an improvement we could make on the algorithm if we could ensure that the list was ORDERED or SORTED in either ascending or descending order. Both of these searches are forms of the same search, referred to as a LINEAR search, because the computation cost increases linearly as the number of elements increase. Although this was an improvement, it turns out that it did not affect the speed of our worst-case scenario. If the element we were looking for was the last in the list, we would still have to go through all N elements of the list before we found what we were looking for. Can we do better than this still? Let's focus on worst-case analysis and try to find something better. 1. Big-O Notation Big-O notation can seem confusing and even unscientific at first, but it is a crucial tool in the field of algorithmic analysis, which is, in turn, a crucial part of computer science. In our previous example, we said that it would cost, worst case, N comparisons to search through our entire list, where N is the number of elements in the list. But, did we ever talk about any cost that we associate with each recursive function call or processing an if statement? It turns out that we could factor these and other items into our cost analysis, if we wanted to. What would that produce? Well, before, we said there were N comparisons, producing a cost of N. This means that if we factor in the processing of the if statement, the predicate function and the actual recursive call as well, it would be 4N, or maybe even 5N or 6N, if we kept looking. HERE'S WHERE IT GETS VERY IMPORTANT TO PAY ATTENTION: As it turns out, these constant factors are not of much interest to us, because our main concern is in computation WITH RESPECT TO N, the number of elements, especially as N grows very large! We say then, that RUNNING TIME (before referred to as computation "cost") of the linear search algorithm is O(N) or "on the order of N" or "order N" or LINEAR, because the amount of time it takes us to compute a result increases linearly as N, the number of elements we are processing, increases. Different algorithms have different running times. We will focus on worst-case running times for our analyses, and you will soon begin to understand why constant factors don't matter. Don't worry if this is still a bit difficult to understand at this point, you will learn through seeing examples. 2. Binary Search Now we will describe a search algorithm that is an ORDER OF MAGNITUDE improvement over our previous search, the linear search. This means that it has a faster or "better" running time, with respect to Big-O notation. So, can we get even more efficient? Hmm . . . well what if I said you could use the following two functions, (and let's assume we're only dealing with lists containing an even number of elements, for you nit-pickers): ;;first-half: list-of-anything -> list-of-anything ;;returns a list that contains the first length/2 elements of the list ;;second-half: list-of-anything -> list-of-anything ;;returns a list that contains the last length/2 elements of the list ;;middle: list-of-anything -> (or number empty) ;;returns the item that is at the beginning of the second half of the list ;;or empty if the second half of the list is empty (meaning the list is empty) We can do the same thing as hi-lo, right? But with a list of numbers. We keep dividing list in half, halving only one of the halves each time, until we find our number!!! This saves us a lot of time; that is, it "costs" us a lot less . . . and we can use a decision tree or other forms of analysis to help us figure out how much it will cost us . . . . Continuing with our example, that is, that N, the number of elements, is 16, and this is our sorted list: (list 2 4 5 6 8 9 10 11 15 18 19 20 21 24 25 26) Let's find 24: we use "middle" first, and ask "is 24 greater than or equal to this number?" Is 24 greater than or equal to 15? It is, so we now call "second-half", and get: (list 15 18 19 20 21 24 25 26) we use "middle" again. Is 24 greater than or equal to 21? It is, so we call "second-half" again, and get: (list 21 24 25 26) We call "middle". Is 24 greater than or equal to 25? No, so now we call "first-half" and get: (list 21 24) Then "middle". Is 24 greater than or equal to 24? Yes, so we call, "second-half" and get: (list 24) Here, the list is finally down to one element, so we check to see if this is the number we've been looking for. If it is, we've found it. If it's not, we know it's not there. Does 24 equal 24? It does. Yay! We found it! Now, how fast did we find that? What is the running time of this algoritm? Well, remembering back to HI-LO, you will find that this is basically the same thing. We're halving the list each time, until there's nothing left. So, like the function you'll do for your homework, it's TimesHalved(N) or nod(N) or log2(N), right? With a list of 16 elements, it takes four times to halve it down to 1, and 2^4 = 16, that is log2(16) = 4, it's all the same. It turns out that it takes approximately log2(N) comparisons, WORST CASE, to find out our answer. I say approximately, because it actually takes one more "halving" to take it down to 0 from 1, so, for you sticklers out there, it's really log2(N) + 1 comparisons. But this really doesn't matter because in our big-O analysis, we remove all constant factors, meaning it is O(log2(N)). On top of that, it turns out that the base of the log is a constant factor as well, so we just say that this algorithm, known as BINARY SEARCH, is O(logN) or "order log N" or "LOGARITHMIC". This means that the amount of time it takes to complete the algorithm increases logarithmicly as the number of elements increase. Again, the worst case running time of a linear search is O(N), and the worst case running time of binary search is much better, O(logN). 3. Insertion Sort We've now talked about how we can search through a list of numbers, but what about sorting a list of numbers? How could we go about doing this? Given a list of unordered numbers, how do we put them in order? Let's say we were being given the numbers in order, and just had to add them into an empty list, one at a time. Wouldn't it be easy to keep this list sorted? (This is kind of like practicing improper card etiquette and picking up your cards one-by-one, then sorting them as you put them into your hand.) Consider this function: ;;insert: number list-of-numbers -> list-of-numbers ;;inserts a number into a sorted list of numbers, and returns the new sorted list ;;WARNING: lon must be sorted!!!! (define (insert n lon) (if (empty? lon) (cons n empty) (if (< n (first lon)) (cons n lon) (cons (first lon) (insert n (rest lon)))))) This type of sort is known as an INSERTION SORT. Assuming we had N numbers to sort, how long, in terms of running time, would it take us to do this? Well, in the interest of time, we'll get a little hand-wavy here. If you want to do an exact analysis, I invite you to do so, if you'd like. (You'll find out that there's just a bunch of extra constant factors and such that you'd throw away anyway.) The way I like to think about it is this: Assume that we're inserting N elements, and i is the ith element we're inserting into the list. Well, similar to our linear search, this means that WORST CASE, we'd have to iterate through i-1 elements before we got to the insertion point for i, and if we added up the "cost" of each insertion this way, we'd get: Summation of i = 1 .. N of (i - 1) = (N-1)/2 * N (Remember from high school algrbra that the summation of x = 1 .. n of x is n/2*(n+1), and it's basically the same thing here . . .) So, in Big-O, we drop our constant factors and (N-1)/2 * N becomes O(N^2) or "order N squared" or QUADRATIC. So, insertion sort runs in POLYNOMIAL TIME, we might say. If you're wondering how we get the sort to actually take in a list, we just use this extra function that goes through the list and inserts the elements one at a time, a linear-time algorithm (which won't really affect the running time of the sort as a whole, since the work done with insert was O(N^2), polynomial, much worse than O(N), linear. ;;insert-sort: list-of-numbers -> list-of-numbers ;;takes in a list-of-numbers and returns a list of the same numbers ;;sorted in ascending order (define (insert-sort lon) (if (empty? lon) empty (insert (first lon) (insert-sort (rest lon))))) More next time . . .