Search Contextures Sites
Accuracy is Important
Excel Formulas for Distance
Creating Excel Code for Distance
Working With Angles in Excel
Converting to Radians
The Final Solution
The Longitude Latitude Code
Excel Distance Calculation Workbook
Excel Function Tutorials
Thanks to Excel MVP, Jerry Latham, who contributed this tutorial for calculating distance.
For those who are in a rush for the solution and don't need all the background information, jump to the longitude latitude code.
The search for an accurate solution to this problem has led me to numerous sites and attempted solutions. A long list of related sites is included at the end of all of this, but the most crucial to what I've found to be the current end of the road are these:
A formula that is accepted to provide results that are accurate to within millimeters is known as Vincenty's formula. Naturally the accuracy of the results depends in large part on the accuracy of the latitude/longitude pairs describing the two points.
Why all the fuss over accuracy? Well, from what I've seen of other formulas, especially those written as a single worksheet function, their values differ quite a bit for what might be considered 'life critical' situations. Typically they are short by some number of meters, typically about 20 to 30 feet per statute mile, and after flying just 30 or 40 miles, I wouldn't care to land several hundred feet short of the approach end of a runway, much less be off by over 7 miles on a trip between Los Angeles and Honolulu.
Since the general tendency in dealing with these types of calculations is to "measure it with a micrometer, mark it with chalk and cut it with an axe", what we have here is a very precise micrometer to begin the measuring process with.
There are numerous Excel worksheet functions that will return an initial heading from one point to the destination point for a Great Circle path between them and similar formulas to return the distance between them.
But based on experience using them and comparing them to known measured distances, the Vincenty method of calculating the distance between the points has not been translated into a single Excel worksheet function, and very likely cannot be at least not easily.
Because it relies on reiterative calculations to deal with points that are very close to being on the exact opposite sides of the world, implementing it as even a series of Excel worksheet formulas is a daunting task.
The basis for the solution presented by T. Vincenty can be found here:
I thought the search was over. Almost, but not quite.
When I moved that code into an Excel code module I was confronted with two sections that Excel not so politely informed me were "too complex" for it to evaluate. I managed to break those down into simpler statements that Excel could deal with and that did not corrupt the final results. That is the sum total of my contribution to the function provided below. I claim nothing more than that small contribution.
But before using the function, there are a few preliminary steps that must be considered. Most significant is that Excel and VB works with angles expressed in radians, not as decimal values of the angles, nor from their initial "plain English" representation.
Consider this situation:
You have a Latitude represented as 10° 27' 36" S (10 degrees, 27 minutes 36 seconds South)
You need to get that into radians, and there is not a direct way to do it, before we can get it to radians it has to be converted into a decimal representation. We need to see it as: 10.46 which is the decimal equivalent to 10° 27' 36" and we need to take into consideration whether it is a North or South latitude, with South latitudes being treated as negative numbers.
Luckily Microsoft provides a couple of handy functions to convert standard angular notations to their decimal equivalent, and back, at this page:
Those routines are included at the code section and one of them has a change made by me to permit you to make a regular angle entry as
10~ 27' 36" S instead of 10° 27' 36" S
After converting the standard notation to a decimal value, we still have to convert that to radians and deal with the sign of the radian result.
The routines and formulas here consider negative latitudes to be South latitudes and negative longitudes to be West longitudes. While this may seem unfair to those of us living in the western hemisphere, what can I say other than deal with it!?
A decimal degree value can be converted to radians in several ways in Excel and for this process, a simple function is used that is also included in the code presented later.
The basic formula is
radians = angleAsDecimal x (Pi / 180)
Copy all of the code below and paste it into a regular code module in your workbook. Instructions for placing the code into a regular module are here: Copy Excel VBA Code to a Regular Module
Set up your worksheet to pass the latitudes and longitudes of the start and end points as standard entries, then enter a formula to pass them to function distVincenty().
Given two points with these coordinates:
- Latitude: 37° 57' 3.7203" S
- Longitude: 144° 25' 29.5244" E
- Latitude: 37° 39' 10.1561" S
- Longitude: 143° 55' 35.3839" E
The general format of the function call is:
=distVincenty( Pt1_LatAsDecimal, Pt1_LongAsDecimal, Pt2_LatAsDecimal, Pt2_LongAsDecimal)
A raw formula would look like this [Note the double-double quotes after the seconds entries]. The SignIt() function, provided as part of the code, converts a standard angular entry to signed decimal value.
=distVincenty(SignIt("37° 57' 3.7203"" S "), SignIt("144° 25' 29.5244"" E"), SignIt("37° 39' 10.1561"" S"), SignIt("143° 55' 35.3839"" E"))
You can use the ~ symbol instead of the ° symbol if it makes it easier for you:
=distVincenty(SignIt("37~ 57' 3.7203"" S "), SignIt("144~ 25' 29.5244"" E"), SignIt("37~ 39' 10.1561"" S"), SignIt("143~ 55' 35.3839"" E"))
If the coordinates for Point 1 are in B2 and C2 and the coordinates for Point 2 are in B3 and C3, then it could be entered as
=distVincenty(SignIt(B2), SignIt(C2), SignIt(B3), SignIt(C3))
The initial code that I began working with generated "formula too complex" errors in two statements. These statements were initially broken into two pieces and then those two separate calculations were "rejoined" in a formula to obtain their final result without the "formula too complex" error.
During the preparation of this document, the package was tested in Excel 2010 64-bit and began returning #VALUE! errors for all input values. Investigation determined that a portion of the already split formula to determine the deltaSigma value was generating an overflow error. This error did not occur in the 32-bit version of Excel 2010, nor in Excel 2003 (a 32-bit application).
The offending line of code was once again broken down into smaller pieces that were eventually rejoined in a mathematically correct process that resulted in the proper values being determined without resorting to any alteration of the original algorithm at all.
These sections are noted in the comments in the code for Function distVincenty() below.
To see the code, and test the formulas, you can download the Excel Distance Calculation sample workbook. The file is in Excel 2003 format, and is zipped. There are macros in the workbook, so enable the macros if you want to test the code.
Excel Video Tutorials for Functions
Contextures Inc., Copyright ©2015
All rights reserved.
Last updated: April 13, 2014 10:13 AM