Search Contextures Sites

## Excel Latitude and Longitude Calculations

Getting Started

Accuracy is Important

Excel Formulas for Distance

Creating Excel Code for Distance

Working With Angles in Excel

Converting to Radians

The Final Solution

Problems Encountered

The Longitude Latitude Code

Excel Distance Calculation Workbook

Excel Function Tutorials

Thanks to Excel MVP, Jerry Latham, who contributed this tutorial for calculating distance.## Getting Started

For those who are in a rush for the solution and don't need all the background information, jump to the longitude latitude code.

The search for an accurate solution to this problem has led me to numerous sites and attempted solutions. A long list of related sites is included at the end of all of this, but the most crucial to what I've found to be the current end of the road are these:

- http://lost-species.livejournal.com/38453.html
- http://www.movable-type.co.uk/scripts/latlong-vincenty.html
- http://en.wikibooks.org/wiki/Programming:Visual_Basic_Classic/Simple_Arithmetic#Trigonometrical_Functions
along with

## Accuracy is Important

A formula that is accepted to provide results that are accurate to within millimeters is known as Vincenty's formula. Naturally the accuracy of the results depends in large part on the accuracy of the latitude/longitude pairs describing the two points.

Why all the fuss over accuracy? Well, from what I've seen of other formulas, especially those written as a single worksheet function, their values differ quite a bit for what might be considered 'life critical' situations. Typically they are short by some number of meters, typically about 20 to 30 feet per statute mile, and after flying just 30 or 40 miles, I wouldn't care to land several hundred feet short of the approach end of a runway, much less be off by over 7 miles on a trip between Los Angeles and Honolulu.

Since the general tendency in dealing with these types of calculations is to "measure it with a micrometer, mark it with chalk and cut it with an axe", what we have here is a very precise micrometer to begin the measuring process with.

## Excel Formulas for Distance

There are numerous Excel worksheet functions that will return an initial heading from one point to the destination point for a Great Circle path between them and similar formulas to return the distance between them.

But based on experience using them and comparing them to known measured distances, the Vincenty method of calculating the distance between the points has not been translated into a single Excel worksheet function, and very likely cannot be at least not easily.

Because it relies on reiterative calculations to deal with points that are very close to being on the exact opposite sides of the world, implementing it as even a series of Excel worksheet formulas is a daunting task.

## Creating Excel Code for Distance

The basis for the solution presented by T. Vincenty can be found here:

http://www.ngs.noaa.gov/PUBS_LIB/inverse.pdf

During my search I first found the movable-type.co.uk page which presented the coded solution in JavaScript. That didn't do me much good as far as coming up with an Excel VBA solution; so I continued the search and finally found the lost-species.livejournal.com page!!

I thought the search was over. Almost, but not quite.

When I moved that code into an Excel code module I was confronted with two sections that Excel not so politely informed me were "too complex" for it to evaluate. I managed to break those down into simpler statements that Excel could deal with and that did not corrupt the final results.

That is the sum total of my contribution to the function provided below. I claim nothing more than that small contribution.## Working With Angles in Excel

But before using the function, there are a few preliminary steps that must be considered. Most significant is that Excel and VB works with angles expressed in radians, not as decimal values of the angles, nor from their initial "plain English" representation.

Consider this situation:

You have a Latitude represented as 10° 27' 36" S (10 degrees, 27 minutes 36 seconds South)

You need to get that into radians, and there is not a direct way to do it, before we can get it to radians it has to be converted into a decimal representation. We need to see it as: 10.46 which is the decimal equivalent to 10° 27' 36" and we need to take into consideration whether it is a North or South latitude, with South latitudes being treated as negative numbers.

Luckily Microsoft provides a couple of handy functions to convert standard angular notations to their decimal equivalent, and back, at this page:

http://support.microsoft.com/kb/213449

Those routines are included at the code section and one of them has a change made by me to permit you to make a regular angle entry as

10~ 27' 36" S

instead of10° 27' 36" Sbecause ~ is directly accessible from the keyboard, while the ° is not.

## Converting to Radians

After converting the standard notation to a decimal value, we still have to convert that to radians and deal with the sign of the radian result.

The routines and formulas here consider negative latitudes to be South latitudes and negative longitudes to be West longitudes. While this may seem unfair to those of us living in the western hemisphere, what can I say other than deal with it!?

A decimal degree value can be converted to radians in several ways in Excel and for this process, a simple function is used that is also included in the code presented later.

The basic formula is

radians = angleAsDecimal x (Pi / 180)## The Final Solution

Copy all of the code below and paste it into a regular code module in your workbook. Instructions for placing the code into a regular module are here: Copy Excel VBA Code to a Regular Module

Set up your worksheet to pass the latitudes and longitudes of the start and end points as standard entries, then enter a formula to pass them to function distVincenty().

Given two points with these coordinates:

Point 1:

- Latitude: 37° 57' 3.7203" S
- Longitude: 144° 25' 29.5244" E
Point 2:

- Latitude: 37° 39' 10.1561" S
- Longitude: 143° 55' 35.3839" E
The general format of the function call is:

=distVincenty( Pt1_LatAsDecimal, Pt1_LongAsDecimal, Pt2_LatAsDecimal, Pt2_LongAsDecimal)A raw formula would look like this [Note the double-double quotes after the seconds entries]. The SignIt() function, provided as part of the code, converts a standard angular entry to signed decimal value.

=distVincenty(SignIt("37° 57' 3.7203"" S "), SignIt("144° 25' 29.5244"" E"), SignIt("37° 39' 10.1561"" S"), SignIt("143° 55' 35.3839"" E"))You can use the ~ symbol instead of the ° symbol if it makes it easier for you:

=distVincenty(SignIt("37~ 57' 3.7203"" S "), SignIt("144~ 25' 29.5244"" E"), SignIt("37~ 39' 10.1561"" S"), SignIt("143~ 55' 35.3839"" E"))If the coordinates for Point 1 are in B2 and C2 and the coordinates for Point 2 are in B3 and C3, then it could be entered as

=distVincenty(SignIt(B2), SignIt(C2), SignIt(B3), SignIt(C3))The result for the 2 sample points used above should be 54972.271, and this result is in Meters.

## Problems Encountered

The initial code that I began working with generated "formula too complex" errors in two statements. These statements were initially broken into two pieces and then those two separate calculations were "rejoined" in a formula to obtain their final result without the "formula too complex" error.

During the preparation of this document, the package was tested in Excel 2010 64-bit and began returning #VALUE! errors for all input values. Investigation determined that a portion of the already split formula to determine the

deltaSigmavalue was generating an overflow error. This error did not occur in the 32-bit version of Excel 2010, nor in Excel 2003 (a 32-bit application).The offending line of code was once again broken down into smaller pieces that were eventually rejoined in a mathematically correct process that resulted in the proper values being determined without resorting to any alteration of the original algorithm at all.

These sections are noted in the comments in the code for Function distVincenty() below.

For any questions regarding all of this, contact Jerry Latham at HelpFrom@JLathamSite.com.

## The Longitude Latitude Code

Now, at last, the code:

'************************************************************* Private Const PI = 3.14159265358979 Private Const EPSILON As Double = 0.000000000001 Public Function distVincenty(ByVal lat1 As Double, ByVal lon1 As Double, _ ByVal lat2 As Double, ByVal lon2 As Double) As Double 'INPUTS: Latitude and Longitude of initial and ' destination points in decimal format. 'OUTPUT: Distance between the two points in Meters. ' '================================================================================= ' Calculate geodesic distance (in m) between two points specified by ' latitude/longitude (in numeric [decimal] degrees) ' using Vincenty inverse formula for ellipsoids '================================================================================= ' Code has been ported by lost_species from www.aliencoffee.co.uk to VBA ' from javascript published at: ' http://www.movable-type.co.uk/scripts/latlong-vincenty.html ' * from: Vincenty inverse formula - T Vincenty, "Direct and Inverse Solutions ' * of Geodesics on the Ellipsoid with application ' * of nested equations", Survey Review, vol XXII no 176, 1975 ' * http://www.ngs.noaa.gov/PUBS_LIB/inverse.pdf 'Additional Reference: http://en.wikipedia.org/wiki/Vincenty%27s_formulae '================================================================================= ' Copyright lost_species 2008 LGPL ' http://www.fsf.org/licensing/licenses/lgpl.html '================================================================================= ' Code modifications to prevent "Formula Too Complex" errors ' in Excel (2010) VBA implementation ' provided by Jerry Latham, Microsoft MVP Excel Group, 2005-2011 ' July 23 2011 '================================================================================= Dim low_a As Double Dim low_b As Double Dim f As Double Dim L As Double Dim U1 As Double Dim U2 As Double Dim sinU1 As Double Dim sinU2 As Double Dim cosU1 As Double Dim cosU2 As Double Dim lambda As Double Dim lambdaP As Double Dim iterLimit As Integer Dim sinLambda As Double Dim cosLambda As Double Dim sinSigma As Double Dim cosSigma As Double Dim sigma As Double Dim sinAlpha As Double Dim cosSqAlpha As Double Dim cos2SigmaM As Double Dim C As Double Dim uSq As Double Dim upper_A As Double Dim upper_B As Double Dim deltaSigma As Double Dim s As Double ' final result, will be returned rounded to 3 decimals (mm). 'added by JLatham to break up "Too Complex" formulas 'into pieces to properly calculate those formulas as noted below 'and to prevent overflow errors when using 'Excel 2010 x64 on Windows 7 x64 systems Dim P1 As Double ' used to calculate a portion of a complex formula Dim P2 As Double ' used to calculate a portion of a complex formula Dim P3 As Double ' used to calculate a portion of a complex formula 'See http://en.wikipedia.org/wiki/World_Geodetic_System 'for information on various Ellipsoid parameters for other standards. 'low_a and low_b in meters ' === GRS-80 === ' low_a = 6378137 ' low_b = 6356752.314245 ' f = 1 / 298.257223563 ' ' === Airy 1830 === Reported best accuracy for England and Northern Europe. ' low_a = 6377563.396 ' low_b = 6356256.910 ' f = 1 / 299.3249646 ' ' === International 1924 === ' low_a = 6378388 ' low_b = 6356911.946 ' f = 1 / 297 ' ' === Clarke Model 1880 === ' low_a = 6378249.145 ' low_b = 6356514.86955 ' f = 1 / 293.465 ' ' === GRS-67 === ' low_a = 6378160 ' low_b = 6356774.719 ' f = 1 / 298.247167 '=== WGS-84 Ellipsoid Parameters === low_a = 6378137 ' +/- 2m low_b = 6356752.3142 f = 1 / 298.257223563 '==================================== L = toRad(lon2 - lon1) U1 = Atn((1 - f) * Tan(toRad(lat1))) U2 = Atn((1 - f) * Tan(toRad(lat2))) sinU1 = Sin(U1) cosU1 = Cos(U1) sinU2 = Sin(U2) cosU2 = Cos(U2) lambda = L lambdaP = 2 * PI iterLimit = 100 ' can be set as low as 20 if desired. While (Abs(lambda - lambdaP) > EPSILON) And (iterLimit > 0) iterLimit = iterLimit - 1 sinLambda = Sin(lambda) cosLambda = Cos(lambda) sinSigma = Sqr(((cosU2 * sinLambda) ^ 2) + _ ((cosU1 * sinU2 - sinU1 * cosU2 * cosLambda) ^ 2)) If sinSigma = 0 Then distVincenty = 0 'co-incident points Exit Function End If cosSigma = sinU1 * sinU2 + cosU1 * cosU2 * cosLambda sigma = Atan2(cosSigma, sinSigma) sinAlpha = cosU1 * cosU2 * sinLambda / sinSigma cosSqAlpha = 1 - sinAlpha * sinAlpha If cosSqAlpha = 0 Then 'check for a divide by zero cos2SigmaM = 0 '2 points on the equator Else cos2SigmaM = cosSigma - 2 * sinU1 * sinU2 / cosSqAlpha End If C = f / 16 * cosSqAlpha * (4 + f * (4 - 3 * cosSqAlpha)) lambdaP = lambda 'the original calculation is "Too Complex" for Excel VBA to deal with 'so it is broken into segments to calculate without that issue 'the original implementation to calculate lambda 'lambda = L + (1 - C) * f * sinAlpha * _ (sigma + C * sinSigma * (cos2SigmaM + C * cosSigma * _ (-1 + 2 * (cos2SigmaM ^ 2)))) 'calculate portions P1 = -1 + 2 * (cos2SigmaM ^ 2) P2 = (sigma + C * sinSigma * (cos2SigmaM + C * cosSigma * P1)) 'complete the calculation lambda = L + (1 - C) * f * sinAlpha * P2 Wend If iterLimit < 1 Then MsgBox "iteration limit has been reached, something didn't work." Exit Function End If uSq = cosSqAlpha * (low_a ^ 2 - low_b ^ 2) / (low_b ^ 2) 'the original calculation is "Too Complex" for Excel VBA to deal with 'so it is broken into segments to calculate without that issue 'the original implementation to calculate upper_A 'upper_A = 1 + uSq / 16384 * (4096 + uSq * _ (-768 + uSq * (320 - 175 * uSq))) 'calculate one piece of the equation P1 = (4096 + uSq * (-768 + uSq * (320 - 175 * uSq))) 'complete the calculation upper_A = 1 + uSq / 16384 * P1 'oddly enough, upper_B calculates without any issues - JLatham upper_B = uSq / 1024 * (256 + uSq * (-128 + uSq * (74 - 47 * uSq))) 'the original calculation is "Too Complex" for Excel VBA to deal with 'so it is broken into segments to calculate without that issue 'the original implementation to calculate deltaSigma 'deltaSigma = upper_B * sinSigma * (cos2SigmaM + upper_B / 4 * _ (cosSigma * (-1 + 2 * cos2SigmaM ^ 2) _ - upper_B / 6 * cos2SigmaM * (-3 + 4 * sinSigma ^ 2) * _ (-3 + 4 * cos2SigmaM ^ 2))) 'calculate pieces of the deltaSigma formula 'broken into 3 pieces to prevent overflow error that may occur in 'Excel 2010 64-bit version. P1 = (-3 + 4 * sinSigma ^ 2) * (-3 + 4 * cos2SigmaM ^ 2) P2 = upper_B * sinSigma P3 = (cos2SigmaM + upper_B / 4 * (cosSigma * (-1 + 2 * cos2SigmaM ^ 2) _ - upper_B / 6 * cos2SigmaM * P1)) 'complete the deltaSigma calculation deltaSigma = P2 * P3 'calculate the distance s = low_b * upper_A * (sigma - deltaSigma) 'round distance to millimeters distVincenty = Round(s, 3) End Function Function SignIt(Degree_Dec As String) As Double 'Input: a string representation of a lat or long in the ' format of 10° 27' 36" S/N or 10~ 27' 36" E/W 'OUTPUT: signed decimal value ready to convert to radians ' Dim decimalValue As Double Dim tempString As String tempString = UCase(Trim(Degree_Dec)) decimalValue = Convert_Decimal(tempString) If Right(tempString, 1) = "S" Or Right(tempString, 1) = "W" Then decimalValue = decimalValue * -1 End If SignIt = decimalValue End Function Function Convert_Degree(Decimal_Deg) As Variant 'source: http://support.microsoft.com/kb/213449 ' 'converts a decimal degree representation to deg min sec 'as 10.46 returns 10° 27' 36" ' Dim degrees As Variant Dim minutes As Variant Dim seconds As Variant With Application 'Set degree to Integer of Argument Passed degrees = Int(Decimal_Deg) 'Set minutes to 60 times the number to the right 'of the decimal for the variable Decimal_Deg minutes = (Decimal_Deg - degrees) * 60 'Set seconds to 60 times the number to the right of the 'decimal for the variable Minute seconds = Format(((minutes - Int(minutes)) * 60), "0") 'Returns the Result of degree conversion '(for example, 10.46 = 10° 27' 36") Convert_Degree = " " & degrees & "° " & Int(minutes) & "' " _ & seconds + Chr(34) End With End Function Function Convert_Decimal(Degree_Deg As String) As Double 'source: http://support.microsoft.com/kb/213449 ' Declare the variables to be double precision floating-point. ' Converts text angular entry to decimal equivalent, as: ' 10° 27' 36" returns 10.46 ' alternative to ° is permitted: Use ~ instead, as: ' 10~ 27' 36" also returns 10.46 Dim degrees As Double Dim minutes As Double Dim seconds As Double ' 'modification by JLatham 'allow the user to use the ~ symbol instead of ° to denote degrees 'since ~ is available from the keyboard and ° has to be entered 'through [Alt] [0] [1] [7] [6] on the number pad. Degree_Deg = Replace(Degree_Deg, "~", "°") ' Set degree to value before "°" of Argument Passed. degrees = Val(Left(Degree_Deg, InStr(1, Degree_Deg, "°") - 1)) ' Set minutes to the value between the "°" and the "'" ' of the text string for the variable Degree_Deg divided by ' 60. The Val function converts the text string to a number. minutes = Val(Mid(Degree_Deg, InStr(1, Degree_Deg, "°") + 2, _ InStr(1, Degree_Deg, "'") - InStr(1, Degree_Deg, "°") - 2)) / 60 ' Set seconds to the number to the right of "'" that is ' converted to a value and then divided by 3600. seconds = Val(Mid(Degree_Deg, InStr(1, Degree_Deg, "'") + _ 2, Len(Degree_Deg) - InStr(1, Degree_Deg, "'") - 2)) / 3600 Convert_Decimal = degrees + minutes + seconds End Function Private Function toRad(ByVal degrees As Double) As Double toRad = degrees * (PI / 180) End Function Private Function Atan2(ByVal X As Double, ByVal Y As Double) As Double ' code nicked from: ' http://en.wikibooks.org/wiki/Programming:Visual_Basic_Classic ' /Simple_Arithmetic#Trigonometrical_Functions ' If you re-use this watch out: the x and y have been reversed from typical use. If Y > 0 Then If X >= Y Then Atan2 = Atn(Y / X) ElseIf X <= -Y Then Atan2 = Atn(Y / X) + PI Else Atan2 = PI / 2 - Atn(X / Y) End If Else If X >= -Y Then Atan2 = Atn(Y / X) ElseIf X <= Y Then Atan2 = Atn(Y / X) - PI Else Atan2 = -Atn(X / Y) - PI / 2 End If End If End Function '=====================================================## Excel Distance Calculation Workbook

To see the code, and test the formulas, you can download the Excel Distance Calculation sample workbook. The file is in Excel 2003 format, and is zipped. There are macros in the workbook, so enable the macros if you want to test the code.

## Excel Function Tutorials

Excel Sum Functions -- Sum Cells

Excel VLOOKUP Function

Excel INDEX / MATCH Functions

Excel Count Functions -- Count Cells

Excel INDIRECT Function## Excel Video Tutorials for Functions

Contextures Inc., Copyright ©2014

All rights reserved.

Last updated: April 13, 2014 10:13 AM