Search Contextures Sites

 

Contextures
Excel news
by email

 

 

 

30 Excel Functions in 30 Days

 

 

 

 

 

 

 

 

 

 

 

Learn how to create Excel dashboards.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Learn how to create Excel dashboards.

Excel Latitude and Longitude Calculations

Getting Started 
Accuracy is Important
Excel Formulas for Distance
Creating Excel Code for Distance
Working With Angles in Excel
Converting to Radians
The Final Solution
Problems Encountered
The Longitude Latitude Code
Excel Distance Calculation Workbook
Excel Function Tutorials

Thanks to Excel MVP, Jerry Latham, who contributed this tutorial for calculating distance.

Getting Started

For those who are in a rush for the solution and don't need all the background information, jump to the longitude latitude code.

The search for an accurate solution to this problem has led me to numerous sites and attempted solutions. A long list of related sites is included at the end of all of this, but the most crucial to what I've found to be the current end of the road are these:

along with

Accuracy is Important

A formula that is accepted to provide results that are accurate to within millimeters is known as Vincenty's formula. Naturally the accuracy of the results depends in large part on the accuracy of the latitude/longitude pairs describing the two points.

Why all the fuss over accuracy? Well, from what I've seen of other formulas, especially those written as a single worksheet function, their values differ quite a bit for what might be considered 'life critical' situations. Typically they are short by some number of meters, typically about 20 to 30 feet per statute mile, and after flying just 30 or 40 miles, I wouldn't care to land several hundred feet short of the approach end of a runway, much less be off by over 7 miles on a trip between Los Angeles and Honolulu.

Since the general tendency in dealing with these types of calculations is to "measure it with a micrometer, mark it with chalk and cut it with an axe", what we have here is a very precise micrometer to begin the measuring process with.

Excel Formulas for Distance

There are numerous Excel worksheet functions that will return an initial heading from one point to the destination point for a Great Circle path between them and similar formulas to return the distance between them.

But based on experience using them and comparing them to known measured distances, the Vincenty method of calculating the distance between the points has not been translated into a single Excel worksheet function, and very likely cannot be at least not easily.

Because it relies on reiterative calculations to deal with points that are very close to being on the exact opposite sides of the world, implementing it as even a series of Excel worksheet formulas is a daunting task.

Creating Excel Code for Distance

The basis for the solution presented by T. Vincenty can be found here:

http://www.ngs.noaa.gov/PUBS_LIB/inverse.pdf

During my search I first found the movable-type.co.uk page which presented the coded solution in JavaScript. That didn't do me much good as far as coming up with an Excel VBA solution; so I continued the search and finally found the lost-species.livejournal.com page!!

I thought the search was over. Almost, but not quite.

When I moved that code into an Excel code module I was confronted with two sections that Excel not so politely informed me were "too complex" for it to evaluate. I managed to break those down into simpler statements that Excel could deal with and that did not corrupt the final results. That is the sum total of my contribution to the function provided below. I claim nothing more than that small contribution.

Working With Angles in Excel

But before using the function, there are a few preliminary steps that must be considered. Most significant is that Excel and VB works with angles expressed in radians, not as decimal values of the angles, nor from their initial "plain English" representation.

Consider this situation:

You have a Latitude represented as 10 27' 36" S (10 degrees, 27 minutes 36 seconds South)

You need to get that into radians, and there is not a direct way to do it, before we can get it to radians it has to be converted into a decimal representation. We need to see it as: 10.46 which is the decimal equivalent to 10 27' 36" and we need to take into consideration whether it is a North or South latitude, with South latitudes being treated as negative numbers.

Luckily Microsoft provides a couple of handy functions to convert standard angular notations to their decimal equivalent, and back, at this page:

http://support.microsoft.com/kb/213449

Those routines are included at the code section and one of them has a change made by me to permit you to make a regular angle entry as

10~ 27' 36" S instead of 10 27' 36" S

because ~ is directly accessible from the keyboard, while the is not.

Converting to Radians

After converting the standard notation to a decimal value, we still have to convert that to radians and deal with the sign of the radian result.

The routines and formulas here consider negative latitudes to be South latitudes and negative longitudes to be West longitudes. While this may seem unfair to those of us living in the western hemisphere, what can I say other than deal with it!?

A decimal degree value can be converted to radians in several ways in Excel and for this process, a simple function is used that is also included in the code presented later.

The basic formula is

radians = angleAsDecimal x (Pi / 180)

where Pi is 3.14159265358979

The Final Solution

Copy all of the code below and paste it into a regular code module in your workbook. Instructions for placing the code into a regular module are here: Copy Excel VBA Code to a Regular Module

Set up your worksheet to pass the latitudes and longitudes of the start and end points as standard entries, then enter a formula to pass them to function distVincenty().

Given two points with these coordinates:

Point 1:

  • Latitude: 37 57' 3.7203" S
  • Longitude: 144 25' 29.5244" E

Point 2:

  • Latitude: 37 39' 10.1561" S
  • Longitude: 143 55' 35.3839" E

The general format of the function call is:

=distVincenty( Pt1_LatAsDecimal, Pt1_LongAsDecimal, Pt2_LatAsDecimal, Pt2_LongAsDecimal)

A raw formula would look like this [Note the double-double quotes after the seconds entries]. The SignIt() function, provided as part of the code, converts a standard angular entry to signed decimal value.

=distVincenty(SignIt("37 57' 3.7203"" S "), SignIt("144 25' 29.5244"" E"), SignIt("37 39' 10.1561"" S"), SignIt("143 55' 35.3839"" E"))

You can use the ~ symbol instead of the symbol if it makes it easier for you:

=distVincenty(SignIt("37~ 57' 3.7203"" S "), SignIt("144~ 25' 29.5244"" E"), SignIt("37~ 39' 10.1561"" S"), SignIt("143~ 55' 35.3839"" E"))

If the coordinates for Point 1 are in B2 and C2 and the coordinates for Point 2 are in B3 and C3, then it could be entered as

=distVincenty(SignIt(B2), SignIt(C2), SignIt(B3), SignIt(C3))

The result for the 2 sample points used above should be 54972.271, and this result is in Meters.

Problems Encountered

The initial code that I began working with generated "formula too complex" errors in two statements. These statements were initially broken into two pieces and then those two separate calculations were "rejoined" in a formula to obtain their final result without the "formula too complex" error.

During the preparation of this document, the package was tested in Excel 2010 64-bit and began returning #VALUE! errors for all input values. Investigation determined that a portion of the already split formula to determine the deltaSigma value was generating an overflow error. This error did not occur in the 32-bit version of Excel 2010, nor in Excel 2003 (a 32-bit application).

The offending line of code was once again broken down into smaller pieces that were eventually rejoined in a mathematically correct process that resulted in the proper values being determined without resorting to any alteration of the original algorithm at all.

These sections are noted in the comments in the code for Function distVincenty() below.

For any questions regarding all of this, contact Jerry Latham at HelpFrom@JLathamSite.com.

The Longitude Latitude Code

Now, at last, the code:

'*************************************************************
Private Const PI = 3.14159265358979
Private Const EPSILON As Double = 0.000000000001

Public Function distVincenty(ByVal lat1 As Double, ByVal lon1 As Double, _
    ByVal lat2 As Double, ByVal lon2 As Double) As Double
'INPUTS: Latitude and Longitude of initial and 
'           destination points in decimal format.
'OUTPUT: Distance between the two points in Meters.
'
'=================================================================================
' Calculate geodesic distance (in m) between two points specified by
' latitude/longitude (in numeric [decimal] degrees)
' using Vincenty inverse formula for ellipsoids
'=================================================================================
' Code has been ported by lost_species from www.aliencoffee.co.uk to VBA
' from javascript published at:
' http://www.movable-type.co.uk/scripts/latlong-vincenty.html
' * from: Vincenty inverse formula - T Vincenty, "Direct and Inverse Solutions
' *       of Geodesics on the Ellipsoid with application
' *       of nested equations", Survey Review, vol XXII no 176, 1975
' *       http://www.ngs.noaa.gov/PUBS_LIB/inverse.pdf
'Additional Reference: http://en.wikipedia.org/wiki/Vincenty%27s_formulae
'=================================================================================
' Copyright lost_species 2008 LGPL 
' http://www.fsf.org/licensing/licenses/lgpl.html
'=================================================================================
' Code modifications to prevent "Formula Too Complex" errors
' in Excel (2010) VBA implementation
' provided by Jerry Latham, Microsoft MVP Excel Group, 2005-2011
' July 23 2011
'=================================================================================

  Dim low_a As Double
  Dim low_b As Double
  Dim f As Double
  Dim L As Double
  Dim U1 As Double
  Dim U2 As Double
  Dim sinU1 As Double
  Dim sinU2 As Double
  Dim cosU1 As Double
  Dim cosU2 As Double
  Dim lambda As Double
  Dim lambdaP As Double
  Dim iterLimit As Integer
  Dim sinLambda As Double
  Dim cosLambda As Double
  Dim sinSigma As Double
  Dim cosSigma As Double
  Dim sigma As Double
  Dim sinAlpha As Double
  Dim cosSqAlpha As Double
  Dim cos2SigmaM As Double
  Dim C As Double
  Dim uSq As Double
  Dim upper_A As Double
  Dim upper_B As Double
  Dim deltaSigma As Double
  Dim s As Double ' final result, will be returned rounded to 3 decimals (mm).
'added by JLatham to break up "Too Complex" formulas
'into pieces to properly calculate those formulas as noted below
'and to prevent overflow errors when using
'Excel 2010 x64 on Windows 7 x64 systems
  Dim P1 As Double ' used to calculate a portion of a complex formula
  Dim P2 As Double ' used to calculate a portion of a complex formula
  Dim P3 As Double ' used to calculate a portion of a complex formula

'See http://en.wikipedia.org/wiki/World_Geodetic_System
'for information on various Ellipsoid parameters for other standards.
'low_a and low_b in meters
' === GRS-80 ===
' low_a = 6378137
' low_b = 6356752.314245
' f = 1 / 298.257223563
'
' === Airy 1830 ===  Reported best accuracy for England and Northern Europe.
' low_a = 6377563.396
' low_b = 6356256.910
' f = 1 / 299.3249646
'
' === International 1924 ===
' low_a = 6378388
' low_b = 6356911.946
' f = 1 / 297
'
' === Clarke Model 1880 ===
' low_a = 6378249.145
' low_b = 6356514.86955
' f = 1 / 293.465
'
' === GRS-67 ===
' low_a = 6378160
' low_b = 6356774.719
' f = 1 / 298.247167

'=== WGS-84 Ellipsoid Parameters ===
  low_a = 6378137       ' +/- 2m
  low_b = 6356752.3142
  f = 1 / 298.257223563
'====================================
  L = toRad(lon2 - lon1)
  U1 = Atn((1 - f) * Tan(toRad(lat1)))
  U2 = Atn((1 - f) * Tan(toRad(lat2)))
  sinU1 = Sin(U1)
  cosU1 = Cos(U1)
  sinU2 = Sin(U2)
  cosU2 = Cos(U2)

  lambda = L
  lambdaP = 2 * PI
  iterLimit = 100 ' can be set as low as 20 if desired.

  While (Abs(lambda - lambdaP) > EPSILON) And (iterLimit > 0)
    iterLimit = iterLimit - 1

    sinLambda = Sin(lambda)
    cosLambda = Cos(lambda)
    sinSigma = Sqr(((cosU2 * sinLambda) ^ 2) + _
        ((cosU1 * sinU2 - sinU1 * cosU2 * cosLambda) ^ 2))
    If sinSigma = 0 Then
      distVincenty = 0  'co-incident points
      Exit Function
    End If
    cosSigma = sinU1 * sinU2 + cosU1 * cosU2 * cosLambda
    sigma = Atan2(cosSigma, sinSigma)
    sinAlpha = cosU1 * cosU2 * sinLambda / sinSigma
    cosSqAlpha = 1 - sinAlpha * sinAlpha

    If cosSqAlpha = 0 Then 'check for a divide by zero
      cos2SigmaM = 0 '2 points on the equator
    Else
      cos2SigmaM = cosSigma - 2 * sinU1 * sinU2 / cosSqAlpha
    End If

    C = f / 16 * cosSqAlpha * (4 + f * (4 - 3 * cosSqAlpha))
    lambdaP = lambda

'the original calculation is "Too Complex" for Excel VBA to deal with
'so it is broken into segments to calculate without that issue
'the original implementation to calculate lambda
'lambda = L + (1 - C) * f * sinAlpha * _
  (sigma + C * sinSigma * (cos2SigmaM + C * cosSigma * _
  (-1 + 2 * (cos2SigmaM ^ 2))))
      'calculate portions
    P1 = -1 + 2 * (cos2SigmaM ^ 2)
    P2 = (sigma + C * sinSigma * (cos2SigmaM + C * cosSigma * P1))
    'complete the calculation
    lambda = L + (1 - C) * f * sinAlpha * P2

  Wend

  If iterLimit < 1 Then
    MsgBox "iteration limit has been reached, something didn't work."
    Exit Function
  End If

  uSq = cosSqAlpha * (low_a ^ 2 - low_b ^ 2) / (low_b ^ 2)

'the original calculation is "Too Complex" for Excel VBA to deal with
'so it is broken into segments to calculate without that issue
  'the original implementation to calculate upper_A
  'upper_A = 1 + uSq / 16384 * (4096 + uSq * _
    (-768 + uSq * (320 - 175 * uSq)))
  'calculate one piece of the equation
  P1 = (4096 + uSq * (-768 + uSq * (320 - 175 * uSq)))
  'complete the calculation
  upper_A = 1 + uSq / 16384 * P1

  'oddly enough, upper_B calculates without any issues - JLatham
  upper_B = uSq / 1024 * (256 + uSq * (-128 + uSq * (74 - 47 * uSq)))

'the original calculation is "Too Complex" for Excel VBA to deal with
'so it is broken into segments to calculate without that issue
  'the original implementation to calculate deltaSigma
  'deltaSigma = upper_B * sinSigma * (cos2SigmaM + upper_B / 4 * _
    (cosSigma * (-1 + 2 * cos2SigmaM ^ 2) _
      - upper_B / 6 * cos2SigmaM * (-3 + 4 * sinSigma ^ 2) * _
        (-3 + 4 * cos2SigmaM ^ 2)))
  'calculate pieces of the deltaSigma formula
  'broken into 3 pieces to prevent overflow error that may occur in
  'Excel 2010 64-bit version.
  P1 = (-3 + 4 * sinSigma ^ 2) * (-3 + 4 * cos2SigmaM ^ 2)
  P2 = upper_B * sinSigma
  P3 = (cos2SigmaM + upper_B / 4 * (cosSigma * (-1 + 2 * cos2SigmaM ^ 2) _
    - upper_B / 6 * cos2SigmaM * P1))
  'complete the deltaSigma calculation
  deltaSigma = P2 * P3

  'calculate the distance
  s = low_b * upper_A * (sigma - deltaSigma)
  'round distance to millimeters
  distVincenty = Round(s, 3)

End Function

Function SignIt(Degree_Dec As String) As Double
'Input:   a string representation of a lat or long in the
'         format of 10 27' 36" S/N  or 10~ 27' 36" E/W
'OUTPUT:  signed decimal value ready to convert to radians
'
  Dim decimalValue As Double
  Dim tempString As String
  tempString = UCase(Trim(Degree_Dec))
  decimalValue = Convert_Decimal(tempString)
  If Right(tempString, 1) = "S" Or Right(tempString, 1) = "W" Then
    decimalValue = decimalValue * -1
  End If
  SignIt = decimalValue
End Function

Function Convert_Degree(Decimal_Deg) As Variant
'source: http://support.microsoft.com/kb/213449
'
'converts a decimal degree representation to deg min sec
'as 10.46 returns 10 27' 36"
'
  Dim degrees As Variant
  Dim minutes As Variant
  Dim seconds As Variant
  With Application
     'Set degree to Integer of Argument Passed
     degrees = Int(Decimal_Deg)
     'Set minutes to 60 times the number to the right
     'of the decimal for the variable Decimal_Deg
     minutes = (Decimal_Deg - degrees) * 60
     'Set seconds to 60 times the number to the right of the
     'decimal for the variable Minute
     seconds = Format(((minutes - Int(minutes)) * 60), "0")
     'Returns the Result of degree conversion
    '(for example, 10.46 = 10 27' 36")
     Convert_Degree = " " & degrees & " " & Int(minutes) & "' " _
         & seconds + Chr(34)
  End With
End Function

Function Convert_Decimal(Degree_Deg As String) As Double
'source: http://support.microsoft.com/kb/213449
   ' Declare the variables to be double precision floating-point.
   ' Converts text angular entry to decimal equivalent, as:
   ' 10 27' 36" returns 10.46
   ' alternative to  is permitted: Use ~ instead, as:
   ' 10~ 27' 36" also returns 10.46
   Dim degrees As Double
   Dim minutes As Double
   Dim seconds As Double
   '
   'modification by JLatham
   'allow the user to use the ~ symbol instead of  to denote degrees
   'since ~ is available from the keyboard and  has to be entered
   'through [Alt] [0] [1] [7] [6] on the number pad.
   Degree_Deg = Replace(Degree_Deg, "~", "")

   ' Set degree to value before "" of Argument Passed.
   degrees = Val(Left(Degree_Deg, InStr(1, Degree_Deg, "") - 1))
   ' Set minutes to the value between the "" and the "'"
   ' of the text string for the variable Degree_Deg divided by
   ' 60. The Val function converts the text string to a number.
   minutes = Val(Mid(Degree_Deg, InStr(1, Degree_Deg, "") + 2, _
             InStr(1, Degree_Deg, "'") - InStr(1, Degree_Deg, "") - 2)) / 60
   ' Set seconds to the number to the right of "'" that is
   ' converted to a value and then divided by 3600.
   seconds = Val(Mid(Degree_Deg, InStr(1, Degree_Deg, "'") + _
           2, Len(Degree_Deg) - InStr(1, Degree_Deg, "'") - 2)) / 3600
   Convert_Decimal = degrees + minutes + seconds
End Function

Private Function toRad(ByVal degrees As Double) As Double
    toRad = degrees * (PI / 180)
End Function

Private Function Atan2(ByVal X As Double, ByVal Y As Double) As Double
 ' code nicked from:
 ' http://en.wikibooks.org/wiki/Programming:Visual_Basic_Classic
 '  /Simple_Arithmetic#Trigonometrical_Functions
 ' If you re-use this watch out: the x and y have been reversed from typical use.
    If Y > 0 Then
        If X >= Y Then
            Atan2 = Atn(Y / X)
        ElseIf X <= -Y Then
            Atan2 = Atn(Y / X) + PI
        Else
        Atan2 = PI / 2 - Atn(X / Y)
    End If
        Else
            If X >= -Y Then
            Atan2 = Atn(Y / X)
        ElseIf X <= Y Then
            Atan2 = Atn(Y / X) - PI
        Else
            Atan2 = -Atn(X / Y) - PI / 2
        End If
    End If
End Function
'=====================================================

Excel Distance Calculation Workbook

To see the code, and test the formulas, you can download the Excel Distance Calculation sample workbook. The file is in Excel 2003 format, and is zipped. There are macros in the workbook, so enable the macros if you want to test the code.

Excel Function Tutorials

Excel Sum Functions -- Sum Cells  
Excel VLOOKUP Function  
Excel INDEX / MATCH Functions  
Excel Count Functions -- Count Cells  
Excel INDIRECT Function  

Excel Video Tutorials for Functions

 

 

 

Privacy Policy

 

Contextures Inc., Copyright 2014
All rights reserved.

 

Last updated: April 13, 2014 10:13 AM