James O'Neill's Blog

July 7, 2010

Working with the image module for PowerShell; part 3, GPS and other data

Filed under: Photography,Powershell — jamesone111 @ 7:59 am

In Part one I showed how my downloadable PowerShell module can tag photos using related data – like GPS position – which was logged as they were being taken, and in part two I showed how I’d extended the module in James Brundage’s  PowerPack for Windows 7. Now I want to explain the extensions which automate the processes of:

  • Getting the data logged by GPS units and similar devices
  • Reading each image file from the memory card and matching it to an entry in the log made at around the same time
  • Building up the set of EXIF filters filters based on the log entry.

The data and pictures are connected by the time stamp on each, but to connect properly the scripts must cope with any time difference between the camera’s clock and time on the logging device – whether that’s a GPS unit or my wrist mounted scuba computer. A few seconds won’t introduce much error, but the devices might be in different time zones –for example GPS works on Universal time (GMT) – so the offset is often hours, not seconds. My quick and dirty way of making a note of the difference is to photograph whatever is doing the logging (assuming it can display its time). The camera will record the time its own clock was set to in the EXIF “Date and time taken” field and subtracting that from the time displayed on the logger in the picture gives an offset to apply to all data points. The following is the core of a function named Set-Offset which could be seen in part one;

$RefDate = ([datetime]( Read-Host ("Please enter the Date & time " +
                                   "in the reference picture, formatted as" + [char]13 +
                        [Char]10 + "Either MM/DD/yyyy HH:MM:SS ±Z or " +
                                   "dd MMMM yyyy HH:mm:ss ±Z"))  
            ).touniversalTime()

$ReferenceImagePath = Read-Host "Please enter the path to the picture"
if ($ReferenceImagePath -and (test-path $ReferenceImagePath) -and $RefDate) {
     $picTime = (get-exif -image $ReferenceImagePath).dateTaken 
     $Global:offset = ($picTime - $refdate).totalSeconds
}

The real Set-Offset can take –refDate and –ReferenceImagePath parameters so the user doesn’t need to be prompted for them.  Most of code you can see is concerned with getting the user to enter the time (in a format that PowerShell can use) and the path to the file. The only part which uses the image module is
(get-exif -image $ReferenceImagePath).dateTaken
Get-Exif is a command I added, and it returns an object which contains all the interesting EXIF data from the image file. Only the value in the DateTaken property is of interest here; it is used to calculate the number of seconds between the camera time and logger time and the result is stored in a global variable named $offset.

The next step is to read the data and applying the offset to it; depending on the how it was logged the next step will be either:
  $Points = Get-NMEAData   -Path $Logpath -offset $offset
Or
  $Points = Get-GPXData    -Path $Logpath -offset $offset
Or 
  $Points = Get-CSVGPSData -Path $Logpath -offset $offset
Or
  $Points = Get-SuuntoData -Path $Logpath -offset $offset

The last one handles the comma separated data exported from the Suunto Dive Manager program which downloads the data from my dive watch. The other 3 deal with different formats of GPS data, it may be in the form of NMEA sentences (comma separated again) or the CSV format used by Efficasoft GPS utilities on my phone or the XML-based GPX format. (GPS data formats are worth another post of their own). You may need to make slight alterations to these functions to work with your own logger, but they are easy to change.  All of them except Get-GPXdata import from a CSV file – and use a feature which is new in PowerShell V2 to specify the CSV column headings when using the import-csv command.  Get-GPXData uses XML documents looking for a hierarchy which goes <gpx> <trk> <trkseg><trkpt><trkpt><trkpt><trkpt>… All the functions use select-object to remove fields which aren’t needed and insert calculated data (for example converting the native speed in knots from GPS to MPH and KM/H )

After running one of these commands there will be a collection of data points stored in the variable $points. Each data point has a time – adjusted by the offset value, so it is time as the camera would have seen it. The Suunto dive computer points have a Description (the name of the dive site and water temperature) and depth, while the GPS points have Speed (GPS works in knots and the script calculates Miles per Hour and Kilometres per hour); bearing, latitude as Degrees, Minutes, Seconds, North or South, Longitude as Degrees, Minutes, Seconds East or West, Latitude & Longitude in their original form from the logger and Altitude in both meters and Feet (NMEA data needs extra processing to get the attitude data and Get-NMEAdata has a –NoAltitude switch to allow processing to be speeded up if it only Latitude and Longitude are needed )

Armed with a collection of points the next step find the one nearest to the time the picture was taken; a function named Get-NearestPoint does this. Given the time stamped on the photo the function returns the data point logged closest to that time. It isn’t very sophisticated, taking 3 parameters: a time, a time-sorted array of points and the name of field on the data points to check for the time, and working through the points until the point being looked at is further away from the target time than the previous point; the core of the function looks like this:

   $variance = [math]::Abs(($dataPoints[0].$columnName - $MatchingTime).totalseconds)
   $i = 1
   do {
        $v = [math]::Abs(($dataPoints[$i].$columnName - $MatchingTime).totalseconds)
        if ($v -le $variance) {$i ++ ; $variance = $v }
      } while (($v -eq $variance) -and ($i -lt $datapoints.count))
   $datapoints[($i -1)]

In use it looks something like this.

$image = Get-Image        –Path "MyPicture.Jpg"
$dt    = Get-ExifItem     -image $image  -ExifID $ExifIDDateTimeTaken
$point = Get-nearestPoint –Data  $points -Column "DateTime" -MatchingTime $dt

$point contains the data used to set the EXIF properties of the picture, a process which requires a series of Exif filters to be created – and I explained EXIF Filters in Part 2.  As well as data retrieved from a log, there are times when I want to tag a picture manually. For example  I took some photos in London’s Trafalgar Square without a GPS logger that I want to tag with 51°30’30” N, 0° 7’40” W  . To make this easier I created a function named Convert-GPStoEXIFFilter which can be invoked like this:

$filter = Convert-GPStoEXIFFilter 51,30,30 "N" 0,7,40 "W"

If you’re not used to PowerShell I should say that in some places 51,30,30 would be the way to write 3 parameters.  In PowerShell  it is one array parameter with 3 members. (Even old hands at Powershell occasionally get confused and put in a comma which turns two parameters into a single array parameter)  I could have explicitly named the parameters and made it clear that these 3 were an array by writing
 -LatDMS @(51,30,30) -NS "N"

Convert-GPStoEXIFFilter returns a chain of up to 7 EXIF Filters for GPS version, Latitude reference (North or South) Longitude reference (East or West), Altitude reference (above or below Sea Level), the Latitude & Longitude (as degrees, Minutes, Second and Decimals) and Altitude in meters (altitude is optional). If $point holds the data logged at the time the picture was taken Convert-GPStoEXIFFilter can be invoked like this:

$filter = Convert-GPStoEXIFFilter -LatDMS $point.Latdms -NS $point.NS `
                   -LONDMS $point.londms -EW $point.ew -AltM $point.altM

At the end of part 2 I showed the Copy-Image command that handles renaming, rotating, and setting keywords & title EXIF fields and mentioned it could be handed a set of filters. All the parameters that Copy-image uses are available to Copy-GPSImage, which takes the the set of points as well . Internally it performs the $image= , $dt= , $point= and $filter= commands seen above before calling Copy-image with the image, the filter chain and the other parameters it was passed. The full set of parameters for Copy-GPSImage is as follows

Image The image to work on – this can be an image object, a file object or a file name can come from the Pipeline.
Points The array of GPS data points from Get-NMEAdata, get-GPXData or Get-CSVGPSData
Keywords Keywords to go into the EXIF Keyword Tags field
Title Text to go into the EXIF Title field
Rotate If specified, adds whatever rotate filter is indicated by the EXIF Orientation field
NoClobber The conventional PowerShell switch to say “Don’t overwrite the file if it already exists”
Destination The FOLDER to which the file should be saved.
Replace

Two values separated by a comma specifying a replacement in the file NAME

ReturnInfo

If specified returns the point(s) matched with the pictures

So now it is possible to use three commands to geotag the images, the first two get time offset , and get the data points, applying that offset in the process.

set-offset "D:\dcim \100Pentx\IMG43272.JPG" –Verbose
$points= Get-CSVGPSData 'F:\My Documents\My GPS\Track Log\20100425115503.log' ‑offset $offset

and the third gets the files on a memory card and push them into Copy-GPSImage

$photoPoints = Dir E:\dcim –include *.jpg –recurse |  Copy-GpsImage -Points $Points ` 
          -verbose  -DestPath "C:\users\jamesone\pictures\oxford"   ` 
          -Keywords "Oxfordshire"  -replace "IMG","OX-"  -returnInfo

This is much as it appeared in Part 1 although third command has changed slightly.Copy-GPSImage now has a –returnInfo switch which returns the points where a photo was taken; to link the point to the image file(s) which patched it an extra property Paths is added to the points.

I mention this because I wanted to show the functions I put added almost for fun at the end. Out-MapPoint and ConvertTo-GPX got brief mentions in part 1: with the data in $photopoints I can push camera symbols through to a map like this : (note the sort –unique to remove duplicate points, 79 is the camera symbol)
  $photopoints | sort dateTime -Unique | Out-MapPoint -symbol {79}

Alternatively I can create a GPX file which can be imported into MapPoint, Google Earth and lots of other tools. GPX files need to be UTF8 text, PowerShell wants to write output files as Unicode – thwarting it isn’t hard but is ugly. 
  $photopoints | sort dateTime -Unique | convertto-gpx | out-file photoPoints.gpx -Encoding utf8

With the photo points logged it would be nice to show the path I walked but that will have too many points so I wrote Merge-GPSPoint which combines all the points for each minute so I can do 
  Merge-GPSPoints $points | Out-MapPoint
or 
  Merge-GPSPoints $points | convertto-gpx | out-file WalkPoints.gpx -Encoding utf8

One thing I should point out here is that the GPX format which I convert to is a series of Waypoints (i.e places that will be navigated to in future), not track points (places which have been visited in the past). The import routine processes the latter.

The last detail of the module for now is that I also gave it a function to find out where an image was taken, like this

PS  > resolve-imageplace 'C:\users\Jamesone\Pictures\Oxford\OX-43624.JPG'
Summertown, Oxford, Oxford, Oxfordshire, England, United Kingdom, Europe, Earth

That’s not a data error when it says Oxford, Oxford. The Geoplaces web service I use returns

ToponymName : name        fcode Desctiption for fcode
Earth Earth AREA a tract of land without homogeneous character or boundaries
Europe Europe CONT

Continent

United Kingdom of Great Britain and Northern Ireland United Kingdom PCLI Independent political entity
England England ADM1 First-order administrative division (US States, England, Scotland etc)
County of Oxfordshire Oxfordshire ADM2 A sub-division of an ADM1 (Counties in the UK)
Oxford District Oxford ADM3 A sub-division of an ADM2 (District level councils in the UK)
Oxford Oxford PPL Populated Place  (Cities, Towns, Villages)
Summertown Summertown PPLX Section of populated place

I haven’t done much to introduce intelligence into processing this. I used Trafalagar square in one part 2 and this returns  Charing Cross, London, City of Westminster, Greater London, England, United Kingdom, Europe, Earth which is correct but difficult to allow for. To make matters worse all sorts of strange geo-political questions come up as well if you say UK is the country, and England is the topmost Administrative division: English people might well think counties are the tier below parliament adminstratively but since the Scottish parliament and Welsh assembly opened, you might find a different view if you step over the border. Software which works to the American model of displaying the Populated place and First admin Division – for example Seattle, Washington; is easily thrown giving Reading, Berkshire it gives Reading, England.  Those are questions to look at another time

This post originally appeared on my technet blog.

July 5, 2010

Exploring the IMAGE PowerShell Module

Filed under: How to,Photography,Powershell — jamesone111 @ 12:57 pm

In part one of this series I showed the finished version of photo-tagging script I’ve been using. I based my work (which is available for download) on James Brundage’s PSImageTools module for PowerShell which is part of the PowerPack included with the Windows 7 Resource kit (and downloadable independently). In this post I want to show the building blocks that were in the original library provide and the ones I added.
Producing a modified image using this module usually means working to the following pattern:

  • Read an image
  • Create a set of filters
  • Apply the filters to the image
  • Save the modified image

If you are wondering what a filter is, that will become clear in a moment. James B’s original module had these commands.

Get-Image Loads an image from a file
Add-CropFilter Creates a filter to crop the image to a given size
Add-OverlayFilter Creates a filter to an overlay such as a watermark or copyright notice
Add-RotateFlipFilter Creates a filter to rotate the image in multiples of 90 degrees or to mirror it vertically or horizontally
Add-ScaleFilter Creates a filter to resize the image
Set-ImageFilter Applies a set of filters to one or more images
Get-ImageProperty Gets Items of EXIF data from an image
ConvertTo-Bitmap Loads a file, applies a conversion filter to it, and saves it as a BMP
ConvertTo-Jpeg Loads a file, applies a conversion filter to it, and saves it as a JPG
Copy-ImageIntoOrganizedFolder Organizes pictures into folders based on EXIF data

You can see there are 4 kinds of filter with their own commands in the list and each one makes some modification to the image: cropping, scaling, rotating, or adding an overlay. Inside the two ConvertTo commands, a 5th kind of filter, conversion, is used and I added a function to create filters to do that. I made some changes to the existing functions to give better flexibility with how they can be called, and added some further functions, mostly to work with EXIF data embedded in the image file. The full list of functions I added is as follows:

Save-Image Not strictly required but it is a logical command to have at the end of a pipe line, instead of calling a method of the image object
New-ImageFilter Not strictly required either but it makes the syntax of adding filters more logical
New-Overlay Takes text and font information and creates a bitmap with the text in that font
Add-ConversionFilter Creates a conversion filter for JPG, GIF, TIF, BMP or PNG format (as used in ConvertTo-Jpeg / Bitmap without applying it to an image or saving it)
Add-ExifFilter Adds a filter to set EXIF data
Copy-Image Copies one or more images, renaming, rotating and setting title keyword tags in the process.
Get-EXIF Returns an object representing the EXIF data of the image
Get-EXIFItem Returns a single item of EXIF data using its EXIF ID (the common IDs are defined as constants
Get-PentaxMakerNoteProperty Decodes information from the Maker-Note Exif field, I have only implemented this for Pentax data
Get-PentaxExif Similar to Get-Exif but with Maker-Note fields for Pentax

The image below was resized and labelled using these commands.  The first step is to create an image to act as an overlay:  I’m going to a copyright notice in Red red text, in 32 point Arial 

PS> $Overlay = New-overlay -text "© James O'Neill 2008" -size 32 -TypeFace "Arial"  `
                           -color "red" -filename "$Pwd\overLay.jpg" 

I’m using a Click for the 800 pixel high versionpicture I took in 2008: and I could have used a more complex command to build the text from the date taken field in the EXIF data.  Next I’m going to create a chain of filters to:

  • Resize my image to be 800 pixels high (the aspect ratio is preserved by default),
  • Add my overlay
  • Set the EXIF fields for the keyword-tags, title and Copyright information
  • Save the image as a JPEG with a 70/100 quality rating

Despite the multi-line formatting here, this is a single PowerShell command:  $filter = new-Filter | add | add | add...

PS> $filter = new-Imagefilter |  
     Add-ScaleFilter      -passThru -height 800 -width 65535  |
     Add-OverlayFilter    -passThru –top    750 –left  0     –image    $Overlay |
     Add-ExifFilter       -passThru -ExifID $ExifIDKeywords  -typeName "vectorofbyte" -string "Ocean" |
     Add-ExifFilter       -passThru -ExifID $ExifIDTitle     -typeName "vectorofbyte" -string "StingRay"  |
     Add-ExifFilter       -passThru -ExifID $ExifidCopyright
-typeName "String" -value "© James O'Neill 2008" |
     Add-ConversionFilter -passThru –typeName jpg -quality 70

Given a set of filters, a script can get an image,  apply the filters to it and save it. Originally these 3 steps needed 3 commands to be piped together like this
PS> Get-Image   C:\Users\Jamesone\Pictures\IMG_3333.JPG  |
      Set-ImageFilter -filter $filter |
         Save-image -fileName {$_.FullName -replace ".jpg$","-small.jpg"}

I streamlined this first by changing James B’s  Set-ImageFilter so that if it is given something other than an image object, it hands it to Get-Image.  In other words Get-Image X | Set-Image is reduced to Set-Image X (and I made sure X could be path, including one with wild cards or one or more file objects) . After processing I added a -savepath parameter so that set-image –SavePath P is the same as Set-Image | Save-Image P . P can be a path, or script block which becomes a path, or empty to over-write the image. Get an image,  apply the filters to it and save it becomes a singe command.
PS> Set-ImageFilter –Image ".\IMG_3333.JPG" -filter $filter `
                    –SaveName {$_.FullName -replace ".jpg$","-small.jpg"}

The workflow for my photos typically begins with copying files from a memory card, replacing the start of the filename – like the “IMG_” in the example above – with text like “DIVE” (I try to keep the sequential numbers the camera stamps on the pictures as a basis for a unique ID). Next, I rotate any which were shot in portrait format so they display correctly and finally I add descriptive information to the EXIF data: keyword tags like “Ocean” and titles like “Stingray”. So it made sense to create a copy-image function which would handle all of that in one command. The only part of this which hasn’t already appeared is rotation. The Orientation EXIF field contains 8 to show the image has been rotated 90 degrees, 6 indicates 270 degrees of rotation, and 1 to show the image is correctly rotated, so it is a question of read the data, and depending on what we find add filters to rotate and reset the orientation data.

$orient = Get-ExifItem -image $image -ExifID $ExifIDOrientation   
if ($orient -eq 8) {Add-RotateFlipFilter -filter $filter -angle 270
                    Add-exifFilter       -filter $filter -ExifID $ExifIDOrientation`
                                         -value  1       -typeid $ExifUnsignedInteger }   

There is similar code to deal with rotation in the opposite direction, and rotation is just another filter like adding the EXIF data for keywords or title, all job of Copy-Image does is to build a chain of filters to add Title and Keyword tags and rotate the image, determine the full path the new copy should be saved to and invoke Set-ImageFilter. To make it more flexible,  gave Copy-Image the ability to add filters to an existing filter chain: in the part one you could see Copy-GPSImage  which finds the GPS data to apply to a picture and produces a series of filters from it: these filters are passed on to Copy-Image which does the rest. 

The last aspect of Copy-Image to look at is renaming:  -Replace has become one of my favourite PowerShell operators. It takes a regular expression and a block of text, and replaces all instances of expression found in a string with the text. Regular expressions can be complex but “IMG” is perfectly valid so if I have a lot of pictures to name as “OX-” for “Oxford”  I can call the function with a replace parameter of "IMG","OX-" . Inside Copy-Image, the parameter $replace is used with the -replace operator (using PowerShell’s ability  to treat “img”,”ox” as one parameter in two parts).   $savePath is worked out as follows:

if ($replace)   {$SavePath= join-path -Path $Destination `
                     -ChildPath ((Split-Path $image.FullName -Leaf) -Replace $replace)}
else            {$SavePath= join-path -Path $Destination `
                     -ChildPath  (Split-Path $image.FullName -Leaf)  }

As mentioned above I went to some trouble to make sure the functions can accept image objects or names of image files or file objects – because at different times, different ones will suit me. So all of the following are valid ways to copy multiple files from my memory card to the current directory ($pwd), renaming, rotating and applying the keyword tag “oxfordshire”

PS[1]> Copy-Image E:\DCIM\100PENTX\img4422*.jpg -Destination $pwd `
           -Rotate -keywords "oxfordshire" -replace "IMG","OX-"
PS[2]> dir  E:\DCIM\100PENTX\img4422*.jpg | Copy-Image -Destination $pwd `
            -Rotate -keywords "oxfordshire" -replace "IMG","OX-"
PS[3]> get-image  E:\DCIM\100PENTX\img4422*.jpg | Copy-Image -Destination $pwd `
            -Rotate -keywords "oxfordshire"   -replace "IMG","OX-"
PS[4]> $i = get-image  E:\DCIM\100PENTX\img4422*.jpg; Copy-Image $i -Destination $pwd `
            -Rotate -keywords "oxfordshire" -replace "IMG","OX-"
PS[5]> dir  E:\DCIM\100PENTX\img4422*.jpg | get-image |  Copy-Image -Destination $pwd `
            -Rotate -keywords "oxfordshire"  -replace "IMG","OX-"

Of course if I have the GPS data from taking the logger with me on a walk I can use Copy-GPSImage to geotag the files as they are copied, and in the next part I’ll look at how the GPS data is processed.

This post originally appeared on my technet blog.

July 1, 2010

GPS, and other kinds of Picture tagging with PowerShell

Filed under: Photography,Powershell — jamesone111 @ 4:38 pm

imageWell… I have been off for a bit and you can have a read of the previous post for some background on that. During that time I’ve done a lot of walking, taking photos as I go.  Having raved about my HTC Touch pro 2 and its GPS I’ve been using it to GeoTag those photos. Naturally (for me) PowerShell figures in here somewhere. I’ve added to the image module in the Windows Powerpack that James Brundage wrote. It now takes data from a log and applies it to pictures. It supports several log formats and incidental things I found I wanted to do with GPS data. And it is available for download

I log my walks using Efficasoft’s GPS  utilities (on the right). I have a picture of the logger to help with the first of the three PowerShell commands I need to use

  1. Set-Offset works out the time difference between the time on the logger and time on the camera and stores it in $offset
  2. Get-CSVGPSData reads the GPS log from a CSV file, using $offset to adjust the time so it matches the camera. I store the result in $points and a successful import can be verified by looking at $points.count.
  3. Copy-GpsImage copies pictures from a memory card to my computer renaming them, rotating them if need be, and tagging them using the GPS data I stored in $points (if copying many files, it is useful to use -verbose switch to see progress)

Steps 2 and 3 might be repeated to tag more than one set of photos provided the camera clock is a consistent interval away from “GPS time”. Here’s what a session looks like in PowerShell


PS > set-offset "D:\dcim \100Pentx\IMG43272.JPG" –Verbose

Click for a Larger VersionPlease enter the Date & time in the reference picture, formatted as
Either MM/DD/yyyy HH:MM:SS ±Z or dd MMMM yyyy HH:mm:ss ±Z: 04/04/2010 16:02:17 +1
VERBOSE: Loading file D:\dcim \100Pentx\IMG43272.JPG
VERBOSE: OffSet = 3607

PS> $points= Get-CSVGPSData 'F:\My Documents\My GPS\Track Log\20100425115503.log' ‑offset $offset
PS> $points.count
593

PS> Dir E:\dcim –include *.jpg –recurse |
      Copy-GpsImage -Points $Points `
          -Keywords "Oxfordshire" `
          -DestPath "C:\users\jamesone\pictures\oxford" `
          -replace "IMG","OX-" `
          -verbose

VERBOSE: Loading file E:\dcim\100PENTX\IMG43624.JPG
VERBOSE: Checking 593 points for one where DateTime is closest to 04/25/2010 12:36:41
VERBOSE: Point 229 matched with variance of 2 seconds
VERBOSE: Performing operation "Write image" on Target " C:\users\jamesone\pictures\oxford\OX-43624".

In my case this will go through a stack of files, ending
VERBOSE: Performing operation "Write image" on Target " C:\users\jamesone\pictures\oxford\OX-43757"


imageThe picture shows a detail from the Radcliffe Observatory building in Oxford (featured in a recent episode of Lewis) with its GPS co-ordinates visible through File/Properties. I also wrote a little bit of code to push the data from the log though to MapPoint – this ended up as a function Out-MapPoint although I later added a function named convertTo-GPX which does the same job more quickly and also works with programs like GoogleEarth.

$MPApp = New-Object -ComObject "Mappoint.Application"
$MPApp.Visible = $true
$map = $mpapp.ActiveMap
ForEach ($point in $points) {
   $location=$map.GetLocation($point.Lat, $point.lon)
   $Pin =$map.AddPushpin( $location, $point.datetime)
   if ( $point.datetime -lt [datetime]"04/25/2010 12:12:00"){$pin.symbol= 6 }
   else                                                     {$pin.symbol = 7}
}
The details behind this this take some explaining, so this will be the first of several parts: if you would like to know more have a look at the next few posts where I will drill into how it all works, but if you want to dive in and play the code is available for download now.

This post originally appeared on my technet blog.

Blog at WordPress.com.