Posterous dies on 4/30/2013 so I've moved all my content and switched to Blogger.

Sunday, August 26, 2012

How to Bulk Download from Garmin Connect Using Ruby Script disconnect.rb

My morning ride just got rained out:-(  I had a fresh installed Unbuntu 12.04 lappy here, so I thought I'd show you how to scape all your rides from Garmin Connect .  

Maybe you want to upload them to Strava using their new bulk uploader.  I used to send the gpx and/or tcx files as attachments in <25MB batches to upload@strava.com before they added the new bulk loader.

[[posterous-content:pid___0]]

So let's get started.

  • First you need a computer running Ubuntu (You can make this work wtih Ruby for Windows too, but it's much harder and it takes much longer to setup)
    • If you don't have an Ubuntu computer you can get one right now without screwing up you current system.  Just run Ubuntu from a Live CD or install Ubuntu along side of your Windows OS by simply installing Wubi on your PC
  • Next you need the diconnect.rb script.  Get it here 
    1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 
    #!/usr/bin/env ruby

    ## disconnect

    # ./disconnect.rb -u yourusername [-o /your/path] [-p yourhttpproxyserver]
    #
    # This is a command-line utility for the bulk-downloading of run data from
    # the connect.garmin.com web application, which has lackluster export
    # capabilities.
    #
    # Using this code is a matter of your own relationship with Garmin Connect
    # and their TOS. I can't imagine this being very destructive to their service,
    # and it's just filling in a hole in their existing service.
    #
    # It's built against Garmin Connect as of September 22, 2011. It's a scraper:
    # thus if Garmin changes, this **will break**.
    #
    # This script requires all of the utilities on the line below: install them
    # with rubygems
    %w{rubygems json fileutils mechanize choice highline/import}.map{|x| require x}

    LOGIN_PAGE = "https://connect.garmin.com/signin"
    ACTIVITIES_SEARCH = "http://connect.garmin.com/proxy/activity-search-service-1.0/json/activities?_dc=1220170621856&start=%d"
    GPX_EXPORT = "http://connect.garmin.com/proxy/activity-service-1.1/gpx/activity/%d?full=true"
    KML_EXPORT = "http://connect.garmin.com/proxy/activity-service-1.0/kml/activity/%d?full=true"
    TCX_EXPORT = "http://connect.garmin.com/proxy/activity-service-1.0/tcx/activity/%d?full=true"

    Choice.options do
        header ''
        header 'Specific options:'

        option :user, :required => true do
            short '-u'
            long '--user=USER'
            desc 'connect.garmin.com username. Required'
        end

        option :dir do
            short '-o'
            long '--output-dir=OUTPUT'
            desc 'the directory to save .tcx files'
            default 'tcx'
        end

    	option :proxy do
    		short '-p'
    		long '--http-proxy'
    		desc 'Use HTTP proxy for remote operations'
    	end
    end

    password = ask("Enter your password: " ) { |q| q.echo = "*" }

    def login(agent, user, password)
        agent.get(LOGIN_PAGE) do |page|
    	    puts "Loaded login page."
            login_form = page.form('login')
            login_form['login:loginUsernameField'] = user
            login_form['login:password'] = password

    		puts "Sent login information."
            page = agent.submit(login_form, login_form.buttons.first)
            raise "Login incorrect!" if page.title().match('Sign In')
    		puts "Login successful!"
            return page
    	end
    end

    def download_run(agent, id)
        print "."
        # This downloads TCX files: you can swap out the constant, or add
        # more lines that download the different kinds of exports. I prefer TCX,
        # because despite being a 'private standard,' it includes all data,
        # including heart rate data.
        agent.get(TCX_EXPORT % (id).to_i).save_as(File.join(Choice[:dir], "%d.tcx" % id))
    end

    def activities(agent)
    	start = 0
    	while true
    		j = agent.get(ACTIVITIES_SEARCH % start)
    		search = JSON.parse(j.content)
    		# Got some JSON content, let's see if there are any activities left to process
    		if search['results']['activities']==nil then break end
    		runs = search['results']['activities'].map {|r|
    			# Get each activity id to insert into the download URL
    			r['activity']['activityId']
    		}.map {|id|			    
    			# Download a run.
    			download_run(agent, id) 
    			start=start+1
    		}
    		puts "|"
    	end
    end

    agent = Mechanize.new

    if Choice[:proxy] != nil then agent.set_proxy(Choice[:proxy], 80, nil, nil) end

    # One needs to log in to get access to private runs. Mechanize will store
    # the session data for the API call that cames next.
    home_page = login(agent, Choice[:user], password)

    FileUtils.mkdir_p(Choice[:dir]) if not File.directory?(Choice[:dir])

    puts "Downloading runs..."

    activities(agent)
    , and extract the tar.gz to your HOME directory (or just download the copy I uploaded here http://ubuntuone.com/3smJsPEVdoOyhoKhuyXl0r)
  • Open a Terminal Window - CTRL + ALT + T
    • Copy each of these commands into terminal, press enter, and choose "y" whenever prompted
      • sudo apt-get install ruby
      • sudo apt-get install rubygems
      • sudo gem install --remote json
      • sudo gem install --remote fileutils
        • ignore it if this one fails - it usually fails for me too but doesn't seem to affect things in the end
      • sudo gem install --remote choice
      • sudo gem install --remote highline
      • sudo gem install --remote import
        • this one takes a long time to finish running -- be patient
  • Open Ubuntu Software Center and install Mechanize Ruby Gem library
    • One way to find Software Center is to press the Windows key and type "software" in the search box
    • Once software center is open type "mechanize ruby" into the search
    • Install this one libwww-mechanize-ruby[[posterous-content:pid___1]]
  • Go back to the Terminal WIndow and follow these directions
    • Type "ls" in prompt and enter to see a list of files in your home directory
      • You should see disconnect.rb listed (If you don't then make make sure to copy the file you downloaded into the HOME directory)
    • Type this command an hit enter
      • ruby disconnect.rb -u <YOUR GARMIN USER NAME> o- tcx_files
        • type your user name without the <>
        • this is going to create a Home/tcx_files directory and download all of your ride gpx files there
      • Type your password when prompted and press enter
      • Your runs are downloading.  YAY!
  • Navigate to Home>tcx_files and watch the gpx files magically appear
Posted by at

4 comments:

Subscribe to: Post Comments (Atom)