Ah, the Tremendous Bowl. Or, as I desire to say, the Superb Owl—that oh-so-American Sunday outlined by infinite nachos, high-budget commercials, and memes that can last us half a decade

As an uncoordinated math geek, I can’t say I’ve ever had a lot connection to the “Football” a part of the Tremendous Bowl. That stated, sports activities, information analytics, and machine studying make a powerful trio: {most professional} groups use this know-how in a technique or one other, from monitoring gamers’ strikes to detecting accidents to studying numbers off gamers’ jerseys. And, for the much less athletic of us, machine studying could even have the ability to assist us enhance our personal abilities.

Which is what we’ll try right this moment. On this submit, I’ll present you how you can use machine studying to investigate your efficiency in your sport of selection (for example, I’ll use my tennis serve, however you may simply undertake the method to different video games). We’ll use the Video Intelligence API to trace posture, AutoML Vision to trace tennis balls, and a few math to tie every little thing collectively in Python.

Wish to do this mission for your self? Comply with alongside within the Qwiklab.

I give full credit score for this concept from my fellow Googler Zack Akil, who used the identical method to investigate penalty kicks in soccer (sorry, “football”).

Utilizing machine studying to investigate my tennis serve

To get began, I got down to seize some video information of my tennis serve. I went to a tennis court docket, arrange a tripod, and captured some footage. Then I despatched the clips to my tennis coach good friend, who gave me some suggestions that regarded like this:


These diagrams have been nice as a result of they analyzed key components of my serve that differed from these {of professional} athletes. I made a decision to make use of this to hone in on what my machine studying app would analyze:

  1. Have been my knees bent as I served?
  2. Was my arm straight after I hit the ball?
  3. How briskly did the ball really journey after I hit it? (This one was only for my private curiosity)

Analyzing posture with pose detection

To compute the angle of my knees and arms, I made a decision to make use of pose detection—a machine studying method that analyzes images or movies of people and tries to find their physique components. There are many instruments you should utilize to do pose detection (like TensorFlow.js), however for this mission, I needed to check out the brand new Individual Detection characteristic of the Google Cloud Video Intelligence API. (You may acknowledge this API from my AI-Powered Video Archive, the place I used it to investigate objects, textual content, and speech in my household movies.) The Individual Detection characteristic acknowledges an entire bunch of physique components, facial options, and clothes. From the docs:


To begin, I clipped the video of my tennis serves down to simply the sections the place I used to be serving. Since I solely caught 17 serves on digital camera, this took me a few minute. Subsequent, I uploaded the video to Google Cloud Storage and ran it by means of the Video Intelligence API. In code, that appears like:

 To name the API, you move the situation in Cloud Storage the place your video is saved in addition to a vacation spot in cloud storage the place the Video Intelligence API can write the outcomes.

When the Video Intelligence API completed analyzing my video, I visualized the outcomes utilizing this neat tool constructed by @wbobeirne. It spits out neat visualization movies like this:


Pose detection makes an amazing pre-processing step for coaching machine studying fashions. For instance, I might use the output of the API (the place of my joints over time) as enter options to a second machine studying mannequin that tries to foretell (for instance) whether or not or not I’m serving, or whether or not or not my serve will go over the online. However for now, I needed to do one thing a lot easier: analyze my serve with highschool math!

For starters, I plotted the y place of my left and proper wrists over time:


It’d look messy, however that information really exhibits fairly clearly the lifetime of a serve. The blue line exhibits the place of my left wrist, which peaks as I throw the tennis ball a couple of seconds earlier than I hit it with my racket (the height in the proper wrist, or orange line).

Utilizing this information, I can inform fairly precisely at what cut-off dates I’m throwing the ball and hitting it. I’d prefer to align that with the angle my elbow is making as I hit the ball. To do this, I’ll must convert the output of the Video Intelligence API–uncooked pixel areas–to angles. 

How do you try this? Clearly utilizing the Regulation of Cosines, duh! (Simply kidding, I positively forgot this and needed to look it up. Right here’s a great explanation of the Regulation of Cosines and a few Python code.)

The Regulation of Cosines is the important thing to changing factors in area to angles. In code, that appears one thing like:

Utilizing these formulae, I plotted the angle of my elbow over time:


By aligning the peak of my wrist and the angle of my elbow, I used to be in a position to decide the angle was round 120 levels (not straight!). If my good friend hadn’t advised me what to search for, it might have been good for an app to catch that my arm angle was completely different from professionals and let me know.

I used the identical method to calculate the angles of my knees and shoulders. (You could find all the main points in the code.)

Computing the pace of my serve

Pose detection let me compute the angles of my physique, however I additionally needed to compute the pace of the ball after I hit it with my racket. To do this, I had to have the ability to monitor the tiny, speedy little tennis ball over time.


As you may see right here, the tennis ball was type of exhausting to establish as a result of it was blurry and much away.

I dealt with this the identical manner Zack did in his Football Pier mission: I educated a customized AutoML Imaginative and prescient mannequin.

When you’re not aware of AutoML Vision, it’s a no-code method to construct laptop imaginative and prescient fashions utilizing deep neural networks. The very best half is, you don’t must know something about ML to make use of it.

AutoML Imaginative and prescient enables you to add your personal labeled information (i.e. with labeled tennis balls) and trains a mannequin for you.

Coaching an object detection mannequin with AutoML Imaginative and prescient

To get began, I took a thirty second clip of me serving and break up it into particular person photos I might use as coaching information to a imaginative and prescient mannequin:

ffmpeg -i filename.mp4 -vf fps=10 -ss 00:00:01 -t 00:00:30 tmp/snapshots/%03d.jpg

You’ll be able to run that command from throughout the notebook I offered, or from the command line you probably have ffmpeg put in. It takes an mp4 and creates a bunch of snapshots (right here at fps=20, i.e. 20 frames per second) as jpgs. The -ss flag controls how far into the video the snapshots ought to begin (i.e. begin “seeking” at 1 second) and the flag -t controls what number of seconds needs to be included (30 on this case).

When you’ve bought all of your snapshots created, you may add them to Google Cloud storage with the command:

gsutil mb gs://my_neat_bucket  # create a brand new bucket

gsutil cp tmp/snapshots/* gs://my_neat_bucket/snapshots

Subsequent, navigate to the Google Cloud console and choose Imaginative and prescient from the left hand menu:


Create a brand new AutoML Imaginative and prescient Mannequin and import your photos.


Fast recap: what’s a machine studying classifier? It’s a kind of mannequin that learns how you can label issues from instance. So to coach our personal AutoML Imaginative and prescient mannequin, we’ll want to offer some labeled coaching information for the mannequin to study from.

As soon as your information has been uploaded, it is best to see it within the AutoML Imaginative and prescient “IMAGES” tab:


Right here, you can begin making use of labels. Click on into a picture. Within the enhancing view (beneath), you’ll have the ability to click on and drag slightly bounding field:

1 gif

For my mannequin, I hand-labeled about 300 pictures which took me ~30 minutes. When you’re executed labeling information, it’s only one click on to coach a mannequin with AutoML–simply click on the “Train New Model” button and wait.


When your mannequin is finished coaching, you’ll have the ability to consider its high quality within the “Evaluate” tab beneath.


As you may see, my mannequin was fairly darn correct, with about 96% precision and recall.

This was greater than sufficient to have the ability to monitor the place of the ball in my photos, and due to this fact calculate its pace:

2 gif

When you’ve educated your mannequin, you should utilize the code in this Jupyter notebook to make a cute little video just like the one I plotted above.

You’ll be able to then use this to plot the place of the ball over time, to calculate pace:


Sadly, I spotted too late I’d made a grave mistake right here. What’s pace? Change in distance over time, proper? However as a result of I didn’t really know the space between me, the participant, and the digital camera, I couldn’t compute distance in miles or meters–solely pixels! So I realized I serve the ball at roughly 200 pixels per second. Good.

So there you will have it–some methods you should utilize to construct your personal sports activities machine studying coach app. And for those who do construct your personal sports activities analyzer, let me know!

Related Article

Baking recipes made by AI

In this post, we’ll show you how to build an explainable machine learning model that analyzes baking recipes, and we’ll even use it to co…

Read Article

Leave a Reply

Your email address will not be published. Required fields are marked *