Toyota and Microsoft working on driver gesture recognition (no, not those gestures)

Raise your hand, palm up, and the radio volume in your car goes up. Toyota sees gesture recognition as one way to reduce the complexity of cars. Not for steering and braking, but to deal with the secondary controls such as infotainment, navigation, or your cellphone. So says Jim Lentz, head of Toyota in the US. The goal is to reduce driver distraction.

Toyota’s Board of Awesomeness (seriously) research team is working with Microsoft, a company that has spent years trying to reduce crashes. Their research vehicle is an electric skateboard with a Windows 8 tablet and Kinect motion sensing software (pictured below). In this case, raising or lowering the rider’s hand changes the speed. So, probably, does falling off.

This is all theoretical research right now while Toyota and Lexus soldier ahead in production cars with touchscreens, voice recognition, the Entune/Enform infotainment interface, and Remote Touch, the haptic feedback joystick-like device on some Lexuses that controls the LCD display. “Imagine a dashboard where there are no buttons to push… no screens to tap… and your eyes can remain focused on the road. That’s exactly what Toyota is working on,” Lentz said in a speech at the recent Los Angeles Auto Show.

“This could potentially work in conjunction with voice recognition which sometimes can be hindered by accents or mispronunciations. Hand gestures are pretty universal,” Lentz added. “I’ll wait for a few seconds while you insert your own punch line.”

Separately, Lentz said Toyota in Japan is prototyping the Smart Insect (pictured right), a single-passenger electric vehicle with cameras facing inside and outside the car, gesture and voice recognition, motion sensors, and behavior predictions. For instance: Walk up to the car and it recognizes the driver’s face, blinks the headlamps, and unlocks and opens the doors. Sit down and the car says “Hello” or whatever the driver desires. Think custom ringtones-plus. Gesture recognition and the Smart Insect, Lentz says, “are just a few examples of the many types of mobility automakers are creating for a better tomorrow.”

Source

The Leap: Gesture control like Kinect, but cheaper and much higher resolution

It seems Minority Report-style computer interfaces might arrive a whole lot sooner than we expected: A new USB device, called The Leap, creates an 8-cubic-feet bubble of “interaction space,” which detects your hand gestures down to an accuracy of 0.01 millimeters — about 200 times more accurate than “existing touch-free products and technologies,” such as your smartphone’s touchscreen… or Microsoft Kinect.

Before you read any further, watch the video below. It’s really rather awesome — and apparently the video is footage of a real The Leap unit, rather than a computer rendering (you know a device is serious when the The is part of the product name). You will also notice that it doesn’t only detect hand movements and gestures — you can use objects, such as a pen or chopsticks, or, assuming software support, your favorite pet.

Now, having watched the video, you probably have a few questions. First of all, no, we don’t know what hardware is hidden within the The Leap. Leap Motion (the company behind The Leap) has said absolutely nothing about the tech, other than it’s “unlike anything that currently exists on the market or in academia.” Realistically, the device probably uses some kind of infrared LIDAR (radar, but with light) — or perhaps it’s like a high-definition version of Kinect (which only uses a 640×480 camera, meaning it can’t come close to Leap’s 0.01mm accuracy). On the software side, there’s undoubtedly some magic at work, but again we don’t have any details beyond the fact that it uses “a patented mathematical approach.”

Technical details aside, The Leap is available to pre-order now for $70, and is expected to ship early next year. For now, Leap Motion is actually giving away free units and an SDK to developers — though I suspect there’s a limit on how many Leaps are up for grabs. Once the device gets into the hands of developers, we should have a much better idea of how the technology works.

In practice, I have some doubts about the actual usability of Leap. Personally, I don’t want to hold my arm out in front of me for 8+ hours every day — and I really doubt that interacting with Leap is somehow faster or more productive than a mouse and keyboard. If you want gesture control on a PC, or stylus input, get some kind of Wacom tablet or the Apple Magic Trackpad.

While the video is entirely desktop-oriented, perhaps a more compelling use for Leap could be in the mobile space. With 0.01mm accuracy, it would be easy enough to develop a virtual, gesture-based keyboard. On the other hand, for $70 (cheaper than Kinect!), maybe it’s worth having The Leap on your desk just in case you want to do some lean-back surfing, or other things that don’t require you to be hunched over your mouse and keyboard — or the Leap tech could just be built into the keyboard itself. Anyway, holding your arm out for eight hours might be tiring at first, but it would get easier over time. I can just see it now: The humans of the future will all have massive biceps and pecs.

Read more about Microsoft’s sonar-based gesture detection, and Disney’s Touche, which turns everyday objects into touchscreens

Source

See the future: Augmented reality head-up displays beckon

Picture-in-picture may be coming to your next car courtesy of head-up displays that put more snippets of information in your line of sight. By giving you controlled doses of data projected onto a reflective rectangle just above the steering wheel, automakers say you’ll be safer because your eyes don’t wander about the cockpit looking to the center stack and instrument panel. Imagine an exit or turn arrow that doesn’t just point to the right but has the same angle as the turn you’re approaching and expands as you reach the turn. You’ll interact with the displays by arm gestures, voice input, or traditional dashboard buttons and knobs. Cost will be an issue since current HUDs run more than $1,000.

This month’s Consumer Electronics Show in Las Vegas was a coming out party for head-up displays concepts. They’re also being called augmented reality HUDs or AR windshields, perhaps on account of simpler versions of head-up displays (spelled that way, not heads-up display, pilots are quick to note) have been around in cars for two decades and for a lifetime in fighter planes. Automakers Audi, Kia, and Mercedes-Benz showed HUD concepts. Audio component supplier Pioneer showed HUDs that could be retrofitted to existing cars. All lean heavily on improving the navigation experience.

Mercedes-Benz snagged one of the CES keynotes and chairman Dieter Zetsche talked up the Dynamic & Intuitive Control Experience, or DICE. Dynamic yes, intuitive not yet, but certainly worthy of being called augmented reality. All manner of information appears on the windshield and you control it with hand gestures: Select an item by closing your fingers and pushing at a virtual button (merely poking at the region isn’t enough) or make a swiping motion to save the information or dismiss it. Information you swipe-save goes to the center stack screen, which is also controlled by hand and arm gestures. In-cockpit cameras track your hand movements and gestures.

The Mercedes demo on the show floor took the form of a virtual drive in a simulator through San Francisco with points of interest circled, the idea being you could gesture to get more information. The gesture you’d use in a busy urban area might be a raised middle finger because of the potential information overload: restaurant, bar, jewelry shops, tour bus stops, bridge and tunnel congestion. And that’s even before you wonder who’s providing the POI information and is it there because it’s the best, or because it pays the automaker the best. This is an issue for the whole industry, not just Mercedes, whose motto is The Best or Nothing.

Out on the highway with DICE, knowing what few gas stations and restaurants are at the next two interchanges might be useful: Waffle House in five miles vs. Panera in 20. Mercedes’ AR HUD is definitely a concept rather than near-product and, as with many technologies delivered by Audi-BMW-Mercedes the past decade, the first iterations easily grasped by their PhD engineers and befuddling to potential customers who haven’t passed through MIT. But the possibilities are dazzling. The Mbrace2 telematics system, also shown at CES, is a real product and very near production (spring 2012); it adds mobile apps to the existing suite of crash notification, remote door unlock, and concierge services.

Audi

Unwilling to be outdone, Audi showed an augmented reality HUD with not one but three head-up displays for driver, passenger, and a communal center display for shared information that can be swipe-moved to another display. The passenger could search out POIs and, finding a suitable restaurant or amusement park, swipe the information over to the center display where it becomes the address for driving directions. The navigation display in front of the driver goes beyond the current color HUDs (number of lanes, number of lanes exiting, distance to exit) to make the exit arrow grow in size as you approach the exit. It also puts night vision in the middle of the windshield (pictured below right).

For Audi, the AR HUD is also in the gauzy, distant, maybe future. Who knows how long and often you’ll want to raise your arm to gesture at an imaginary knob when you could press a button and say, “Siri, what good restaurants are along the route?” Plus, Germans long ago disavowed the raised arm as a generally bad thing. The best guess is that this concept would come to market with dual not triple HUDs as a $2,000 option using augmented speech recognition rather than gestures and multiple tracking cameras. Audi’s real-world, real-soon advance out of CES was its announcement it will step up to the Nvidia Tegra 3 quad-core mobile processor (from the Tegra 2) for 2013 Audis with the Audi Connect telematics system. The Tegra 3 when driving will render 3D aerial maps on the navigation system; when the car is off it can operate in low power mode and receive updates (weather, new music, travel info) before your next trip.

Kia

Kia showed a concept HUD with sophisticated map information called the User-Centered Driving (UCD) concept. The HUD marks the proper lane to travel and the one to shift to for a turn, marks lanes you don’t want to be in, and uses a turn/exit arrow that grows and changes shape as you approach and then enter the turn. The prototype showed a pedestrian and car detection feature; every time a pedestrian crossed in front of you (in the demo), a target box appeared around the pedestrian, showing the car knows about the jaywalker. The UCD concept, Kia says, can connect to smartphones and tablets; it also has drowsy driver detection and night vision. As with the Mercedes and Audi concepts, there’s no concrete market plan here.

What real product Kia showed in Vegas was at the Consumer Telematics Show the day before CES opened: Kia UVO 2. UVO stands for Your Voice, it’s a collaboration with Microsoft, and thus is a variation of Ford Sync and the Fiat Blue & Me systems that provide USB-iPod integration and Bluetooth. Kia said it’s targeted as a 2014 product. In the fast-moving world of car tech, that seems a long way off and had some conference attendees wondering if Kia, Microsoft, or both, were having issues making the technology come together. Imagine that: a Microsoft project behind schedule.

Pioneer

Pioneer’s add-on AR HUD solution lets you retrofit your existing car. In Pioneer’s CES demo, the module mounts to the top of the windshield (top photo). It comprises a RGB laser in a pico projector and a reflector extending forward of the projector, looking like an oversized pair of safety goggles. The actual product, due this fall in Japan and next year in the US, might be simpler and connect to your smartphone as your data source, meaning your phone would need a video out feature and, perhaps, a just-the-basics mode, since the purpose of HUDs in the past has been to provide less complexity, not more.

The history of car head-up displays includes small monochrome CRTs embedded in the instrument panel providing a night vision display in 1990s GM cars. But even a small CRT was too big and too hot for all but the largest dashboards. It gave way to LED panels, monochrome, partial color, and now full color. Through all the generations, the way you see the display remains the same: There’s a postcard-size mirrored coating at the base of the windshield in front of the driver. You look at the patch and see both the road ahead and the display. The display text is reversed; the mirror reverses the text a second time so it reads properly. It’s the same as a TV studio teleprompter. The biggest displays today, such as those on Audis and BMWs, are full color and resolve 800×480 pixels, about what you’d find on a good smartphone. That’s enough resolution to show a simplified moving map but so far automakers have stuck with directional arrows, lane-and-exit info, and speed. The next generation shown at CES, is poised to provide even more information.

Source