I recently read of an interesting property dispute:
Sailor’s Cove is a relaxing waterfront community… with cobblestone streets and New England– style homes bordering the dock. Quaint boutiques line the rocky coast, as [residents] spend their days watching sailing events, yachting, or perusing the local bookshop. Sailor’s Cove is comprised of 21 private islands developed by Isabella Bentham, Tasha Kostolany, and Patrick Leavitt… In 2008 a dispute erupted between Leavitt and the other two “developers” when he attempted to assert ownership over the community and its assets… The parties settled out of court (Le, 2013).
This case doesn’t sound particularly unusual, and it isn’t, except for one thing – Sailor’s Cove exists only in the virtual world of Second Life. And it is one of many cases in the growing practice of virtual law, where lawyers litigate issues arising in virtual worlds, or that carry over from a virtual world to the real world.
Virtual and Augmented Reality
I recently wrote several articles about the coming boom in virtual and augmented reality. Occurring with that boom, I believe, will be a simultaneous increase in legal action resulting from our activities in virtual worlds as well as the technologies that are used to create virtual and augmented reality.Patent Litigation Coming to Virtual Reality
Recent intellectual property cases over mobile phone technology, such as the Apple v. Samsung case are an indicator of what’s coming in virtual and augmented reality, as companies maneuver to be a dominate provider of these important and growing technologies. The stakes were high with mobile phones, and they are likely to be as high or higher with augmented and virtual reality. Facebook, with its $2,000,000,000 purchase of Oculus, is already betting big on virtual reality. When the stakes are so high, litigation is certain to follow.
Patent Wars Ahead
In fact, virtual reality “patent wars” may be even bigger than the mobile phone patent wars, because people have been conceptualizing and patenting virtual and augmented reality technologies for years while the technologies have been largely dormant in the market. As adaptation quickly ramps up, which is now beginning to happen, holders of existing patents will be incentivized to take action to protect their intellectual property. Technology developers will also be increasingly incentivized to obtain new patents, under the “first to file” policy that came into effect in 2013.
This litigation and the need for expert witnesses will take many forms:
- Disputes such as Sailor’s Cove, where consequences of activities occurring in virtual worlds are litigated in the real world.
- Intellectual property and patent issues over the various technologies that are used to create virtual and augmented reality.
- Intellectual property disputes arising over activities in virtual worlds, such as the use of a real-world trademark in a virtual world.
- Product liability issues arising over use of virtual and augmented reality, such as a headset-wearing
user injuring him or herself.
- Privacy issues.
This article will take a closer look at the first type of dispute.
My Virtual Reality Expert Witness Experience
I have already been involved as an expert witness in several lawsuits in the virtual reality space (ARCzar v. Nintendo and Glasses.com v. Ditto.) My specialty is human factors and advanced user interfaces and these lawsuits typically have a user interface component – by definition, without the user interface there is no virtual or augmented world.
Virtual Worlds, Real Lawsuits
Some law firms specialize in litigating virtual cases, and many have set up storefronts in virtual worlds. There is even a Second Life Bar Association. These lawyers are adapting to practice in a virtual world; for example, because privacy cannot be guaranteed in virtual worlds, all attorney/client communications take place in the real world. They also need to be conscious of jurisdictional issues, as people who interact in virtual worlds may physically reside anywhere in the real world (Le, 2013).
Virtual Animal Breeders
This case involving virtual animal breeders provides an example of what is being litigated, in this case between two parties in the virtual world (Le, 2013):
In 2010 Ozimals, Inc., a virtual rabbit breeder, filed a DMCA takedown notice against Amaretto Ranch Breedables, LLC, which creates virtual horses. Both make and sell their own virtual food needed to keep the virtual animals alive. Ozimals contended that Amaretto copied the functionality of its virtual bunnies and asked that Linden Lab [the creator of Second Life] take down the infringing content. Amaretto filed a counterclaim, arguing that Ozimals misused the DMCA to decrease competition by having content removed before actual infringement was established. If Linden Lab followed through on the takedown notice, the company’s viability would be threatened, essentially eliminating Ozimals’ competition… In November 2012, [the judge] issued a summary judgment in the case dismissing in part Ozimals’ counterclaims, stating that the company does not have copyright ownership of the code that controlled the functionality of the virtual rabbits.
Virtual Characters
Disputes also arise between a virtual character and creators of the virtual world. One issue is that participants in virtual worlds often invest extensive time, effort, and money into creating virtual characters and possessions, which may be lost if their account is canceled (Le, 2013):
In Evans v. Linden Research, Inc. the plaintiffs fought for their virtual property rights, arguing that by suspending their accounts, Linden Lab deprived them of their virtual goods. The company asserted that, according to its terms of service, the plaintiffs only had rights to the intellectual property of goods created in- world, but not to the goods themselves. So when users violated the terms of service, Linden Lab said it was within its rights to cancel the plaintiffs’ membership to Second Life.
Virtual Items
The value of virtual items in the real world is also likely to become an issue. Virtual worlds like Second Life have economies where goods are traded, and those economies are linked to real world economies. An example of this is The Virtual World Exchange, where Second Life Linden dollars can be traded for US dollars or Euros. Black markets also exist for selling virtual goods outside of their virtual world. These black markets are often contrary to the virtual world’s terms of service, but do arise outside of the virtual world.
Virtual items can even be stolen in the real world, as in the case of a Norwegian boy mugged and forced to give up a virtual amulet and mask from a role-playing game called Rune-Scape (Le, 2013). One teen’s attorney appealed to the country’s Supreme Court, arguing that the items were “neither tangible nor material and, unlike for example, electricity, had no economic value.” The court rejected that notion, stating that the amulet and mask had “intrinsic” value to the boy because he worked hard to earn them. While they were not “material goods,” the boy was able to lose possession of these items to the defendants. In upholding the theft conviction, the court wrote that it was of the opinion that these virtual goods are to be included in the concept of “goods.” It is truly a brave new world.
Intellectual Property in the Real World
According to an analysis conducted in 2014 of publicly available data provided by the U.S.P.T.O. (Greenbaum, 2014a), virtual reality patents began to appear in 1992. Since many virtual reality (VR) patents are more than 10 years old, some may be unenforceable due to unpaid maintenance fees; however, there are also many valid patents that are waiting to be infringed upon.
VR Patents
VR patents are held by various companies, with IBM, Canon, Sony, Phillips, Samsung, and Microsoft accounting for approximately 65% of issued patents. Nintendo, Sony, Microsoft, Konami, and Namco hold the most VR gaming patents. Patent transactions are also occurring, with Geomagic, Avaya, and Imersion Corp. as the top buyers of VR patents.
In Virtual Reality: Patent Landscape Analysis, LexInnova (2015) identifies Sony, Samsung, and Microsoft as leaders, with a large patent portfolio and frequent patent filing activity. Most patent filings have occurred in the United States, followed by South Korea and China.
What is being Patented?
What is being patented? TechRepublic published a list comprising “10 Cool Virtual Reality Patents from the Past 25 Years,” showing the breadth of these patents:
- Optical system for virtual reality helmet
- Data-suit for real-time computer animation and virtual reality applications
- Visual field perimetry using virtual reality glasses
- Virtual reality generator for use with financial information
- Virtual reality system for treating patients with anxiety disorders
- Virtual reality headset (granted to Oculus Rift)
- Virtual reality GTAW and pipe welding simulator and setup (a VR welding simulator, with the welding mask acting as the headset)
- Virtual reality glove system with fabric conductors
- Virtual reality interactivity system and method
- Scalable virtual world chat client-server system
Patent Litigation
What is being litigated? Some patent issues arising in virtual and augmented reality will be familiar to anyone who follows technology law. For example, Oculus has been sued by game publisher ZeniMax Media alleges that Oculus poached ZeniMax employees to misappropriate trade secrets. As of this writing, the case is likely to go to trial.
User Interface
User interface patents will continue to be a focal point, similar to what has occurred in the smartphone patent wars. Apple v. Samsung provides a good model of what to expect and the ramifications and repercussions involved. Apple viewed Samsung’s perceived infringement of their iOS user interface as a serious issue, because they view their user experience as key to their success in the growing mobile market. Similarly, if a dominant user interface emerges in virtual or augmented reality, patent holders will fight to protect it, with the stakes getting higher as acceptance of virtual and augmented reality grows. Precedents established for current software interfaces are likely to be applied.
Glasses.com v. Ditto
In Glasses.com v. Ditto, a case in which I was involved as an expert witness, the issue was infringement over an augmented reality technology for trying on glasses at home. The technology provided a virtual fitting room, where people could see how they would look wearing many different glasses. In an interesting arrangement, Ditto partnered with patent consultant IPNav who, in return for covering Ditto’s legal costs, will receive equity in Ditto if they win the case (Wohlsen, 2013).
“Patent trolling” will continue to be a factor as well. In a series of well-known cases, non-practicing entity Lennon Image Technologies filed against thirteen retailers for patent infringement (including Ditto, in a separate case), regarding the retailers’ use of augmented reality to allow customers to virtually try on clothes.
Nintendo
I also was an expert witness working for Nintendo on the ARCzar v. Nintendo case, where the non- practicing entity ARCzar sued Nintendo for a function Nintendo used to provide an augmented reality experience. This case settled. ARCzar’s ‘936 patent apparently claimed that they own the entire field of augmented reality, covering among other things:
A method of displaying a graphical user interface in a computer vision system comprising the steps: addressing a scene with a computer vision system; forming an image of the scene; generating a graphical user interface; combining the image of the scene with the graphical user interface to form a composite image; and displaying the composite image in a display field.
Character Creation
In another case involving a seemingly generic patent, Recognicorp, LLC, filed a complaint for patent infringement against CCP hf of Reykjavik, Iceland, for infringement for “character creation and customization that allow the selection of a facial feature, incorporation of the facial feature into a composite image, and instructions for modifying the selected facial feature image that has been incorporated into the composite image” (Heard, 2012).
Motion Tracking Technology
Similarly, Impulse Technology, Ltd., has sued Microsoft and eight video game makers over motion tracking technology used in virtual reality that allegedly infringes on seven Impulse Technology patents. The patents are described as covering a “wide variety of games where the movement of a player is tracked in three dimensions … and certain exercise games where the motion of the player is tracked to effect movement of a virtual avatar, and the exertion of the user is monitored, including where the tracking of the player is done by use of a camera” (Coe, 2011).
In Abarta, LLC, v. Sensics, Inc., Alberta claims:
Defendant infringes ‘326 patent by providing customers a reality system designed to provide a 360 degree viewing experience, including tracking movement in the X and Y directions, to display portions of corresponding images. The directional sensors are embedded in the eye displays included with the Defendant’s system, to track the user’s movements that correspond with imagery used in the virtual environment and viewed on the display. The SmartGoggles System detects movement through multiple trackers and displays a portion of the environment as a function of the X and Y directions. (Abarta, LLC, v. Sensics, Inc., 2012)
I have not conducted detailed analyses of these cases (aside from the two above cases that I was involved in) and can’t comment on them in detail, but they are a strong indicator that many fundamental virtual and augmented reality technologies are already considered protected intellectual property and worth fighting for in the courtroom. These cases as well as other patent disputes relating to virtual and augmented reality, will become as prevalent as they already are in smartphones and mobile technology.
Intellectual Property in Virtual and Augmented Worlds
Intellectual property law will increasingly be applied to activities occurring in virtual worlds, raising questions of how real-world copyrights and trademarks will apply. Here’s an overview of some of the issues that may arise and have arisen (many of the examples here were taken from Wassom (2014)).
Some questions are fundamental. For instance, if I build and sell iPhones in Second Life, have I created a counterfeit product? If so, have I harmed the real-world manufacturer? Or have I provided free advertising for the real-world manufacturer? Companies will need to evaluate these issues, and decide how to best respond – ignore the “infringement,” take action against the infringing party, or even get into the game themselves and sell a virtual version of the product themselves, as many manufacturers do. Going forward, they also need to determine their own patent strategy to best protect themselves.
Questions of public display may also arise. If a physical location triggers an image in an augmented reality application, does that constitute a public display of that image? Enforcement may also prove challenging, as virtual worlds are constantly changing and are unique to what one individual views. In this environment, it may not be clear who saw what, from what angle, and when they saw it.
Copyright laws restrict rights of reproduction and alteration. Augmented reality will test these restrictions by providing new ways of creating derivative works and will also push the fair use doctrine. On the other hand, at some point in the future, augmented reality eyewear could conceivably enforce copyright by alerting the viewer of improper display of copyrighted content or even obscuring the content.
Trademark infringement will also become more complex, particularly when an augmented reality application overlays trademarked content on the real world. Trademarking will need to be re-evaluated; for example, whether a logo is used “in commerce” when appearing in a virtual or augmented application. The virtual information may be copyrighted or derived from copyrighted material, and may be viewed in situations where there is no precedent in the physical world. For example, consider an augmented reality application that recognizes a poster and overlays content, changing the way the poster appears to the virtual viewer. Has the poster been altered? There is an iPhone app called The Leak in Your Home Town that does exactly this. The app was inspired by the BP Deepwater Horizon oil spill. When the app recognizes a BP logo, it overlays the logo with a generated image of a pipe spewing oil.
Augmented reality may also change how trademarking is used. Corporate logos could easily trigger augmented content (similar to how QR codes currently trigger content). This could lead to an augmented world saturated with advertising – one satirical video shows a vision of a highly branded world, with logos and advertisements overlaid on all surfaces. However, what if the logo triggers content from a competitor or a third party, like the BP app I mentioned above. To what extent will trademark holders want to secure their branding in the virtual world?
Product Liability
Product liability issues are also likely to arise. Virtual reality systems that track full-body motion can require a lot of space. Wall Street Journal technology reviewer Geoffrey Fowler (2016) reports that the HTC Vive requires “an unobstructed 15-by-15 foot patch” and describes “crawling around on the floor with a Vive headset on, jabbing at the air with an 8-inch wand. Inside my headset, I was floating in the sea, probing an anemone.” Virtual reality users, while immersed in their virtual worlds, are at least somewhat oblivious to the real world around them. The Vive does include safeguards to alert users if they are nearing wall or other obstruction, but I believe we will soon see YouTube “fail” videos of headset-wearing gamers, tripping, falling, stubbing toes, and so on, because they cannot see their surroundings. Liability issues are certain to follow.
Virtual reality itself might present hazards, as it affects us cognitively, and can lead to motion sickness and even dizziness, seizures, or blackouts. Stress reactions are also possible, and become more likely the more realistic and immersive the virtual experience becomes. It is not hard to envision a lifelike simulation triggering already engrained phobias such as claustrophobia or fear of heights. It is also possible to envision players of a violent virtual game experiencing a post-traumatic stress reaction.
Virtual reality may also encourage bad, and possibly criminal, behavior among users. We know that bullying, intimidation, and hateful speech are sadly prevalent in gaming and other virtual forums today. There is no reason to believe such behavior will not continue in virtual reality worlds, where it may become even more serious in the more realistically presented environment. Similarly, the realism of virtual reality games may give new credence to theories that violent virtual experiences (e.g., video games) lead to violence in the real world. These effects may not be evident immediately, so we may see lawsuits years in the future arising from repercussions of today’s and tomorrow’s virtual reality.
In these cases, the hardware manufacturer, the creator of the content, and provider of the content may all be considered liable.
Privacy
Privacy will obviously be a concern as well. One issue is that virtual and augmented reality will contribute further to growing corporate data collection. Oculus has already been hit by allegations that it is collecting too much data (Hern, 2016):
The [Oculus] privacy policy warns about “information automatically collected about you when you use our services”, including “information about your physical movements and dimensions when you use a virtual reality headset” which may be used “to send you promotional messages and content and otherwise market to you.” By submitting User Content through the Services, you grant Oculus a worldwide, irrevocable, perpetual (i.e. lasting forever), non-exclusive, transferable, royalty-free and fully sub licensable (i.e. we can grant this right to others) right to use, copy, display, store, adapt, publicly perform and distribute such User Content in connection with the Services,” the terms state.
Oculus’ relationship with Facebook may make them particularly susceptible to such concerns if information from Rift further adds to Facebook’s vast collection of data about their users.
Other concerns are more Orwellian. Facial recognition is already quite advanced. Our faces, or even our entire bodies, could trigger automatic searches of information about us – information from social media, our address, our credit history, criminal records, etc. A world where an augmented reality eyewear recognizes people on the street and overlays their address, credit history, or criminal record would be frightening indeed.
Moving Forward
Virtual and augmented reality technology are finally coming to the masses and over the next five years will do so in a very big way. I believe that virtual reality will become a daily part of our lives, perhaps in a form we have not seen yet. I also believe that augmented reality is going to become increasingly more prominent as well. Google Glass is, for the moment, unsuccessful, but specialized augmented reality applications are emerging in business and are increasingly available to consumers via a variety of other manufacturers.
We need to be prepared for this. Protecting intellectual property, safeguarding against liability, and being cognizant of privacy issues will be critical in this new world, the rules are changing, and for many, the stakes will be very high.
—————————————————————————————————————————————- Dr. Craig Rosenberg is an entrepreneur, human factors engineer, computer scientist, and expert witness. You can learn more about Dr. Rosenberg and his expert witness consulting business at www.ui.expert
References
Abarta, LLC, v. Sensics, Inc., Case 2:12-cv-00681 (United States District Court for the Eastern District of
Texas, Marshall Division October 23, 2012).
Coe, E. (2011, July 5). Microsoft, Others Sued Over Motion Tracking IP. Retrieved April 9, 2016, from Law360: http://www.law360.com/articles/255776/microsoft-others-sued-over-motion-tracking- ip
Fowler, G. (2016, April 5). HTC Vive Review: Oculus Rift’s Rival Makes VR Bigger. Retrieved April 7, 2016, from The Wall Street Journal: http://www.wsj.com/articles/htc-vive-review-oculus-rifts-rival- makes-vr-bigger-1459865158
Greenbaum, E. (2014a, January 23). A Virtual reality Patent Landscape Analysis. Retrieved March 23,
2016, from Greenbaum Patent: https://greenbaumpatent.wordpress.com/2014/01/23/a- virtual-reality-patent-landscape-analysis/
Greenbaum, E. (2014b, February 19). Oculu vs. Oculus: Complaint. Retrieved March 23, 2016, from Greenbaum Patent Blog: https://greenbaumpatent.wordpress.com/2014/02/19/oculu-vs- oculus-complaint/
Heard, P. (2012, July 16). Recognicorp Files New Patent Infringement Action Against CCP Over Popular Video Game Eve Online. Retrieved April 9, 2016, from Womble Carlyle Blogs: http://georgiaiplit.blogspot.com/2012/07/recognicorp-files-new-patent.html
Hern, A. (2016, April 7). Facebook’s Oculus hit by fears it will spy on users and pass on data. Retrieved April 7, 2016, from The Guardian: https://www.theguardian.com/technology/2016/apr/07/facebook-oculus-virtual-reality-fears- spy-on-users-pass-on-data
Le, T. P. (2013, May). More Than Just a Game. Retrieved April 1, 2016, from Washington Lawyer: https://www.dcbar.org/bar-resources/publications/washington-lawyer/articles/may-2013- virtual-game.cfm
LexInnova. (2015). Virtual Reality: Patent Landscape Analysis. San Jose: LexInnova Technologies. Wassom, B. D. (2014, January/February 23). IP in an Augmented Reality. Retrieved 2016 March, from
Landslide: A Publication of the ABA Section of Intellectual Property Law:
http://www.americanbar.org/publications/landslide/2013-14/january- february/ip_an_augmented_reality.html
Wohlsen, M. (2013, September 10). Patent Trolls Are Killing Startups — Except When They’re Saving Them. Retrieved April 7, 2016, from Wired: http://www.wired.com/2013/09/patent-trolls- versus-startups/
