Next Steps: Smartphone technologies at the leading edge ready to apply to news
By Jordan Wirfs-Brock
Mobile technology offers new ways to create, distribute, and interact with news, content, and people. This section explores the current uses and applications of emerging mobile technologies such as geo-location, augmented reality, voice-to-text, financial transactions, push reminders, social incentives, multi-touch, and gesture. It also discusses the potential unexplored possibilities for using these technologies in mobile news Web sites and applications (apps).
Geo-location and location-based content
Geographic journalism is hardly a new concept. By necessity and design, media started locally — information was distributed by hand and on foot — before evolving into 20th Century mass media forms. While the Internet transcends geographic borders, allowing communication to cross continents, it also offers an unprecedented opportunity for the union of information and place at the hyper-local level. Smartphones allow location-based information to be both abstract and concrete at the same time: They connect the digital with the tangible. Mobile technology offers journalism a chance to return to its location-based roots.
In some respects, the mobile Web provides the capacity for news to become more ‘journalistic’ than ever
Alberto Ibargüen, CEO of the Knight Foundation (which funds and has funded hyper-local and geographic journalism projects such as EveryBlock, now owned by MSNBC.com), has discussed how journalism has lost its sense of place. In a column on Poynter Online, Bill Mitchell describes a 2009 speech by Ibargüen [1]: “He began with a discussion of what’s being lost, and pegged it to geography: ‘For the first time in the history of the republic, the delivery of news and information is not happening in the same space as democracy.’”
Unless somebody can devise a sustainable geographic model for journalism, Ibargüen argued, the United States needs to figure out “how to structure democracy in a different way not rooted in geography.”
How that might happen, he acknowledged, he has no idea.
In some respects, the mobile Web provides the capacity for news to become more “journalistic” than ever: People will have a greater ability to connect place and information to their personal lives. Location-based services for smartphones saw an explosion in 2009. The research firm Gartner reported that mobile location-based services users and revenues worldwide more than doubled from 2008 to 2009, from 41 million to 95.7 million and $998.3 million to $2.2 billion, respectively [2]. This boom is a result of the increasing number of phone handsets with GPS capabilities and changes in phone-use habits. Users can now find news, movie times, directions, restaurants, friends, business inventories, and other information about their immediate surroundings.
Many location-based media Web sites, mobile sites, and mobile apps exist. Some use professional content (Geocommons’ news maps); some use crowd-sourced content (Inauguration Report, Flickr, Yelp); and some use a hybrid of the two (Washington Post’s TimeSpace Web site, EveryBlock, Center for Locative Media). TimeSpace places articles and photos from the Post’s news Web site and external sources like Flickr onto a world map and tracks their development over time. It also has ongoing topic-based maps (entertainment, sports, politics, and auto racing) as well as headline-specific maps such as for the BP oil spill in the Gulf of Mexico. GeoCommons has created similar map-based news packages for specific stories.
 Everyblock’s iPhone app allows the user to filter a neighborhood by data type, such as police reports. |
Geographic journalism also is used in “citizen reporting.” Inauguration Report, a joint effort by NPR, CBS News, American University, Twittervision, GeoCommons, and Zerion, documented President Obama’s January 2009 inauguration by visually mapping tweets (i.e., posts to Twitter) in and around Washington, D.C. EveryBlock, a data-driven hyper-local news site targeting select U.S. cities, combines professionally written news articles, public information such as police reports, and user-generated photos (mainly via the Flickr photo-sharing service owned by Yahoo!) displayed on a block-by-block map. Besides its regular Website, Everyblock offers a free iPhone app and a mobile Web site (m.everyblock.com).
Location-based services aren’t limited to content consumption. Users also can create their own geo-tagged content. Twitter introduced geo-location capabilites in Fall 2009 and popular blogging platform WordPress added geo-tagging to its iPhone app in March 2010, allowing users to include location information in mobile blog posts.
Location-based content creation and consumption raises some privacy concerns for users and publishers, both on the PC and mobile platforms, but especially the latter. Some have suggested that the privacy measures outlined in the Electronic Communications Privacy Act, passed by the U.S. Congress in 1986, need to be updated to account for location-based services.
Emerging location-based technologies, such as augmented reality and indoor mapping, continue to expand the realm of possibilities. Geographic content and location-based services present unexplored opportunities for mobile journalism, including data visualization, interactive narratives, crowd-sourced on-location eyewitness news, and new ways to explore content based on location.
Augmented reality
Augmented reality is an emerging technology that allows information previously limited to presentation as Web pages, lists, or maps — public transportation routes and schedules, historic-building information, restaurant reviews, publicly posted photos, etc. — to be presented as a visual layer overlain onto images on a smartphone’s screen as seen with its camera.
There are two types of augmented reality: One uses GPS and a digital compass to generate spherical coordinates with the user at the origin; the other relies on the recognition of visual markers (Hubbard, 2009) [3]. Augmented reality applications experienced a surge in popularity after the release of the Android platform and Motorola’s Droid phone, which included a standard built-in digital compass (as did iPhone models from the 3GS model on). Augmented reality applications include Layar’s mobile-augmented reality browser; Wikitude’s crowd-sourced “world browser” by Mobilizy; Nokia’s Mobile Augmented Reality Applications research project; Yelp’s augmented reality business search; Lonely Planet’s Compass augmented reality travel guides; and TagWhat, a free network and app where iPhone (3GS and beyond) and Android device users can create and share location-based messages and content in mobile augmented reality or on the desktop Web.
Yelp CEO Jeremy Stoppelman talks about the company’s augmented-reality application for the iPhone, which lets users combine Yelp reviews with the physical world, in this ZDNet video. Not yet fully explored: news applications for this powerful mobile technology. |
Because context is vital for journalism in the digital age, augmented reality and journalism are a natural fit
Augmented reality offers considerable potential for journalism and storytelling, but few traditional news providers have adopted the technology in significant ways. A notable exception is Steven Fiener’s Computer Graphics and User Interfaces Lab at Columbia University [4] (“Computer Graphics & User Interfaces Lab,” n.d.), which has worked on creating augmented reality documentaries (Höller, Pavlik, & Feiner, 1999) [5]. Layar has released some content layers that have journalistic applications, such as a map of financial-bailout spending created with Recovery.gov data. Some non-journalism fields, such as tourism [6], crime-scene investigation [7], and the military [8], have explored applications that could be relevant to journalism. Educational applications [9] have revealed benefits that mobile devices have for learning: portability, social interactivity, sensitivity to context, connectivity, and individuality/customization. These features overlap with the ones Pavlik identified as essential to journalism’s future in his book, Journalism and New Media [10]. Pavlik describes the transition to a more contextualized journalism through five elements: a range of communication channels, the use of hypermedia, increased audience involvement, dynamic content, and customization. Augmented reality is a digital embodiment of context. Because context is vital for journalism in the digital age, augmented reality and journalism are a natural fit.
Potential news applications for mobile augmented reality include:
- Multimedia interactive tours as editorial features, which seamlessly integrate information on-location in real time. Such tours could be engaged by news consumers on-site using a smartphone, or on a PC or tablet Web browser viewed from anywhere. Google Earth, timed to align with the United Nations’ COP15 Climate Conference in Copenhagen, released a series of educational climate-change tours that help users visualize the effects of climate change using geographic imagery. This concept could be adapted to mobile augmented reality, where users could be truly immersed in information or news. Augmented reality has the potential to be used to develop innovative methods for storytelling that unite the digital and physical worlds [11].
- On-location aggregated news. Mobile augmented reality apps like Wikitude World Browser allow you to point a smartphone in the direction of a known object (mountain range, historical building, etc.), and the app will identify it. With Wikitude, it takes your physical location, matches the image in the phone’s camera, and pulls information from multiple sources, including Wikipedia and Flickr. News applications could include augmented reality building recognition able to pull up news as the user points a smartphone at a government building. E.g., during a stroll in Denver’s Civic Center Park, pointing the phone at the State Capitol Building to the east could show links to news stories about the legislature and news releases from the governor’s office from the last 24 hours; turning around and pointing at the Denver City & County Building to the west could show city government news and the mayor’s blog entries.
This smartphone view identifying Alcatraz Island in San Francisco Bay was taken using Wikitude World Browser; a news application of this augmented reality technique might include historical news items that appear when the phone’s camera is pointed at a recognizable city landmark, to accompany an article on tourism. - Crowd-sourced eyewitness accounts of news events. Eyewitnesses or attendees of news events using an augmented reality mobile app (e.g., TagWhat) could snap photos or send in brief text or voice accounts, which are accompanied by geo-location data sent by the app. Online news consumers soon will be able to navigate a Web map to the site of a news event, then view the combined eyewitness accounts and multimedia recorded at the event for an immersive news experience fed by the crowd on hand at the scene.
Voice-to-text and text-to-voice
As the rise of the smartphone erases the division between computer and phone, it also erodes the division between voice and text. User habits, which favor SMS (text messaging) to voice calls, at least among the young-adult demographic [12], underscore a preference for textual information. But as keyboards shrink to fit newer, smaller digital devices and state and federal restrictions on texting while driving expand, the market for improved speech recognition and voice-to-text technologies has grown. In November 2008, IBM named “You will talk to the Web … and the Web will talk back” among its five innovative technology predictions for the next five years [13]. While the Web has not yet reached that point, voice search engines (e.g., Bing, Google) and dictation software have become mainstream. Midomi, a Web and mobile audio search engine, finds songs based on a hum or sung command from users or recorded music. Some products, like Nuance‘s Dragon (Dictation and Search), rely solely on automated speech recognition software. Others, like voice-to-text note-taking service Jott (acquired by Nuance), use a combination of automated speech recognition software and human customer service specialists. Voice-to-text applications still have many limitations, including processing speed and accuracy, especially with homonyms, punctuation, and proper nouns, but the technology has improved markedly in recent years. Such technology is a boon to some disabled computer and smartphone users.
A user could listen to a news story that was dictated by a journalist and relayed via text; respond to the story with his or her voice; and post those comments to a blog or Twitter
Commercial voice-to-text services for smartphones are flourishing. CNN reported, “Voice-activated technologies have entered a renaissance of sorts” (March 19, 2010). Popular mobile applications that offer transcriptions of voicemails and turn voice notes into texts include pay services Jott, QuickTate, Dial2Do, and ShoutOut, and free services Google Voice, Dragon Dictate, and TweetCall. YouTube is now capable of auto-captioning for English videos using speech recognition software.
Most people can speak faster than they can type, and read faster than they can speak. Ultimately, voice-to-text technology will make the transition from the fastest method of content creation (voice) to the fastest method of content consumption (text) seamless. This has the potential to help both the creation and consumption of mobile news. Voice-to-text technology could allow journalists to record field notes and interviews, skip the tedious process of transcription, and deliver content to audiences faster. It could allow readers of the news to annotate, comment, and share their reactions to the news hands-free and in real time.
Some services — like Jott, Dial2Do, and Amazon’s Kindle e-reader — have text-to-speech capabilities and allow users to hear the news. Online media consultant Amy Gahran predicts that more and more people will start listening to their news, whether it is created in audio or text. She wrote for Poynter Online: “As text-to-speech technology continues to improve and proliferate, I’d suggest that text news publishers consider how well their online and Kindle content ‘reads’ in the audible sense” [14]. (It should be noted that Gahran’s advice seems to conflict with the results of this report’s survey of college students, which showed little appetite for audio news.)
As this technology evolves, a user could potentially listen to a news story that was dictated by a journalist and relayed via text; respond to the story with his or her voice; and immediately post those comments to a blog or Twitter, or send them via SMS.
Further, those consuming news content on a smartphone could opt to read the text version on the device’s small screen, or listen to an alternative audio version that is either machine-voiced or recorded manually by the journalist for those preferring to listen (for example, while the news consumer is driving).
User-generated content
User-generated content and so-called “citizen journalism” predate the smartphone revolution; however, smartphones make the creation and sharing of text, pictures, video, and audio possible in real-time. Twitter has been used via mobile phone by eyewitnesses providing live breaking accounts of events such as the US Airways Hudson River plane crash in January 2009. Often, mainstream news media pick up those accounts. CNN’s iReport, YouTube’s CitizenTube, and Yahoo!’s You Witness News feature vetted video news content from users. Sites like Yelp and Google’s Near Me Now offer user-generated business reviews, and Trip Advisor and newcomer Tripwolf offer combinations of editorial and user-generated travel reviews and information.
Smartphone user-generated content also can play a role in news after initial publication. This is especially relevant to in-depth news coverage, since smartphone-toting citizens can send in photos, video, audio, or text documenting their experience of the issue to which they’ve just been exposed. A news package about a growing graffiti problem, for instance, could ask smartphone-using readers to take photos of new instances of graffiti — to fill out an interactive map to better show the scope of a city’s graffiti problem, and/or to pass on to municipal authorities so that they can clean it up. Such an assignment also could be made to readers while the reporter is working on the investigation, in order to have more evidence for the published package.
User-generated mobile content also will play a role more and more as augmented reality is put to applications for news. See the earlier section on augmented reality for more on this topic.
Mobile financial transactions
Mobile transactions could increase the opportunities for news entities to generate revenue
Technology that allows smartphone users to transfer money, paying for everything from bills to campaign donations, is widely available. SMS technology has made donating to charities, political organizations, and other groups possible via mobile phone in a matter of seconds. This was used widely after the Haiti earthquake in early 2010 and the 2008 presidential election campaigns, mostly using short codes (shorter-than-normal phone numbers to dial from a mobile phone to donate money, which is added to the user’s phone bill). Mobile apps and mobile Web sites increasingly are allowing users to make purchases using a credit card, Paypal, Amazon, or other account. Some iPhone apps, for example, allow purchasing within the app and the amount is charged to the mobile user’s Apple iTunes account (which every iPhone owner must have).
For news applications, mobile transactions could be used for instantly collecting subscription fees, one-off micropayments, and/or user donations. (In summer 2010, donations cannot be accepted with apps for Apple’s iPhone.) Alternative fincancial models for journalism — such as Spot.us, which relies on crowd-sourced funds to pay content creators to produce specific news stories — could employ mobile transactions to increase the opportunities for generating revenue. News providers also could connect readers to non-profits that are relevant to their content, providing a direct link for reader involvement and financial support for charitable and public-service entities. This opportunity is not without its caveats, however, since some news organizations will want to avoid the appearance of advocacy.
Civic engagement
Smartphones can enable reader involvement that goes beyond financial contributions. The mobile Web offers opportunities to take civic engagement out of offices and homes and into a community’s streets, transit system, parks, and city centers. Applications like DIY Democracy for the iPhone enable users to report civic observations — such as maintenance problems, crimes, or safety hazards — to their city or state governments directly from their smartphones. The Extraordinaries facilitates “micro-volunteering” on the smartphone itself (using a mobile app) during periods when the phone user has some idle time, as well as on a PC or tablet device.
Mobile news applications could utilize the civic engagement potential of apps to connect readers to local volunteer opportunities that are related to news content, or facilitate expression of a reader’s opinion on an issue covered to relevant officials or leaders. Such features would fit well in mobile versions of some in-depth and investigative journalism packages.
 DIY Democracy is a mobile app that encourages civic engagement by connecting users directly to their local government officials. |
 The Extraordinaries invites users to do volunteer work on a smartphone for causes they support, and motivate their social networks to join. |
Social incentives
A growing number of non-news Web sites and mobile applications have implemented social incentives and gaming elements to motivate users to increase their participation. Incentive systems are designed to reinforce and encourage ideal user behaviors, and often are left vague enough that they evolve beyond the initial intended purpose as the application grows. News applications for social incentives via smartphone use are just beginning to be recognized.
Robert Scoble, a prominent blogger who analyzes social media, has outlined nine tips for successful incentive models:
- Serve your users’ narcissism
- Measure behavior and report it
- Add status for behaviors
- Make multiple status reports
- Make undesired behavior seem lame
- Make it easy to share success with others
- Make an API for studying behavior
- Make it easy to bring in other users [from other social networks]
- Give people more “hooks” to addict their friends
Two successful social media sites that utilize these practices (for mobile as well as the desktop Web) are Yelp and Foursquare. Yelp offers an “Elite Squad” program where users gain membership — which includes exclusive social events bridging online and physical communities — by writing reviews that other users deem useful or funny. This incentive system reinforces desired user behaviors, such as transparency (Elite Squad members must use their real names and pictures). Yelp also awards users for being the first to review a business, a measure that ensures coverage of as many businesses as possible. Fourquare, a geography-based mobile application where users can “check in” with their location, uses a blend of incentive models that connects private users and businesses. Individual users can unlock “badges” that reward diverse behaviors, such as visiting multiple locations in one day, returning to the same location, traveling to locations far from home, or discovering new locations. The user who has spent the most time at a given place becomes its “mayor.” Foursquare encourages businesses to offer free drinks, food, or services to mayors and frequent visitors. (See the images below.)
 Location-based app Foursquare offers incentives that connect its users to business owners. |
Media innovators are beginning to apply these concepts to the news. At Stanford University, 2009-10 Knight Journalism Fellows Susanne Rust and John Duncan developed a smartphone app called HearSay, which integrates social sharing and gaming with news consumption. Rust described it this way: “Think FourSquare for news” [15].
If gaming and social incentive strategies are to be built into mobile news Web sites or applications to persuade users to read or view more content, incentive programs could track:
- How many articles the user reads
- The user’s progress through an article
- How many comments the user makes
- How many people the user has shared an article with
- Secondary sharing (i.e., did the person the user shared it with pass the article on?)
Citations
[1] Mitchell, Bill. “Knight CEO Says Journalism Losing its Geographic Roots.” Poynter Online. (Feb. 27, 2009) http://www.poynter.org/column.asp?id=131&aid=159170
[2] Press release. Gartner Research. “Gartner Says Consumer Location-Based Services Market Will More Than Double in 2009.” (July 7, 2009) http://www.gartner.com/it/page.jsp?id=1059812
[3] Hubbard, Sid Gabriel. “Augmented Reality: A Human Interface for Ambient Intelligence.” ReadWriteWeb. (Aug. 12, 2009; Sept. 13, 2009) http://www.readwriteweb.com/archives/augmented_reality_human_interface_for_ambient_intelligence.php
[4] Feiner, S. (n.d.) “Computer Graphics and User Interfaces Lab.” Retrieved from http://graphics.cs.columbia.edu/
[5] Höllerer, T.; Feiner, S.; Pavlik, J. In Proceedings from ISWC ’99: International Symposium on Wearable Computers. San Francisco, CA. “Situated documentaries: embedding multimedia presentations in the real world.” (1999)
[6] Reitmayr, G.; Schmalsteig, D. In Proceedings from Symposium Location Based Services and TeleCartography. Vienna, Austra: Wiley. “Collaborative augmented reality for outdoor navigation and information browsing.” (2004)
[7] Sandvik, K.; Waade, A.M.In G. Agger (Ed.), Working Papers from Crime Fiction and Crime Journalism in Scandinavia 5: 1 – 17. “Crime scenes as augmented reality on screen online and offline.” (2008)
[8] Livingston, M.A.; Brown, D.G.; Julier, S.J.; Schmidt, G.S. In Proceedings from Virtual Media for Military Applications. Neuilly-sur-Seine, France: RTO. “Mobile augmented reality: applications and human factors evaluations.” (2006)
[9] Klopfer, E.; Squire, K. Education Technology Research and Development 56 (2): 203-228. “Environmental Detectives: the development of an augmented reality platform for environmental simulations.” (2007)
[10] Pavlik, J. V. New York: Columbia University Press. “Journalism and New Media.” (2001)
[11] Clark, Krissy. Nieman Reports. “Journalism on the Map: A Case for Location-Aware Storytelling.” (Summer 2010) http://www.nieman.harvard.edu/reportsitem.aspx?id=102425
[12] Dean, Jenny. University of Colorado School of Journalism & Mass Communication. (2010) http://testkitchen.colorado.edu/projects/reports/smartphone/smartphone-survey/
[13] IBM. Press release. “IBM Reveals Five Innovations That Will Change Our Lives in the Next Five Years.” (Nov. 25, 2008) http://www-03.ibm.com/press/us/en/pressrelease/26170.wss
[14] Gahran, Amy. Poynter Online. “How News Outlets Can Benefit from Kindle 2′s Text-to-Speech Function” (March 27, 2009) http://www.poynter.org/column.asp?id=31&aid=160829
[15] Maples, Pam. Knight Journalism Fellowships at Stanford University. “Knight Fellowships 2.0 (Beta).” (Viewed: June 15, 2010) http://knight.stanford.edu/news/2010/knight2.0/
PDF
- Comment help
- Get Livefyre
