The Rashomon Project: Online Multi-Perspective Chronologies
What problem are you intending to solve?
Letting the public obtain a comprehensive understanding of an event from many video accounts.
What is the technological approach, or development roadmap?
The Rashomon Project will develop an open-source, online toolkit that facilitates rapid assembly and display of Multi-Perspective Chronologies. Initial experiments with standard iPhone and Android video show that temporal metadata embedded in the digital files can be used to do a very close alignment. Once they are closely aligned, audio signals from video can be processed to further refine the alignment. Eventually, the tool may be able to align simultaneous streaming video, allowing users to gain a complete perspective of a given event in real time. Next steps include: • Design and develop the software in HTML5 to quickly assemble MPCs from a set of videos and photos, using metadata and audio signals. Develop the project’s back end: toolsets for video transcoding, metadata extraction and analysis, facial obscuring to protect the privacy of participants and bystanders, dynamic time alignment. • Design and develop a website with secure repository for users to upload videos and photos. • Design and develop a reliable, intuitive, and well-documented web-based display and user interface for the public to view and analyze MPCs, and perhaps share insights.
How will end users interact with it, and how will they benefit?
Users will upload video files to a central server for alignment and online publication by the Rashomon team. Because security and protection of activists can be of vital concern, the Rashomon toolkit incorporates face-blurring technology. This tool will allow the public (potentially hundreds of thousands of viewers) to gain a much better understanding of contested events from user-generated photos and video than is currently possible. It will provide citizen journalists with needed tactical technology to organize and publish photos and videos that comprise community coverage of an event. Public understanding will be enhanced and individuals will feel more engaged in their communities if they know their video footage could contribute to greater awareness of how complicated events unfolded.
How will your app leverage the 1Gbps, sliceable and deeply programmable network?
Access to smart gigabit networks enables Rashomon to address use cases with far more simultaneous video streams (for instance: footage from multiple occupy protests happening in multiple cities at the same time or election documentation at various polling places in a national election). In anticipation of this, we are stress testing our current framework to identify and adapt to both CPU-based and network-based bottlenecks. Where network bottlenecks exist, we anticipate implementing HTML5 adaptive streaming (as-of-yet unsupported in browsers) and other solutions to deliver both higher quality (resolution) and quantities (when available) of video streams.
Further application information
Additional supporting information, materials and resources
Take a look at the existing code - project repository
Will your work be beta-ready by the end of the Development Challenge?
Yes, we anticipate launching a demonstration by Human Rights Day, December 10.
How much effort do you expect this work to take?
We plan to announce a prototype of the Rashomon tool by early December 2012, but it will take another 6-9 months of 1.5 full-time staff plus student support to fully develop and automate its capacity and document the process.
Do you need help?
We welcome support and advice from industry experts to develop a reliable, intuitive, and well-documented web-based display and user interface for the public to view and analyze Rashomon’s multi-perspective chronologies (MPC). We will also need support staff to assemble 5-10 examples of MPCs from footage of contested events, such as election-related protests or polling station violations. Assistance to publicize the system and resource widely would be helpful.
If you can help let them know in the comments below.
and team members
Development of Rashomon is led by UC Berkeley professor Ken Goldberg, who teaches and supervises research in Robotics, Automation, and New Media. He joined the UC Berkeley faculty in 1995 and is craigslist Distinguished Professor of New Media. He is a Professor of Industrial Engineering and Operations Research, with secondary appointments in Electrical Engineering/Computer Science and the School of Information. Ken has published over 150 peer-reviewed technical papers on algorithms for robotics, automation, and social information filtering; his inventions have been awarded eight U.S. patents. Ken has worked closely with the U.S. State Department and is Faculty Director of the Data and Democracy Initiative at CITRIS. Ken and his students have designed and developed dozens of Internet interfaces, and the Rashomon team includes computer scientists and experts in video and human computer interfaces. A key developer on the project is Abram “Aphid” Stern, a researcher, artist and web developer. In 2005 he cofounded Metavid.org, an open archive of U.S. legislative video. He has a research appointment at UC Santa Cruz and has published papers on open video and open government. Abram has collaborated with artists, academics and government transparency advocates on a variety of web video projects which focus on enriching the public domain and citizen media with metadata-rich interfaces. Dr. Camille Crittenden directs the CITRIS Data and Democracy Initiative and facilitates communications, fundraising, and partnerships for Rashomon, including with organizations or individuals who may be end users. As Executive Director of the UC Berkeley Human Rights Center from 2006-12, she brings connections to international human rights advocacy organizations and courts that stand to benefit from the Rashomon toolkit. Several of these organizations will advise on its development and possible use cases. For example, WITNESS, a human rights and video advocacy organization (http://witness.org), is working with pro bono lawyers affiliated with the International Bar Association to research the legal and evidentiary value of a multi-perspectival tool for authenticating claims of human rights violations. The Human Rights Center at UC Berkeley is also providing guidance on this topic. Additional members of the project team come from UC Davis and UC Santa Cruz. See http://automation.berkeley.edu/rashomon/ for a list of contributors. Team Captain for purposes of this submission is Cora Basada. The UC Berkeley Team accepts the current Terms and Conditions, with the exception of the “Legal Rights in Submission Clause”.