“Wouldn’t it be great if we had an algorithm that was smart enough to read the world news and anticipate conflict before it happens,” says Jonathan Morgan, Co-Founder and Technical Director for Crisis.Net, a new Application Program Interface (API) developed by Ushahidi. “But that’s not just a limitation of technology,” he explains. 

“There are just too many variables and it’s important to have deep domain expertise of the area, the groups and ethnicities engaged in conflict, and even the topology of the region,” he says. “There’s a massive amount of knowledge that’s required to truly understand how a conflict is unfolding that’s quite specific to a particular region.”

But times have changed since Ushahidi first launched its crisis mapping platform in the violent aftermath of the 2007 elections in Kenya. With the use of social media now widespread, so too has the way in which many media, international organisations, and local NGOs now work. Ushahidi has had to go social.

But consuming already existing data can prove quite a hurdle for many journalists and analysts without additional sources added into the mix. Even if experienced in doing so, they often have to relearn and adapt the same process when they move to a crisis or conflict in another region. And those not used to retrieving and using data have an even steeper learning curve.  

“That’s the real barrier to acting quickly when conflicts or other emergency situations arise,” Morgan concludes. “Some data will already exist while other data is generated in real-time from streaming information sources such as social media, sensors, or even quasi-real-time data that comes out of traditional media, RSS feeds, and stuff like that.”

And that’s where Crisis.Net comes in with its ability to deliver data pulled from a variety of sources in Javascript Object Notation (JSON) or Comma-Separated Values (CSV) formats. The API should make it possible for media organisations and developers alike to build their own applications without the need to spend days locating, identifying, and processing data.

“What we found with Ushahidi was that we had to rely on information directly submitted by users interested in a particular crisis, disaster, or a conflict, but that when we wanted to do any  analysis on data on a particular conflict, in the humanitarian space at least, it was tucked away in tiny corners scattered all over the Internet,” says Morgan.

“There might be a non-profit doing some work in a particular region or there’s collected data stashed in reports sitting on a server somewhere,” he explains. “Or maybe there’s a list of reports in a spreadsheet on Google Drive or a hashtag that we want to follow for information relevant to a crisis, but there’s no good way to collect all of this disparate data in a single place.”

To demonstrate the potential of Crisis.Net, the Facebook pages of Palestinian supporters were sourced to visualise recent attacks on Gaza. Tweets have also been analysed to assess attitudes to the Israeli Defence Force (IDF). Crisis.Net has also created a timeline charting the Islamic State (of Iraq and the Levant) in Northern Iraq based on YouTube, Twitter, and Facebook conversations. 

Partnering with AlchemyAPI, facial recognition technology was used to filter content of IS members from Twitter. But, as its name suggests, Crisis.Net not just about conflict.

“The World Policy Journal is also building on a map based on data we retrieved for them about the spread of Ebola in West Africa,” says Morgan, “and we’ve also been in discussion with the African Union and some other international NGOs about combining different data for early warning systems that’s more humanitarian focused.”

As with Ushahidi itself, verification is vital, but that is left up to end users. It’s also why Morgan worked with Eliot Higgins, better known online as Brown Moses. Working on the conflict in Syria by monitoring social media sites, Higgins had already managed to put together collect and collate precise information about militant groups in the conflict.

Working with the 1,700 Facebook pages and YouTube channels Higgins monitors and piping them into Crisis.Net, Morgan was able to demonstrate that maps of the conflict derived from information on social media could be as accurate if not as detailed as those produced by the BBC using data manually collected by humanitarian organisations working in the region.

“We’re not trying to be the arbiters of truth or what conflict or crisis data is,” he explains. “There might be interest in highly subjective content from a particular community in order to examine exactly that kind of polarisation. We also try to attach enough metadata so that users can determine whether or not a piece of information is valid for the sort of analysis they want.”

“What’s the right level of abstraction?” he muses. “At Ushahidi in general we work hard to make platforms that make just enough choices to make sure that we’re building tools that are useful for as broad an audience as possible, but without making so many choices that we’re limiting the possibilities of the platform as well.”

Language is another problem.

“Machine translation is not useful for understanding what somebody is trying to say,” he says, “but enough of marker keywords are consistently translated. Things like ‘killing’ or ‘barrel bombs,’ an important one in Syria, are pretty reliably translated and enough to give our algorithms what they need in order to categorise content and understand the names of places or people.”

It also depends on the situation in terms of social media use itself. Nigeria, for example, is a ‘black hole,’ according to Morgan while there’s an abundance of sources in the Middle East.   

One issue still being determined, however, is the interface and whether it should be made drag and drop so that users don’t need any technical expertise to interact with raw data. Crisis.Net is also slated to be incorporated into the third major release of Ushahidi in order to allow its own visualisations to be populated by data derived from the API.

Morgan is also advocating for international organisations to become  more open in their data policies. For now, much of it is locked away and only some data is released publicly. This is a valuable resource in its own right, he explains, but especially when reliable information from social media is absent or completely lacking.

“I’m always surprised by the stories a dataset, or a different combination of datasets, can tell and I don’t think there’s an appreciation of that in those large organisations about the potential power for change,” notes Morgan. “They really have a valuable resource, but they’re just not very receptive to the idea that it can be shared publicly in a way that’s safe.”

Ushahidi hopes that Crisis.Net will encourage others to build their own applications with its API. “The long-term goal of the platform is to make data so accessible that those with beginner to intermediate technical expertise can build their own applications to benefit their communities instead of, say, another app to share photos with friends,” he says.

“I think that people are really attracted to that idea, but have no place to start. There’s just no way for them to do so without having a lot of knowledge about where all that information can come from. I hope we can move people to a higher level of social engagement if we can make the repetitive process of finding and combining information sources easier.”