Introducing Data Models for Human(itarian) Services

Source: Introducing Data Models for Human(itarian) Services

9377515138_892d7834af_z-300x200Immediately after a disaster, information managers collect information about who is doing what, where, and turn it into “3W Reports.” While some groups have custom software for collecting this information, the most popular software tool for this work is the spreadsheet. Indeed, the spreadsheet is still the “lingua franca” of the humanitarian aid community, which is why UNOCHA’s Humanitarian Data Exchange project is designed to support people using this popular software tool.

After those critical first few days, nonprofits and government agencies often transition their efforts from ad hoc emergency relief and begin to provide more consistent “services” to the affected population.

The challenge of organizing this type of “humanitarian/human services” information is a bit different than the challenges associated with disaster-related 3W reports, and similar to the work being done by people who manage and maintain persistent nonprofit services directories. In the US, these types of providers are often called “211” because you can dial “211” in many communities in the US to be connected to a call center with access to a directory of local nonprofit service information.

During the ongoing migrant crisis facing Europe, a number of volunteer technical communities (VTCs)  in the Digital Humanitarian Network engaged in the work of managing data about these humanitarian services. They quickly realized they needed to come up with a shared template for this information so they could more easily merge data with their peers, and also so that during the next disaster, they didn’t have to reinvent the wheel all over again.

Since spreadsheets are the most popular information management tool, the group decided to focus on creating a standard set of column headers for spreadsheets with the following criteria:

To create this shared data model, we analyzed a number of existing service data models, including:

  • Stand By Task Force’s services spreadsheet
  • Advisor.UNHCR services directory
  • Open Referral Human Service Data Standard (HSDS)

The first two data models came from the humanitarian sector and were relatively simple and easy to analyze. The third, Open Referral, comes from a US-based nonprofit service directory project that did not assume that spreadsheets would be an important medium for sharing and viewing data.

To effectively incorporate Open Referral into our analysis, we had to convert it into something that could be viewed in a single sheet of a spreadsheet (we call it “flat”). During the process we also made it compliant with the Humanitarian Exchange Language (HXL), which will enable Open Referral to collaborate more with the international humanitarian aid community on data standards work. Check out the Open Referral HSDS_flat sheet to see the work product.

We’re excited about the possibility that Open Referral will take this “flat” version under their wing and maintain it going forward.

Once we had a flat version of Open Referral, we could do some basic analysis of the three models to create a shared data model. You can learn about our process in our post “10 Steps to Create a Shared Data Model with Spreadsheets.”

The results of that work is what we’re calling the Humanitarian Service Data Model (HSDM). The following documents and resources (hopefully) make it useful to you and your organizations.

We hope the HSDM will be used by the various stakeholders who were involved in the process of making it, as well as other groups that routinely manage this type of data, such as:

  • member organizations of the Digital Humanitarian Network
  • grassroots groups that come together to collate information after disasters
  • big institutions like UNOCHA who maintain services datasets
  • software developers who make apps to organize and display service information

I hope that the community that came together to create the HSDM will continue to work together to create a taxonomy for #service+type (what the service does) and #service+eligibility (who the service is for). If and when that work is completed, digital humanitarians will be able to more easily create and share critical information about services available to people in need.


* Photo credits: John Englart (Takver)/Flickr CC-by-SA


Creating a Shared Data Model with a Spreadsheet

Source: Creating a Shared Data Model with a Spreadsheet

Over the last year, a number of clients have tasked me with bringing datasets from many different sources together. It seems many people and groups want to work more closely with their peers to  not only share and merge data resources, but to also work with them to arrive at a “shared data model” that they can all use to manage data in compatible ways going forward.

Since spreadsheets are, by far, the most popular data collection and management tool, using spreadsheets for this type of work is a no-brainer.

After doing this task a few times, I’ve gotten confident enough to document my process for taking a bunch of different spreadsheet data models and turning them in a single shared one.

Here is the 10-step process:

  1. Create a spreadsheet. First column is for field labels. You can add additional columns for other information you’d like to analyze about the field such as its data type, database name and/or reference taxonomies (i.e. HXL Tag).
  2. Place the names of the data models you’ve selected to analyze in the column headers to the right of the field labels.
  3. List all the fields of the longest data model on the left side of the sheet under the “Field Label” heading.
  4. Place an “x” in the cells of the data model that contain the field to indicate it contains all the fields documented in the left hand column.
  5. Working left to right, place an  “x” to indicate when a data model has a field label contained therein. If the data model has that field but uses a different label, place that label in the cell(4a). If it doesn’t have that field, leave the cell blank. Add any additional fields not in the first data model to the bottom of the Field Labels column (4b).

    This is a sheet comparing three different data models with a set of field labels and a “taxonomy convention”.
  6. Do the same thing for the next data models.
  7. Once you have all the data models documented in this way, then you can look and see what the most popular fields are by seeing which have the most “x”s. Drag those rows to the top, so the most popular fields are on the top, and the least popular fields are on the bottom. I like to color code them, so the most popular fields are one color (green), the moderately popular ones are another (yellow) and the least popular but still repeated fields are another (red).
  8. Once you have done all this, you should present it to your stakeholder community and ask them for feedback. Some good questions are: (a) If our data model were just the colored fields, would that be sufficient? Why or why not? What fields should we add or subtract? (b) Data model #1 uses label x for a field while data model #2 uses label y. What label should we use for this and why?
  9. Once people start engaging with these questions, layout the emerging data model in a new sheet, horizontally in the first row. Call this sheet a “draft template”. Bring the color coding with it to make it easier for people to recognize that the models are the same. As people give feedback, make the changes to the “template” sheet while leaving the “comparison” sheet as a reference. Encourage people to make their comment directly in the cell they’re referencing.
  10. Once all comments have been addresses and everyone is feeling good about the template sheet, announce that sheet is the “official proposal” of a shared data model/standard. Give people a deadline to make their comments and requests for changes. If no comments/changes are requested – congratulations: you have created a shared data model! Good luck getting people to use it. 😉

Do you find yourself creating shared data models? Do you have other processes for making them? Did you try out this process and have some feedback? Is this documentation clear? Tell me what you’re thinking in the comments below.


2015 in Review: who’s doing what, where and why

Source: 2015 in Review: who’s doing what, where and why

OpenReferral_Logo_GreenWe recently posted Open Referral’s 2015 Year in Review report. [You can view and download the document here.]

In the last blog post, we discussed the different technological products that have emerged through Open Referral.

In this post, we’ll discuss the different projects in which people are using these tools to find new ways to share and use information about the health, human, and social services available to people in need.


The community resource directory data problem is a global one: pretty much everywhere one might look, there are people struggling to keep track of which institutions provide what services to whom.

Yet each community has its own institutional, demographic, and cultural landscape. Any or all of these unique characteristics might determine the viability of various answers to the key question at the core of our work:

“How can we ensure that community resource data can be made openly accessible in a reliable and sustainable way?”

In Open Referral we believe this means that our ‘global’ problem needs, to some degree, local solutions. So this initiative is designed around the interests of people on the proverbial ‘ground,’ working together in their own communities. In pilot projects, lead stakeholders use our data format and associated tools to pursue their own objectives. As they evaluate their own progress, their insights will guide the iterative development of our specification and our tools.


Code for San Francisco's Open Referral team designing the new resource data editing system at the National Day of Civic Hacking

Code for San Francisco’s Open Referral team designing the new resource data editing system at the National Day of Civic Hacking

Progress to date: a diversity of communities

Our initial pilot projects — San Francisco and DC — have seen mixed results in their first year.

Key institutions have made ambitious pledges, and focused projects have emerged with transformative potential. And in one instance (see below: ‘A Success Story in DC’) a lead stakeholder achieved a significant local victory, in large part through smarter use of resource data.


A drawing from DC Open211: a vision of an open resource data ecosystem

However, these pilots have yet to fully cohere around formal teams with clearly defined processes. (In part, this is because pilots acquired funding in minor project-specific grants, which has limited the scope and rhythm of work accordingly.) Additional funding will likely be necessary to support stakeholders and drive the development of infrastructure that serves them all.

In the meantime, innovators in a number of other communities have joined up with Open Referral, taking steps to build open, interoperable resource directory data ecosystems — and demonstrating exciting progress along the way.


Ontario: a resource data federation

ontario-logoOntario’s information-and-referral sector presents a remarkable precedent of a ‘federated’ resource data system. Across the province, dozens of local referral providers have integrated with each other in a network that is stewarded by Ontario 211. Local providers update their own records in their own databases, and Ontario 211 aggregates these updates into a province-wide system.

xiJr1nqoNow, Ontario 211 is using the HSDS to develop an open data platform that will make resource data from across the province easier to access for third-parties. Regional advocacy organizations such as PoweredByData and the Ontario Non-Profit Network are developing strategic plans around the use of this open resource data. And other local technology providers such as the Community Information Online Consortium are adapting their software in accordance with HSDS to preserve interoperability across this ecosystem.


Boston: transforming a website into a platform is a resource referral application maintained by the Boston Children’s Hospital. Helpsteps offers both a basic service search and also a brief survey that screens users for factors that can yield tailored recommendation results. Yet this valuable resource data can be used in many more ways.

So the Helpsteps team is using the Ohana API to redeploy this software, improving its internal flexibility while offering up its data data for new uses to the Boston community at large.


Chicago: an emerging resource data ecosystem

Screen-Shot-2015-01-21-at-3.32.23-PMPurple Binder, a for-profit company with a premium product for social workers and other service providers, now offers their resource data for use by third parties, by way of their own API which was modeled upon HSDS and the Ohana API.

Purple Binder first deployed the open-source Ohana Web Search as a public-facing website that enables anyone to search for services in their covered communities. Now, several third party applications — designed for quite different purposes — are able to query Purple Binder’s API to deliver its data to their users. It’s our first glimpse of an interoperable ecosystem of various apps sharing the same data.


A success story in DC: resource data used for justice

bread-for-the-city-logoWe’ve also seen that resource data isn’t just useful to help people find where to go to get help. It’s also critical for assessing the effectiveness of our safety net as a whole — and imagining systemic improvements to it.

Towards this end, the community champion of the DC Open211 project — Bread for the City — has already demonstrated a small but illuminating data-driven success story. As Bread described on their blog — and which was also reported upon at the Huffington Post — by enabling smarter use of their day-to-day use of resource directory data, Bread for the City’s social workers were also able to advocate for tangible improvements in food accessibility for thousands of DC residents. The result is a victory that Bread for the City describes as ‘racial justice.’


Evaluating this progress

On a personal note, this work has been a great privilege. I am grateful for the opportunity to learn from a tremendous range of leaders who’ve joined the Open Referral network — government officials, data scientists, civic technologists, non-profit executives, community leaders, and more. We’ve only had the opportunity to blog about some of their inspiring work, and I’m looking forward to many stories yet to come.

That said, I also want to acknowledge one of the biggest challenges in this process: it’s hard to consistently “work in the open,” especially when engaging so many different kinds of stakeholders, separated by various kinds of distance, and operating under various kinds of institutional constraints. Projects in this domain face a fair amount of uncertainty, often with few direct precedents for success. All of this makes it hard to strike the right balance between transparency and discretion.

Members of our network have been able to participate in all deliberations and decisions about Open Referral itself. However, much deliberation around specific partnerships and projects has happened out of view. Along the way, we’ve often allowed extended periods of silence in Open Referral’s public channels of communication.

We also did not allocate time and resources to gather various parts of our network together for a major in-person event, as we did in 2014.

In 2016, we should rise to meet this need for more regular communication, including at least one large in-person gathering.


More broadly, this kind of work — aligning many stakeholders around shared goals, building infrastructure, facilitating learning and deliberation — is resource intensive and often unfamiliar to conventional funders. Almost every civic data standardization initiative (such as Open311, Open Contracting Data Standard, BLDS, etc) seems to share a need for resources to support research, development, and implementation.

There has been some promising discussion of cooperation across various civic data standardization initiatives. Yet we also recognize a need to educate funders, governments, and other civic leaders around the costs and benefits of open standardization. If you’re interested in sponsoring this work, and/or exploring opportunities to engage in such education, please be in touch.


Moving forward: what’s next?

In our next blog post, we’ll explore specific ways to move this work forward through the year(s) to come.

Meanwhile, we’re still eager to collect feedback of various kinds. If we haven’t yet been in touch, and you have observations, suggestions, or questions, we want to hear from you.

And if you are interested in starting a pilot project in your community, we also want to hear from you!

Please leave a comment on this blog post — or email us at

Many thanks again to all who have been engaged in re-imagining a safety net for the 21st century.

The building blocks of an open ecosystem

Source: 2015 in Review: the building blocks of an open ecosystem

Last week we posted Open Referral’s 2015 Year in Review report. [You can view and download the document here.] We’ll unpack the key parts of the report here on the blog. This post will cover our technological accomplishments and feedback; the next post will cover our projects in the field; and a final post will consider the path ahead.

A world in which information about community resources is easy for anyone to find, trust, and effectively use — in whatever way works best for them. This is Open Referral’s hopeful vision of the future.

In 2015, we saw the first glimmers of such a world. Let’s take a look:

We published the Human Services Data Specification

DC_Open_211_pdf__page_7_of_10_Drafted by Sophia Parafina, through nine iterations of public comment and testing, the Human Services Data Specification (HSDS) is an open format that enables the exchange of machine-readable directory data about health, human, and social services among different kinds of information systems — including conventional calling center systems, emerging user-centered applications, and the major web platforms such as Google, Facebook, Yelp, etc.

HSDS v1.0 was published in March 2015, and is already under adoption by a range of referral software providers, from Purple Binder to iCarol. HSDS is even being adapted for the specific needs of particular subdomains, such as legal services (through the leadership of LegalServer) to emergency response services (through the leadership of Sarapis).

The Ohana Project delivered an open source directory platform

Ohana APIIn March of 2015, the Ohana Project formally concluded the work that was funded by the Knight Foundation’s 2013 Health Data Challenge.

Ohana’s products include the Ohana API, which enables any resource directory to be transformed into an open platform, and the Ohana Web Search, a free open source mobile-friendly front-end application that enables simple searching for services.

Ohana has been redeployed by a number of initiatives, from the Boston Children’s Hospital to NYC:Prepared, to the SociSalud resource locator (launched by Medialab-Prado with data from Madrid’s open data portal).

We fostered an emerging ecosystem of open source tools

Link-SF offers a deliberately simple, mobile-friendly, visually-aided browsing experience.

Link-SF offers a deliberately simple, mobile-friendly, visually-aided browsing experience.

Taken together, Ohana and HSDS offer up the elemental building blocks of an open ecosystem of interoperable information systems. Over the course of the year, we saw that ecosystem start to take shape.

For example, Link-SF is a mobile resource locator initially developed by Zendesk for St Anthony’s in San Francisco, and is now being adapted to ‘speak’ the HSDS format and read from APIs such as Ohana. It has already been redeployed in other localities such as Queens.

In D.C., Social Impact Lab — in partnership with the DC Public Library — has prototyped a ‘Logic Library’ that enables users to build screening toolkits which can help people determine which services might be relevant and available to them.


Evaluating our progress

Open Referral is engaged in an iterative process of experimentation and discovery — like a laboratory. Over the course of this past year, we’ve gathered a considerable amount of feedback about what’s working and what can be improved.

HSDS: technically apt, with room to grow

The good: We’ve received positive feedback about HSDS from technical partners at enterprise software vendors (especially regarding HSDS’s approach to normalizing the relationship between services, sites, and organizational entities). Several vendors are using it as a model for structuring new products, in addition to facilitating data exchange.

Needs improvement: Feedback indicates that HSDS falls short of one of our initial principles of simplicity. The specification is daunting for relatively non-technical users (who, for example, shouldn’t be expected to be familiar with JSON datapackages, foreign keys, etc). For example, managing HSDS-compliant resource data in a spreadsheet would be prohibitively difficult unless additional tools are developed to ease the process. Additional issues regarding HSDS have been logged in the HSDS Github repo.

What we can change: In future iterations, we might consider a dual approach to formatting flatstructured data (a precedent that has already been set by the Open Contracting Data Standard). I’ve also proposed adopting a ‘Share Alike’ license that will encourage adaptation, while preserving the open source nature of new contributions. [Read more about this licensing proposal here.] Note that future development of HSDS is pending the re-establishment of technical leadership. In the meantime, we encourage users to adapt as needed, while documenting your work and reporting back to the community.

Ohana: transform any resource directory into an open platform

The good: We’ve been told that Ohana (the API and associated tools) is easy enough to redeploy and, for the most part, developers have approved of the code and documentation. Several sites have been able to be deployed with pre-existing data presented in new and more effective ways. In situations where resource directory data is already being diligently maintained, then Ohana can be useful in exposing that data to more tools and people.

Needs improvement: We have received ample feedback that the current version of Ohana doesn’t address the challenges of directory data maintenance. Ohana’s basic admin interface is challenging for users, especially those who are tasked with understanding and improving the quality of records which may be out-of-date, incomplete or otherwise inaccurate. Users have articulated the need for feedback loops from external apps to enable users to comment on data accuracy, as well as workflow tools to make record verification as easy as possible. Additional requests for features have been logged here.

What we can change: Ohana could evolve to meet these needs, or something new could emerge to take its place. This is up for us to decide. Like almost all open source civic tech projects, Ohana’s future will depend upon the value it brings to users. We invite those who have deployed Ohana and/or have experience with it to join us in this deliberation.


Evaluating our Process

The good: We conducted an iterative, public comment process that was driven by hands-on research in partnership with stakeholders in our pilot projects. This process received generally positive feedback, especially in dealing with key points of disagreement along the way. (Most prominently: approval of HSDS v1.0 hinged upon an architectural decision that ignited a month-long debate about data normalization vs denormalization. This debate underscored the tensions inherent in any effort to construct a useful model out of a complex reality, as well as technical tradeoffs between efficiency of systems and granularity of information, etc.) Our objective was not to ascertain The Truth so much as to ensure that we heard a range of perspectives, prioritized the interests of end users, and sought an outcome which would at least be acceptable to everyone involved.

Needs improvement: Quite a few participants observed that the workflow of conversations was confusingly fragmented between our community listserv, Google Docs, and Github; this sometimes made discussion around particular issues hard to follow. The commenting feature in Google Docs, in particular, proved unwieldy especially for managing complex discussions and for recalling previous conversations.

What we can change: We should be able to streamline the commenting process into a single collaboration tool (such as Github); however, we should also ensure that such a tool is made accessible to non-technical users, through custom interfaces and/or training.

We’re still learning!

Your feedback is essential to this process. Do you have insights to share?

Would you like to participate in the next iteration of the Open Referral workgroup?

Might you or your institution be interested in sponsoring this initiative?

Please be in touch.

Meanwhile, stay tuned for the next post — about various projects that are implementing these technologies.

Great video of Disaster Recovery efforts in Napal using UAVs for mapping


I had the honor of spearheading this disaster recovery UAV mission in Nepal a few weeks ago as part of Kathmandu Flying Labs. I’ve been working on this new initiative (in my own time) with Kathmandu Living Labs (KLL), Kathmandu University (KU), DJI and Pix4D. This Flying Lab is the first of several local UAV innovation labs that I am setting up (in my personal capacity and during my holiday time) with friends and colleagues in disaster-prone countries around the world. The short film documentary above was launched just minutes ago by DJI and describes how we teamed up with local partners in Kathmandu to make use of aerial robotics (UAVs) to map Nepal’s recovery efforts.

Here are some of the 3D results, courtesy of Pix4D (click to enlarge):


Screen Shot 2015-11-02 at 5.17.58 PM

Why work in 3D? Because disaster damage is a 3D phenomenon. This newfound ability to work in 3D has important implications for Digital Humanitarians. To be sure, the analysis of these 3D models could potentially be crowdsourced and eventually analyzed entirely within a Virtual Reality environment.

Since most of our local partners in Nepal don’t have easy access to computers or VR headsets, I found another way to unlock and liberate this digital data by printing our high-resolution maps on large, rollable banners.



We brought these banner maps back to the local community and invited them to hack the map. How? Directly, by physically adding their local knowledge to the map; knowledge about the location of debris, temporary shelters, drinking water and lots more. We brought tape and color-coded paper with us to code this knowledge so that the community could annotate the map themselves.


In other words, we crowdsourced a crisis map of Panga, which was highly participatory. The result was a rich, contextual social layer on top of the base map, which further inform community discussions on strategies and priorities guiding their recovery efforts. For the first time ever, the community of Panga was working off the one and same dataset to inform their rebuilding. In short, our humanitarian mission combined aerial robotics, computer vision, water-proof banners, local knowledge, tape, paper and crowdsourcing to engage local communities on the reconstruction process.


I’m now spending my evenings & weekends working with friends and colleagues to plan a follow-up mission in early 2016. We’ll be returning to Kathmandu Flying Labs with new technology partners to train our local partners on how to use fixed-wing UAVs for large scale mapping efforts. In the meantime, we’re also exploring the possibility of co-creating Jakarta Flying Labs, Monrovia Flying Labs and Santiago Flying Labs in 2016.

I’m quitting my day job next week to devote myself full time to these efforts. Fact is, I’ve been using all of my free time (meaning evenings, weekends and many, many weeks of holiday time) to pursue my passion in aid robotics and to carry out volunteer-based UAV missions like the one in Nepal. I’ve also used holiday time (and my own savings) to travel across the globe to present this volunteer-work at high-profile events, such as the 2015 Web Summit here in Dublin where the DJI film documentary was just publicly launched.


My Nepali friends & I need your help to make sure that Kathmandu Flying Labs take-off and become a thriving and sustainable center of social entrepreneur-ship. To this end, we’re actively looking for both partners and sponsors to make all this happen, so please do get in touch if you share our vision. And if you’d like to learn more about how UAVs other emerging technologies are changing the face of humanitarian action, then check out my new book Digital Humanitarians.

In the meantime, big, big thanks to our Nepali partners and technology partners for making our good work in Kathmandu possible!

Source: Great video of Disaster Recovery efforts in Napal using UAVs for mapping

PressForward 3.6 Release

The PressForward team is excited to announce the release of PressForward 3.6, an update to our WordPress plugin that helps you collect, discuss, and share content from the web.  This update increases the available methods of aggregating and organizing content with the introduction of new OPML (Outbound Processor Markup Language) features.

OPML files allow users to automatically exchange lists of RSS / Atom feeds, like blogrolls, and OPML file subscriptions allow followers to receive automatic updates to that list. Now, with PressForward 3.6, users can import OPML files with associated tags, subscribe to an OPML link, and generate outbound OPML files for others to follow.  OPML file subscription improves PressForward administrators’ ability to manage their feed lists by automatically detecting additions or changes made to an OPML without requiring an update by the PressForward administrator. When administrators import an OPML file, it will recognize feed tags and create folders for them in the PressForward All Content area, improving users’ ability to organize and find available content.

By allowing administrators to publish their subscription list as an OPML link, PressForward publications can more easily share and update their feed lists and folders, making changes automatically visible to followers.

In addition to these new features, 3.6 also includes numerous bug fixes and small enhancements. See below for a full listing of all features, enhancements, and bug fixes, or you can check out the GitHub milestone for more information.

Props to open-source developers who contributed to this milestone: James DiGioia and Douglas Linsmeyer.


New Features

Added the ability to subscribe to an OPML file.
OPML subscription imports any folders included in the OPML file.
Added preference field for minutes between feed retrieval cycles.
Added an OPML file link to Tools menu. Allows administrators to share their subscription list as an OPML file.
Users can select what post type and status nominations take on when they are sent to the next step.
User can subscribe to a site from the Nominate This bookmarklet.
If the site gives the META author property, that value will now automatically populate in the Nominate This bookmarklet.


Added edit link to the success/fail message when a feed is added.
Added an in browser screen that indicates loading when a large OPML is uploaded.
Added Folders and sorting by folders and feeds to the “Nominated” panel.
Layout enhancements to the Add Feeds and Tools panels.

Bug Fixes

Fixed bug where alerts weren’t closed upon successful feed retrieval.
Fixed bug causing “Nominated” usernames to repeat.
‘Save Inactive’ in the “Edit Feed” page no longer saves the feed as ‘Pending’.
When sorting by ‘Last Time Feed Checked’ results now include unchecked feeds.
Fixed in-modal comments in Nominated panel.
Fixed bug where the Open Graph image was not being pulled in through nomination process.
Fixed bug related to insure PressForward properly hooks to the Yoast SEO plugin’s Open Graph and Canonical values.
Fixed bug where the author custom field was no longer overriding WordPress author.
Fixed bug where clean-up of old posts after user-set period was not cleaning up posts.

Source: PressForward 3.6 Release