Leveraging Google App Engine for Humanitarian Project Monitoring

Bedatadriven is a company that helps clients leverage data and analysis to achieve their goals with software development, consulting, and training. One of their core projects is ActivityInfo, a database platform for humanitarian relief operations and development assistance. Originally developed for UNICEF’s emergency program in eastern Congo, today the system is used by over 75 organizations working in Africa and Asia, tracking relief and development activities, across more than 10,000 project sites. With ActivityInfo, project managers can quickly establish an online database that reports the results of educational projects, maps activities that improve water and hygiene, tracks the delivery of equipment to clinics or any other humanitarian activities a project undertakes.

Bedatadriven, a company that leverages data and analysis to achieve their clients' goals, was facing challenges with their core project, ActivityInfo. ActivityInfo is a database platform for humanitarian relief operations and development assistance, used by over 75 organizations across Africa and Asia. The platform was initially developed for UNICEF’s emergency program in eastern Congo and is now used to track relief and development activities across more than 10,000 project sites. The challenge was to ensure that the system was highly available, given the challenging environments that ActivityInfo users work in and the nature of the crises. They also needed minimal system administration, allowing Bedatadriven to focus on product development. The platform also needed to scale up and down according to the load, with minimal human intervention. They also required clear monitoring tools to help pinpoint performance problems.

Bedatadriven chose to use Google App Engine and Google Cloud SQL for ActivityInfo. Google App Engine provided the high availability they needed, with minimal system administration. It also scaled up and down with the load appropriately, without having to monkey with configuration or specify triggers through trial and error. New instances came online to serve requests in under 30 seconds, keeping request latency low even when they experienced very sudden spikes in utilization. Google App Engine also provided strong monitoring tools that helped them quickly find and eliminate performance bottlenecks. It collected logs from all running instances in near real time and had a clean interface that allowed them to review and search logs, aggregated by request. This allowed them to flag all requests that exceed a certain latency and drill down to the causes very quickly. Google Cloud SQL service was also used, which helped them pinpoint the MySQL queries that needed tuning.

• The move to Google App Engine and Google Cloud SQL has proven to be successful for Bedatadriven. The high availability, scalability, and strong monitoring tools provided by these platforms have improved the quality of service to their users and allowed them to focus on software development. The limitations imposed by App Engine, such as the 30-second request limit, have turned out to be beneficial, as they shut down problematic requests after thirty seconds and flagged them in the logs, allowing other requests to complete normally and giving them time to optimize the queries. This has resulted in ActivityInfo running better than ever before, with less time spent on server administration and more time spent on delivering practical aid and support to vulnerable people and communities.

• New instances come online to serve requests in under 30 seconds, keeping request latency low even during sudden spikes in utilization.

• ActivityInfo is now used by over 75 organizations across Africa and Asia, tracking relief and development activities across more than 10,000 project sites.

• Google App Engine collects logs from all running instances in near real time, allowing for quick identification and resolution of performance bottlenecks.