Big Data solution

Objective

The project was commissioned by the regulatory authority of a city with a smart bus card system. At the time of project implementation there was no way to proactively determine when a commuter had been charged a wrong fare because of faulty bus hardware or manual intervention by the bus captain.  A lengthy investigation process was required to validate claims made by passengers about possible wrong fare occurrences.

The objective of this project was to devise an automated system to identify potential wrong fare occurrences and emerging fault trends in buses.

Solution

The project involved handling very large data volumes as there were more 15 million transactions per day translating to more than 5 billion historical transactions per year to be processed hence the decision to adopt Big Data technology. The solution stack included:

  • Hadoop for data storage and processing and MySQL for rapid reporting
  • Hive and NoSQL as query tools
  • Java and R Map Reduce for wrong fare detection algorithms
  • Pentaho for dashboards and reporting.

Benefits

  • Automated detection of wrong fare incidents and flagging of affected commuters’ cards.
  • Early warning of emerging fault trends in buses and other fare related equipment or operations, so that corrective action could be taken in a timely manner.