Building a data pipeline for the management and processing of urban data streams

Elarbi Badidi, Nouf El Neyadi, Meera Al Saeedi, Fatima Al Kaabi, Muthucumaru Maheswaran

Research output: Chapter in Book/Report/Conference proceedingChapter

2 Citations (Scopus)


Urban data streams (UDS) originate from various sensors and Internet of Things (IoT) devices deployed in smart cities as well as social media sources such as Twitter and Facebook. The large volumes of urban data need to be harnessed to help smart city stakeholders and applications make informed decisions on the fly. Furthermore, effective management and governance of smart city components relies on the ability to integrate and federate their data, process urban data streams locally, and use big data analytics. Data integration and interoperability is a challenging problem that smart cities are facing today. Successful data integration is crucial for improved services and governance. This chapter describes a framework that aims to serve in building a data pipeline for the acquisition and processing of urban data streams, urban data analytics, and creation of value-added services. The framework relies on latest technologies for data processing including IoT, edge computing, data integration techniques, cloud computing, and data analytics. The proposed platform will facilitate real-time event detection, notification of alerts, mining the opinions of citizens regarding the governance of their city, and building monitoring dashboards. A prototype of the platform is being implemented using the Kafka messaging platform.

Original languageEnglish
Title of host publicationHandbook of Smart Cities
Subtitle of host publicationSoftware Services and Cyber Infrastructure
PublisherSpringer International Publishing
Number of pages17
ISBN (Electronic)9783319972718
ISBN (Print)9783319972701
Publication statusPublished - Nov 15 2018


  • Data integration
  • Data interoperability
  • Data streams processing
  • Internet of things
  • Messaging queue
  • Smart cities

ASJC Scopus subject areas

  • General Computer Science
  • General Engineering


Dive into the research topics of 'Building a data pipeline for the management and processing of urban data streams'. Together they form a unique fingerprint.

Cite this