Microsoft CityNext Big Data Solution Accelerator


Table of Contents

Introduction

The Microsoft CityNext Big Data Solution Accelerator is middleware that hosts big data-enabled smart city solutions on Microsoft’s cloud platform. It aims to solve key big data issues for smart city scenarios, including data ingestion, data model, city analytics, services integration and information dissemination.
001.png
The Microsoft CityNext Big Data Solution Accelerator offers the following key value propositions to cities:
  • The Microsoft CityNext Big Data Solution Accelerator is built on a modular architecture that allows cities to build smart city solutions with minimal development and quick deployment. Building and deploying a big data architecture seems daunting to even the best IT organizations, and “doing more with less” is a key imperative for cities in their budget-constrained environment. Using a modular architecture that consists of five key modules, the Microsoft CityNext Big Data Solution Accelerator effectively reduces the complexity involved in building big data solutions, and can significantly reduce the investment needed while enabling cities to bring innovative new solutions to their citizens.
  • The Microsoft CityNext Big Data Solution Accelerator provides a common framework that allows data and services to be shared across multiple departments and domains. This helps a city to truly utilize the power of big data to perform cross-domain data analysis, and to offer integrated services across the entire set of city functions. The scalability and versatility of the solution accelerator ensures that it can meet the needs of a wide range of city services and handle data in various forms –from operational systems, sensor networks, as well as social media. The cross-domain nature of the solution accelerator enables cross-agency collaboration and helps uncover insights that lead to positive changes to city services.
  • The Microsoft CityNext Big Data Solution Accelerator is built on Microsoft technologies and products that have consistently endured the test of the marketplace; offering the performance, reliability, and security that a smart city seeks. Together with our partners, we aim to develop the best-of-breed big data ecosystem with enhanced solutions for city big data, city analytics, and city service integration. Through customer POCs (proofs-of-concept), the solution accelerator has demonstrated how cities around the world can utilize cloud computing and big data technologies to ingest, manage and analyze data, disseminate information, and offer a wide range of big data powered services effectively.

With this solution accelerator, a variety of mechanisms are specifically orchestrated to control the overall flow of data so that the end-to-end devices and services capabilities are functioning at the highest levels of efficiency. The Microsoft CityNext Big Data Solution Accelerator connects the dots of the flow of data between the cloud, mobile devices, social media and big data. This essentially cultivates an environment which ensures that valued and vital information is constantly available to every important contributor within the overall city infrastructure so that the notion of “one city” is not just an idea, but a living reality.

Significance and Importance - The Microsoft CityNext Big Data Solution Accelerator uses various methods during the data collection process to consolidate city data from the eight city domains in order to store it in databases. No longer do city employees have to worry about the troubling and exhausting process of going through multiple channels in order to access city data to make decisions.
Frequently, city employees need to address issues that affect multiple functions of the city in real-time. In order for city-wide data to flow across an infrastructure, a solution accelerator must be put into place so that the data can be easily accessible to various domains. A variety of constraints, as well as administrative complexity, can be crippling and can hinder a city’s government from being transparent and accessible.
By unlocking value from data, Microsoft CityNext formulates a new way to optimize, standardize, and sustain an underlying innovative solution accelerator that all city functions can use. Whether the data is structured or unstructured, in the size of gigabytes to petabytes, or at rest or in motion, the Microsoft CityNext Big Data Solution Accelerator will manage countless amounts of city data and disseminate that data accordingly.
002.png

Data Flow Process - The Microsoft CityNext Big Data Solution Accelerator is designed to ingest large volumes of city data, current and historical, and also outside data generated from citizens through social media. It is made up of five main components: data ingestion, city artifacts management, city analytics, city services management and info dissemination.
Initially, the data is acquired from streaming or batch data through the sensor network within the city infrastructure -whether it be through messages, databases or other relevant city files- from city services and from citizens through various clouds (private, hybrid or MS public) and social media networks. Once the data is retrieved, it is distributed to either push or pull channels and routed to the formatter so that the data can be consolidated into the operational data store, or sent directly to the real-time analytics runtime manger based on urgency. After the data is stored, it is analyzed and processed within the repository portal in order to determine through which service the data should be published for rendering.
By using the CityNext Analytics Repository Portal, a hub is created that allows developers and researchers the freedom to publish, share and discover city analytics modules and algorithms, and to connect these modules to specified CityNext data; as well as CityNext analytics-related datasets with predefined data models, for future analysis and subsequent rendering. City researchers and developers also have the opportunity to train and develop their modules and algorithms using a batch processing engine with connected CityNext predefined models. They can also develop and implement their city analytics services using asserts from the CityNext analytic repository.
According to the parameters set, the incoming data can provide batch analysis using city analytics modules and algorithms by Hadoop for unstructured data, and by a pre-built BI data warehouse for structured data with a batch processing engine. Real-time analysis using built-in CEP modules and algorithms through framework based on city streaming data with a real-time processing engine can also be delivered.
Subsequently, in order for the data to be further distributed, based on the nature of the content, the data is rerouted to either the batch-based or real-time analytics runtime manager. Finally, through the service determined by the analysis, data is disseminated through web coverage, a data portal or notification services. The notification services are primarily rendered via notifications and alerts, while web coverage is released mostly based on browsing and queries.

Back to top

The value of the Microsoft CityNext Big Data Solution Accelerator

003.png
The necessity and importance of having a big data platform is clear and unmistakably evident: it is the engine that powers the initiative towards building a smart city. Without a big data platform, a city would be severely limited in what level of services it offers to its citizens and how creatively it utilizes the available data to fully unleash the power of big data solutions. By using the Microsoft CityNext Big Data Solution Accelerator to supplement a big data platform, unlocking the power to generate big data solutions can be realized. To better understand how the solution accelerator can enhance a big data platform, a deeper look into the components of the solution accelerator can help provide some clarification.

In order for the data to be used, the Data Ingestion component interacts with external and internal data sources to ensure that diverse data be ingested into the solution accelerator. With using the communication channels and data formats that the solution accelerator supports, it also provides a flexible framework allowing 3rd party developers to develop additional channels and data formats according to different cities’ demands.

The main data storage of the solution accelerator can be found within the City Artifacts Management component. This particular component supports data enrichment, discovery and aggregation; which is essential for indexing and retrieval, as well as providing OData response of both metadata and result sets with publisher/subscriber (pub/sub) capabilities.

Another facet of the Microsoft CityNext Big Data Solution Accelerator is that it can support both real-time and batch-based data. Besides attaining, ingesting and managing city data, the solution accelerator has another component with the unique ability to analyze the stored data to provide and relay readily available solutions, as well as give developers consistent development experience when analyzing data on top of the Microsoft CityNext Big Data Solution Accelerator for more business insights.

Just as vital to the solution accelerator is City Services Management. Service lifecycle management (CSM) enables city operators to manage the big data empowered services registration and release processes. As a result, this component powers various applications in the city so that a collection of services are able to provide essential runtime brokering capability to route the service requests, search the catalog, and utilize services to speed up application development and allow the sampling of smart city services for the city to use.

The final component is equally as indispensable as to why the Microsoft CityNext Big Data Solution Accelerator is necessary in helping a big data platform create a smart city. With the massive amount of incoming data being retrieved, stored and managed and analyzed, it must subsequently be rendered in some present-day fashion. It is here, where the Information Dissemination component shines just as brightly as the others. Information Dissemination (ID) enables the 3rd party to disseminate data through push and pull channels with multiple types of adapters. It also enables a government to control the data that is disseminated and validates the dissemination request from the 3rd party, ultimately ensuring that the data is disseminated to the intentionally targeted users, via the correct channels, at the right time.


Back to top

How can the solution accelerator assist a big data platform in making a smart city?

004.png
To be a smart city, a city must have a people-first approach, use modern technology and methods to transform in a new age -which will decisively enable engagement between city and citizen- and accelerate innovation and opportunity. The Microsoft CityNext Big Data Solution Accelerator will lead your city towards this path and guide you on the journey.

How, what kind, and how much city data is retrieved is significant in the makings of a smart city. With the Microsoft CityNext Big Data Solution Accelerator, all ingested data is parsed into a key-value pair and stored into permanent ODS storage with extremely high performance. The solution accelerator supports Web HTTP/HTTPS, MS SQL Database, Websphere MQ, and SMB File Share as the pull channels, and HTTP POST as the push channel. Also, the supported data formats include XML, CSV, JSON, and XLS. This ensures that the type of data acquired is not limited to a small variety of data sources, nor the format in which the data comes in.

The capability of the solution accelerator to handle a variety of data and data models is a key to how expandable and scalable it can be. The Microsoft CityNext Big Data Solution Accelerator manages data using NoSQL storage. This NoSQL storage features optimized entity-attribute-value (EAV) schema for HBase with multi-cluster supported indexing and retrieval. The solution accelerator also includes a data warehouse with pre-defined multi-dimension and tabular dimensioned schema based on the Urban Data Model.

Being restricted to analyzing just batch-based data can limit a city’s effectiveness. The Microsoft CityNext Big Data Solution Accelerator provides analytics runtime job management that supports either real-time Complex Event Processing (CEP) jobs, batch-based Win32 or Hadoop jobs for data analysis. The solution accelerator also provides a PowerShell-based Insight Catalog service to create and manage the analytics jobs. The Data Publication Service (DPS) publishes the results from the analytics jobs, Data Warehouse, and OLAP cubes to other developers for further consumption.

Accessibility to data, service information and analytics plays a huge role as to how a city responds to citizens’ needs, optimize services and offer solutions to predictable issues. The Microsoft CityNext Big Data Solution Accelerator is in touch with current-day accessibility standards by allowing service access usage and data access usage with using Live ID, FB, Yahoo, and Gmail for external access and AD for federated login. The solution accelerator also provides tools for the city to create a service model which describes the interfaces, categorizations, and relationships between data/services that provides advanced discovery capability and business opportunities to the developers.

It is impossible for a city to be fully engaged with its citizens without having open channels that encourage and provide constant communication. The Microsoft CityNext Big Data Solution Accelerator enables third parties to disseminate data through push and pull channels with multiple types of adapters including Signal-R, Windows 8, Windows Phone 8, eMail and Windows 8 Live Tile.


Back to top

For more information about the solution accelerator's architecture and design, please refer to the High Level Design document on the download page.

Last edited Jul 15, 2014 at 7:30 AM by gheadd, version 22